Use Rust To Scale back The Measurement Of Your SQLite Database

Nirina Zubir

Meet sqlite-zstd, a Rust library that compresses your database many fold, resulting in nice financial savings in dimension whereas conserving its search capabilities intact.

As identified in “In Reward Of SQLite”, SQLite will not be a toy database:

Regardless of its compact dimension and absence of the consumer server mannequin, SQLite is a RDBMS with all of the options that make one thing relational – that’s tables, indexes, constraints, triggers, transactions and the like. Nonetheless, there are few built-in features in comparison with PostgreSQL or Oracle. 

SQLite doesn’t have any compression options. This has modified with the Rust-based library, sqlite-zstd which guarantees to:

present clear dictionary-based row-level compression that permits compressing a sqlite database’s entries nearly in addition to in case you have been compressing the entire DB file, whereas on the identical time retaining random entry.

Suffice it to say that we’re all conscious of the advantages of compressing knowledge, be it that of a PDF doc, a humble ZIP file or on this case a database. As a proof of idea and walkthrough of the instrument, I will work with a pattern database utilized by the Joplin observe taking app. Nicely work with Home windows as a result of it presents a extra simple expertise.

The pattern database.sqlite is 2.6 GB in dimension.Some would say ‘what sort of notes do you’ve that occupy that a lot area?’. It’s true that that quantity sounds extravagant. It’s like that as a result of Joplin permits you to scrape any net web page you encounter and retailer it as markdown contained in the database. Since I am an individual with a wide selection of pursuits, after I discover one thing attention-grabbing I make a copy of it. As such the variety of pages I dump into Joplin accumulates and the dimensions of the database will increase. Subsequently sqlite-zstd was a godsend. After all your use case is likely to be completely different; the library’s foremost use case instance on-line showcases compressing a database with 7 million e book entries in JSON format, and does so by decreasing its dimension from 2.2GB to 550KB!

However so as to get to that, the primary hurdle was to search out the 64-bit model of sqlite’s CLI, sqlite3, for the reason that official construct is obtainable solely in 32-bit, and since zstd is a 64-bit library it wanted the corresponding model. When you can construct it manually, why go to that bother when somebody has already finished it for you? Jump over to the SQLite shell builder Github repo and obtain the most recent 64-bit launch for Home windows (Ubuntu, MacOS as properly).

Having bought maintain of the CLI it is time to execute it to load each the library and pattern database. After that we are going to allow the clear row-level compression of the ‘physique’ column, the column that retains the majority of the textual content of the ‘notes’ desk. Saying that,you’ll be able to invoke the clear row-level compression perform a number of instances on the identical desk with completely different columns to compress.

You name it like this:

zstd_enable_transparent('{"desk": "objects", "column": "data1", "compression_level": 19, "dict_chooser": "''a''"}'),

and on account of that calling, the desk can be renamed to ‘_table_name_zstd’, whereas ‘table_name’ will change into a view that may be queried as usually, together with SELECT, INSERT, UPDATE, and DELETE queries. Needless to say this perform won’t compress any knowledge by itself, that you must name zstd_incremental_maintenance afterwards.

The next variations apply when compression is energetic: 

  • The compressed column might solely comprise blob or textual content knowledge, relying on the affinity of the declared knowledge sort (e.g. VARCHAR(10) is ok, however int will not be).
  • The first key should not be null for any row, in any other case updating might not work as anticipated
    sqlite3_changes() will return 0 for modifying queries (see right here).
  • The SQLite streaming blob studying API can be considerably ineffective for the reason that blob is absolutely copied into reminiscence anyhow.
  • Attaching a database containing compressed tables utilizing ATTACH ‘foo.db’ will not be supported.
  • DDL statements (like ALTER TABLE and CREATE INDEX) are solely partially supported 

Additionally, ‘dict_chooser’ is an SQL expression that decides easy methods to partition the info. Instance partitioning keys:

I went with the best case of ‘a’ which implies that all of the rows are compressed with the identical dictionary.

And now in code:

$ sqlite3

#opening database
.open database.sqlite

#loading extension
.load sqlite_zstd

#enabling enabling clear row-level compression
choose zstd_enable_transparent('{"desk": "notes", "column": "physique", "compression_level": 19, "dict_chooser": "''a''"}');

Sadly I bought an error again :

Runtime error: Couldn't create insert set off

Brought on by:
0: close to "order": syntax error
1: Error code

Enabling the debug log of the library reveals what occurs behind the covers (many columns eliminated for brevity):

[2022-08-22T16:57:46Z INFO sqlite_zstd::create_extension] [sqlite-zstd] initialized
sqlite> choose zstd_enable_transparent('{"desk": "notes", "column": "physique", "compression_level": 19, "dict_chooser": "''a''"}');

[2022-08-22T16:57:59Z DEBUG sqlite_zstd::transparent] cols=[ColumnInfo { name: "id", coltype: "TEXT", is_primary_key: true, to_compress: false, is_dict_id: false }, ColumnInfo { name: "parent_id", coltype: "TEXT", is_primary_key: false, to_compress: false, is_dict_id: false }, ColumnInfo { name: "title", coltype: "TEXT", is_primary_key: false, to_compress: false, is_dict_id: false }, ColumnInfo { name: "*body", coltype: "TEXT", is_primary_key: false, to_compress: true, is_dict_id: false }, ColumnInfo { name: "order", }]

[2022-08-22T16:57:59Z DEBUG sqlite_zstd::transparent] [run] alter desk `notes` rename to `_notes_zstd`

[2022-08-22T16:57:59Z DEBUG sqlite_zstd::transparent] [run]
create view `notes` as
choose `id`, `parent_id`, `title`, zstd_decompress_col(`physique`, 1, `_body_dict`, true) as `physique`, `created_time`, `updated_time`, `is_conflict`, `latitude`, `longitude`, `order`, `master_key_id`
from `_notes_zstd`

[2022-08-22T16:57:59Z DEBUG sqlite_zstd::transparent] [run]
create set off `notes_insert_trigger`
as a substitute of insert on `notes`
for every row
insert into `_notes_zstd`(id, parent_id, title, physique, _body_dict, created_time, order, user_created_time, user_updated_time) choose new.`id`,
new.`physique` as `physique`, null as `_body_dict`,

After some digging I understood that the unquoted order entry was being thought of as a reserved SQL phrase and thus the error. Altering order to ‘order’ made the SQL Create Set off assertion go.This, after all, was an edge case because of the particular column title that the Joplin app is utilizing and if it have been one other app I may not even have observed. However in any case, the quoting mechanism ought to be fastened library-side, subsequently I opened a difficulty on its GitHub repo.

Till it’s fastened and a brand new model is launched, what are you able to do? I renamed the column simply to make it go:

alter desk notes rename 'order' to 'take a look at'

Then run sqlite_zstd::clear once more, which now runs to completion.

As already stated, sqlite_zstd::clear simply allows the compression. The actual work is completed by:

choose zstd_incremental_maintenance(null, 1);

adopted by:


so as to reclaim the free area.

After the couple of minutes that it takes for the operations to finish, let’s observe the good points. From 2,663,996 KB unique dimension to 1,912,900 KB. And that’s with out tweaking the settings and utilizing the default dictionary chooser. Not unhealthy!

Let’s examine if the search works too :

choose * from notes the place physique like ‘python%’ restrict 1 ;

It really works like a allure!

Compression allows you to save cupboard space by decreasing the dimensions of the database, nevertheless it has a efficiency impression as a result of the info have to be compressed and decompressed when being accessed. As such the library have to be loaded every time the database is accessed. For instance, let’s see how Joplin behaves when making an attempt to load the compressed database (assuming the renaming of the order column and the that ‘notes’ has grew to become a view as a substitute of a desk) – it has no impression on its performance.

Sure there is no such perform. Nonetheless I can run SQL queries in opposition to it by the CLI.

Bonus materials

As a substitute of fidgeting with the sqlite3 CLI you are able to do the identical simply and GUI-based through the use of Db browser for SQLite. Simply load the database and the extension by the GUI.Then run your SQL on it.

The large conclusion right here is that any software that may side-load this library can scale back the dimensions of its database by 50 to 95% whereas not affecting its primary performance. Certain the efficiency impression is there, however contemplating most operations nonetheless run at over 50k per seconds you’ll most likely produce other bottlenecks.

There’s different optimizations to be finished, however the identical technique ought to work for different databases, with barely any modifications required for say PostgreSQL.

Is that this the common-or-garden begin of a higher impression on the state of database know-how ?

 Rust Use Continues To Increase

Extra Data


Associated Articles

In Reward Of SQLite

Take The Newbie’s Sequence to Rust


To be told about new articles on I Programmer, join our weekly publication, subscribe to the RSS feed and comply with us on Twitter, Fb or Linkedin.






or e-mail your remark to: [email protected]

Leave a Reply

Your email address will not be published. Required fields are marked *

Next Post

Iranian hackers lurked in Albania’s govt community for 14 months

The Federal Bureau of Investigation (FBI) and CISA stated that one of many Iranian menace teams behind the damaging assault on the Albanian authorities’s community in July lurked inside its programs for roughly 14 months. “A FBI investigation signifies Iranian state cyber actors acquired preliminary entry to the sufferer’s community […]
Iranian hackers lurked in Albania’s govt community for 14 months WordPress Theme: Seek by ThemeInWP