Hi, I'm totally new to sqlite and I'd like to use it for some logging application on an embedded linux-based device. Data comes from multiple (~10), similar sources at a steady rate. The rolling data set would be in the size of 20 GB. Such an amount of storage would suffice to retain data from the previous week or so.
Reading the documentation https://www.sqlite.org/whentouse.html somehow suggests the usage of sharding: >Concurrency is also improved by "database sharding": using separate database files for > different subdomains. For example, the server might have a separate SQLite database for each > user, so that the server can handle hundreds or thousands of simultaneous connections, but > each SQLite database is only used by one connection. In my case I would be doing sharding on the data source and/or the day of the timestamp, so to have individual files in the size of a few hundreds MB. This way, deleting the oldest data would be as easy as deleting the corresponding file. However, I did not find any reference whatsoever on sharding being available _within_ sqlite. Ideally, I would like to have a way of "seeing" the whole dataset with a single query spanning all available databases. Would that be at all feasible? I saw the "attach database" statement which seems closely related but whose use-case I honestly don't get. If not, is there any state-of-the-art adapter layer that would be performing (and hide) the underlying sharding? I don't really care about query performance (e.g. if such a global query spanning 20 different databases is indeed performed serially, thereby take 20 times longer), I just need a way of hiding this detail. I saw some reference to SPHiveDB https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg43575.html but the project looks stale (9 years since the last commit). I also looked into AtomDB but it looks overly complicated for my use-case (single, embedded server), plus it somehow requires the underlying sharding to be totally exposed. Any ideas? Gerlando _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
