> Ideally, I would like to have a way of "seeing" the whole dataset with a > single query spanning all available databases.
I think swarmvtab may be helpful. https://www.sqlite.org/swarmvtab.html 2018-07-29 10:34 GMT+02:00, Gerlando Falauto <gerlando.fala...@gmail.com>: > Hi, > > I'm totally new to sqlite and I'd like to use it for some logging > application on an embedded > linux-based device. Data comes from multiple (~10), similar sources at a > steady rate. > The rolling data set would be in the size of 20 GB. Such an amount of > storage would suffice to retain data from the previous week or so. > > Reading the documentation https://www.sqlite.org/whentouse.html somehow > suggests the usage of sharding: > >>Concurrency is also improved by "database sharding": using separate > database files for >> different subdomains. For example, the server might have a separate > SQLite database for each >> user, so that the server can handle hundreds or thousands of simultaneous > connections, but >> each SQLite database is only used by one connection. > > In my case I would be doing sharding on the data source and/or the day of > the timestamp, so to have individual files in the size of a few hundreds MB. > This way, deleting the oldest data would be as easy as deleting the > corresponding file. > > However, I did not find any reference whatsoever on sharding being > available _within_ sqlite. > Ideally, I would like to have a way of "seeing" the whole dataset with a > single query spanning all available databases. > > Would that be at all feasible? I saw the "attach database" statement which > seems closely related but whose use-case I honestly don't get. > If not, is there any state-of-the-art adapter layer that would be > performing (and hide) the underlying sharding? I don't really care about > query performance (e.g. if such a global query spanning 20 different > databases is indeed performed serially, thereby take 20 times longer), I > just need a way of hiding this detail. > > I saw some reference to SPHiveDB > https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg43575.html > but the project looks stale (9 years since the last commit). > > I also looked into AtomDB but it looks overly complicated for my use-case > (single, embedded server), plus it somehow requires the underlying sharding > to be totally exposed. > > Any ideas? > Gerlando > _______________________________________________ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
