> On Nov 6, 2019, at 9:25 AM, Digital Dog <digitald...@gmail.com> wrote: > > If there are millions or billions of rows > in the data set I consider it big data and the only reasonable format for > storing it is a column store format.
There are many types of stores for "big data". My employer, Couchbase, has customers who would disagree with you; they use our document database to store millions or billions of rows. It depends on the data type. For highly structured data with many sparse columns you may be right. For less structured data a key-value or document store is better. I'm sure there are other types of big-data storage I'm unaware of. In the original poster's case, it didn't seem like the data set really had zillions of columns, since [IIRC] they didn't need to be queried separately. You could put something like that in a key-value store with each value an encoded C array, for example. —Jens _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
