On Tue, 23 Jun 2009 22:01:26 +0200, Misza wrote: > I wonder if anyone used SQLite extensively with big datasets and could > provide some insight into performance? > In a nutshell, I am writing an ETL framework and need a good (read: > performing) engine for the "T"ransform part. > I suppose I could use flat files for that, but I'd like to have some SQL > capabilities at my disposal, which is why I'm poking around file-based, > serverless engines. > The question is, how does SQLite perform when faced with huge datasets, > where "huge" means 10s of gigabytes in size (typical for a Data > Warehouse's staging area)? > Most common operations (after unload) would include multi-table joins > (mostly merge joins), field transformations (contatenation, casting) and > record filtering.
You might want to consider Hive. http://www.facebook.com/note.php?note_id=89508453919 ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________ _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
