Thanks for the thoughts, useful to know. In relation to your question Joseph, historically the scripts were written to only put the raw data into the database and then the Views read it, but that was too slow. So now I've got an extra step to turn the raw data into easily Viewable data, with the bonus it uses considerably less disk space. The alternative of hard-coding the analysis and getting Python to write only the Processed Tables seemed like an ugly design, so I went with this.
Cheers -----Original Message----- From: sqlite-users-bounces at mailinglists.sqlite.org [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Joseph T. Sent: Tuesday, April 14, 2015 12:17 PM To: General Discussion of SQLite Database Subject: Re: [sqlite] Best way to temporarily store data before processing Mr. Moules, why not skip the raw tables entirely? Or failing that a separate in memory db would probably be the best and quickest option. Sent from my Samsung Epic? 4G TouchJonathan Moules <J.Moules at hrwallingford.com> wrote:Hi List, I'm wondering if anyone can offer me a "best practice" way of doing this. I'm doing some log analysis using Python/SQLite. Python parses a log file and splits the raw data from each line in the log into one of about 40 tables in an SQLite database (I'll call them Raw Tables). Once a log file has been processed like this, I run some SQL which takes the data from the Raw Tables and aggregates/processes it into about 10 different "Prepared Tables" which are read with a bunch of Views. The aggregation/processing doesn't take long, and the SQL for it is simple. I'd like to update the Prepared Tables after each log file is read because there are thousands of files and I don't want to have to rely on having GB of disk space sitting around for temporary Raw Tables. Once the Prepared Tables have been created, there's no real need to keep the data in the Raw Tables. The Prepared Tables don't have to be in the same database as the Raw Tables. I'm happy to use ATTACH. So my question: What's the best way to do this with the minimum overhead? Options that have come to mind (probably missed a lot): - Some sort of empty template database for the Raw Tables which is copied/cloned/overwritten for each file processed. - And/Or use "DELETE FROM Raw_Tables" to truncate it after each file (there are no indexes). - And/Or place it into :memory:. - And/Or just CREATE the Raw Tables for each file? - And/Or do it within the Prepared Tables database and use "DELETE FROM Raw_Tables". (That file you wouldn't want in :memory: of course). Thoughts welcome, thanks for your time, Jonathan ________________________________ HR Wallingford and its subsidiaries uses faxes and emails for confidential and legally privileged business communications. They do not of themselves create legal commitments. Disclosure to parties other than addressees requires our specific consent. We are not liable for unauthorised disclosures nor reliance upon them. If you have received this message in error please advise us immediately and destroy all copies of it. HR Wallingford Limited Howbery Park, Wallingford, Oxfordshire, OX10 8BA, United Kingdom Registered in England No. 02562099 ________________________________ _______________________________________________ sqlite-users mailing list sqlite-users at mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users at mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users This message has been scanned for viruses by MailControl - www.mailcontrol.com Click https://www.mailcontrol.com/sr/OtIwePPILzPGX2PQPOmvUtzroFmuoUEuyo!5uyiNNRAv6vRKBGbHjbCMljIjlDona9wHUfmUgVH3KeH4!zzvzA== to report this email as spam. ________________________________ HR Wallingford and its subsidiaries uses faxes and emails for confidential and legally privileged business communications. They do not of themselves create legal commitments. Disclosure to parties other than addressees requires our specific consent. We are not liable for unauthorised disclosures nor reliance upon them. If you have received this message in error please advise us immediately and destroy all copies of it. HR Wallingford Limited Howbery Park, Wallingford, Oxfordshire, OX10 8BA, United Kingdom Registered in England No. 02562099 ________________________________
