For native SQLite tables, DROP TABLE is much faster than DELETE FROM.
-----Urspr?ngliche Nachricht-----
Von: Jonathan Moules [mailto:J.Moules at hrwallingford.com]
Gesendet: Dienstag, 14. April 2015 11:40
An: 'sqlite-users at mailinglists.sqlite.org'
Betreff: [sqlite] Best way to temporarily store data before processing
Hi List,
I'm wondering if anyone can offer me a "best practice" way of doing this.
I'm doing some log analysis using Python/SQLite. Python parses a log file and
splits the raw data from each line in the log into one of about 40 tables in an
SQLite database (I'll call them Raw Tables).
Once a log file has been processed like this, I run some SQL which takes the
data from the Raw Tables and aggregates/processes it into about 10 different
"Prepared Tables" which are read with a bunch of Views. The
aggregation/processing doesn't take long, and the SQL for it is simple.
I'd like to update the Prepared Tables after each log file is read because
there are thousands of files and I don't want to have to rely on having GB of
disk space sitting around for temporary Raw Tables.
Once the Prepared Tables have been created, there's no real need to keep the
data in the Raw Tables.
The Prepared Tables don't have to be in the same database as the Raw Tables.
I'm happy to use ATTACH.
So my question:
What's the best way to do this with the minimum overhead?
Options that have come to mind (probably missed a lot):
- Some sort of empty template database for the Raw Tables which is
copied/cloned/overwritten for each file processed.
- And/Or use "DELETE FROM Raw_Tables" to truncate it after each
file (there are no indexes).
- And/Or place it into :memory:.
- And/Or just CREATE the Raw Tables for each file?
- And/Or do it within the Prepared Tables database and use "DELETE
FROM Raw_Tables". (That file you wouldn't want in :memory: of course).
Thoughts welcome, thanks for your time,
Jonathan
________________________________
HR Wallingford and its subsidiaries uses faxes and emails for confidential and
legally privileged business communications. They do not of themselves create
legal commitments. Disclosure to parties other than addressees requires our
specific consent. We are not liable for unauthorised disclosures nor reliance
upon them.
If you have received this message in error please advise us immediately and
destroy all copies of it.
HR Wallingford Limited
Howbery Park, Wallingford, Oxfordshire, OX10 8BA, United Kingdom Registered in
England No. 02562099
________________________________
_______________________________________________
sqlite-users mailing list
sqlite-users at mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___________________________________________
Gunter Hick
Software Engineer
Scientific Games International GmbH
FN 157284 a, HG Wien
Klitschgasse 2-4, A-1130 Vienna, Austria
Tel: +43 1 80100 0
E-Mail: hick at scigames.at
This communication (including any attachments) is intended for the use of the
intended recipient(s) only and may contain information that is confidential,
privileged or legally protected. Any unauthorized use or dissemination of this
communication is strictly prohibited. If you have received this communication
in error, please immediately notify the sender by return e-mail message and
delete all copies of the original communication. Thank you for your cooperation.
