[sqlite] AUTO: Bret Patterson/Austin/IBM is out of the office (returning 10/22/2012)
I am out of the office until 10/22/2012. I'm out of the office but checking email once a day and will respond to any high importance issues as quickly as possible. Note: This is an automated response to your message "sqlite-users Digest, Vol 58, Issue 13" sent on 10/13/2012 10:00:02 AM. This is the only notification you will receive while this person is away. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] AUTO: Bret Patterson/Austin/IBM is out of the office (returning 11/28/2011)
I am out of the office until 11/28/2011. I'm out of the office but checking email once or twice a day and will respond to any high importance issues as quickly as possible. Note: This is an automated response to your message "sqlite-users Digest, Vol 47, Issue 18" sent on 11/18/2011 10:00:01. This is the only notification you will receive while this person is away. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] AUTO: Bret Patterson/Austin/IBM is out of the office (returning 08/22/2011)
I am out of the office until 08/22/2011. I'm out of the office but checking email once or twice a day and will respond to any high importance issues as quickly as possible. Note: This is an automated response to your message "sqlite-users Digest, Vol 44, Issue 12" sent on 08/12/2011 6:00:01. This is the only notification you will receive while this person is away. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] AUTO: Bret Patterson/Austin/IBM is out of the office (returning 06/17/2011)
I am out of the office until 06/17/2011. I'm out of the office but checking email once or twice a day and will respond to any high importance issues as quickly as possible. Note: This is an automated response to your message "sqlite-users Digest, Vol 42, Issue 16" sent on 06/16/2011 6:00:01. This is the only notification you will receive while this person is away. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] AUTO: Bret Patterson/Austin/IBM is out of the office (returning 03/21/2011)
I am out of the office until 03/21/2011. I'm out of the office but checking email once or twice a day and will respond to any high importance issues as quickly as possible. Note: This is an automated response to your message "sqlite-users Digest, Vol 39, Issue 17" sent on 3/17/2011 6:00:01. This is the only notification you will receive while this person is away. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] AUTO: Bret Patterson/Austin/IBM is out of the office (returning 07/01/2010)
I am out of the office until 07/01/2010. I'm out of the office but checking email once or twice a day and will respond to any high importance issues as quickly as possible. Note: This is an automated response to your message "sqlite-users Digest, Vol 30, Issue 27" sent on 6/28/10 6:00:02. This is the only notification you will receive while this person is away. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] AUTO: Bret Patterson/Austin/IBM is out of the office (returning 03/08/2010)
I am out of the office until 03/08/2010. I'm out of the office but checking email once or twice a day and will respond to any high importance issues as quickly as possible. Note: This is an automated response to your message "sqlite-users Digest, Vol 27, Issue 4" sent on 3/4/10 5:00:02. This is the only notification you will receive while this person is away. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] AUTO: Bret Patterson/Austin/IBM is out of the office (returning 01/04/2010)
I am out of the office until 01/04/2010. I'm out of the office but checking email once or twice a day and will respond to any high importance issues as quickly as possible. Note: This is an automated response to your message "sqlite-users Digest, Vol 24, Issue 11" sent on 12/11/09 5:00:01. This is the only notification you will receive while this person is away. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] AUTO: Bret Patterson/Austin/IBM is out of the office (returning 11/30/2009)
I am out of the office until 11/30/2009. I'm out of the office but checking email once or twice a day and will respond to any high importance issues as quickly as possible. Note: This is an automated response to your message "sqlite-users Digest, Vol 23, Issue 24" sent on 11/24/09 5:00:01. This is the only notification you will receive while this person is away. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Disk activity on Linux
Thanks Walter and Paul I'll look into the things you pointed out and let you guys know how it turns out. Open Source approval at IBM is version specific and the approval process is quite a bit of work so I'd like to avoid it as long as possible. Bret Patterson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Disk activity on Linux
We're seeing a lot more disk activity than expected on Linux when using sqlite3. We've run this same series of test on windows and the disk IO is much lower, which is the opposite of what I really expected. Below is my scenario and perhaps someone can point out what I can do to fix this problem. We are using sqlite version 3.5.7 . Shared cache is enabled using at program startup time: sqlite3_enable_shared_cache(1); We create a new database every hour. This database has 2 indexes and a ton of columns. The maximum size of each row is about 3,400 bytes. A lot of that is UTF8 strings. The database was actually pre-created on a windows machine and is installed with the product. The stored database template file is then copied to the real database instead of actually "creating" it every hour. The database is opened in read/write mode and we issue the following pragma's after opening: PRAGMA read_uncommitted=ON PRAGMA synchronous=OFF I also call: sqlite3_busy_timeout( db,TM_DB_BUSY_TIMEOUT); // 30,000 is the value of the busy timeout. We are then attempting to insert about 278 rows per second over the course of an hour. The batch size is 8,192 rows or every 5 minutes, whichever comes first. In this case we are committing the transaction approximately every 30-40 seconds. The database is grows from it's initial creation size of about 11,000 bytes to about 1.5 gigabytes in the course of the hour. As the database size increases we are seeing significant increases in the CPU IOWait times that eventually slow down the insertion thread enough that our reception queue fills up and we have to throw out data to prevent a queue overflow. The disk we are using is 2 disks in Raid 0 configuration. On windows we had a similar problem but we fixed it by issuing the PRAGMA synchronous=OFF to speed up the disk io. A corrupt database is a problem but performance is very important and we can tolerate a potential database corrupt in low chance scenarios. I've tried everything I can think of to reduce the IOWait times and am at a loss. I've lowered the batch size and even tried the pragma PRAGMA journal_mode=MEMORY but that didn't seem to have any effect that I could determine. If I watch the database directory as the test is run the Database file and the journal file are being written to constantly. Anyone have any ideas? This problem is limiting the scale of our product significantly and I need to find a solution to it. We are pretty much hitting a wall around 277 records per second and we need to be more around 500-1000 records per second if possible. Bret Patterson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
