Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Simon Slavin
On 14 Feb 2011, at 7:33am, Max Vlasov wrote: The first variation was similar to the one with bus-powered, this time I used hard button on the box that is equivalent to unplugging both connection and power. The difference for a single test was really impressive 5355 - 4445 = -910. And when I

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Max Vlasov
On Mon, Feb 14, 2011 at 12:05 PM, Simon Slavin slav...@bigfraud.org wrote: The second variation was just unplugging the cord but keeping the power intact, so if it's drive that caches, it would end its operations completely. This time the results were perfect, for example 4822 - 4822, and

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Simon Slavin
On 14 Feb 2011, at 10:30am, Max Vlasov wrote: On Mon, Feb 14, 2011 at 12:05 PM, Simon Slavin slav...@bigfraud.org wrote: Can you expand upon your conclusion ? I like your test, and am interested to know precisely where you think the lag is happening. Simon, the conclusion came from the

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Black, Michael (IS)
SOLVED!! Marcus Grimm and I went back forth a number of times trying to figure out why my benchmarks were so much faster than his. Found it... My SATA RAID setup had Enable Advanced Performance on by default (I had never turned it on). My secondary tests on an IDE drive showed similar

Re: [sqlite] BUG (3.7.5): identifier quotation characters get part of column names for certain statements

2011-02-14 Thread Nißl Reinhard
Hi, looks like the patch did got stripped. I've attached it again. And please find attached a further patch which addresses TK_AGG_COLUMNS. Bye. -- Reinhard Nißl -Ursprüngliche Nachricht- Von: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] Im Auftrag von Nißl

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Black, Michael (IS)
Also...here's another part of the benchmark which shows triggers aren't as evil as I thought. Trigger for this example was 2X the manual update. F:\d:batch 50 1000 12 using wal mode using update trigger Sqlite Version: 3.7.5 Inserting 50 rows using a bulk of 1000 commits per second:

Re: [sqlite] BUG (3.7.5): identifier quotation characters get part of column names for certain statements

2011-02-14 Thread Nißl Reinhard
Hi, sorry for the noise. Again trying to supply attachments. Bye. -- Reinhard Nißl -Ursprüngliche Nachricht- Von: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] Im Auftrag von Nißl Reinhard Gesendet: Montag, 14. Februar 2011 13:56 An: General Discussion of

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Jim Wilcoxson
On Mon, Feb 14, 2011 at 2:33 AM, Max Vlasov max.vla...@gmail.com wrote: On Sun, Feb 13, 2011 at 8:53 PM, Jim Wilcoxson pri...@gmail.com wrote: But I thought about how it would be possible to test this explanation . I'm going to do some tests that works like this. The same id updating,

Re: [sqlite] Question about having 2 db connections on same

2011-02-14 Thread Jeff Archer
Date: Sun, 13 Feb 2011 22:14:29 -0500 From: Pavel Ivanov paiva...@gmail.com I do a transaction on connection 1 using BEGIN IMMEDIATE, some rows updated, COMMIT. Then, when I attempt same sequence on connection 2, when I do first call to step to execute begin transaction and it never returns.

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Pavel Ivanov
So my question is, does it maintain the other 3 parts of ACID, so that the database will never be in a corrupted state after a power loss, even though it may be missing some writes that were confirmed? Jim, I think the answer to your question is already in Max's tests: the USB drive is

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Simon Slavin
On 14 Feb 2011, at 3:56pm, Pavel Ivanov wrote: So my question is, does it maintain the other 3 parts of ACID, so that the database will never be in a corrupted state after a power loss, even though it may be missing some writes that were confirmed? Jim, I think the answer to your question

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Jim Wilcoxson
On Mon, Feb 14, 2011 at 10:56 AM, Pavel Ivanov paiva...@gmail.com wrote: So my question is, does it maintain the other 3 parts of ACID, so that the database will never be in a corrupted state after a power loss, even though it may be missing some writes that were confirmed? Jim, I think the

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Simon Slavin
On 14 Feb 2011, at 4:48pm, Jim Wilcoxson wrote: I believe (though am not sure) that the drive can reorder individual sector/record writes between syncs however it likes, as long as it doesn't allow writes to cross a sync and also executes the sync groups (for lack of a better term - the

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Max Vlasov
On Mon, Feb 14, 2011 at 7:48 PM, Jim Wilcoxson pri...@gmail.com wrote: On Mon, Feb 14, 2011 at 10:56 AM, Pavel Ivanov paiva...@gmail.com wrote: So my question is, does it maintain the other 3 parts of ACID, so that the database will never be in a corrupted state after a power loss, even

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Jim Wilcoxson
On Mon, Feb 14, 2011 at 10:56 AM, Pavel Ivanov paiva...@gmail.com wrote: So my question is, does it maintain the other 3 parts of ACID, so that the database will never be in a corrupted state after a power loss, even though it may be missing some writes that were confirmed? Jim, I think the

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Max Vlasov
On Mon, Feb 14, 2011 at 8:42 PM, Jim Wilcoxson pri...@gmail.com wrote: On Mon, Feb 14, 2011 at 10:56 AM, Pavel Ivanov paiva...@gmail.com wrote: So my question is, does it maintain the other 3 parts of ACID, so that the database will never be in a corrupted state after a power loss, even

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Black, Michael (IS)
And you think Jim's timings are wrong because.. I've already shown you can get speed just like he's showing. That's what you get on a good write-cache-enabled drive. And if you want to talk about data reliability...BACK UP YOUR DATA. The likely failure points I can think of are: #1

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Max Vlasov
On Mon, Feb 14, 2011 at 11:50 PM, Black, Michael (IS) michael.bla...@ngc.com wrote: And if you want to talk about data reliability...BACK UP YOUR DATA. The likely failure points I can think of are: #1 Power supply (redundant supplies available) #2 Hard drive smoked (and your data is toast

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Jim Wilcoxson
On Mon, Feb 14, 2011 at 3:02 PM, Max Vlasov max.vla...@gmail.com wrote: On Mon, Feb 14, 2011 at 8:42 PM, Jim Wilcoxson pri...@gmail.com wrote: On Mon, Feb 14, 2011 at 10:56 AM, Pavel Ivanov paiva...@gmail.com wrote: So my question is, does it maintain the other 3 parts of ACID, so that the

Re: [sqlite] EXT :Re: UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Black, Michael (IS)
I ran some tests on my USB stick and pulled-while-writing...no corruption at all. Integrity check worked fine every time. I checked and write caching is not enabled for that drive (Windows XP 64) For Linux I believe you can force synchronous mode with a mount like this: /dev/sdb1 /media/usb

Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected

2011-02-14 Thread Simon Slavin
On 14 Feb 2011, at 8:50pm, Black, Michael (IS) wrote: And if you want to talk about data reliability...BACK UP YOUR DATA. And keep the backups off-site. And once a year try to actually restore one of them. I've earned /lots/ of money from companies with backup systems that claimed to work

[sqlite] BUG: test scripts corrupt*.test need do_not_use_codec

2011-02-14 Thread Noah Hart
corruptA.test corruptD.test need the following changes: -source $testdir/tester.tcl +source $testdir/tester.tcl + +# Do not use a codec for tests in this file, as the database file is +# manipulated directly using tcl scripts (using the [hexio_write] command). +# +do_not_use_codec

[sqlite] BUG: test script vacuum2.test needs do_not_use_codec

2011-02-14 Thread Noah Hart
vacuum2.test reads directly from the database using hexio_read It need the following changes: -source $testdir/tester.tcl +source $testdir/tester.tcl + +# Do not use a codec for tests in this file, as the database file is +# manipulated directly using tcl scripts (using the [hexio_read]

[sqlite] BUG: test script exclusive2.test needs do_not_use_codec

2011-02-14 Thread Noah Hart
exclusive2.test reads directly from the database using binary read It needs the following changes: -source $testdir/tester.tcl +source $testdir/tester.tcl + +# Do not use a codec for tests in this file, as the database file is +# manipulated directly using tcl scripts (using binary read) +#