Re: [sqlite] Solving the Sudoku with SQlite 3.8.3 trunk

2014-01-19 Thread big stone
Hi Keith, Indeed just removing the CTE creation of the DIGITS makes Dan's version up to speed. Would the wholenumber external SQLite module help : - to make SQLite code cleaner ? (like generate_series of Postgresql, or dual of Oracle) - still provide the same speed-up ? Portfolio of typical

[sqlite] Dramatic performance difference between different PRAGMA synchronous settings for bulk inserts

2014-01-19 Thread Mario M. Westphal
I have a performance effect which I don't quite understand. Maybe I'm using the wrong settings or something. Sorry for the long post, but I wanted to include all the info that may be important. My software is written in C++, runs on Windows 7/8, the SQLite database file is either on a local SATA

Re: [sqlite] Dramatic performance difference between different PRAGMA synchronous settings for bulk inserts

2014-01-19 Thread Simon Slavin
On 19 Jan 2014, at 2:00pm, Mario M. Westphal m...@mwlabs.de wrote: I logged the execution times of various operations in this phase to a text file. Everything was fast, the processing, the INSERTs etc. But COMMIT operations sometimes took 20s, then 0.2s, then again 10s. That's the time

[sqlite] Sqlie from c#: encoding bug?

2014-01-19 Thread cinema cinema
I'm reading a text from c# saving it through a insert parameter in a varchar column of a table. Both the file and the sqlite db encoding is utf-8 but I see different characters (it seems due to a bad encoding) while reading data from the sqlite3 command line or from other clients... Any

[sqlite] Command line shell not flushing stderr when interactive

2014-01-19 Thread Christopher Wellons
When the shell is set to interactive (i.e. -interactive), the output (stdout) is flushed with every prompt (shell.c:422) but stderr is not. In some situations this leads to no error messages being displayed until the stderr buffer fills. This happens when running the official sqlite3 binary as

Re: [sqlite] Command line shell not flushing stderr when interactive

2014-01-19 Thread Richard Hipp
On Sun, Jan 19, 2014 at 11:10 AM, Christopher Wellons well...@nullprogram.com wrote: When the shell is set to interactive (i.e. -interactive), the output (stdout) is flushed with every prompt (shell.c:422) but stderr is not. Stderr is suppose to be unbuffered so that flushing is not

Re: [sqlite] Dramatic performance difference between different PRAGMA synchronous settings for bulk inserts

2014-01-19 Thread Richard Hipp
In WAL mode with synchronous=NORMAL, SQLite only syncs (FlushFileBuffers() on windows) when it does a checkpoint operation. Checkpoints should be happening automatically whenever the WAL file exceeds about 1MB in size. For an 8GB database, probably there are about 8000 sync operations,

Re: [sqlite] Dramatic performance difference between different PRAGMA synchronous settings for bulk inserts

2014-01-19 Thread Petite Abeille
On Jan 19, 2014, at 3:00 PM, Mario M. Westphal m...@mwlabs.de wrote: Also FTS4 is used, which also creates large tables. (Unrelated to your question, but, take a look at external content FTS4 table… they dramatically cut down the amount of duplicated data [1]) During an ingest phase, my

Re: [sqlite] Command line shell not flushing stderr when interactive

2014-01-19 Thread Christopher Wellons
When the shell is set to interactive (i.e. -interactive), the output (stdout) is flushed with every prompt (shell.c:422) but stderr is not. Stderr is suppose to be unbuffered so that flushing is not required. Or is that different for windows? According to the stderr Linux man page stderr

Re: [sqlite] Solving the Sudoku with SQlite 3.8.3 trunk

2014-01-19 Thread Keith Medcalf
soduko1.sql and soduko2.sql are the two originals. soduko3.sql removes the digits view to an actual table (from soduko2.sql) and soduko3.sql puts digits back in as a CTE but is a select from the wholenumber module rather than generating the digits recursively. So, the fastest one uses digits

Re: [sqlite] Command line shell not flushing stderr when interactive

2014-01-19 Thread Luuk
On 19-01-2014 19:59, Christopher Wellons wrote: When the shell is set to interactive (i.e. -interactive), the output (stdout) is flushed with every prompt (shell.c:422) but stderr is not. Stderr is suppose to be unbuffered so that flushing is not required. Or is that different for windows?

Re: [sqlite] Command line shell not flushing stderr when interactive

2014-01-19 Thread Simon Slavin
On 19 Jan 2014, at 7:32pm, Luuk luu...@gmail.com wrote: It is acceptable—and normal—for standard output and standard error to be directed to the same destination, such as the text terminal. Messages appear in the same order as the program writes them, unless buffering is involved. (For

Re: [sqlite] Dramatic performance difference between different PRAGMA synchronous settings for bulk inserts

2014-01-19 Thread Mario M. Westphal
If you want to try running with synchronous=NORMAL, you might try setting PRAGMA wal_autocheckpoint=10; (from the default of 1000) which will make for dramatically larger WAL files, but also dramatically fewer syncs. Then the syncs will use just 5 or 6 minutes instead of 4.5 hours.

Re: [sqlite] Dramatic performance difference between different PRAGMA synchronous settings for bulk inserts

2014-01-19 Thread Mario M. Westphal
Unrelated to your question, but, take a look at external content FTS4 table they dramatically cut down the amount of duplicated data [1]) Thanks for the tip. I'll definitely check that. Currently I build the contents for FTS dynamically from several other tables, combining, splitting,

Re: [sqlite] Mutally dependent JOIN clauses

2014-01-19 Thread Hick Gunter
-Ursprüngliche Nachricht- Von: Rob Golsteijn [mailto:rob.golste...@mapscape.eu] Gesendet: Freitag, 17. Jänner 2014 11:38 An: sqlite-users@sqlite.org Betreff: Re: [sqlite] Mutally dependent JOIN clauses ... My statement: SELECT * FROM C LEFT JOIN A ON A.a*A.a + B.b*B.b = C.c*c.c