Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread jungle Boogie
Hi there, Just because I'm interested, I'm wondering if you can identify your hardware, and how long it takes your system to do your desired operation on such a large number of records. Do let us know which option you performed the query with as well. P.S. For best results, I'd recommend using t

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Nico Williams
On Tue, Sep 26, 2017 at 01:37:42PM -0700, Jens Alfke wrote: > > On Sep 26, 2017, at 1:17 PM, Guy Harris wrote: > > A user wouldn't know what to do with "you've exceeded your stored data > > quota”? > > A Turkish or Chinese user likely wouldn’t. (SQLite’s error messages > are not localized.) And

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Jens Alfke
> On Sep 26, 2017, at 3:17 PM, Guy Harris wrote: > > It shows a whole bunch of codes, none of which are "something that > distinguishes EIO from other errors such as EFBIG, EDQUOT, etc.". > > I'm not asking for something that indicates what xXYZZY method reported the > error. I'm asking for

[sqlite] CREATE FOREIGN KEY support

2017-09-26 Thread Igor Korot
Hi, ALL, I believe the creation a foreign key on the existing tables is not supported on SQLite. However, I can issue a series of the SQL command which will emulate the creation of foreign key. 1. BEGIN 2. CREATE TEMPORARY TABLE temp AS SELECT * FROM ; 3. DROP TABLE ; 4. CREATE TABLE (, FOREIGN K

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Guy Harris
On Sep 26, 2017, at 3:11 PM, Simon Slavin wrote: > On 26 Sep 2017, at 10:53pm, Guy Harris wrote: >> >> I *would* suggests an additional API to get a *separate* extended error >> code, so that if, for example, a write() fails and that failure is turned >> into SQLITE_IOERR, you can get somethi

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Simon Slavin
On 26 Sep 2017, at 10:53pm, Guy Harris wrote: > > I *would* suggests an additional API to get a *separate* extended error code, > so that if, for example, a write() fails and that failure is turned into > SQLITE_IOERR, you can get something that distinguishes EIO from other errors > such as

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Ron Barnes
😊 Than you! -Original Message- From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Stephen Chrzanowski Sent: Tuesday, September 26, 2017 4:31 PM To: SQLite mailing list Subject: Re: [sqlite] Is there a way to perform a muti-level sort and extract of la

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Guy Harris
On Sep 26, 2017, at 2:22 PM, Jens Alfke wrote: >> On Sep 26, 2017, at 1:57 PM, Guy Harris wrote: >> >> Which means "for stuff that would be shown to the user, for the user to >> read, either localize your error messages, or make sure your API returns >> error codes that the application can tu

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Guy Harris
On Sep 26, 2017, at 2:08 PM, Scott Robison wrote: > There are physical errors and there are logical errors. If an error is > generated from write, it's not unreasonable to classify it as an > "output error". From read as an "input error". "Output error", yes, although it'd be useful to provide m

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Guy Harris
On Sep 26, 2017, at 2:16 PM, Simon Slavin wrote: > On 26 Sep 2017, at 9:57pm, Guy Harris wrote: > >> On Sep 26, 2017, at 1:37 PM, Jens Alfke wrote: >> On Sep 26, 2017, at 1:17 PM, Guy Harris wrote: A user wouldn't know what to do with "you've exceeded your stored data q

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Jens Alfke
> On Sep 26, 2017, at 1:57 PM, Guy Harris wrote: > > Which means "for stuff that would be shown to the user, for the user to read, > either localize your error messages, or make sure your API returns error > codes that the application can turn into localized error messages". Um, that’s what

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Guy Harris
On Sep 26, 2017, at 1:43 PM, Simon Slavin wrote: > On 26 Sep 2017, at 9:17pm, Guy Harris wrote: > >> The *number* might annoy the support staff; right off the top of your head, >> what's the error number for "file system quota exceeded" or "I/O error"? >> (No cheating by looking it up in a m

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Simon Slavin
On 26 Sep 2017, at 9:57pm, Guy Harris wrote: > On Sep 26, 2017, at 1:37 PM, Jens Alfke wrote: > >>> On Sep 26, 2017, at 1:17 PM, Guy Harris wrote: >>> >>> A user wouldn't know what to do with "you've exceeded your stored data >>> quota”? >> >> A Turkish or Chinese user likely wouldn’t. (S

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Scott Robison
There are physical errors and there are logical errors. If an error is generated from write, it's not unreasonable to classify it as an "output error". From read as an "input error". There is a lot of sqlite source code that already exists and has been written to work with the current interface. T

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Guy Harris
On Sep 26, 2017, at 1:37 PM, Jens Alfke wrote: >> On Sep 26, 2017, at 1:17 PM, Guy Harris wrote: >> >> A user wouldn't know what to do with "you've exceeded your stored data >> quota”? > > A Turkish or Chinese user likely wouldn’t. (SQLite’s error messages are not > localized.) Which means

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Simon Slavin
On 26 Sep 2017, at 9:17pm, Guy Harris wrote: > The *number* might annoy the support staff; right off the top of your head, > what's the error number for "file system quota exceeded" or "I/O error"? (No > cheating by looking it up in a man page or include file!) My support staff are allowed

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Jens Alfke
> On Sep 26, 2017, at 1:17 PM, Guy Harris wrote: > > A user wouldn't know what to do with "you've exceeded your stored data quota”? A Turkish or Chinese user likely wouldn’t. (SQLite’s error messages are not localized.) And there are plenty of messages that are much less understandable to a

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Stephen Chrzanowski
uh.. UNIQUE... DISTINCT... , On Tue, Sep 26, 2017 at 4:30 PM, Stephen Chrzanowski wrote: > > > On Tue, Sep 26, 2017 at 1:36 PM, Simon Slavin > wrote: > >> >> >> My one concern in reading your post is how your dates are formatted. >> When putting your date fields into your SQL table you will hav

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Stephen Chrzanowski
On Tue, Sep 26, 2017 at 1:36 PM, Simon Slavin wrote: > > > My one concern in reading your post is how your dates are formatted. When > putting your date fields into your SQL table you will have to ensure that > dates are saved as a day number, or as text which naturally sorts into date > order,

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Guy Harris
On Sep 26, 2017, at 1:05 PM, Simon Slavin wrote: > On 26 Sep 2017, at 8:47pm, Guy Harris wrote: > >> On Sep 26, 2017, at 8:22 AM, Jens Alfke wrote: >> >>> The basic error code is SQLITE_IOERR, which just means "Some kind of disk >>> I/O error occurred” according to the comment. Which is true

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Simon Slavin
On 26 Sep 2017, at 8:47pm, Guy Harris wrote: > On Sep 26, 2017, at 8:22 AM, Jens Alfke wrote: > >> The basic error code is SQLITE_IOERR, which just means "Some kind of disk >> I/O error occurred” according to the comment. Which is true in this case; an >> I/O operation returned an error. >

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Ron Barnes
That is exactly what I want to do -Original Message- From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Keith Medcalf Sent: Tuesday, September 26, 2017 3:04 PM To: SQLite mailing list Subject: Re: [sqlite] Is there a way to perform a muti-level sort

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Ron Barnes
Thank you very much! I will research the two suggestions below. As for your sort assumption, you are correct. A A 1 2 A A 2 1 A B 1 2 A B 1 3 A C 1 1 -Original Message- From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Warren Young S

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Ron Barnes
Thank you so much - I will test this as soon as I get home! -Original Message- From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Simon Slavin Sent: Tuesday, September 26, 2017 1:37 PM To: SQLite mailing list Subject: Re: [sqlite] Is there a way to perf

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Guy Harris
On Sep 26, 2017, at 8:22 AM, Jens Alfke wrote: > The basic error code is SQLITE_IOERR, which just means "Some kind of disk I/O > error occurred” according to the comment. Which is true in this case; an I/O > operation returned an error. But the *disk* didn't - the *operating system* did, so if

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Keith Medcalf
SELECT DISTINCT * FROM table ORDER BY field1,field2,field3,field4 DESC,field5; to do the whole sorting and de-duplication in one step ... assuming you want to report duplicate entire rows only once ... --- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Warren Young
On Sep 26, 2017, at 11:24 AM, Ron Barnes wrote: > > I have approximately 600 million records that need to be sorted Where is the data now? > There are 18 table entries. You mean 18 columns per row, right? > I also need to deduplicate the records based upon the sorted output file. You speak o

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Simon Slavin
On 26 Sep 2017, at 6:24pm, Ron Barnes wrote: > I need to sort them as follows... > > Sort Field 1 Ascending > Sort Field 2 Ascending WITHIN field 1 > Sort Field 3 Ascending WITHIN field 2 WITHIN field 1 > Sort Field 4 Descending WITHIN field 3 WITHIN field 2 WITHIN field 1 <== This > is a Dat

[sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Ron Barnes
Hello All, I have approximately 600 million records that need to be sorted and then extracted to a flat file. I am unable to code a solution using visual Basic .NET. It was suggested to me that a DB engine could perform my task for me. Is there a way to accomplish this using the multi-level s

Re: [sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread Jens Alfke
> On Sep 25, 2017, at 4:39 AM, KRECKEL Richard (AREVA) > wrote: > > Remove the write permission of a SQLite database's journal file. Then, try > write-accessing the database. The error reported is "disk I/O error". (This > happened to me when two user tried to share a DB and had their umask

Re: [sqlite] ADO.NET driver upgrade puzzle

2017-09-26 Thread Barry Smith
> On 26 Sep 2017, at 12:14 am, Clemens Ladisch wrote: > > Roberts, Barry (FINTL) wrote: >> As per my original post, all C# access code is making extensive use of >> "using" statements. However we do obviously rely on the connection pool >> being thread safe, because many threads are writing to

Re: [sqlite] 'database disk image is malformed' only on the mac

2017-09-26 Thread David Raymond
The terminology for that option has always caught me out. The best phrase is the one below. http://www.sqlite.org/compile.html#threadsafe "When compiled with SQLITE_THREADSAFE=2, SQLite can be used in a multithreaded program so long as no two threads attempt to use the same database connection (

Re: [sqlite] All of a sudden I must commit after every single change and I don't know what happened.

2017-09-26 Thread R Smith
Web App you say I imagine you are using some wrapper, possibly a JAVA one, PHP or a PEARL one, I think your web service may have updated so the supported wrapper updated with possibly a new default setting or such. Those wrappers usually have a setting/property called "AutoCommit" which /

Re: [sqlite] All of a sudden I must commit after every single change and I don't know what happened.

2017-09-26 Thread Simon Slavin
Are the changes actually making it to the file on disk ? In other words, if you do _open(), INSERT, _close(), does the disk file get updated ? What PRAGMAs are you using ? Simon. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http:

Re: [sqlite] All of a sudden I must commit after every single change and I don't know what happened.

2017-09-26 Thread Richard Hipp
On 9/24/17, bensonbear wrote: > > But today, I found that a sequence of actions done in one function with one > database connection/cursor will not work because the later ones do not see > the changes of the earlier ones. Is my understanding correct that as long > as it is the same connection/cu

[sqlite] bug: failure to write journal reported as "disk I/O error"

2017-09-26 Thread KRECKEL Richard (AREVA)
Remove the write permission of a SQLite database's journal file. Then, try write-accessing the database. The error reported is "disk I/O error". (This happened to me when two user tried to share a DB and had their umask set wrong.) The error message reported by SQLite is inappropriate. A "perm

Re: [sqlite] 'database disk image is malformed' only on the mac

2017-09-26 Thread Fahad
I don't think so: https://sqlite.org/threadsafe.html "With -DSQLITE_THREADSAFE=2 the threading mode is multi-thread." Setting it to 0 disables all mutexes (assumes single threaded) So I've set it to be multi-threaded. Okay so I've wrapped @synchronized(..) around my database usage, stopped cach

[sqlite] All of a sudden I must commit after every single change and I don't know what happened.

2017-09-26 Thread bensonbear
I have an sqlite3 database I use for a web app that I am mostly the sole user of. It has been working fine for years, but all of a sudden today, I find that the app cannot insert and delete items from the database when it needs to. This is an app with a single thread, and for each command the w

Re: [sqlite] ADO.NET driver upgrade puzzle

2017-09-26 Thread Clemens Ladisch
Roberts, Barry (FINTL) wrote: > As per my original post, all C# access code is making extensive use of > "using" statements. However we do obviously rely on the connection pool > being thread safe, because many threads are writing to different > databases (connections) concurrently. > > There is no