Re: [sqlite] Importing a cvs file

2014-07-18 Thread RSmith
On 2014/07/19 03:10, Keith Medcalf wrote: Mostly the problems experienced by people is that they make some home- brew CSV importer that does not realise how to correctly read output from a standards-based exporter such as Excel, and then try to change things like separation or quoting methods to

Re: [sqlite] Importing a cvs file

2014-07-18 Thread Jonathan Leslie
I tested excel 2010 on a windows 7 machine 64 bit.   I entered the data, and did a "save as" choosing the type .csv MS-DOS version. All worked out with proper quotes for data fields, and quotes in the data field delimited.  On Friday, July 18, 2014 9:11 PM, Keith Medcalf wrote: > > >>Mostly

Re: [sqlite] Importing a cvs file

2014-07-18 Thread Keith Medcalf
>Mostly the problems experienced by people is that they make some home- >brew CSV importer that does not realise how to correctly read >output from a standards-based exporter such as Excel, and then try to >change things like separation or quoting methods to "fix" it >after the fact. Which version

Re: [sqlite] Testing performance hypothesis of scanning a table with no indexes versus reading same data from a file

2014-07-18 Thread Simon Slavin
On 18 Jul 2014, at 11:50pm, Richard Hipp wrote: > Hard to say. There are a lot of dependencies. A lot. There's no point in us publishing figures of any kind since they'll be useless for you. All someone could do is look at comparisons between the two things on one hardware setup. > In a t

[sqlite] YACC/Bison vs. Lemon vs. standard input

2014-07-18 Thread Ed Davis
I'm trying to convert a calculator from Bison to Lemon. I ran into an unexpected problem involving standard input where the two programs behave quite differently.  The Bison version prints the result immediately after pressing [Enter]. With the Lemon version, the result is delayed until I type a n

Re: [sqlite] Testing performance hypothesis of scanning a table with no indexes versus reading same data from a file

2014-07-18 Thread Richard Hipp
On Fri, Jul 18, 2014 at 6:44 PM, David Canterbrie wrote: > Hello, > > I've been tasked with trying to understand how much of a performance hit > one would get if one had to scan a table in its entirety versus reading the > same data stored as a new-line (or some sort like that) from a file. > > T

[sqlite] Testing performance hypothesis of scanning a table with no indexes versus reading same data from a file

2014-07-18 Thread David Canterbrie
Hello, I've been tasked with trying to understand how much of a performance hit one would get if one had to scan a table in its entirety versus reading the same data stored as a new-line (or some sort like that) from a file. The hypothesis I suppose we're trying to understand is that reading sequ

Re: [sqlite] Keeping a history of structured data

2014-07-18 Thread Tristan Van Berkom
On Thu, 2014-07-17 at 21:43 +0100, Simon Slavin wrote: On 17 Jul 2014, at 8:43pm, Tristan Van Berkom wrote: > The objective is to keep a revisioned history of 'E' whenever 'E' has > changed, or any of it's 'P' counterparts have changed, ideally without > storing a duplicate row for the one '

Re: [sqlite] Creating fiels

2014-07-18 Thread Rui Fernandes
Thank you very much Ryan :) I'll try it tomorrow. Kind regards, Miguel On Fri, Jul 18, 2014 at 10:37 PM, RSmith wrote: > > On 2014/07/18 22:56, Rui Fernandes wrote: > >> I already know how to import a csv file, and save it in SQLite format. >> But how can I define the time of variable in the

Re: [sqlite] Creating fiels

2014-07-18 Thread RSmith
On 2014/07/18 22:56, Rui Fernandes wrote: I already know how to import a csv file, and save it in SQLite format. But how can I define the time of variable in the fields since it assumes all of them are TEXT? I assume you mean "type" of variable (and not "time") - else see the reply from Mr. G

Re: [sqlite] Importing a cvs file

2014-07-18 Thread Jonathan Leslie
"My statements tested for accuracy, nice!" What, you thought I was gonna take a programmers word on it??? ;) Like I said, I never knew there was a formal way to handle the CSV character in the data field, this is far more a problem of the rules/protocol never being stated completely rather

Re: [sqlite] Creating fiels

2014-07-18 Thread Donald Griggs
Regarding "But how can I define the time of variable in the fields since it assumes all of them are TEXT?" In case you are asking about how to manipulate date and/or time data, see: http://www.sqlite.org/lang_datefunc.html Regards, Donald ___ sql

Re: [sqlite] Creating fiels

2014-07-18 Thread Kees Nuyt
On Fri, 18 Jul 2014 21:56:31 +0100, Rui Fernandes wrote: > I already know how to import a csv file, and save it in SQLite format. > But how can I define the time of variable in the fields since it assumes > all of them are TEXT? > > Where can I find this information? And do I create this before t

Re: [sqlite] Importing a cvs file

2014-07-18 Thread RSmith
On 2014/07/18 21:32, Jonathan Leslie wrote: Ryan, You are quite right, it is not a perfect solution. Thank you for your detailed explanation your's is a much more complete solution. I never knew about the quotes syntax being formal. that is very good to know. Sure enough, it works just as y

[sqlite] Creating fiels

2014-07-18 Thread Rui Fernandes
I already know how to import a csv file, and save it in SQLite format. But how can I define the time of variable in the fields since it assumes all of them are TEXT? Where can I find this information? And do I create this before the input - how? Kind regards, Miguel Fernandes ___

Re: [sqlite] Importing a cvs file

2014-07-18 Thread Jonathan Leslie
Ryan,  You are quite right, it is not a perfect solution.  Thank you for your detailed explanation your's is a much more complete solution.  I never knew about the   quotes syntax being formal. that is very good to know.   Sure enough, it works just as you described: 1ºfield 1ºfield 2ºthis is

Re: [sqlite] Sometimes when my process restarts, it returns error "database is locked"

2014-07-18 Thread Mayank Kumar (mayankum)
Thanks Simon. Now sqlite3_config is the first call we are making followed by sqlite3_open. I am checking the return code for all calls. In one of the sqlite3_step calls later, I am getting an error "file is encrypted or is not a database" but I don't see this callback getting called. Immediatel

Re: [sqlite] Vacuum command fails

2014-07-18 Thread Simon Slavin
On 18 Jul 2014, at 4:45pm, Carlos Ferreira wrote: > This is on a standard windows 7 64 bits laptop. > > I am not using a shell command. I am using the win32/win64 precompiled Dlls. Can you please check to see if using VACUUM in the shell tool has the same problem with the same database ? Thi

Re: [sqlite] Importing a cvs file

2014-07-18 Thread RSmith
On 2014/07/18 17:17, Rui Fernandes wrote: It's the header, right? Must contain the fields names. Am I right? Yes, definitely the header that is needed in your case. The rest all look perfect. The header should have 13 fields (that is 13 identifiers that are not duplicates and contain nothing

[sqlite] Importing CSV file

2014-07-18 Thread Rui Fernandes
Hi, I think I'm getting a bunch of attached fields. Everything is added to the same line, without separators. I've seen it when I read it. I've used: .mode csv table1 .import data.csv table1 .save Data.dat Kind regards, Miguel Fernandes ___ sqlite-us

Re: [sqlite] Importing a cvs file

2014-07-18 Thread Rui Fernandes
Hi, I think I'm getting a bunch of attached fields. Everything is added to the same line, without separators. I've seen it when I read it. Kind regards, Miguel Fernandes On Fri, Jul 18, 2014 at 4:49 PM, RSmith wrote: > > On 2014/07/18 16:53, Jonathan Leslie wrote: > >> I have a similar situa

Re: [sqlite] Importing a cvs file

2014-07-18 Thread RSmith
On 2014/07/18 16:53, Jonathan Leslie wrote: I have a similar situation about to happen. I also have the issue where one of the data fields is a text section that the user fills in and of course, he is free to put commas in this field. I believe that this will result in higgly-piggly (that's

Re: [sqlite] Vacuum command fails

2014-07-18 Thread Carlos Ferreira
Hi, This is on a standard windows 7 64 bits laptop. I am not using a shell command. I am using the win32/win64 precompiled Dlls. I will check the pragma integrity check... One thing I noticed before was that the maximum size for a blob inside a record field is much smaller than what I imagined.

Re: [sqlite] Vacuum command fails

2014-07-18 Thread Simon Slavin
On 18 Jul 2014, at 4:11pm, Carlos Ferreira wrote: > I also have a problem with VACCUM. ( both 32 and 64 bits ) > > Whenever I deal with small databases, it works fine, but as soon as the DB > is more than 200 Mb, the vaccum command bails out with code 21. > or Is this on a standard type of

Re: [sqlite] Importing a cvs file

2014-07-18 Thread Richard Hipp
On Fri, Jul 18, 2014 at 11:15 AM, Rui Fernandes wrote: > Hi, > > Thaks for the help, and the links > Here are the 2 first lines of the cvs file: > > Costa de Xurius,T.SLP,42.5,1.48333,AD,,,0,,Europe/Andorra,1.0,2.0,1.0 > Font de la Xona,H.SPNG,42.55003,1.44986,AD,Parroquia de la > Massana,,0,,Eur

Re: [sqlite] [bug] Optimizer chooses suboptimal plan for correlated subqueries

2014-07-18 Thread Richard Hipp
On Fri, Jul 18, 2014 at 10:58 AM, Stephen Broberg wrote: > > As for the .fullschema thing, a few questions: > > 1. Does just produce the DDL, or does it generate data as well? Just > the DDL in this case would not be particularly useful, as you'd need the db > populated to accurately reproduce

Re: [sqlite] Importing a cvs file

2014-07-18 Thread Rui Fernandes
It's the header, right? Must contain the fields names. Am I right? Kind regards, Miguel Fernandes On Fri, Jul 18, 2014 at 4:15 PM, Rui Fernandes wrote: > Hi, > > Thaks for the help, and the links > Here are the 2 first lines of the cvs file: > > Costa de Xurius,T.SLP,42.5,1.48333,AD,,,0,,Eur

Re: [sqlite] Importing a cvs file

2014-07-18 Thread Rui Fernandes
Hi, Thaks for the help, and the links Here are the 2 first lines of the cvs file: Costa de Xurius,T.SLP,42.5,1.48333,AD,,,0,,Europe/Andorra,1.0,2.0,1.0 Font de la Xona,H.SPNG,42.55003,1.44986,AD,Parroquia de la Massana,,0,,Europe/Andorra,1.0,2.0,1.0 I've used mode, and import. But I've got the e

Re: [sqlite] Vacuum command fails

2014-07-18 Thread Carlos Ferreira
Hi, I also have a problem with VACCUM. ( both 32 and 64 bits ) Whenever I deal with small databases, it works fine, but as soon as the DB is more than 200 Mb, the vaccum command bails out with code 21. or Any idea ? I can make my own copy and rename ..that is probably what Vaccum does...but

Re: [sqlite] Importing a cvs file

2014-07-18 Thread Jonathan Leslie
I have a similar situation about to happen.  I also have the issue where one of the data fields is a text section that the user fills in and of course, he is free to put commas in this field.   I believe that this will result in higgly-piggly (that's a technical term) when I do the import.   Wha

Re: [sqlite] Importing a cvs file

2014-07-18 Thread Richard Hipp
On Fri, Jul 18, 2014 at 10:23 AM, Rui Fernandes wrote: > Greetings from Portugal, > > I made my firt import of a cvs file to the SQLite, and save it as a file. > My newbie question is that Sqlite is assuming every line as a record of > text, not separating the fields > Must the text fiels be

Re: [sqlite] Importing a cvs file

2014-07-18 Thread RSmith
On 2014/07/18 16:23, Rui Fernandes wrote: Greetings from Portugal, Olá Rui, I made my firt import of a cvs file to the SQLite, and save it as a file. My newbie question is that Sqlite is assuming every line as a record of text, not separating the fields Must the text fiels be surrounded b

Re: [sqlite] Vacuum command fails

2014-07-18 Thread RSmith
On 2014/07/18 16:30, Nelson, Erik - 2 wrote: veeresh kumar wrote: Now I am running the the application on D: Drive (it has 841 GB free space) and C: drive has space (333 GB free). It looks like VACUUM uses C:drive space ??? Also command seems to be working fine... Maybe I'm missing some

Re: [sqlite] Vacuum command fails

2014-07-18 Thread Nelson, Erik - 2
veeresh kumar wrote: > Now I am running the the application on D: Drive (it has 841 GB free > space) and C: drive has space (333 GB free). It looks like VACUUM uses > C:drive space ??? Also command seems to be working fine... > Maybe I'm missing something, but I felt like this was pretty clea

[sqlite] Importing a cvs file

2014-07-18 Thread Rui Fernandes
Greetings from Portugal, I made my firt import of a cvs file to the SQLite, and save it as a file. My newbie question is that Sqlite is assuming every line as a record of text, not separating the fields Must the text fiels be surrounded by "? And it will assume the . as decimal point? Another

Re: [sqlite] [bug] Optimizer chooses suboptimal plan for correlated subqueries

2014-07-18 Thread Richard Hipp
Thanks for the detailed report. We always like reproducible test cases! Having not yet looked into your problem, let me first bring out a couple of *minor* details: (1) We prefer to reserve the word "bug" for situations where it gets the wrong answer. Getting the correct answer more slowly tha

[sqlite] [bug] Optimizer chooses suboptimal plan for correlated subqueries

2014-07-18 Thread Stephen Broberg
Hi, We encountered an issue in our system where we had two nearly identical subqueries in a statement, which differed only in one table (both of which had the same definition). One table had 4 rows, the other table had zero rows. The subquery with the 4-row table ran about 10,000 times faster

[sqlite] Curious re: autoindexing v. index creation

2014-07-18 Thread Isaac Hodes
Hello SQLiters, I've recently run into a couple things which I found odd with respect to how SQLite decided to execute a particular query (and those with some similar characteristics). These oddities appear in both in how the query planner works and in the generated opcodes. I profess the opposite

Re: [sqlite] Vacuum command fails

2014-07-18 Thread veeresh kumar
Hi Simon,, Happy to provide you with answers...Answers Inline... One thing i would like to know is does the VACUUM command uses C: drive space or the drive where my application is running...It seems like it requires C: drive to have enough space. On Thursday, 17 July 2014 2:14 PM, Simon Sla

Re: [sqlite] [bug] sqlite memory corruption (use by digikam)

2014-07-18 Thread Mathieu Clabaut
Thank you Dan, as it is well behind my understanding of Digikam, I've asked one of the lead developper to have a look upon your answer and its implications. Best regards, -Mathieu 2014-07-15 22:27 GMT+02:00 Dan Kennedy : > On 07/16/2014 03:22 AM, Dan Kennedy wrote: > >> On 07/15/2014 09:06 P

Re: [sqlite] An Explain Query Plan that I cannot explain

2014-07-18 Thread Staffan Tylen
Thanks for the tip, ANALYZE is the cause. Without it the index will be searched. Staffan On Thu, Jul 17, 2014 at 7:38 PM, Simon Slavin wrote: > > On 17 Jul 2014, at 6:34pm, Staffan Tylen wrote: > > > I'm unable to understand why T2 isn't using the index. Is it the fact > that > > the table T

Re: [sqlite] sqlite & multi process usage

2014-07-18 Thread Howard Chu
Richard Hipp wrote: On Thu, Jul 17, 2014 at 5:10 AM, Micka wrote: Well, All of my process are using Mutex to protect sqlite from concurrent access. What kind of mutex are you using that works across processs? All the mutexes I know about only work for a single process. pthreads supports

Re: [sqlite] sqlite & multi process usage

2014-07-18 Thread Micka
Sorry, I mean semaphore => sem_open(pszSemaphoreName, O_CREAT, S_IRWXU | S_IRWXG | S_IRWXO, 0); On Fri, Jul 18, 2014 at 11:45 AM, Richard Hipp wrote: > On Thu, Jul 17, 2014 at 5:10 AM, Micka wrote: > > > Well, > > > > All of my process are using Mutex to protect sqlite from concurrent > access

Re: [sqlite] sqlite & multi process usage

2014-07-18 Thread Richard Hipp
On Thu, Jul 17, 2014 at 5:10 AM, Micka wrote: > Well, > > All of my process are using Mutex to protect sqlite from concurrent access. > What kind of mutex are you using that works across processs? All the mutexes I know about only work for a single process. -- D. Richard Hipp d...@sqlite.org