Re: [sqlite] Does sqlite has db file-size restriction on Solaris 10?
Hi, Simon, The running server and sqlite database are run in same server; it is not access by network. It is local query and writes. Regards, Liang Kunming. -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Simon Slavin Sent: 2013年11月25日 10:27 To: General Discussion of SQLite Database Subject: Re: [sqlite] Does sqlite has db file-size restriction on Solaris 10? On 25 Nov 2013, at 2:08am, Liang Kunming wrote: > Based on my test, create a >3GB files on my Solaris 10 is ok, do you have any > others suggestions? Sorry, no. I am intrigued, however, but the fact that your trace shows IP calls when you didn't mention you were accessing the storage medium over a network. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Does sqlite has db file-size restriction on Solaris 10?
On 25 Nov 2013, at 2:08am, Liang Kunming wrote: > Based on my test, create a >3GB files on my Solaris 10 is ok, do you have any > others suggestions? Sorry, no. I am intrigued, however, but the fact that your trace shows IP calls when you didn't mention you were accessing the storage medium over a network. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Does sqlite has db file-size restriction on Solaris 10?
Hi, Simon, Based on my test, create a >3GB files on my Solaris 10 is ok, do you have any others suggestions? bash-3.00# du -hs logs/* 7.0G logs/SilentAlarm.log 25K logs/SilentAlarmError.log 10M logs/systemErr1.txt 4.4M logs/systemErr10.txt 10M logs/systemErr2.txt 10M logs/systemErr3.txt 10M logs/systemErr4.txt 10M logs/systemErr5.txt 10M logs/systemErr6.txt 10M logs/systemErr7.txt 10M logs/systemErr8.txt 10M logs/systemErr9.txt 256K logs/systemOut1.txt bash-3.00# ls -lt database/* -rw-r--r-- 1 root root1544 Nov 25 15:25 database/silentalarm.db-journal -rw-r--r-- 1 root root 2147483648 Nov 25 15:25 database/silentalarm.db bash-3.00# ulimit -a core file size(blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited open files(-n) 8192 pipe size (512 bytes, -p) 10 stack size(kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes(-u) 29995 virtual memory(kbytes, -v) unlimited bash-3.00# Regards, Liang Kunming. -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Liang Kunming Sent: 2013年11月23日 9:52 To: General Discussion of SQLite Database Subject: Re: [sqlite] Does sqlite has db file-size restriction on Solaris 10? Hi, Simon, I use the 64bit Solaris 10 and the fs type is ufs. The trace as below: bash-3.00# df -v Mount Dir Filesystemblocks used free %used /dev/dsk/c0t5000CCA01286BB88d0s0 / 94601683 4847315 88808352 6% /devices /devices 000 0% /system/co ctfs 000 0% /proc proc 000 0% /etc/mntta mnttab 000 0% /etc/svc/v swap 4676759 211 4676548 1% /system/ob objfs 000 0% /etc/dfs/s sharefs000 0% /platform/sun4v/lib/libc_psr/libc_psr_hwcap2.so.1 /platform/ 94601683 4847315 88808352 6% /platform/sun4v/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1 /platform/ 94601683 4847315 88808352 6% /dev/fdfd 000 0% /tmp swap 4679532 2984 4676548 1% /var/run swap 4676558 10 4676548 1% /dev/dsk/c0t5000CCA01286BB88d0s1 /opt 161363185 80324957 7942459751% bash-3.00# fstyp /dev/dsk/c0t5000CCA01286BB88d0s1 ufs bash-3.00# isainfo -v 64-bit sparcv9 applications hpc vis3 fmaf asi_blk_init vis2 vis popc 32-bit sparc applications hpc vis3 fmaf asi_blk_init vis2 vis popc v8plus div32 mul32 I will test the case if can create a 3GB file on this file system and share the result here. Thanks. Regards, Liang Kunming. -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Simon Slavin Sent: 2013年11月23日 9:23 To: General Discussion of SQLite Database Subject: Re: [sqlite] Does sqlite has db file-size restriction on Solaris 10? On 23 Nov 2013, at 1:03am, Liang Kunming wrote: > I meet some issue when use the sqlite on Solaris 10. The db file is made by > the sqlite R3.4.2 version and the sqlite3 is compiled on Solaris 10 platform > (has attached). When the db file meet 2147483648 bytes (2Gigabytes), the file > size can not increase anymore, and query/write also error. When query or > write the data, the exception as below. Who know the solution of this issue > and can share me, thanks very much. Your trace indicates that you are using a network file system (perhaps the one called NFS) to access the drive your database is on. Which file system and/or network file system are you using to access that drive ? Can you please test that it is possible to make a 3GB file of any kind on that drive. A long text file would do fine, but not a file completely filled with hex zeros (0x00). Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite version 3.8.2 running 2.5x faster for some queries.
On Sun, 24 Nov 2013 16:53:01 +0200 RSmith wrote: > Similarly if one was to add a function which returns a date-dependant > value, such as 'dayOfWeek(x)' and mark it as deterministic for inside > a single query, that would make sense, even though it would be very > indeterministic (or referentially opaque, if you will) between > queries. This is all dandy unless you have queries (as seen on this > forum) that run for three days or more... would it matter then? A SQL statement is atomic irrespective of how long it runs. Your wday() function would return to the day of the week as of the time the statement was evaluated, even if it took a month of Sundays to execute. ;-) Another way to think about it: a deterministic function always returns the same output for a given input. In an SQL statement, a function is provided input only once, regardless of how long it runs. --jkl ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite version 3.8.2 running 2.5x faster for some queries.
On Sun, 24 Nov 2013 09:17:20 -0500 Doug Currie wrote: > in computer science we have referential transparency > > http://en.wikipedia.org/wiki/Referential_transparency_ > (computer_science) > > and pure functions > > http://en.wikipedia.org/wiki/Pure_function https://en.wikipedia.org/wiki/Deterministic_algorithm In computer science we also have deterministic and nondeterministic. Those are also IIRC the terms used in the SQL standard. --jkl ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite version 3.8.2 running 2.5x faster for some queries.
On Nov 24, 2013, at 7:10 PM, Valentin Davydov wrote: > Wait a second... and you'll get different value of datetime('now'). In this > sense datetime() is as deterministic as random(): it may give the same result > next invocation or may not, dependng on various circumstances not related to > the function itself. Nah… in SQL… ‘now’ is supposed to be computed once per statement. And that’s all. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite version 3.8.2 running 2.5x faster for some queries.
On Sat, Nov 23, 2013 at 06:18:29AM -0500, Richard Hipp wrote: > On Sat, Nov 23, 2013 at 5:26 AM, Pepijn Van Eeckhoudt < > pep...@vaneeckhoudt.net> wrote: > > > Is datetime special in thuis context or will constant expression hoisting > > like this happen for any function? > > SQLite must know that the function always gives the same output given the > same inputs. No every function works that way. Counterexamples include > random() and last_insert_rowid(). But most built-in functions are > factorable in the same way that datetime() is. Wait a second... and you'll get different value of datetime('now'). In this sense datetime() is as deterministic as random(): it may give the same result next invocation or may not, dependng on various circumstances not related to the function itself. > Currently there is no API to designate an application-defined function as > being "constant" in the sense that it always generates the same output > given the same inputs. Hence, SQLite assumes the worst about > application-defined functions and never tries to factor them out of the > inner loop. Probably we should add a new API that allows the application > to state auxiliary properties about application-defined functions (such as > whether or not it is "constant", whether or not it can return NULL, whether > or not it might change the encoding of its input parameters, etc.) But > that has not been done yet. To my opinion, the most general solution is to let to the application programmer to decide whether to calcucale the function once (say, at the beginning of a transaction), store the result and then access the stored value, or to make the new call to the function each iteration, depending on the application semantics. Valentin Davydov. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Insert with multiple nested selects.
> Perhaps: > > INSERT OR IGNORE INTO table_a > ( > col_a, > col_b, > col_c, > col_d > ) VALUES ( >(SELECT id FROM table_b WHERE name=?) > ,? > ,? > ,(SELECT id FROM table_c WHERE name=?) > ); Hah, I need some time away from this one, not sure what I was thinking my nested selects... Thanks Kees, jlc ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Insert with multiple nested selects.
On Sun, 24 Nov 2013 16:10:58 +, "Joseph L. Casale" wrote: >Hey guys, >Trying to create a statement for use with parameters in a Python execute method >when performing inserts with multiple nested selects. I can adjust it for use >with Python, but I am having issues when there is more than one nested select. > >Something such as: > >INSERT OR IGNORE INTO table_a >( >col_a, >col_b, >col_c, >col_d >) >SELECT col_a FROM (SELECT id FROM table_b WHERE name=?) >,? >,? >,SELECT col_d FROM (SELECT id FROM table_c WHERE name=?); > >Anyone have a hint on how to perform such as query? Perhaps: INSERT OR IGNORE INTO table_a ( col_a, col_b, col_c, col_d ) VALUES ( (SELECT id FROM table_b WHERE name=?) ,? ,? ,(SELECT id FROM table_c WHERE name=?) ); >Thanks, >jlc HTH -- Groet, Cordialement, Pozdrawiam, Regards, Kees Nuyt ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Insert with multiple nested selects.
On 24-11-2013 17:10, Joseph L. Casale wrote: Hey guys, Trying to create a statement for use with parameters in a Python execute method when performing inserts with multiple nested selects. I can adjust it for use with Python, but I am having issues when there is more than one nested select. Something such as: INSERT OR IGNORE INTO table_a ( col_a, col_b, col_c, col_d ) SELECT col_a FROM (SELECT id FROM table_b WHERE name=?) ,? ,? ,SELECT col_d FROM (SELECT id FROM table_c WHERE name=?); Anyone have a hint on how to perform such as query? sqlite> create table test (a integer, b integer, c integer); sqlite> insert into test (a,b,c) values (1,2,3); sqlite> insert into test (a,b,c) values (select 4,2,3); Error: near "select": syntax error sqlite> insert into test (a,b,c) values ((select 4),2,3); sqlite> select * from test; 1|2|3 4|2|3 sqlite> ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Insert with multiple nested selects.
Hey guys, Trying to create a statement for use with parameters in a Python execute method when performing inserts with multiple nested selects. I can adjust it for use with Python, but I am having issues when there is more than one nested select. Something such as: INSERT OR IGNORE INTO table_a ( col_a, col_b, col_c, col_d ) SELECT col_a FROM (SELECT id FROM table_b WHERE name=?) ,? ,? ,SELECT col_d FROM (SELECT id FROM table_c WHERE name=?); Anyone have a hint on how to perform such as query? Thanks, jlc ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite version 3.8.2 running 2.5x faster for some queries.
On Sun, Nov 24, 2013 at 4:30 PM, Petite Abeille wrote: > > On Nov 24, 2013, at 3:17 PM, Doug Currie wrote: > > > There is value in compatibility, but those adjectives are awful. > > FWIW, DETERMINISTIC is what Oracle uses: > > > http://docs.oracle.com/cd/E11882_01/appdev.112/e25519/function.htm#LNPLS1183 > There's also this: http://docs.oracle.com/cd/B13789_01/appdev.101/b10807/13_elems039.htm that specifies various levels of "purity", i.e. if the package function reads from or writes to the database or global state. Each level allows for more aggressive optimization when the function takes part in a query. It's been a while since I've coded in PL/SQL but I think that if the RESTRICT_REFERENCES pragma is not specified for a package function the SQL engine will not accept calls to it from an SQL statement. Also, if the implementation of a function violates its RESTRICT_REFERENCES pragma the PL/SQL compiler will not compile it. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite version 3.8.2 running 2.5x faster for some queries.
Ugh, my last thought was not well-formed - apologies. When I said: "...can add a function to replace an SQL function to improve it many times for the specific purpose". This would of course hardly matter if the SQL (or SQLite specifically) function was already deterministic (read: cached). My thinking was more towards enforcing determinism on something that isn't usually, if it does not affect your query's outcome (i.e specific purpose). Hope that is more clear - thanks. Ryan ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite version 3.8.2 running 2.5x faster for some queries.
Agreed - also some functions might not be intrinsically deterministic, but it may well be so for the duration of a query. There may need to be some thinking on this. I refer back to a discussion earlier (and subsequent SQLite adaption) which made a date-time reference deterministic within a single query for transactional integrity. Similarly if one was to add a function which returns a date-dependant value, such as 'dayOfWeek(x)' and mark it as deterministic for inside a single query, that would make sense, even though it would be very indeterministic (or referentially opaque, if you will) between queries. This is all dandy unless you have queries (as seen on this forum) that run for three days or more... would it matter then? I for one would very much like the ability to specify added functions as deterministic or not (or whatever term would indicate 'cacheability of the result') as this might be an enormous efficiency improvement in itself, and also allow you to now add things to improve speed in some standard queries. Think of the people who have queries running for ages and can add a function to replace an SQL function to improve it many times for the specific purpose.* Actually, thinking about it, this was always achievable with some clever programming and managing the result cache yourself - but I still think this addition would be an improvement. That's my 2 cents. Have a great day all! Ryan *This is not to say the SQL way is not good, but it often has to cater for a wide variety of things where a user-added function might be very tuned to a specific purpose - ridding a lot of CPU-time fat. On 2013/11/24 16:30, Petite Abeille wrote: On Nov 24, 2013, at 3:17 PM, Doug Currie wrote: There is value in compatibility, but those adjectives are awful. FWIW, DETERMINISTIC is what Oracle uses: http://docs.oracle.com/cd/E11882_01/appdev.112/e25519/function.htm#LNPLS1183 I would personally stick to that if such functionality was ever introduced in SQLite. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite version 3.8.2 running 2.5x faster for some queries.
On Nov 24, 2013, at 3:17 PM, Doug Currie wrote: > There is value in compatibility, but those adjectives are awful. FWIW, DETERMINISTIC is what Oracle uses: http://docs.oracle.com/cd/E11882_01/appdev.112/e25519/function.htm#LNPLS1183 I would personally stick to that if such functionality was ever introduced in SQLite. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite version 3.8.2 running 2.5x faster for some queries.
On Nov 24, 2013, at 6:47 AM, Alek Paunov wrote: > > BTW, I see the term "deterministic" in the SQL99 BNFs: > … > but different in PostgreSQL ("immutable", "stable", etc): There is value in compatibility, but those adjectives are awful. In computer science we have referential transparency http://en.wikipedia.org/wiki/Referential_transparency_(computer_science) and pure functions http://en.wikipedia.org/wiki/Pure_function e ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Fw: Timestamps from different timezones
Hello Christopher, > You absolutely cannot do this automatically unless you know > the source of > each timestamp, any maybe not even then. > Just as a f'rinstance, there are at least 3 different meanings > for EST, and > 2 of them are in the same country: > try -0500, +1000, +1100 for a start. Thanks for your reply. I was afraid that this would be the answer... Thanks again bernie >-Original Message- >From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users- >boun...@sqlite.org] On Behalf Of Bernie Reiter >Sent: Saturday, 23 November, 2013 16:54 >To: sqlite-users@sqlite.org >Subject: [sqlite] Timestamps from different timezones > >Dear List, > >I receive data records from various sources. I can't influence their >formats. These data records are containing datetimestamps like this >(without the doublequotes): > >"06-May-12 18:57:41 BST" >"Nov-22-13 22:58:10 PST" >"23-Nov-13 08:56:57 GMT" >"22.11.13 00:33:32 MEZ" >"23-Nov-13 18:57:40 AEDST" >"23-Nov-13 01:58:10 EST" > >I want to automatically convert these datetimestamps into standard UTC. > >Currently I am using a "manual approach" by: >a) converting the datetimestamp into the common format " -MM- >DDTHH:MM" >b) reading the time zone code (e.g., BST) >c) looking-up the timezone indicator for this time zone code (e.g. - >01:00) >(see paragraph "Formats 2 through 10 may be optionally followed by a >timezone indicator..." >on page http://sqlite.org/lang_datefunc.html ) >d) and appending this indicator to an SQL statement. > >My SQL statements look like this: >SELECT datetime('2012-05-06T18:57:41-01:00'); > >Is there a clever way to convert these datetimestamps into standard UTC ? > > >I don't want to "mess around" with the local timezone of the server >on which sqlite3 is running. This approach is not portable. > >The Unix/Linux 'date' utility is not portable either. > >Thanks a lot > >bernie >___ >sqlite-users mailing list >sqlite-users@sqlite.org >http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Timestamps from different timezones
Hello Keith, Thanks for your reply. > The select returns a timestamp in UTC, not in localtime. > If you want localtime you need to specify that: > SELECT datetime('2012-05-06T18:57:41-01:00', 'localtime'); Yes, I do want timestamps in UTC only. Otherwise the same SELECT would return a different value when run on a server located in the UK compared to one located in the USA. => Not portable. > or do you mean something like > SELECT strftime('%s', '2012-05-06T18:57:41-01:00'); > which returns the number of seconds since the unixepoch? No, because what is principally the difference when I store a datetime value in a SQLite field whether it is expressed as UTC time or unixepoch time based on UTC? My application still would need to remember that this field does contain a datetime value based on UTC... Thanks again bernie On Sunday, 24 November 2013, 5:51, Keith Medcalf wrote: >My SQL statements look like this: >SELECT datetime('2012-05-06T18:57:41-01:00'); > >Is there a clever way to convert these datetimestamps into standard UTC ? The select returns a timestamp in UTC, not in localtime. If you want localtime you need to specify that: SELECT datetime('2012-05-06T18:57:41-01:00', 'localtime'); or do you mean something like SELECT strftime('%s', '2012-05-06T18:57:41-01:00'); which returns the number of seconds since the unixepoch? >-Original Message- >From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users- >boun...@sqlite.org] On Behalf Of Bernie Reiter >Sent: Saturday, 23 November, 2013 16:54 >To: sqlite-users@sqlite.org >Subject: [sqlite] Timestamps from different timezones > >Dear List, > >I receive data records from various sources. I can't influence their >formats. These data records are containing datetimestamps like this >(without the doublequotes): > >"06-May-12 18:57:41 BST" >"Nov-22-13 22:58:10 PST" >"23-Nov-13 08:56:57 GMT" >"22.11.13 00:33:32 MEZ" >"23-Nov-13 18:57:40 AEDST" >"23-Nov-13 01:58:10 EST" > >I want to automatically convert these datetimestamps into standard UTC. > >Currently I am using a "manual approach" by: >a) converting the datetimestamp into the common format " -MM- >DDTHH:MM" >b) reading the time zone code (e.g., BST) >c) looking-up the timezone indicator for this time zone code (e.g. - >01:00) >(see paragraph "Formats 2 through 10 may be optionally followed by a >timezone indicator..." >on page http://sqlite.org/lang_datefunc.html ) >d) and appending this indicator to an SQL statement. > >My SQL statements look like this: >SELECT datetime('2012-05-06T18:57:41-01:00'); > >Is there a clever way to convert these datetimestamps into standard UTC ? > > >I don't want to "mess around" with the local timezone of the server >on which sqlite3 is running. This approach is not portable. > >The Unix/Linux 'date' utility is not portable either. > >Thanks a lot > >bernie >___ >sqlite-users mailing list >sqlite-users@sqlite.org >http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite version 3.8.2 running 2.5x faster for some queries.
On 23.11.2013 13:18, Richard Hipp wrote: SQLite must know that the function always gives the same output given the same inputs. No every function works that way. Counterexamples include random() and last_insert_rowid(). But most built-in functions are factorable in the same way that datetime() is. BTW, I see the term "deterministic" in the SQL99 BNFs: http://savage.net.au/SQL/sql-99.bnf.html#deterministic%20characteristic http://savage.net.au/SQL/sql-2003-2.bnf.html#deterministic%20characteristic aslo found in MySQL: http://dev.mysql.com/doc/refman/5.6/en/create-procedure.html but different in PostgreSQL ("immutable", "stable", etc): http://www.postgresql.org/docs/9.3/static/sql-createfunction.html I think "deterministic" is used also in the Prolog, whit the same meaning. Kind Regards, Alek ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users