Re: [sqlite] SQLite really SLOW on Solaris?
Hi Eric, Hi Roger, Interesting, I found the issue with the slow solaris issue. You see in my queue implementation I add in a: sqlite3_sleep(1); Essentially I wanted to sleep for 1 millisecond between sends to the queue (in the send function itself), so that if the sender is sending too fast it allows receivers to get messages On Debian it sleeps for 1 millisecond, BUT on the Solaris VM it sleeps for 1 second!!! Not one millisecond..why??? Why would that be? Could it be that the OS clocks are out? The moment I took this out of the code on Solaris it was super fast just like it was on Debian. Any ideas? Lynton -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Eric Smith Sent: 20 November 2010 09:13 PM To: General Discussion of SQLite Database Subject: Re: [sqlite] SQLite really SLOW on Solaris? Lynton Grice wrote: Many thanks for your feedback, much appreciated ;-) But why would that happen on Solaris and not Debian? Did you try it on real OSs not VMs? Are you sure the host OS is giving the same compute disk slices to the VMs? Any other differences in those mappings? Are the VMs running on the same host box? If not, are they running on the same hardware and host OS? Along the lines of Roger's comment, some VM implementations have OS clocks that simply don't work right -- they run too fast or too slow (I have personally observed differences of up to 3x), because they assume they are on real hardware. Have you tested that? Did you try compiling it with profiling data turned on to see where the bottlenecks are (using gcc's -pg flag plus gprof(1))? Or, as a simpler test, you could run your app from strace(1) (assuming solaris has such a thing) to see if you are spending loads of time in system calls as Roger hypothesized. Also, how would I pass the -D flags? Do I do that when I compile SQLite? Yes -- those are arguments to GCC that are similar to writing a #define compiler directive into your code. So saying gcc -DFOO=bar ... would be like writing #define FOO bar into the code. The sqlite.org docs describe what they do pretty well iirc. Eric -- Eric A. Smith You can observe a lot by watching. -- Yogi Berra ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite really SLOW on Solaris?
On 21 Nov 2010, at 10:10am, Lynton Grice wrote: in my queue implementation I add in a: sqlite3_sleep(1); Essentially I wanted to sleep for 1 millisecond between sends to the queue (in the send function itself), so that if the sender is sending too fast it allows receivers to get messages On Debian it sleeps for 1 millisecond, BUT on the Solaris VM it sleeps for 1 second!!! Not one millisecond..why??? See http://www.sqlite.org/c3ref/sleep.html Apparently the Solaris implementation of xSleep() is weird. Just out of interest, do int theReturn = sqlite3_sleep(1); (or some equivalent) on both platforms and look at the values you get returned. Do you get 1 on one platform and 1000 on the other ? Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite really SLOW on Solaris?
Hi Simon, Yup, on Solaris I ran this and it was 1000 milliseconds for the following: int theReturn = sqlite3_sleep(1); Any way you know I can make it sleep on Solaris for 1 millisecond? Any other C tricks you know of? Thanks ;-) Lynton -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Simon Slavin Sent: 21 November 2010 12:23 PM To: General Discussion of SQLite Database Subject: Re: [sqlite] SQLite really SLOW on Solaris? On 21 Nov 2010, at 10:10am, Lynton Grice wrote: in my queue implementation I add in a: sqlite3_sleep(1); Essentially I wanted to sleep for 1 millisecond between sends to the queue (in the send function itself), so that if the sender is sending too fast it allows receivers to get messages On Debian it sleeps for 1 millisecond, BUT on the Solaris VM it sleeps for 1 second!!! Not one millisecond..why??? See http://www.sqlite.org/c3ref/sleep.html Apparently the Solaris implementation of xSleep() is weird. Just out of interest, do int theReturn = sqlite3_sleep(1); (or some equivalent) on both platforms and look at the values you get returned. Do you get 1 on one platform and 1000 on the other ? Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite really SLOW on Solaris?
On Sun, 21 Nov 2010 13:08:31 +0200, Lynton Grice lynton.gr...@logosworld.com wrote: Hi Simon, Yup, on Solaris I ran this and it was 1000 milliseconds for the following: int theReturn = sqlite3_sleep(1); Any way you know I can make it sleep on Solaris for 1 millisecond? Any other C tricks you know of? Standard C Library Functions usleep(3C) NAME usleep - suspend execution for interval in microseconds SYNOPSIS #include unistd.h int usleep(useconds_t useconds); DESCRIPTION The usleep() function suspends the caller from execution for the number of microseconds specified by the useconds argu- ment. The actual suspension time might be less than requested because any caught signal will terminate usleep() following execution of that signal's catching routine. The suspension time might be longer than requested by an arbi- trary amount because of the scheduling of other activity in the system. If the value of useconds is 0, then the call has no effect. The use of the usleep() function has no effect on the action or blockage of any signal. In a multithreaded process, only the invoking thread is suspended from execution. RETURN VALUES On completion, usleep() returns 0. There are no error returns. ERRORS No errors are returned. USAGE The usleep() function is included for its historical usage. The nanosleep(3C) function is preferred over this function. -- ( Kees Nuyt ) c[_] ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite really SLOW on Solaris?
Thanks Kees, much appreciated ;-) -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Kees Nuyt Sent: 21 November 2010 01:47 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] SQLite really SLOW on Solaris? On Sun, 21 Nov 2010 13:08:31 +0200, Lynton Grice lynton.gr...@logosworld.com wrote: Hi Simon, Yup, on Solaris I ran this and it was 1000 milliseconds for the following: int theReturn = sqlite3_sleep(1); Any way you know I can make it sleep on Solaris for 1 millisecond? Any other C tricks you know of? Standard C Library Functions usleep(3C) NAME usleep - suspend execution for interval in microseconds SYNOPSIS #include unistd.h int usleep(useconds_t useconds); DESCRIPTION The usleep() function suspends the caller from execution for the number of microseconds specified by the useconds argu- ment. The actual suspension time might be less than requested because any caught signal will terminate usleep() following execution of that signal's catching routine. The suspension time might be longer than requested by an arbi- trary amount because of the scheduling of other activity in the system. If the value of useconds is 0, then the call has no effect. The use of the usleep() function has no effect on the action or blockage of any signal. In a multithreaded process, only the invoking thread is suspended from execution. RETURN VALUES On completion, usleep() returns 0. There are no error returns. ERRORS No errors are returned. USAGE The usleep() function is included for its historical usage. The nanosleep(3C) function is preferred over this function. -- ( Kees Nuyt ) c[_] ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite really SLOW on Solaris?
On 21 Nov 2010, at 11:46am, Kees Nuyt wrote: lynton.gr...@logosworld.com wrote: Hi Simon, Yup, on Solaris I ran this and it was 1000 milliseconds for the following: int theReturn = sqlite3_sleep(1); Any way you know I can make it sleep on Solaris for 1 millisecond? Any other C tricks you know of? Standard C Library Functions usleep(3C) NAME usleep - suspend execution for interval in microseconds Good note. But the original question remains: why does the SQLite VM under Solaris not implement sqlite3_sleep() with a resolution of at least milliseconds ? It's not like Solaris is hampered by requirements for a tiny library ! I found some code in 'os_unix.c' that says static int unixSleep(sqlite3_vfs *NotUsed, int microseconds){ #if OS_VXWORKS [...] #elif defined(HAVE_USLEEP) HAVE_USLEEP which suggests that OS_VXWORKS and HAVE_USLEEP are consulted to work out what resolution is available. But then we descend into compile-time parameters and I don't know anything about them. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] SQLite really SLOW on Solaris?
Hi there, I compiled SQLite on Debain (VM) first using the following: /usr/local/bin/gcc -c -fPIC sqlite3.c /usr/local/bin/gcc -shared -o libsqlite3.so sqlite3.o And then I created my own shared lib using SQLite and when I use the lib on Debian it is SUPER FAST. Then I installed SQLIte on Solaris (VM) using the same compile commands above. I copied my shared lib code from Debian to Solaris and ran the same test code.and it is SO SLOWlike ridiculously slow. On Debian it processes a couple hundred messages per second, on Solaris using the SAME code it does like ONE message per second. What could be wrong here? Must I change the compile options on Solaris? Any ideas? Chat later Lynton ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite really SLOW on Solaris?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/20/2010 07:12 AM, Lynton Grice wrote: On Debian it processes a couple hundred messages per second, on Solaris using the SAME code it does like ONE message per second. One possible cause is the busy handler but this only applies if you have concurrent access to the file and have set a busy timeout. The default SQLite busy handler keeps trying again and if it fails sleeps, repeating the process. It will sleep for longer and longer periods until the overall busy timeout period has expired. If you do not provide any -D flags then the call used to do the sleeping has a granularity of one second - ie even if SQLite wants to sleep for 50ms it will sleep for one second. If you have the usleep() function and supply - -DHAVE_USLEEP then microsecond resolution sleeps are available. These are the sleep intervals in milliseconds that sqliteDefaultBusyCallback tries when USLEEP is available: { 1, 2, 5, 10, 15, 20, 25, 25, 25, 50, 50, 100 } Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzoBeIACgkQmOOfHg372QTXHQCfQ3HIoXF5FcqP9AttZ7gNdPEZ IKYAoMVuIqoSa4e08nsbVQk5ABWZdq1G =+TOO -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite really SLOW on Solaris?
Hi Roger, Many thanks for your feedback, much appreciated ;-) But why would that happen on Solaris and not Debian? Also, how would I pass the -D flags? Do I do that when I compile SQLite? Thanks again Lynton -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Roger Binns Sent: 20 November 2010 07:31 PM To: General Discussion of SQLite Database Subject: Re: [sqlite] SQLite really SLOW on Solaris? -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/20/2010 07:12 AM, Lynton Grice wrote: On Debian it processes a couple hundred messages per second, on Solaris using the SAME code it does like ONE message per second. One possible cause is the busy handler but this only applies if you have concurrent access to the file and have set a busy timeout. The default SQLite busy handler keeps trying again and if it fails sleeps, repeating the process. It will sleep for longer and longer periods until the overall busy timeout period has expired. If you do not provide any -D flags then the call used to do the sleeping has a granularity of one second - ie even if SQLite wants to sleep for 50ms it will sleep for one second. If you have the usleep() function and supply - -DHAVE_USLEEP then microsecond resolution sleeps are available. These are the sleep intervals in milliseconds that sqliteDefaultBusyCallback tries when USLEEP is available: { 1, 2, 5, 10, 15, 20, 25, 25, 25, 50, 50, 100 } Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzoBeIACgkQmOOfHg372QTXHQCfQ3HIoXF5FcqP9AttZ7gNdPEZ IKYAoMVuIqoSa4e08nsbVQk5ABWZdq1G =+TOO -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite really SLOW on Solaris?
Lynton Grice wrote: Many thanks for your feedback, much appreciated ;-) But why would that happen on Solaris and not Debian? Did you try it on real OSs not VMs? Are you sure the host OS is giving the same compute disk slices to the VMs? Any other differences in those mappings? Are the VMs running on the same host box? If not, are they running on the same hardware and host OS? Along the lines of Roger's comment, some VM implementations have OS clocks that simply don't work right -- they run too fast or too slow (I have personally observed differences of up to 3x), because they assume they are on real hardware. Have you tested that? Did you try compiling it with profiling data turned on to see where the bottlenecks are (using gcc's -pg flag plus gprof(1))? Or, as a simpler test, you could run your app from strace(1) (assuming solaris has such a thing) to see if you are spending loads of time in system calls as Roger hypothesized. Also, how would I pass the -D flags? Do I do that when I compile SQLite? Yes -- those are arguments to GCC that are similar to writing a #define compiler directive into your code. So saying gcc -DFOO=bar ... would be like writing #define FOO bar into the code. The sqlite.org docs describe what they do pretty well iirc. Eric -- Eric A. Smith You can observe a lot by watching. -- Yogi Berra ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite really SLOW on Solaris?
Hi Eric, I will be trying it on real machines next week but for now I have one Debian and one Solaris VM on my laptop, each allocated the same memory etc I will try profiling what is really going on and will let you know, but I hope it is just a VM thing Thanks ;-) Lynton -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Eric Smith Sent: 20 November 2010 09:13 PM To: General Discussion of SQLite Database Subject: Re: [sqlite] SQLite really SLOW on Solaris? Lynton Grice wrote: Many thanks for your feedback, much appreciated ;-) But why would that happen on Solaris and not Debian? Did you try it on real OSs not VMs? Are you sure the host OS is giving the same compute disk slices to the VMs? Any other differences in those mappings? Are the VMs running on the same host box? If not, are they running on the same hardware and host OS? Along the lines of Roger's comment, some VM implementations have OS clocks that simply don't work right -- they run too fast or too slow (I have personally observed differences of up to 3x), because they assume they are on real hardware. Have you tested that? Did you try compiling it with profiling data turned on to see where the bottlenecks are (using gcc's -pg flag plus gprof(1))? Or, as a simpler test, you could run your app from strace(1) (assuming solaris has such a thing) to see if you are spending loads of time in system calls as Roger hypothesized. Also, how would I pass the -D flags? Do I do that when I compile SQLite? Yes -- those are arguments to GCC that are similar to writing a #define compiler directive into your code. So saying gcc -DFOO=bar ... would be like writing #define FOO bar into the code. The sqlite.org docs describe what they do pretty well iirc. Eric -- Eric A. Smith You can observe a lot by watching. -- Yogi Berra ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users