[sqlite] Bug Report: All database opening blocked awaiting wal index rebuild

[Brian Vincent]

I've managed to reproduce a scenario where one database, database1.db is
rebuilding a large wal-index.  Meanwhile, all other threads attempting to
open completely unrelated databases (e.g. database2.db) are blocked until
the wal-index on database1.db is finished rebuilding.  This is obviously
undesirable.

Here is how I reproduced it.  There are at least 3 threads involved here.
Shared cache mode must be enabled and database1.db must have a large wal
file that needs its wal-index rebuilt.

Thread #1 succeeds in opening database1.db.  It proceeds to call, in my
case "PRAGMA cache_size = ...".  I believe this thread acquires a btree
lock of database1.db and starts rebuilding the wal-index.

Thread #2 attempts to open database1.db, and inside of the function
sqlite3BtreeOpen, proceeds to acquire the SQLITE_MUTEX_STATIC_OPEN lock,
and then call sqlite3BtreeSchema, which calls sqlite3BtreeEnter and waits
on the same btree lock that Thread #1 has.

Thread #3 can attempt to open any database and it will block attempting to
acquire the SQLITE_MUTEX_STATIC_OPEN lock.

Does this make sense?  I'm not 100% sure what the SQLITE_MUTEX_STATIC_OPEN
is needed for, but since it's a global lock, it seems that it should do a
little as possible while holding this lock, and everything goes wrong when
it attempts to acquire a btree lock while holding this lock.

Here are some actual stack traces reproduced from sqlite 3.8.7.2.  From
looking at the source, it appears the latest version has the same problem.
Let me know if you need any additional information from me.

Thread #1

#0  0x7f2b92f4c9cd in read () at ../sysdeps/unix/syscall-template.S:84
#1  0x7f2b73019b04 in unixRead ()
#2  0x7f2b7302abe8 in walIndexReadHdr ()
#3  0x7f2b7302ae4f in walTryBeginRead ()
#4  0x7f2b73031886 in sqlite3PagerSharedLock ()
#5  0x7f2b73031a26 in sqlite3BtreeBeginTrans ()
#6  0x7f2b73059978 in sqlite3InitOne ()
#7  0x7f2b73059c17 in sqlite3Init ()
#8  0x7f2b73059cb2 in sqlite3ReadSchema ()
#9  0x7f2b73060b48 in sqlite3Pragma ()
#10 0x7f2b730663d1 in sqlite3Parser ()
#11 0x7f2b730680a1 in sqlite3RunParser ()
#12 0x7f2b73068450 in sqlite3Prepare ()
#13 0x7f2b7306877c in sqlite3LockAndPrepare ()
#14 0x7f2b7306897d in sqlite3_prepare_v2 ()

Thread #2.  I'm fairly confident that it's inside of a call to
sqlite3BtreeSchema
which has been inlined.

#0  __lll_lock_wait () at
../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x7f2b92824dfd in __GI___pthread_mutex_lock (mutex=0x7f2b080018d8)
at ../nptl/pthread_mutex_lock.c:80
#2  0x7f2b7301c873 in btreeLockCarefully ()
#3  0x7f2b7304804b in sqlite3BtreeOpen ()
#4  0x7f2b7306bc77 in openDatabase ()

Thread #3

#0  __lll_lock_wait () at
../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x7f2b92824dfd in __GI___pthread_mutex_lock (mutex=0x7f2b7328e350
) at ../nptl/pthread_mutex_lock.c:80
#2  0x7f2b7304777a in sqlite3BtreeOpen ()
#3  0x7f2b7306bc77 in openDatabase ()

Thanks,
Brian Vincent
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Lemon doesn't reset `yyerrcnt` after error

[Richard Hipp]

Please try again with the latest version of Lemon.  Thanks.

On 7/6/16, Nick Wellnhofer  wrote:
> On 05/07/2016 18:12, Richard Hipp wrote:
>> Please try https://www.sqlite.org/src/info/2683b375ad129117 and verify
>> that the changes on trunk are working.  Thanks.
>
> Still doesn't work for me. The structure of the #ifdefs in `Parse` is:
>
>  #ifdef YYERRORSYMBOL
>...
>  #elif defined(YYNOERRORRECOVERY)
>...
>  #else  /* YYERRORSYMBOL is not defined */
>...
>  #endif
>
> Your first check-in modifies the first branch, your second check-in the
> second
> branch, resulting in:
>
>  #ifdef YYERRORSYMBOL
>...
>#ifndef YYNOERRORRECOVERY
>  yypParser->yyerrcnt = -1;
>#endif
>...
>  #elif defined(YYNOERRORRECOVERY)
>...
>#ifndef YYNOERRORRECOVERY
>  yypParser->yyerrcnt = -1;
>#endif
>...
>  #else  /* YYERRORSYMBOL is not defined */
>...
>  #endif
>
> The change to the second branch has no effect because YYNOERRORRECOVERY is
> always defined. My patch modifies the third branch ("YYERRORSYMBOL is not
> defined"). This fixes code that defines neither YYERRORSYMBOL nor
> YYNOERRORRECOVERY. I think the code should look like this:
>
>  #ifdef YYERRORSYMBOL
>...
>#ifndef YYNOERRORRECOVERY
>  yypParser->yyerrcnt = -1;
>#endif
>...
>  #elif defined(YYNOERRORRECOVERY)
>...
>  #else  /* YYERRORSYMBOL is not defined */
>...
>yypParser->yyerrcnt = -1;
>...
>  #endif
>
> (Another check for YYNOERRORRECOVERY isn't really needed in the third
> branch.
> It will always be undef.)
>
> Nick
>
>


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Downloading SQLite issue

[David Empson]

I think we need more details to understand what is going wrong for you.

Exactly what did you download and where from?

This is the main download page for SQLite:

https://www.sqlite.org/download.html

On that page is a section titled "Precompiled Binaries for Mac OS X (x86)”. 
Under that is a single link to the current version of the bundle of precompiled 
Mac tools: https://www.sqlite.org/2016/sqlite-tools-osx-x86-313.zip

If that isn’t what you are trying to run, you’ll need to explain what you 
downloaded.

Assuming it was that file:

After expanding the zip file you should have a folder containing three files: 
sqldiff, sqlite3, and sqlite3_analyzer.

These are command line tools which would normally be run from a shell inside 
Terminal.app. If you try to run one from Finder (by double clicking the icon) 
on a recent version of OS X, you should get a warning from Gatekeeper that they 
can’t be opened because they are from an unidentified developer (not a problem 
if they are launched from the shell). If you get past the Gatekeeper warning 
using one of the standard methods, Terminal should be launched, open a new 
window, and run the program. (That’s what happens for me.)

The warning about a damaged file is an odd one to get for command line tools, 
because it usually appears when an application’s digital signature doesn’t 
match the application bundle (indicating the application was modified). The 
SQLite command line tools are not digitally signed, which suggests Gatekeeper 
is getting confused somehow.

I’ve seen mention of cases where this message can appear for documents as well 
as applications, e.g. the following case where someone was using a third party 
file system driver via OSX Fuse - all downloaded files (with the 
com.apple.quarantine extended attribute) produced that message if 
double-clicked when the file was on the unusual file system, but were OK if 
subsequently moved to a normal HFS+ volume.

http://apple.stackexchange.com/questions/129966/files-wrongly-considered-as-damaged-in-encfs-volume

> On 9/07/2016, at 1:19 AM, Alex Fender  wrote:
> 
> I tried with Chrome and Firefox. I am using the default unzip tool in Mac.
> I double click the zip folder and then it unpacks a new folder to my
> desktop. When I open that folder, I then see the Sqlite programs. I double
> click to open and it says damaged.
> 
> On Fri, Jul 8, 2016 at 7:52 AM, Simon Slavin  wrote:
> 
>> 
>> On 8 Jul 2016, at 1:20pm, Alex Fender  wrote:
>> 
>>> After I download and unzip. I click and open Sqlite3 by double clicking.
>>> When I open the file, it says damaged and move to the trash.
>> 
>> Works fine on my Mac.
>> 
>> Which web browser are you using to download ?
>> Which program is used when you unzip the file ?

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Downloading SQLite issue

[Alex Fender]

I tried with Chrome and Firefox. I am using the default unzip tool in Mac.
I double click the zip folder and then it unpacks a new folder to my
desktop. When I open that folder, I then see the Sqlite programs. I double
click to open and it says damaged.

On Fri, Jul 8, 2016 at 7:52 AM, Simon Slavin  wrote:

>
> On 8 Jul 2016, at 1:20pm, Alex Fender  wrote:
>
> > After I download and unzip. I click and open Sqlite3 by double clicking.
> > When I open the file, it says damaged and move to the trash.
>
> Works fine on my Mac.
>
> Which web browser are you using to download ?
> Which program is used when you unzip the file ?
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>



-- 

Alex Fender

Funnel Scientist

*Funnel Science Internet Marketing LLC *

*Office* 972-867-3100 <%28972%29%20867-3100> *| Cell* 214-625-9023 *| *
FunnelScience.com 
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Downloading SQLite issue

[Simon Slavin]

On 8 Jul 2016, at 1:20pm, Alex Fender  wrote:

> After I download and unzip. I click and open Sqlite3 by double clicking.
> When I open the file, it says damaged and move to the trash.

Works fine on my Mac.

Which web browser are you using to download ?
Which program is used when you unzip the file ?

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Help Installing SQLite on a Windows 8.1 PC

[Simon Slavin]

On 6 Jul 2016, at 8:29pm, Krista M Whipple  wrote:

> I have downloaded the two 64-bit Windows files on my Windows 8.1 PC, but I
> cannot get SQLite to install on my PC.
> 
> 
> 
> Any help or directions would be greatly appreciated,

I understand the source of your confusion.  Unlike other database systems there 
is no 'SQLite installation'.  Each programmer who wants to use SQLite includes 
it in their program.  There's no central library, no SQLite server, and no 
place to look for configuration files.  Programs which use SQLite have 
everything they need inside the program and don't depend on any external files.

If you want to play with SQLite yourself without having to write your own 
software then you should download the 'Precompiled Binaries' from the web site. 
This includes a SQLITE3.EXE which is a command line shell which lets you issue 
your own SQL commands and see the results.  Just like other programs it doesn't 
depend on a library or server: everything it uses is compiled into the 
application.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Downloading SQLite issue

[Alex Fender]

After I download and unzip. I click and open Sqlite3 by double clicking.
When I open the file, it says damaged and move to the trash.

On Thu, Jul 7, 2016 at 9:27 PM, Richard Hipp  wrote:

> On 7/7/16, Alex Fender  wrote:
> > When I download SQLite for Mac, when I extract the file out of the zip
> and
> > open, the files are damaged. How can I download a file that is not
> damaged?
>
> I click on the link using Firefox and it downloads and unzips to my
> Mac just fine.
>
> What are you doing that is not working for you?
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>



-- 

Alex Fender

Funnel Scientist

*Funnel Science Internet Marketing LLC *

*Office* 972-867-3100 <%28972%29%20867-3100> *| Cell* 214-625-9023 *| *
FunnelScience.com 
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users