Sure - I loose my locks when calling daemon(). I already have changed my program to close and re-open the DB when daemonizing. I was not aware, that daemon() by itself would do a fork(). There is no explicit fork() in my program...
Anyway, it is now clear what should be avoided: carrying an open DB over the daemon() call. Maybe it would be worth noting that in the section about "How to corrupt the DB". B.T.W.: You are right about the fork(). I just read the manual page for daemon(3) again and there is stated: RETURN VALUE (This function forks, and if the fork(2) succeeds, the parent calls _exit(2), so that further errors are seen by the child only.) On success (...) So it is clear, what happens here. But it was not obvious to me, that daemon() uses fork(). I was aware, that a fork() would be a good way to ruin the DB, but I thought I was not doing that :-D Anyway, thank you all for your support and the helpful hints. SQLite is a real cool and handy software - right what I needed. Andreas Am 16.07.2019 um 13:19 schrieb Hick Gunter: > I think your file locks are getting stranded in the parent process during > your daemon() call. > > When you open the SQLite db, you are opening file descriptors and generating > memory state. > Accessing the db to read config creates further memory state plus some file > level locks. > Which will persist until you commit the transaction. > > The first step in creating a daemon is to fork() from the parent process. > This inherits the memory state and open file descriptors BUT NOT the file > record locks. > > If you continue running the same image in the child process, SQLite will > THINK it still has those locks and/or is still within a transaction. > > The parent process then needs to terminate, releasing the locks it still had. > > -----Original Message----- > From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On > Behalf Of Andreas Kretzer > Sent: Dienstag, 16. Juli 2019 12:40 > To: sqlite-users@mailinglists.sqlite.org > Subject: [EXTERNAL] Re: [sqlite] sqlite3_close() drops wal and shm files > despite of other processes holding DB open > > ... and here is the solution! > > I daemonize my programs using > > daemon (1, 1); > > If I do that, before I open the DB, then everything is alright, locks are > held - fine! > > In the other case I was first opening the DB to fetch some configuration > options from it before I took care of program args and then possibly > daemonize, if the "do not run in background"-option is not set. > > And so, if I use this foreground option, my process wild hold the locks just > fine! -> Never use daemon() with open DB. > > Thanks for the hints to how to have a look at locks and that it is worth > taking that look! > > I think, this problem is solved - I have to change the order of the startup. > Maybe I even open the DB, read the configuration and close it again, before > opening it again after being daemonized. > > Andreas > > Am 16.07.2019 um 12:21 schrieb Andreas Kretzer: >> Interessting insights! >> >> Dan pointed out, that I should check the locks on the DB files and >> report that information back. Unluckily, my 'lsof' just reports PIDs, >> executable name and open file. It is a Busybox multicall binary. >> >> The good thing is: I have 'lslocks' on my system, which gives me >> exactly the information I need. It turns out, that one of the two main >> processes doesn't apply a lock at all (still under investigation why). >> The process, that uses exactly the same procedure to open the DB helds >> locks on the DB itself and the '-shm' file, the other does not: >> >> root:~> lsof |fgrep ecc.db >> 698 /usr/bin/lcd_manager /data/ecc/ecc.db >> 698 /usr/bin/lcd_manager /data/ecc/ecc.db-wal >> 698 /usr/bin/lcd_manager /data/ecc/ecc.db-shm >> 706 /usr/bin/ecc_core /data/ecc/ecc.db >> 706 /usr/bin/ecc_core /data/ecc/ecc.db-wal >> 706 /usr/bin/ecc_core /data/ecc/ecc.db-shm root:~> lslocks >> COMMAND PID TYPE SIZE MODE M START END PATH >> lcd_manager 698 POSIX 146.2M READ 0 1073741826 1073742335 >> /data/ecc/ecc.db lcd_manager 698 POSIX 32K READ 0 128 >> 128 /data/ecc/ecc.db-shm root:~> >> >> At least, this explains why closing the second process (lcd_manager) >> will close the DB completely, removing the '-shm' and '-wal' file. >> >> I will now investigate, why the first process doesn't acquire the locks. >> >> I will report back, when I know more! >> >> Andreas >> >> Am 11.07.2019 um 18:07 schrieb Andreas Kretzer: >>> I'm using SQLITE3 (V3.29.0) on an arm embedded linux (2.6.39) on an >>> ext3 filesystem. >>> >>> Several processes hold the DB open and the "-wal" and "-shm" files exist. >>> if I use 'lsof | fgrep <name of DB>' I can see all processes having >>> all three files open. At least one of the processes uses threads, but >>> every process has just one single DB connection active which is >>> shared among all threads. >>> >>> The compilation of sqlite3 is done with multithreading in mind: >>> >>> sqlite> pragma compile_options; >>> COMPILER=gcc-6.2.0 >>> ENABLE_DBSTAT_VTAB >>> ENABLE_FTS4 >>> ENABLE_JSON1 >>> ENABLE_RTREE >>> ENABLE_STAT3 >>> ENABLE_STMTVTAB >>> ENABLE_UNKNOWN_SQL_FUNCTION >>> ENABLE_UPDATE_DELETE_LIMIT >>> HAVE_ISNAN >>> THREADSAFE=1 >>> >>> I can check, that the database is threadsafe (mode == 1) and is >>> switched to WAL-mode. >>> >>> So far I never noticed any problems dealing with concurrent updates or so. >>> The only thing (tested in depth with V3.15.2 and V3.29.0) is when one >>> process stops and closes the database using sqlite3_close(). This may >>> even be the sqlite3 CLI. That process closes DB (lsof shows that this >>> process has closed its filedescriptors and is not in the listing >>> anymore). Right at the next write access to the DB in the still >>> running process (at least I think that this is exactly the point) the >>> "-wal" and "-shm" files are removed. >>> The sqlite3_exec() function still returns SQLITE3_OK on all following >>> actions, but 'lsof' reports, that this process has opened the "-wal" >>> and "-shm" >>> files, >>> but marked as "deleted". And they are really deleted and none of the >>> upcoming DB changes will ever reach the real DB. >>> >>> What is wrong? I already checked, that my kernel supports POSIX file >>> locking (CONFIG_FILE_LOCKING=yes). What else can I check? Two or more >>> sqlite3 CLI processes started in parallel don't exhibit this >>> behavior. >>> >>> Thanks >>> >>> Andreas > > -- > > Mit freundlichen Grüßen > > Andreas Kretzer > > > > ETB Electronic Team > > Beratungs- und Vertriebs GmbH > > > > Berliner Straße 8a > > 15537 Erkner > > > > FON +49 3362 889349-12 > > FAX +49 3362 889349-23 > > > > > > email: a.kret...@etb-electronic.de > > > > AG Potsdam HRB 16532; Sitz der Gesellschaft: Am Mellensee > > Geschäftsführer: Marco Runge, Jürgen Gentzsch > > > > _______________________________________________ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > > > ___________________________________________ > Gunter Hick | Software Engineer | Scientific Games International GmbH | > Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) > +43 1 80100 - 0 > > May be privileged. May be confidential. Please delete if not the addressee. > _______________________________________________ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users -- Mit freundlichen Grüßen Andreas Kretzer ETB Electronic Team Beratungs- und Vertriebs GmbH Berliner Straße 8a 15537 Erkner FON +49 3362 889349-12 FAX +49 3362 889349-23 email: a.kret...@etb-electronic.de AG Potsdam HRB 16532; Sitz der Gesellschaft: Am Mellensee Geschäftsführer: Marco Runge, Jürgen Gentzsch _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users