Irad K wrote: > Hi Howard and thanks for the info.=C2=A0 >=20 > I've read your insights about the corruption nature and I'd like to see= if I can write some sort of integrity check before I try to open the fil= e.=C2=A0 > Perhaps you could guide me where to look for the file format, where can= I find all the meta-pages and in which offset can I find the used pages = ?
Read the source yourself if you want to know the underlying format. Other= wise, just use mdb_env_info(). =C2=A0 > Basically, I wish to check the latest meta-page and see if the number o= f pages correspond with the file actual size.=C2=A0 >=20 > Second, as I've stated, my target operating system is macOS.. are you f= amiliar of any known issues with file atomicity in this particular OS ?=C2= =A0 > Perhaps you could tell me how it's implemented under the hood to achiev= e data consistency if I'm using the database using MDB_NOMETASYNC, so I f= urther > investigate this issue and address it to Apple if necessary.=20 I've seen other reports of corruptions on MacOSX. Since they're in SQLite= I haven't investigated further. http://sqlite.1065341.n5.nabble.com/SQLITE-vs-OSX-mmap-inevitable-catalog= -corruption-td85620.html http://sqlite.1065341.n5.nabble.com/Re-Database-corruption-and-PRAGMA-ful= lfsync-on-macOS-td95366i20.html >=20 > Irad >=20 > On Mon, Sep 10, 2018 at 8:44 PM Howard Chu <[email protected] <mailto:hyc@s= ymas.com>> wrote: >=20 > [email protected] <mailto:[email protected]> wrote: > > Full_Name: Irad.k > > Version: recent from GitHub. > > OS: macOS > > URL: ftp://ftp.openldap.org/incoming/ > > Submission from: (NULL) (82.81.84.130) > > > > > > Hi, > > > > I'm working with LMDB with the following flags :=C2=A0 MDB_NOMETA= SYNC | MDB_NOSUBDIR > > | MDB_NOTLS. > > > > Somehow, even though the DB should be ACI(without the D),=C2=A0 i= t got corrupted > > after recovering from kernel panic, and It crashes my process whe= n trying to > > access one of the records (see crash log below). > > > > Here's a link to the file : > > > > https://drive.google.com/file/d/12Q3KYYrapiJOgiaccnDL3tQACkqrM3C5= /view?usp=3Dsharing >=20 > This file is only 6 pages long, but the latest meta page claims tha= t it's using 11 pages. > Looking at the previous meta page, it claims that 9 pages were used= . >=20 > Clearly your OS failed to write the complete contents out before it= panicked. At the > very least the file should have been 9 pages long. Sounds like you = have an OS bug. >=20 > > According to the crash log from the process, It can clearly be se= en that the > > invalid address reside inside the mapped file region which is the= lmdb mapped > > file, but still I get KERN_MEMORY_ERROR on that address. > > > >>From what I know, an attempt to access address within the mapped = range can > > either retrieve the page contents directly from memory (if it's a= lready there), > > or trigger page fault trap that eventually lead to reading the mi= ssing data from > > disk and return it to process as well. > > > > One thing that raise some concerns is that the file size is only = 24k and the > > mapping spans over 256M. However, the file's meta data seems to b= e coherent to > > file contents. > > > > Any idea how did it happen, and what exactly in the file cause th= is corruption ? > > > > > > CRASH LOG : > > -----------------------------------------------------------------= ----- > > Exception Type:=C2=A0 =C2=A0 =C2=A0 =C2=A0 EXC_BAD_ACCESS (SIGBUS= ) > > Exception Codes:=C2=A0 =C2=A0 =C2=A0 =C2=A0KERN_MEMORY_ERROR at 0= x000000010648800a > > Exception Note:=C2=A0 =C2=A0 =C2=A0 =C2=A0 EXC_CORPSE_NOTIFY > > > > Termination Signal:=C2=A0 =C2=A0 Bus error: 10 > > Termination Reason:=C2=A0 =C2=A0 Namespace SIGNAL, Code 0xa > > Terminating Process:=C2=A0 =C2=A0exc handler [0] > > > > VM Regions Near 0x10648800a: > > > > __LINKEDIT=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00000000= 106464000-000000010647f000 [=C2=A0 108K] r--/rwx SM=3DCOW > >=C2=A0 ^Z^C [/usr/lib/dyld] > > --> mapped file=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 00000001= 0647f000-000000011647f000 [256.0M] r--/rwx > > SM=3DPRV=C2=A0 Object_id=3D9034edd9 > > STACK GUARD=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 000070000f2b= 3000-000070000f2b4000 [=C2=A0 =C2=A0 4K] ---/rwx SM=3DNUL > >=C2=A0 stack guard for thread 1 > > > > Thread 0 Crashed:: Dispatch queue: com.apple.main-thread > > 0=C2=A0 =C2=A0myprog=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00x0000000101666756 mdb_page_search_root + = 39 > > 1=C2=A0 =C2=A0myprog=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00x00000001016660f7 mdb_page_search + 182 > > 2=C2=A0 =C2=A0myprog=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00x00000001016614de mdb_cursor_set + 88 > > 3=C2=A0 =C2=A0myprog=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00x0000000101661476 mdb_get + 134 > > > > > > >=20 >=20 > --=20 > =C2=A0 -- Howard Chu > =C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0htt= p://www.symas.com > =C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0http://highlandsun= .com/hyc/ > =C2=A0 Chief Architect, OpenLDAP=C2=A0 http://www.openldap.org/proj= ect/ >=20 --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
