According to the QNX mmap page
http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html
MAP_NOINIT
When specified, the POSIX requirement that the memory be zeroed is relaxed. The
physical memory being used for this allocation must have been previously freed
with UNMAP_INIT_OPTIONAL for this flag to have any effect.
________________________________
This flag was added in the QNX Neutrino Core OS 6.3.2.
Interesting that this claims it's a POSIX requirement but I don't think most
any others do this.
Michael D. Black
Senior Scientist
Advanced Analytics Directorate
Northrop Grumman Information Systems
________________________________
From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
Sent: Fri 10/8/2010 9:44 AM
To: General Discussion of SQLite Database
Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS
Hello,
I debugged the SQLite functions and here is my finding:
The call to "mmap" in the function "unixShmMap" is causing the issue.
void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
MAP_SHARED, pShmNode->h, iRegion*szRegion);
It is setting the previous memory region/regions to zero while mapping the new
ones. Mmap call internally uses the QNX API mmap64() to map the required memory
region. Not sure on what is happening here. Just need to dig into memory
mapping to find whats happening and hopefully find a solution.
Dan - Do you have any idea on why this could be happening?
Thanks,
Praveen
-----Original Message-----
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org]
On Behalf Of Dan Kennedy
Sent: Wednesday, October 06, 2010 9:05 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS
On Oct 6, 2010, at 9:39 PM, Raj, Praveen wrote:
> Hi Dan,
>
> For debugging purpose we use the QNX Momentics which is the IDE
> provided by QNX. I believe to certain extent
> we should be able to debug the issue using IDE. In what situations
> does the shared memory block be zeroed.
> I believe it should be zeroed when the db is closed or when WAL file
> is checkpointed. Any other scenarios?
Following a checkpoint when the writer wraps around to
the start of the log file.
> Please provide pointers on what needs to be checked along with
> "unixShmMap()" when the problem occurs.
There is only one call to sqlite3OsShmMap() in wal.c. Try
to stop the debugger when this function is called with iPage==1.
At this point there should be a pointer to the first 32KB of
mapped memory in pWal->apWiData[0]. It should not be full of
zeroes as it is in the dumps that you sent. Check if this is
the case. More specifically - 100 bytes or so into the buffer
should not contain all zeroes. It should contain a bunch of
32-bit non-zero frame numbers.
Then check if it is still the case after the call to
sqlite3OsShmMap() has returned. Either way, you're looking
for the point where that buffer pWal->apWiData[0] is
overwritten with zeroes.
Dan.
> Thanks,
> Praveen
>
> -----Original Message-----
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
> ] On Behalf Of Dan Kennedy
> Sent: Wednesday, October 06, 2010 7:37 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
> OS
>
>
> On Oct 6, 2010, at 8:15 PM, Raj, Praveen wrote:
>
>> Hello,
>>
>> I'm new to SQLite, so not sure what is the behavior of these failed
>> tests on Unix/Linux. Are they successfully passing all the WAL
>> journal mode tests on Unix?
>>
>> What could be happening when the shared memory (-shm file) is
>> increased from 32k to 64k?
>> Is a new 64k memory chunk allocated and old 32k data is copied here
>> and then 32k is purged?
>
> At some point in the transaction, the first 32KB of shared-memory
> are being zeroed. I'm not sure why this would happen.
>
> The WAL file looks Ok. And the header in the shared-memory looks
> Ok. But most of the first 32KB of shared-memory has been zeroed
> (it is likely that the whole thing was zeroed and then the header
> rewritten).
>
> Maybe something about running the unixShmMap() function on QNX to
> mmap() the second 32KB of space (we use a separate mapping
> for each 32KB block) is zeroing the first. Somehow. Do you have
> a good debugger for the platform?
>
> Dan.
>
>
>
>
>> Any insights on whats happening here would be of great help.
>>
>> Thanks,
>> Praveen
>>
>> -----Original Message-----
>> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
>> ] On Behalf Of Raj, Praveen
>> Sent: Sunday, October 03, 2010 7:42 PM
>> To: General Discussion of SQLite Database
>> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
>> OS
>>
>> Hi,
>>
>> Yes the files sizes of "test.db", "test.db-wal" and "test.db-shm"
>> are 1024, 4333512 and 65536 bytes respectively as specified below.
>>
>> Most of the test scripts are failing when the -shm file size grows
>> from 32kb to 64kb (though the "test.db-wal"
>> file size varies for different test scripts).
>> But there are cases in WAL tests where the size increases to 64kb,
>> but the problem is not seen.
>> Is there any problem while allocating/increasing shared memory, or
>> is the WAL page indexes getting corrupted when shared memory
>> increases?
>>
>> Please suggest some tips to find out the root cause for this issue.
>>
>> Thanks,
>> Praveen
>> ________________________________________
>> From: sqlite-users-boun...@sqlite.org [sqlite-users-
>> boun...@sqlite.org] On Behalf Of Dan Kennedy [danielk1...@gmail.com]
>> Sent: Saturday, October 02, 2010 2:39 PM
>> To: General Discussion of SQLite Database
>> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
>> OS
>>
>> On Oct 1, 2010, at 2:20 PM, Raj, Praveen wrote:
>>
>>> Hi,
>>>
>>> I'm validating SQLite test suite version 3.7.2 on QNX operating
>>> system. I have built the testfixture using SQLite amalgation file
>>> and other related files/libraries.
>>>
>>> I ran the full test suite and most of the test cases are passing,
>>> except some run in journal_mode = WAL. Test cases whose
>>> autocheckpoint is turned off and large amount of
>>> data is being inserted without checkpointing are failing. Other WAL
>>> tests are passing. I tried increasing the stack size of the
>>> testfixture process, but that didn't help.
>>>
>>> If i reduce the number of inserts or increase the WAL page size,
>>> test cases are passing. I believe there is some corruption happening
>>> with the WAL index while
>>> inserting large number of rows. Here is an example (from SQLite test
>>> suite) of the failure
>>> (Purposefully added "PRAGMA integrity_check" at end of wal3-1.0)
>>
>> How large are the "test.db", "test.db-wal" and "test.db-shm" files
>> on your system after the transaction in wal3-1.0? i.e. stick
>>
>> puts [file size test.db]
>> puts [file size test.db-wal]
>> puts [file size test.db-shm]
>>
>> or similar after the test case.
>>
>> Here the files are 1024, 4333512 and 65536 bytes. From what you are
>> saying it sounds like the problem is occurring when the -shm file
>> grows from 32KB to 64KB.
>>
>> Dan.
>>
>>
>> _______________________________________________
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>> The information contained in this message may be confidential and
>> legally protected under applicable law. The message is intended
>> solely for the addressee(s). If you are not the intended recipient,
>> you are hereby notified that any use, forwarding, dissemination, or
>> reproduction of this message is strictly prohibited and may be
>> unlawful. If you are not the intended recipient, please contact the
>> sender by return e-mail and destroy all copies of the original
>> message.
>>
>> _______________________________________________
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>> _______________________________________________
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> The information contained in this message may be confidential and
> legally protected under applicable law. The message is intended
> solely for the addressee(s). If you are not the intended recipient,
> you are hereby notified that any use, forwarding, dissemination, or
> reproduction of this message is strictly prohibited and may be
> unlawful. If you are not the intended recipient, please contact the
> sender by return e-mail and destroy all copies of the original
> message.
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
The information contained in this message may be confidential and legally
protected under applicable law. The message is intended solely for the
addressee(s). If you are not the intended recipient, you are hereby notified
that any use, forwarding, dissemination, or reproduction of this message is
strictly prohibited and may be unlawful. If you are not the intended recipient,
please contact the sender by return e-mail and destroy all copies of the
original message.
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
