On Oct 14, 2010, at 5:53 PM, Raj, Praveen wrote:
> Hi Dan,
>
> I did some more investigation on the issue and i feel there is
> synchronization problem happening here.
>
> After mmapping the shm (wal index) file to process memory, the WAL
> indexes are written into the mmapped area, and this data is not
> getting synchronized with physical (shm) file. As a result when the
> mmap() function is called the second time to map the 32k-64k memory
> region, it is synchronizing the complete mmapped region (previous
> 32k regions) with physical file, even though a valid offset is
> passed. Not sure if this is the actual behaviour of mmap() call.
>
> While debugging, before the mmap() call i checked mmapped region and
> it had valid indexes, whereas after the call all became 0's. Also i
> found that the shm file is always filled with 0's even after commits.
>
> When i added the msync() statement (to sync the shm file) before
> mmap call as shown below, the problem is not seen. In this case the
> shm file has valid 32-bit indexes, as data is synchronized before
> next mmap call is executed.
>
> while(pShmNode->nRegion<=iRegion){
> int ret = msync( apNew[0], iRegion*szRegion, MS_SYNC);
> void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
> MAP_SHARED, pShmNode->h, pShmNode->nRegion*szRegion
> );
>
> With the above msync() call all my failed test cases are passing.
>
> I don't see any msync() call in the SQLite amalgamation/wal.c file.
> I believe the data in mapped region and physical file are not
> synched automatically. We need to explicitly do it using msync() call.
> Don't know if there is any other mechanism in SQLite through which
> the data is synchronized. Does the call to sqlite3OsSync() sync
> the shm file as well? or is the shm file not syned purposefully?
It's true that the mapped region and physical file are not
synced automatically. But mmap() still should not be zeroing
regions that have already been mapped. This is a bug in mmap().
We don't sync it because we don't care if that file is
written to persistent storage or not. The only reason
we use a file located next to the database in the file-system
instead of in /tmp or something is because it happens to be
a convenient way to make sure all clients access the same
shared memory. See the section entitled "Implementation Of
Shared-Memory For The WAL-Index" here for more details:
http://www.sqlite.org/wal.html
It seems plausible that adding the msync() might work around
the mmap() problem. Hard to be really confident though - there
may be race conditions lurking...
Thanks for looking into this.
Dan.
>
> This is all my understanding and not sure if this is causing the
> actual issue. Please guide me if my approach/understanding is
> incorrect.
>
>
> Thanks,
> Praveen
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]
> ] On Behalf Of Dan Kennedy
> Sent: Friday, October 08, 2010 9:33 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
> OS
>
>
> On Oct 8, 2010, at 9:44 PM, Raj, Praveen wrote:
>
>> Hello,
>>
>> I debugged the SQLite functions and here is my finding:
>>
>> The call to "mmap" in the function "unixShmMap" is causing the issue.
>> void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
>> MAP_SHARED, pShmNode->h, iRegion*szRegion);
>>
>> It is setting the previous memory region/regions to zero while
>> mapping the new ones. Mmap call internally uses the QNX API mmap64()
>> to map the required memory region. Not sure on what is happening
>> here. Just need to dig into memory mapping to find whats happening
>> and hopefully find a solution.
>>
>> Dan - Do you have any idea on why this could be happening?
>
> Sounds like a bug in QNX to me.
>
> _______________________________________________
> sqlite-users mailing list
> [email protected]
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> The information contained in this message may be confidential and
> legally protected under applicable law. The message is intended
> solely for the addressee(s). If you are not the intended recipient,
> you are hereby notified that any use, forwarding, dissemination, or
> reproduction of this message is strictly prohibited and may be
> unlawful. If you are not the intended recipient, please contact the
> sender by return e-mail and destroy all copies of the original
> message.
>
> _______________________________________________
> sqlite-users mailing list
> [email protected]
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[email protected]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
