On Oct 15, 2010, at 10:24 PM, Black, Michael (IS) wrote:
> I'm not sure but I suspect sqlite is not calling unmap before
> extending the area.
> That would explain why it still gets zeroed out even with the flags.
>
> Put a break point in the unixShmUnmap call and see if it gets called
> before mmap.
We're doing this:
ftruncate(fd, 32*1024);
mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
<write into the shared memory we just mapped>
ftruncate(fd, 64*1024);
mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 32*1024);
No unmap calls.
Praveen's investigations suggest that the second mmap() call is
zeroing the memory mapped by the first mmap() call. Which is,
as you might expect, confusing SQLite.
I guess it could also be the second ftruncate() call that is
zeroing our mapped memory. That would be even odder though...
Dan.
> May just need some QNX logic that says "if we're extending an area
> unmap it first with flags".
>
> I think the mmap should honor the NOINIT flag when extending an area
> but apparently it doesn't -- only unmap can set the flag to make it
> honor it. QNX must be about the only one that does this.
>
> QNX mmap claims to be POSIX 1003.1 compliant but I don't see this
> behavior defined in there.
>
> The only zero-fill reference I see in POSIX is
> "The system shall always zero-fill any partial page at the end of an
> object"
> which doesn't fit this behavior at all.
>
> Perhaps you should report this as a bug or non-desirable/non-
> compliant behavior compared to every other OS in the world (and the
> POSIX standard which doesn't call for this behavior).
>
> Michael D. Black
> Senior Scientist
> Advanced Analytics Directorate
> Northrop Grumman Information Systems
>
>
> ________________________________
>
> From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
> Sent: Fri 10/15/2010 9:55 AM
> To: General Discussion of SQLite Database
> Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests
> fail on QNX OS
>
>
>
> Hi Michael,
>
> Yes I added the "MAP_NOINIT" to mmap() and "UNMAP_INIT_OPTIONAL"
> flag to munmap_flags() call. Don't know where i might be going wrong
> in SQLite.
>
> As you suggested, I wrote a small application to check if this
> works. Fortunately it worked as desired (as given below).
>
> MAP_NOINIT
> When specified, the POSIX requirement that the memory be zeroed is
> relaxed. The physical memory being used for this allocation must
> have been previously freed with UNMAP_INIT_OPTIONAL for this flag to
> have any effect.
>
> _________________________________________________________________________________
> int main(int argc, char *argv[]) {
> printf("Welcome to the QNX Momentics IDE\n");
>
> //Open a file
> int fd = open("/tmp/mmaptest",O_RDWR|O_CREAT, 0777);
>
> //Truncate the file
> int size = ftruncate(fd, 2048);
>
> const char buff[]="Testing the mmap API";
> write(fd,buff,sizeof(buff));
>
> //Mapping the disk file to memory
> void *pMem = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED, fd, 0);
>
> //Check the mmapped memory contents
> printf("Memory Mapped Data: %s\n", pMem);
>
> //Set all mmapped locations to 'A'
> memset(pMem,'A',2048);
>
> //Unmap the memory
> //munmap(pMem, 2048);
> munmap_flags(pMem, 2048, UNMAP_INIT_OPTIONAL);
>
> //Buffer to read disk data
> char disk_read[2048];
>
> //Read the data from the file to check if its synced
> read(fd,disk_read,sizeof(disk_read));
> printf("Reading disk file data:%s\n",disk_read);
>
> //Resetting the contents of disk file
> size = ftruncate(fd, 0);
> close(fd);
>
> //Open the file again to check to re-map the memory
> fd = open("/tmp/mmaptest",O_RDWR, 0777);
> size = ftruncate(fd, 2048);
>
> //Map the memory region again
> //int pMem1 = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED, fd,
> 0);
> int pMem1 = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED|
> MAP_NOINIT, fd, 0);
>
> //Print the re-mapped region
> printf("New Memory Mapped Data: %s\n", pMem1);
>
> //Read the disk contents again to check if "MAP_NOINIT" works
> read(fd,disk_read,sizeof(disk_read));
> printf("Disk file data during re-map: %s\n", disk_read);
>
> //Close the disk file
> close(fd);
>
> return EXIT_SUCCESS;
> }
>
> ---------------------------------------------------------------
> Output:
>
> Welcome to the QNX Momentics IDE
> Memory Mapped Data: Testing the mmap API
> Reading disk file data:AAAAAAAAAAAAAAAAAAAAAA [2048 times]
> New Memory Mapped Data: AAAAAAAAAAAAAAAAAAAAA [2048 times]
> Disk file data during re-map:
> ---------------------------------------------------------------
>
> Also found that the mmapped data is written back to disk file on
> munmap()/munmap_flags() call.
>
> Maybe i need to look into Sqlite amalgamation file in detail to
> analyze what's happening here during unmapping of memory.
> I believe we can use these flags under all scenarios without any pre-
> conditions. Not sure if POSIX (on QNX) have some limitations on
> their usage.
>
> Thanks,
> Praveen
> ______________________________________________________________________________________________________________________
>
> -----Original Message-----
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
> ] On Behalf Of Black, Michael (IS)
> Sent: Thursday, October 14, 2010 9:22 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
> OS
>
> Did you also add the MAP_NOINIT to the mmap() call?
>
> It sounds like exactly the behavior you're seeing.
>
> Did you try writing a stand-alone app to test this idea?
>
> Michael D. Black
> Senior Scientist
> Advanced Analytics Directorate
> Northrop Grumman Information Systems
>
>
> ________________________________
>
> From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
> Sent: Thu 10/14/2010 10:44 AM
> To: General Discussion of SQLite Database
> Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests
> fail on QNX OS
>
>
>
> Hi Micheal,
>
> Thanks Dan and Michael for all your inputs.
>
> I tried this approach as well, but didn't find any success.
> During unmapping i used the API munmap_flags() with
> "UNMAP_INIT_OPTIONAL" flag to avoid the zero initialization during
> the next mmaping.
>
> Another thought I have here is that the old mmapped regions may not
> be initialized with zeros, but instead the regions are getting
> synced with the data in disk file (which is full of zeros). Not sure
> if can happen with mmap() API though?
>
>
> Thanks,
> Praveen
>
> -----Original Message-----
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
> ] On Behalf Of Black, Michael (IS)
> Sent: Thursday, October 14, 2010 5:21 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
> OS
>
> I sent this before...have you tried this?
>
> According to the QNX mmap page
> http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html
>
> <https://owa1.ngc.com/exchweb/bin/redir.asp?URL=http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html
>
> >
>
> MAP_NOINIT
> When specified, the POSIX requirement that the memory be zeroed is
> relaxed. The physical memory being used for this allocation must
> have been previously freed with UNMAP_INIT_OPTIONAL for this flag to
> have any effect.
> ________________________________
>
> This flag was added in the QNX Neutrino Core OS 6.3.2.
>
> Interesting that this claims it's a POSIX requirement but I don't
> think most any others do this.
>
>
> Michael D. Black
> Senior Scientist
> Advanced Analytics Directorate
> Northrop Grumman Information Systems
>
>
> ________________________________
>
> From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
> Sent: Thu 10/14/2010 5:53 AM
> To: General Discussion of SQLite Database
> Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests
> fail on QNX OS
>
>
>
> Hi Dan,
>
> I did some more investigation on the issue and i feel there is
> synchronization problem happening here.
>
> After mmapping the shm (wal index) file to process memory, the WAL
> indexes are written into the mmapped area, and this data is not
> getting synchronized with physical (shm) file. As a result when the
> mmap() function is called the second time to map the 32k-64k memory
> region, it is synchronizing the complete mmapped region (previous
> 32k regions) with physical file, even though a valid offset is
> passed. Not sure if this is the actual behaviour of mmap() call.
>
> While debugging, before the mmap() call i checked mmapped region and
> it had valid indexes, whereas after the call all became 0's. Also i
> found that the shm file is always filled with 0's even after commits.
>
> When i added the msync() statement (to sync the shm file) before
> mmap call as shown below, the problem is not seen. In this case the
> shm file has valid 32-bit indexes, as data is synchronized before
> next mmap call is executed.
>
> while(pShmNode->nRegion<=iRegion){
> int ret = msync( apNew[0], iRegion*szRegion, MS_SYNC);
> void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
> MAP_SHARED, pShmNode->h, pShmNode->nRegion*szRegion
> );
>
> With the above msync() call all my failed test cases are passing.
>
> I don't see any msync() call in the SQLite amalgamation/wal.c file.
> I believe the data in mapped region and physical file are not
> synched automatically. We need to explicitly do it using msync() call.
> Don't know if there is any other mechanism in SQLite through which
> the data is synchronized. Does the call to sqlite3OsSync() sync the
> shm file as well? or is the shm file not syned purposefully?
>
> This is all my understanding and not sure if this is causing the
> actual issue. Please guide me if my approach/understanding is
> incorrect.
>
>
> Thanks,
> Praveen
>
> -----Original Message-----
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
> ] On Behalf Of Dan Kennedy
> Sent: Friday, October 08, 2010 9:33 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
> OS
>
>
> On Oct 8, 2010, at 9:44 PM, Raj, Praveen wrote:
>
>> Hello,
>>
>> I debugged the SQLite functions and here is my finding:
>>
>> The call to "mmap" in the function "unixShmMap" is causing the issue.
>> void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
>> MAP_SHARED, pShmNode->h, iRegion*szRegion);
>>
>> It is setting the previous memory region/regions to zero while
>> mapping
>> the new ones. Mmap call internally uses the QNX API mmap64() to map
>> the required memory region. Not sure on what is happening here. Just
>> need to dig into memory mapping to find whats happening and hopefully
>> find a solution.
>>
>> Dan - Do you have any idea on why this could be happening?
>
> Sounds like a bug in QNX to me.
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> The information contained in this message may be confidential and
> legally protected under applicable law. The message is intended
> solely for the addressee(s). If you are not the intended recipient,
> you are hereby notified that any use, forwarding, dissemination, or
> reproduction of this message is strictly prohibited and may be
> unlawful. If you are not the intended recipient, please contact the
> sender by return e-mail and destroy all copies of the original
> message.
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
>
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
>
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
