Hello Dan,
To answer your question, the 2nd ftruncate() is not zeroing the 1st mmapped
region. I had checked it using debugger, and it happens only during
2nd mmap call.
But from my investigation, i found that the 2nd ftruncate()(and further calls)
is the one from where the problem is arising.
The ftruncate() call is setting some flag (it could be time field of file)
which will make the 1st region to remap with disk file contents, when 2nd
mmap() is called.
I avoided the second ftruncate call (hardcoded size to high value) and 2nd
mmap() call worked as desired - mapped only the 2nd 32k region retaining the
1st 32k region data intact.
-----------------------------------------------------------------------------------
int ftruncate( int fildes,
off_t length );
int ftruncate64( int fildes,
off64_t length );
Arguments:
fildes
The descriptor for the file that you want to truncate.
length
The length that you want the file to be, in bytes.
Library:
libc
Use the -l c option to qcc to link against this library. This library is
usually included automatically.
Description:
These functions cause the file referenced by fildes to have a size of length
bytes. If the size of the file previously exceeded length, the extra data is
discarded (this is similar to using the F_FREESP option with fcntl()). If the
size of the file was previously shorter than length, the file size is extended
with NUL characters (similar to the F_ALLOCSP option to fcntl()).
The value of the seek pointer isn't modified by a call to ftruncate().
Upon successful completion, the ftruncate() function marks the st_ctime and
st_mtime fields of the file for update. If the ftruncate() function is
unsuccessful, the file is unaffected.
------------------------------------------------------------------------------------
The API tells that the time fields of file are updated. This could be the
triggering point for our issue, not sure though.
Michael/Dan,
Any inputs from your side?
Thanks for all your valuable inputs till now.
Thanks,
Praveen
-----Original Message-----
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org]
On Behalf Of Dan Kennedy
Sent: Friday, October 15, 2010 9:08 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX OS
On Oct 15, 2010, at 10:24 PM, Black, Michael (IS) wrote:
> I'm not sure but I suspect sqlite is not calling unmap before
> extending the area.
> That would explain why it still gets zeroed out even with the flags.
>
> Put a break point in the unixShmUnmap call and see if it gets called
> before mmap.
We're doing this:
ftruncate(fd, 32*1024);
mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
<write into the shared memory we just mapped>
ftruncate(fd, 64*1024);
mmap(0, 32*1024, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 32*1024);
No unmap calls.
Praveen's investigations suggest that the second mmap() call is
zeroing the memory mapped by the first mmap() call. Which is,
as you might expect, confusing SQLite.
I guess it could also be the second ftruncate() call that is
zeroing our mapped memory. That would be even odder though...
Dan.
> May just need some QNX logic that says "if we're extending an area
> unmap it first with flags".
>
> I think the mmap should honor the NOINIT flag when extending an area
> but apparently it doesn't -- only unmap can set the flag to make it
> honor it. QNX must be about the only one that does this.
>
> QNX mmap claims to be POSIX 1003.1 compliant but I don't see this
> behavior defined in there.
>
> The only zero-fill reference I see in POSIX is
> "The system shall always zero-fill any partial page at the end of an
> object"
> which doesn't fit this behavior at all.
>
> Perhaps you should report this as a bug or non-desirable/non-
> compliant behavior compared to every other OS in the world (and the
> POSIX standard which doesn't call for this behavior).
>
> Michael D. Black
> Senior Scientist
> Advanced Analytics Directorate
> Northrop Grumman Information Systems
>
>
> ________________________________
>
> From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
> Sent: Fri 10/15/2010 9:55 AM
> To: General Discussion of SQLite Database
> Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests
> fail on QNX OS
>
>
>
> Hi Michael,
>
> Yes I added the "MAP_NOINIT" to mmap() and "UNMAP_INIT_OPTIONAL"
> flag to munmap_flags() call. Don't know where i might be going wrong
> in SQLite.
>
> As you suggested, I wrote a small application to check if this
> works. Fortunately it worked as desired (as given below).
>
> MAP_NOINIT
> When specified, the POSIX requirement that the memory be zeroed is
> relaxed. The physical memory being used for this allocation must
> have been previously freed with UNMAP_INIT_OPTIONAL for this flag to
> have any effect.
>
> _________________________________________________________________________________
> int main(int argc, char *argv[]) {
> printf("Welcome to the QNX Momentics IDE\n");
>
> //Open a file
> int fd = open("/tmp/mmaptest",O_RDWR|O_CREAT, 0777);
>
> //Truncate the file
> int size = ftruncate(fd, 2048);
>
> const char buff[]="Testing the mmap API";
> write(fd,buff,sizeof(buff));
>
> //Mapping the disk file to memory
> void *pMem = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED, fd, 0);
>
> //Check the mmapped memory contents
> printf("Memory Mapped Data: %s\n", pMem);
>
> //Set all mmapped locations to 'A'
> memset(pMem,'A',2048);
>
> //Unmap the memory
> //munmap(pMem, 2048);
> munmap_flags(pMem, 2048, UNMAP_INIT_OPTIONAL);
>
> //Buffer to read disk data
> char disk_read[2048];
>
> //Read the data from the file to check if its synced
> read(fd,disk_read,sizeof(disk_read));
> printf("Reading disk file data:%s\n",disk_read);
>
> //Resetting the contents of disk file
> size = ftruncate(fd, 0);
> close(fd);
>
> //Open the file again to check to re-map the memory
> fd = open("/tmp/mmaptest",O_RDWR, 0777);
> size = ftruncate(fd, 2048);
>
> //Map the memory region again
> //int pMem1 = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED, fd,
> 0);
> int pMem1 = mmap(0, 2048, PROT_READ|PROT_WRITE,MAP_SHARED|
> MAP_NOINIT, fd, 0);
>
> //Print the re-mapped region
> printf("New Memory Mapped Data: %s\n", pMem1);
>
> //Read the disk contents again to check if "MAP_NOINIT" works
> read(fd,disk_read,sizeof(disk_read));
> printf("Disk file data during re-map: %s\n", disk_read);
>
> //Close the disk file
> close(fd);
>
> return EXIT_SUCCESS;
> }
>
> ---------------------------------------------------------------
> Output:
>
> Welcome to the QNX Momentics IDE
> Memory Mapped Data: Testing the mmap API
> Reading disk file data:AAAAAAAAAAAAAAAAAAAAAA [2048 times]
> New Memory Mapped Data: AAAAAAAAAAAAAAAAAAAAA [2048 times]
> Disk file data during re-map:
> ---------------------------------------------------------------
>
> Also found that the mmapped data is written back to disk file on
> munmap()/munmap_flags() call.
>
> Maybe i need to look into Sqlite amalgamation file in detail to
> analyze what's happening here during unmapping of memory.
> I believe we can use these flags under all scenarios without any pre-
> conditions. Not sure if POSIX (on QNX) have some limitations on
> their usage.
>
> Thanks,
> Praveen
> ______________________________________________________________________________________________________________________
>
> -----Original Message-----
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
> ] On Behalf Of Black, Michael (IS)
> Sent: Thursday, October 14, 2010 9:22 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
> OS
>
> Did you also add the MAP_NOINIT to the mmap() call?
>
> It sounds like exactly the behavior you're seeing.
>
> Did you try writing a stand-alone app to test this idea?
>
> Michael D. Black
> Senior Scientist
> Advanced Analytics Directorate
> Northrop Grumman Information Systems
>
>
> ________________________________
>
> From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
> Sent: Thu 10/14/2010 10:44 AM
> To: General Discussion of SQLite Database
> Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests
> fail on QNX OS
>
>
>
> Hi Micheal,
>
> Thanks Dan and Michael for all your inputs.
>
> I tried this approach as well, but didn't find any success.
> During unmapping i used the API munmap_flags() with
> "UNMAP_INIT_OPTIONAL" flag to avoid the zero initialization during
> the next mmaping.
>
> Another thought I have here is that the old mmapped regions may not
> be initialized with zeros, but instead the regions are getting
> synced with the data in disk file (which is full of zeros). Not sure
> if can happen with mmap() API though?
>
>
> Thanks,
> Praveen
>
> -----Original Message-----
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
> ] On Behalf Of Black, Michael (IS)
> Sent: Thursday, October 14, 2010 5:21 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
> OS
>
> I sent this before...have you tried this?
>
> According to the QNX mmap page
> http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html
>
> <https://owa1.ngc.com/exchweb/bin/redir.asp?URL=http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/m/mmap.html
> >
>
> MAP_NOINIT
> When specified, the POSIX requirement that the memory be zeroed is
> relaxed. The physical memory being used for this allocation must
> have been previously freed with UNMAP_INIT_OPTIONAL for this flag to
> have any effect.
> ________________________________
>
> This flag was added in the QNX Neutrino Core OS 6.3.2.
>
> Interesting that this claims it's a POSIX requirement but I don't
> think most any others do this.
>
>
> Michael D. Black
> Senior Scientist
> Advanced Analytics Directorate
> Northrop Grumman Information Systems
>
>
> ________________________________
>
> From: sqlite-users-boun...@sqlite.org on behalf of Raj, Praveen
> Sent: Thu 10/14/2010 5:53 AM
> To: General Discussion of SQLite Database
> Subject: EXTERNAL:Re: [sqlite] TestFixture 3.7.2 - Some WAL tests
> fail on QNX OS
>
>
>
> Hi Dan,
>
> I did some more investigation on the issue and i feel there is
> synchronization problem happening here.
>
> After mmapping the shm (wal index) file to process memory, the WAL
> indexes are written into the mmapped area, and this data is not
> getting synchronized with physical (shm) file. As a result when the
> mmap() function is called the second time to map the 32k-64k memory
> region, it is synchronizing the complete mmapped region (previous
> 32k regions) with physical file, even though a valid offset is
> passed. Not sure if this is the actual behaviour of mmap() call.
>
> While debugging, before the mmap() call i checked mmapped region and
> it had valid indexes, whereas after the call all became 0's. Also i
> found that the shm file is always filled with 0's even after commits.
>
> When i added the msync() statement (to sync the shm file) before
> mmap call as shown below, the problem is not seen. In this case the
> shm file has valid 32-bit indexes, as data is synchronized before
> next mmap call is executed.
>
> while(pShmNode->nRegion<=iRegion){
> int ret = msync( apNew[0], iRegion*szRegion, MS_SYNC);
> void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
> MAP_SHARED, pShmNode->h, pShmNode->nRegion*szRegion
> );
>
> With the above msync() call all my failed test cases are passing.
>
> I don't see any msync() call in the SQLite amalgamation/wal.c file.
> I believe the data in mapped region and physical file are not
> synched automatically. We need to explicitly do it using msync() call.
> Don't know if there is any other mechanism in SQLite through which
> the data is synchronized. Does the call to sqlite3OsSync() sync the
> shm file as well? or is the shm file not syned purposefully?
>
> This is all my understanding and not sure if this is causing the
> actual issue. Please guide me if my approach/understanding is
> incorrect.
>
>
> Thanks,
> Praveen
>
> -----Original Message-----
> From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org
> ] On Behalf Of Dan Kennedy
> Sent: Friday, October 08, 2010 9:33 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] TestFixture 3.7.2 - Some WAL tests fail on QNX
> OS
>
>
> On Oct 8, 2010, at 9:44 PM, Raj, Praveen wrote:
>
>> Hello,
>>
>> I debugged the SQLite functions and here is my finding:
>>
>> The call to "mmap" in the function "unixShmMap" is causing the issue.
>> void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE,
>> MAP_SHARED, pShmNode->h, iRegion*szRegion);
>>
>> It is setting the previous memory region/regions to zero while
>> mapping
>> the new ones. Mmap call internally uses the QNX API mmap64() to map
>> the required memory region. Not sure on what is happening here. Just
>> need to dig into memory mapping to find whats happening and hopefully
>> find a solution.
>>
>> Dan - Do you have any idea on why this could be happening?
>
> Sounds like a bug in QNX to me.
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> The information contained in this message may be confidential and
> legally protected under applicable law. The message is intended
> solely for the addressee(s). If you are not the intended recipient,
> you are hereby notified that any use, forwarding, dissemination, or
> reproduction of this message is strictly prohibited and may be
> unlawful. If you are not the intended recipient, please contact the
> sender by return e-mail and destroy all copies of the original
> message.
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
>
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
>
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
>
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
The information contained in this message may be confidential and legally
protected under applicable law. The message is intended solely for the
addressee(s). If you are not the intended recipient, you are hereby notified
that any use, forwarding, dissemination, or reproduction of this message is
strictly prohibited and may be unlawful. If you are not the intended recipient,
please contact the sender by return e-mail and destroy all copies of the
original message.
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
