Continuously running daemon process CFConstantStringRefs build up over time
Greetings Cocoa Developers, I have run into a snag while writing my background daemon and I was hoping that there might be some people on this list that could point me in the right direction or suggest a workaround. I am writing a background daemon to handle to field incoming software update requests for several client applications running in the wild. My implementation is working great, but I seem to have a slow buildup of CFConstantStringRefs that are being created by internal Foundation calls and CFSTR macros and are never released. This slowly builds up over time and there doesn't appear to be any way to release these strings. I have run the daemon through the clang static analyzer and the Instruments leaks tool but none are reporting any leaks. I have even downloaded a fresh copy of valgrind from svn and it too is not finding anything. The instruments allocation monitor is reporting that there are CFConstantStringRefs that Foundation is allocating from internal methods and CFSTR macros that I am using in some functions. I'm happy to provide more details of the actual call-stacks and code if necessary. Does anyone have any idea about what could be causing this or what I can do about it? While this isn't a big deal in the short term, I have seen it build up fairly quickly during stress testing. I have even tried setting hard and soft resource limits using the launchd plist by setting the MemoryLock and ResidentSetSize but these are not having the expected effect of limiting the memory buildup. A work around I was contemplating was to have an idle timer that kills the app after a time if there are no active or incoming connections so it will then be restarted by launchd and reset the memory. This seems like a hack to get around the problem and I think I must be overlooking something. Any help would be greatly appreciated as I've tried googling and re-reading the background daemon tech notes and have turned up nothing. Thank you for your help. Sincerely, Kevin Ross ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Continuously running daemon process CFConstantStringRefs build up over time
Since you're writing a daemon, you'll need to handle autorelease-pool creation and draining manually (something that's normally handled by NSApplication in standard AppKit apps.) Perhaps objects are autoreleased and placed in the root autorelease pool (that you might be creating in main() or the like) which is never drained? Also, how many strings are leaking? I know the frameworks cache NSNumber instances; I'm not sure about immutable strings. I have run the daemon through the clang static analyzer and the Instruments leaks tool but none are reporting any leaks. I have even downloaded a fresh copy of valgrind from svn and it too is not finding anything. The instruments allocation monitor is reporting that there are CFConstantStringRefs that Foundation is allocating from internal methods and CFSTR macros that I am using in some functions. I'm happy to provide more details of the actual call-stacks and code if necessary. A pedantic detail: note that strings created with CFSTR exist statically within your executable (they aren't dynamically allocated) and therefore aren't leaks. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Continuously running daemon process CFConstantStringRefs build up over time
Thanks for your insight Dave,
In my current implementation I do have an autorelease pool being created and
released at the end of main, as well as inside each posted notification that
NSFileHandle posts on each background thread. Upon further inspection the
largest culprits are not CFConstantStringRefs but instead non-object
allocations inside libsqlite3.dylib being called by sqlite3MemMalloc. The
constant strings are indeed building but are not the largest allocations.
With one save of the managedObjectContext libsqlite3.dylib mallocs 35 objects
that are still considered live even after the NSOperation that is the root of
the call stack has since been freed.
Here is the call-stack for one of the allocations since they all seem to follow
the same call stack:
0 libsqlite3.dylib sqlite3MemMalloc
1 libsqlite3.dylib mallocWithAlarm
2 libsqlite3.dylib pcache1Fetch
3 libsqlite3.dylib sqlite3PcacheFetch
4 libsqlite3.dylib sqlite3PagerAcquire
5 libsqlite3.dylib sqlite3BtreeBeginTrans
6 libsqlite3.dylib btreeCursor
7 libsqlite3.dylib sqlite3InitOne
8 libsqlite3.dylib sqlite3Init
9 libsqlite3.dylib sqlite3Pragma
10 libsqlite3.dylib yy_reduce
11 libsqlite3.dylib sqlite3Parser
12 libsqlite3.dylib sqlite3RunParser
13 libsqlite3.dylib sqlite3LockAndPrepare
14 CoreData -[NSSQLiteConnection _executeSQLString:]
15 CoreData -[NSSQLiteConnection _configurePragmaOptions]
16 CoreData -[NSSQLiteConnection connect]
17 CoreData -[NSSQLCore _loadOrSetMetadata]
18 CoreData -[NSSQLCore _ensureMetadataLoaded]
19 CoreData -[NSSQLCore
initWithPersistentStoreCoordinator:configurationName:URL:options:]
20 CoreData -[NSPersistentStoreCoordinator
addPersistentStoreWithType:configuration:URL:options:error:]
21 MAID -[MIMaidServer persistentStoreCoordinator]
/Users/kevin/Desktop/MAID_Workspace/MAID/MAID/Source/MIMaidServer.m:466
22 MAID -[MIMaidServer managedObjectContext]
/Users/kevin/Desktop/MAID_Workspace/MAID/MAID/Source/MIMaidServer.m:492
23 MAID -[MIMaidOperation managedObjectContext]
/Users/kevin/Desktop/MAID_Workspace/MAID/MAID/Source/MIMaidOperation.m:432
24 MAID -[MIMaidOperation logClientData:]
/Users/kevin/Desktop/MAID_Workspace/MAID/MAID/Source/MIMaidOperation.m:242
25 MAID -[MIMaidOperation handleIncommingData:]
/Users/kevin/Desktop/MAID_Workspace/MAID/MAID/Source/MIMaidOperation.m:230
26 MAID -[MIMaidOperation dataReadFromClient:]
/Users/kevin/Desktop/MAID_Workspace/MAID/MAID/Source/MIMaidOperation.m:210
27 Foundation _nsnote_callback
28 CoreFoundation __CFXNotificationPost
29 CoreFoundation _CFXNotificationPostNotification
30 Foundation -[NSNotificationCenter postNotificationName:object:userInfo:]
31 Foundation _performFileHandleSource
32 CoreFoundation __CFRunLoopDoSources0
33 CoreFoundation __CFRunLoopRun
34 CoreFoundation CFRunLoopRunSpecific
35 Foundation -[NSRunLoop(NSRunLoop) runMode:beforeDate:]
36 Foundation -[NSRunLoop(NSRunLoop) run]
37 MAID -[MIMaidOperation main]
/Users/kevin/Desktop/MAID_Workspace/MAID/MAID/Source/MIMaidOperation.m:165
38 Foundation __NSThread__main__
39 libSystem.B.dylib _pthread_start
40 libSystem.B.dylib thread_start
With regard to the CFSTR being static within the executable
CFStringRef MICreateVersionStringFromVersionValue( UInt32 versionValue )
{
UniChar upperBase = 0;
UniChar lowerBase = 0;
UniChar major = 0;
UniChar minor = 0;
CFStringRef hexStringRef = CFStringCreateWithFormat( NULL, NULL,
CFSTR(%x), versionValue ); - Instruments is hi-lighting this line as
an allocation.
CFIndex sizeMatters = CFStringGetLength( hexStringRef );
//CFIndex charactersReturned = 0;
if ( sizeMatters 3 )
{
CFStringGetCharacters( hexStringRef, CFRangeMake( sizeMatters -
4, 1 ), upperBase );
}
CFStringGetCharacters( hexStringRef, CFRangeMake( sizeMatters - 3, 1 ),
lowerBase );
CFStringGetCharacters( hexStringRef, CFRangeMake( sizeMatters - 2, 1 ),
major );
CFStringGetCharacters( hexStringRef, CFRangeMake( sizeMatters - 1, 1 ),
minor );
CFStringRef versionStringRef = CFStringCreateWithFormat( NULL, NULL,
CFSTR(%c%c.%c.%c), upperBase, lowerBase, major, minor ); - Instruments
is hi-lighting this line as an allocation even though it is being released by
the caller of the function.
CFRelease( hexStringRef );
return versionStringRef;
}
The returned versionStringRef is being released from the caller of the function
and none of the other tools are reporting this as a leak so I'm not sure why
the system is still considers any of the memory live.
Thanks for your help in this matter, it's been driving me crazy for a few days.
Sincerely,
Kevin Ross
On Oct 1, 2010, at 1:00 AM, Dave Keck wrote:
Since you're writing a daemon, you'll need to handle autorelease-pool
creation and draining manually
Re: Continuously running daemon process CFConstantStringRefs build up over time
On Oct 1, 2010, at 2:15 PM, Kevin Ross wrote: libsqlite3.dylib mallocs 35 objects that are still considered live Sqlite manages its own cache. -- Scott Ribe scott_r...@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Continuously running daemon process CFConstantStringRefs build up over time
Thanks a million Scott, I've started reading the sqlite documentation and it looks like I can adjust the PRAGMA cache_size to let me tweak the size of the cache. I am only writing to the database with this daemon and I'll try to dig up any other options that might help optimize it for this scenario. Those were the largest allocations in my testing and setting the cache to 10 seems to eliminate the outstanding malloc'ed objects. I still have some outstanding allocations from -[NSAutoeleasePool init] and +[NSRunLoop(NSRunLoop) currentRunLoop] that Foundation must be caching, but the memory overhead has drastically reduced under stress-testing. While I'm on the subject, are there any caveats to calling [[NSRunLoop currentRunLoop] run]; in the -(void)main of my NSOperation subclass? In my testing it works wonderfully but I haven't read anything in the docs that say to avoid it. Thanks again for your help, Kevin Ross On Oct 1, 2010, at 1:20 PM, Scott Ribe wrote: On Oct 1, 2010, at 2:15 PM, Kevin Ross wrote: libsqlite3.dylib mallocs 35 objects that are still considered live Sqlite manages its own cache. -- Scott Ribe scott_r...@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Continuously running daemon process CFConstantStringRefs build up over time
CFStringRef hexStringRef = CFStringCreateWithFormat( NULL, NULL, CFSTR(%x), versionValue ); - Instruments is hi-lighting this line as an allocation. It sounds like you found the cause of your persisting string objects so the details of CFSTR is irrelevant, but to clarify anyway: the result of CFStringCreateWithFormat() is of course an allocation as Instruments reported, but the format string that you supplied CFStringCreateWithFormat() – that is, the CFString whose value is %x – exists statically within your executable (in the __DATA/__cfstring segment/section) and is therefore not an allocation. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
