Re: Memory not being released in a timely manner

[Greg Parker]

On Jun 7, 2013, at 9:44 AM, Jonathan Taylor  
wrote:
> Your suggestion of the VM Tracker instrument (which I had not spotted before) 
> did bring up some interesting results though. These autoreleased image 
> buffers that were causing the problem are definitely NOT reported at all by 
> Allocations ("live" bytes stays stable at 250MB), but they do show up under 
> VM Tracker as "CG image" and "CG raster data" - although I don't think 
> there's a way of getting any further details about the buffers they provide 
> the backing for?

You're probably seeing something like this:
1. Some code runs a loop that generates autorelease garbage without spinning 
any autorelease pools.
2. Some of the autoreleased but not deallocated objects are NSImage objects.
3. The NSImage objects allocate their pixel data using something other than 
malloc(). For example, the memory is allocated and shared with the WindowServer 
process using the Mach memory machinery.
4. The NSImage objects are visible in the Allocations instrument, but they are 
small and hard to see. The lost memory is mostly the NSImage pixel data which 
is visible only to the VM Tracker instrument.


-- 
Greg Parker gpar...@apple.com Runtime Wrangler



___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Jens Alfke]

On Jun 10, 2013, at 9:15 AM, Quincey Morris 
 wrote:

> It's quite possible that the image files are being mapped into memory, rather 
> than being read into allocated buffers. That would produce the results you've 
> described (an increase in VM usage, but no big allocation footprint).

I don’t think r/o memory-mapped address space shows up under RPRVT. (It’s not 
‘private’ — any other process that mmap’ed the same file would be sharing that 
address space.)

—Jens
___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Ken Thomases]

On Jun 10, 2013, at 10:23 AM, Jonathan Taylor wrote:

> Thanks for your comments Ken.

You're welcome.

> It was really good to learn about how to use heapshots effectively, that's 
> definitely something I'm going to use again in future. In this case it did 
> provide ~some~ more information on what is going on.
> 
> Just to be clear, the problem is now solved by wrapping the correct bit of 
> code in an explicit autorelease pool, but I'm still very interested in 
> understanding what's going on and why Allocations is not reporting large 
> chunks of memory - I still maintain that this is not happening. If I 
> understood your reply correctly, you're saying this isn't what you would 
> expect.

> However, if I understand you correctly, you are convinced that the (large) 
> backing buffer for the bitmap data should be being reported by Allocations.

No, you've misunderstood.  I was suggesting that there were large allocations 
that don't show up as objects.  My theory was that they were not allocated via 
malloc but by something like vm_allocate(), which the Allocations instrument 
doesn't track.

Regards,
Ken


___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Quincey Morris]

On Jun 10, 2013, at 08:23 , Jonathan Taylor  
wrote:

> However, if I understand you correctly, you are convinced that the (large) 
> backing buffer for the bitmap data should be being reported by Allocations. 
> As you can hopefully see from these screenshots, while the NS objects 
> themselves are being reported as live, the backing memory itself really 
> doesn't seem to be showing up in Allocations (though the VM tracker knows 
> about it.

It's quite possible that the image files are being mapped into memory, rather 
than being read into allocated buffers. That would produce the results you've 
described (an increase in VM usage, but no big allocation footprint).
___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Jonathan Taylor]

Thanks for your comments Ken. It was really good to learn about how to use 
heapshots effectively, that's definitely something I'm going to use again in 
future. In this case it did provide ~some~ more information on what is going on.

Just to be clear, the problem is now solved by wrapping the correct bit of code 
in an explicit autorelease pool, but I'm still very interested in understanding 
what's going on and why Allocations is not reporting large chunks of memory - I 
still maintain that this is not happening. If I understood your reply 
correctly, you're saying this isn't what you would expect. The following 
results are with the explicit autorelease pool (bug fix for my issue) disabled 
so that the problem manifests itself.

With the use of heapshots, it does turn out there are in fact ~small~ objects 
accumulating that are reported by Allocations. An example screenshot can be 
viewed here:

https://docs.google.com/file/d/0Bye8FKbpg3dYRWF0YWo1djNmZVU/edit?usp=sharing
However this is only reporting relatively small amounts of memory still 
outstanding from during the image processing run. The fact that there are 
Image/Bitmap objects in there should ring alarm bells, and watching one of 
those objects would have given me a very strong clue as to what was going 
wrong, had I done it earlier. For example:

https://docs.google.com/file/d/0Bye8FKbpg3dYcnlQRWt0ZDlVdjA/edit?usp=sharing
is pretty strong evidence that it's a delay in autorelease pool drain that is 
causing all this memory to sit around.

However, if I understand you correctly, you are convinced that the (large) 
backing buffer for the bitmap data should be being reported by Allocations. As 
you can hopefully see from these screenshots, while the NS objects themselves 
are being reported as live, the backing memory itself really doesn't seem to be 
showing up in Allocations (though the VM tracker knows about it. When you wrote:

> So, the Allocations instrument should have been showing you those living 
> objects.  True, it couldn't show you the size of memory ultimately dependent 
> on those objects, but that's often the case (i.e. a small object that owns a 
> large NSData or something; Allocations would sort the NSData to the top but 
> you'd have to hunt to find the small object that owns it).

... it seems that my observations match that in terms of the top-level objects, 
but there really doesn't seem to be any sign at all of the backing memory. 
Weird.

Cheers
Jonny
___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Ken Thomases]

On Jun 7, 2013, at 11:44 AM, Jonathan Taylor wrote:

> So, wrapping that single line of code in an autorelease pool has fixed it.

> I do think it's interesting though (and a bit worrying) that the only way I 
> could pinpoint the actual problem was by reading through the relevant bits of 
> my code over and over - I wasn't able to glean any info from Instruments that 
> really narrowed things down, other than to confirm that there were definitely 
> image buffers accumulating somewhere.

These two statements contradict each other.  The large chunks of memory were 
not objects or even malloc blocks (probably obtained using vm_allocate() or the 
like), but they were owned by regular objects.  Those regular objects were 
still alive and keeping the large chunks around.  The fact that the use of an 
autorelease pool fixed the problem is proof of that.

So, the Allocations instrument should have been showing you those living 
objects.  True, it couldn't show you the size of memory ultimately dependent on 
those objects, but that's often the case (i.e. a small object that owns a large 
NSData or something; Allocations would sort the NSData to the top but you'd 
have to hunt to find the small object that owns it).

You might have had better luck if you had taken heapshots between two points in 
your app's lifetime and observed what objects had been created after the first 
and still living at the second.
http://www.friday.com/bbum/2010/10/17/when-is-a-leak-not-a-leak-using-heapshot-analysis-to-find-undesirable-memory-growth/

Regards,
Ken


___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Jens Alfke]

On Jun 7, 2013, at 9:44 AM, Jonathan Taylor  
wrote:

> Your suggestion of the VM Tracker instrument (which I had not spotted before) 
> did bring up some interesting results though. These autoreleased image 
> buffers that were causing the problem are definitely NOT reported at all by 
> Allocations ("live" bytes stays stable at 250MB), but they do show up under 
> VM Tracker as "CG image" and "CG raster data" - although I don't think 
> there's a way of getting any further details about the buffers they provide 
> the backing for?

Those sorts of buffers are probably magical at the kernel level — one of the 
jobs of CG/Quartz is to shuffle pixmaps between CPU and GPU memory, and doing 
that efficiently is important to overall graphics performance. I wouldn’t be 
surprised if there were special types of memory pages that were optimized for 
this, that don’t live in normal address space.

I just took a look at the Quartz Debug app and didn’t find any window for 
viewing memory usage, but someone might know of other tools. Sounds like it 
would be a good idea to add that to the regular memory-tracking instruments.

—Jens
___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Fritz Anderson]

On 7 Jun 2013, at 11:44 AM, Jonathan Taylor  
wrote:

> I do think it's interesting though (and a bit worrying) that the only way I 
> could pinpoint the actual problem was by reading through the relevant bits of 
> my code over and over - I wasn't able to glean any info from Instruments that 
> really narrowed things down, other than to confirm that there were definitely 
> image buffers accumulating somewhere.

This is the tragedy of the developer's existence. New tools make the day-to-day 
stuff much easier, but sometimes you must back off to 1970-era techniques, and 
audit your code. Tragedy never leaves us.



-- 
Fritz Anderson
Xcode 4 Unleashed: 4.5 supplement for free!
http://www.informit.com/store/xcode-4-unleashed-9780672333279


___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Jonathan Taylor]

>  It smells like you're doing a lot of processing with temporary objects, in a 
> loop, without bracketing the loop body in @autoreleasepool{}, but I remember 
> you saying you're not.

Oh dear. Oh dear.

You are right. I know this is what everybody has been telling me all along. I 
have a tiny one-line GCD block that is re-posted back onto the main event loop 
(for thread safety reasons), which I had overlooked. It was setting off a chain 
reaction of binding notifications etc that resulted in the image as displayed 
in the GUI being updated. An autoreleased image buffer involved in this wasn't 
being caught by an explicit pool, and so was falling foul of the problem 
described by Jeff Johnson whereby autoreleased objects weren't being cleaned up 
in a timely manner.

So, wrapping that single line of code in an autorelease pool has fixed it. I'm 
so sorry for taxing everyone's patience on this one!

Your suggestion of the VM Tracker instrument (which I had not spotted before) 
did bring up some interesting results though. These autoreleased image buffers 
that were causing the problem are definitely NOT reported at all by Allocations 
("live" bytes stays stable at 250MB), but they do show up under VM Tracker as 
"CG image" and "CG raster data" - although I don't think there's a way of 
getting any further details about the buffers they provide the backing for?

I do think it's interesting though (and a bit worrying) that the only way I 
could pinpoint the actual problem was by reading through the relevant bits of 
my code over and over - I wasn't able to glean any info from Instruments that 
really narrowed things down, other than to confirm that there were definitely 
image buffers accumulating somewhere.


Jonny.
___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Fritz Anderson]

On 7 Jun 2013, at 6:05 AM, Jonathan Taylor  
wrote:

> [Greg Parker:]
>> "The Allocations instrument should report objects with pending autoreleases 
>> as ordinary live objects"
> 
> Do you happen to know if that's true of Allocations on 10.6? Is that 
> definitely true for objects that have actually passed out of scope, and it 
> would therefore be forbidden to re-retain them (if you know what I mean), but 
> which haven't actually been freed "behind the scenes"?

Greg Parker's lightest word would annihilate even my deepest thoughts, but this 
is my no-inside-information understanding of the semantics of the autorelease 
pool:

It's just a broker with no insight into the state of the objects it holds. It 
does a retain when an object is presented, counts the retains it does, and does 
N releases when the pool is drained. Nothing special, and no way to decide 
whether all of the ownership count relates to what may be a stack of 
autorelease pools.

There are optimizations — ARC includes a lot of them — but people have relied 
on those semantics for decades, and there's not much scope to change them. It 
has to act-as-if.

If you already can pick out the object you're interested in in the Allocations 
history, and you turn retain tracking on, you can study the retain count and 
see how many times the object has been autoreleased. But Instruments has no way 
of flagging whether a retain pattern is other than what you expect/intend.

The out-of-scope compile-time condition you suggest would be very hard to 
detect reliably. Clang may be insightful enough that it could instrument the 
object code to give the Allocations instrument clues, but that would be the 
next OS at the earliest, and doesn't help you on Snow Leopard.

I have no clue about your root problem, which is bizarre according to what 
you've told us. My approach to such things is to blunder around until something 
suggests itself. It smells like you're doing a lot of processing with temporary 
objects, in a loop, without bracketing the loop body in @autoreleasepool{}, but 
I remember you saying you're not.

[I remember vaguely that CG operates its own heap for image buffers, which is 
what your error message complains about. I dealt with it a long time ago, and I 
may be misremembering. What does the VM Tracker instrument say?]

— F

-- 
Fritz Anderson
Xcode 4 Unleashed: 4.5 supplement for free!
http://www.informit.com/store/xcode-4-unleashed-9780672333279


___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Jonathan Taylor]

> The Allocations instrument should report objects with pending autoreleases as 
> ordinary live objects. (Note that many objects with retain count == pending 
> autorelease count will be retained again before the autorelease pool pops.)
> 
> In OS X 10.8 and iOS 6 simulator, you can set environment variable 
> OBJC_PRINT_POOL_HIGHWATER=YES to get debugging logs of the autorelease pool 
> high-water mark on each thread. This can detect code that accumulates lots of 
> autorelease garbage without spinning any pools. (The high-water mark is not 
> checked until the pool is popped, so you'd have to actually finish the work 
> to see the result.) 

Thanks for your comments Greg, very interesting. Unfortunately (in a sense!) 
the problem doesn't seem to happen on 10.8, so I can't reproduce the problem 
and get those debug logs.

> "The Allocations instrument should report objects with pending autoreleases 
> as ordinary live objects"

Do you happen to know if that's true of Allocations on 10.6? Is that definitely 
true for objects that have actually passed out of scope, and it would therefore 
be forbidden to re-retain them (if you know what I mean), but which haven't 
actually been freed "behind the scenes"?

I'm not seeing Allocations report large amounts of memory. I assume that it 
should be reporting anything that is present within my 32-bit address space 
(i.e. I assume there isn't any way that external libraries could allocate 
memory that is in some way hidden from Allocations)? If that is the case, I 
suppose the only other possibility I can think of is that I am fragmenting the 
address space to such an extent that there is no way of allocating any further 
buffers (size of order 1MB). 

Currently my understanding of the problem is pretty much limited to:
- Allocations claims only 300MB allocated memory
- 1.2MB buffer cannot be allocated ("insufficient memory")
- Occurs when program is in background, and "tickling" autorelease pools at 1 
second intervals makes the problem go away.
- Occurs on 10.6, not on 10.8

I do have a solution in the form of the "tickling", but I'd be very interested 
if you had any suggestions on ways I could dig further into the underlying 
cause.

Cheers
Jonny
___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Greg Parker]

On Jun 6, 2013, at 4:41 AM, Jonathan Taylor  
wrote:
> As far as I can see, the Allocations tool does not report memory that is 
> pending release from an autorelease pool. It would be interesting to know if 
> there was a way of monitoring that. Re your suggestion about wrapping code 
> with my own autorelease pools, I have done so but I am pretty sure the movie 
> generation APIs that I am using launch their own threads, ones that I don't 
> have control over. Without being able to see exactly what allocations are 
> still pending release, I can't be sure, but I am as certain as I can be that 
> that's where the memory-pending-autorelease must be accumulating.

The Allocations instrument should report objects with pending autoreleases as 
ordinary live objects. (Note that many objects with retain count == pending 
autorelease count will be retained again before the autorelease pool pops.)

In OS X 10.8 and iOS 6 simulator, you can set environment variable 
OBJC_PRINT_POOL_HIGHWATER=YES to get debugging logs of the autorelease pool 
high-water mark on each thread. This can detect code that accumulates lots of 
autorelease garbage without spinning any pools. (The high-water mark is not 
checked until the pool is popped, so you'd have to actually finish the work to 
see the result.) 


-- 
Greg Parker gpak...@apple.com Runtime Wrangler



___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Jonathan Taylor]

Thankyou all for your replies. A few responses -

On 4 Jun 2013, at 16:30, Jeff Johnson wrote:
> We've run into this issue in a number of apps. As a workaround, we add a 
> timer to the main thread and have it fire periodically. The timer's action 
> method does the following:

Thanks again! I've implemented your code and that has "fixed" my problem - 
great news.


On 4 Jun 2013, at 18:33, Jens Alfke wrote:
> I don’t know if ObjectAlloc is supported anymore; you should be using 
> Instruments.
> 
> It’s been a long time since I used that app, so I don’t know if it reports 
> all heap allocations or just Obj-C objects. The problem is that a small 
> object (like an NSImageRep) can be hanging onto huge memory buffers for 
> pixels.

Apologies, I meant the Allocations tool within Instruments. [Incidentally, in 
my install ObjectAlloc just launches Instruments/Allocations]

As far as I can see, the Allocations tool does not report memory that is 
pending release from an autorelease pool. It would be interesting to know if 
there was a way of monitoring that. Re your suggestion about wrapping code with 
my own autorelease pools, I have done so but I am pretty sure the movie 
generation APIs that I am using launch their own threads, ones that I don't 
have control over. Without being able to see exactly what allocations are still 
pending release, I can't be sure, but I am as certain as I can be that that's 
where the memory-pending-autorelease must be accumulating.

>> The problem seems to be that, even though ObjectAlloc thinks the memory has 
>> been released, it is not actually being properly freed up until some time 
>> later. The memory usage as reported in Activity Monitor climbs higher and 
>> higher until, if left unattended, there is apparently no more memory 
>> available.
> 
> There are a lot of different numbers measuring different kinds of memory 
> usage, and not all of them are relevant to this. What specific value in 
> Activity Monitor are you talking about? The most relevant one is Private Mem 
> (RPRVT).

Yep, that's the one. (I was trying to keep my original email reasonably 
concise!)


Cheers
Jonny.
___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Graham Cox]

On 06/06/2013, at 11:03 AM, Jeff Johnson  
wrote:

> In some cases, yes, although the architecture sometimes makes that difficult.

Can you give an example?


> And even code that allocates only a small amount of memory can become very 
> significant if it is called repeatedly over the course of the app's running 
> for days or weeks at a time.


That would surely be a leak? That's quite different from just being slow to 
autorelease something. An app that runs for weeks without once spinning its run 
loop? And if that's the case, it's your responsibility to use autorelease pools 
correctly.

--Graham


___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Jeff Johnson]

On Jun 4, 2013, at 12:35 PM, Jens Alfke  wrote:

> On Jun 4, 2013, at 8:30 AM, Jeff Johnson  
> wrote:
> 
>> The "end" of the event loop cycle is immediately before the beginning. Thus, 
>> for example, if your app is in the background and not receiving events, then 
>> the autorelease pool will not be drained.
> 
> You can work around this by wrapping your own autorelease pools around code 
> that allocates significant amounts of memory, rather than relying on the 
> runloop’s outer autorelease pool to free it for you.
> 
> —Jens

In some cases, yes, although the architecture sometimes makes that difficult. 
And even code that allocates only a small amount of memory can become very 
significant if it is called repeatedly over the course of the app's running for 
days or weeks at a time.

-Jeff

___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Jens Alfke]

On Jun 4, 2013, at 8:30 AM, Jeff Johnson  
wrote:

> The "end" of the event loop cycle is immediately before the beginning. Thus, 
> for example, if your app is in the background and not receiving events, then 
> the autorelease pool will not be drained.

You can work around this by wrapping your own autorelease pools around code 
that allocates significant amounts of memory, rather than relying on the 
runloop’s outer autorelease pool to free it for you.

—Jens

___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Jens Alfke]

On Jun 3, 2013, at 8:59 AM, Jonathan Taylor  
wrote:

> The program loads and processes a large number (~2000) image files, and I 
> believe that I am disposing of memory and draining autorelease pools 
> correctly. Running the program under ObjectAlloc, it never reports memory 
> usage over 300MB.

I don’t know if ObjectAlloc is supported anymore; you should be using 
Instruments.

It’s been a long time since I used that app, so I don’t know if it reports all 
heap allocations or just Obj-C objects. The problem is that a small object 
(like an NSImageRep) can be hanging onto huge memory buffers for pixels.

> The problem seems to be that, even though ObjectAlloc thinks the memory has 
> been released, it is not actually being properly freed up until some time 
> later. The memory usage as reported in Activity Monitor climbs higher and 
> higher until, if left unattended, there is apparently no more memory 
> available.

There are a lot of different numbers measuring different kinds of memory usage, 
and not all of them are relevant to this. What specific value in Activity 
Monitor are you talking about? The most relevant one is Private Mem (RPRVT).

—Jens

smime.p7s
Description: S/MIME cryptographic signature
___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Jonathan Taylor]

Hi Jeff,
Thanks very much for your reply. That's brilliant. I had tried playing around 
with a few "tricks" I thought might prompt a pool drain, but to no joy. It's 
great to have a bit of code that will do the job for that - I'll give it a go 
tomorrow.
Cheers
Jonny

On 4 Jun 2013, at 16:30, Jeff Johnson  wrote:

> Hi Johnny.
> 
> This is a long-standing problem with AppKit. According to the documentation, 
> "The Application Kit creates an autorelease pool on the main thread at the 
> beginning of every cycle of the event loop, and drains it at the end, thereby 
> releasing any autoreleased objects generated while processing an event." 
> However, this is somewhat misleading. The "end" of the event loop cycle is 
> immediately before the beginning. Thus, for example, if your app is in the 
> background and not receiving events, then the autorelease pool will not be 
> drained. That's why your memory drops significantly when you click the mouse 
> or switch applications.
> 
> We've run into this issue in a number of apps. As a workaround, we add a 
> timer to the main thread and have it fire periodically. The timer's action 
> method does the following:
> 
> NSEvent *event = [NSEvent otherEventWithType:NSApplicationDefined 
> location:NSZeroPoint modifierFlags:0 timestamp:[NSDate 
> timeIntervalSinceReferenceDate] windowNumber:0 context:nil subtype:0 data1:0 
> data2:0]
> [NSApp postEvent:event atStart:YES];
> 
> This basically "tickles" the event loop and causes the autorelease pool to 
> drain.
> 
> -Jeff
> 
> 
> On Jun 3, 2013, at 10:59 AM, Jonathan Taylor  
> wrote:
> 
>> Hi all,
>> 
>> Can anybody advise what tools I should be using to debug a particular out of 
>> memory condition - and maybe even how I can fix it? The error that I 
>> ultimately encounter (in a 32-bit application on Snow Leopard) is:
>> 2013-06-03 15:44:30.271 MyApplication[25115:a0f] NSImage: Insufficient 
>> memory to allocate pixel data buffer of 1228800 bytes
>> However as I'll explain I don't feel that I am doing anything that should 
>> result in running out of memory.
>> 
>> The program loads and processes a large number (~2000) image files, and I 
>> believe that I am disposing of memory and draining autorelease pools 
>> correctly. Running the program under ObjectAlloc, it never reports memory 
>> usage over 300MB. The problem seems to be that, even though ObjectAlloc 
>> thinks the memory has been released, it is not actually being properly freed 
>> up until some time later. The memory usage as reported in Activity Monitor 
>> climbs higher and higher until, if left unattended, there is apparently no 
>> more memory available. Clicking the mouse or switching applications causes 
>> an immediate and significant drop in memory usage.
>> 
>> Thus the situation seems to be that I believe I am doing everything I can 
>> for good memory management, but the OS is not actually freeing things up for 
>> re-use. I encountered a similar problem that I asked about on here last year:
>> http://lists.apple.com/archives/cocoa-dev/2012/Jul/msg00602.html
>> Although in that case the eventual conclusion was that I should be using a 
>> different approach in my code, it involved the same issue of memory not 
>> being released when I would have expected it to be. In that case from last 
>> year, I was wrong to rely on it being released when I expected, but I feel 
>> that this time it should be reasonable to expect that I won't run out of 
>> memory entirely, given that I am releasing all the buffers that I am 
>> finished with and only keeping a small working set allocated!
>> 
>> This is of course partly just speculation because ObjectAlloc isn't giving 
>> any info about this unavailable but apparently not-allocated memory. So...
>> 1. Can anybody recommend a way of debugging this problem to get more 
>> concrete evidence for what is actually happening?
>> 2. Assuming my interpretation is correct, can anybody suggest a solution?
>> 
>> Many thanks
>> Jonny
> 


___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Seth Willits]

On Jun 3, 2013, at 8:59 AM, Jonathan Taylor wrote:

> ...The memory usage as reported in Activity Monitor climbs higher and higher 
> until, if left unattended, there is apparently no more memory available. 
> Clicking the mouse or switching applications causes an immediate and 
> significant drop in memory usage….

I've seen this problem with GCD's handling of autorelease pools. It sounds very 
similar to what Jeff said, but I've only experienced it on 10.6, with 10.7+ 
behaving as desired.


--
Seth Willits





___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Memory not being released in a timely manner

[Jeff Johnson]

Hi Johnny.

This is a long-standing problem with AppKit. According to the documentation, 
"The Application Kit creates an autorelease pool on the main thread at the 
beginning of every cycle of the event loop, and drains it at the end, thereby 
releasing any autoreleased objects generated while processing an event." 
However, this is somewhat misleading. The "end" of the event loop cycle is 
immediately before the beginning. Thus, for example, if your app is in the 
background and not receiving events, then the autorelease pool will not be 
drained. That's why your memory drops significantly when you click the mouse or 
switch applications.

We've run into this issue in a number of apps. As a workaround, we add a timer 
to the main thread and have it fire periodically. The timer's action method 
does the following:

NSEvent *event = [NSEvent otherEventWithType:NSApplicationDefined 
location:NSZeroPoint modifierFlags:0 timestamp:[NSDate 
timeIntervalSinceReferenceDate] windowNumber:0 context:nil subtype:0 data1:0 
data2:0]
[NSApp postEvent:event atStart:YES];

This basically "tickles" the event loop and causes the autorelease pool to 
drain.

-Jeff


On Jun 3, 2013, at 10:59 AM, Jonathan Taylor  
wrote:

> Hi all,
> 
> Can anybody advise what tools I should be using to debug a particular out of 
> memory condition - and maybe even how I can fix it? The error that I 
> ultimately encounter (in a 32-bit application on Snow Leopard) is:
> 2013-06-03 15:44:30.271 MyApplication[25115:a0f] NSImage: Insufficient memory 
> to allocate pixel data buffer of 1228800 bytes
> However as I'll explain I don't feel that I am doing anything that should 
> result in running out of memory.
> 
> The program loads and processes a large number (~2000) image files, and I 
> believe that I am disposing of memory and draining autorelease pools 
> correctly. Running the program under ObjectAlloc, it never reports memory 
> usage over 300MB. The problem seems to be that, even though ObjectAlloc 
> thinks the memory has been released, it is not actually being properly freed 
> up until some time later. The memory usage as reported in Activity Monitor 
> climbs higher and higher until, if left unattended, there is apparently no 
> more memory available. Clicking the mouse or switching applications causes an 
> immediate and significant drop in memory usage.
> 
> Thus the situation seems to be that I believe I am doing everything I can for 
> good memory management, but the OS is not actually freeing things up for 
> re-use. I encountered a similar problem that I asked about on here last year:
> http://lists.apple.com/archives/cocoa-dev/2012/Jul/msg00602.html
> Although in that case the eventual conclusion was that I should be using a 
> different approach in my code, it involved the same issue of memory not being 
> released when I would have expected it to be. In that case from last year, I 
> was wrong to rely on it being released when I expected, but I feel that this 
> time it should be reasonable to expect that I won't run out of memory 
> entirely, given that I am releasing all the buffers that I am finished with 
> and only keeping a small working set allocated!
> 
> This is of course partly just speculation because ObjectAlloc isn't giving 
> any info about this unavailable but apparently not-allocated memory. So...
> 1. Can anybody recommend a way of debugging this problem to get more concrete 
> evidence for what is actually happening?
> 2. Assuming my interpretation is correct, can anybody suggest a solution?
> 
> Many thanks
> Jonny


___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Memory not being released in a timely manner

[Jonathan Taylor]

Hi all,

Can anybody advise what tools I should be using to debug a particular out of 
memory condition - and maybe even how I can fix it? The error that I ultimately 
encounter (in a 32-bit application on Snow Leopard) is:
2013-06-03 15:44:30.271 MyApplication[25115:a0f] NSImage: Insufficient memory 
to allocate pixel data buffer of 1228800 bytes
However as I'll explain I don't feel that I am doing anything that should 
result in running out of memory.

The program loads and processes a large number (~2000) image files, and I 
believe that I am disposing of memory and draining autorelease pools correctly. 
Running the program under ObjectAlloc, it never reports memory usage over 
300MB. The problem seems to be that, even though ObjectAlloc thinks the memory 
has been released, it is not actually being properly freed up until some time 
later. The memory usage as reported in Activity Monitor climbs higher and 
higher until, if left unattended, there is apparently no more memory available. 
Clicking the mouse or switching applications causes an immediate and 
significant drop in memory usage.

Thus the situation seems to be that I believe I am doing everything I can for 
good memory management, but the OS is not actually freeing things up for 
re-use. I encountered a similar problem that I asked about on here last year:
http://lists.apple.com/archives/cocoa-dev/2012/Jul/msg00602.html
Although in that case the eventual conclusion was that I should be using a 
different approach in my code, it involved the same issue of memory not being 
released when I would have expected it to be. In that case from last year, I 
was wrong to rely on it being released when I expected, but I feel that this 
time it should be reasonable to expect that I won't run out of memory entirely, 
given that I am releasing all the buffers that I am finished with and only 
keeping a small working set allocated!

This is of course partly just speculation because ObjectAlloc isn't giving any 
info about this unavailable but apparently not-allocated memory. So...
1. Can anybody recommend a way of debugging this problem to get more concrete 
evidence for what is actually happening?
2. Assuming my interpretation is correct, can anybody suggest a solution?

Many thanks
Jonny




___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com