Re: Removing Observers eats up memory

[Markus Spoettl]

On 25/01/16 18:47, Ken Thomases wrote:

When the array changes, I remove all observers from these objects before
re-observing the new objects.


If at all possible, you should do this in a more targeted fashion.  Add and
remove observers only from the objects which were actually added or removed
from the array.  If you have properly encapsulated modifications of the array
in indexed collection mutating accessors, this is trivial.


OK, that's certainly an option, but the brute force way it is currently working 
with surely should work, too, and not cause the symptoms I see.



Has anyone any idea how the removing of observers can cause this kind of
death spiral?


You should use Instruments to trace exactly what's going on during one of the
runaway cases.


When I break in the debugger I end up in various different library functions 
concerned with hash tables and the like, such as


#0  0x7fff9ddc6d03 in weak_entry_for_referent(weak_table_t*, 
objc_object*) ()
#1  0x7fff9ddc84b6 in weak_read_no_lock ()
#2  0x7fff9ddc8473 in objc_loadWeakRetained ()
#3  0x7fff9ddc861f in objc_loadWeak ()
#4  0x7fff9cbc81c5 in hashProbe ()
#5  0x7fff9cbcbf3a in -[NSConcreteHashTable rehashAround:] ()
#6  0x7fff9cbc8245 in hashProbe ()
#7  0x7fff9cbc8014 in -[NSConcreteHashTable addObject:] ()
#8  0x7fff9cbf3f48 in _NSKeyValueObservationInfoCreateByRemoving ()
#9	0x7fff9cbf3933 in -[NSObject(NSKeyValueObserverRegistration) 
_removeObserver:forProperty:] ()
#10	0x7fff9cbf3837 in -[NSObject(NSKeyValueObserverRegistration) 
removeObserver:forKeyPath:] ()
#11	0x0001002e71f2 in -[TrackEditorViewController 
unregisterObserverForDataPoint:] at 
/Users/markus/Projects/rubiTrack/src/TrackEditorViewController.m:639


Not sure which instrument would help me finding out what is going on. I was 
using the Time Profiler but it didn't really help, partly because it's really 
difficult to know if I'm hitting the bug or not (it doesn't happen every time). 
Also recreating the issue under the debugger is much easier.



By the way, you should consider implementing the observationInfo property on
your observed class.  This avoids KVO storing observation info in a side
table.  Note that the property is a void pointer, not an object pointer, and
therefore does no memory management.


This is interesting. When I implement this property in the proxy object, the 
problem goes away (at least I wasn't able to duplicate it so far). Turn off the 
implementation, it comes back. Not really sure what this tells me, though.


Regards
Markus
--
__
Markus Spoettl
___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Removing Observers eats up memory

[Gary L. Wade]

You should look more at your design. Having that many objects being unobserved 
and reobserved (your word although in the context of new objects) reminds me of 
what I've called a genocidal refresh, an antipattern that can be fixed by only 
refreshing (or in your case observing) the things actually changed; in your 
case that may be only the items in the array that were added or removed.
--
Gary L. Wade (Sent from my iPad)
http://www.garywade.com/

> On Jan 25, 2016, at 1:10 AM, Markus Spoettl  wrote:
> 
> Hi,
> 
>  I have a view controller with a table view that is bound to an array 
> containing around 1000-1 model objects. The same view controller 
> registers itself as observer of all the objects' properties. These objects 
> are actually proxies for other model objects and are created by my view 
> controller (there is no outside reference to these objects).
> 
> When the array changes, I remove all observers from these objects before 
> re-observing the new objects.
> 
> Sometimes (on El Capitan) this removing of observers causes my app to freeze 
> and eat memory at an alarming rate. Sometimes this spirals completely out of 
> control until all memory is exhausted, sometimes it stops after a few 
> gigabytes of mystery allocations and my app continues.
> 
> The bug doesn't surface all the time, only 30-50% of all tries.
> 
> I have tried to wrap the observer removing code within an @autorelease {} 
> block, which helps a little. The problem is now harder to recreate, but it is 
> still there.
> 
> This is on El Capitan using Xcode 6.4 built against the 10.10 SDK.
> 
> Has anyone any idea how the removing of observers can cause this kind of 
> death spiral?
> 
> Regards
> Markus
> -- 
> __
> Markus Spoettl
> ___

___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Removing Observers eats up memory

[Ken Thomases]

On Jan 25, 2016, at 3:10 AM, Markus Spoettl  wrote:
> 
>  I have a view controller with a table view that is bound to an array 
> containing around 1000-1 model objects. The same view controller 
> registers itself as observer of all the objects' properties. These objects 
> are actually proxies for other model objects and are created by my view 
> controller (there is no outside reference to these objects).
> 
> When the array changes, I remove all observers from these objects before 
> re-observing the new objects.

If at all possible, you should do this in a more targeted fashion.  Add and 
remove observers only from the objects which were actually added or removed 
from the array.  If you have properly encapsulated modifications of the array 
in indexed collection mutating accessors, this is trivial.


> Sometimes (on El Capitan) this removing of observers causes my app to freeze 
> and eat memory at an alarming rate. Sometimes this spirals completely out of 
> control until all memory is exhausted, sometimes it stops after a few 
> gigabytes of mystery allocations and my app continues.
> 
> The bug doesn't surface all the time, only 30-50% of all tries.

> Has anyone any idea how the removing of observers can cause this kind of 
> death spiral?

You should use Instruments to trace exactly what's going on during one of the 
runaway cases.

By the way, you should consider implementing the observationInfo property on 
your observed class.  This avoids KVO storing observation info in a side table. 
 Note that the property is a void pointer, not an object pointer, and therefore 
does no memory management.

Regards,
Ken


___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Removing Observers eats up memory

[Quincey Morris]

On Jan 25, 2016, at 01:10 , Markus Spoettl  wrote:
> 
> Has anyone any idea how the removing of observers can cause this kind of 
> death spiral?

Genocidal refreshes aside, are you absolutely sure that you’re always 
removing/adding observers on an appropriate thread? There are two dangers here:

1. You may be doing things on a background thread that are supposed to be done 
on the main thread.

2. You may be doing things on multiple threads. In particular, doing massive 
quantities of retains and releases (via ARC) on multiple threads simultaneously 
has terrible performance characteristics, because (I think) there’s a lock 
involved. I would also assume that massive quantities of multi-threaded allocs 
and frees would be bad, because there has to be some kind of lock there too.

Without contention, these locks are likely zero-cost (more or less). With 
parallelism, the results can be bad.

There’s no particular reason to think that any of these things are happening in 
your scenario, but they might be worth looking into.

___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Removing Observers eats up memory

[Richard Charles]

> On Jan 25, 2016, at 2:10 AM, Markus Spoettl  wrote:
> 
> Sometimes (on El Capitan) this removing of observers causes my app to freeze 
> and eat memory at an alarming rate. Sometimes this spirals completely out of 
> control until all memory is exhausted, sometimes it stops after a few 
> gigabytes of mystery allocations and my app continues.

Sounds to me like the observations are not being removed. I have had no issues 
with KVO on El Capitan.

--Richard Charles


___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Removing Observers eats up memory

[Markus Spoettl]

On 25/01/16 20:23, Ken Thomases wrote:

This is interesting. When I implement this property in the proxy object,
the problem goes away (at least I wasn't able to duplicate it so far). Turn
off the implementation, it comes back. Not really sure what this tells me,
though.


Implementing the property avoids the need to use the global hash table to
look up the observation info by the object's address.  That, of course,
sidesteps the whole issue of any performance issues with the hash table
reorganizing itself.  It's also just plain faster even without this specific
issue with the hash table.  The only thing you stand to lose is making your
objects slightly larger.  Since it sounds like these objects are basically
guaranteed to be key-value observed, though, it ends up saving memory.


Thanks for the suggestion and explanation, I would never have thought of using 
this.


I wonder if there's a downside to implementing this in all my model objects 
other than adding an additional pointer to their data size.


Regards
Markus
--
__
Markus Spoettl
___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Removing Observers eats up memory

[Markus Spoettl]

On 25/01/16 17:31, Gary L. Wade wrote:

You should look more at your design. Having that many objects being
unobserved and reobserved (your word although in the context of new objects)
reminds me of what I've called a genocidal refresh, an antipattern that can
be fixed by only refreshing (or in your case observing) the things actually
changed; in your case that may be only the items in the array that were added
or removed.


This no doubt would be the "right" way of doing it, but it would not address the 
issue at hand because the observations have to be removed when the view 
controller goes away (all of them at the same time), and that's where the 
problem happens.


Regards
Markus
--
__
Markus Spoettl
___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Removing Observers eats up memory

[Markus Spoettl]

On 25/01/16 18:34, Quincey Morris wrote:

On Jan 25, 2016, at 01:10 , Markus Spoettl > wrote:


Has anyone any idea how the removing of observers can cause this kind of death
spiral?


Genocidal refreshes aside, are you absolutely sure that you’re always
removing/adding observers on an appropriate thread? There are two dangers here:

1. You may be doing things on a background thread that are supposed to be done
on the main thread.

2. You may be doing things on multiple threads. In particular, doing massive
quantities of retains and releases (via ARC) on multiple threads simultaneously
has terrible performance characteristics, because (I think) there’s a lock
involved. I would also assume that massive quantities of multi-threaded allocs
and frees would be bad, because there has to be some kind of lock there too.

Without contention, these locks are likely zero-cost (more or less). With
parallelism, the results can be bad.

There’s no particular reason to think that any of these things are happening in
your scenario, but they might be worth looking into.


Yes, true, worth a shot. Unfortunately not the problem, it all happens on the 
main thread. The code also doesn't use ARC (though the standard libs might, I 
don't know).


Ken's suggestion implementing observationInfo solves this issue for me.

Regards
Markus
--
__
Markus Spoettl
___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Removing Observers eats up memory

[Ken Thomases]

On Jan 25, 2016, at 3:22 PM, Markus Spoettl  wrote:
> 
> Thanks for the suggestion and explanation, I would never have thought of 
> using this.

You're welcome.  I'm glad I could help.

> I wonder if there's a downside to implementing this in all my model objects 
> other than adding an additional pointer to their data size.

Not much.  I guess there's also a bit of code bloat since the compiler 
synthesizes (or you implement) accessors for the property.  Should be 
negligible, though.

Regards,
Ken


___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Removing Observers eats up memory

[Ken Thomases]

On Jan 25, 2016, at 1:05 PM, Markus Spoettl  wrote:
> 
> When I break in the debugger I end up in various different library functions 
> concerned with hash tables and the like, such as
> 
> #00x7fff9ddc6d03 in weak_entry_for_referent(weak_table_t*, 
> objc_object*) ()
> #10x7fff9ddc84b6 in weak_read_no_lock ()
> #20x7fff9ddc8473 in objc_loadWeakRetained ()
> #30x7fff9ddc861f in objc_loadWeak ()
> #40x7fff9cbc81c5 in hashProbe ()
> #50x7fff9cbcbf3a in -[NSConcreteHashTable rehashAround:] ()
> #60x7fff9cbc8245 in hashProbe ()
> #70x7fff9cbc8014 in -[NSConcreteHashTable addObject:] ()
> #80x7fff9cbf3f48 in _NSKeyValueObservationInfoCreateByRemoving ()
> #90x7fff9cbf3933 in -[NSObject(NSKeyValueObserverRegistration) 
> _removeObserver:forProperty:] ()
> #10   0x7fff9cbf3837 in -[NSObject(NSKeyValueObserverRegistration) 
> removeObserver:forKeyPath:] ()
> #11   0x0001002e71f2 in -[TrackEditorViewController 
> unregisterObserverForDataPoint:] at 
> /Users/markus/Projects/rubiTrack/src/TrackEditorViewController.m:639

This looks like manipulation of the side table I mentioned when you don't 
implement observationInfo.  Of necessity, this side table is global, for all 
objects which don't implement observationInfo.  So, adding and removing large 
numbers of objects seems like it sometimes triggers a reorganization of the 
hash table.


> Not sure which instrument would help me finding out what is going on. I was 
> using the Time Profiler but it didn't really help, partly because it's really 
> difficult to know if I'm hitting the bug or not (it doesn't happen every 
> time).

Well, since you were seeing runaway memory allocation, I figured the 
Allocations instrument would be useful.  It would show the spike in memory 
usage and what was responsible for the allocations.

>> By the way, you should consider implementing the observationInfo property on
>> your observed class.  This avoids KVO storing observation info in a side
>> table.  Note that the property is a void pointer, not an object pointer, and
>> therefore does no memory management.
> 
> This is interesting. When I implement this property in the proxy object, the 
> problem goes away (at least I wasn't able to duplicate it so far). Turn off 
> the implementation, it comes back. Not really sure what this tells me, though.

Implementing the property avoids the need to use the global hash table to look 
up the observation info by the object's address.  That, of course, sidesteps 
the whole issue of any performance issues with the hash table reorganizing 
itself.  It's also just plain faster even without this specific issue with the 
hash table.  The only thing you stand to lose is making your objects slightly 
larger.  Since it sounds like these objects are basically guaranteed to be 
key-value observed, though, it ends up saving memory.

Regards,
Ken


___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Removing Observers eats up memory

[Markus Spoettl]

Hi,

  I have a view controller with a table view that is bound to an array 
containing around 1000-1 model objects. The same view controller registers 
itself as observer of all the objects' properties. These objects are actually 
proxies for other model objects and are created by my view controller (there is 
no outside reference to these objects).


When the array changes, I remove all observers from these objects before 
re-observing the new objects.


Sometimes (on El Capitan) this removing of observers causes my app to freeze and 
eat memory at an alarming rate. Sometimes this spirals completely out of control 
until all memory is exhausted, sometimes it stops after a few gigabytes of 
mystery allocations and my app continues.


The bug doesn't surface all the time, only 30-50% of all tries.

I have tried to wrap the observer removing code within an @autorelease {} block, 
which helps a little. The problem is now harder to recreate, but it is still there.


This is on El Capitan using Xcode 6.4 built against the 10.10 SDK.

Has anyone any idea how the removing of observers can cause this kind of death 
spiral?


Regards
Markus
--
__
Markus Spoettl
___

Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)

Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com

Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com