Re: Core data fetch and multithreading (sequel)
Le 24 nov. 2010 à 05:53, Chris Hanson a écrit : > If your code actually looks like this, it's a bug: You're using entityInMOC1 > in another thread. Pass JUST its object ID, not the object itself, from one > thread to the other. Well, it turns out it is caused by something else. But never mind, I have pinpointed it and will investigate. I had a further question: is there any possible way to force a newly created object in a MOC to be visible to other MOCs except a -save: operation? Suppose I have a single varying entity in a 1:n relationship to a set of a ten thousand, and I want to keep track of the changes in two separated MOCs. Surely, there has to be a more efficient way than -save: which will attempt to write to the disk each time it is called… Something in between. No? Thanks, Vincent___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core data fetch and multithreading
Le 24 nov. 2010 à 05:53, Chris Hanson a écrit : > If your code actually looks like this, it's a bug: You're using entityInMOC1 > in another thread. Pass JUST its object ID, not the object itself, from one > thread to the other. I had not thought about this. Thanks for pointing it out. I have no time today, but I'll tomorrow and let you know. Cheers, have a nice day! Vincent ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core data fetch and multithreading
On Nov 23, 2010, at 12:38 PM, vincent habchi wrote: > However, I have now a repeated memory management crash when I release the moc > (apparently, one managed object get released one time too much: it is > destroyed in the main MOC, while the secondary MOC has a copy of it. When I > release the secondary MOC, it sends somehow an autorelease message to the > destroyed entity and that leads to a crash). > > Wherefore the question I asked to Quincey, that I'm repeating here: > > If I have two MOCs, let's say MOC1, MOC2, and I get in MOC2 the copy of a > MOC1 object: > > Entity * entityInMOC2 = [MOC2 objectWithID:[entityInMOC1 objectID]]; > > and then I traverse a relationship (or execute a fetch request): > > OtherEntity * someOtherEntity = [entityInMOC2 someRelationship]; > > do the variable someOtherEntity refer to an object in MOC1 or in MOC2? The variable someOtherEntity refers to an object in the same context as entityInMOC2, therefore MOC2. I think I see where your issue is though. On this line: > Entity * entityInMOC2 = [MOC2 objectWithID:[entityInMOC1 objectID]]; If your code actually looks like this, it's a bug: You're using entityInMOC1 in another thread. Pass JUST its object ID, not the object itself, from one thread to the other. -- Chris ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core data fetch and multithreading
Chris, thanks for your answer. > Don't do this. If you're using bindings or KVO at all in your main thread, > you CANNOT lock your context every place you need to in order to make this > safe. Furthermore, I suspect you may not be locking your context on the > background thread because you're "just reading" - that's also incorrect. I don't use bindings and lock also on the background thread, but I followed the general advice, threw this code away and replaced it by a dual MOC, one on the main thread, and the other one being created at the beginning of the GCD block, and released at the end. However, I have now a repeated memory management crash when I release the moc (apparently, one managed object get released one time too much: it is destroyed in the main MOC, while the secondary MOC has a copy of it. When I release the secondary MOC, it sends somehow an autorelease message to the destroyed entity and that leads to a crash). Wherefore the question I asked to Quincey, that I'm repeating here: If I have two MOCs, let's say MOC1, MOC2, and I get in MOC2 the copy of a MOC1 object: Entity * entityInMOC2 = [MOC2 objectWithID:[entityInMOC1 objectID]]; and then I traverse a relationship (or execute a fetch request): OtherEntity * someOtherEntity = [entityInMOC2 someRelationship]; do the variable someOtherEntity refer to an object in MOC1 or in MOC2? Thanks a lot, Vincent___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core data fetch and multithreading
On Nov 22, 2010, at 1:08 PM, Quincey Morris wrote: > On Nov 22, 2010, at 07:58, Hunter Hillegas wrote: > >> I think someone somewhere told me that if you create a MOC on the main >> thread, there's some special runloop integration that is included, hence one >> of the reasons it's important to not create one on the main thread and then >> pass it around. > > Perhaps that's true, but it sounds like an urban myth. If it's true, the MOC > locking approach can't work *at all* either -- because for the duration of > the lock the MOC is in fact being used on a background thread. But the MOC > locking approach *can* work, ergo ... etc ... QED. You're right that context locking between a main-thread and non-main-thread use of a single context can't work; this isn't an "urban myth." Context locking can only really work for contexts created on background threads, but you shouldn't need context locking anyway; just use one context per thread/queue and share the coordinator instead. Life will be much easier. > It seems feasible to design your app so that when something non-UI-related > needs to happen, you can dispatch the work as a GCD-based operation. All NSOperations are "GCD operations." I generally recommend not using raw GCD in Cocoa code, instead preferring the higher level abstraction offered by NSOperation (which supports dependencies, priorities, cancellation, and so on). Using GCD doesn't even really get you much in terms of lines-of-code savings, especially once you start subclassing NSOperation and building your own abstractions atop it. > For operations that need to modify the data, it's easy to just pass the main > MOC as a parameter into the GCD block, isn't it? Don't do this; it isn't safe to just pass a main-thread context to a background thread. Instead, pass the persistent store coordinator and the IDs of any needed managed objects, and create a managed object context within your operation with. Contexts are cheap, you can use them as scratch pads like this. -- Chris ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core data fetch and multithreading
On Nov 21, 2010, at 11:47 AM, vincent habchi wrote: > briefly speaking, I have a Core Data Entity bearing a to-many relationship > (therefore an NSSet * iVar). A dialog on the main thread can modify this set, > while it may be simultaneously enumerated on a background GCD thread. I have > therefore used the managed object context provided mutex to protect the > respective snippets. Don't do this. If you're using bindings or KVO at all in your main thread, you CANNOT lock your context every place you need to in order to make this safe. Furthermore, I suspect you may not be locking your context on the background thread because you're "just reading" - that's also incorrect. Use a separate context on the background thread, attached to the same NSPersistentStoreCoordinator, and only pass managed object IDs between threads. Also, be sure to create your background thread's context on the thread or queue which you wish to use it; don't create one on the main thread and then pass it in to your background thread. (The reason for is is that a context can care about eg the run loop it's on. You don't want to use a context attached to the main thread's run loop from a background thread.) This is pretty much the standard for Core Data multithreading: thread isolation of object graphs (as represented by contexts), passing only object IDs( and did-save notifications containing object IDs) between threads. -- Chris ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core data fetch and multithreading
On Nov 22, 2010, at 07:58, Hunter Hillegas wrote: > I think someone somewhere told me that if you create a MOC on the main > thread, there's some special runloop integration that is included, hence one > of the reasons it's important to not create one on the main thread and then > pass it around. Perhaps that's true, but it sounds like an urban myth. If it's true, the MOC locking approach can't work *at all* either -- because for the duration of the lock the MOC is in fact being used on a background thread. But the MOC locking approach *can* work, ergo ... etc ... QED. On Nov 22, 2010, at 04:34, vincent habchi wrote: >> I think maybe you have more design options here. For example, you can [in >> principle, I think] multithread with a single MOC without locks if you pass >> "ownership" of the MOC around between threads that make changes, so that >> ownership serializes access. That requires the ownership passing to be >> thread safe, and you already have [I think] the > > What do you mean exactly? Could you elaborate just a little bit? It seems feasible to design your app so that when something non-UI-related needs to happen, you can dispatch the work as a GCD-based operation. For operations that need to modify the data, it's easy to just pass the main MOC as a parameter into the GCD block, isn't it? All you need to do is ensure that only one such MOC-using block can execute at once. That's easy too, using NSOperation to limit the queue to one executing operation at a time, or using a separate one-at-a-time queue for just the operations that modify data, or using operation dependencies -- isn't it? Secondarily, you'd want each operation to be short, if your main thread might need to wait for updates to complete, for UI reasons, so you probably don't want any big enumeration loops in a single block. Maybe you also need a way of prioritizing operations, so that an operation that the main thread is waiting on can execute ahead of true background operations in the queue. Other than that there's no locking or thread synchronizing for you to code, because you're seamlessly using the locking/synchronizing that's built into the GCD and NSOperation mechanisms. In effect, Apple's done the heavy lifting for you. This sort of thing would be especially attractive if you also have big background operations that *don't* modify the data (which I think you said you do). Those operations can run fully concurrently, because they can use their own MOC. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core data fetch and multithreading
I think someone somewhere told me that if you create a MOC on the main thread, there's some special runloop integration that is included, hence one of the reasons it's important to not create one on the main thread and then pass it around. This is also important to keep in mind re: NSOperation, since it's init is called on the main thread. http://www.duckrowing.com/2010/03/11/using-core-data-on-multiple-threads/ On Nov 22, 2010, at 12:16 AM, Quincey Morris wrote: > I think maybe you have more design options here. For example, you can [in > principle, I think] multithread with a single MOC without locks if you pass > "ownership" of the MOC around between threads that make changes, so that > ownership serializes access. That requires the ownership passing to be thread > safe, and you already have [I think] the perfect mechanism for that: GCD. > You'd probably also want to break down your background operations so that > enumeration of the relationship doesn't take place within a single GCD block > execution, but where each block execution is one iteration of the > enumeration. (Isn't that more GCD-like anyway?) ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core data fetch and multithreading
Quincey: I am a bit in a hurry, so I will answer quickly: > I think maybe you have more design options here. For example, you can [in > principle, I think] multithread with a single MOC without locks if you pass > "ownership" of the MOC around between threads that make changes, so that > ownership serializes access. That requires the ownership passing to be thread > safe, and you already have [I think] the What do you mean exactly? Could you elaborate just a little bit? > Finally, when struggling with Core Data like this, it's worthwhile to > repeatedly ask yourself if Core Data is the correct technology to use. Just > because a solution to your application's problem can be describe in > functional Core Data terms, that doesn't necessarily mean that Core Data is > the best (or even a good) actual solution. Well, since I may deal with sets of more than 100,000 entities, Core Data is appealing because of its embedded SQLite code. Fetching through CD is certainly way faster than the blind solution which would involve enumerate each candidate at each operation… > FWIW, which may not be much. I *very much* appreciate your help and the time you take to answer. Cheers! Vincent___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core data fetch and multithreading
On Nov 21, 2010, at 23:14, vincent habchi wrote: > I intended to lock, fetch the entity, read the corresponding attribute, and > unlock. That's all I've to do. On the main thread, I lock when I mutate the > set, then unlock. That's all in one place, so it's not so difficult to figure > out. *That's* not difficult to figure out, but that doesn't make it correct or complete. What if Core Data implementation within the lock-protected code does a performSelector:...afterDelay:... to make something happen later? > The problem is that I need some kind of real time behavior. Having two MOC > implies saving each time there is a mutation. And since the mutating code is > tied to NSColorPanel in continuous mode, there can be many mutations as long > as the user moves the color cursor until he has found the correct hue. The > cycle goes like this: > > User picks color -> modify the set (on the main thread) -> redraw CALayer in > a GCD queue -> read the set (in the GCD queue) -> displays. > > On the other hand, I could create a private pool of color outside Core Data > and somehow insert it only when the user acknowledges its choice. That would > mean copy the Core Data set, and then later replace it by the modified > version. But since this set can hold several thousand elements, this looks an > unnecessary waste of memory to me. Or I could have a separate memory-only > persistent store with its own MOCs. But, once again, it seems unreasonably > complex for the goal. On Nov 21, 2010, at 22:26, vincent habchi wrote: > At that point, there are, I think, two possibilities: > > 1. Use a single MOC and its provided mutex for accessing shared ressources > (but this is strongly discouraged); > > 2. Create a private pool of memory in which you duplicate new objects until > they are saved. [I'm quoting you from the other thread.] I think maybe you have more design options here. For example, you can [in principle, I think] multithread with a single MOC without locks if you pass "ownership" of the MOC around between threads that make changes, so that ownership serializes access. That requires the ownership passing to be thread safe, and you already have [I think] the perfect mechanism for that: GCD. You'd probably also want to break down your background operations so that enumeration of the relationship doesn't take place within a single GCD block execution, but where each block execution is one iteration of the enumeration. (Isn't that more GCD-like anyway?) If you follow an approach like that, you can create a second MOC for read-only access, for purposes such as concurrent updating of the UI, without any need to merge contexts -- you can just throw away the read-only context when it's out of date. Actually, you can have as many read-only contexts as you need. The primary MOC accumulates all the changes no matter how produced, and that's the one that's saved when the document is saved. Of course, the specific nature of your application might make such an approach infeasible, but I'm just throwing out an example of what I referred to earlier as additional design work. The problem with straight-ahead locking is that it often tends to be micro-synchronization (same as atomicity), which can fail at the macro level. Far better to analyze the problem and devise a solution via a global strategy. Finally, when struggling with Core Data like this, it's worthwhile to repeatedly ask yourself if Core Data is the correct technology to use. Just because a solution to your application's problem can be describe in functional Core Data terms, that doesn't necessarily mean that Core Data is the best (or even a good) actual solution. FWIW, which may not be much. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core data fetch and multithreading
Quincey: > I think the point is that it's far *less* complex because the code to deal > with a second MOC should be a lot simpler than the thread locking code. Plus, > the chances are that your locking code is wrong. :) (Trust me, that's not a > dig at you personally. But multithreading interlocks are *hard*. Such code > can work almost all of the time and still be wrong.) Oh, I know that too ;) Back when I was in U. I worked on a custom 4 x MC68000 machine, with TAS semaphores in assembler. That was fun, hardware AND software (rewriting code is fairly free, but burning PALs was not! ;)) > You've added important information. If you're *changing* the objects > enumerated in the background thread, you're going to run into difficulties > with the NSDocument metaphor. (You can't -[save:] the MOC in the background > thread because […] No, fortunately, that's the contrary: I'm reading objects in the background, to get them displayed, while the user can modify their properties in the foreground, so only the main thread alters the relationship entity members. Besides, the background processes are transitory (only -displayInContext: in GCD queues). I try to keep it short here, but if you desire more info, I'll happily send it to you in a separate mail. Thanks again (I'm going to bed this time)! Vincent ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core data fetch and multithreading
On Nov 21, 2010, at 12:50, vincent habchi wrote: > Exactly, that's why I want to change. Note that my needs are fairly basic, so > I didn't see the point of entering into a more complex scheme. I think the point is that it's far *less* complex because the code to deal with a second MOC should be a lot simpler than the thread locking code. Plus, the chances are that your locking code is wrong. :) (Trust me, that's not a dig at you personally. But multithreading interlocks are *hard*. Such code can work almost all of the time and still be wrong.) > I know I should have two MOCs, but that leads to further difficulties linked > to mutator use. I see fairly clearly how to get a copy of the root entity > (the one that has the NSSet *), but how about the entities at the other end > of the relationship? Should I enumerate all those to get copies, and then > perform the mutation on the copies? Can I directly mutate the set from the > root copy object without making this expensive copying operation? Is the > change immediately reflected in the other MOC (I need real time interaction), > etc. You've added important information. If you're *changing* the objects enumerated in the background thread, you're going to run into difficulties with the NSDocument metaphor. (You can't -[save:] the MOC in the background thread because you shouldn't be changing the document file without the user having a say in it, and you can't release the background MOC without losing the changes. You might be forced to merge the changes back into the original context.) Approach #3 *might* be the best way after all, but I think if you're trying to get multithreaded data changing within a NSDocument/NSPersistentDocument scenario (which is kind of a hack to begin with) you might need to go deeper into the design process anyway. For example, you might be able to break everything (including your current "foreground" and "background" object modification loops) down into smaller steps that you schedule with GCD. Speaking from very painful experience, I'd recommend you be careful trying to design for performance if you need Core Data to behave in certain ways to make the design work. What seems like a small operation from your side of the Core Data API can be something entirely different in the Core Data implementation. (Note that I'm not saying anything against Core Data performance -- it's highly optimized -- but rather against making assumptions about how Core Data implements its behavior. You can't observe Core Data to find out, either. Its implementation characteristics really do change in dramatic ways between Mac OS versions.) ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core data fetch and multithreading
Le 21 nov. 2010 à 21:39, Quincey Morris a écrit : > It's not an ivar, it's a property. You're perfectly right. > You can't modify a NSSet. I suspect you mean "modify this relationship", but > the ambiguity leads me to wonder if you're trying to do something funky here. > Not that it's relevant to your actual question ... I mean that I replace an entity in the set, so I add one new entity and I remove an old one via the appropriate mutators (-addEntityObject, -removeEntityObject). > under the heading "General Guidelines". If I understand your scenario > correctly, you've been using approach #3, the one labeled "This approach is > strongly discouraged". Exactly, that's why I want to change. Note that my needs are fairly basic, so I didn't see the point of entering into a more complex scheme. > Use approach #1 (except in the unlikely case that performance is an issue > without full concurrency, in which case use approach #2). I know I should have two MOCs, but that leads to further difficulties linked to mutator use. I see fairly clearly how to get a copy of the root entity (the one that has the NSSet *), but how about the entities at the other end of the relationship? Should I enumerate all those to get copies, and then perform the mutation on the copies? Can I directly mutate the set from the root copy object without making this expensive copying operation? Is the change immediately reflected in the other MOC (I need real time interaction), etc. Thanks for answering, have a nice Sunday! Vincent___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core data fetch and multithreading
On Nov 21, 2010, at 11:47, vincent habchi wrote: > briefly speaking, I have a Core Data Entity bearing a to-many relationship > (therefore an NSSet * iVar). It's not an ivar, it's a property. > A dialog on the main thread can modify this set, You can't modify a NSSet. I suspect you mean "modify this relationship", but the ambiguity leads me to wonder if you're trying to do something funky here. Not that it's relevant to your actual question ... > while it may be simultaneously enumerated on a background GCD thread. I have > therefore used the managed object context provided mutex to protect the > respective snippets. > > I am improving the background thread, replacing the enumeration by a fetch > operation. Do I still need the lock, or is it provided internally by the Core > Data framework? Look at: http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/CoreData/Articles/cdMultiThreading.html under the heading "General Guidelines". If I understand your scenario correctly, you've been using approach #3, the one labeled "This approach is strongly discouraged". Use approach #1 (except in the unlikely case that performance is an issue without full concurrency, in which case use approach #2). ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
