Re: Core Data fetch performance
On 12 Nov 2012, at 14:15, Joerg Simon j_si...@mac.com wrote: This summs it up really nicely: http://wbyoung.tumblr.com/post/27851725562/core-data-growing-pains Links to radar bug reports are within the blog post. The conclusion it is not usable does not hold under iOS6, since nested contexts work quite wonderfully there, but it shows problems under iOS5. Generally if you use performBlock and not performBlockAndWhait and do not use a fetched results controller at all and some other tricks, you can get it working quite well also under iOS5, but actually the code you produce is uglier than doing it manually the old way using thread confinment and merge by hand. At least in my experience. But under iOS6 nested contexts rock! So to summarise: - Child contexts synchronously route through their parent to perform fetches. The author thinks this is a mistake; I consider it an expected consequence of the design - Making changes into a parent context can be slow with no mention of a radar or details of any possible fix - Child contexts in iOS5/OS X 10.7 are pretty buggy when it comes to propogating changes *down* from their parent. This is where iOS 6/OS X 10.8 appears to have improved matters. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data fetch performance
Hi Hunter, I am very interested in what bugs they fixed. Could you please share more about it , e.g. an official bug fix report ? Thanks Bob The API changes came in iOS 5 but in iOS 6 they fixed enough bugs to make it work right. :-) On Nov 11, 2012, at 4:50 PM, Rick Mann rm...@latencyzero.com wrote: GCD, along with the Core Data changes in iOS 5 and 6 for handling concurrency, make this much easier than it used to be (or at least cleaner and harder to screw up as badly). By the way, what were the changes in iOS 6? I can't find specific Core Data notes for iOS 6, only iOS 5. Thanks! ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/bob.cromwell2012%40gmail.com This email sent to bob.cromwell2...@gmail.com ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data fetch performance
This summs it up really nicely: http://wbyoung.tumblr.com/post/27851725562/core-data-growing-pains Links to radar bug reports are within the blog post. The conclusion it is not usable does not hold under iOS6, since nested contexts work quite wonderfully there, but it shows problems under iOS5. Generally if you use performBlock and not performBlockAndWhait and do not use a fetched results controller at all and some other tricks, you can get it working quite well also under iOS5, but actually the code you produce is uglier than doing it manually the old way using thread confinment and merge by hand. At least in my experience. But under iOS6 nested contexts rock! Cheers, Jörg On Nov 12, 2012, at 1:58 PM, Bob Cromwell bob.cromwell2...@gmail.com wrote: Hi Hunter, I am very interested in what bugs they fixed. Could you please share more about it , e.g. an official bug fix report ? Thanks Bob The API changes came in iOS 5 but in iOS 6 they fixed enough bugs to make it work right. :-) On Nov 11, 2012, at 4:50 PM, Rick Mann rm...@latencyzero.com wrote: GCD, along with the Core Data changes in iOS 5 and 6 for handling concurrency, make this much easier than it used to be (or at least cleaner and harder to screw up as badly). By the way, what were the changes in iOS 6? I can't find specific Core Data notes for iOS 6, only iOS 5. Thanks! ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/bob.cromwell2012%40gmail.com This email sent to bob.cromwell2...@gmail.com ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/j_simon%40mac.com This email sent to j_si...@mac.com ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Core Data fetch performance
I'm seeing a substantial degradation in Core Data fetch performance after just
a few thousand objects, in the simulator on a beefy MacBook Pro.
My app loads data from the web, then uses an integer key in each downloaded
record to find the local record in Core Data. So I do a lot of queries like
this:
{
NSFetchRequest* req = [[NSFetchRequest alloc] init];
req.fetchLimit = 1;
NSManagedObjectContext* moc = self.managedObjectContext;
NSEntityDescription* entity = [NSEntityDescription entityForName:
@MissionEvent
inManagedObjectContext: moc];
req.entity = entity;
NSPredicate* pred = [NSPredicate predicateWithFormat: @keyShadow == %ld,
inMEP.mId];
req.predicate = pred;
NSError*error = nil;
NSArray* events = [self.managedObjectContext executeFetchRequest: req
error: error];
...
}
Basically, I query, the insert, over and over. As the number of records grows
from 0 to 6000, the time it takes to execute this query grows from about 0.0007
s to about 0.0126 s. Indexing is on for keyShadow:
sqlite explain query plan SELECT 0, t0.Z_PK, t0.Z_OPT, t0.ZCLOCKSHADOW,
t0.ZDESC, t0.ZDISABLEDSHADOW, t0.ZDURATIONSHADOW, t0.ZKEYSHADOW,
t0.ZLASTMODIFIED, t0.ZNAME, t0.ZSEQUENCESHADOW, t0.ZTIMESHADOW, t0.ZTYPESHADOW,
t0.ZMISSION FROM ZMISSIONEVENT t0 WHERE t0.ZKEYSHADOW = ?
... ;
sele order from deta
-
0 0 0 SEARCH TABLE ZMISSIONEVENT AS t0 USING INDEX
ZMISSIONEVENT_ZKEYSHADOW_INDEX (ZKEYSHADOW=?) (~10 rows)
This seems like pretty poor performance. Is this the best I can hope for?
Thanks,
--
Rick
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: Core Data fetch performance
Do you have keyShadow set as indexed in the model? If not, you probably want to
give that a try.
If you have all of the keys up front (i.e. this is a big batch update or insert
from the Web Service data), you might consider loading all of the records at
once and then using a predicate to find what you need. Something like:
//load all possible matches at once
NSFetchRequest *possibleEventsFetchRequest = [[NSFetchRequest alloc]
initWithEntityName:@MissionEvent];
[possibleEventsFetchRequest setPredicate:[NSPredicate
predicateWithFormat:@keyShadow IN %@, allPossibleKeyShadows]];
NSArray *possibleEvents = [self.managedObjectContext executeFetchRequest:
possibleEventsFetchRequest error:nil];
//create a filter predicate
NSPredicate *indexMatchPredicate = [NSPredicate predicateWithFormat:@keyShadow
== %ld, inMEP.mId];
//this now happens in memory
NSManagedObject *matchingObject = [[possibleEvents filteredArrayUsingPredicate:
indexMatchPredicate] lastObject];
Hunter
On Nov 11, 2012, at 6:42 AM, Rick Mann rm...@latencyzero.com wrote:
I'm seeing a substantial degradation in Core Data fetch performance after
just a few thousand objects, in the simulator on a beefy MacBook Pro.
My app loads data from the web, then uses an integer key in each downloaded
record to find the local record in Core Data. So I do a lot of queries like
this:
{
NSFetchRequest* req = [[NSFetchRequest alloc] init];
req.fetchLimit = 1;
NSManagedObjectContext* moc = self.managedObjectContext;
NSEntityDescription* entity = [NSEntityDescription entityForName:
@MissionEvent
inManagedObjectContext: moc];
req.entity = entity;
NSPredicate* pred = [NSPredicate predicateWithFormat: @keyShadow == %ld,
inMEP.mId];
req.predicate = pred;
NSError*error = nil;
NSArray* events = [self.managedObjectContext executeFetchRequest: req
error: error];
...
}
Basically, I query, the insert, over and over. As the number of records grows
from 0 to 6000, the time it takes to execute this query grows from about
0.0007 s to about 0.0126 s. Indexing is on for keyShadow:
sqlite explain query plan SELECT 0, t0.Z_PK, t0.Z_OPT, t0.ZCLOCKSHADOW,
t0.ZDESC, t0.ZDISABLEDSHADOW, t0.ZDURATIONSHADOW, t0.ZKEYSHADOW,
t0.ZLASTMODIFIED, t0.ZNAME, t0.ZSEQUENCESHADOW, t0.ZTIMESHADOW,
t0.ZTYPESHADOW, t0.ZMISSION FROM ZMISSIONEVENT t0 WHERE t0.ZKEYSHADOW = ?
... ;
sele order from deta
-
0 0 0 SEARCH TABLE ZMISSIONEVENT AS t0 USING INDEX
ZMISSIONEVENT_ZKEYSHADOW_INDEX (ZKEYSHADOW=?) (~10 rows)
This seems like pretty poor performance. Is this the best I can hope for?
Thanks,
--
Rick
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/lists%40lastonepicked.com
This email sent to li...@lastonepicked.com
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: Core Data fetch performance
On Nov 11, 2012, at 9:57 , Hunter Hillegas li...@lastonepicked.com wrote: Do you have keyShadow set as indexed in the model? If not, you probably want to give that a try. Yeah, the key is indexed (as I pointed out in the original post :-)) If you have all of the keys up front (i.e. this is a big batch update or insert from the Web Service data), you might consider loading all of the records at once and then using a predicate to find what you need. Something like: I do. I was thinking about this last night. I'm concerned that this may require too much memory, and building a string of 6400 IDs for the query to then format...sounds expensive. Can one build IN queries directly? I could do it in small batches. In truth, there will generally be more new records than updated records, and by adding a created-date (I currently use a last-modified-date), I can determine if the record is new or updated. Then I can skip the fetch for most of the downloaded records. But that's such a big change. I currently download on a separate thread, then call back to the main thread for each record. The intent was avoid doing Core Data work on a separate thread, and keep the UI responsive. But that doesn't really enable the second core, and it adds a lot of overhead. Maybe it's time to do the Core Data on the second thread, too. Pretty sure that will require substantial changes to the way my UI keeps up with updates. -- Rick ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data fetch performance
Do you have keyShadow set as indexed in the model? If not, you probably want to give that a try. Yeah, the key is indexed (as I pointed out in the original post :-)) Heh, I re-read your post like four times and only just now saw that notation. Whoops. If you have all of the keys up front (i.e. this is a big batch update or insert from the Web Service data), you might consider loading all of the records at once and then using a predicate to find what you need. Something like: I do. I was thinking about this last night. I'm concerned that this may require too much memory, and building a string of 6400 IDs for the query to then format...sounds expensive. It's possible that doing them all in one hit might be too much… As with most workloads, you may need to experiment to find a happy medium. Can one build IN queries directly? I'm not entirely sure I know what you mean by 'directly' in this context. I currently download on a separate thread, then call back to the main thread for each record. The intent was avoid doing Core Data work on a separate thread, and keep the UI responsive. But that doesn't really enable the second core, and it adds a lot of overhead. Maybe it's time to do the Core Data on the second thread, too. Pretty sure that will require substantial changes to the way my UI keeps up with updates. GCD, along with the Core Data changes in iOS 5 and 6 for handling concurrency, make this much easier than it used to be (or at least cleaner and harder to screw up as badly). ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data fetch performance
I do. I was thinking about this last night. I'm concerned that this may require too much memory, and building a string of 6400 IDs for the query to then format...sounds expensive. Can one build IN queries directly? I should note, I've done this quite successfully with workloads in the ~500 ID/possible matches range but never something quite as big as you're suggesting. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data fetch performance
On Nov 11, 2012, at 15:54 , Hunter Hillegas li...@lastonepicked.com wrote: Heh, I re-read your post like four times and only just now saw that notation. Whoops. It happens. Can one build IN queries directly? I'm not entirely sure I know what you mean by 'directly' in this context. Instead of using -predicateWithFormat:, creating an NSComparisonPredicate, perhaps type NSInPredicateOperatorType, and pass an array of NSNumbers to it. Not sure that's how it works. I currently download on a separate thread, then call back to the main thread for each record. The intent was avoid doing Core Data work on a separate thread, and keep the UI responsive. But that doesn't really enable the second core, and it adds a lot of overhead. Maybe it's time to do the Core Data on the second thread, too. Pretty sure that will require substantial changes to the way my UI keeps up with updates. GCD, along with the Core Data changes in iOS 5 and 6 for handling concurrency, make this much easier than it used to be (or at least cleaner and harder to screw up as badly). I'll have to look into it. I also need my UI to populate as the loads are happening, at least partly, which means no longer doing one big UI update at the end of the operation. Thanks, -- Rick ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data fetch performance
On Nov 11, 2012, at 15:54 , Hunter Hillegas li...@lastonepicked.com wrote: GCD, along with the Core Data changes in iOS 5 and 6 for handling concurrency, make this much easier than it used to be (or at least cleaner and harder to screw up as badly). By the way, what were the changes in iOS 6? I can't find specific Core Data notes for iOS 6, only iOS 5. Thanks! -- Rick ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data fetch performance
On Nov 11, 2012, at 23:16 , Hunter Hillegas li...@lastonepicked.com wrote: The API changes came in iOS 5 but in iOS 6 they fixed enough bugs to make it work right. :-) Hrm, does that mean it won't work on iOS 5.1? -- Rick ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data fetch performance
The API changes came in iOS 5 but in iOS 6 they fixed enough bugs to make it work right. :-) On Nov 11, 2012, at 4:50 PM, Rick Mann rm...@latencyzero.com wrote: GCD, along with the Core Data changes in iOS 5 and 6 for handling concurrency, make this much easier than it used to be (or at least cleaner and harder to screw up as badly). By the way, what were the changes in iOS 6? I can't find specific Core Data notes for iOS 6, only iOS 5. Thanks! ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Core Data fetch performance, design pattern sought
Hi all, I have 2 CoreData entities, Foo and Bar. Both have 'name' string attributes. There are textfields allowing the user to rename any Foo and any Bar. Assume no 2 Foos can have the same name; likewise for Bars. There is a relationship between Foo and Bar. I want a Foo to be related to a Bar if and only if their names match. That is, when the user renames a Foo (or a Bar), I want to add/break a relationship between the Bar (or Foo) of the same name. Up to now I've done this by implementing Foo's setName: to 1) change self's name and 2) fetch Bars of the same name. Likewise for Bar. This works. But it's slow if a single user action changes many names at once because many redundant fetches are performed. Any suggestions on a good pattern to solve this? Thanks, -- Sean McBride, B. Eng s...@rogue-research.com Rogue Researchwww.rogue-research.com Mac Software Developer Montréal, Québec, Canada ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data fetch performance, design pattern sought
On Sep 2, 2010, at 14:46, Sean McBride wrote: I have 2 CoreData entities, Foo and Bar. Both have 'name' string attributes. There are textfields allowing the user to rename any Foo and any Bar. Assume no 2 Foos can have the same name; likewise for Bars. There is a relationship between Foo and Bar. I want a Foo to be related to a Bar if and only if their names match. That is, when the user renames a Foo (or a Bar), I want to add/break a relationship between the Bar (or Foo) of the same name. Up to now I've done this by implementing Foo's setName: to 1) change self's name and 2) fetch Bars of the same name. Likewise for Bar. This works. But it's slow if a single user action changes many names at once because many redundant fetches are performed. According to the above, what you really want is an implicit relationship between Foos and Bars, based on name identity. That's relationship in the design sense, not in the Core Data sense. a. Have you asked the question whether you need an explicit (Core Data) relationship as well? What are the implications of traversing your object graph by name (i.e. by fetching according to name) instead of by explicit relationship? b. Assuming you do need the explicit relationship, I'd suggest it's functioning as a cache for the implicit relationship. So, you could implement it that way. Override the Core Data relationship property accessors, and try using the cached primitive relationship first. If that fails, fetch the related object by name and cache the result in the primitive relationship. (Specifically, I mean: if the object has an existing related object, check the related object's name. If it doesn't match, or if there is no related object, then fetch the related object.) If the related object is in memory, this access is slower but not incredibly slower than the built-in accessor (unless you have very stringent performance requirements). If the related object is not in memory, or it the relationship is stale, then the access is also going to cost you a fetch. There's probably another viable intermediate option, too. When changing a name, you could defer refreshing explicit relationships until any explicit relationship is, er, explicitly accessed. At that point you could refresh all accumulated stale relationships. That would bunch up the update processing, but might reduce unnecessary fetching, compared to your original proposal. It all kind of depends on your performance requirements. Possibly even, redundant fetches are cheaper than anything I'm suggesting, in your usage scenario. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
