Re: Core Data dog-slow when using first time after boot
Hello all,
I saw an interesting reply from Dr. Hipp posted today:
http://www.mail-archive.com/sqlite-users@sqlite.org/msg62618.html
This seems to explain why SQLite (and Core Data) might sometimes launch and
execute very slowly.
-- Tito
On Sep 3, 2009, at 5:00 PM, Ben Trumbull wrote:
On Sep 3, 2009, at 4:49 AM, Ruotger Skupin wrote:
Since it's not a many to many, you can perform the prefetching
effectively by hand using a fetch request to preload the relevant
destination rows with an IN query based on the data you initially fetched
for the source entity. You shouldn't have to, but if you've run into a
bug, that's how you could workaround it.
You still haven't described the NSPredicate you were using with
filteredArrayUsingPredicate. Being more forthcoming about the context
and details of your problem will help us get you answers more quickly.
This is the predicate I was using for the test of the original post, but
since I use Smart Folders predicates can look a lot different (i.e.:
complex):
(additionalInfo.onTheFlyIsInternal == 0 AND additionalInfo.isSuppressed !=
1) AND (account.uniqueID IN {D1AB3788-00DF-4475-A979-CE3EFC3987B5} OR
FALSEPREDICATE)
You'll want to prefetch additionalInfo and account.
Hm. Problem here is: As mentioned I use the predicate for a smart group. So
the predicate can vary wildly and can reference basically any entity in the
model. So what is the best strategy here? Decompile the predicate (which
is built by an NSPredicateEditor) and prefetch the keypaths it takes?
Prefetch everything?
Prefetching everything is usually undesirable. The easiest approximate
solution is to just track the hot button relationships for each of your core
entities, and look up which keypaths to prefetch by entity name. Of course,
you can also do a lot less filtering in memory if you pass the predicates to
the database with the fetch request. You only need to prefetch relationships
you are going to use in memory. You don't need to prefetch anything that's
simply used as part of the predicate in the fetch request itself.
Walking through the predicate and gathering up the keypaths used is very
tedious, but not especially difficult, if you find yourself wanting a
complete solution.
- Ben
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/cocoa-dev/tciuro%40mac.com
This email sent to tci...@mac.com
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
Am 26.08.2009 um 01:21 schrieb Ben Trumbull:
When I use setRelationshipKeyPathsForPrefetching the fetch throws:
-[NSSQLAttribute inverseRelationship]: unrecognized selector sent
to instance 0x10ee150
Can you provide the entire stack trace at this point ? gdb use
future-break objc_exception_throw
Thanks. What is the actual set of keypaths you set in the fetch
request ? setRelationshipKeyPathsForPrefetching expects all the
keypaths to terminate in a relationship object, not the attribute
off that related object.
Yes, that was the problem.
|-|
| Transaction |
|-| ||
|net | -- | Amount | -- (one other entity)
|gross| -- ||
|fee | -- |value | -
|account | - |currency||
|... |||||
|-|| |
| |
-- |
| |
||| |
|| Account| |
||| |
||balances| --
-- |transactions|
|... |
||
So basically I use Amount only as a container for a currency
descriptor and a value. I would have to model 7 inverse
relationships on that entity to make all relationships two-way.
Hmm. I'd recommend consider whether or not it makes sense to de-
normalize Amount into Transaction instead of making it a separate
entity. I assume the net/gross/fee are all going to be in the same
currency, yes ? Amount is very fine grained, and not (apparently)
offering very much value for the abstraction cost.
Will do that. At least for the amounts in Transaction.
Since it's not a many to many, you can perform the prefetching
effectively by hand using a fetch request to preload the relevant
destination rows with an IN query based on the data you initially
fetched for the source entity. You shouldn't have to, but if
you've run into a bug, that's how you could workaround it.
You still haven't described the NSPredicate you were using with
filteredArrayUsingPredicate. Being more forthcoming about the
context and details of your problem will help us get you answers
more quickly.
This is the predicate I was using for the test of the original
post, but since I use Smart Folders predicates can look a lot
different (i.e.: complex):
(additionalInfo.onTheFlyIsInternal == 0 AND
additionalInfo.isSuppressed != 1) AND (account.uniqueID IN
{D1AB3788-00DF-4475-A979-CE3EFC3987B5} OR FALSEPREDICATE)
You'll want to prefetch additionalInfo and account.
Hm. Problem here is: As mentioned I use the predicate for a smart
group. So the predicate can vary wildly and can reference basically
any entity in the model. So what is the best strategy here?
Decompile the predicate (which is built by an NSPredicateEditor) and
prefetch the keypaths it takes? Prefetch everything?
Kind Regards
Ruotger
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Sep 3, 2009, at 4:49 AM, Ruotger Skupin wrote:
Since it's not a many to many, you can perform the prefetching
effectively by hand using a fetch request to preload the relevant
destination rows with an IN query based on the data you initially
fetched for the source entity. You shouldn't have to, but if
you've run into a bug, that's how you could workaround it.
You still haven't described the NSPredicate you were using with
filteredArrayUsingPredicate. Being more forthcoming about the
context and details of your problem will help us get you answers
more quickly.
This is the predicate I was using for the test of the original
post, but since I use Smart Folders predicates can look a lot
different (i.e.: complex):
(additionalInfo.onTheFlyIsInternal == 0 AND
additionalInfo.isSuppressed != 1) AND (account.uniqueID IN
{D1AB3788-00DF-4475-A979-CE3EFC3987B5} OR FALSEPREDICATE)
You'll want to prefetch additionalInfo and account.
Hm. Problem here is: As mentioned I use the predicate for a smart
group. So the predicate can vary wildly and can reference basically
any entity in the model. So what is the best strategy here?
Decompile the predicate (which is built by an NSPredicateEditor)
and prefetch the keypaths it takes? Prefetch everything?
Prefetching everything is usually undesirable. The easiest
approximate solution is to just track the hot button relationships for
each of your core entities, and look up which keypaths to prefetch by
entity name. Of course, you can also do a lot less filtering in
memory if you pass the predicates to the database with the fetch
request. You only need to prefetch relationships you are going to use
in memory. You don't need to prefetch anything that's simply used as
part of the predicate in the fetch request itself.
Walking through the predicate and gathering up the keypaths used is
very tedious, but not especially difficult, if you find yourself
wanting a complete solution.
- Ben
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
Am 24.08.2009 um 23:13 schrieb Ben Trumbull:
When I use setRelationshipKeyPathsForPrefetching the fetch throws:
-[NSSQLAttribute inverseRelationship]: unrecognized selector sent
to instance 0x10ee150
Can you provide the entire stack trace at this point ? gdb use
future-break objc_exception_throw
#0 0x97503f26 in objc_exception_throw ()
#1 0x944279db in -[NSObject(NSObject) doesNotRecognizeSelector:] ()
#2 0x94383026 in ___forwarding___ ()
#3 0x94382bf2 in __forwarding_prep_0___ ()
#4 0x97b3acae in -[NSSQLCore
_prefetchRelationshipKey:sourceEntityDescription:sourceObjectIDs:prefetchRelationshipKeys:inContext
:] ()
#5 0x97b3ab87 in -[NSSQLCore
_prefetchWithFetchRequest:withObjectIDs:inContext:] ()
#6 0x97b3d159 in -[NSSQLCore
_prefetchRelationshipKey:sourceEntityDescription:sourceObjectIDs:prefetchRelationshipKeys:inContext
:] ()
#7 0x97b3ab87 in -[NSSQLCore
_prefetchWithFetchRequest:withObjectIDs:inContext:] ()
#8 0x97b2c106 in newFetchedRowsForFetchPlan_MT ()
#9 0x97b164e7 in -[NSSQLCore newRowsForFetchPlan:] ()
#10 0x97b15d75 in -[NSSQLCore objectsForFetchRequest:inContext:] ()
#11 0x97b15991 in -[NSSQLCore executeRequest:withContext:] ()
#12 0x97b14ae1 in -[NSPersistentStoreCoordinator(_NSInternalMethods)
executeRequest:withContext:] ()
#13 0x97b11fff in -[NSManagedObjectContext executeFetchRequest:error:]
()
#14 0x00010b6e in +[GPPayPalTransaction
transactionsWithPredicate:sortDescriptors:returnObjectsAsFaults:relationshipKeyPathsForPrefetching:inManagedObjectContext:error
:] (self=0x10ebd4, _cmd=0x6a180, inPredicate=0x0,
inSortDescriptors=0x10adb80, inAsFaults=0 '\000',
inPrefetchKeyPaths=0x10ff990, inContext=0x10ad7e0,
outError=0xbfffe73c) at /Users/roddi/Development/GaragePay/
GPPayPalTransaction.m:100
(this is 10A432)
For the record: I do use one-way relationships
That should be okay as long as you're not pretending to use a no
inverse to-many relationship as a de facto many to many
relationship. No inverse relationships can only behave as if the
inverse were to-one. Specifically, each row in the destination
entity may only be associated with a single row in the source entity
if the relationship on the source entity does not have an inverse.
(e.g. you only get join tables in fully modeled bi-directional
relationships)
Do I understand that correctly? No two (or more) Accounts or
Transactions should point to the same Amount (see simplified
schema). Unless there's a bug somewhere that's not the case.
|-|
| Transaction |
|-| ||
|net | -- | Amount | -- (one other entity)
|gross| -- ||
|fee | -- |value | -
|account | - |currency||
|... |||||
|-|| |
| |
-- |
| |
||| |
|| Account| |
||| |
||balances| --
-- |transactions|
|... |
||
So basically I use Amount only as a container for a currency
descriptor and a value. I would have to model 7 inverse relationships
on that entity to make all relationships two-way.
Since it's not a many to many, you can perform the prefetching
effectively by hand using a fetch request to preload the relevant
destination rows with an IN query based on the data you initially
fetched for the source entity. You shouldn't have to, but if you've
run into a bug, that's how you could workaround it.
You still haven't described the NSPredicate you were using with
filteredArrayUsingPredicate. Being more forthcoming about the
context and details of your problem will help us get you answers
more quickly.
This is the predicate I was using for the test of the original post,
but since I use Smart Folders predicates can look a lot different
(i.e.: complex):
(additionalInfo.onTheFlyIsInternal == 0 AND
additionalInfo.isSuppressed != 1) AND (account.uniqueID IN
{D1AB3788-00DF-4475-A979-CE3EFC3987B5} OR FALSEPREDICATE)
AdditionalInfo has a one-to-one two-way relationship with Transaction
(not depicted above).
Hope that helps
Ruotger
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 25, 2009, at 7:19 AM, Ruotger Skupin wrote:
Am 24.08.2009 um 23:13 schrieb Ben Trumbull:
When I use setRelationshipKeyPathsForPrefetching the fetch throws:
-[NSSQLAttribute inverseRelationship]: unrecognized selector sent
to instance 0x10ee150
Can you provide the entire stack trace at this point ? gdb use
future-break objc_exception_throw
Thanks. What is the actual set of keypaths you set in the fetch
request ? setRelationshipKeyPathsForPrefetching expects all the
keypaths to terminate in a relationship object, not the attribute off
that related object.
(this is 10A432)
For the record: I do use one-way relationships
That should be okay as long as you're not pretending to use a no
inverse to-many relationship as a de facto many to many
relationship. No inverse relationships can only behave as if the
inverse were to-one. Specifically, each row in the destination
entity may only be associated with a single row in the source
entity if the relationship on the source entity does not have an
inverse. (e.g. you only get join tables in fully modeled bi-
directional relationships)
Do I understand that correctly? No two (or more) Accounts or
Transactions should point to the same Amount (see simplified
schema). Unless there's a bug somewhere that's not the case.
Correct. For each separate modeled no inverse relationship, the
destination row (Amount) can only be pointed to by one source row.
|-|
| Transaction |
|-| ||
|net | -- | Amount | -- (one other entity)
|gross| -- ||
|fee | -- |value | -
|account | - |currency||
|... |||||
|-|| |
| |
-- |
| |
||| |
|| Account| |
||| |
||balances| --
-- |transactions|
|... |
||
So basically I use Amount only as a container for a currency
descriptor and a value. I would have to model 7 inverse
relationships on that entity to make all relationships two-way.
Hmm. I'd recommend consider whether or not it makes sense to de-
normalize Amount into Transaction instead of making it a separate
entity. I assume the net/gross/fee are all going to be in the same
currency, yes ? Amount is very fine grained, and not (apparently)
offering very much value for the abstraction cost.
Since it's not a many to many, you can perform the prefetching
effectively by hand using a fetch request to preload the relevant
destination rows with an IN query based on the data you initially
fetched for the source entity. You shouldn't have to, but if
you've run into a bug, that's how you could workaround it.
You still haven't described the NSPredicate you were using with
filteredArrayUsingPredicate. Being more forthcoming about the
context and details of your problem will help us get you answers
more quickly.
This is the predicate I was using for the test of the original post,
but since I use Smart Folders predicates can look a lot different
(i.e.: complex):
(additionalInfo.onTheFlyIsInternal == 0 AND
additionalInfo.isSuppressed != 1) AND (account.uniqueID IN
{D1AB3788-00DF-4475-A979-CE3EFC3987B5} OR FALSEPREDICATE)
You'll want to prefetch additionalInfo and account.
- Ben
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
Am 20.08.2009 um 22:28 schrieb Ben Trumbull: setRelationshipKeyPathsForPrefetching When I use setRelationshipKeyPathsForPrefetching the fetch throws: -[NSSQLAttribute inverseRelationship]: unrecognized selector sent to instance 0x10ee150 For the record: I do use one-way relationships Ruotger ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 24, 2009, at 3:20 AM, Ruotger Skupin wrote: Am 20.08.2009 um 22:28 schrieb Ben Trumbull: setRelationshipKeyPathsForPrefetching When I use setRelationshipKeyPathsForPrefetching the fetch throws: -[NSSQLAttribute inverseRelationship]: unrecognized selector sent to instance 0x10ee150 Can you provide the entire stack trace at this point ? gdb use future- break objc_exception_throw For the record: I do use one-way relationships That should be okay as long as you're not pretending to use a no inverse to-many relationship as a de facto many to many relationship. No inverse relationships can only behave as if the inverse were to- one. Specifically, each row in the destination entity may only be associated with a single row in the source entity if the relationship on the source entity does not have an inverse. (e.g. you only get join tables in fully modeled bi-directional relationships) Since it's not a many to many, you can perform the prefetching effectively by hand using a fetch request to preload the relevant destination rows with an IN query based on the data you initially fetched for the source entity. You shouldn't have to, but if you've run into a bug, that's how you could workaround it. You still haven't described the NSPredicate you were using with filteredArrayUsingPredicate. Being more forthcoming about the context and details of your problem will help us get you answers more quickly. - Ben ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
Am 20.08.2009 um 04:38 schrieb M Pulis: I got that, Bill, thank you. The OP did not specify what particular sql store (we still do NOT know) nor if he is compiled Universal, has any helper programs, etc, he did however, describe symptoms I have seen with Rosetta. So I simply proposed checking that no PPC code was being engaged. Simple to do. For the record: I compiled Universal (debug version: Intel-only) 32bit- only. No helper programs. So it's pretty save to say, I'm not using Rosetta. Ruotger ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
Am 20.08.2009 um 11:34 schrieb Ruotger Skupin: Am 19.08.2009 um 22:00 schrieb Ben Trumbull: I debugged it with some Snow Leopard magic and found out, that firing faults is very slow the first time after boot. When I use -[NSArray filteredArrayUsingPredicate:] the fault firing is killing me. What's the predicate you are using ? I'm not using a predicate for the first fetch. (Ok, I have learned now, I should do so.) You're almost certainly filtering in memory after fetching only a subset of the data needed to perform your in memory filtering (specifically, only the source entity and not its relationships). Yes I do some on-the-fly filtering which depends on other objects. (Ok, lesson learned: Bad idea) Since you didn't fetch the related objects, Core Data goes and retrieves them on demand. However, mind reading is technically challenging, and Plan B, retrieving the related objects one at a time, is a rather inefficient I/O pattern. It would be vastly faster to filter the data in the database using a predicate on the fetch request. So, would it be better to perform the on-the-fly filtering calculation every time the object changes and put flags in the root object? Then fetch with a predicate testing these flags? Probably. (model change! yay!) If that's not possible, then you should use a fetch request predicate that approximates what you need, and filter those results more precisely in memory. That's what I'm doing now because the most common use case is: I need almost everything. So the approximation is: I fetch everything and filter out some objects. Like a bounding rect in views. If you have already fetched the objects, but want to identity different subsets, you can use objectID fetch results to simply identify the matching objects without pulling back all the row data. The subsets depend on the data of the objects. (Smart Folders) Complex locale aware Unicode text queries can be slow. If you find yourself spending time with such a query, you should consider some of the techniques shown in the DerivedProperty example available on ADC. Isn't all text Unicode? I don't understand. This shouldn't be a special case. But I will have a look at the sample. In my case I'd guess that at least half of the objects contain unicode strings (international names and addresses). What I want to say: write anything in German or French and you end up with Unicode. Typically, apps that launch slowly once, and then run quickly afterwards are doing far too much I/O. On successive launches, the kernel has cached most of the data, so instead of doing to-disk I// O the app is doing to-kernel-memory I/O. As stated before I need almost all data anyway. Is there a way to bulk-load the whole database into mem in one go? That's not what sql is about, is it? The most common problem in Core Data apps is when people allow faulting to lazily fetch data one row at a time, even when the app needs all the data immediately. They should prefetch the related objects with the original fetch request by setting the prefetching keypaths. Ok, will try to do that. Another question: I'm using Garbage Collection. Should I look out for something special there? Keep managed objects on purpose? Letting them go as soon as possible? Thanks for the insight Ruotger ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 19, 2009, at 10:38 PM, M Pulis wrote: My whole deal was about any PPC tasks in the OP's product's chain of execution, as I.S. selectively missed then exaggerated the off topic point (#5 of 8) in his surgical excerpt of my detailed response to his questioning of my suggestion to the OP proving me wrong, perhaps for even suggesting PPC in the first place. Yeccch. :-P (sigh) You're being obnoxious. You accuse me of focusing on one thing, unrelated to the main point, just to prove you wrong. Ignoring the egotism of that, let's get to the meat (since this is a technical list, not digg.com). I asked for clarification since your original message only implicated Rosetta because of what you (and others) assumed to be a slow-performing library startup. It was your choice to continue citing sqlite since both posters might have this in common. You said, A guess based on logic and deduction, my Dear Watson. Okay logic and deduction it is. If the code was built for PPC and run on PPC, Rosetta is not needed. If the code was built for Intel and run on Intel, Rosetta is not needed. Since you can't mix architectures (something someone who knows enough about Rosetta to imagine the need to warm it up on login could be reasonably expected to have encountered in the documentation), that leaves only compiling for PPC and running on Intel, or forcing the Universal Binary to run under Rosetta on Intel. Since the OP said in his second message, I debugged it with some Snow Leopard magic..., that tells us that it's fairly unlikely he'd be purposefully compiling only for PPC or forcibly running the application under Rosetta to use the PPC code. Unlikely but not impossible, I'll grant you that. But extremely unlikely. A quick verification of one's build settings is all that's needed, sure, but that's not what I was asking. I was asking you to explain your reasoning out of curiosity (and a bit of confusion). Your detailed response (a third of which contained things like, I'm bored and repeated information) condenses down to about 5 or 6 interrelated points to explain your reasoning. Points 4 and 6 read more like assertions and are based on a complete assumption. Really, the only point that needed addressing (considering the above) was point 5. My original question was just that - a question. A request for you to explain what appeared to me to be an interesting leap of logic. Nobody said or implied your suggestion was unwelcome (you pulled that one out of thin air). I asked you to elaborate and you became offended when your suggestion / elaboration was shown to be wrong. It's as simple as that. Can we please stop the crying and keep things on a technical level? -- I.S. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
ok best, gary ToothPics Company 1.800.218.0531 On Aug 20, 2009, at 2:16 AM, Ruotger Skupin li...@chipmunk-app.com wrote: Am 20.08.2009 um 04:38 schrieb M Pulis: I got that, Bill, thank you. The OP did not specify what particular sql store (we still do NOT know) nor if he is compiled Universal, has any helper programs, etc, he did however, describe symptoms I have seen with Rosetta. So I simply proposed checking that no PPC code was being engaged. Simple to do. For the record: I compiled Universal (debug version: Intel-only) 32bit-only. No helper programs. So it's pretty save to say, I'm not using Rosetta. Ruotger ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 20, 2009, at 2:34 AM, Ruotger Skupin wrote: Am 19.08.2009 um 22:00 schrieb Ben Trumbull: I debugged it with some Snow Leopard magic and found out, that firing faults is very slow the first time after boot. When I use -[NSArray filteredArrayUsingPredicate:] the fault firing is killing me. What's the predicate you are using ? I'm not using a predicate for the first fetch. (Ok, I have learned now, I should do so.) Okay, but what was the predicate you were passing to filteredArrayUsingPredicate: ? You're almost certainly filtering in memory after fetching only a subset of the data needed to perform your in memory filtering (specifically, only the source entity and not its relationships). Yes I do some on-the-fly filtering which depends on other objects. (Ok, lesson learned: Bad idea) Dynamic filtering isn't necessarily bad. The goal is always to perform the least amount of I/O and consume the least amount of memory. This produces the best performance. Since you didn't fetch the related objects, Core Data goes and retrieves them on demand. However, mind reading is technically challenging, and Plan B, retrieving the related objects one at a time, is a rather inefficient I/O pattern. It would be vastly faster to filter the data in the database using a predicate on the fetch request. So, would it be better to perform the on-the-fly filtering calculation every time the object changes and put flags in the root object? Then fetch with a predicate testing these flags? Probably. (model change! yay!) Uhm, I think we're getting way ahead of ourselves. That's probably serious overkill for the problem you're currently facing. If that's not possible, then you should use a fetch request predicate that approximates what you need, and filter those results more precisely in memory. That's what I'm doing now because the most common use case is: I need almost everything. So the approximation is: I fetch everything and filter out some objects. Well, we can provide better suggestions with the actual filtering predicate you're using. But the primary point here is that you're NOT fetching everything. You're only fetching the objects on one side of a relationship, but filtering based on information across a relationship. Like a bounding rect in views. If you have already fetched the objects, but want to identity different subsets, you can use objectID fetch results to simply identify the matching objects without pulling back all the row data. The subsets depend on the data of the objects. (Smart Folders) Complex locale aware Unicode text queries can be slow. If you find yourself spending time with such a query, you should consider some of the techniques shown in the DerivedProperty example available on ADC. Isn't all text Unicode? I don't understand. This shouldn't be a special case. But I will have a look at the sample. You can HAVE as much Unicode text as you want. That's just bytes. But performing regex or CONTAINS searches on Unicode text is very expensive. In my case I'd guess that at least half of the objects contain unicode strings (international names and addresses). What I want to say: write anything in German or French and you end up with Unicode. Typically, apps that launch slowly once, and then run quickly afterwards are doing far too much I/O. On successive launches, the kernel has cached most of the data, so instead of doing to-disk I// O the app is doing to-kernel-memory I/O. As stated before I need almost all data anyway. Is there a way to bulk-load the whole database into mem in one go? That's not what sql is about, is it? setRelationshipKeyPathsForPrefetching The most common problem in Core Data apps is when people allow faulting to lazily fetch data one row at a time, even when the app needs all the data immediately. They should prefetch the related objects with the original fetch request by setting the prefetching keypaths. Ok, will try to do that. Another question: I'm using Garbage Collection. Should I look out for something special there? Keep managed objects on purpose? Letting them go as soon as possible? One thing at a time. Add the appropriate prefetching to your fetch request and then reprofile the app. - Ben ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 20, 2009, at 02:35, Ruotger Skupin wrote:
Complex locale aware Unicode text queries can be slow. If you
find yourself spending time with such a query, you should consider
some of the techniques shown in the DerivedProperty example
available on ADC.
Isn't all text Unicode?
No. Not all apps are Unicode based, and many of the ones that aren't
will put things on the pasteboard quite happily. The web (and thus
anything copied out of a web browser) is definitely not all Unicode,
especially the older pages. And even within Unicode there are
multiple encoding formats (8. 16, and 32 bit). In addition to the
varying encoding sizes, Unicode also has multiple ways to represent
conceptual characters. Characters that have diacritics for example,
can be represented as either one Unichar ('é') or two ('´' + 'e').
I don't understand. This shouldn't be a special case. But I will
have a look at the sample.
In my case I'd guess that at least half of the objects contain
unicode strings (international names and addresses). What I want to
say: write anything in German or French and you end up with Unicode.
Due to the multiplicity of representations, text comparisons in
Unicode can be slow, since instead of just doing a byte by byte
comparison, you end needing to calculate character sizes, check for
compositions/decompositions, check for analogues between different
symbol systems used to represent a single language (ie kana and
kanjii), recognize and drop punctuation, etc. For apps that do
repeated comparisons against a set of strings, it can be worth it to
preprocess all strings into one canonical format to minimize the
amount of work that needs to be done during a comparison (make all
strings UTF8/16/32, make all characters lowercase, strip all
diacritics or ensure characters that have them are always in either
their composed or decomposed forms, etc) and then use a less expensive
collation for the comparison.
As a side node, if you want to use regular expressions on Unicode
strings, you generally need to do the normalization anyway, since
regex languages operate at the Unichar level rather than at the
conceptual character level.
+Melissa
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Core Data dog-slow when using first time after boot
Hi, when fetching about 5000 objects from an sql store, Core Data is very slow the very first time after a boot. When running the app the first time it takes 50 to 90 seconds and when starting it the second time it is well below one second. What is going on here? Has anyone noticed this too? I thought of writing a background app to load the store and fetch the objects when logging in, just in case the user might launch the app... But that's madness of course! I'm talking about Leopard 10.5.7 and 10.5.8 here. Don't know about Tiger (can't talk about SL) regards Ruotger Skupin ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 19, 2009, at 4:13 AM, Ruotger Skupin wrote: when fetching about 5000 objects from an sql store, Core Data is very slow the very first time after a boot. When running the app the first time it takes 50 to 90 seconds and when starting it the second time it is well below one second. What is going on here? Has anyone noticed this too? Did you try using Shark to find out why this is happening? Nick Zitzmann http://www.chronosnet.com/ ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
Am 19.08.2009 um 18:38 schrieb Nick Zitzmann: On Aug 19, 2009, at 4:13 AM, Ruotger Skupin wrote: when fetching about 5000 objects from an sql store, Core Data is very slow the very first time after a boot. When running the app the first time it takes 50 to 90 seconds and when starting it the second time it is well below one second. What is going on here? Has anyone noticed this too? Did you try using Shark to find out why this is happening? According to shark basically nothing plus some time spend in faults. Shark is useless when perf-debugging Core Data. I debugged it with some Snow Leopard magic and found out, that firing faults is very slow the first time after boot. When I use -[NSArray filteredArrayUsingPredicate:] the fault firing is killing me. So I recoded it to fetch everything upfront with [fetchRequest setReturnsObjectsAsFaults:NO]; and the fetching of 5000 objects still takes more than 15 seconds. On second launch that's about 3 seconds. This seems to be slow. Ruotger ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 19, 2009, at 1:14 PM, Ruotger Skupin wrote: I debugged it with some Snow Leopard magic and found out, that firing faults is very slow the first time after boot. When I use - [NSArray filteredArrayUsingPredicate:] the fault firing is killing me. So I recoded it to fetch everything upfront with [fetchRequest setReturnsObjectsAsFaults:NO]; and the fetching of 5000 objects still takes more than 15 seconds. On second launch that's about 3 seconds. This seems to be slow. Hmm ... time to hit the books if you haven't already: http://developer.apple.com/documentation/Cocoa/Conceptual/CoreData/Articles/cdPerformance.html#//apple_ref/doc/uid/TP40003468 Have you tried anything suggested there? -- I.S. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
Am 19.08.2009 um 19:18 schrieb I. Savant: Hmm ... time to hit the books if you haven't already: http://developer.apple.com/documentation/Cocoa/Conceptual/CoreData/Articles/cdPerformance.html#/ /apple_ref/doc/uid/TP40003468 Have you tried anything suggested there? Fetch Limits: Not tried The App basically loads objects and displays them in a table, so I can't fetch only half of them as sorting depends on the contents of the objects. I do fetches with sort descriptors though, to keep sorting time down (always below 0.1 sec). Batch faulting: tried It helps but it is still too slow. Pre-fetching: not tried But if I understand the document right it's just a special case of Batch Faulting and I don't see any faults fired since I use Batch Faulting anyway Reducing Memory Overhead: most don't apply (Garbage Collection / no temporary managed objects) I set the undo manager to nil Large Data Objects (BLOBs): not used I'd say all object are under 1k Would switching to a different store type (other than SQL) help? The database is not that large (in my example 55MB) maybe an atomic store has its advantages there? Ruotger ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
Ruotger, Interestingly enough, I experienced this behavior in my latest app which doesn't use Core Data. It uses SQLite directly instead. I recalled I had experienced this a long time ago (years ago) and someone (I don't remember who and where) mentioned a solution/ workaround/hack, which involves reading the file. Let's call this function warmUpFile(): FILE *my_stream = NULL; const char *my_filename = ...; size_t object_size = 1024; size_t object_count = 1024; char buf[object_size * object_count]; my_stream = fopen (my_filename, r); while (fread (buf, object_size, object_count, my_stream) 0); fclose (my_stream); I have timed it both modes, always after rebooting my computer: Reboot Launch app sqlite3_open_v2 do stuff == 35 seconds Reboot Launch app warmUpFile() sqlite3_open_v2 do stuff == 7 seconds Once warmUpFile() has been executed, let Core Data open the store and see if it helps. -- Tito On Aug 19, 2009, at 10:45 AM, Ruotger Skupin wrote: Am 19.08.2009 um 19:18 schrieb I. Savant: Hmm ... time to hit the books if you haven't already: http://developer.apple.com/documentation/Cocoa/Conceptual/CoreData/Articles/cdPerformance.html#//apple_ref/doc/uid/TP40003468 Have you tried anything suggested there? Fetch Limits: Not tried The App basically loads objects and displays them in a table, so I can't fetch only half of them as sorting depends on the contents of the objects. I do fetches with sort descriptors though, to keep sorting time down (always below 0.1 sec). Batch faulting: tried It helps but it is still too slow. Pre-fetching: not tried But if I understand the document right it's just a special case of Batch Faulting and I don't see any faults fired since I use Batch Faulting anyway Reducing Memory Overhead: most don't apply (Garbage Collection / no temporary managed objects) I set the undo manager to nil Large Data Objects (BLOBs): not used I'd say all object are under 1k Would switching to a different store type (other than SQL) help? The database is not that large (in my example 55MB) maybe an atomic store has its advantages there? Ruotger ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/tciuro%40mac.com This email sent to tci...@mac.com ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
Folks, This sounds like Rosetta. If you are seeing this on an intel machine, and if _any_ code in the execution path is non-intel, then Rosetta will startup, blocking until ready. I have an app called Rosetta Booster; when set as a login item, Rosetta is engaged, initialized, and ready to go, specifically reducing non-native first-app-startup-time to virtually the same as a native app. As this really might not be a Cocoa issue, drop me an off-list request if you would like to see/test Rosetta Booster - it is freeware. Gary On Aug 19, 2009, at 11:10 AM, Tito Ciuro wrote: Ruotger, Interestingly enough, I experienced this behavior in my latest app which doesn't use Core Data. It uses SQLite directly instead. I recalled I had experienced this a long time ago (years ago) and someone (I don't remember who and where) mentioned a solution/ workaround/hack, which involves reading the file. Let's call this function warmUpFile(): FILE *my_stream = NULL; const char *my_filename = ...; size_t object_size = 1024; size_t object_count = 1024; char buf[object_size * object_count]; my_stream = fopen (my_filename, r); while (fread (buf, object_size, object_count, my_stream) 0); fclose (my_stream); I have timed it both modes, always after rebooting my computer: Reboot Launch app sqlite3_open_v2 do stuff == 35 seconds Reboot Launch app warmUpFile() sqlite3_open_v2 do stuff == 7 seconds Once warmUpFile() has been executed, let Core Data open the store and see if it helps. -- Tito On Aug 19, 2009, at 10:45 AM, Ruotger Skupin wrote: Am 19.08.2009 um 19:18 schrieb I. Savant: Hmm ... time to hit the books if you haven't already: http://developer.apple.com/documentation/Cocoa/Conceptual/CoreData/Articles/cdPerformance.html#/ /apple_ref/doc/uid/TP40003468 Have you tried anything suggested there? Fetch Limits: Not tried The App basically loads objects and displays them in a table, so I can't fetch only half of them as sorting depends on the contents of the objects. I do fetches with sort descriptors though, to keep sorting time down (always below 0.1 sec). Batch faulting: tried It helps but it is still too slow. Pre-fetching: not tried But if I understand the document right it's just a special case of Batch Faulting and I don't see any faults fired since I use Batch Faulting anyway Reducing Memory Overhead: most don't apply (Garbage Collection / no temporary managed objects) I set the undo manager to nil Large Data Objects (BLOBs): not used I'd say all object are under 1k Would switching to a different store type (other than SQL) help? The database is not that large (in my example 55MB) maybe an atomic store has its advantages there? Ruotger ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/tciuro%40mac.com This email sent to tci...@mac.com ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/toothpic%40fastq.com This email sent to tooth...@fastq.com ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 19, 2009, at 3:09 PM, M Pulis wrote: This sounds like Rosetta. If you are seeing this on an intel machine, and if _any_ code in the execution path is non-intel, then Rosetta will startup, blocking until ready. I'm not sure how you've arrived at this conclusion based on the messages in this thread so far. Can you explain? -- I.S. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
(1) The subject: Re: Core Data dog-slow when using first time after boot (2) contains the key phrase dog slow when using first time after boot (3) I know that Rosetta only starts once and takes a lot of time, but not until the first non-native code hits. (4) also, one post from tito indicating seeing slow sqlite without core data, this logically changes the subject to dog-slow when using first time after boot (5) tito was close with warmupfile workaround (is sqlite native???) (6) none of the apple performance tips attempted apply - they won't because its a environment startup issue not an app runtime issue (7) Extremely easy to test - less time than this email. (8) I got bored watching no progress (9) I thought I would venture a guess based on Rosetta experience (10) no number ten, bored now. sounds like is a guess, not a conclusion A guess based on logic and deduction, my Dear Watson. :-) Gary On Aug 19, 2009, at 12:16 PM, I. Savant wrote: On Aug 19, 2009, at 3:09 PM, M Pulis wrote: This sounds like Rosetta. If you are seeing this on an intel machine, and if _any_ code in the execution path is non-intel, then Rosetta will startup, blocking until ready. I'm not sure how you've arrived at this conclusion based on the messages in this thread so far. Can you explain? -- I.S. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
re: Core Data dog-slow when using first time after boot
when fetching about 5000 objects from an sql store, Core Data is very slow the very first time after a boot. When running the app the first time it takes 50 to 90 seconds and when starting it the second time it is well below one second. Properly done, you could fetch 5000 objects on an original iPhone in a couple of seconds (or even less, depending on the NSFetchRequest options). On a Mac Pro, your performance is off by two orders of magnitude. If you wanted a result set, but didn't need all of it at once, you could use setBatchSize and handle about 3 million rows per second on a Mac Pro. Have you profiled your application ? Have you run Instruments on your app with the Core Data template ? It'll provide Core Data specific information about faulting, fetching and saving which pairs well with the more general CPU sampler. According to shark basically nothing plus some time spend in faults. Shark is useless when perf-debugging Core Data. Shark is very good at telling you where you are doing something expensive. Shark has just given you really valuable information (too many faults being tripped) and where in your code you are doing this expensive thing. Instead of describing that as useless, perhaps you should consider reducing the number of faults being tripped ... I debugged it with some Snow Leopard magic and found out, that firing faults is very slow the first time after boot. When I use -[NSArray filteredArrayUsingPredicate:] the fault firing is killing me. What's the predicate you are using ? You're almost certainly filtering in memory after fetching only a subset of the data needed to perform your in memory filtering (specifically, only the source entity and not its relationships). Since you didn't fetch the related objects, Core Data goes and retrieves them on demand. However, mind reading is technically challenging, and Plan B, retrieving the related objects one at a time, is a rather inefficient I/O pattern. It would be vastly faster to filter the data in the database using a predicate on the fetch request. If that's not possible, then you should use a fetch request predicate that approximates what you need, and filter those results more precisely in memory. Like a bounding rect in views. If you have already fetched the objects, but want to identity different subsets, you can use objectID fetch results to simply identify the matching objects without pulling back all the row data. Complex locale aware Unicode text queries can be slow. If you find yourself spending time with such a query, you should consider some of the techniques shown in the DerivedProperty example available on ADC. What is going on here? Typically, apps that launch slowly once, and then run quickly afterwards are doing far too much I/O. On successive launches, the kernel has cached most of the data, so instead of doing to-disk I//O the app is doing to-kernel-memory I/O. The most common problem in Core Data apps is when people allow faulting to lazily fetch data one row at a time, even when the app needs all the data immediately. They should prefetch the related objects with the original fetch request by setting the prefetching keypaths. Have you read through the Core Data Programming Guide's chapter on Performance ? https://developer.apple.com/documentation/Cocoa/Conceptual/CoreData/Articles/cdPerformance.html#//apple_ref/doc/uid/TP40003468 So I recoded it to fetch everything upfront with [fetchRequest setReturnsObjectsAsFaults:NO]; This isn't solving the problem you expect. You'll want to use prefetching and should read the programming guide. Pre-fetching: not tried But if I understand the document right it's just a special case of Batch Faulting and I don't see any faults fired since I use Batch Faulting anyway No, prefetching is not a special case of batch faulting. Batch faulting is about taking N objects (or objectIDs) and telling Core Data I want all of these, now. Prefetching is about telling Core Data I want the objects that match this fetch request, plus all of their related friends X and Y.Z - Ben ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 19, 2009, at 3:55 PM, M Pulis wrote: (5) tito was close with warmupfile workaround (is sqlite native???) Is SQLite native seems to be the one point on which all your others hinge. I'd be dutifully surprised if it were not. -- I.S. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
Interestingly enough, I experienced this behavior in my latest app which doesn't use Core Data. It uses SQLite directly instead. I recalled I had experienced this a long time ago (years ago) and someone (I don't remember who and where) mentioned a solution/ workaround/hack, which involves reading the file. Let's call this function warmUpFile(): This is treating the symptom instead of the disease of doing far too much I/O. Ruotger has a lot of more valuable optimization work to do first, before worrying about whether or not it makes sense to warm up the UBC cache. Nearly all the time, it makes more sense to perform fewer I/O operations, or otherwise reduce the quantity of I/O or improve their locality of reference. This kind of hack put a lot more memory pressure on the system, and is prone to induce VM paging. That would be even worse for performance. It's possible for SQLite databases to become fragmented, both on disk in traditional file fragmentation and internally as tables grow and shrink, they can lose their own locality of reference (imagine like malloc heap fragmentation). Fragmented database files can be reconstructed with better internal locality by using the vacuum pragma. This can be extremely expensive for very large files, however. File system fragmentation is usually handled at the same time, or by recopying the file elsewhere. Unless you are low on free space, in which case you are SOL. HFS+ volumes provide better performance with more free space. I'm not certain what the specific recommendations are, but I believe people are encouraged to keep at least 10% of the disk space available for optimal performance. It's possible to reduce internal db fragmentation by avoiding the auto- vacuum pragma and using incremental vacuum instead. Core Data does this now for all its clients. In general, we try to tune database performance out from underneath our clients by adopting new features both in SQLite and in Mac OS X. - Ben ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 19, 2009, at 1:05 PM, I. Savant wrote: On Aug 19, 2009, at 3:55 PM, M Pulis wrote: (5) tito was close with warmupfile workaround (is sqlite native???) Is SQLite native seems to be the one point on which all your others hinge. I'd be dutifully surprised if it were not. It is flat out impossible. A task -- a running program -- is either all PPC or all Intel or all 32 bit or all 64 bit. Period. No mixing allowed. b.bum ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 19, 2009, at 4:56 PM, Bill Bumgarner wrote: Is SQLite native seems to be the one point on which all your others hinge. I'd be dutifully surprised if it were not. It is flat out impossible. A task -- a running program -- is either all PPC or all Intel or all 32 bit or all 64 bit. Period. No mixing allowed. Hence my surprise. Wouldn't it surprise the hell out of YOU? :-D -- I.S. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
The sqlite question was a reference to tito saying his experience was years ago when the possibility of a non-native sqlite on an intel machine was absolutely real and significantly less surprising. You may know, but I have no idea what versions are installed on any given system, much less the OP's, responders, etc (you can execute PPC frameworks onto an Intel machine, yes?) what frameworks are in play, etc; and simply pose the question. Actually, I, #3 is the real hinge point you missed, totally irrespective of sqlite: (3) I know that Rosetta only starts once and takes a lot of time, but not until the first non-native code hits. Any non-native code will trigger Rosetta. Plus, it was just a suggestion. A very easy to test suggestion that can eliminate the Rosetta possibility in two reboots and potentially a lot of needless work=manhours=$$$. If it ain't Rosetta, fine... off into the depths of analysis and recoding (always a good idea) ... if it is Rosetta then one can better identify the offending ppc code chunk, update it and move on. Sorry to waste your time shooting down my suggestion - obviously experts know somehow that it can't possibly be Rosetta and are looking elsewhere in solving this with greater rapidity. gary On Aug 19, 2009, at 1:05 PM, I. Savant wrote: On Aug 19, 2009, at 3:55 PM, M Pulis wrote: (5) tito was close with warmupfile workaround (is sqlite native???) Is SQLite native seems to be the one point on which all your others hinge. I'd be dutifully surprised if it were not. -- I.S. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 19, 2009, at 2:21 PM, M Pulis wrote: The sqlite question was a reference to tito saying his experience was years ago when the possibility of a non-native sqlite on an intel machine was absolutely real and significantly less surprising. You may know, but I have no idea what versions are installed on any given system, much less the OP's, responders, etc (you can execute PPC frameworks onto an Intel machine, yes?) what frameworks are in play, etc; and simply pose the question. Even though I experienced this issue years ago in PPC, I'm currently running Intel and this behavior can still be seen. For the record, the database size is ~133MB. -- Tito ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 19, 2009, at 2:21 PM, M Pulis wrote: responders, etc (you can execute PPC frameworks onto an Intel machine, yes?) No, not from an native app. Sorry 'bout that. Extremely unlikely that Rosetta is the OP's problem unless there is something like a helper app that is getting launched also. Gary ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
On Aug 19, 2009, at 5:21 PM, M Pulis wrote: You may know, but I have no idea what versions are installed on any given system, much less the OP's, responders, etc (you can execute PPC frameworks onto an Intel machine, yes?) what frameworks are in play, etc; and simply pose the question. ... Plus, it was just a suggestion. ... Sorry to waste your time shooting down my suggestion - Your tone suggests you've been offended (hurt?) by the fact that your suggestion was questioned for clarity then ultimately proven wrong (or at least not relevant to the problem). You suggested a technical solution (warming up Rosetta) on a technical mailing list. One which I thought was wrong. Because I'm not omniscient, my ego permits me to ask you to elaborate on your suggestion in case you knew something I didn't. After all, if someone schools me, I'm the better for it. I gave you the benefit of doubt but it turned out your assumption was wrong. There was no berating, no beating you up, just a technical correction. That's all there is to it. Again, it's a *technical mailing list*. I'm not going to withhold the correct information for fear of hurting someone's feelings. -- I.S. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
No offense taken... as you saw, I have no problem even proving myself wrong to correct my postings. I suggested testing for the possibility of non-native code. Warming up Rosetta was NOT presented not as a solution but an aid to test the presence of rather than assuming not. My ego permits me to consider reasonable possibilities out of the box and pose the questions for consideration. I'd rather be wrong quickly than to be silent and come to find out I could have helped From the thread up to that point, not much else had helped the OP either until Ben, who may be the most direct help. I have seen native apps that launch helper programs that are still PPC to do grunt work via Rosetta. Not knowing or assuming the structure of the OP's app, development environment, build settings and what else may be going on... the technical details not readily apparent in the posts make it difficult to nail every problem. It will be interesting to see what ultimately helps the OP, should he share it. Gary On Aug 19, 2009, at 4:04 PM, I. Savant wrote: On Aug 19, 2009, at 5:21 PM, M Pulis wrote: You may know, but I have no idea what versions are installed on any given system, much less the OP's, responders, etc (you can execute PPC frameworks onto an Intel machine, yes?) what frameworks are in play, etc; and simply pose the question. ... Plus, it was just a suggestion. ... Sorry to waste your time shooting down my suggestion - Your tone suggests you've been offended (hurt?) by the fact that your suggestion was questioned for clarity then ultimately proven wrong (or at least not relevant to the problem). You suggested a technical solution (warming up Rosetta) on a technical mailing list. One which I thought was wrong. Because I'm not omniscient, my ego permits me to ask you to elaborate on your suggestion in case you knew something I didn't. After all, if someone schools me, I'm the better for it. I gave you the benefit of doubt but it turned out your assumption was wrong. There was no berating, no beating you up, just a technical correction. That's all there is to it. Again, it's a *technical mailing list*. I'm not going to withhold the correct information for fear of hurting someone's feelings. -- I.S. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data dog-slow when using first time after boot
I got that, Bill, thank you. The OP did not specify what particular sql store (we still do NOT know) nor if he is compiled Universal, has any helper programs, etc, he did however, describe symptoms I have seen with Rosetta. So I simply proposed checking that no PPC code was being engaged. Simple to do. My whole deal was about any PPC tasks in the OP's product's chain of execution, as I.S. selectively missed then exaggerated the off topic point (#5 of 8) in his surgical excerpt of my detailed response to his questioning of my suggestion to the OP proving me wrong, perhaps for even suggesting PPC in the first place. Yeccch. :-P I should not have questioned sqllite --- even I.S. knows that came from a different person sharing their experience, NOT the OP... the real task is improving the OP's startup time after boot. For that he needs more than just native code. Gary On Aug 19, 2009, at 6:21 PM, Bill Bumgarner wrote: The point was that sqlite could not possibly be non-native. On Aug 19, 2009, at 5:20 PM, M Pulis wrote: On Aug 19, 2009, at 1:56 PM, Bill Bumgarner wrote: On Aug 19, 2009, at 1:05 PM, I. Savant wrote: On Aug 19, 2009, at 3:55 PM, M Pulis wrote: (5) tito was close with warmupfile workaround (is sqlite native???) Is SQLite native seems to be the one point on which all your others hinge. I'd be dutifully surprised if it were not. It is flat out impossible. A task -- a running program -- is either all PPC or all Intel or all 32 bit or all 64 bit. Period. No mixing allowed. b.bum True, and a Intel app can launch a PPC app on intel, invoking Rosetta for the PPC app/task. Not knowing or assuming the structure of the OP's app, development environment and what else may be going on... In any event, even if the OP's app was not Universal, it may only improve by 10-15 seconds or so, a small dent. He has some bigger fish to fry. Gary ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
