RE: Speeding up commit
Actually on that point I think I agree that I have seen this very behaviour myself. OpenJPA gets confused if change B then change A then commit A (with cascade all set on A), I commonly get detached entity errors in reference to B. I ended up working around that by persisting B, reread it then persisting A. I got so tired of this issue I have been using Hibernate a lot more lately. Chris -Original Message- From: Aron Lurie [mailto:a...@cambridgesemantics.com] Sent: Thursday, 16 June 2011 5:44 AM To: users@openjpa.apache.org Subject: Re: Speeding up commit Also, When class A has a property of type class B, and both instances are persisted, the B instance does not need to be proxied because any mutation made on the B instance will be noticed by the entity manager since it is persisted. However, I believe the current behavior is to create a proxy for the B instance. If I am right, this is a source of redundancy that could be eliminated. -Aron On 6/15/2011 3:11 PM, Aron Lurie wrote: > Based on the documentation I've read, it didn't occur to me as a user > that proxying stemmed from detachment. I had the concept in mind that > proxying was a two part process: attachment - or putting the proxy in > place of the original object, and detachment - or replacing the > original object. > > Just my 2 cents. > > Anyways, I've found a way to speed up the proxying process, by caching > the result of my objects getters, since they take time to produce the > return value. > > -Aron > > On 6/15/2011 11:47 AM, Pinaki Poddar wrote: >>> it still attaches proxies during commit, >> It was my bad. Looks like your environment is ready to absorb changes. >> Please update on trunk -- that has a correct NONE logic that will >> bypass proxy on commit. >> >> - >> Pinaki >> -- >> View this message in context: >> http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479 >> 404.html Sent from the OpenJPA Users mailing list archive at >> Nabble.com. >
Re: Speeding up commit
Aron - Can you post your Entities?
Re: Speeding up commit
Looks like more than simple bulk insertion to me :) The interesting way you have chosen should not make first-class managed object be proxied. That is my remote observation. However, you can inspect that actual runtime class after an instance has entered a persistence context, to see for yourself. A a = new A(); B b = new B(); a.setB(b); em.persist(a); em.persist(b); assertSame(B.class, a.getB().getClass()); - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6480642.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
I am enhancing during runtime, but with the build-time enhancer. I know that sounds kind of strange, but I am using PCEnhance.run and a custom BytecodeWriter that writes to a custom Class Loader that allows me to add and use the enhanced classes during runtime. Would this cause managed objects to be proxied? Would it matter if the declared return type is different from the type I specify as targetEntity? -Aron On 6/15/2011 4:43 PM, Pinaki Poddar wrote: Proxies are *not* created for managed objects, in general. This brings me to my favorite question: are you enhancing the persistent classes at build-time? If not, please do. It will save you ton of trouble. For normal (I mean build-time enhanced) persistence mode, OpenJPA creates proxies for 'second-class objects' -- the objects that are "managed" by the runtime in some sense but *could* mutate without OpenJPA knowing about such mutation. Examples are instance of java.util.Date or java.util.Collection. OpenJPA proxies them because then it can be aware of their mutation and act appropriately to capture these changes. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6480502.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Proxies are *not* created for managed objects, in general. This brings me to my favorite question: are you enhancing the persistent classes at build-time? If not, please do. It will save you ton of trouble. For normal (I mean build-time enhanced) persistence mode, OpenJPA creates proxies for 'second-class objects' -- the objects that are "managed" by the runtime in some sense but *could* mutate without OpenJPA knowing about such mutation. Examples are instance of java.util.Date or java.util.Collection. OpenJPA proxies them because then it can be aware of their mutation and act appropriately to capture these changes. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6480502.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Also, When class A has a property of type class B, and both instances are persisted, the B instance does not need to be proxied because any mutation made on the B instance will be noticed by the entity manager since it is persisted. However, I believe the current behavior is to create a proxy for the B instance. If I am right, this is a source of redundancy that could be eliminated. -Aron On 6/15/2011 3:11 PM, Aron Lurie wrote: Based on the documentation I've read, it didn't occur to me as a user that proxying stemmed from detachment. I had the concept in mind that proxying was a two part process: attachment - or putting the proxy in place of the original object, and detachment - or replacing the original object. Just my 2 cents. Anyways, I've found a way to speed up the proxying process, by caching the result of my objects getters, since they take time to produce the return value. -Aron On 6/15/2011 11:47 AM, Pinaki Poddar wrote: it still attaches proxies during commit, It was my bad. Looks like your environment is ready to absorb changes. Please update on trunk -- that has a correct NONE logic that will bypass proxy on commit. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479404.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Based on the documentation I've read, it didn't occur to me as a user that proxying stemmed from detachment. I had the concept in mind that proxying was a two part process: attachment - or putting the proxy in place of the original object, and detachment - or replacing the original object. Just my 2 cents. Anyways, I've found a way to speed up the proxying process, by caching the result of my objects getters, since they take time to produce the return value. -Aron On 6/15/2011 11:47 AM, Pinaki Poddar wrote: it still attaches proxies during commit, It was my bad. Looks like your environment is ready to absorb changes. Please update on trunk -- that has a correct NONE logic that will bypass proxy on commit. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479404.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
There was an error in initial commit. Please try Revision 1135857. Also please let us know your observation with this changes. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479537.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
> Adding another property to > the ProxyManager to not use proxies from the get-go makes sense to me. Actually, I have considered that option. The problem with it is that ProxyManager is per unit, not per context. Going via the route that I have taken allows the user to deactivate proxy for one context (say when it is loading data in bulk) without affecting behavior for other contexts. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479527.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
@Mike will get it sorted. Please allow some time. I am out of practice and my old laptop is gone :) I think Hudson build is quiet now -- last time it was unhappy was about 7:18PM night before. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479506.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
@Mike, OPENJPA-2017 The use case is important and useful. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479476.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Yes proxy could track changes in second-class objects, if you tell them to do so. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479464.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
> it still attaches proxies during commit, It was my bad. Looks like your environment is ready to absorb changes. Please update on trunk -- that has a correct NONE logic that will bypass proxy on commit. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6479404.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Unfortunately we don't have a good answer for your use case. I spent a little (very little) time looking into removing the proxies altogether and it wasn't as simple as just putting the non proxies into the entity. There should be a jira created for this use case, but I haven't been able to find one. Do you have a JIRA acount, and would you mind opening one? -mike On Wed, Jun 15, 2011 at 9:52 AM, Aron Lurie wrote: > We also need the proxies for for managed entities to track the state of >> non >> primitives (Dates, collections etc.). >> > So in my application, I know that the state of the non primitives will not > be changing. All I need OpenJPA to do is extract the value of the > non-primitive once when it is creating its insert statement. > > > It will not proxy with for AutoDetachType.NONE. >> > I have already set this property, and it allows me to close the entity > manager much faster because it does not bother detaching the proxies, but it > still attaches proxies during commit, so commit does not run any faster. > > -Aron > > > On 6/15/2011 10:37 AM, Michael Dick wrote: > >> Rick's right on both counts. TrackChanges doesn't eliminate proxies - it >> should just make them no-op. >> >> We also need the proxies for for managed entities to track the state of >> non >> primitives (Dates, collections etc.). I don't think we have code in place >> that falls back and does a more thorough comparison if the proxies are not >> found though. >> >> Pinaki, >> >> The code changes are definitely untested - it's currently breaks the >> TestEnumToKernelConstantMappings test (which is rather banal, but probably >> there for a good reason). >> >> I'm not sure what you mean about not having a regression test environment. >> This problem would be found in a rather quick maven build. I understand >> not >> having multiple databases available, but running the regression bucket >> with >> derby should be doable. >> >> You can skip the long running locking tests with this arg: >> -Dsurefire.excludes.locking=**, if time is a concern. >> >> -mike >> >> On Wed, Jun 15, 2011 at 9:11 AM, Rick Curtis wrote: >> >> Javadoc from ProxyManagerImpl >>> >>>/** >>> * Whether proxies produced by this factory will use {@link >>> ChangeTracker}s >>> * to try to cut down on data store operations at the cost of some >>> extra >>> * bookkeeping overhead. Defaults to true. >>> */ >>>public boolean getTrackChanges() { >>>return _trackChanges; >>>} >>> >>> It sounds like this property is used to determine whether the proxies are >>> tracking changes... not to toggle the creation. Adding another property >>> to >>> the ProxyManager to not use proxies from the get-go makes sense to me. >>> >>> On Wed, Jun 15, 2011 at 8:58 AM, Kevin Sutter >>> wrote: >>> >>> Hi guys, Shouldn't this property setting turn off the proxy usage? That's the way I read the documentation, but it doesn't seem to work that way. We still get the proxies created. Actually, I don't see much difference in processing whether this is set to True or False. Is this a bug, or am I reading the documentation wrong? Thanks, Kevin On Tue, Jun 14, 2011 at 7:53 PM, Rick Curtis wrote: Aren't proxies also used to track changes while a persistence context > is >>> active? > > Rick Curtis > > >>> >>> -- >>> *Rick Curtis* >>> >>> >
Re: Speeding up commit
We also need the proxies for for managed entities to track the state of non primitives (Dates, collections etc.). So in my application, I know that the state of the non primitives will not be changing. All I need OpenJPA to do is extract the value of the non-primitive once when it is creating its insert statement. It will not proxy with for AutoDetachType.NONE. I have already set this property, and it allows me to close the entity manager much faster because it does not bother detaching the proxies, but it still attaches proxies during commit, so commit does not run any faster. -Aron On 6/15/2011 10:37 AM, Michael Dick wrote: Rick's right on both counts. TrackChanges doesn't eliminate proxies - it should just make them no-op. We also need the proxies for for managed entities to track the state of non primitives (Dates, collections etc.). I don't think we have code in place that falls back and does a more thorough comparison if the proxies are not found though. Pinaki, The code changes are definitely untested - it's currently breaks the TestEnumToKernelConstantMappings test (which is rather banal, but probably there for a good reason). I'm not sure what you mean about not having a regression test environment. This problem would be found in a rather quick maven build. I understand not having multiple databases available, but running the regression bucket with derby should be doable. You can skip the long running locking tests with this arg: -Dsurefire.excludes.locking=**, if time is a concern. -mike On Wed, Jun 15, 2011 at 9:11 AM, Rick Curtis wrote: Javadoc from ProxyManagerImpl /** * Whether proxies produced by this factory will use {@link ChangeTracker}s * to try to cut down on data store operations at the cost of some extra * bookkeeping overhead. Defaults to true. */ public boolean getTrackChanges() { return _trackChanges; } It sounds like this property is used to determine whether the proxies are tracking changes... not to toggle the creation. Adding another property to the ProxyManager to not use proxies from the get-go makes sense to me. On Wed, Jun 15, 2011 at 8:58 AM, Kevin Sutter wrote: Hi guys, Shouldn't this property setting turn off the proxy usage? That's the way I read the documentation, but it doesn't seem to work that way. We still get the proxies created. Actually, I don't see much difference in processing whether this is set to True or False. Is this a bug, or am I reading the documentation wrong? Thanks, Kevin On Tue, Jun 14, 2011 at 7:53 PM, Rick Curtis wrote: Aren't proxies also used to track changes while a persistence context is active? Rick Curtis -- *Rick Curtis*
Re: Speeding up commit
Rick's right on both counts. TrackChanges doesn't eliminate proxies - it should just make them no-op. We also need the proxies for for managed entities to track the state of non primitives (Dates, collections etc.). I don't think we have code in place that falls back and does a more thorough comparison if the proxies are not found though. Pinaki, The code changes are definitely untested - it's currently breaks the TestEnumToKernelConstantMappings test (which is rather banal, but probably there for a good reason). I'm not sure what you mean about not having a regression test environment. This problem would be found in a rather quick maven build. I understand not having multiple databases available, but running the regression bucket with derby should be doable. You can skip the long running locking tests with this arg: -Dsurefire.excludes.locking=**, if time is a concern. -mike On Wed, Jun 15, 2011 at 9:11 AM, Rick Curtis wrote: > Javadoc from ProxyManagerImpl > >/** > * Whether proxies produced by this factory will use {@link > ChangeTracker}s > * to try to cut down on data store operations at the cost of some extra > * bookkeeping overhead. Defaults to true. > */ >public boolean getTrackChanges() { >return _trackChanges; >} > > It sounds like this property is used to determine whether the proxies are > tracking changes... not to toggle the creation. Adding another property to > the ProxyManager to not use proxies from the get-go makes sense to me. > > On Wed, Jun 15, 2011 at 8:58 AM, Kevin Sutter wrote: > > > Hi guys, > > Shouldn't this property setting turn off the proxy usage? > > > > > > > > That's the way I read the documentation, but it doesn't seem to work that > > way. We still get the proxies created. Actually, I don't see much > > difference in processing whether this is set to True or False. Is this a > > bug, or am I reading the documentation wrong? > > > > Thanks, > > Kevin > > > > On Tue, Jun 14, 2011 at 7:53 PM, Rick Curtis wrote: > > > > > Aren't proxies also used to track changes while a persistence context > is > > > active? > > > > > > Rick Curtis > > > > > > > > > -- > *Rick Curtis* >
Re: Speeding up commit
Javadoc from ProxyManagerImpl /** * Whether proxies produced by this factory will use {@link ChangeTracker}s * to try to cut down on data store operations at the cost of some extra * bookkeeping overhead. Defaults to true. */ public boolean getTrackChanges() { return _trackChanges; } It sounds like this property is used to determine whether the proxies are tracking changes... not to toggle the creation. Adding another property to the ProxyManager to not use proxies from the get-go makes sense to me. On Wed, Jun 15, 2011 at 8:58 AM, Kevin Sutter wrote: > Hi guys, > Shouldn't this property setting turn off the proxy usage? > > > > That's the way I read the documentation, but it doesn't seem to work that > way. We still get the proxies created. Actually, I don't see much > difference in processing whether this is set to True or False. Is this a > bug, or am I reading the documentation wrong? > > Thanks, > Kevin > > On Tue, Jun 14, 2011 at 7:53 PM, Rick Curtis wrote: > > > Aren't proxies also used to track changes while a persistence context is > > active? > > > > Rick Curtis > > > -- *Rick Curtis*
Re: Speeding up commit
Hi guys, Shouldn't this property setting turn off the proxy usage? That's the way I read the documentation, but it doesn't seem to work that way. We still get the proxies created. Actually, I don't see much difference in processing whether this is set to True or False. Is this a bug, or am I reading the documentation wrong? Thanks, Kevin On Tue, Jun 14, 2011 at 7:53 PM, Rick Curtis wrote: > Aren't proxies also used to track changes while a persistence context is > active? > > Rick Curtis >
Re: Speeding up commit
Aren't proxies also used to track changes while a persistence context is active? Rick Curtis
Re: Speeding up commit
Proxying/Deproxying stems from detachment. We proxy because we want to tract changes in detached objects. AutoDetach.NONE basically tells: 'do not bother detaching at all, not going to use these objects anyway ever'. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6476649.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
> It will not proxy with for AutoDetachType.NONE. That is confusing my simple mind Why do I need to set an AutoDetach flag to tell OpenJPA to not use proxies? >These stuff is highly untested because I do not have access to a ... I'm not hung up on the right/wrong-ness of your change... I'm still thinking about it conceptually. On Tue, Jun 14, 2011 at 4:43 PM, Pinaki Poddar wrote: > It will not proxy with for AutoDetachType.NONE. > See StateManagerImpl commit log for further details. > > These stuff is highly untested because I do not have access to a regression > test environment right now. I am still committing so that it gets tested > with other commits. I will rollback if things just start breaking. > > - > Pinaki > > -- *Rick Curtis*
Re: Speeding up commit
It will not proxy with for AutoDetachType.NONE. See StateManagerImpl commit log for further details. These stuff is highly untested because I do not have access to a regression test environment right now. I am still committing so that it gets tested with other commits. I will rollback if things just start breaking. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6476127.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Is the problem when OpenJPA tries to proxy user supplied objects on transaction commit(flush)? Or is the problem when the persistence context detaches and OpenJPA spends cycles trying to remove the proxies that we just put in? Pinaki - Did you solve the both problems, or just the second? ... I thought the user Aron was reporting a problem with the fact that we ever put proxies in.? On Tue, Jun 14, 2011 at 3:32 PM, Pinaki Poddar wrote: > Hello, > For batch insertion, > 1. Set openjpa.RetainState=false > this might gain few extra cycles > > I had introduced a tentative option [1] for automatic detachment (your > observation of cycles being spent on StateManager.proxyFields(...) is > related to detachment process that OpenJPA runtime invokes at > user-configurable points in the lifetime of a persistence context such as > transaction endpoints or when EntityManager.close() is called). > > If the user application does not intend to refer to the managed entities > later (as it seems to fits the use case you are describing), then we can > bypass the overhead of detachment. > > In a quick Person-has-many-Address type model, I notice a ~20% reduction > in data insert with this new NONE option. > > To access, and if possible verify efficacy of this new feature, please do > the following > > 1. get a nightly build or build from trunk locally. > 2. Ensure that revision 1135776 is included. Easy way to do this >$ java -jar openjpa.jar > will print the latest revision included in the jar > 3. In user application, > em.setProperty("openjpa.AutoDetach", AutoDetachType.NONE); >Note that the option must be set on an instantiated EntityManager, and > *not* in persistence.xml > > [1] https://issues.apache.org/jira/browse/OPENJPA-2017 > > - > Pinaki > > -- *Rick Curtis*
Re: Speeding up commit
Hello, For batch insertion, 1. Set openjpa.RetainState=false this might gain few extra cycles I had introduced a tentative option [1] for automatic detachment (your observation of cycles being spent on StateManager.proxyFields(...) is related to detachment process that OpenJPA runtime invokes at user-configurable points in the lifetime of a persistence context such as transaction endpoints or when EntityManager.close() is called). If the user application does not intend to refer to the managed entities later (as it seems to fits the use case you are describing), then we can bypass the overhead of detachment. In a quick Person-has-many-Address type model, I notice a ~20% reduction in data insert with this new NONE option. To access, and if possible verify efficacy of this new feature, please do the following 1. get a nightly build or build from trunk locally. 2. Ensure that revision 1135776 is included. Easy way to do this $ java -jar openjpa.jar will print the latest revision included in the jar 3. In user application, em.setProperty("openjpa.AutoDetach", AutoDetachType.NONE); Note that the option must be set on an instantiated EntityManager, and *not* in persistence.xml [1] https://issues.apache.org/jira/browse/OPENJPA-2017 - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6475817.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Unfortunately it's looking like a this is a code path which we don't have an optimizations for Essentially you're using OpenJPA to put data into a DB and then once your tran commits you're not going to use the Entities any longer.? If you're looking for a workaround, you could noop org.apache.openjpa.kernel.SingleFieldManager.proxy(...) and just return true. Not ideal, but it might work for you in the short term. On Tue, Jun 14, 2011 at 10:35 AM, C N Davies wrote: > Hi Aron, > > Yes that is what I meant, I persist and commit 200 at a time. When my > objects are highly complex I threw in em.clear() as I was getting stack > issues as well. > > Chris > > -Original Message- > From: Aron Lurie [mailto:a...@cambridgesemantics.com] > Sent: Wednesday, 15 June 2011 1:06 AM > To: users@openjpa.apache.org > Subject: Re: Speeding up commit > > Chris, > What do you mean by batching? Do you mean persisting 200, committing, > persisting another 200, committing, etc? If so, I don't have the option of > doing this because my objects reference one another so the cascading > persist > would blow up my batch size. > > Thanks, > Aron > > On 6/14/2011 10:54 AM, C N Davies wrote: > > I had a load of issues with committing large quantities of entities > > (OpenJPA 2.0), in the end I batched them into batches of 200 and the > > overall performance was significantly better. This is not a > > scientific analysis of the issue I know, but might be of use to you :) > > > > Chris > > > > > > -Original Message- > > From: Aron Lurie [mailto:a...@cambridgesemantics.com] > > Sent: Wednesday, 15 June 2011 12:50 AM > > To: users@openjpa.apache.org > > Subject: Re: Speeding up commit > > > > 2.1.0 > > > > On 6/14/2011 10:47 AM, Rick Curtis wrote: > >> What version of OpenJPA are you running? > >> > >> On Tue, Jun 14, 2011 at 9:45 AM, Aron > > Luriewrote: > >>> Hello, > >>> I have a situation where I am trying to persist ~2500 objects all at > > once. > >>> After the objects are persisted, the program ends, so there is no > >>> need for any functionality after the objects have been persisted. I > >>> am just trying to dump these objects to the DB as fast as possible. > >>> > >>> Using Eclipse's TPTP profiler, I have timed the > >>> EntityManager.commit() operation to take 586 seconds. Of that, > >>> cumulatively almost 455 seconds are spent inside of ~2500 calls to > > StateManagerImpl.proxyFields(bool, bool). > >>>From the point that I start commit(), those 2nd class objects are > >>> not mutated by my program, and I do not need to use any of these > >>> objects after > >>> commit() finishes. From what I have read, it would seem that I have > >>> no need for my objects to be proxied, and from my tests, it seems > >>> that removing proxies would speed things up significantly. Am I > >>> missing something? Or is there a way to turn off proxies? And if > >>> not, how can I speed up the proxying operation? > >>> > >>> Thanks, > >>> Aron > >>> > >> > > -- *Rick Curtis*
RE: Speeding up commit
Hi Aron, Yes that is what I meant, I persist and commit 200 at a time. When my objects are highly complex I threw in em.clear() as I was getting stack issues as well. Chris -Original Message- From: Aron Lurie [mailto:a...@cambridgesemantics.com] Sent: Wednesday, 15 June 2011 1:06 AM To: users@openjpa.apache.org Subject: Re: Speeding up commit Chris, What do you mean by batching? Do you mean persisting 200, committing, persisting another 200, committing, etc? If so, I don't have the option of doing this because my objects reference one another so the cascading persist would blow up my batch size. Thanks, Aron On 6/14/2011 10:54 AM, C N Davies wrote: > I had a load of issues with committing large quantities of entities > (OpenJPA 2.0), in the end I batched them into batches of 200 and the > overall performance was significantly better. This is not a > scientific analysis of the issue I know, but might be of use to you :) > > Chris > > > -Original Message- > From: Aron Lurie [mailto:a...@cambridgesemantics.com] > Sent: Wednesday, 15 June 2011 12:50 AM > To: users@openjpa.apache.org > Subject: Re: Speeding up commit > > 2.1.0 > > On 6/14/2011 10:47 AM, Rick Curtis wrote: >> What version of OpenJPA are you running? >> >> On Tue, Jun 14, 2011 at 9:45 AM, Aron > Luriewrote: >>> Hello, >>> I have a situation where I am trying to persist ~2500 objects all at > once. >>> After the objects are persisted, the program ends, so there is no >>> need for any functionality after the objects have been persisted. I >>> am just trying to dump these objects to the DB as fast as possible. >>> >>> Using Eclipse's TPTP profiler, I have timed the >>> EntityManager.commit() operation to take 586 seconds. Of that, >>> cumulatively almost 455 seconds are spent inside of ~2500 calls to > StateManagerImpl.proxyFields(bool, bool). >>>From the point that I start commit(), those 2nd class objects are >>> not mutated by my program, and I do not need to use any of these >>> objects after >>> commit() finishes. From what I have read, it would seem that I have >>> no need for my objects to be proxied, and from my tests, it seems >>> that removing proxies would speed things up significantly. Am I >>> missing something? Or is there a way to turn off proxies? And if >>> not, how can I speed up the proxying operation? >>> >>> Thanks, >>> Aron >>> >>
Re: Speeding up commit
Chris, What do you mean by batching? Do you mean persisting 200, committing, persisting another 200, committing, etc? If so, I don't have the option of doing this because my objects reference one another so the cascading persist would blow up my batch size. Thanks, Aron On 6/14/2011 10:54 AM, C N Davies wrote: I had a load of issues with committing large quantities of entities (OpenJPA 2.0), in the end I batched them into batches of 200 and the overall performance was significantly better. This is not a scientific analysis of the issue I know, but might be of use to you :) Chris -Original Message- From: Aron Lurie [mailto:a...@cambridgesemantics.com] Sent: Wednesday, 15 June 2011 12:50 AM To: users@openjpa.apache.org Subject: Re: Speeding up commit 2.1.0 On 6/14/2011 10:47 AM, Rick Curtis wrote: What version of OpenJPA are you running? On Tue, Jun 14, 2011 at 9:45 AM, Aron Luriewrote: Hello, I have a situation where I am trying to persist ~2500 objects all at once. After the objects are persisted, the program ends, so there is no need for any functionality after the objects have been persisted. I am just trying to dump these objects to the DB as fast as possible. Using Eclipse's TPTP profiler, I have timed the EntityManager.commit() operation to take 586 seconds. Of that, cumulatively almost 455 seconds are spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool, bool). From the point that I start commit(), those 2nd class objects are not mutated by my program, and I do not need to use any of these objects after commit() finishes. From what I have read, it would seem that I have no need for my objects to be proxied, and from my tests, it seems that removing proxies would speed things up significantly. Am I missing something? Or is there a way to turn off proxies? And if not, how can I speed up the proxying operation? Thanks, Aron
RE: Speeding up commit
I had a load of issues with committing large quantities of entities (OpenJPA 2.0), in the end I batched them into batches of 200 and the overall performance was significantly better. This is not a scientific analysis of the issue I know, but might be of use to you :) Chris -Original Message- From: Aron Lurie [mailto:a...@cambridgesemantics.com] Sent: Wednesday, 15 June 2011 12:50 AM To: users@openjpa.apache.org Subject: Re: Speeding up commit 2.1.0 On 6/14/2011 10:47 AM, Rick Curtis wrote: > What version of OpenJPA are you running? > > On Tue, Jun 14, 2011 at 9:45 AM, Aron Luriewrote: > >> Hello, >> I have a situation where I am trying to persist ~2500 objects all at once. >> After the objects are persisted, the program ends, so there is no >> need for any functionality after the objects have been persisted. I >> am just trying to dump these objects to the DB as fast as possible. >> >> Using Eclipse's TPTP profiler, I have timed the >> EntityManager.commit() operation to take 586 seconds. Of that, >> cumulatively almost 455 seconds are spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool, bool). >> From the point that I start commit(), those 2nd class objects are >> not mutated by my program, and I do not need to use any of these >> objects after >> commit() finishes. From what I have read, it would seem that I have >> no need for my objects to be proxied, and from my tests, it seems >> that removing proxies would speed things up significantly. Am I >> missing something? Or is there a way to turn off proxies? And if not, >> how can I speed up the proxying operation? >> >> Thanks, >> Aron >> > >
Re: Speeding up commit
2.1.0 On 6/14/2011 10:47 AM, Rick Curtis wrote: What version of OpenJPA are you running? On Tue, Jun 14, 2011 at 9:45 AM, Aron Luriewrote: Hello, I have a situation where I am trying to persist ~2500 objects all at once. After the objects are persisted, the program ends, so there is no need for any functionality after the objects have been persisted. I am just trying to dump these objects to the DB as fast as possible. Using Eclipse's TPTP profiler, I have timed the EntityManager.commit() operation to take 586 seconds. Of that, cumulatively almost 455 seconds are spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool, bool). From the point that I start commit(), those 2nd class objects are not mutated by my program, and I do not need to use any of these objects after commit() finishes. From what I have read, it would seem that I have no need for my objects to be proxied, and from my tests, it seems that removing proxies would speed things up significantly. Am I missing something? Or is there a way to turn off proxies? And if not, how can I speed up the proxying operation? Thanks, Aron
Re: Speeding up commit
What version of OpenJPA are you running? On Tue, Jun 14, 2011 at 9:45 AM, Aron Lurie wrote: > Hello, > I have a situation where I am trying to persist ~2500 objects all at once. > After the objects are persisted, the program ends, so there is no need for > any functionality after the objects have been persisted. I am just trying to > dump these objects to the DB as fast as possible. > > Using Eclipse's TPTP profiler, I have timed the EntityManager.commit() > operation to take 586 seconds. Of that, cumulatively almost 455 seconds are > spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool, bool). > From the point that I start commit(), those 2nd class objects are not > mutated by my program, and I do not need to use any of these objects after > commit() finishes. From what I have read, it would seem that I have no need > for my objects to be proxied, and from my tests, it seems that removing > proxies would speed things up significantly. Am I missing something? Or is > there a way to turn off proxies? And if not, how can I speed up the proxying > operation? > > Thanks, > Aron > -- *Rick Curtis*