Re: concurrent thread exception plus deadlock on delete statement using slice
Hi, that solves the deadlock and exception problems, but when I remove that line from my configuration, I still have that rollback issue, that is OpenJPA sends the delete statements to the database but issues a rollback only, no commit (see my example project with the ROLLBACK_persistence.xml configuration where the BrokerImpl property is commented out) Do you have any idea why this is so? Regards, Mike -Ursprüngliche Nachricht- Von: Pinaki Poddar [mailto:ppod...@apache.org] Gesendet: Montag, 13. Juni 2011 22:35 An: users@openjpa.apache.org Betreff: Re: concurrent thread exception plus deadlock on delete statement using slice Hi, Slice uses its own Broker (DistributedBrokerImpl) implementaion. Whereas your configuration tries to overwrite that with property name=openjpa.BrokerImpl value=non-finalizing/property For a quick turnaround, I would advise that you remove that from your configuration. If the problem persists still, I will examine it further. I will add this caveat/limitation in our documentation. Regards -- - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/concurrent-thread-exception-plus-deadlock-on-delete-statement-using-slice-tp6449143p6471718.html Sent from the OpenJPA Users mailing list archive at Nabble.com. ___ SMA Solar Technology AG Aufsichtsrat: Guenther Cramer (Vorsitzender) Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner Handelsregister: Amtsgericht Kassel HRB 3972 Sitz der Gesellschaft: 34266 Niestetal USt-ID-Nr. DE 113 08 59 54 WEEE-Reg.-Nr. DE 95881150 ___
Speeding up commit
Hello, I have a situation where I am trying to persist ~2500 objects all at once. After the objects are persisted, the program ends, so there is no need for any functionality after the objects have been persisted. I am just trying to dump these objects to the DB as fast as possible. Using Eclipse's TPTP profiler, I have timed the EntityManager.commit() operation to take 586 seconds. Of that, cumulatively almost 455 seconds are spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool, bool). From the point that I start commit(), those 2nd class objects are not mutated by my program, and I do not need to use any of these objects after commit() finishes. From what I have read, it would seem that I have no need for my objects to be proxied, and from my tests, it seems that removing proxies would speed things up significantly. Am I missing something? Or is there a way to turn off proxies? And if not, how can I speed up the proxying operation? Thanks, Aron
Re: Speeding up commit
What version of OpenJPA are you running? On Tue, Jun 14, 2011 at 9:45 AM, Aron Lurie a...@cambridgesemantics.comwrote: Hello, I have a situation where I am trying to persist ~2500 objects all at once. After the objects are persisted, the program ends, so there is no need for any functionality after the objects have been persisted. I am just trying to dump these objects to the DB as fast as possible. Using Eclipse's TPTP profiler, I have timed the EntityManager.commit() operation to take 586 seconds. Of that, cumulatively almost 455 seconds are spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool, bool). From the point that I start commit(), those 2nd class objects are not mutated by my program, and I do not need to use any of these objects after commit() finishes. From what I have read, it would seem that I have no need for my objects to be proxied, and from my tests, it seems that removing proxies would speed things up significantly. Am I missing something? Or is there a way to turn off proxies? And if not, how can I speed up the proxying operation? Thanks, Aron -- *Rick Curtis*
Re: Speeding up commit
2.1.0 On 6/14/2011 10:47 AM, Rick Curtis wrote: What version of OpenJPA are you running? On Tue, Jun 14, 2011 at 9:45 AM, Aron Luriea...@cambridgesemantics.comwrote: Hello, I have a situation where I am trying to persist ~2500 objects all at once. After the objects are persisted, the program ends, so there is no need for any functionality after the objects have been persisted. I am just trying to dump these objects to the DB as fast as possible. Using Eclipse's TPTP profiler, I have timed the EntityManager.commit() operation to take 586 seconds. Of that, cumulatively almost 455 seconds are spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool, bool). From the point that I start commit(), those 2nd class objects are not mutated by my program, and I do not need to use any of these objects after commit() finishes. From what I have read, it would seem that I have no need for my objects to be proxied, and from my tests, it seems that removing proxies would speed things up significantly. Am I missing something? Or is there a way to turn off proxies? And if not, how can I speed up the proxying operation? Thanks, Aron
RE: Speeding up commit
I had a load of issues with committing large quantities of entities (OpenJPA 2.0), in the end I batched them into batches of 200 and the overall performance was significantly better. This is not a scientific analysis of the issue I know, but might be of use to you :) Chris -Original Message- From: Aron Lurie [mailto:a...@cambridgesemantics.com] Sent: Wednesday, 15 June 2011 12:50 AM To: users@openjpa.apache.org Subject: Re: Speeding up commit 2.1.0 On 6/14/2011 10:47 AM, Rick Curtis wrote: What version of OpenJPA are you running? On Tue, Jun 14, 2011 at 9:45 AM, Aron Luriea...@cambridgesemantics.comwrote: Hello, I have a situation where I am trying to persist ~2500 objects all at once. After the objects are persisted, the program ends, so there is no need for any functionality after the objects have been persisted. I am just trying to dump these objects to the DB as fast as possible. Using Eclipse's TPTP profiler, I have timed the EntityManager.commit() operation to take 586 seconds. Of that, cumulatively almost 455 seconds are spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool, bool). From the point that I start commit(), those 2nd class objects are not mutated by my program, and I do not need to use any of these objects after commit() finishes. From what I have read, it would seem that I have no need for my objects to be proxied, and from my tests, it seems that removing proxies would speed things up significantly. Am I missing something? Or is there a way to turn off proxies? And if not, how can I speed up the proxying operation? Thanks, Aron
Re: Speeding up commit
Chris, What do you mean by batching? Do you mean persisting 200, committing, persisting another 200, committing, etc? If so, I don't have the option of doing this because my objects reference one another so the cascading persist would blow up my batch size. Thanks, Aron On 6/14/2011 10:54 AM, C N Davies wrote: I had a load of issues with committing large quantities of entities (OpenJPA 2.0), in the end I batched them into batches of 200 and the overall performance was significantly better. This is not a scientific analysis of the issue I know, but might be of use to you :) Chris -Original Message- From: Aron Lurie [mailto:a...@cambridgesemantics.com] Sent: Wednesday, 15 June 2011 12:50 AM To: users@openjpa.apache.org Subject: Re: Speeding up commit 2.1.0 On 6/14/2011 10:47 AM, Rick Curtis wrote: What version of OpenJPA are you running? On Tue, Jun 14, 2011 at 9:45 AM, Aron Luriea...@cambridgesemantics.comwrote: Hello, I have a situation where I am trying to persist ~2500 objects all at once. After the objects are persisted, the program ends, so there is no need for any functionality after the objects have been persisted. I am just trying to dump these objects to the DB as fast as possible. Using Eclipse's TPTP profiler, I have timed the EntityManager.commit() operation to take 586 seconds. Of that, cumulatively almost 455 seconds are spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool, bool). From the point that I start commit(), those 2nd class objects are not mutated by my program, and I do not need to use any of these objects after commit() finishes. From what I have read, it would seem that I have no need for my objects to be proxied, and from my tests, it seems that removing proxies would speed things up significantly. Am I missing something? Or is there a way to turn off proxies? And if not, how can I speed up the proxying operation? Thanks, Aron
RE: Speeding up commit
Hi Aron, Yes that is what I meant, I persist and commit 200 at a time. When my objects are highly complex I threw in em.clear() as I was getting stack issues as well. Chris -Original Message- From: Aron Lurie [mailto:a...@cambridgesemantics.com] Sent: Wednesday, 15 June 2011 1:06 AM To: users@openjpa.apache.org Subject: Re: Speeding up commit Chris, What do you mean by batching? Do you mean persisting 200, committing, persisting another 200, committing, etc? If so, I don't have the option of doing this because my objects reference one another so the cascading persist would blow up my batch size. Thanks, Aron On 6/14/2011 10:54 AM, C N Davies wrote: I had a load of issues with committing large quantities of entities (OpenJPA 2.0), in the end I batched them into batches of 200 and the overall performance was significantly better. This is not a scientific analysis of the issue I know, but might be of use to you :) Chris -Original Message- From: Aron Lurie [mailto:a...@cambridgesemantics.com] Sent: Wednesday, 15 June 2011 12:50 AM To: users@openjpa.apache.org Subject: Re: Speeding up commit 2.1.0 On 6/14/2011 10:47 AM, Rick Curtis wrote: What version of OpenJPA are you running? On Tue, Jun 14, 2011 at 9:45 AM, Aron Luriea...@cambridgesemantics.comwrote: Hello, I have a situation where I am trying to persist ~2500 objects all at once. After the objects are persisted, the program ends, so there is no need for any functionality after the objects have been persisted. I am just trying to dump these objects to the DB as fast as possible. Using Eclipse's TPTP profiler, I have timed the EntityManager.commit() operation to take 586 seconds. Of that, cumulatively almost 455 seconds are spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool, bool). From the point that I start commit(), those 2nd class objects are not mutated by my program, and I do not need to use any of these objects after commit() finishes. From what I have read, it would seem that I have no need for my objects to be proxied, and from my tests, it seems that removing proxies would speed things up significantly. Am I missing something? Or is there a way to turn off proxies? And if not, how can I speed up the proxying operation? Thanks, Aron
Re: Speeding up commit
Unfortunately it's looking like a this is a code path which we don't have an optimizations for Essentially you're using OpenJPA to put data into a DB and then once your tran commits you're not going to use the Entities any longer.? If you're looking for a workaround, you could noop org.apache.openjpa.kernel.SingleFieldManager.proxy(...) and just return true. Not ideal, but it might work for you in the short term. On Tue, Jun 14, 2011 at 10:35 AM, C N Davies c...@cndavies.com wrote: Hi Aron, Yes that is what I meant, I persist and commit 200 at a time. When my objects are highly complex I threw in em.clear() as I was getting stack issues as well. Chris -Original Message- From: Aron Lurie [mailto:a...@cambridgesemantics.com] Sent: Wednesday, 15 June 2011 1:06 AM To: users@openjpa.apache.org Subject: Re: Speeding up commit Chris, What do you mean by batching? Do you mean persisting 200, committing, persisting another 200, committing, etc? If so, I don't have the option of doing this because my objects reference one another so the cascading persist would blow up my batch size. Thanks, Aron On 6/14/2011 10:54 AM, C N Davies wrote: I had a load of issues with committing large quantities of entities (OpenJPA 2.0), in the end I batched them into batches of 200 and the overall performance was significantly better. This is not a scientific analysis of the issue I know, but might be of use to you :) Chris -Original Message- From: Aron Lurie [mailto:a...@cambridgesemantics.com] Sent: Wednesday, 15 June 2011 12:50 AM To: users@openjpa.apache.org Subject: Re: Speeding up commit 2.1.0 On 6/14/2011 10:47 AM, Rick Curtis wrote: What version of OpenJPA are you running? On Tue, Jun 14, 2011 at 9:45 AM, Aron Luriea...@cambridgesemantics.comwrote: Hello, I have a situation where I am trying to persist ~2500 objects all at once. After the objects are persisted, the program ends, so there is no need for any functionality after the objects have been persisted. I am just trying to dump these objects to the DB as fast as possible. Using Eclipse's TPTP profiler, I have timed the EntityManager.commit() operation to take 586 seconds. Of that, cumulatively almost 455 seconds are spent inside of ~2500 calls to StateManagerImpl.proxyFields(bool, bool). From the point that I start commit(), those 2nd class objects are not mutated by my program, and I do not need to use any of these objects after commit() finishes. From what I have read, it would seem that I have no need for my objects to be proxied, and from my tests, it seems that removing proxies would speed things up significantly. Am I missing something? Or is there a way to turn off proxies? And if not, how can I speed up the proxying operation? Thanks, Aron -- *Rick Curtis*
Re: Speeding up commit
Hello, For batch insertion, 1. Set openjpa.RetainState=false this might gain few extra cycles I had introduced a tentative option [1] for automatic detachment (your observation of cycles being spent on StateManager.proxyFields(...) is related to detachment process that OpenJPA runtime invokes at user-configurable points in the lifetime of a persistence context such as transaction endpoints or when EntityManager.close() is called). If the user application does not intend to refer to the managed entities later (as it seems to fits the use case you are describing), then we can bypass the overhead of detachment. In a quick Person-has-many-Address type model, I notice a ~20% reduction in data insert with this new NONE option. To access, and if possible verify efficacy of this new feature, please do the following 1. get a nightly build or build from trunk locally. 2. Ensure that revision 1135776 is included. Easy way to do this $ java -jar openjpa.jar will print the latest revision included in the jar 3. In user application, em.setProperty(openjpa.AutoDetach, AutoDetachType.NONE); Note that the option must be set on an instantiated EntityManager, and *not* in persistence.xml [1] https://issues.apache.org/jira/browse/OPENJPA-2017 - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6475817.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Is the problem when OpenJPA tries to proxy user supplied objects on transaction commit(flush)? Or is the problem when the persistence context detaches and OpenJPA spends cycles trying to remove the proxies that we just put in? Pinaki - Did you solve the both problems, or just the second? ... I thought the user Aron was reporting a problem with the fact that we ever put proxies in.? On Tue, Jun 14, 2011 at 3:32 PM, Pinaki Poddar ppod...@apache.org wrote: Hello, For batch insertion, 1. Set openjpa.RetainState=false this might gain few extra cycles I had introduced a tentative option [1] for automatic detachment (your observation of cycles being spent on StateManager.proxyFields(...) is related to detachment process that OpenJPA runtime invokes at user-configurable points in the lifetime of a persistence context such as transaction endpoints or when EntityManager.close() is called). If the user application does not intend to refer to the managed entities later (as it seems to fits the use case you are describing), then we can bypass the overhead of detachment. In a quick Person-has-many-Address type model, I notice a ~20% reduction in data insert with this new NONE option. To access, and if possible verify efficacy of this new feature, please do the following 1. get a nightly build or build from trunk locally. 2. Ensure that revision 1135776 is included. Easy way to do this $ java -jar openjpa.jar will print the latest revision included in the jar 3. In user application, em.setProperty(openjpa.AutoDetach, AutoDetachType.NONE); Note that the option must be set on an instantiated EntityManager, and *not* in persistence.xml [1] https://issues.apache.org/jira/browse/OPENJPA-2017 - Pinaki -- *Rick Curtis*
Re: Native SQL Query
If possible, try revision 1135806 from trunk. I have committed a draft version that would execute SQL in Slice. I have not tested it and I have concerns about sorting etc. If possible, let me know whether it gets you out of NPE. Also check for ORDER By or SUM() type support which perhaps will simply not work. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Native-SQL-Query-tp6432918p6476092.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
It will not proxy with for AutoDetachType.NONE. See StateManagerImpl commit log for further details. These stuff is highly untested because I do not have access to a regression test environment right now. I am still committing so that it gets tested with other commits. I will rollback if things just start breaking. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6476127.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Update statements not generated for a @OneToMany mapping
OpenJPA 1.2.2
Spring 3
HSQLDB 2.0
JUnit Testing with SpringJUnit4ClassRunner Spring @Transactional on
Testclass
--
class Settlement {
@OneToMany(mappedBy = settlement, Cascade=CascadeType.REFRESH)
private ListVoucherProcessed vouchersProcessed = new
ArrayListVoucherProcessed(0);
...
}
public class VoucherProcessed {
@OneToOne
@JoinColumn
private Voucher voucher;
@ManyToOne
@JoinColumn
private Settlement settlement;
...
}
The service class passes a new Settlement object along with a list of
vocherIds (since VoucherProcessed
are already persisted)
// dao
@Transactional
public void save(Settlement settlement, ListLong voucherIds) {
// Load VoucherProcessed
ListVoucherProcessed vouchersProcessed = (ListVoucherProcessed) em
.createQuery(SELECT vp FROM VoucherProcessed vp WHERE
vp.voucher.voucherId IN (:voucherIds))
.setParameter(voucherIds, voucherIds).getResultList();
if (vouchersProcessed != null) {
settlement.setVouchersProcessed(vouchersProcessed);
em.persist(settlement);
}
}
The above code generates INSERT for settlement but there's no update for
Voucher like:
UPDATE Voucher SET settlement.id=?
However, if I pass Settlement with a persistent VoucherList to the dao,
Update is generated.
What am I missing?
--
View this message in context:
http://openjpa.208410.n2.nabble.com/Update-statements-not-generated-for-a-OneToMany-mapping-tp6476296p6476296.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
It will not proxy with for AutoDetachType.NONE. That is confusing my simple mind Why do I need to set an AutoDetach flag to tell OpenJPA to not use proxies? These stuff is highly untested because I do not have access to a ... I'm not hung up on the right/wrong-ness of your change... I'm still thinking about it conceptually. On Tue, Jun 14, 2011 at 4:43 PM, Pinaki Poddar ppod...@apache.org wrote: It will not proxy with for AutoDetachType.NONE. See StateManagerImpl commit log for further details. These stuff is highly untested because I do not have access to a regression test environment right now. I am still committing so that it gets tested with other commits. I will rollback if things just start breaking. - Pinaki -- *Rick Curtis*
Re: Speeding up commit
Proxying/Deproxying stems from detachment. We proxy because we want to tract changes in detached objects. AutoDetach.NONE basically tells: 'do not bother detaching at all, not going to use these objects anyway ever'. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Speeding-up-commit-tp6474341p6476649.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Update statements not generated for a @OneToMany mapping (with persistent entity)
What am I missing? @OneToMany(mappedBy = settlement, Cascade=CascadeType.REFRESH) Correct cascade type. (also the capital case on Cascade looks a suspect) - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Update-statements-not-generated-for-a-OneToMany-mapping-with-persistent-entity-tp6476296p6476693.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Update statements not generated for a @OneToMany mapping (with persistent entity)
It's a typo in this post. code compiles just fine. Cascade type shouldn't matter because I'm not persisting Voucher here (it's already a persistent entity as it's reloaded in dao). As a test, CascadeType.All doesn't make any difference. -- View this message in context: http://openjpa.208410.n2.nabble.com/Update-statements-not-generated-for-a-OneToMany-mapping-with-persistent-entity-tp6476296p6476707.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Update statements not generated for a @OneToMany mapping (with persistent entity)
add cascade type MERGE, and call em.merge(settlement). Remember that merge returns a Settlement different than its input argument. Looks like a non-owning, owning side issue. Settlement is the non-owning side of the relation. Owning side is the one that has the Foreign Key for the relation in the database. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Update-statements-not-generated-for-a-OneToMany-mapping-with-persistent-entity-tp6476296p6476786.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Update statements not generated for a @OneToMany mapping (with persistent entity)
Also you should call voucher.setSettlement(s). JPA, by default, will not manage bi-directional referential integrity. The application has to do it. OpenJPA has a inverse manager -- you can activate that and most likely with the current code you will see an exception. But asking OpenJPA to check referential integrity of bi-directional relation is costly. - Pinaki -- View this message in context: http://openjpa.208410.n2.nabble.com/Update-statements-not-generated-for-a-OneToMany-mapping-with-persistent-entity-tp6476296p6476807.html Sent from the OpenJPA Users mailing list archive at Nabble.com.
Re: Speeding up commit
Aren't proxies also used to track changes while a persistence context is active? Rick Curtis
