1) I'm using tomcat, so no Java EE container. I'm using Spring to handle the
transactions.
2) I'm using one EMF for the whole app from app start, and one EM for the
transaction or sometimes 2, launched by a tomcat filter which opens it at the
start of the HTTP request and closes it at the end. And the two transactions I'm
talking about are 2 distinct user operations, so each gets one EM.
3) Build-time enhancement.
4) Actually no because in OpenJPA 1.2.2 there is a bug that throws NPEs and
means I can't run with it enabled. I just upgraded to 2.0.0 I think I've got the
same bug - albeit a different stack trace - but a quick google didn't show up
any immediately available work-around. I guess this is the key issue, right?
Here's my persistence xml
<persistence-unit name="OpenJpaJdbc">
<description>Pattern Repo JPA Config with OpenJPA</description>
<provider>org.apache.openjpa.persistence.PersistenceProviderImpl</provider>
<mapping-file>org/permacode/atomic/domain/entity/AtomicEntity.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/BrokerAccount.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Category.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/DollarReturn.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Exchange.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/ExchangeSymbol.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Fill.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Market.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/MarketSymbol.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/MarketSystem.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Order.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Pattern.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Portfolio.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/PortfolioItem.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/RequiredParam.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/TestAnalysis.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/TestRun.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Trade.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/TradingParam.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/TradingSystem.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/Weighting.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/WeightingGroup.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/MiniAnalysis.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/MiniResult.xml</mapping-file>
<mapping-file>org/permacode/patternrepo/orm/TestQueries.xml</mapping-file>
<properties>
<property name="openjpa.RuntimeUnenhancedClasses" value="unsupported" />
<property name="openjpa.Log" value="SQL=TRACE"/>
<property name="openjpa.ConnectionFactoryProperties"
value="PrettyPrint=true, PrettyPrintLineLength=72" />
</properties>
</persistence-unit>
Donald Woods on 18/06/10 13:36, wrote:
Some quick initial questions -
1) Are you using Java SE or a Java EE container (more of a transactions
and XA question)?
2) Are you using one EMF and EM for both the parent, child and
grandchild operations? Are you calling flush() anywhere?
3) Are you using build time or run time enhancement of entities?
4) Have you enabled the DataCache or QueryCache?
http://openjpa.apache.org/builds/2.0.0/apache-openjpa-2.0.0/docs/manual/manual.html#ref_guide_cache_query
Also, what do the entities and your persistence.xml look like?
-Donald
On 6/18/10 8:11 AM, Adam Hardy wrote:
I have a transaction that submits around 20K records, followed by
another transaction in the user process which inserts the same into a
related table.
The performance issues I talked about below initially caused processing
times of 45 mins, but I worked on the Java in the code and I tweaked the
mySQL database and reduced it to 15 mins.
This though is still a problem - I've been over the optimization
guidelines in the documentation and there's nothing there that I can
implement that I'm not already.
The first transaction I mentioned takes 5 mins, but the second takes
15mins and is inserting child records of the records created in the
first transaction. It looks like OpenJPA is fetching all of those first
records again. Shouldn't they already be in memory?
Thanks
Adam
Adam Hardy on 12/06/10 13:21, wrote:
I am trying to get a handle on what I should be able to achieve.
Can someone give me some idea of the metrics I should be able to get
optimistically when persisting an object that has a child list with
20,000 child objects and 20,000 grandchildren? (one-to-one child ->
grandchild)
Can I reasonably expect to get this done in under a minute?
I think that would work out at a rate of about 1.5 milliseconds per
object.
Thanks
Adam
Adam Hardy on 11/06/10 17:34, wrote:
I have performance problems with large lists of beans due to the base
class I am using for my entities.
This is slightly non-OpenJPA specific, so I hope nobody minds, but it
is Friday afternoon so I'm hoping you give me a bit of slack here.
The problem arises when I start building lists with over 10,000 items
on a parent class.
The trouble is in the base class for the entities, which is quite
clever (but obviously not clever enough) and it has non-facile
equals() and hashcode() algorithms which make use of reflection. It's
here that the slow-down comes.
When I link the child with a parent that already has 10,000 children,
the equals() method is called by ArrayList before the new child is
placed in the index.
As far as I can tell I have a couple of options.
(1) ditch the reflection-based equals method and hard-code an equals
method.
(2) don't use ArrayList but find a Collection-based class that uses
hashes or similar to identify items instead of equals. This is just
speculation - perhaps there is no such thing or it wouldn't help anyway:
- would a collection using hashes caches the hashes of the items
already indexed?
- would such a collection be persistable?
If anyone has been in this situation before, or has an idea about it
, I'd really appreciate the help.
