RE: T5 and hibernate slowing with more records
Hello, Thank you for everyone's input. I had originally taken out the sub queries from the main method and moved them to another method(IE check if city is there if not add it and link the foreign key) and then moved it to a map held in memory(still putting unknown ones in the db) so there were less calls to the db. This didn't help but then I moved back to the @CommitAfter(which was the original way I had started) and it worked fine. I like your clear idea and creating my own transactions. I think I'll give that a go as well. --James -Original Message- From: Jonathan Barker [mailto:jonathan.theit...@gmail.com] Sent: January-16-09 12:31 AM To: 'Tapestry users' Subject: RE: T5 and hibernate slowing with more records Been there, hit that. You need to clear() the session. Or, you can also evict() as you go. Basically, every time Hibernate needs to commit something, it needs to flush to the database, and it looks at all of the Hibernate objects that it knows about - not just those to which you have current references. For every object you handle, you add one more that it needs to check to see if it needs to have changes flushed. The clear() gets rid of anything you are not holding a reference to. I tend to do it every ten records or so, but I've never tried to optimize it. The other option, as someone suggested, is to commit a batch rather than single records, but that can be nasty for large sets. I'm working with an import of about 250,000 records as I write this. Jonathan > -Original Message- > From: James Sherwood [mailto:jsherw...@rgisolutions.com] > Sent: Thursday, January 15, 2009 13:16 > To: 'Tapestry users' > Subject: T5 and hibernate slowing with more records > > Hello, > > > > I am using POI to take an excel file and put it into a Mysql database > using > hibernate. > > > > I put a counter on the records going into the database and it starts off > fast but then slows to over 15 seconds a record and I cannot figure out > why. > > > > My code is basically this: > > > > Public class DatabaseDump{ > > > > @Inject > > Private Session _session; > > > > Public void onAction(){ //they hit the go button > > Loop rows{ > > > _session.beginTransaction(); > > Loop cells{ > > Fill the > object from the cells > > > _session.save(Object); > > } > > > _session.getTransaction().commit(); > > } > > } > > > > Everything works fine as in the database gets filled for the rows it has > completed but gradually slows to a crawl and at around 2700 records of > 10500 > records it's a record every 15 seconds. > > > > I run the project under eclipse using Tomcat. The memory starts at around > 80mb and only goes up to around 110mb. > > > > What am I doing wrong? > > > > --James > > > > - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org
RE: T5 and hibernate slowing with more records
Been there, hit that. You need to clear() the session. Or, you can also evict() as you go. Basically, every time Hibernate needs to commit something, it needs to flush to the database, and it looks at all of the Hibernate objects that it knows about - not just those to which you have current references. For every object you handle, you add one more that it needs to check to see if it needs to have changes flushed. The clear() gets rid of anything you are not holding a reference to. I tend to do it every ten records or so, but I've never tried to optimize it. The other option, as someone suggested, is to commit a batch rather than single records, but that can be nasty for large sets. I'm working with an import of about 250,000 records as I write this. Jonathan > -Original Message- > From: James Sherwood [mailto:jsherw...@rgisolutions.com] > Sent: Thursday, January 15, 2009 13:16 > To: 'Tapestry users' > Subject: T5 and hibernate slowing with more records > > Hello, > > > > I am using POI to take an excel file and put it into a Mysql database > using > hibernate. > > > > I put a counter on the records going into the database and it starts off > fast but then slows to over 15 seconds a record and I cannot figure out > why. > > > > My code is basically this: > > > > Public class DatabaseDump{ > > > > @Inject > > Private Session _session; > > > > Public void onAction(){ //they hit the go button > > Loop rows{ > > > _session.beginTransaction(); > > Loop cells{ > > Fill the > object from the cells > > > _session.save(Object); > > } > > > _session.getTransaction().commit(); > > } > > } > > > > Everything works fine as in the database gets filled for the rows it has > completed but gradually slows to a crawl and at around 2700 records of > 10500 > records it's a record every 15 seconds. > > > > I run the project under eclipse using Tomcat. The memory starts at around > 80mb and only goes up to around 110mb. > > > > What am I doing wrong? > > > > --James > > > > - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org
RE: Re: T5 and hibernate slowing with more records
a quick way is: instead of creating a service, make something like this @CommitAfter private void saveThem(List lst) { // put code here to save objects, don't beging a transaction as it has been started by Tapestry-Hibernate } how long? James Sherwood wrote: > > Hello, > > It took 13 seconds to create the List with 14000 objects. > > At 1 hour 45 minutes I was at 8600 records committing each one. > > For the service, I haven't went that far in T5 yet so I will have to look > it > up. > > If there was a way to save the whole list at once using the session that > might be faster as well. > > --James > > -Original Message- > From: Angelo Chen [mailto:angelochen...@yahoo.com.hk] > Sent: January-15-09 9:33 PM > To: users@tapestry.apache.org > Subject: [Spam Rating:3.18] Re: T5 and hibernate slowing with more records > > > Hi James, > > I do not see any reason why it is slow, but would suggest to do following: > > 1) try to put everything into a list of objects without saving to the > database, slow? > 2) use a service to accept the list from #1, beging a transaction, save, > commit, slow? > > > Angelo > > > James Sherwood wrote: >> >> Hello, >> >> >> >> I am using POI to take an excel file and put it into a Mysql database >> using >> hibernate. >> >> >> >> I put a counter on the records going into the database and it starts off >> fast but then slows to over 15 seconds a record and I cannot figure out >> why. >> >> >> >> My code is basically this: >> >> >> >> Public class DatabaseDump{ >> >> >> >> @Inject >> >> Private Session _session; >> >> >> >> Public void onAction(){ //they hit the go button >> >> Loop rows{ >> >> >> _session.beginTransaction(); >> >> Loop cells{ >> >> Fill the >> object from the cells >> >> >> _session.save(Object); >> >> } >> >> >> _session.getTransaction().commit(); >> >> } >> >> } >> >> >> >> Everything works fine as in the database gets filled for the rows it has >> completed but gradually slows to a crawl and at around 2700 records of >> 10500 >> records it's a record every 15 seconds. >> >> >> >> I run the project under eclipse using Tomcat. The memory starts at >> around >> 80mb and only goes up to around 110mb. >> >> >> >> What am I doing wrong? >> >> >> >> --James >> >> >> >> >> >> >> > > -- > View this message in context: > http://www.nabble.com/T5-and-hibernate-slowing-with-more-records-tp21483684p > 21490838.html > Sent from the Tapestry - User mailing list archive at Nabble.com. > > > - > To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org > For additional commands, e-mail: users-h...@tapestry.apache.org > > > - > To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org > For additional commands, e-mail: users-h...@tapestry.apache.org > > > -- View this message in context: http://www.nabble.com/T5-and-hibernate-slowing-with-more-records-tp21483684p21491243.html Sent from the Tapestry - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org
Re: T5 and hibernate slowing with more records
Can you avoid committing the transaction on every item? Committing takes more database effort than the initial update/insert, so batching these within one transaction is highly advisable if it's possible. Christian. On 15-Jan-09, at 21:02 , James Sherwood wrote: Hello, It took 13 seconds to create the List with 14000 objects. At 1 hour 45 minutes I was at 8600 records committing each one. For the service, I haven't went that far in T5 yet so I will have to look it up. If there was a way to save the whole list at once using the session that might be faster as well. --James -Original Message- From: Angelo Chen [mailto:angelochen...@yahoo.com.hk] Sent: January-15-09 9:33 PM To: users@tapestry.apache.org Subject: [Spam Rating:3.18] Re: T5 and hibernate slowing with more records Hi James, I do not see any reason why it is slow, but would suggest to do following: 1) try to put everything into a list of objects without saving to the database, slow? 2) use a service to accept the list from #1, beging a transaction, save, commit, slow? Angelo James Sherwood wrote: Hello, I am using POI to take an excel file and put it into a Mysql database using hibernate. I put a counter on the records going into the database and it starts off fast but then slows to over 15 seconds a record and I cannot figure out why. My code is basically this: Public class DatabaseDump{ @Inject Private Session _session; Public void onAction(){ //they hit the go button Loop rows{ _session.beginTransaction(); Loop cells{ Fill the object from the cells _session.save(Object); } _session.getTransaction().commit(); } } Everything works fine as in the database gets filled for the rows it has completed but gradually slows to a crawl and at around 2700 records of 10500 records it's a record every 15 seconds. I run the project under eclipse using Tomcat. The memory starts at around 80mb and only goes up to around 110mb. What am I doing wrong? --James -- View this message in context: http://www.nabble.com/T5-and-hibernate-slowing-with-more-records-tp21483684p 21490838.html Sent from the Tapestry - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org Christian Edward Gruber christianedwardgru...@gmail.com - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org
RE: Re: T5 and hibernate slowing with more records
Hello, It took 13 seconds to create the List with 14000 objects. At 1 hour 45 minutes I was at 8600 records committing each one. For the service, I haven't went that far in T5 yet so I will have to look it up. If there was a way to save the whole list at once using the session that might be faster as well. --James -Original Message- From: Angelo Chen [mailto:angelochen...@yahoo.com.hk] Sent: January-15-09 9:33 PM To: users@tapestry.apache.org Subject: [Spam Rating:3.18] Re: T5 and hibernate slowing with more records Hi James, I do not see any reason why it is slow, but would suggest to do following: 1) try to put everything into a list of objects without saving to the database, slow? 2) use a service to accept the list from #1, beging a transaction, save, commit, slow? Angelo James Sherwood wrote: > > Hello, > > > > I am using POI to take an excel file and put it into a Mysql database > using > hibernate. > > > > I put a counter on the records going into the database and it starts off > fast but then slows to over 15 seconds a record and I cannot figure out > why. > > > > My code is basically this: > > > > Public class DatabaseDump{ > > > > @Inject > > Private Session _session; > > > > Public void onAction(){ //they hit the go button > > Loop rows{ > > > _session.beginTransaction(); > > Loop cells{ > > Fill the > object from the cells > > > _session.save(Object); > > } > > > _session.getTransaction().commit(); > > } > > } > > > > Everything works fine as in the database gets filled for the rows it has > completed but gradually slows to a crawl and at around 2700 records of > 10500 > records it's a record every 15 seconds. > > > > I run the project under eclipse using Tomcat. The memory starts at around > 80mb and only goes up to around 110mb. > > > > What am I doing wrong? > > > > --James > > > > > > > -- View this message in context: http://www.nabble.com/T5-and-hibernate-slowing-with-more-records-tp21483684p 21490838.html Sent from the Tapestry - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org
Re: T5 and hibernate slowing with more records
Hi James, I do not see any reason why it is slow, but would suggest to do following: 1) try to put everything into a list of objects without saving to the database, slow? 2) use a service to accept the list from #1, beging a transaction, save, commit, slow? Angelo James Sherwood wrote: > > Hello, > > > > I am using POI to take an excel file and put it into a Mysql database > using > hibernate. > > > > I put a counter on the records going into the database and it starts off > fast but then slows to over 15 seconds a record and I cannot figure out > why. > > > > My code is basically this: > > > > Public class DatabaseDump{ > > > > @Inject > > Private Session _session; > > > > Public void onAction(){ //they hit the go button > > Loop rows{ > > > _session.beginTransaction(); > > Loop cells{ > > Fill the > object from the cells > > > _session.save(Object); > > } > > > _session.getTransaction().commit(); > > } > > } > > > > Everything works fine as in the database gets filled for the rows it has > completed but gradually slows to a crawl and at around 2700 records of > 10500 > records it's a record every 15 seconds. > > > > I run the project under eclipse using Tomcat. The memory starts at around > 80mb and only goes up to around 110mb. > > > > What am I doing wrong? > > > > --James > > > > > > > -- View this message in context: http://www.nabble.com/T5-and-hibernate-slowing-with-more-records-tp21483684p21490838.html Sent from the Tapestry - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org
RE: T5 and hibernate slowing with more records
Hello, Sorry I forgot to mention that I do not believe indexing is an issue as it is only an 'over time' issue that it gets slow not a total number in the db that causes the slowness. I can have 0 records or 40k records in the db and the transactions are still blazing fast at the start. --James -Original Message- From: Szemere Szemere [mailto:szemereszem...@googlemail.com] Sent: January-15-09 6:30 PM To: Tapestry users Subject: Re: T5 and hibernate slowing with more records Feels like a Hibernate question, not Tapestry. Does your table have the right index, do you need to have 10k transactions in a single web-request? Szemere - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org
RE: T5 and hibernate slowing with more records
Hello, I suppose I could restrict the number of records allowed however I do not believe that I need to. I believe there is a problem somewhere, maybe in how I am doing the transactions or something. Before using t5/hibernate we were using t3/torque and if that system would have no problem with this I cannot see the newer technology having any problem. I guess I will ask the hibernate forums but I thought it would have something to do with the way I use the session/transactions. --James -Original Message- From: Szemere Szemere [mailto:szemereszem...@googlemail.com] Sent: January-15-09 6:30 PM To: Tapestry users Subject: Re: T5 and hibernate slowing with more records Feels like a Hibernate question, not Tapestry. Does your table have the right index, do you need to have 10k transactions in a single web-request? Szemere - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org
Re: T5 and hibernate slowing with more records
Feels like a Hibernate question, not Tapestry. Does your table have the right index, do you need to have 10k transactions in a single web-request? Szemere
T5 and hibernate slowing with more records
Hello, I am using POI to take an excel file and put it into a Mysql database using hibernate. I put a counter on the records going into the database and it starts off fast but then slows to over 15 seconds a record and I cannot figure out why. My code is basically this: Public class DatabaseDump{ @Inject Private Session _session; Public void onAction(){ //they hit the go button Loop rows{ _session.beginTransaction(); Loop cells{ Fill the object from the cells _session.save(Object); } _session.getTransaction().commit(); } } Everything works fine as in the database gets filled for the rows it has completed but gradually slows to a crawl and at around 2700 records of 10500 records it's a record every 15 seconds. I run the project under eclipse using Tomcat. The memory starts at around 80mb and only goes up to around 110mb. What am I doing wrong? --James