Adrian Thanks for the response. In my case, I do not think that data consistancy is an issue because a single thread is reading the incoming data.
The worst of these is a price change operation where a wholesaler sends this customer a list of all products and their current selling price (as well as other info). This application scans through this list and does lots of stuff to it like change price, add products if they do not exist, expire ones that do and are no longer supplied, etc. The only way I can see to do this "breaking up the set into smaller chunks would be to" Transaction.begin iter = findListIteratorByCondition(..) List thisBatch = iter.getPartialList(..) iter.close() Transaction end Rinse and repeat Each time, you are reading over from the beginning of the file. With typically 1.4 million records to process for one of these price change operations, I would think that this would add a considerable time to the total process. It currently takes 7 hours to run on a spiffy 32 gig machine. Is there an alternate way to avoid reading from the beginning of the file for each set? Skip -----Original Message----- From: Adrian Crum [mailto:adrian.c...@sandglass-software.com] Sent: Saturday, June 15, 2013 1:09 AM To: user@ofbiz.apache.org Subject: Re: findListIteratorByCondition in 12.04 Transactions are recommended for queries so you can ensure consistent data. Put your large processing jobs inside a loop that processes smaller sets of data - like 100 records or so. -Adrian On 6/15/2013 1:21 AM, Skip wrote: > I have used findListIteratorByCondition successfully in the past to process > large files with no transaction in place. > > By putting a TransactionUtil.begin/end around the > findListIteratorByCondition and a second TransactionUtil.begin/end around > the actual writing of the data. > > This all worked just fine. Now, with 12.04 I get a (ERROR: portal "C_xxx" > does not exist) after processing about 500 or so records (not sure of the > actual number) on the iterator.next() call. > > If I remove the begin/end around the findListIteratorByCondition, everything > runs fine. The only problem is that I get an exception report (no actual > exception) from GenericDelegator that sez (ERROR: Cannot do a find that > returns an EntityListIterator with no transaction in place.) > > This is not a thrown exception, only one that is displayed in the log. > > So, I have two questions. > > 1. Why is it dangerous enough to not have a transaction in place to print > this exception in the log but not throw an actual exception. > > 2. Given my need above (where I might spend an hour processing huge amounts > of data), what is the recommended way to use findListIteratorByCondition and > only have a transaction around the actual commits? > > Thanks in Advance > > Skip > >
