Interesting - I don't think I used a groupBy with my large volume
RowHandlers.
If you're writing to the output stream from the RowHandler and the user
Cancels the download, wouldn't the stream get closed and cause an exception
the next time you tried to write?
It seems like you're going through a lot of ugly work to handle the 'Cancel'
action - is it absolutely necessary? Worst case is you have a Thread
spinning through the millions of rows for no reason because the user said
'No' halfway through the operation. How often is that expected to happen?
How many users do you have that are expected to do the download? How loaded
up is your application that having a Thread finish the processing would be
noticeable and cause a problem?
Maybe you could have something like this:
private boolean noop = false;
public void handleRow (Object valueObject)
throws SQLException {
if (noop) {
// if first time to hit this, do any cleanup here
return;
}
// Rest of your code
}
public void setNoop(boolean b) {
this.noop = b;
}
When you detect the Cancel operation you call 'setNoop(true)' on the
RowHandler and it just spins through the rest of the records. That Thread
would then be busy running through the rest of the results, but might be ok
since no user is waiting on it.
Generally I tend to think that any time you find yourself messing around
with the connections iBATIS is using, you're doing something wrong.
On 1/30/08 10:18 AM, "Harvey Kim" <[EMAIL PROTECTED]> wrote:
> Thanks - that's good to know. That gives me a warm fuzzy feeling.
>
> I guess the issue now is to figure out what type of query in ibatis is
> causing the memory leak. And there is absolutely no doubt that the
> culprit is ibatis. JDBC with the same query left no residue of memory
> leaks. And rearranging the query by eliminating the groupBy clause in
> the ibatis xml file eliminated 90% of the memory leak. Also, straight
> forward query from one table that returns millions of rows from ibatis
> yielded NO memory leaks. So there's something about joins but more
> significantly - group by clause. My guess is that ibatis must be using
> some sort of a Map to implement the groupBy clause and is simply letting
> it grow indefinitely.
>
> One more bit of information. Just throwing an Exception alone in the
> "handleRow" will not get you out of the "loop". I had to issue
> "sqlMap.endConnection()" to force the calling routine within ibatis to
> throw a NullPtrException in order to exit. If anybody has any other
> ideas, I'd appreciate it. It does work but it's very ugly.
>
> On Tue, 29 Jan 2008 20:24:13 -0700, "Chris Lamey"
> <[EMAIL PROTECTED]> said:
>> I have processed millions of rows with a rowhandler and no memory leaks
>> in several different applications.
>>
>>
>> -----Original Message-----
>> From: Harvey Kim [mailto:[EMAIL PROTECTED]
>> Sent: Tue 1/29/2008 5:23 PM
>> To: ibatis user
>> Subject: Re: Exit out of rowHandler
>>
>> Thanks, I'm currently throwing an exception but that is really ugly. I
>> was hoping for more "graceful" solution. But I guess I can live with
>> it. Oh, and "Cancel" button won't be a problem - I got that covered.
>>
>> Another thing - I've issued this question few months back but there ws
>> no reply. Does anybody know anything about memory leak problem when
>> using the RowHandler? After processing about 200,000 rows or so, I
>> noticed that "available memory" starts to decrease. This is more
>> prevalent when using a "groupBy" clause. When I took out the groupBy
>> clause, memory leak appeared to go have gone away but it slowly started
>> appearing at 200,000th row (give or take few thousands).
>>
>> Just for kicks, I converted everything to JDBC with exactly the same
>> code and it successfully processed 1 million rows with absolutely no
>> memory leaks. Meaning, instead of calling ibatis with RH, I simply put
>> it in a loop after issuing the same SQL generated by ibatis via JDBC.
>> Meaning, my code was exactly the same except instead of being inside the
>> "handleRow", it was inside the loop. With this evidence, I became
>> suspicious of the ibatis RH.
>>
>> I guess what I'm asking is if anybody ever successfully processed
>> millions of row using RH without a memory leak . I'd hate to resort
>> back to JDBC when ibatis makes everything so much easier.
>>
>> I'd appreciate any input,
>>
>> thanks,
>>
>> -Harvey
>>
>>
>> On Tue, 29 Jan 2008 16:52:02 -0700, "Larry Meadors"
>> <[EMAIL PROTECTED]> said:
>>> On Jan 29, 2008 4:34 PM, Harvey Kim <[EMAIL PROTECTED]> wrote:
>>>> Anybody know how to gracefully exit out of "handleRow" method? Before
>>>> anybody suggests using "queryFor*" methods, I need to be able to query
>>>> for potentially 12 million rows. It's a straight download to an excel
>>>> file so nothing is being displayed (thank god). Anyway, I'm trying to
>>>> exit out of "handleRow" method when the user clicks on the "cancel"
>>>> button from the standard download dialogue box.
>>>
>>> Hm, red flag there - I assume you're talking web app, but I don't
>>> think the request is canceled when the user clicks on the cancel
>>> button, it'll still going to run.
>>>
>>> In any case, the only current way to cancel RH processing is to throw
>>> an exception from the handleRow method. Not sexy, but it works. :-)
>>>
>>> Larry
>>
>> --
>> http://www.fastmail.fm - Email service worth paying for. Try it for free
>>
>>
