We have seen a few posts about using RowHandlers in the past few months.
Though I have not tried it with the optional groupBy attribute I can
definitively say that I have processed millions of row through a row handler
and my application never user more the 10mg of memory.
On Fri, Apr 11, 2008 at 11:57 AM, Harvey Kim <[EMAIL PROTECTED]> wrote:
> Is the app running for days/years or is the query running for
> days/years? If the query returns 24 million rows, I get out of error
> memory every single time. In theory, no query should really take days
> to return but we just happen to run into this case because our DBA
> forgot to put the index on the tables :-). After the indexing, the
> analyzing, paralleling, ... the query came back much much faster. But
> it also ran out of memory much much faster. I think it's the function
> of how many rows get processed on a single query - not really how long
> the apps been running. And I doubt there are many application out there
> that returns 24 millions rows on a single query where each object
> returned can be thousands of bytes. Or maybe there are... If there are,
> I'd love to hear if there were memory leak problems on their app.
>
> I'm just guessing here but internally, doesn't groupBy allocate HashMap
> to keep track of what it is grouping?
>
> On Fri, 11 Apr 2008 11:42:10 -0600, "Clinton Begin"
> <[EMAIL PROTECTED]> said:
> > I can't imagine why that would happen. If this was a bug, I would think
> > far
> > more people would be experiencing it. I have apps that have run for
> > months,
> > and in some cases years, without running out of memory. GroupBy doesn't
> > do
> > anything special really. It just skips duplicate rows for a parent
> > class.
> >
> > Try iBATIS 2.3.1. If there is a problem, it might have to do with
> > session/transaction management. 2.3.1 improves the handling of such.
> >
> > Clinton
> >
> > On Fri, Apr 11, 2008 at 10:10 AM, Harvey Kim <[EMAIL PROTECTED]>
> wrote:
> >
> > > If you use groupBy, you will eventually run out of memory. Few months
> > > ago, I e-mailed a sample program which showed my application doing
> > > absolute nothing except issuing a ibatis query. It was using
> > > queryWithRowHandler. Literally, all it did was call
> > > "queryWithRowHandler" and my rowHanlder did absolutely nothing.
> > > Literally this:
> > >
> > > public class IbatisDownloadThread implements Runnable, Serializable,
> > > RowHandler
> > > {
> > > private SqlMapClient sqlMapClient = null;
> > >
> > > public void run()
> > > {
> > > try
> > > {
> > > sqlMapClient = BaseDAO.getSqlMapClient();
> > > sqlMapClient.queryWithRowHandler("getIdList", queryDto, this);
> > > }
> > > }
> > >
> > > public void handleRow()
> > > {
> > > }
> > > }
> > >
> > > The above code eventually gets "Out of Memory" error and the query is
> > > using a groupBy. When I took out the "groupBy", it went MUCH MUCH
> MUCH
> > > longer. But even that ran out of memory after few hours of running.
> > > However, once I converted everything to a JDBC loop, "Out of Memory"
> > > error went away. When I examined the available memory at runtime, I
> > > could see the available memory decreasing with every call to
> > > "handleRow()". With JDBC, I examined the heap size inside the loop
> and
> > > it held steady for hours and finished without a problem.
> > >
> > > The moral of the story - you could probably get away with ibatis if
> you
> > > take out the groupBy (that was my quick fix). I took out the groupBy
> > > and simply issued another ibatis command to get the list inside
> > > "handleRow()". That was my quick fix and it is probably sufficient
> for
> > > now.
> > >
> > > You can still run out of memory if you use "queryWithRowHandler" but
> > > highly unlikely unless you have an application like mine which could
> > > have a thread running for days. But to be really safe, you have to go
> > > back to JDBC. Since I was on a tight deadline, I didn't bother going
> > > through the ibatis code but I may do that one of these days because it
> > > is a wonderful tool.
> > >
> > >
> > > On Tue, 8 Apr 2008 12:27:29 -0700 (PDT), "Carlos de Luna Saenz"
> > > <[EMAIL PROTECTED]> said:
> > > > It seems like there is not enough data in your mail...
> > > > what data base are you accessing? how are you dealing with it
> > > (DataSource
> > > > is JDBC or via the App Server) your class is keeping the data for
> any
> > > > reason (maybe in a session or application bean)? what's the App
> Server?
> > > > etc, etc... if i were you i should look all that kind of variables
> and
> > > > monitor with a profiler to see if there's somehint wrong with the
> query,
> > > > the server, the managed memory, or all of the above.
> > > > Greetings
> > > > Carlos de Luna
> > > >
> > > > ----- Mensaje original ----
> > > > De: Fernando Simonetti <[EMAIL PROTECTED]>
> > > > Para: "[email protected]" <[email protected]>
> > > > Enviado: martes, 8 de abril, 2008 7:51:26
> > > > Asunto: groupby x out of memory
> > > >
> > > > Eventually my aplication server is going down by "Out of Memory".
> > > > Analizing
> > > > de heap dumping, it's point for the same class allocating 600 Mb.
> > > >
> > > > This event can occur 3 hours or 3 days after I restart my server.
> This
> > > > class
> > > > is used thounsands times in the period. Monitoring the memory of the
> > > > server,
> > > > I could see 300MB to 700MB of memory been allocated in 3 minutes.
> It
> > > > hapens
> > > > eventualy but not in a determined period.
> > > >
> > > > The unique diference for this class for others, is the groupby
> clause in
> > > > Ibatis. First I thouth it could be a cartesian product, but I ran
> the
> > > > query
> > > > without parameters and found nothing.
> > > >
> > > > I going to implement this class without groupby clause, but first I
> > > would
> > > > like to see if there are other options.
> > > >
> > > >
> > > > thanks,
> > > > Fernando
> > > >
> > > >
> > > >
> > >
> ____________________________________________________________________________________
> > > > ¡Capacidad ilimitada de almacenamiento en tu correo!
> > > > No te preocupes más por el espacio de tu cuenta con Correo Yahoo!:
> > > > http://correo.espanol.yahoo.com/
> > > >
> > >
> > > --
> > > http://www.fastmail.fm - Accessible with your email software
> > > or over the web
> > >
> > >
>
> --
> http://www.fastmail.fm - mmm... Fastmail...
>
>
