Am Tuesday 02 December 2008 schrieb Pratik:
> Hey Andreas,
>
> I just quickly went through your changes and it looks like a good
> start. A few things :
>
> - Could you please open a ticket at
> http://rails.lighthouseapp.com/projects/8994-ruby-on-rails/overview
> and upload a .diff file with the patch written for Rails ? Check
> http://rails.lighthouseapp.com/projects/8994-ruby-on-rails/overview
> for more help
> - Patch is missing tests
> - Those JDBC patches don't really belong to Rails :)
Well, it's just that people would wonder about the JDBC behavior. You risk
an OOM killed process if you work without the patches for large sets of
data.
If the adapter maintainers picked up the changes, the adapter would work out
of the box.
Reading the other comments about the incompleteness of the #find_each part,
I do wonder if I better focus on the first part for now. #find_each_by_sql
is pretty straight forward. Probably the other part can be improved, let me
see.
In general I see the main benefit in the impressively decreased memory need
when working with large result sets. It's still a single transaction which
is what I missed in the other plugins which work around the memory
limitation by multiple selects.
> Also, it'll really help if you could join the mailing list while we
> talk about your patch. Most of the people are likely to forget CCing
> you. But if you don't want to join our lil group, that's not a big
> deal :)
You're right. ;-)
I hope to find some more time over the Xmas holidays to put some more effort
into the current implementation. I certainly get back then.
> Thanks for working on this.
>
> On Sat, Nov 29, 2008 at 3:14 PM, Andreas Gungl <[EMAIL PROTECTED]> wrote:
> > Hello,
> >
> > For an internal project we were looking for a solution which extends
> > ActiveRecord by an iterator function. Using find(:all) is no fun if you
> > have to process some 10.000 records.
> >
> > Until now there have been two ways of dealing with that scenario:
> > - write your logic a second times (e.g. use stored procedure)
> > - bet on AR plugins which work around that limitation by fetching IDs
> > and processing them in groups, or by using :offset and :limit
> >
> > Rewriting logic is something we wanted to avoid, and the plugins don't
> > fully respect transactional contexts. So we started to implement our
> > own true iterator support for AR. Our project is on Oracle, but in the
> > meantime we have added support for MySQL. Probably other adapters can
> > be extended easily, too. We also tried JRuby 1.1.x, which is sometimes
> > faster than Ruby 1.8.6, but a patch is needed to bring the Java part of
> > the connection adapter into shape for a result set iteration.
> >
> > Okay, you're about to ask: how does it work. Here we go:
> >
> > MyTable.find_each_by_sql("some SQL here") { |my_table| ... }
> >
> > MyTable.find_each(:all, :conditions => ...,
> >
> > :include => ...) { |my_table| ... }
> >
> > Attached you find the magic code which can be used as a plugin for
> > Rails. When testing, please keep in mind that only Oracle and MySQL is
> > fully supported. JDBC will take lots of RAM for large result sets until
> > you have patched the JdbcConnectionAdapter.
> >
> > Some figures with JRuby:
> > I've tested the code for an export of ~80.000 customer data records.
> > Originally I couldn't run the export with heap space less than 2 GB
> > (JRuby 1.1.4 without extensive garbage collection). After having
> > patched the connection adapter, it works with less than 128 MB heap
> > space (JRuby 1.1).
> >
> >
> > I'd be happy if our idea would be picked up and AR would get these
> > iterator methods integrated. I've seen lots of people asking for
> > exactly this behavior. It's possible to implement, it's easy to
> > implement, and IMHO it doesn't break the AR metaphor.
> >
> > If you like the idea and want to send feedback, please CC me. I'm not
> > subscribed to the list.
> >
> > Regards,
> > Andreas
> > --
> > OTTO Software Partner GmbH, Freiberger Str. 35, 01067 Dresden, Germany
> > Andreas Gungl (e-mail: [EMAIL PROTECTED])
> > Tel. +49-351-49723-140 - Fax: +49-351-49723-119
> >
> > AG Dresden, HRB 2475
> > Geschäftsführer Burkhard Arrenberg, Jens Gruhl
> >
> >
--
OTTO Software Partner GmbH, Freiberger Str. 35, 01067 Dresden, Germany
Andreas Gungl (e-mail: [EMAIL PROTECTED])
Tel. +49-351-49723-140 - Fax: +49-351-49723-119
AG Dresden, HRB 2475
Geschäftsführer Burkhard Arrenberg, Jens Gruhl
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Ruby
on Rails: Core" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---
