Hey Andreas, I just quickly went through your changes and it looks like a good start. A few things :
- Could you please open a ticket at http://rails.lighthouseapp.com/projects/8994-ruby-on-rails/overview and upload a .diff file with the patch written for Rails ? Check http://rails.lighthouseapp.com/projects/8994-ruby-on-rails/overview for more help - Patch is missing tests - Those JDBC patches don't really belong to Rails :) Also, it'll really help if you could join the mailing list while we talk about your patch. Most of the people are likely to forget CCing you. But if you don't want to join our lil group, that's not a big deal :) Thanks for working on this. On Sat, Nov 29, 2008 at 3:14 PM, Andreas Gungl <[EMAIL PROTECTED]> wrote: > Hello, > > For an internal project we were looking for a solution which extends > ActiveRecord by an iterator function. Using find(:all) is no fun if you > have to process some 10.000 records. > > Until now there have been two ways of dealing with that scenario: > - write your logic a second times (e.g. use stored procedure) > - bet on AR plugins which work around that limitation by fetching IDs and > processing them in groups, or by using :offset and :limit > > Rewriting logic is something we wanted to avoid, and the plugins don't fully > respect transactional contexts. So we started to implement our own true > iterator support for AR. Our project is on Oracle, but in the meantime we > have added support for MySQL. Probably other adapters can be extended > easily, too. We also tried JRuby 1.1.x, which is sometimes faster than Ruby > 1.8.6, but a patch is needed to bring the Java part of the connection > adapter into shape for a result set iteration. > > Okay, you're about to ask: how does it work. Here we go: > > MyTable.find_each_by_sql("some SQL here") { |my_table| ... } > > MyTable.find_each(:all, :conditions => ..., > :include => ...) { |my_table| ... } > > Attached you find the magic code which can be used as a plugin for Rails. > When testing, please keep in mind that only Oracle and MySQL is fully > supported. JDBC will take lots of RAM for large result sets until you have > patched the JdbcConnectionAdapter. > > Some figures with JRuby: > I've tested the code for an export of ~80.000 customer data records. > Originally I couldn't run the export with heap space less than 2 GB (JRuby > 1.1.4 without extensive garbage collection). After having patched the > connection adapter, it works with less than 128 MB heap space (JRuby 1.1). > > > I'd be happy if our idea would be picked up and AR would get these iterator > methods integrated. I've seen lots of people asking for exactly this > behavior. It's possible to implement, it's easy to implement, and IMHO it > doesn't break the AR metaphor. > > If you like the idea and want to send feedback, please CC me. I'm not > subscribed to the list. > > Regards, > Andreas > -- > OTTO Software Partner GmbH, Freiberger Str. 35, 01067 Dresden, Germany > Andreas Gungl (e-mail: [EMAIL PROTECTED]) > Tel. +49-351-49723-140 - Fax: +49-351-49723-119 > > AG Dresden, HRB 2475 > Geschäftsführer Burkhard Arrenberg, Jens Gruhl > > > > -- Cheers! - Pratik http://m.onkey.org --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
