The one problem I see with this approach is that it does not allow for
the eager-loading of polymorphic associations. In fact, as the code
stands now I couldn't get vanilla :include options to work,
only :joins (but I think that's just a little bug in the parsing of
the options hash).
The problem is, when Rails uses the eager include approach rather than
the join approach it looks through the entire result set to see which
associations need to be queried and then grabs those associations in
separate queries. This is relatively efficient because the object
graph is being built out for all objects at once, my guess is that
doing the same thing one object at a time would be very difficult to
do efficiently.
As an alternative approach, the project I work on uses a home-grown
version of what Andreas mentions as one of the two options -- we fetch
IDs and process them in groups. The one thing I really like about that
approach is all the same finder options still work as expected with no
additional programming required. IMO losing eager includes of
polymorphic associations is a non-starter.
just my 2 cents :)
-ac
On Nov 29, 6:14 am, Andreas Gungl <[EMAIL PROTECTED]> wrote:
> Hello,
>
> For an internal project we were looking for a solution which extends
> ActiveRecord by an iterator function. Using find(:all) is no fun if you
> have to process some 10.000 records.
>
> Until now there have been two ways of dealing with that scenario:
> - write your logic a second times (e.g. use stored procedure)
> - bet on AR plugins which work around that limitation by fetching IDs and
> processing them in groups, or by using :offset and :limit
>
> Rewriting logic is something we wanted to avoid, and the plugins don't fully
> respect transactional contexts. So we started to implement our own true
> iterator support for AR. Our project is on Oracle, but in the meantime we
> have added support for MySQL. Probably other adapters can be extended
> easily, too. We also tried JRuby 1.1.x, which is sometimes faster than Ruby
> 1.8.6, but a patch is needed to bring the Java part of the connection
> adapter into shape for a result set iteration.
>
> Okay, you're about to ask: how does it work. Here we go:
>
> MyTable.find_each_by_sql("some SQL here") { |my_table| ... }
>
> MyTable.find_each(:all, :conditions => ...,
> :include => ...) { |my_table| ... }
>
> Attached you find the magic code which can be used as a plugin for Rails.
> When testing, please keep in mind that only Oracle and MySQL is fully
> supported. JDBC will take lots of RAM for large result sets until you have
> patched the JdbcConnectionAdapter.
>
> Some figures with JRuby:
> I've tested the code for an export of ~80.000 customer data records.
> Originally I couldn't run the export with heap space less than 2 GB (JRuby
> 1.1.4 without extensive garbage collection). After having patched the
> connection adapter, it works with less than 128 MB heap space (JRuby 1.1).
>
> I'd be happy if our idea would be picked up and AR would get these iterator
> methods integrated. I've seen lots of people asking for exactly this
> behavior. It's possible to implement, it's easy to implement, and IMHO it
> doesn't break the AR metaphor.
>
> If you like the idea and want to send feedback, please CC me. I'm not
> subscribed to the list.
>
> Regards,
> Andreas
> --
> OTTO Software Partner GmbH, Freiberger Str. 35, 01067 Dresden, Germany
> Andreas Gungl (e-mail: [EMAIL PROTECTED])
> Tel. +49-351-49723-140 - Fax: +49-351-49723-119
>
> AG Dresden, HRB 2475
> Geschäftsführer Burkhard Arrenberg, Jens Gruhl
>
> active_record_iterator.tgz
> 9KViewDownload
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Ruby
on Rails: Core" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---
