No time to respond in detail
Break data into chunks (order by ID or something like that)
something like this (untested - set limit to something bigger than 1000 -
but this also stops you eating all the memory)
def output_from_sql(sql_file,target_file)
raise ArgumentError, "SQL file not found : #{sql_file}" unless
File.exists?(sql_file)
keys = data.first.keys
FasterCSV.open(target_file,'w') do |csv|
csv << keys
last_key = 0
while true
data =
ActiveRecord::Base.connection.select_all(File.open(sql_file).read,:order =>
'id',:limit => 2000, :conditions => ['id > ?',last_key])
data.each do |row|
csv << keys.map {|key| row[key]}
end
break if data.empty?
last_key = data.last.id
end
end
end
On Thu, Oct 2, 2008 at 11:11 AM, Mike Barton <[EMAIL PROTECTED]>wrote:
>
> Hi,
>
> I'm having trouble getting data out of my database through
> activerecord when the amount data returned by the sql is very large. I
> could use plain mysql at the command line, but would prefer to keep it
> in Ruby.
>
> This gist shows the helper I'm using at the moment, works for datasets
> in the tens of thousands, but not in the millions. The error message I
> get is that activerecord timed out.
>
> http://gist.github.com/14333
>
> Any suggestions would be great.
>
> Thanks
>
> Mike
> >
>
--
Thanks and regards,
Francis Fish
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"NWRUG" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/nwrug-members?hl=en
-~----------~----~----~----~------~----~------~--~---
