Another possible solution would be this: Upload your data in CSV/TSV/whatever form to S3. Write a rake task that does the following:
* download from S3 to RAILS_ROOT/tmp * use the psql command line tool (it's on our dyno grid) or one of the ActiveRecord bulk import extensions to read the file and import to your database Then you can run it with "heroku rake my_import_task" If this is going to be a regular process, you'll likely want to wrap all of this up as something you can run from a worker using DJ or its' ilk. On Dec 10, 2010, at 10:51 AM, Zach Bailey wrote: > > Thanks John, that's a great suggestion. Unfortunately it's looking like it > will take about 7.5 hours to import 3.12M rows: > > 1 tables, 3,123,800 records > companies: 1% | | ETA: > 07:25:34 > > I'm wondering if there's a more expedient route... in the past I've used the > postgres COPY command [1] to do bulk imports of large data sets quickly, but > that requires that the server be able to read a file off the server's local > filesystem. I don't suppose that's feasible given how the Heroku platform > works, but would love to be pleasantly surprised :) > > Anyone from Heroku able to pipe up and offer any other possible suggestions? > Just to restate the problem, I have a single table with about 3.12M records > that I'm wanting to transfer from a local DB to my remote Heroku DB without > touching the other Heroku app data. It's ok if the table gets blown away on > the Heroku side as it has nothing in it (new model I just added). > > Happy Friday, > Zach > > [1] http://www.postgresql.org/docs/8.4/interactive/sql-copy.html > > On Thursday, December 9, 2010 at 4:36 AM, johnb wrote: > >> If it's just a single table and you have it in a db locally then db:push >> --tables <tablename> would get it up to heroku - but this will replace the >> contents of the remote table with the local table and not append to it. If >> the application is live you could put it into maintenance mode, db:pull >> --tables <tablename> append your rows to it and then push the table back and >> put the app live... >> >> perhaps? >> >> John. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Heroku" group. >> To post to this group, send email to her...@googlegroups.com. >> To unsubscribe from this group, send email to >> heroku+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/heroku?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
