Try doing Legacy.objects.all().iterator(), this isn't technically the public API, but this grabs the results at a lower level, so it doesn't get cached.
On Nov 23, 5:44 pm, derek73 <[EMAIL PROTECTED]> wrote: > I have written a management script to migrate data from an existing > site into our new Django-powered site. The script runs fine with one > exception, memory consumption. A few of the tables I need to migrate > have around 250,000 records, and it seems that the Django ORM caches > each record in memory as it migrates each record. > > I have been able to work around it by only migrating about 50,000 > records at a time. By the time it reaches 50,000 records the python > process has consumed about 2 GB of RAM. If I let the process end, > python releases all that RAM, then I start a new management command > with the next set of 50,000. > > It seems there's probably a better way to manage this memory problem > but I'm a bit of a Python/Django newb. Any insight would be greatly > appreciated. I'm running the migration script on my MacBookPro with > 2GB of RAM and 30GB free HD at the moment. > > Here's an example of how I've set up my migration script. I cycle over > the ids cause when I tried to do "for row in Legacy.objects.all():" it > tried to swap out like 8GB of RAM then died. > > ids = [x[0] for x in Legacy.objects.order_by('time').values_list('id') > [0:50000]] > for row_id in ids: > row = Legacy.objects.get(id=row_id) > new = ShinyNewObj() > new.stuff = row.oldstuff > new.save() > > And here's the error I get when it runs out of memory: > > ----- > > --> Importing #172000 id:172000 date:'2008-11-06 10:19:49' > Python(4125) malloc: *** mmap(size=16777216) failed (error code=12) > *** error: can't allocate region > File "manage.py", line 27, in <module> > execute_from_command_line() > File "/Library/Python/2.5/site-packages/django/core/management/ > __init__.py", line 331, in execute_from_command_line > utility.execute() > File "/Library/Python/2.5/site-packages/django/core/management/ > __init__.py", line 295, in execute > self.fetch_command(subcommand).run_from_argv(self.argv) > File "/Library/Python/2.5/site-packages/django/core/management/ > base.py", line 77, in run_from_argv > self.execute(*args, **options.__dict__) > File "/Library/Python/2.5/site-packages/django/core/management/ > base.py", line 96, in execute > output = self.handle(*args, **options) > File "/Library/Python/2.5/site-packages/django/core/management/ > base.py", line 178, in handle > return self.handle_noargs(**options) > File "/Users/derek/projects/my_project/apps/legacy/management/ > commands/migrate_legacy.py", line 161, in handle_noargs > obj._save() > File "/Users/derek/projects/my_project/apps/ShinyNew/models.py", > line 286, in _save > super(ShinyNew, self).save(**kwargs) > File "/Library/Python/2.5/site-packages/django/db/models/base.py", > line 307, in save > self.save_base(force_insert=force_insert, > force_update=force_update) > File "/Library/Python/2.5/site-packages/django/db/models/base.py", > line 354, in save_base > manager.filter(pk=pk_val).extra(select={'a': 1}).values > ('a').order_by())): > File "/Library/Python/2.5/site-packages/django/db/models/query.py", > line 185, in __nonzero__ > iter(self).next() > File "/Library/Python/2.5/site-packages/django/db/models/query.py", > line 179, in _result_iter > self._fill_cache() > File "/Library/Python/2.5/site-packages/django/db/models/query.py", > line 612, in _fill_cache > self._result_cache.append(self._iter.next()) > File "/Library/Python/2.5/site-packages/django/db/models/query.py", > line 653, in iterator > for row in self.query.results_iter(): > File "/Library/Python/2.5/site-packages/django/db/models/sql/ > query.py", line 206, in results_iter > for rows in self.execute_sql(MULTI): > File "/Library/Python/2.5/site-packages/django/db/models/sql/ > query.py", line 1700, in execute_sql > cursor.execute(sql, params) > File "/Library/Python/2.5/site-packages/django/db/backends/util.py", > line 19, in execute > return self.cursor.execute(sql, params) > File "/Library/Python/2.5/site-packages/django/db/backends/mysql/ > base.py", line 83, in execute > return self.cursor.execute(query, args) > File "build/bdist.macosx-10.5-i386/egg/MySQLdb/cursors.py", line > 166, in execute > File "build/bdist.macosx-10.5-i386/egg/MySQLdb/connections.py", line > 35, in defaulterrorhandler > _mysql_exceptions.InterfaceError: (0, '') --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---