Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  closed
Component:  Core |  Version:  master
  (Serialization)|   Resolution:  fixed
 Severity:  Normal   | Triage Stage:  Ready for
 Keywords:  dumpdata fixtures|  checkin
  memory |  Needs documentation:  0
Has patch:  1|  Patch needs improvement:  0
  Needs tests:  0|UI/UX:  0
Easy pickings:  0|
-+-

Comment (by claudep):

 Replying to [comment:61 barry.melton@…]:
 > With these changes, serialized objects now seem to start with "[ ,",
 which is wrong (including a comma, even on the first result.)

 Looking at the test cases, I don't think your assertion is True. However,
 it is possible your use case is not covered by test cases. So please
 provide us more details so as we can try to reproduce the faulty JSON
 result.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  closed
Component:  Core |  Version:  master
  (Serialization)|   Resolution:  fixed
 Severity:  Normal   | Triage Stage:  Ready for
 Keywords:  dumpdata fixtures|  checkin
  memory |  Needs documentation:  0
Has patch:  1|  Patch needs improvement:  0
  Needs tests:  0|UI/UX:  0
Easy pickings:  0|
-+-

Comment (by barry.melton@…):

 With these changes, serialized objects now seem to start with "[ ,", which
 is wrong (including a comma, even on the first result.)

 This patch allows me to export my data sets correctly, but does so with an
 invalid JSON schema, and does not allow me to import them.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  closed
Component:  Core |  Version:  master
  (Serialization)|   Resolution:  fixed
 Severity:  Normal   | Triage Stage:  Ready for
 Keywords:  dumpdata fixtures|  checkin
  memory |  Needs documentation:  0
Has patch:  1|  Patch needs improvement:  0
  Needs tests:  0|UI/UX:  0
Easy pickings:  0|
-+-
Changes (by Claude Paroz ):

 * status:  assigned => closed
 * resolution:   => fixed


Comment:

 In [3b5083bee5e96539dec599106aece9889e70ce05]:
 {{{
 #!CommitTicketReference repository=""
 revision="3b5083bee5e96539dec599106aece9889e70ce05"
 Fixed #5423 -- Made dumpdata output one row at a time.

 This should prevent storing all rows in memory when big sets of
 data are dumped.
 See ticket for heroic contributors.
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  assigned
Component:  Core |  Version:  SVN
  (Serialization)|   Resolution:
 Severity:  Normal   | Triage Stage:  Ready for
 Keywords:  dumpdata fixtures|  checkin
  memory |  Needs documentation:  0
Has patch:  1|  Patch needs improvement:  0
  Needs tests:  0|UI/UX:  0
Easy pickings:  0|
-+-
Changes (by charettes):

 * needs_better_patch:  1 => 0
 * stage:  Accepted => Ready for checkin


Comment:

 The patch looks good to me. I tried running it against a database that use
 to exhaust all memory and never achieved to actually finish dumping the
 data and it succeed.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  assigned
Component:  Core |  Version:  SVN
  (Serialization)|   Resolution:
 Severity:  Normal   | Triage Stage:  Accepted
 Keywords:  dumpdata fixtures|  Needs documentation:  0
  memory |  Patch needs improvement:  1
Has patch:  1|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-

Comment (by rm_):

 As promised on IRC to cramm (xrmx here) here's some results with / without
 latest lukeplant's patch on top of django 1.4b1.
 First my environment is a bit peculiar, i have a fixed limit of address
 space (the sum of all kind of memory) a process can use, in this case
 96MB. So if my process consume more than 96MB it gets killed.

 Before patch:
 {{{
 ../venv14/bin/python manage.py dumpdata store > store.json
 /venv14/lib/python2.6/site-packages/django/conf/__init__.py:75:
 DeprecationWarning: The ADMIN_MEDIA_PREFIX setting has been removed; use
 STATIC_URL instead.
   "use STATIC_URL instead.", DeprecationWarning)
 Error: Unable to serialize database:
 }}}
 process got killed so no backtrace printed, store.json is empty

 After patch:

 {{{
 ../venv14/bin/python manage.py dumpdata store > store.json
 /venv14/lib/python2.6/site-packages/django/conf/__init__.py:75:
 DeprecationWarning: The ADMIN_MEDIA_PREFIX setting has been removed; use
 STATIC_URL instead.
   "use STATIC_URL instead.", DeprecationWarning)
 }}}
 {{{
 $ ls -l store.json
 17496440 2012-03-05 10:02 store.json
 }}}

 To conclude the patch makes a huge difference in my environment, from not
 working to working.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  assigned
Component:  Core |  Version:  SVN
  (Serialization)|   Resolution:
 Severity:  Normal   | Triage Stage:  Accepted
 Keywords:  dumpdata fixtures|  Needs documentation:  0
  memory |  Patch needs improvement:  1
Has patch:  1|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-
Changes (by nickname123):

 * cc: wgordonw1@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  assigned
Component:  Core |  Version:  SVN
  (Serialization)|   Resolution:
 Severity:  Normal   | Triage Stage:  Accepted
 Keywords:  dumpdata fixtures|  Needs documentation:  0
  memory |  Patch needs improvement:  1
Has patch:  1|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-

Comment (by lukeplant):

 Attempted to update for trunk - will one of the original authors please
 check this, and then mark it Ready-For-Checkin and I will commit.

 I confirm that this helps a lot - one dumpdata process I tried with went
 from max 247MB to max 77MB.

 I agree with toofishes that we should not wait for the much bigger changes
 that would be required to address the further caching problems with the DB
 drivers.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  assigned
Component:  Core |  Version:  SVN
  (Serialization)|   Resolution:
 Severity:  Normal   | Triage Stage:  Accepted
 Keywords:  dumpdata fixtures|  Needs documentation:  0
  memory |  Patch needs improvement:  1
Has patch:  1|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-
Changes (by mitar):

 * cc: mmitar@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  assigned
Component:  Core |  Version:  SVN
  (Serialization)|   Resolution:
 Severity:  Normal   | Triage Stage:  Accepted
 Keywords:  dumpdata fixtures|  Needs documentation:  0
  memory |  Patch needs improvement:  1
Has patch:  1|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-
Changes (by Cerin):

 * cc: Cerin (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  assigned
Component:  Core |  Version:  SVN
  (Serialization)|   Resolution:
 Severity:  Normal   | Triage Stage:  Accepted
 Keywords:  dumpdata fixtures|  Needs documentation:  0
  memory |  Patch needs improvement:  1
Has patch:  1|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-
Changes (by hakon.erichsen@…):

 * cc: hakon.erichsen@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  assigned
Component:  Core |  Version:  SVN
  (Serialization)|   Resolution:
 Severity:  Normal   | Triage Stage:  Accepted
 Keywords:  dumpdata fixtures|  Needs documentation:  0
  memory |  Patch needs improvement:  1
Has patch:  1|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-

Comment (by JannKleen):

 Ah, I forgot to post my memory usage ... I have about 50 models with about
 3.5m rows in total...

 It results in a 2.2G json file and peaks at about 173m of used memory
 during the process.

 We're using PostgreSQL by the way.

 Replying to [comment:51 pzinovkin]:
 > Just tried new patch with mysql. For me it's still consumes about 250
 megs. But it's way more better than "out of memory".
 > {{{
 > 311m 257m 4040 R   79 25.8   4:36.47 python
 > }}}
 >
 > {{{
 > 236M 2011-10-02 11:28 dump.json
 > }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  assigned
Component:  Core |  Version:  SVN
  (Serialization)|   Resolution:
 Severity:  Normal   | Triage Stage:  Accepted
 Keywords:  dumpdata fixtures|  Needs documentation:  0
  memory |  Patch needs improvement:  1
Has patch:  1|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-

Comment (by pzinovkin):

 Just tried new patch with mysql. For me it's still consumes about 250
 megs. But it's way more better than "out of memory".
 {{{
 311m 257m 4040 R   79 25.8   4:36.47 python
 }}}

 {{{
 236M 2011-10-02 11:28 dump.json
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  assigned
Component:  Core |  Version:  SVN
  (Serialization)|   Resolution:
 Severity:  Normal   | Triage Stage:  Accepted
 Keywords:  dumpdata fixtures|  Needs documentation:  0
  memory |  Patch needs improvement:  1
Has patch:  1|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-
Changes (by JannKleen):

 * cc: jann@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
 Reporter:  adrian   |Owner:  ramiro
 Type:  New feature  |   Status:  assigned
Component:  Core |  Version:  SVN
  (Serialization)|   Resolution:
 Severity:  Normal   | Triage Stage:  Accepted
 Keywords:  dumpdata fixtures|  Needs documentation:  0
  memory |  Patch needs improvement:  1
Has patch:  1|UI/UX:  0
  Needs tests:  0|
Easy pickings:  0|
-+-

Comment (by JannKleen):

 When using patch  5423.1.diff, 5423.2.diff or 5423.3.diff, we're still
 saving all the data in a StreamIO object instead of writing it directly to
 sys.stdout, which caused huge memory usage again...
 I combined 5423.diff and 5423.3.diff which solved the problem for me.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |  Owner:  ramiro
   Type:  New| Status:  assigned
  feature|  Component:  Core
  Milestone: |  (Serialization)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:  dumpdata fixtures
   Triage Stage:  Accepted   |  memory
Needs documentation:  0  |  Has patch:  1
Patch needs improvement:  1  |Needs tests:  0
  UI/UX:  0  |  Easy pickings:  0
-+-
Changes (by toofishes):

 * cc: dpmcgee@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |  Owner:  ramiro
   Type:  New| Status:  assigned
  feature|  Component:  Core
  Milestone: |  (Serialization)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:  dumpdata fixtures
   Triage Stage:  Accepted   |  memory
Needs documentation:  0  |  Has patch:  1
Patch needs improvement:  1  |Needs tests:  0
  UI/UX:  0  |  Easy pickings:  0
-+-

Comment (by toofishes):

 New ticket opened with preliminary patch for server-side cursors: #16614.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |  Owner:  ramiro
   Type:  New| Status:  assigned
  feature|  Component:  Core
  Milestone: |  (Serialization)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:  dumpdata fixtures
   Triage Stage:  Accepted   |  memory
Needs documentation:  0  |  Has patch:  1
Patch needs improvement:  1  |Needs tests:  0
  UI/UX:  0  |  Easy pickings:  0
-+-

Comment (by toofishes):

 My last comment looks like it isn't totally correct- Django does give you
 a new cursor each time you ask for one. Apparently we never close cursors?
 Not sure if this is a potential problem in some DBMSs.

 With that said, I still stand by my view that server-side cursors is
 fundamentally a different problem and should not hold this patch from
 going in.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |  Owner:  ramiro
   Type:  New| Status:  assigned
  feature|  Component:  Core
  Milestone: |  (Serialization)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:  dumpdata fixtures
   Triage Stage:  Accepted   |  memory
Needs documentation:  0  |  Has patch:  1
Patch needs improvement:  1  |Needs tests:  0
  UI/UX:  0  |  Easy pickings:  0
-+-

Comment (by toofishes):

 Replying to [comment:40 ramiro]:
 > Unfortunately the patch isn't ready, some comments and further testing
 showed that in the PostgreSQL case big ORM queries are still consuming too
 mucho memory.
 >
 > I suspect the patch solves things in the Django side because it
 implemenst usage of QuerySet.iterator(). But we still need to add to e.g.
 outr PostgreSQL backend the ability to signal psycopg2/RDBMS it shouldn't
 get all the queryset from the server, maybe using server-side cursors.
 >

 I'm going to respectfully disagree with your switch away from "Ready for
 checkin".
 1. This patch still improves the situation heavily, as it eliminates the
 double caching problems that existed in both database drivers and the
 Django querysets.
 2. Looking into server side cursors, I was going to attempt to implement
 this, but it turns out the Django ORM is built with a fundamental
 limitation- that you have one connection with one cursor to a given
 database, period. Take a look at `django/db/backends/__init__.py`, you
 will see the cursor() method always returns the single cached cursor
 object from the BaseDatabaseWrapper object. This works if every query is
 fully executed and the cursor is available for immediate reuse, but with
 any sort of iteration, you can't touch the shared cursor object and would
 need to create a new one solely for that purpose. This brings with it a
 whole new bag of worms- now you have to encapsulate the cursor and close
 it when done, among other things, which is a non-trivial addition to the
 current database code.

 So in short, I think holding off on applying this is the wrong decision,
 and another bug should be opened, related to #13869 and this one, that
 adds full support for server side cursors and the management of them, or
 their equivalent, in every database backend. Then it should be a simple 1
 or 2 line adjustment to make dumpdata take advantage of this new feature.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |  Owner:  ramiro
   Type:  New| Status:  assigned
  feature|  Component:  Core
  Milestone: |  (Serialization)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:  dumpdata fixtures
   Triage Stage:  Accepted   |  memory
Needs documentation:  0  |  Has patch:  1
Patch needs improvement:  1  |Needs tests:  0
  UI/UX:  0  |  Easy pickings:  0
-+-
Changes (by ben@…):

 * cc: ben@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |  Owner:  ramiro
   Type:  New| Status:  assigned
  feature|  Component:  Core
  Milestone: |  (Serialization)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:  dumpdata fixtures
   Triage Stage:  Accepted   |  memory
Needs documentation:  0  |  Has patch:  1
Patch needs improvement:  1  |Needs tests:  0
  UI/UX:  0  |  Easy pickings:  0
-+-
Changes (by znick):

 * cc: znick (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |  Owner:  ramiro
   Type:  New| Status:  assigned
  feature|  Component:  Core
  Milestone: |  (Serialization)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:  dumpdata fixtures
   Triage Stage:  Accepted   |  memory
Needs documentation:  0  |  Has patch:  1
Patch needs improvement:  1  |Needs tests:  0
  UI/UX:  0  |  Easy pickings:  0
-+-
Changes (by pelletier.thomas@…):

 * cc: pelletier.thomas@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |  Owner:  ramiro
   Type:  New| Status:  assigned
  feature|  Component:  Core
  Milestone: |  (Serialization)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:  dumpdata fixtures
   Triage Stage:  Accepted   |  memory
Needs documentation:  0  |  Has patch:  1
Patch needs improvement:  1  |Needs tests:  0
  UI/UX:  0  |  Easy pickings:  0
-+-

Comment (by ramiro):

 (above comment was from me)

 Some links that might prove useful for anybody wishing to attack this:

 * http://www.initd.org/psycopg/docs/usage.html#server-side-cursors
 * http://thebuild.com/blog/2010/12/13/very-large-result-sets-in-django-
 using-postgresql/
 * http://thebuild.com/blog/2010/12/14/using-server-side-postgresql-
 cursors-in-django/
 * http://djangosnippets.org/snippets/2375/

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |  Owner:  ramiro
   Type:  New| Status:  assigned
  feature|  Component:  Core
  Milestone: |  (Serialization)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:  dumpdata fixtures
   Triage Stage:  Accepted   |  memory
Needs documentation:  0  |  Has patch:  1
Patch needs improvement:  1  |Needs tests:  0
  UI/UX:  0  |  Easy pickings:  0
-+-
Changes (by anonymous):

 * needs_better_patch:  0 => 1
 * stage:  Ready for checkin => Accepted


Comment:

 Unfortunately the patch isn't ready, some comments and further etsting
 showed it is still consuming

 I suspect the patch solves things in the Django side because it implemenst
 usage of QuerySet.iterator(). But we still need to add to e.g. outr
 PostgreSQL backend the ability to signal psycopg2/RDBMS it shouldn't get
 all the queryset from the server, maybe using server-side cursors.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |  Owner:  ramiro
   Type:  New| Status:  assigned
  feature|  Component:  Core
  Milestone: |  (Serialization)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:  dumpdata fixtures
   Triage Stage:  Ready for  |  memory
  checkin|  Has patch:  1
Needs documentation:  0  |Needs tests:  0
Patch needs improvement:  0  |  Easy pickings:  0
  UI/UX:  0  |
-+-
Changes (by pzinovkin):

 * ui_ux:   => 0
 * stage:  Accepted => Ready for checkin


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |  Owner:  ramiro
   Type:  New| Status:  assigned
  feature|  Component:  Core
  Milestone: |  (Serialization)
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:  dumpdata fixtures
   Triage Stage:  Accepted   |  memory
Needs documentation:  0  |  Has patch:  1
Patch needs improvement:  0  |Needs tests:  0
 |  Easy pickings:  0
-+-
Changes (by rm_):

 * cc: riccardo.magliocchetti@… (added)
 * easy:   => 0


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |Owner:  ramiro
   Type:  New|   Status:  assigned
  feature|Component:  Core (Serialization)
  Milestone: | Severity:  Normal
Version:  SVN| Keywords:  dumpdata fixtures
 Resolution: |  memory
   Triage Stage:  Accepted   |Has patch:  1
Needs documentation:  0  |  Needs tests:  0
Patch needs improvement:  0  |
-+-
Changes (by gremmie):

 * cc: bgneal@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |Owner:  ramiro
 Status:  assigned   |Milestone:
  Component: |  Version:  SVN
  Serialization  | Keywords:  dumpdata fixtures
 Resolution: |  memory
   Triage Stage:  Accepted   |Has patch:  1
Needs documentation:  0  |  Needs tests:  0
Patch needs improvement:  0  |
-+-
Changes (by tomchristie):

 * cc: tomchristie (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |Owner:  ramiro
 Status:  assigned   |Milestone:
  Component: |  Version:  SVN
  Serialization  | Keywords:  dumpdata fixtures
 Resolution: |  memory
   Triage Stage:  Accepted   |Has patch:  1
Needs documentation:  0  |  Needs tests:  0
Patch needs improvement:  0  |
-+-
Changes (by kmike):

 * cc: kmike84@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |Owner:  ramiro
 Status:  assigned   |Milestone:
  Component: |  Version:  SVN
  Serialization  | Keywords:  dumpdata fixtures
 Resolution: |  memory
   Triage Stage:  Accepted   |Has patch:  1
Needs documentation:  0  |  Needs tests:  0
Patch needs improvement:  0  |
-+-

Comment (by Boris Savelev ):

 thanks for formating-)

 I use MySQL backend.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |Owner:  ramiro
 Status:  assigned   |Milestone:
  Component: |  Version:  SVN
  Serialization  | Keywords:  dumpdata fixtures
 Resolution: |  memory
   Triage Stage:  Accepted   |Has patch:  1
Needs documentation:  0  |  Needs tests:  0
Patch needs improvement:  0  |
-+-

Comment (by ramiro):

 Replying to [comment:31 Boris Savelev ]:

 Thanks for testing the patch. I've reformatted your comment so it is more
 readable. Can you tell us which database backend are you using?

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+-
   Reporter:  adrian |Owner:  ramiro
 Status:  assigned   |Milestone:
  Component: |  Version:  SVN
  Serialization  | Keywords:  dumpdata fixtures
 Resolution: |  memory
   Triage Stage:  Accepted   |Has patch:  1
Needs documentation:  0  |  Needs tests:  0
Patch needs improvement:  0  |
-+-

Comment (by Boris Savelev ):

 I try dumpdata with latest patch
 python manage.py dumpdata -a > dump.json

 at the end of work from `top`:
 446m 429m 3460 R 96.4 12.3   2:54.07 python manage.py dumpdata -a

 ls -lha dump.json
 -rw-r--r-- 1 boris boris 77M Мар 10 17:50 dump.json

 I din't think that it is ok.

 Also, dumpdata doesn't flush data on disk while working
 data flush only at the end

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+--
   Reporter:  adrian | Owner:  ramiro   
   
 Status:  assigned   | Milestone:   
   
  Component:  Serialization  |   Version:  SVN  
   
 Resolution: |  Keywords:  dumpdata fixtures 
memory
   Triage Stage:  Accepted   | Has patch:  1
   
Needs documentation:  0  |   Needs tests:  0
   
Patch needs improvement:  0  |  
-+--

Comment (by ramiro):

 Results obtained testing with the setup described at and downloadable from
 https://bitbucket.org/cramm/serializers_memory_usage/src (serialization of
 50,000 instances of
 
[https://bitbucket.org/cramm/serializers_memory_usage/src/dc88a9b44de7/serializers_memory_usage/ticket5423/models.py
 this model ] to JSON). Results with XML, YAMS and MySQL, SQLite3 are
 similar.

 == Unpatched trunk ==

 [[Image(json-postgresq-trunk.png)]]

 == Patched trunk ==

 [[Image(json-postgresq-trunk-patched.png)]]

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
-+--
   Reporter:  adrian | Owner:  ramiro   
   
 Status:  assigned   | Milestone:   
   
  Component:  Serialization  |   Version:  SVN  
   
 Resolution: |  Keywords:  dumpdata fixtures 
memory
   Triage Stage:  Accepted   | Has patch:  1
   
Needs documentation:  0  |   Needs tests:  0
   
Patch needs improvement:  0  |  
-+--
Changes (by ramiro):

  * keywords:  dumpdata loaddata fixtures memory => dumpdata fixtures
   memory


Comment:

 Removing ''loaddata'' keyword. Loaddata has its own ticket tracking memory
 its usage issues (#12007).

 Please test the latest patch and report back your experiences. I have
 created a test environment to measure, graph and compare memory usage:
 https://bitbucket.org/cramm/serializers_memory_usage/src

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
+---
  Reporter:  adrian | Owner:  ramiro
   
Status:  assigned   | Milestone:
   
 Component:  Serialization  |   Version:  SVN   
   
Resolution: |  Keywords:  dumpdata loaddata 
fixtures memory
 Stage:  Accepted   | Has_patch:  1 
   
Needs_docs:  0  |   Needs_tests:  0 
   
Needs_better_patch:  0  |  
+---
Changes (by ramiro):

  * needs_better_patch:  1 => 0

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
+---
  Reporter:  adrian | Owner:  ramiro
   
Status:  assigned   | Milestone:
   
 Component:  Serialization  |   Version:  SVN   
   
Resolution: |  Keywords:  dumpdata loaddata 
fixtures memory
 Stage:  Accepted   | Has_patch:  1 
   
Needs_docs:  0  |   Needs_tests:  0 
   
Needs_better_patch:  1  |  
+---
Changes (by ramiro):

  * owner:  nobody => ramiro
  * status:  new => assigned

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
+---
  Reporter:  adrian | Owner:  nobody
   
Status:  new| Milestone:
   
 Component:  Serialization  |   Version:  SVN   
   
Resolution: |  Keywords:  dumpdata loaddata 
fixtures memory
 Stage:  Accepted   | Has_patch:  1 
   
Needs_docs:  0  |   Needs_tests:  0 
   
Needs_better_patch:  1  |  
+---
Changes (by bronger):

 * cc: bronger (added)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-upda...@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
+---
  Reporter:  adrian | Owner:  nobody
   
Status:  new| Milestone:
   
 Component:  Serialization  |   Version:  SVN   
   
Resolution: |  Keywords:  dumpdata loaddata 
fixtures memory
 Stage:  Accepted   | Has_patch:  1 
   
Needs_docs:  0  |   Needs_tests:  0 
   
Needs_better_patch:  1  |  
+---
Comment (by russellm):

 #10301 marked as a duplicate. It has a patch that could be an alternative
 solution.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
+---
  Reporter:  adrian | Owner:  nobody
   
Status:  new| Milestone:
   
 Component:  Serialization  |   Version:  SVN   
   
Resolution: |  Keywords:  dumpdata loaddata 
fixtures memory
 Stage:  Accepted   | Has_patch:  1 
   
Needs_docs:  0  |   Needs_tests:  0 
   
Needs_better_patch:  1  |  
+---
Comment (by phaesler):

 Bah - half these patches do exactly that.  I'll shut up now.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
+---
  Reporter:  adrian | Owner:  nobody
   
Status:  new| Milestone:
   
 Component:  Serialization  |   Version:  SVN   
   
Resolution: |  Keywords:  dumpdata loaddata 
fixtures memory
 Stage:  Accepted   | Has_patch:  1 
   
Needs_docs:  0  |   Needs_tests:  0 
   
Needs_better_patch:  1  |  
+---
Comment (by phaesler):

 Oops.  Should have used preview.  That should read:

 One approach would be to have the format Serializers handle the "list of
 objects" logic manually. E.g. for JSON, have start_serialization write
 open-square-bracket to the stream; have end_object write a comma to stream
 if needed and dump the object tree to stream using simplejson;
 end_serialization needs to write the close-square-bracket; the existing
 objects member should be removed all together.

 I might have a bash at a patch this weekend.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
+---
  Reporter:  adrian | Owner:  nobody
   
Status:  new| Milestone:
   
 Component:  Serialization  |   Version:  SVN   
   
Resolution: |  Keywords:  dumpdata loaddata 
fixtures memory
 Stage:  Accepted   | Has_patch:  1 
   
Needs_docs:  0  |   Needs_tests:  0 
   
Needs_better_patch:  1  |  
+---
Changes (by phaesler):

  * needs_better_patch:  0 => 1
  * component:  django-admin.py => Serialization

Comment:

 OK, then we're agreed this is a Serializer issue.

 One approach would be to have the format Serializers handle the "list of
 objects" logic manually.  E.g. for JSON, have start_serialization write
 "[" to the stream; have end_object write a comma to stream if needed and
 dump the object tree to stream using simplejson; end_serialization needs
 to write the "]"; the existing objects member should be removed all
 together.  I might have a bash at a patch this weekend.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
--+-
  Reporter:  adrian   | Owner:  nobody  
 
Status:  new  | Milestone:  
 
 Component:  django-admin.py  |   Version:  SVN 
 
Resolution:   |  Keywords:  dumpdata loaddata 
fixtures memory
 Stage:  Accepted | Has_patch:  1   
 
Needs_docs:  0|   Needs_tests:  0   
 
Needs_better_patch:  0|  
--+-
Comment (by russellm):

 Dumpdata is a light wrapper around the serializers. Any problem with
 dumpdata that goes deeper than command arguments not being parsed
 correctly points to a deeper issue with the serialiers themselves.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
--+-
  Reporter:  adrian   | Owner:  nobody  
 
Status:  new  | Milestone:  
 
 Component:  django-admin.py  |   Version:  SVN 
 
Resolution:   |  Keywords:  dumpdata loaddata 
fixtures memory
 Stage:  Accepted | Has_patch:  1   
 
Needs_docs:  0|   Needs_tests:  0   
 
Needs_better_patch:  0|  
--+-
Comment (by phaesler):

 I just opened this: #11258

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
--+-
  Reporter:  adrian   | Owner:  nobody  
 
Status:  new  | Milestone:  
 
 Component:  django-admin.py  |   Version:  SVN 
 
Resolution:   |  Keywords:  dumpdata loaddata 
fixtures memory
 Stage:  Accepted | Has_patch:  1   
 
Needs_docs:  0|   Needs_tests:  0   
 
Needs_better_patch:  0|  
--+-
Comment (by phaesler):

 The problem is not in dumpdata, it's in the Django Serialisation code.
 The core.serializers.python.Serializer doesn't start writing to the output
 stream until all objects to be serialised have been read into memory.

 The attached patches won't work until the above is addressed.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
--+-
  Reporter:  adrian   | Owner:  nobody  
 
Status:  new  | Milestone:  
 
 Component:  django-admin.py  |   Version:  SVN 
 
Resolution:   |  Keywords:  dumpdata loaddata 
fixtures memory
 Stage:  Accepted | Has_patch:  1   
 
Needs_docs:  0|   Needs_tests:  0   
 
Needs_better_patch:  0|  
--+-
Comment (by peterbe):

 @jmcvetta I don't think this has yet been added to trunk. Just checked
 today, 7 Apr 2009.
 Glad to hear that you got it working with your patch but I don't think I
 can use that patch any more on the trunk since it appears to be very
 different.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
--+-
  Reporter:  adrian   | Owner:  nobody  
 
Status:  new  | Milestone:  
 
 Component:  django-admin.py  |   Version:  SVN 
 
Resolution:   |  Keywords:  dumpdata loaddata 
fixtures memory
 Stage:  Accepted | Has_patch:  1   
 
Needs_docs:  0|   Needs_tests:  0   
 
Needs_better_patch:  0|  
--+-
Changes (by jmcvetta):

 * cc: jmcvetta (added)
  * keywords:  => dumpdata loaddata fixtures memory

Comment:

 It appears [attachment:dumpdata_streamed_output_django1.0-fix.diff the
 latest patch] has already been applied to trunk:
 {{{
 $ patch -p1 < ~/incoming/dumpdata_streamed_output_django1.0-fix.diff
 patching file django/core/management/commands/dumpdata.py
 Reversed (or previously applied) patch detected!  Assume -R? [n]
 }}}

 I experienced no problems using {{{dumpdata}}} to export a ~160,000 record
 fixture, in both json and yaml formats.  However, while I was able run
 {{{loaddata}}} on the json fixture without incident, attempting to load
 the yaml fixture quickly overwhelmed the memory on my 4GB workstation.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
--+-
  Reporter:  adrian   | Owner:  nobody
Status:  new  | Milestone:
 Component:  django-admin.py  |   Version:  SVN   
Resolution:   |  Keywords:
 Stage:  Accepted | Has_patch:  1 
Needs_docs:  0|   Needs_tests:  0 
Needs_better_patch:  0|  
--+-
Comment (by Martin):

 I installed the patch, but still have to watch django eat up all the
 memory without any output, when I try to dump my database (>700.000
 records) :(
 Debug is False. Is there anything else where I could go wrong? What's the
 format with the least memory use?

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
--+-
  Reporter:  adrian   | Owner:  nobody
Status:  new  | Milestone:
 Component:  django-admin.py  |   Version:  SVN   
Resolution:   |  Keywords:
 Stage:  Accepted | Has_patch:  1 
Needs_docs:  0|   Needs_tests:  0 
Needs_better_patch:  0|  
--+-
Comment (by emes):

 I mean ''dumpdata_streamed_output_django1.0-fix.diff'' fixes the bugs
 mentioned above and there are no more bugs known to me :)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #5423: "dumpdata" should stream output one row at a time

[Django]

#5423: "dumpdata" should stream output one row at a time
--+-
  Reporter:  adrian   | Owner:  nobody
Status:  new  | Milestone:
 Component:  django-admin.py  |   Version:  SVN   
Resolution:   |  Keywords:
 Stage:  Accepted | Has_patch:  1 
Needs_docs:  0|   Needs_tests:  0 
Needs_better_patch:  0|  
--+-
Comment (by emes):

 This patch has bugs:
  * Ordering models by model._meta.pk.attname fails on models where primary
 key is OneToOneField ('parent' vs 'parent_id').
  * JSON separator is not placed between model entries, due to first=True
 inside loop  The attached patch corrects them.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---