Re: [Django] #16393: CookieWizard incompatible with simlejson 2.0.9

[Django]

#16393: CookieWizard incompatible with simlejson 2.0.9
---+
 Reporter:  kmtracey   |Owner:  kmtracey
 Type:  Bug|   Status:  closed
Component:  contrib.formtools  |  Version:  SVN
 Severity:  Normal |   Resolution:  fixed
 Keywords: | Triage Stage:  Accepted
Has patch:  0  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by Alex):

 * status:  new => closed
 * resolution:   => fixed


Comment:

 In [17014]:
 {{{
 #!CommitTicketReference repository="" revision="17014"
 Fixed #16393 -- FormWizard's cookie storage backend now works with all
 versions of simplejson and the standard library json module.
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #16393: CookieWizard incompatible with simlejson 2.0.9

[Django]

#16393: CookieWizard incompatible with simlejson 2.0.9
+---
   Reporter:  kmtracey  |  Owner:  kmtracey
   Type:  Bug   | Status:  new
  Milestone:|  Component:  contrib.formtools
Version:  SVN   |   Severity:  Normal
 Resolution:|   Keywords:
   Triage Stage:  Accepted  |  Has patch:  0
Needs documentation:  0 |Needs tests:  0
Patch needs improvement:  0 |  Easy pickings:  0
  UI/UX:  0 |
+---
Changes (by kmtracey):

 * owner:  nobody => kmtracey


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #16393: CookieWizard incompatible with simlejson 2.0.9

[Django]

#16393: CookieWizard incompatible with simlejson 2.0.9
+---
   Reporter:  kmtracey  |  Owner:  nobody
   Type:  Bug   | Status:  new
  Milestone:|  Component:  contrib.formtools
Version:  SVN   |   Severity:  Normal
 Resolution:|   Keywords:
   Triage Stage:  Accepted  |  Has patch:  0
Needs documentation:  0 |Needs tests:  0
Patch needs improvement:  0 |  Easy pickings:  0
  UI/UX:  0 |
+---

Comment (by steph):

 The patch fixes the failing tests but doesn't add new tests for the exact
 problem.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #16393: CookieWizard incompatible with simlejson 2.0.9

[Django]

#16393: CookieWizard incompatible with simlejson 2.0.9
+---
   Reporter:  kmtracey  |  Owner:  nobody
   Type:  Bug   | Status:  new
  Milestone:|  Component:  contrib.formtools
Version:  SVN   |   Severity:  Normal
 Resolution:|   Keywords:
   Triage Stage:  Accepted  |  Has patch:  0
Needs documentation:  0 |Needs tests:  0
Patch needs improvement:  0 |  Easy pickings:  0
  UI/UX:  0 |
+---
Changes (by jezdez):

 * component:  Forms => contrib.formtools


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #16393: CookieWizard incompatible with simlejson 2.0.9

[Django]

#16393: CookieWizard incompatible with simlejson 2.0.9
+
   Reporter:  kmtracey  |  Owner:  nobody
   Type:  Bug   | Status:  new
  Milestone:|  Component:  Forms
Version:  SVN   |   Severity:  Normal
 Resolution:|   Keywords:
   Triage Stage:  Accepted  |  Has patch:  0
Needs documentation:  0 |Needs tests:  0
Patch needs improvement:  0 |  Easy pickings:  0
  UI/UX:  0 |
+

Comment (by jezdez):

 Replying to [comment:3 kmtracey]:
 > This is pdb info from a session where simplejson 2.0.9 was installed. I
 added breakpoints to the django/contrib/formtools/wizard/storage/cookie.py
 file, methods load_data and update_response to see what was getting stored
 in the cookie and what was subsequently loaded:
 >
 > {{{
 > --> ./runtests.py --settings=testdb.sqlite -v0
 formtools.CookieWizardTests.test_cleaned_data
 > > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
 > -> response.set_signed_cookie(self.prefix,
 self.encoder.encode(self.data))
 > (Pdb) self.data
 > {'step_files': {}, 'step': u'form1', 'extra_data': {}, 'step_data': {}}
 > (Pdb) c
 > > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
 > -> return json.loads(data, cls=json.JSONDecoder)
 > (Pdb) n
 > --Return--
 > > /home/kmtracey/django/hg-
 
django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()->{u'extra_data':
 {}, u'step': u'form1', u'step_data': {}, u'step_files': {}}
 > -> return json.loads(data, cls=json.JSONDecoder)
 >
 > # No problem above, first calls to store/load the cookie data have no
 data that causes a problem
 >
 > # Next iteration is where there's a problem. The step_data for form1 is
 a QueryDict
 >
 > (Pdb) c
 > > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
 > -> response.set_signed_cookie(self.prefix,
 self.encoder.encode(self.data))
 > (Pdb) self.data
 > {u'step_files': {u'form1': {}}, u'step': u'form2', u'extra_data': {},
 u'step_data': {u'form1': }}
 > (Pdb) self.data['step_data']['form1']
 > 
 > (Pdb) c
 > > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
 > -> return json.loads(data, cls=json.JSONDecoder)
 > (Pdb) n
 > --Return--
 > > /home/kmtracey/django/hg-
 
django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()->{u'extra_data':
 {}, u'step': u'form2', u'step_data': {u'form1': {u'cookie...ent_step':
 [u'form1'], u'form1-name': [u'Pony'], u'form1-thirsty': [u'2'],
 u'form1-user': [u'1']}}, u'step_files': {u'form1': {}}}
 > -> return json.loads(data, cls=json.JSONDecoder)
 > (Pdb) n
 >
 > # When step_data for form1 is re-constituted, the values for all the
 keys are wrapped in lists
 > # This ultimately triggers errors like: TypeError: int() argument must
 be a string or a number, not 'list'
 > # when the wizard code uses the re-constituted step data as data for the
 form.
 >
 > > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(14)__init__()
 > -> if self.data is None:
 > (Pdb) self.data['step_data']['form1']
 > {u'form1-user': [u'1'], u'cookie_contact_wizard-current_step':
 [u'form1'], u'form1-name': [u'Pony'], u'form1-thirsty': [u'2']}
 > (Pdb)
 > }}}
 >
 > This is with simplejson 2.1.6 installed:
 >
 > {{{
 > --> ./runtests.py --settings=testdb.sqlite -v0
 formtools.CookieWizardTests.test_cleaned_data
 > > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
 > -> response.set_signed_cookie(self.prefix,
 self.encoder.encode(self.data))
 > (Pdb) c
 > > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
 > -> return json.loads(data, cls=json.JSONDecoder)
 > (Pdb) c
 > > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
 > -> response.set_signed_cookie(self.prefix,
 self.encoder.encode(self.data))
 > (Pdb) self.data
 > {u'step_files': {u'form1': {}}, u'step': u'form2', u'extra_data': {},
 u'step_data': {u'form1': }}
 > (Pdb) self.data['step_data']['form1']
 > 
 > (Pdb) c
 > > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
 > -> return json.loads(data, cls=json.JSONDecoder)
 > (Pdb) n
 > --Return--
 > > /home/kmtracey/django/hg-
 
django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()->{u'extra_data':
 {}, u'step': u'form2', u'step_data': {u'form1': {u'cookie...ent_step':
 u'form1', u'form1-name': u'Pony', u'form1-thirsty': u'2', u'form1-user':
 u'1'}}, u'step_files': {u'form1': {}}}
 > -> return json.loads(data, cls

Re: [Django] #16393: CookieWizard incompatible with simlejson 2.0.9

[Django]

#16393: CookieWizard incompatible with simlejson 2.0.9
+
   Reporter:  kmtracey  |  Owner:  nobody
   Type:  Bug   | Status:  new
  Milestone:|  Component:  Forms
Version:  SVN   |   Severity:  Normal
 Resolution:|   Keywords:
   Triage Stage:  Accepted  |  Has patch:  0
Needs documentation:  0 |Needs tests:  0
Patch needs improvement:  0 |  Easy pickings:  0
  UI/UX:  0 |
+

Comment (by kmtracey):

 This is pdb info from a session where simplejson 2.0.9 was installed. I
 added breakpoints to the django/contrib/formtools/wizard/storage/cookie.py
 file, methods load_data and update_response to see what was getting stored
 in the cookie and what was subsequently loaded:

 {{{
 --> ./runtests.py --settings=testdb.sqlite -v0
 formtools.CookieWizardTests.test_cleaned_data
 > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
 -> response.set_signed_cookie(self.prefix, self.encoder.encode(self.data))
 (Pdb) self.data
 {'step_files': {}, 'step': u'form1', 'extra_data': {}, 'step_data': {}}
 (Pdb) c
 > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
 -> return json.loads(data, cls=json.JSONDecoder)
 (Pdb) n
 --Return--
 > /home/kmtracey/django/hg-
 
django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()->{u'extra_data':
 {}, u'step': u'form1', u'step_data': {}, u'step_files': {}}
 -> return json.loads(data, cls=json.JSONDecoder)

 # No problem above, first calls to store/load the cookie data have no data
 that causes a problem

 # Next iteration is where there's a problem. The step_data for form1 is a
 QueryDict

 (Pdb) c
 > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
 -> response.set_signed_cookie(self.prefix, self.encoder.encode(self.data))
 (Pdb) self.data
 {u'step_files': {u'form1': {}}, u'step': u'form2', u'extra_data': {},
 u'step_data': {u'form1': }}
 (Pdb) self.data['step_data']['form1']
 
 (Pdb) c
 > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
 -> return json.loads(data, cls=json.JSONDecoder)
 (Pdb) n
 --Return--
 > /home/kmtracey/django/hg-
 
django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()->{u'extra_data':
 {}, u'step': u'form2', u'step_data': {u'form1': {u'cookie...ent_step':
 [u'form1'], u'form1-name': [u'Pony'], u'form1-thirsty': [u'2'],
 u'form1-user': [u'1']}}, u'step_files': {u'form1': {}}}
 -> return json.loads(data, cls=json.JSONDecoder)
 (Pdb) n

 # When step_data for form1 is re-constituted, the values for all the keys
 are wrapped in lists
 # This ultimately triggers errors like: TypeError: int() argument must be
 a string or a number, not 'list'
 # when the wizard code uses the re-constituted step data as data for the
 form.

 > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(14)__init__()
 -> if self.data is None:
 (Pdb) self.data['step_data']['form1']
 {u'form1-user': [u'1'], u'cookie_contact_wizard-current_step': [u'form1'],
 u'form1-name': [u'Pony'], u'form1-thirsty': [u'2']}
 (Pdb)
 }}}

 This is with simplejson 2.1.6 installed:

 {{{
 --> ./runtests.py --settings=testdb.sqlite -v0
 formtools.CookieWizardTests.test_cleaned_data
 > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
 -> response.set_signed_cookie(self.prefix, self.encoder.encode(self.data))
 (Pdb) c
 > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
 -> return json.loads(data, cls=json.JSONDecoder)
 (Pdb) c
 > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
 -> response.set_signed_cookie(self.prefix, self.encoder.encode(self.data))
 (Pdb) self.data
 {u'step_files': {u'form1': {}}, u'step': u'form2', u'extra_data': {},
 u'step_data': {u'form1': }}
 (Pdb) self.data['step_data']['form1']
 
 (Pdb) c
 > /home/kmtracey/django/hg-
 django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
 -> return json.loads(data, cls=json.JSONDecoder)
 (Pdb) n
 --Return--
 > /home/kmtracey/django/hg-
 
django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()->{u'extra_data':
 {}, u'step': u'form2', u'step_data': {u'form1': {u'cookie...ent_step':
 u'form1', u'form1-name': u'Pony', u'form1-thirsty': u'2', u'form1-user':
 u'1'}}, u'step_files': {u'form1': {}}}
 -> return json.loads(data, cls=json.JSONDecoder)
 (Pdb) n


 # With this level of simplejson, the step_data for form1 does not have its
 values wrapped in lists

 > /home/kmtracey/django/hg-
 django/django/c

Re: [Django] #16393: CookieWizard incompatible with simlejson 2.0.9

[Django]

#16393: CookieWizard incompatible with simlejson 2.0.9
+
   Reporter:  kmtracey  |  Owner:  nobody
   Type:  Bug   | Status:  new
  Milestone:|  Component:  Forms
Version:  SVN   |   Severity:  Normal
 Resolution:|   Keywords:
   Triage Stage:  Accepted  |  Has patch:  0
Needs documentation:  0 |Needs tests:  0
Patch needs improvement:  0 |  Easy pickings:  0
  UI/UX:  0 |
+

Comment (by jezdez):

 I was able to reproduce the error messages and am rather baffled by them,
 to be honest. Can you elaborate why `QueryDict` is supposed to be the
 problem?

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #16393: CookieWizard incompatible with simlejson 2.0.9

[Django]

#16393: CookieWizard incompatible with simlejson 2.0.9
+
   Reporter:  kmtracey  |  Owner:  nobody
   Type:  Bug   | Status:  new
  Milestone:|  Component:  Forms
Version:  SVN   |   Severity:  Normal
 Resolution:|   Keywords:
   Triage Stage:  Accepted  |  Has patch:  0
Needs documentation:  0 |Needs tests:  0
Patch needs improvement:  0 |  Easy pickings:  0
  UI/UX:  0 |
+
Changes (by aaugustin):

 * component:  Uncategorized => Forms
 * stage:  Unreviewed => Accepted


Comment:

 `django.http.QueryDict` inherits
 `django.utils.datastructures.MultiValueDict` where a custom `iteritems` is
 defined.

 The changelog of simplejson 2.1.0 (which is the version that comes just
 after 2.0.9) contains this line:
 > Encoder changed to use iteritems rather than PyDict_Next in order to
 support dict subclasses that have a well defined ordering
 http://bugs.python.org/issue6105

 This is a plausible explanation for the change in the serialization
 output.

 Also, the implementation with PyDict_Next in <= 2.0.9 can be used only
 when the C module (speedups) is compiled, and it isn't in the version
 bundled with Django — it's the pure Python implementation. This would
 explain why you don't see the bug when simplejson isn't installed.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

[Django] #16393: CookieWizard incompatible with simlejson 2.0.9

[Django]

#16393: CookieWizard incompatible with simlejson 2.0.9
--+-
 Reporter:  kmtracey  |Owner:  nobody
 Type:  Bug   |   Status:  new
Milestone:|Component:  Uncategorized
  Version:  SVN   | Severity:  Normal
 Keywords:| Triage Stage:  Unreviewed
Has patch:  0 |  Needs documentation:  0
  Needs tests:  0 |  Patch needs improvement:  0
Easy pickings:  0 |UI/UX:  0
--+-
 On an Ubuntu 10.04 machine, the formtools tests produce 5 errors (listed
 below). I eventually tracked this down to having the Ubuntu-repository
 version of python-simplejson installed. The version there is 2.0.9. If I
 instead pip install the current (2.1.16) one the tests pass. (They also
 pass on a machine with no simplejson installed, so falling back to the
 2.0.7 level bundled with Django.) The difference seems to be in how
 !QueryDicts are serialized: with the 2.0.9 level all the key values stored
 by the wizard are wrapped in lists; this list-wrapping does not seem to
 happen with the other levels.

 It seems like a bug in the 2.0.9 level (at least...I did not try anything
 besides 2.0.7, 2.0.9, and 2.1.16, so the problem may appear in other in-
 between levels, I don't know). The reason I'm opening a ticket here is
 Ubuntu 10.04 is the current Unbuntu long-term service release, and its
 system-install level of this package is one that doesn't seem to work
 properly, so I fear a fair number of people may run with it. One
 possibility to "fix" it would be to document a recommended level for this
 package if it is installed (we kind of do that for sqlite:
 https://docs.djangoproject.com/en/dev/ref/databases/#sqlite-3-3-6-or-
 newer-strongly-recommended). We don't currently seem to have any good
 place to put such info for simplejson though.

 Alternatively maybe something could be changed in !QueryDict code to get
 them to correctly serialize even on this 2.0.9 level of
 simplejson...though I have no idea if that is feasible.

 These are the test errors; I'm assuming they reflect problems that would
 be seen in real code and they are not just test setup artifacts:

 {{{
 --> ./runtests.py --settings=testdb.sqlite -v2 formtools
 Importing application formtools
 Creating test database for alias 'default' (':memory:')...
 Creating tables ...
 Creating table django_content_type
 Creating table auth_permission
 Creating table auth_group_permissions
 Creating table auth_group
 Creating table auth_user_user_permissions
 Creating table auth_user_groups
 Creating table auth_user
 Creating table django_site
 Creating table django_flatpage_sites
 Creating table django_flatpage
 Creating table django_redirect
 Creating table django_session
 Creating table django_comments
 Creating table django_comment_flags
 Creating table django_admin_log
 Installing custom SQL ...
 Installing indexes ...
 No fixtures found.
 Creating test database for alias 'other' ('other_db')...
 Destroying old test database 'other'...
 Creating tables ...
 Creating table django_content_type
 Creating table auth_permission
 Creating table auth_group_permissions
 Creating table auth_group
 Creating table auth_user_user_permissions
 Creating table auth_user_groups
 Creating table auth_user
 Creating table django_site
 Creating table django_flatpage_sites
 Creating table django_flatpage
 Creating table django_redirect
 Creating table django_session
 Creating table django_comments
 Creating table django_comment_flags
 Creating table django_admin_log
 Installing custom SQL ...
 Installing indexes ...
 No fixtures found.
 test_init
 (django.contrib.formtools.wizard.tests.formtests.CookieFormTests) ... ok
 test_cleaned_data
 (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests)
 ... ERROR
 test_form_finish
 (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests)
 ... ERROR
 test_form_post_error
 (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests)
 ... ok
 test_form_post_success
 (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests)
 ... ok
 test_form_refresh
 (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests)
 ... ERROR
 test_form_stepback
 (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests)
 ... ok
 test_initial_call
 (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests)
 ... ok
 test_manipulated_data
 (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests)
 ... ok
 test_template_context
 (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests)
 ... ok
 test_done (django.contrib.formtools.wizard.tests.formtests.FormTests) ...
 ok
 test_first_step
 (django.contrib.formtools.wizard.test