Re: textile mysql unicode and newforms
Hi Waylan
I tried it the uicode branch on my local machine and it didn't make any
difference. I'm still keeping it on my local box though.
So, I tweaked the code a bit, so it looks like this at the top of the save
overide now, not at the bottom and it's working as expected.
if self.summary:
self.summary = self.summary.encode('ascii', 'ignore')
self.summary_html = textile.textile(self.summary)
That way we simply don't pass the pasted apostrophes at all, and textile
works, what i was after in the first place.
This works on the production server that isn't running the unicode branch
yet. Soon though, when we get some time to test.
I think I might have some trouble when my boss tells me they want to use the
same code base to do a welsh site though, lots of funny characters in there.
And the country
:-)
On 19/06/07, Waylan Limberg <[EMAIL PROTECTED]> wrote:
>
>
> I suspect the Unicode branch [1] addresses the issues you are having.
> A few weeks back there was a call for testers as it is now feature
> complete. I'd suggest giving that a try.
>
> [1]: http://code.djangoproject.com/wiki/UnicodeBranch
>
>
> On 6/19/07, vanderkerkoff <[EMAIL PROTECTED]> wrote:
> >
> > Hello everyone
> >
> > I'm running django from the trunk, so using the most up to date
> > version, python 2.5 with PyTextile 2.0.10.
> > mysql5.0.2 with all settings to utf-8 and django content type is utf-8
> >
> > I'm overwriting the save command on events using newforms, we're
> > textiling the input for an html field, here's what I mean.
> >
> > def save(self):
> > import textile
> > if self.body:
> > self.body_html = textile.textile(self.body)
> > super(Event, self).save()
> >
> >
> > it fails with this error
> >
> > Exception Value:'ascii' codec can't decode byte 0xb4 in position
> 0:
> > ordinal not in range(128)
> > Exception Location:
> /usr/local/lib/python2.5/site-packages/textile.py
> > in glyphs, line 2418
> >
> > My textile settings are
> > # Set your encoding here.
> > ENCODING = 'utf8'
> >
> > # Output? Non-ASCII characters will be automatically
> > # converted to XML entities if you choose ASCII.
> > OUTPUT = 'utf8'
> >
> > I tried changing my OUTPUT to ascii in textile but got the same error,
> > so to me it looks like the form is sending a unicode
> > series of bytes to textile which it can't understand.
> >
> > One way around this is to manipulate the self.summary prior to passing
> > it to textile, like this.
> >
> > self.body = self.body.decode('utf-8')
> > self.body = self.body.encode('ascii', 'ignore')
> >
> > This forces the passing of ascii to textile and it likes that alot,
> > and works.
> >
> > But if a user now copies and pastes the dreaded apostrophe form word
> > or another special character unique to word,
> > it fails with this error.
> >
> > Exception Value:'ascii' codec can't encode character u'\u2019'
> in
> > position 5: ordinal not in range(128)
> > Exception Location: /usr/local/lib/python2.5/encodings/utf_8.py in
> > decode, line 16
> >
> >
> > If I run the super save earlier in the save definition after removing
> > the textiling of the body section, and then
> > call the data out of the database further down in the save definition,
> > and then save it again like this
> >
> > e = Event.objects.get(id=new_id)
> > if e.body:
> > e.body_html = textile.textile(e.body)
> > super(Event, e).save()
> >
> > It all works fine, no encoding or decoding needed for pasted
> > apostrophes or anything.
> >
> > Here's the paste of the relevant part of the form with certain
> > sections commented out so you can see what I mean.
> >
> > http://pastie.textmate.org/71702
> >
> > I found this on the google groups form Ivan Sagalev
> > To summarizes: your storage (a database) and your input/output (the
> > web)
> > really should use utf-8 to avoid problems with "strange" characters.
> > If
> > you deal internally with unicode (which newforms produce for you) then
> > for now you should explicitly encode from it to utf-8 until Django
> > starts doing it automatically.
> >
> > I've also been reading this thread on the google developers group, and
> > I'm now completely confused as to what is going on.
> >
> > unicode issues in multiple tickets
> >
> > If anyone can tell me if there is some current status on this, or how
> > it works right now I'd be really grateful. If I have to encode and
> > decode then I
> > don't mind, not much anyway :-)
> >
> >
> > >
> >
>
>
> --
>
> Waylan Limberg
> [EMAIL PROTECTED]
>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
Re: textile mysql unicode and newforms
I suspect the Unicode branch [1] addresses the issues you are having.
A few weeks back there was a call for testers as it is now feature
complete. I'd suggest giving that a try.
[1]: http://code.djangoproject.com/wiki/UnicodeBranch
On 6/19/07, vanderkerkoff <[EMAIL PROTECTED]> wrote:
>
> Hello everyone
>
> I'm running django from the trunk, so using the most up to date
> version, python 2.5 with PyTextile 2.0.10.
> mysql5.0.2 with all settings to utf-8 and django content type is utf-8
>
> I'm overwriting the save command on events using newforms, we're
> textiling the input for an html field, here's what I mean.
>
> def save(self):
> import textile
> if self.body:
> self.body_html = textile.textile(self.body)
> super(Event, self).save()
>
>
> it fails with this error
>
> Exception Value:'ascii' codec can't decode byte 0xb4 in position 0:
> ordinal not in range(128)
> Exception Location: /usr/local/lib/python2.5/site-packages/textile.py
> in glyphs, line 2418
>
> My textile settings are
> # Set your encoding here.
> ENCODING = 'utf8'
>
> # Output? Non-ASCII characters will be automatically
> # converted to XML entities if you choose ASCII.
> OUTPUT = 'utf8'
>
> I tried changing my OUTPUT to ascii in textile but got the same error,
> so to me it looks like the form is sending a unicode
> series of bytes to textile which it can't understand.
>
> One way around this is to manipulate the self.summary prior to passing
> it to textile, like this.
>
> self.body = self.body.decode('utf-8')
> self.body = self.body.encode('ascii', 'ignore')
>
> This forces the passing of ascii to textile and it likes that alot,
> and works.
>
> But if a user now copies and pastes the dreaded apostrophe form word
> or another special character unique to word,
> it fails with this error.
>
> Exception Value:'ascii' codec can't encode character u'\u2019' in
> position 5: ordinal not in range(128)
> Exception Location: /usr/local/lib/python2.5/encodings/utf_8.py in
> decode, line 16
>
>
> If I run the super save earlier in the save definition after removing
> the textiling of the body section, and then
> call the data out of the database further down in the save definition,
> and then save it again like this
>
> e = Event.objects.get(id=new_id)
> if e.body:
> e.body_html = textile.textile(e.body)
> super(Event, e).save()
>
> It all works fine, no encoding or decoding needed for pasted
> apostrophes or anything.
>
> Here's the paste of the relevant part of the form with certain
> sections commented out so you can see what I mean.
>
> http://pastie.textmate.org/71702
>
> I found this on the google groups form Ivan Sagalev
> To summarizes: your storage (a database) and your input/output (the
> web)
> really should use utf-8 to avoid problems with "strange" characters.
> If
> you deal internally with unicode (which newforms produce for you) then
> for now you should explicitly encode from it to utf-8 until Django
> starts doing it automatically.
>
> I've also been reading this thread on the google developers group, and
> I'm now completely confused as to what is going on.
>
> unicode issues in multiple tickets
>
> If anyone can tell me if there is some current status on this, or how
> it works right now I'd be really grateful. If I have to encode and
> decode then I
> don't mind, not much anyway :-)
>
>
> >
>
--
Waylan Limberg
[EMAIL PROTECTED]
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: textile mysql unicode and newforms
Here's the add_event in my views which uses the newforms and calls the save.
http://pastie.textmate.org/71706
It will probably help :-)
On 19/06/07, vanderkerkoff <[EMAIL PROTECTED]> wrote:
>
> Hello everyone
>
> I'm running django from the trunk, so using the most up to date
> version, python 2.5 with PyTextile 2.0.10.
> mysql5.0.2 with all settings to utf-8 and django content type is utf-8
>
> I'm overwriting the save command on events using newforms, we're
> textiling the input for an html field, here's what I mean.
>
> def save(self):
> import textile
> if self.body:
> self.body_html = textile.textile(self.body)
> super(Event, self).save()
>
>
> it fails with this error
>
> Exception Value:'ascii' codec can't decode byte 0xb4 in position
> 0:
> ordinal not in range(128)
> Exception Location: /usr/local/lib/python2.5/site-packages/textile.py
> in glyphs, line 2418
>
> My textile settings are
> # Set your encoding here.
> ENCODING = 'utf8'
>
> # Output? Non-ASCII characters will be automatically
> # converted to XML entities if you choose ASCII.
> OUTPUT = 'utf8'
>
> I tried changing my OUTPUT to ascii in textile but got the same error,
> so to me it looks like the form is sending a unicode
> series of bytes to textile which it can't understand.
>
> One way around this is to manipulate the self.summary prior to passing
> it to textile, like this.
>
> self.body = self.body.decode('utf-8')
> self.body = self.body.encode('ascii', 'ignore')
>
> This forces the passing of ascii to textile and it likes that alot,
> and works.
>
> But if a user now copies and pastes the dreaded apostrophe form word
> or another special character unique to word,
> it fails with this error.
>
> Exception Value:'ascii' codec can't encode character u'\u2019' in
> position 5: ordinal not in range(128)
> Exception Location: /usr/local/lib/python2.5/encodings/utf_8.py in
> decode, line 16
>
>
> If I run the super save earlier in the save definition after removing
> the textiling of the body section, and then
> call the data out of the database further down in the save definition,
> and then save it again like this
>
> e = Event.objects.get(id=new_id)
> if e.body:
> e.body_html = textile.textile(e.body)
> super(Event, e).save()
>
> It all works fine, no encoding or decoding needed for pasted
> apostrophes or anything.
>
> Here's the paste of the relevant part of the form with certain
> sections commented out so you can see what I mean.
>
> http://pastie.textmate.org/71702
>
> I found this on the google groups form Ivan Sagalev
> To summarizes: your storage (a database) and your input/output (the
> web)
> really should use utf-8 to avoid problems with "strange" characters.
> If
> you deal internally with unicode (which newforms produce for you) then
> for now you should explicitly encode from it to utf-8 until Django
> starts doing it automatically.
>
> I've also been reading this thread on the google developers group, and
> I'm now completely confused as to what is going on.
>
> unicode issues in multiple tickets
>
> If anyone can tell me if there is some current status on this, or how
> it works right now I'd be really grateful. If I have to encode and
> decode then I
> don't mind, not much anyway :-)
>
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
