Re: bug? Issue with markdown, encoding and mysql
On Wed, May 21, 2008 at 2:57 PM, Mike Chambers <[EMAIL PROTECTED]> wrote: > > Sorry. I passed some UTF-8 text to the > django.contrib.markup.templatetags.markup.markdown function (which is > included with django), and got what appeared to be an incorrectly > encoded string in return. Ah, OK, I guess you were trying to use django's wrapper to deal with encoding. Personally, I find that a little strange as it's meant specifically for the template system, but anyway... Django passes text around as unicode and I believe the wrapper function expects unicode as well. That's the thing, by the time text reaches the template, it's expected to already be unicode. As your trying to use the filter in an unintended fashion, you would need to convert your UTF-8 text to unicode first - or at least tell django that you have UTF-8 text. It's usually easier to convert to unicode. See the unicode documentation I linked to in my previous response. Of course, if you have to do that, you might as well use markdown directly. > > I thought this MIGHT be a bug with the django markdown function. If it > was, then I would log it. The "MIGHT" is your clue. If your not sure you have a bug, it's always best to assume user error. If a discussion on the users list confirms a bug exists, then it's time to report a bug and/or discuss ways to fix the bug in the dev list. > > From what you suggested, it appears that it is a bug with the > underlying markdown library, and not the django markdown function. > Therefore, I won't log it. > > I apologize for the noise this added to the list, and will be more > circumspect in the future when trying to contribute to improving django. No apology needed. Its an easy mistake for newer users to make. Now that you've been filled in on the details, you'll know how to proceed next time. Don't ever hesitate to contribute to Django. -- Waylan Limberg [EMAIL PROTECTED] --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: bug? Issue with markdown, encoding and mysql
Sorry. I passed some UTF-8 text to the
django.contrib.markup.templatetags.markup.markdown function (which is
included with django), and got what appeared to be an incorrectly
encoded string in return.
I thought this MIGHT be a bug with the django markdown function. If it
was, then I would log it.
From what you suggested, it appears that it is a bug with the
underlying markdown library, and not the django markdown function.
Therefore, I won't log it.
I apologize for the noise this added to the list, and will be more
circumspect in the future when trying to contribute to improving django.
mike
Waylan Limberg wrote:
> This appears to be a usage question. This list is for the development
> of Django itself, not developing projects that use Django. Usage
> questions should be directed to the django-users list [1].
>
> [1]: http://groups.google.com/group/django-users
>
> That said, it appears that you are taking issue with markdown, which
> is a separate library not included with django. Issues with markdown
> should be addressed on that projects mailing list [2]. However, keep
> in mind that markdown knows (almost) nothing about encodings. It only
> works with unicode (or ascii) text. You *must* give markdown unicode
> text and it *only* outputs unicode text. It is your responsibility to
> deal with whatever encodings you need. It would be almost imposable
> for markdown to support every possibility, so it doesn't even try.
>
> [2]:
> http://sourceforge.net/mailarchive/forum.php?forum_name=python-markdown-discuss
>
> However, Django does have some handy mechanisms [3] for dealing with
> this sort of thing. You might want to check them out.
>
> [3]: http://www.djangoproject.com/documentation/unicode/
>
> Also, why are you importing markdown from the template filter? Why not
> just import markdown directly?
>
> On Wed, May 21, 2008 at 2:11 PM, Mike Chambers <[EMAIL PROTECTED]> wrote:
>> I just ran into an issue where i was getting unicode errors when trying
>> to insert data into mysql (via a model).
>>
>> I had this code:
>>
>> --
>> from django.contrib.markup.templatetags.markup import markdown
>>
>> def save(self):
>>self.content_html = markdown(self.content_source)
>>
>>super(Chapter, self).save()
>> --
>>
>> self.content_source is utf-8
>>
>> This would cause a unicode error when the code tried to save the string
>> in the DB (mysql) if content_source contained any non-ascii chars.
>>
>> I was able to fix this by explicitly setting the encoding on the string
>> returned from markdown()
>>
>> --
>> from django.contrib.markup.templatetags.markup import markdown
>>
>> def save(self):
>>self.content_html = markdown(self.content_source).encode("utf-8")
>>
>>super(Chapter, self).save()
>> --
>>
>> However, I would have expected the markdown function to return the
>> correctly encoded string.
>>
>> Here is a simple script that I believe shows the difference in output:
>>
>> --
>> from django.contrib.markup.templatetags.markup import markdown
>>
>> tmp = u'Andr\202'
>> m = markdown(tmp)
>> print m
>> m = markdown(tmp).encode("utf-8")
>> print m
>> --
>>
>> Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17)
>> django.VERSION (0, 97, 'pre')
>>
>> I am pretty new to django, and dont have much experience working with
>> unicode, so I wanted to post here to see if anyone thought that this
>> looked like a bug? If so I will log it.
>>
>> mike
>>
>>
>
>
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: bug? Issue with markdown, encoding and mysql
On Wed, May 21, 2008 at 2:32 PM, Waylan Limberg <[EMAIL PROTECTED]> wrote: [snip] > in mind that markdown knows (almost) nothing about encodings. It only > works with unicode (or ascii) text. You *must* give markdown unicode > text and it *only* outputs unicode text. It is your responsibility to > deal with whatever encodings you need. It would be almost imposable > for markdown to support every possibility, so it doesn't even try. > Oh, I should mention that this policy has only been strictly enforced on the most recent release (1.7) of Python-Markdown. Previous versions may have appeared to work with some encodings, but it was always buggy. Your encouraged to upgrade to the latest version. -- Waylan Limberg [EMAIL PROTECTED] --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: bug? Issue with markdown, encoding and mysql
This appears to be a usage question. This list is for the development
of Django itself, not developing projects that use Django. Usage
questions should be directed to the django-users list [1].
[1]: http://groups.google.com/group/django-users
That said, it appears that you are taking issue with markdown, which
is a separate library not included with django. Issues with markdown
should be addressed on that projects mailing list [2]. However, keep
in mind that markdown knows (almost) nothing about encodings. It only
works with unicode (or ascii) text. You *must* give markdown unicode
text and it *only* outputs unicode text. It is your responsibility to
deal with whatever encodings you need. It would be almost imposable
for markdown to support every possibility, so it doesn't even try.
[2]:
http://sourceforge.net/mailarchive/forum.php?forum_name=python-markdown-discuss
However, Django does have some handy mechanisms [3] for dealing with
this sort of thing. You might want to check them out.
[3]: http://www.djangoproject.com/documentation/unicode/
Also, why are you importing markdown from the template filter? Why not
just import markdown directly?
On Wed, May 21, 2008 at 2:11 PM, Mike Chambers <[EMAIL PROTECTED]> wrote:
>
> I just ran into an issue where i was getting unicode errors when trying
> to insert data into mysql (via a model).
>
> I had this code:
>
> --
> from django.contrib.markup.templatetags.markup import markdown
>
> def save(self):
>self.content_html = markdown(self.content_source)
>
>super(Chapter, self).save()
> --
>
> self.content_source is utf-8
>
> This would cause a unicode error when the code tried to save the string
> in the DB (mysql) if content_source contained any non-ascii chars.
>
> I was able to fix this by explicitly setting the encoding on the string
> returned from markdown()
>
> --
> from django.contrib.markup.templatetags.markup import markdown
>
> def save(self):
>self.content_html = markdown(self.content_source).encode("utf-8")
>
>super(Chapter, self).save()
> --
>
> However, I would have expected the markdown function to return the
> correctly encoded string.
>
> Here is a simple script that I believe shows the difference in output:
>
> --
> from django.contrib.markup.templatetags.markup import markdown
>
> tmp = u'Andr\202'
> m = markdown(tmp)
> print m
> m = markdown(tmp).encode("utf-8")
> print m
> --
>
> Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17)
> django.VERSION (0, 97, 'pre')
>
> I am pretty new to django, and dont have much experience working with
> unicode, so I wanted to post here to see if anyone thought that this
> looked like a bug? If so I will log it.
>
> mike
>
>
> >
>
--
Waylan Limberg
[EMAIL PROTECTED]
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
bug? Issue with markdown, encoding and mysql
I just ran into an issue where i was getting unicode errors when trying
to insert data into mysql (via a model).
I had this code:
--
from django.contrib.markup.templatetags.markup import markdown
def save(self):
self.content_html = markdown(self.content_source)
super(Chapter, self).save()
--
self.content_source is utf-8
This would cause a unicode error when the code tried to save the string
in the DB (mysql) if content_source contained any non-ascii chars.
I was able to fix this by explicitly setting the encoding on the string
returned from markdown()
--
from django.contrib.markup.templatetags.markup import markdown
def save(self):
self.content_html = markdown(self.content_source).encode("utf-8")
super(Chapter, self).save()
--
However, I would have expected the markdown function to return the
correctly encoded string.
Here is a simple script that I believe shows the difference in output:
--
from django.contrib.markup.templatetags.markup import markdown
tmp = u'Andr\202'
m = markdown(tmp)
print m
m = markdown(tmp).encode("utf-8")
print m
--
Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17)
django.VERSION (0, 97, 'pre')
I am pretty new to django, and dont have much experience working with
unicode, so I wanted to post here to see if anyone thought that this
looked like a bug? If so I will log it.
mike
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
