Keeping things as seen is the point. If the data contains any characters outside of the ASCII range, then you can't (directly) store it (reliably) in a db set for ASCII. Unicode encompasses all the characters that you will find on the web. UTF-8 is a relatively trouble free way to store unicode in a database, and, if a substantial majority of your characters actually are in the ASCII set, then UTF-8 is more compact than some of the other encodings.
You are much better off figuring out where your errors came from when the db was UTF-8 (or another unicode encoding). On 4/2/12, Ali Mesdaq <ali.mes...@fireeye.com> wrote: > I did have UTF-8 in the db before but then when I was inserting(done by a > script) values in I would get errors. I really want to avoid doing > conversions before I insert the data so I can maintain the data in the > exact format as it was seen in. I may be forced into doing that it seems > to make django even work. I tried to get my app to work by creating the > __unicode__ method return a string of "blah" and it still dies on me. > > Thanks, > Ali Mesdaq > Security Researcher > Cell: +1 (619) 952-8488 | Fax: +1 (408) 321-9818 > Email: ali.mes...@fireeye.com > > > Next Generation Threat Protection > http://www.FireEye.com <http://www.fireeye.com/> > > > > > > > > On 4/1/12 3:04 PM, "Bill Freeman" <ke1g...@gmail.com> wrote: > >>If you can, switch to UTF-8 in the db. Web traffic can arrive in a >>variety of >>encodings, and while the headers tell the server how to make unicode out >>of it, but by having the db set for ASCII, you limit what you can store >>(since >>not all unicode characters can be converted to ASCII. >> >>On Fri, Mar 30, 2012 at 7:45 PM, Ali Mesdaq <ali.mes...@fireeye.com> >>wrote: >>> I have a situation where I am reading data that I have no control over >>>and >>> inserting it into a db. The data is http headers. I am storing them in >>> postgres db in a text field and the db encoding is SQL_ASCII. Since the >>> data can be anything even non compliant http headers with anything for >>>its >>> values I don't want to modify the data before I store it in the db. >>> However this is causing issues with certain values causing the >>> UnicodeDecodeError. For example I have a specific case where the user >>> agent is set to 'KC\xd4\xda\xcf\xdf\xc9\xfd\xbc\xb6'. I have been trying >>> to look for a way to deal with these cases gracefully in the models >>> __unicode__ method but nothing I have tried has worked. >>> >>> Thanks, >>> Ali Mesdaq >>> Security Researcher >>> Cell: +1 (619) 952-8488 | Fax: +1 (408) 321-9818 >>> Email: ali.mes...@fireeye.com >>> >>> >>> Next Generation Threat Protection >>> http://www.FireEye.com <http://www.fireeye.com/> >>> >>> >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>>Groups "Django users" group. >>> To post to this group, send email to django-users@googlegroups.com. >>> To unsubscribe from this group, send email to >>>django-users+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>>http://groups.google.com/group/django-users?hl=en. >>> >> >>-- >>You received this message because you are subscribed to the Google Groups >>"Django users" group. >>To post to this group, send email to django-users@googlegroups.com. >>To unsubscribe from this group, send email to >>django-users+unsubscr...@googlegroups.com. >>For more options, visit this group at >>http://groups.google.com/group/django-users?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
