On Monday, April 2, 2012, Ali Mesdaq wrote: > I did have UTF-8 in the db before but then when I was inserting(done by a > script) values in I would get errors. I really want to avoid doing > conversions before I insert the data so I can maintain the data in the > exact format as it was seen in.
This, in general, isn't going to be possible. Even if you get it to the point where it looks like its working, you may still end up with silent data corruption, and not be able to retrieve your data when you need it. If your database column only supports ASCII, and you cant change the database, then your best option is probably going to be to use a strict ASCII encoding, like base64 or base85. If you are pretty sure that the data is *mostly* ASCII, then you could go with something like quoted-printable. The point, though, is that you need to do *something* to your data to coerce it into the ASCII range before you put it in the database. As long as you remember to encode the data on save, and decode it (once!) before displaying it, you should be able to get it to work. Ian > I may be forced into doing that it seems > to make django even work. I tried to get my app to work by creating the > __unicode__ method return a string of "blah" and it still dies on me. > > Thanks, > Ali Mesdaq > Security Researcher > Cell: +1 (619) 952-8488 | Fax: +1 (408) 321-9818 > Email: ali.mes...@fireeye.com > > > Next Generation Threat Protection > http://www.FireEye.com <http://www.fireeye.com/> > > > > > > > > On 4/1/12 3:04 PM, "Bill Freeman" <ke1g...@gmail.com> wrote: > > >If you can, switch to UTF-8 in the db. Web traffic can arrive in a > >variety of > >encodings, and while the headers tell the server how to make unicode out > >of it, but by having the db set for ASCII, you limit what you can store > >(since > >not all unicode characters can be converted to ASCII. > > > >On Fri, Mar 30, 2012 at 7:45 PM, Ali Mesdaq <ali.mes...@fireeye.com> > >wrote: > >> I have a situation where I am reading data that I have no control over > >>and > >> inserting it into a db. The data is http headers. I am storing them in > >> postgres db in a text field and the db encoding is SQL_ASCII. Since the > >> data can be anything even non compliant http headers with anything for > >>its > >> values I don't want to modify the data before I store it in the db. > >> However this is causing issues with certain values causing the > >> UnicodeDecodeError. For example I have a specific case where the user > >> agent is set to 'KC\xd4\xda\xcf\xdf\xc9\xfd\xbc\xb6'. I have been trying > >> to look for a way to deal with these cases gracefully in the models > >> __unicode__ method but nothing I have tried has worked. > >> > >> Thanks, > >> Ali Mesdaq > >> Security Researcher > >> Cell: +1 (619) 952-8488 | Fax: +1 (408) 321-9818 > >> Email: ali.mes...@fireeye.com > >> > >> > >> Next Generation Threat Protection > >> http://www.FireEye.com <http://www.fireeye.com/> > >> > >> > >> > >> > >> > >> -- > >> You received this message because you are subscribed to the Google > >>Groups "Django users" group. > >> To post to this group, send email to django-users@googlegroups.com. > >> To unsubscribe from this group, send email to > >>django-users+unsubscr...@googlegroups.com. > >> For more options, visit this group at > >>http://groups.google.com/group/django-users?hl=en. > >> > > > >-- > >You received this message because you are subscribed to the Google Groups > >"Django users" group. > >To post to this group, send email to django-users@googlegroups.com. > >To unsubscribe from this group, send email to > >django-users+unsubscr...@googlegroups.com. > >For more options, visit this group at > >http://groups.google.com/group/django-users?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
