Re: Enable unicode
On Tue, Jan 28, 2014 at 7:26 PM, Igor Korot ikoro...@gmail.com wrote: Hi, ALL, In here: http://stackoverflow.com/questions/21397035/set-utf8-on-mysql, I got a suggestion to enable use_unicode. Problem is I'm developing on Windows and it's not that I can recompile my python. I'm using Python2.7 on Windows XP. Any pointer on how do I enable use_unicode? Before you go any further: MySQL has a broken interpretation of utf8 that allows only a subset of the full Unicode range. Instead, use utf8mb4, which is what the rest of the world calls UTF-8. As far as I know, you can just switch in utf8mb4 everywhere that you're currently using utf8 and it'll work. According to [1] the use_unicode flag is a keyword parameter to connect(). As much as possible, I'd recommend using those parameters rather than explicitly executing SQL statements to reconfigure the connection - it's clearer, and the local client might want to reconfigure itself in response to the change too. Be aware that MySQL has a number of issues with Unicode and sorting (or at least, it did the last time I checked, which was a while ago now), not to mention other problems with its default MyISAM format. You may want to consider PostgreSQL instead. ChrisA [1] http://mysql-python.sourceforge.net/MySQLdb.html -- https://mail.python.org/mailman/listinfo/python-list
Re: Enable unicode
On Tue, Jan 28, 2014 at 7:56 PM, Igor Korot ikoro...@gmail.com wrote: Hi, Chris, Hi! I'm hoping it was oversight that led to this email coming to me personally instead of to the list, and hoping that you won't mind me responding on-list. On Tue, Jan 28, 2014 at 12:35 AM, Chris Angelico ros...@gmail.com wrote: On Tue, Jan 28, 2014 at 7:26 PM, Igor Korot ikoro...@gmail.com wrote: Hi, ALL, In here: http://stackoverflow.com/questions/21397035/set-utf8-on-mysql, I got a suggestion to enable use_unicode. Problem is I'm developing on Windows and it's not that I can recompile my python. I'm using Python2.7 on Windows XP. Any pointer on how do I enable use_unicode? Before you go any further: MySQL has a broken interpretation of utf8 that allows only a subset of the full Unicode range. Instead, use utf8mb4, which is what the rest of the world calls UTF-8. As far as I know, you can just switch in utf8mb4 everywhere that you're currently using utf8 and it'll work. So instead of using 'utf8' just use 'utf8mb4'? Yes, that's right. Unless utf8mb4 isn't supported, in which case try utf8 and see if you can use the full range (something might be translating it for you, which would probably be a good thing). According to [1] the use_unicode flag is a keyword parameter to connect(). As much as possible, I'd recommend using those parameters rather than explicitly executing SQL statements to reconfigure the connection - it's clearer, and the local client might want to reconfigure itself in response to the change too. Is it supported on all versions of MySQLDB? No idea! I don't use MySQLDB, so just give it a shot and see if it works. Be aware that MySQL has a number of issues with Unicode and sorting (or at least, it did the last time I checked, which was a while ago now), not to mention other problems with its default MyISAM format. You may want to consider PostgreSQL instead. I'm not using MyISAM, only InnoDB. ;-) That's good, but it doesn't cover everything. You may find that non-ASCII strings get mis-sorted. I strongly prefer PostgreSQL for anything where I actually care about the data I'm storing. And yes, that's everything that I store. So I don't use MySQL anywhere any more :) So, how do I properly write the connection lines? I've no idea - I don't actually use MySQLDB, I just looked at the docs :) But try adding use_unicode=True to your connect() call. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Enable unicode
Hi, Chris, On Tue, Jan 28, 2014 at 1:00 AM, Chris Angelico ros...@gmail.com wrote: On Tue, Jan 28, 2014 at 7:56 PM, Igor Korot ikoro...@gmail.com wrote: Hi, Chris, Hi! I'm hoping it was oversight that led to this email coming to me personally instead of to the list, and hoping that you won't mind me responding on-list. Sorry about that. I keep forgetting that gmail web mail is stupid and does not use the list address on Reply. Does not happen to other list, only this one. On Tue, Jan 28, 2014 at 12:35 AM, Chris Angelico ros...@gmail.com wrote: On Tue, Jan 28, 2014 at 7:26 PM, Igor Korot ikoro...@gmail.com wrote: Hi, ALL, In here: http://stackoverflow.com/questions/21397035/set-utf8-on-mysql, I got a suggestion to enable use_unicode. Problem is I'm developing on Windows and it's not that I can recompile my python. I'm using Python2.7 on Windows XP. Any pointer on how do I enable use_unicode? Before you go any further: MySQL has a broken interpretation of utf8 that allows only a subset of the full Unicode range. Instead, use utf8mb4, which is what the rest of the world calls UTF-8. As far as I know, you can just switch in utf8mb4 everywhere that you're currently using utf8 and it'll work. So instead of using 'utf8' just use 'utf8mb4'? Yes, that's right. Unless utf8mb4 isn't supported, in which case try utf8 and see if you can use the full range (something might be translating it for you, which would probably be a good thing). So you mean just try it in the python console and see if it works? According to [1] the use_unicode flag is a keyword parameter to connect(). As much as possible, I'd recommend using those parameters rather than explicitly executing SQL statements to reconfigure the connection - it's clearer, and the local client might want to reconfigure itself in response to the change too. Is it supported on all versions of MySQLDB? No idea! I don't use MySQLDB, so just give it a shot and see if it works. Problem is it might work on my machine, but the person I'm working with/for might have earlier version of the driver. That's actually why I asked on SO - is it supported on all versions of MySQLDB. Be aware that MySQL has a number of issues with Unicode and sorting (or at least, it did the last time I checked, which was a while ago now), not to mention other problems with its default MyISAM format. You may want to consider PostgreSQL instead. I'm not using MyISAM, only InnoDB. ;-) That's good, but it doesn't cover everything. You may find that non-ASCII strings get mis-sorted. I strongly prefer PostgreSQL for anything where I actually care about the data I'm storing. And yes, that's everything that I store. So I don't use MySQL anywhere any more :) Hehe. Originally the software was written for SQLite. Then mySQL support was added and now the new script should be made based on the existing one and this new script will be used in a web based app - Django + jQuery/jQWidgets. The person is not a developer, rather have a management background so he is going with the market trend, which is currently mySQL for web app. ;-) I'd very much prefer to use SQLite as the data we use are text of the different length which is more suitable for SQLite. But So, how do I properly write the connection lines? I've no idea - I don't actually use MySQLDB, I just looked at the docs :) But try adding use_unicode=True to your connect() call. OK, will do, thx. ChrisA -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Enable unicode
On Tue, Jan 28, 2014 at 8:20 PM, Igor Korot ikoro...@gmail.com wrote: So instead of using 'utf8' just use 'utf8mb4'? Yes, that's right. Unless utf8mb4 isn't supported, in which case try utf8 and see if you can use the full range (something might be translating it for you, which would probably be a good thing). So you mean just try it in the python console and see if it works? Yep. Play around with four categories of character: ASCII, Latin-1, Basic Multilingual Plane, and Supplementary Multilingual Plane. They correspond to codepoints 128, 256, 65536, and 1114112. Is it supported on all versions of MySQLDB? No idea! I don't use MySQLDB, so just give it a shot and see if it works. Problem is it might work on my machine, but the person I'm working with/for might have earlier version of the driver. That's actually why I asked on SO - is it supported on all versions of MySQLDB. Try it on yours, try it on theirs; or dig through the docs to see when it was added. Hard to say without knowing exactly what version your client is using. I strongly prefer PostgreSQL for anything where I actually care about the data I'm storing. And yes, that's everything that I store. So I don't use MySQL anywhere any more :) Hehe. Originally the software was written for SQLite. Then mySQL support was added and now the new script should be made based on the existing one and this new script will be used in a web based app - Django + jQuery/jQWidgets. The person is not a developer, rather have a management background so he is going with the market trend, which is currently mySQL for web app. ;-) I'd very much prefer to use SQLite as the data we use are text of the different length which is more suitable for SQLite. But Market trend is deceptive AND meaningless. Yes, there's a general notion that low-grade web servers will give you Apache, MySQL, and PHP, while not offering anything else; but that's like saying that a dirt-cheap car comes with a radio and no CD player, ergo you should be listening to the radio even in a car that has (or could have) a CD player. The reason cheap web hosts offer MySQL is because lots of people ask for it, and the reason people ask for it is because lots of hosts offer it. There's no other reason. If you're the expert and he's of management background, then it's your job to sell the technically superior option despite his initial gut feeling... be an advocate, not an automaton. :) And hopefully he'll respect your skills enough to listen to your advice. I don't have much experience with SQLite, but I would suspect that text of different lengths should be able to be handled equally cleanly by any database engine - at least the major ones. You may want to look into whether you use VARCHAR, TEXT, or some other data type, but it should work just fine. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Enable unicode
Igor Korot ikoro...@gmail.com writes: On Tue, Jan 28, 2014 at 1:00 AM, Chris Angelico ros...@gmail.com wrote: Hi! I'm hoping it was oversight that led to this email coming to me personally instead of to the list, and hoping that you won't mind me responding on-list. Sorry about that. I keep forgetting that gmail web mail is stupid and does not use the list address on Reply. The “reply” command is for replying only to the article author So that is working correctly. Does not happen to other list, only this one. Many mailing lists are misconfigured, breaking the “reply” command so it doesn't do what it's meant to (and you lose the facility to reply only to the author of the message). Instead, for replying to a list, you need the “reply to list” command URL:http://david.woodhou.se/reply-to-list.html. Sadly, it appears GMail does not have this command; you may want to use a better mail client which implements this feature. -- \ “I must say that I find television very educational. The minute | `\ somebody turns it on, I go to the library and read a book.” | _o__)—Groucho Marx | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Enable unicode
On Tuesday, January 28, 2014 2:50:20 PM UTC+5:30, Igor Korot wrote: Hi, Chris, On Tue, Jan 28, 2014 at 1:00 AM, Chris Angelico wrote: On Tue, Jan 28, 2014 at 7:56 PM, Igor Korot wrote: Hi, Chris, Hi! I'm hoping it was oversight that led to this email coming to me personally instead of to the list, and hoping that you won't mind me responding on-list. Sorry about that. I keep forgetting that gmail web mail is stupid and does not use the list address on Reply. Does not happen to other list, only this one. In gmail: Gear-icon - Settings - Default-reply behavior select all -- https://mail.python.org/mailman/listinfo/python-list
Re: Enable unicode
On Wed, Jan 29, 2014 at 3:06 AM, Rustom Mody rustompm...@gmail.com wrote: Sorry about that. I keep forgetting that gmail web mail is stupid and does not use the list address on Reply. Does not happen to other list, only this one. In gmail: Gear-icon - Settings - Default-reply behavior select all That's an imperfect solution, but it's the one I use. You then have to remember to delete the individual sender's address, or that person will get a duplicate copy. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Enable unicode
On 29Jan2014 02:47, Ben Finney ben+pyt...@benfinney.id.au wrote: Igor Korot ikoro...@gmail.com writes: On Tue, Jan 28, 2014 at 1:00 AM, Chris Angelico ros...@gmail.com wrote: Hi! I'm hoping it was oversight that led to this email coming to me personally instead of to the list, and hoping that you won't mind me responding on-list. Sorry about that. I keep forgetting that gmail web mail is stupid and does not use the list address on Reply. [...] Instead, for replying to a list, you need the “reply to list” command URL:http://david.woodhou.se/reply-to-list.html. Sadly, it appears GMail does not have this command; you may want to use a better mail client which implements this feature. For example, choose and use a real desktop mail client and connect to GMail's IMAP interface. At least then you can pick something with the feature set you want. -- Cameron Simpson c...@zip.com.au -- https://mail.python.org/mailman/listinfo/python-list
