Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Dave Angel
On 02/12/2013 07:20 PM, Magnus Pettersson wrote: You don't show the code that actually does the io.open(), nor the url.encode, so I'm not going to guess what you're actually doing. Hmm im not sure what you mean but I wrote all code needed in a previous post so maybe you missed that one :) I

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
Thanks a lot Steven, you gave me a good AHA experience! :) Now I understand why I had to use encoding when calling the urllib2! So basically Eclipse PyDev does this in the background for me, and its console supports utf-8, so thats why i never had to think about it before (and why some scripts

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
> You don't show the code that actually does the io.open(), nor the > > url.encode, so I'm not going to guess what you're actually doing. Hmm im not sure what you mean but I wrote all code needed in a previous post so maybe you missed that one :) In short I basically just have: import io io.op

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Steven D'Aprano
Magnus Pettersson wrote: > # This made the fetching of the website work. Why did i have to write > # url.encode("UTF-8") when url already is unicode? I feel i dont have a > # good understanding of this. > page = urllib2.urlopen(url.encode("UTF-8")) Start here: "The Absolute Minimum Every Softw

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread MRAB
On 2013-02-12 14:24, Magnus Pettersson wrote: I have tried now to take away printing to terminal and just keeping the writing to a .txt file to disk (which is what the scripts purpose is): with open(filepath,"a") as f: for card in cardlist: f.write(card+"\n") The file it writes t

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Dave Angel
On 02/12/2013 12:12 PM, Magnus Pettersson wrote: < snip > #Here kanji = u"私" baseurl = u"http://www.romajidesu.com/kanji/"; url = baseurl+kanji savefile([url]) #this test works now. uses: io.open(filepath, "a",encoding="UTF-8") as f: # This made the fetching of the website work. You don't sh

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Fabio Zadrozny
Just to note, PyDev does something behind the scenes (it sets the encoding for the console). You may specify which encoding you want at your launch configuration (in the 'common' tab you can set the encoding you want for the shell). Cheers, Fabio On Tue, Feb 12, 2013 at 3:12 PM, Magnus Petters

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
> What encoding is this file? Since you're appending to it, you really > > need to match the pre-existing encoding, or the next program to deal > > with it is in big trouble. So using the io.open() without the encoding= > > keyword is probably a mistake. The .txt file is in UTF-8 I have g

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Terry Reedy
On 2/12/2013 7:34 AM, Magnus Pettersson wrote: Ahh so its the actual printing that makes it error out outside of eclipse because its a different terminal that its printing to. Its the default DOS terminal in windows that runs then i run the script with python.exe and i guess its the same when i r

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Dave Angel
On 02/12/2013 10:29 AM, Magnus Pettersson wrote: Are you sure you are writing the same data? That would mean that pydev changes the default encoding -- which is evil. A portable approach would be to use codecs.open() or io.open() instead of the built-in: import io with io.open(filepath,

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Peter Otten
Magnus Pettersson wrote: >> io.open() uses UTF-8 by default, but you can specify other encodings with >> >> io.open(filepath, mode, encoding=whatever). > > > Interesting. Pydev must be doing something behind the scenes because when > i changed open() to io.open() i get error inside of eclipse n

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
> Are you sure you are writing the same data? That would mean that pydev > > changes the default encoding -- which is evil. > > > > A portable approach would be to use codecs.open() or io.open() instead of > > the built-in: > > > > import io > > with io.open(filepath, "a") as f: > >

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Peter Otten
Magnus Pettersson wrote: > I have tried now to take away printing to terminal and just keeping the > writing to a .txt file to disk (which is what the scripts purpose is): > > with open(filepath,"a") as f: > for card in cardlist: > f.write(card+"\n") > > The file it writes to exists

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
I have tried now to take away printing to terminal and just keeping the writing to a .txt file to disk (which is what the scripts purpose is): with open(filepath,"a") as f: for card in cardlist: f.write(card+"\n") The file it writes to exists and im just appending to it, but when i r

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
Ahh so its the actual printing that makes it error out outside of eclipse because its a different terminal that its printing to. Its the default DOS terminal in windows that runs then i run the script with python.exe and i guess its the same when i run with pythonw.exe just that the terminal win

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Steven D'Aprano
Magnus Pettersson wrote: > I am using Eclipse to write my python scripts and when i run them from > inside eclipse they work fine without errors. > > But almost in every script that handle some form of special characters > like swedish åäö and chinese characters etc A comment: they are not "spec

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Andrew Berg
On 2013.02.12 04:43, Magnus Pettersson wrote: > I am using Eclipse to write my python scripts and when i run them from inside > eclipse they work fine without errors. > > But almost in every script that handle some form of special characters like > swedish åäö and chinese characters etc i get U

UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
I am using Eclipse to write my python scripts and when i run them from inside eclipse they work fine without errors. But almost in every script that handle some form of special characters like swedish åäö and chinese characters etc i get Unicode errors when running the script externally with p