Re: Problem writing some strings (UnicodeEncodeError)
Paulo da Silva wrote:
Em 12-01-2014 20:29, Peter Otten escreveu:
Paulo da Silva wrote:
but I have not tried it myself. Also, some bytes may need to be
escaped, either to be understood by the shell, or to address security
concerns:
Since I am puting the file names between , the only char that needs to
be escaped is the itself.
What about the escape char?
Just this fn=fn.replace('','\\')
So far I didn't find any problem, but the script is still running.
To be a bit more explicit:
for filename in os.listdir():
... print(template.replace(fn, filename.replace('', '\\')))
...
ls \\; rm whatever; ls \
--
https://mail.python.org/mailman/listinfo/python-list
Re: Problem writing some strings (UnicodeEncodeError)
Peter Otten wrote:
Paulo da Silva wrote:
Em 12-01-2014 20:29, Peter Otten escreveu:
Paulo da Silva wrote:
but I have not tried it myself. Also, some bytes may need to be
escaped, either to be understood by the shell, or to address security
concerns:
Since I am puting the file names between , the only char that needs
to be escaped is the itself.
What about the escape char?
Just this fn=fn.replace('','\\')
So far I didn't find any problem, but the script is still running.
To be a bit more explicit:
for filename in os.listdir():
... print(template.replace(fn, filename.replace('', '\\')))
...
ls \\; rm whatever; ls \
The complete session:
import os
template = 'ls fn'
with open('\\; rm whatever; ls \\', w) as f: pass
...
for filename in os.listdir():
... print(template.replace(fn, filename.replace('', '\\')))
...
ls \\; rm whatever; ls \
Shell variable substitution is another problem. c.l.py is probably not the
best place to get the complete list of possibilities.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Problem writing some strings (UnicodeEncodeError)
Em 13-01-2014 08:58, Peter Otten escreveu:
Peter Otten wrote:
Paulo da Silva wrote:
Em 12-01-2014 20:29, Peter Otten escreveu:
Paulo da Silva wrote:
but I have not tried it myself. Also, some bytes may need to be
escaped, either to be understood by the shell, or to address security
concerns:
Since I am puting the file names between , the only char that needs
to be escaped is the itself.
What about the escape char?
Just this fn=fn.replace('','\\')
So far I didn't find any problem, but the script is still running.
To be a bit more explicit:
for filename in os.listdir():
... print(template.replace(fn, filename.replace('', '\\')))
...
ls \\; rm whatever; ls \
The complete session:
import os
template = 'ls fn'
with open('\\; rm whatever; ls \\', w) as f: pass
...
for filename in os.listdir():
... print(template.replace(fn, filename.replace('', '\\')))
...
ls \\; rm whatever; ls \
Shell variable substitution is another problem. c.l.py is probably not the
best place to get the complete list of possibilities.
I see what you mean.
This is a tedious problem. Don't know if there is a simple solution in
python for this. I have to think about it ...
On a more general and serious application I would not produce a bash
script. I would do all the work in python.
That's not the case, however. This is a few times execution script for a
very special purpose. The only problem was the occurrence of some
Portuguese characters in old filenames encoded in another code than
utf-8. Very few also include the .
The worst thing that could happen was the bash script to abort. Then it
would be easy to fix it using a simple editor.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Problem writing some strings (UnicodeEncodeError)
Paulo da Silva wrote:
Em 13-01-2014 08:58, Peter Otten escreveu:
Peter Otten wrote:
Paulo da Silva wrote:
Em 12-01-2014 20:29, Peter Otten escreveu:
Paulo da Silva wrote:
but I have not tried it myself. Also, some bytes may need to be
escaped, either to be understood by the shell, or to address
security concerns:
Since I am puting the file names between , the only char that needs
to be escaped is the itself.
What about the escape char?
Just this fn=fn.replace('','\\')
So far I didn't find any problem, but the script is still running.
To be a bit more explicit:
for filename in os.listdir():
... print(template.replace(fn, filename.replace('', '\\')))
...
ls \\; rm whatever; ls \
The complete session:
import os
template = 'ls fn'
with open('\\; rm whatever; ls \\', w) as f: pass
...
for filename in os.listdir():
... print(template.replace(fn, filename.replace('', '\\')))
...
ls \\; rm whatever; ls \
Shell variable substitution is another problem. c.l.py is probably not
the best place to get the complete list of possibilities.
I see what you mean.
This is a tedious problem. Don't know if there is a simple solution in
python for this. I have to think about it ...
On a more general and serious application I would not produce a bash
script. I would do all the work in python.
That's not the case, however. This is a few times execution script for a
very special purpose. The only problem was the occurrence of some
Portuguese characters in old filenames encoded in another code than
utf-8. Very few also include the .
The worst thing that could happen was the bash script to abort. Then it
would be easy to fix it using a simple editor.
I looked around in the stdlib and found shlex.quote(). It uses ' instead of
which simplifies things, and special-cases only ':
print(shlex.quote(alpha'beta))
'alpha'''beta'
So the answer is simpler than I had expected.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Problem writing some strings (UnicodeEncodeError)
Em 13-01-2014 17:29, Peter Otten escreveu: Paulo da Silva wrote: Em 13-01-2014 08:58, Peter Otten escreveu: I looked around in the stdlib and found shlex.quote(). It uses ' instead of which simplifies things, and special-cases only ': print(shlex.quote(alpha'beta)) 'alpha'''beta' So the answer is simpler than I had expected. Yes, it should work, at least in this case. Although python oriented, it seems to work to bash also. I need to remove the from the templates and use shlex.quote for filenames. I'll give it a try. Thanks -- https://mail.python.org/mailman/listinfo/python-list
Problem writing some strings (UnicodeEncodeError)
Hi! I am using a python3 script to produce a bash script from lots of filenames got using os.walk. I have a template string for each bash command in which I replace a special string with the filename and then write the command to the bash script file. Something like this: shf=open(bashfilename,'w') filenames=getfilenames() # uses os.walk for fn in filenames: ... cmd=templ.replace(fn,fn) shf.write(cmd) For certain filenames I got a UnicodeEncodeError exception at shf.write(cmd)! I use utf-8 and have # -*- coding: utf-8 -*- in the source .py. How can I fix this? Thanks for any help/comments. -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem writing some strings (UnicodeEncodeError)
On Sun, 1/12/14, Paulo da Silva p_s_d_a_s_i_l_...@netcabo.pt wrote: Subject: Problem writing some strings (UnicodeEncodeError) To: python-list@python.org Date: Sunday, January 12, 2014, 4:36 PM Hi! I am using a python3 script to produce a bash script from lots of filenames got using os.walk. I have a template string for each bash command in which I replace a special string with the filename and then write the command to the bash script file. Something like this: shf=open(bashfilename,'w') filenames=getfilenames() # uses os.walk for fn in filenames: ... cmd=templ.replace(fn,fn) shf.write(cmd) For certain filenames I got a UnicodeEncodeError exception at shf.write(cmd)! I use utf-8 and have # -*- coding: utf-8 -*- in the source .py. How can I fix this? Thanks for any help/comments. == what is the output of locale.getpreferredencoding(False)? That is the default value of the encoding parameter of the open function. shf=open(bashfilename,'w', encoding='utf-8') might work, though on my Linux macine locale.getpreferredencoding(False) returns utf-8. help(open) ... In text mode, if encoding is not specified the encoding used is platform dependent: locale.getpreferredencoding(False) is called to get the current locale encoding. (For reading and writing raw bytes use binary mode and leave encoding unspecified.) ... -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem writing some strings (UnicodeEncodeError)
Paulo da Silva wrote: I am using a python3 script to produce a bash script from lots of filenames got using os.walk. I have a template string for each bash command in which I replace a special string with the filename and then write the command to the bash script file. Something like this: shf=open(bashfilename,'w') filenames=getfilenames() # uses os.walk for fn in filenames: ... cmd=templ.replace(fn,fn) shf.write(cmd) For certain filenames I got a UnicodeEncodeError exception at shf.write(cmd)! I use utf-8 and have # -*- coding: utf-8 -*- in the source .py. How can I fix this? Thanks for any help/comments. You make it harder to debug your problem by not giving the complete traceback. If the error message contains 'surrogates not allowed' like in the demo below with open(tmp.txt, w) as f: ... f.write(\udcef) ... Traceback (most recent call last): File stdin, line 2, in module UnicodeEncodeError: 'utf-8' codec can't encode character '\udcef' in position 0: surrogates not allowed you have filenames that are not valid UTF-8 on your harddisk. A possible fix would be to use bytes instead of str. For that you need to open `bashfilename` in binary mode (wb) and pass bytes to the os.walk() call. Or you just go and fix the offending names. -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem writing some strings (UnicodeEncodeError)
On 01/12/2014 07:36 AM, Paulo da Silva wrote: Hi! I am using a python3 script to produce a bash script from lots of filenames got using os.walk. I have a template string for each bash command in which I replace a special string with the filename and then write the command to the bash script file. Something like this: shf=open(bashfilename,'w') filenames=getfilenames() # uses os.walk for fn in filenames: ... cmd=templ.replace(fn,fn) shf.write(cmd) For certain filenames I got a UnicodeEncodeError exception at shf.write(cmd)! I use utf-8 and have # -*- coding: utf-8 -*- in the source .py. How can I fix this? Not sure exactly, but I'd try shf=open(bashfilename,'wb') as a start. HTH, Emile -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem writing some strings (UnicodeEncodeError)
Em 12-01-2014 16:23, Peter Otten escreveu: Paulo da Silva wrote: I am using a python3 script to produce a bash script from lots of filenames got using os.walk. I have a template string for each bash command in which I replace a special string with the filename and then write the command to the bash script file. Something like this: shf=open(bashfilename,'w') filenames=getfilenames() # uses os.walk for fn in filenames: ... cmd=templ.replace(fn,fn) shf.write(cmd) For certain filenames I got a UnicodeEncodeError exception at shf.write(cmd)! I use utf-8 and have # -*- coding: utf-8 -*- in the source .py. How can I fix this? Thanks for any help/comments. You make it harder to debug your problem by not giving the complete traceback. If the error message contains 'surrogates not allowed' like in the demo below with open(tmp.txt, w) as f: ... f.write(\udcef) ... Traceback (most recent call last): File stdin, line 2, in module UnicodeEncodeError: 'utf-8' codec can't encode character '\udcef' in position 0: surrogates not allowed That is the situation. I just lost it and it would take a few houres to repeat the situation. Sorry. you have filenames that are not valid UTF-8 on your harddisk. A possible fix would be to use bytes instead of str. For that you need to open `bashfilename` in binary mode (wb) and pass bytes to the os.walk() call. This is my 1st time with python3, so I am confused! As much I could understand it seems that os.walk is returning the filenames exactly as they are on disk. Just bytes like in C. My template is a string. What is the result of the replace command? Is there any change in the filename from os.walk contents? Now, if the result of the replace has the replaced filename unchanged how do I convert it to bytes type, without changing its contents, so that I can write to the bashfile opened with wb? Or you just go and fix the offending names. This is impossible in my case. I need a bash script with the names as they are on disk. -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem writing some strings (UnicodeEncodeError)
Paulo da Silva wrote: Em 12-01-2014 16:23, Peter Otten escreveu: Paulo da Silva wrote: I am using a python3 script to produce a bash script from lots of filenames got using os.walk. I have a template string for each bash command in which I replace a special string with the filename and then write the command to the bash script file. Something like this: shf=open(bashfilename,'w') filenames=getfilenames() # uses os.walk for fn in filenames: ... cmd=templ.replace(fn,fn) shf.write(cmd) For certain filenames I got a UnicodeEncodeError exception at shf.write(cmd)! I use utf-8 and have # -*- coding: utf-8 -*- in the source .py. How can I fix this? Thanks for any help/comments. You make it harder to debug your problem by not giving the complete traceback. If the error message contains 'surrogates not allowed' like in the demo below with open(tmp.txt, w) as f: ... f.write(\udcef) ... Traceback (most recent call last): File stdin, line 2, in module UnicodeEncodeError: 'utf-8' codec can't encode character '\udcef' in position 0: surrogates not allowed That is the situation. I just lost it and it would take a few houres to repeat the situation. Sorry. you have filenames that are not valid UTF-8 on your harddisk. A possible fix would be to use bytes instead of str. For that you need to open `bashfilename` in binary mode (wb) and pass bytes to the os.walk() call. This is my 1st time with python3, so I am confused! As much I could understand it seems that os.walk is returning the filenames exactly as they are on disk. Just bytes like in C. No, they are decoded with the preferred encoding. With UTF-8 that can fail, and if it does the surrogateescape error handler replaces the offending bytes with special codepoints: import os with open(b\xe4\xf6\xfc, w) as f: f.write(whatever) ... 8 os.listdir() ['\udce4\udcf6\udcfc'] You can bypass the decoding process by providing a bytes argument to os.listdir() (or os.walk() which uses os.listdir() internally): os.listdir(b.) [b'\xe4\xf6\xfc'] To write these raw bytes into a file the file has of course to be binary, too. My template is a string. What is the result of the replace command? Is there any change in the filename from os.walk contents? Now, if the result of the replace has the replaced filename unchanged how do I convert it to bytes type, without changing its contents, so that I can write to the bashfile opened with wb? Or you just go and fix the offending names. This is impossible in my case. I need a bash script with the names as they are on disk. I think instead of the hard way sketched out above it will be sufficient to specify the error handler when opening the destination file shf = open(bashfilename, 'w', errors=surrogateescape) but I have not tried it myself. Also, some bytes may need to be escaped, either to be understood by the shell, or to address security concerns: import os template = ls fn for filename in os.listdir(): ... print(template.replace(fn, filename)) ... ls foo; rm bar -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem writing some strings (UnicodeEncodeError)
I think instead of the hard way sketched out above it will be sufficient to specify the error handler when opening the destination file shf = open(bashfilename, 'w', errors=surrogateescape) This seems to fix everything! I tried with a small test set and it worked. but I have not tried it myself. Also, some bytes may need to be escaped, either to be understood by the shell, or to address security concerns: Since I am puting the file names between , the only char that needs to be escaped is the itself. I'm gonna try with the real thing. Thank you very much for the fixing and for everything I have learned here. -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem writing some strings (UnicodeEncodeError)
Paulo da Silva wrote: but I have not tried it myself. Also, some bytes may need to be escaped, either to be understood by the shell, or to address security concerns: Since I am puting the file names between , the only char that needs to be escaped is the itself. What about the escape char? -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem writing some strings (UnicodeEncodeError)
Em 12-01-2014 20:29, Peter Otten escreveu:
Paulo da Silva wrote:
but I have not tried it myself. Also, some bytes may need to be escaped,
either to be understood by the shell, or to address security concerns:
Since I am puting the file names between , the only char that needs to
be escaped is the itself.
What about the escape char?
Just this fn=fn.replace('','\\')
So far I didn't find any problem, but the script is still running.
--
https://mail.python.org/mailman/listinfo/python-list
