On Jul 25, 8:50 pm, Dave Angel <d...@davea.name> wrote: > On 07/25/2012 08:09 AM, jaroslav.dob...@gmail.com wrote: > > > > > > > > > > > On Wednesday, July 25, 2012 1:35:09 PM UTC+2, Philipp Hagemeister wrote: > >> Hi Jaroslav, > > >> you can catch a UnicodeDecodeError just like any other exception. Can > >> you provide a full example program that shows your problem? > > >> This works fine on my system: > > >> import sys > >> open('tmp', 'wb').write(b'\xff\xff') > >> try: > >> buf = open('tmp', 'rb').read() > >> buf.decode('utf-8') > >> except UnicodeDecodeError as ude: > >> sys.exit("Found a bad char in file " + "tmp") > > > Thank you. I got it. What I need to do is explicitly decode text. > > > But I think trial and error with moving files around will in most cases be > > faster. Usually, such a problem occurs with some (usually complex) program > > that I wrote quite a long time ago. I don't like editing old and complex > > programs that work under all normal circumstances. > > > What I am missing (especially for Python3) is something like: > > > try: > > for line in sys.stdin: > > except UnicodeDecodeError: > > sys.exit("Encoding problem in line " + str(line_number)) > > > I got the point that there is no such thing as encoding-independent lines. > > But if no line ending can be found, then the file simply has one single > > line. > > i can't understand your question. if the problem is that the system > doesn't magically produce a variable called line_number, then generate > it yourself, by counting > in the loop.
That was just a very incomplete and general example. My problem is solved. What I need to do is explicitly decode text when reading it. Then I can catch exceptions. I might do this in future programs. I dislike about this solution that it complicates most programs unnecessarily. In programs that open, read and process many files I don't want to explicitly decode and encode characters all the time. I just want to write: for line in f: or something like that. Yet, writing this means to *implicitly* decode text. And, because the decoding is implicit, you cannot say try: for line in f: # here text is decoded implicitly do_something() except UnicodeDecodeError(): do_something_different() This isn't possible for syntactic reasons. The problem is that vast majority of the thousands of files that I process are correctly encoded. But then, suddenly, there is a bad character in a new file. (This is so because most files today are generated by people who don't know that there is such a thing as encodings.) And then I need to rewrite my very complex program just because of one single character in one single file. -- http://mail.python.org/mailman/listinfo/python-list