[issue9291] mimetypes initialization fails on Windows because of non-Latin characters in registry

[adamhj]

adamhj added the comment:

 The encoding is wrong. We should read the registry using Unicode, or at least 
 use the correct encoding. The correct encoding is the ANSI code page: 
 sys.getfilesystemencoding().

 Can you please try with: default_encoding = sys.getfilesystemencoding() ?

This does not work. In fact it doesn't matter what default_encoding is. The 
variable ctype, which is returned by _winreg.EnumKey(), is a byte 
string(b'blahblah'), at least on my computer(win2k3sp2, python 2.7.6). Because 
the interpreter is asked to encode a byte string, it tries to convert the byte 
string to unicode string first, by calling decode implicitly with 'ascii' 
encoding, so the exception UnicodeDecodeError.

the variable ctype, which is read from registry key name, can be decoded 
correctly with sys.getfilesystemencoding()(which returns 'mbcs'), but in fact 
what we need is a byte string, so there should be neither encoding nor decoding 
here.

if there is a case that _winreg.EnumKey() returns unicode string, then a type 
check should be added before the encode. Or maybe the case is that the return 
type of _winreg.EnumKey() is different in 2.x and 3.x?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9291
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19567] mimetypes.init() raise unhandled excption in windows

[adamhj]

New submission from adamhj:

my system is windows 2k3 sp2, python version is 2.7.6

i found this bug when trying to install the newest setuptools


X:\xxxez_setup.py
Extracting in d:\docume~1\xxx\locals~1\temp\tmpcyxs8s
Now working in d:\docume~1\xxx\locals~1\temp\tmpcyxs8s\setuptools-1.3.2
Installing Setuptools
Traceback (most recent call last):
  File setup.py, line 17, in module
exec(init_file.read(), command_ns)
  File string, line 8, in module
  File 
d:\docume~1\xxx\locals~1\temp\tmpcyxs8s\setuptools-1.3.2\setuptools\__init__.py,
 line 11, in module
from setuptools.extension import Extension
  File 
d:\docume~1\xxx\locals~1\temp\tmpcyxs8s\setuptools-1.3.2\setuptools\extension.py,
 line 5, in module
from setuptools.dist import _get_unpatched
  File 
d:\docume~1\xxx\locals~1\temp\tmpcyxs8s\setuptools-1.3.2\setuptools\dist.py, 
line 15, in module
from setuptools.compat import numeric_types, basestring
  File 
d:\docume~1\xxx\locals~1\temp\tmpcyxs8s\setuptools-1.3.2\setuptools\compat.py,
 line 19, in module
from SimpleHTTPServer import SimpleHTTPRequestHandler
  File D:\Python27\lib\SimpleHTTPServer.py, line 27, in module
class SimpleHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
  File D:\Python27\lib\SimpleHTTPServer.py, line 208, in 
SimpleHTTPRequestHandler
mimetypes.init() # try to read system mime.types
  File D:\Python27\lib\mimetypes.py, line 358, in init
db.read_windows_registry()
  File D:\Python27\lib\mimetypes.py, line 258, in read_windows_registry
for subkeyname in enum_types(hkcr):
  File D:\Python27\lib\mimetypes.py, line 249, in enum_types
ctype = ctype.encode(default_encoding) # omit in 3.x!
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 33: 
ordinal not in range(128)
Something went wrong during the installation.
See the error message above.


then i see into the code, the exception is raised in this function


def enum_types(mimedb):
i = 0
while True:
try:
ctype = _winreg.EnumKey(mimedb, i)
except EnvironmentError:
break
try:
ctype = ctype.encode(default_encoding) # omit in 3.x!
except UnicodeEncodeError:
pass
else:
yield ctype
i += 1


i checked my registry, there is an key in HKCR whose name is in 
Chinese(encoding GBK), which causes this problem(btw, the default_encoding is 
'ascii', assigned from sys.getdefaultencoding())

i don't know is it legal to use a non-ascii string as the name of a registry 
key in windows, but i think there is some problem in these piece of code. why 
the variable ctype need to be decoded here? i checked the _winreg.EnumKey() 
function, it returns byte string:

 _winreg.EnumKey(key,8911)
'\xbb\xad\xb0\xe5\xa1\xa3\xce\xc4\xb5\xb5'
#this is the problem key which cause the exception

so python tries to encode it(with ascii encoding) first, and then the exception 
is raised(and unhandled)

shouldn't we just remove the try..encode..except paragraph?

--
components: Library (Lib)
messages: 202731
nosy: adamhj
priority: normal
severity: normal
status: open
title: mimetypes.init() raise unhandled excption in windows
type: behavior
versions: Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19567
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13928] bug in asyncore.dispatcher_with_send

[adamhj]

adamhj ada...@gmail.com added the comment:

 A non connected socket must be writable in order to connect.
i can't understand this, does it means that one may use self.connect() in 
handle_write()? and in fact i found something seems opposite on this page: 
http://docs.python.org/howto/sockets.html

If a socket is in the output readable list, you can be 
as-close-to-certain-as-we-ever-get-in-this-business that a recv on that socket 
will return something. Same idea for the writable list. You’ll be able to send 
something.

If you have created a new socket to connect to someone else, put it in the 
potential_writers list. If it shows up in the writable list, you have a decent 
chance that it has connected.

from the latter paragraph may i assume that a writable socket should always has 
been connected?

 Not sure what you mean here. Why would you call connect() twice?
sorry for the typo, it should be if we call dispatcher.send() immediately 
after .connect(), socket error 10057 may be raised, this happens if you 
connect to a high delay remote port, when the .send() is called before the SYN 
ACK is received. 

i think it maybe the programmer's responsibility to check the connection 
availability when using dispatcher class, but at least for dispatcher_with_send 
class, programmer should not need do anything special to use send() after a 
successful connect(), if .send() is called before connection established 
successfully, send() should only buffer the sending data and wait the socket to 
become writable

by the way, i found this behavior on windows 7 x32/python 2.7.2

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13928
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13928] bug in asyncore.dispatcher_with_send

[adamhj]

adamhj ada...@gmail.com added the comment:

here is a script emulating what is happened in dispatcher class when the second 
bug triggered. you can use either a non-exist host/port, or a high delay remote 
port as target(see the comments in the script) and you may use a 
sniffer(tcpdump/wireshark e.g.) with the script to see what is happen

--
Added file: http://bugs.python.org/file24417/bug-emu.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13928
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13928] bug in asyncore.dispatcher_with_send

[adamhj]

adamhj ada...@gmail.com added the comment:

ah just ignore my previous msg as i post it without seeing yours.

 Nope. It means that *before* showing up the socket was *not* connected.
ok, i read more in the asyncore source and finally understand what do you mean 
by A non connected socket must be writable in order to connect. i think this 
is a little confusing if without any explaining, shouldn't this be written into 
the asyncore reference page as both readable() and writable() may be override 
by user?

 Of course it does: you're not connected yet (10057 = WSAENOTCONN). 
 You're supposed to use send() in handle_connect(), when the connection has 
 already been established.
 This:

 self.connect()
 self.send('hello')

 ...is not asyncore is supposed to be used.

oh it seems i forgot handle_connect(), thank you for your patient explanation 
and sorry to have troubled you. i think there is no problem anymore

--
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13928
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13928] bug in asyncore.dispatcher_with_send

[adamhj]

New submission from adamhj ada...@gmail.com:

i found 2 relative bugs in asyncore.dispatcher_with_send class:

one is in the asyncore.dispatcher_with_send.writable():
def writable(self):
return (not self.connected) or len(self.out_buffer)
why is a not connected connection writable? i think this is definitely a bug
my fix:
def writable(self):
return self.connected and len(self.out_buffer)

another bug is more obscure, i'm not sure is it a bug or something should be 
handled by user(programmer)

the bug is also in asyncore.dispatcher_with_send class, and maybe also in 
asyncore.dispatcher class. asyncore.dispatcher uses unblocking socket to handle 
network missions, when we use the connect() method of dispatcher to establish 
the socket, it will call socket.connect_ex() method to create the connection, 
however, socket.connect_ex() may return 10035(EWOULDBLOCK) as it is an 
unblocking socket indicates that the connection creating is not finished yet, 
if we call dispatcher.connect() immediately after .connect(), socket error 
10057 may be raised, indicating that the socket is not established yet, then 
the asyncore main loop catches this exception, and calls handle_error(in my 
case i close the connection in handle_error so the connection which would be 
established with no problem breaks), i think there should be a connection state 
check in asyncore.dispatcher.send(), or at least in 
asyncore.dispatcher_with_send.send.

my fix for asyncore.dispatcher_with_send.send():
def send(self, data):
if self.debug:
self.log_info('sending %s' % repr(data))
self.out_buffer = self.out_buffer + data
if self.connected:# do not call send() if connection is
self.initiate_send()  # not established yet, just put data 
  # in buffer

for the second bug, to reproduce it, just create a unblocking socket to a 
remote, long delay port with socket.connect_ex and call send immediately

--
components: Library (Lib)
messages: 152494
nosy: adamhj
priority: normal
severity: normal
status: open
title: bug in asyncore.dispatcher_with_send
type: behavior
versions: Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13928
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com