[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Steve Dower]

Steve Dower added the comment:

I just created issue27603 for any work that goes into adapting the IDLE context 
menu. Pretty good chance I'll get to look at it for 3.6.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Terry J. Reedy]

Terry J. Reedy added the comment:

Having "Edit in IDLE' work uniformly for all installed versions would be great.

> set the working directory properly

This needs some discussion and coordination.  I am thinking of patching IDLE to 
switch to $HOME when cwd is the python install dir, but this would very likely 
only be done for 3.6 as part of re-writing the startup code.

> and any environment variables that may be needed/helpful.
Can't think of anything at the moment, but this has never been an option.

> treating Shift+Click differently from a regular click, if that was any value 
> (run in IDLE and leave the interactive session open, perhaps?).

IDLE's "-r file.py" runs the file and leaves Shell open.  I believe this is 
meant as a substitute for setting $IDLESTARTUP or $PYTHONSTARTUP and using -s.  
For development, one should open in the editor and hit f5.   There might be a 
controversy whether to run in IDLE or 'python -i'.  If control or alt click 
could be distinguished, both would be possible.

Are you going to open issues, or should I?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Steve Dower]

Steve Dower added the comment:

The "Edit in IDLE" submenu we implemented could be moved from a collection of 
registry keys into the shell extension, which would also let us detect all 
Python installations, as right now we only have 3.5 and later (we could even 
filter out those that are missing IDLE for whatever reason). We'd also be able 
to fully control launching it, so we could set the working directory properly 
and any environment variables that may be needed/helpful.

I believe we can also do other tricks like treating Shift+Click differently 
from a regular click, if that was any value (run in IDLE and leave the 
interactive session open, perhaps?).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Terry J. Reedy]

Terry J. Reedy added the comment:

> Certainly some of the requests for IDLE could be accommodated).

Steve, what did you have in mind here?

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Steve Dower]

Steve Dower added the comment:

Fixed. This should get out into the wild first with 3.6.0a4, and then 3.5.3[rc1]

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Roundup Robot]

Roundup Robot added the comment:

New changeset 3005fc6cff8a by Steve Dower in branch '3.5':
Issue #27469: Adds a shell extension to the launcher so that drag and drop 
works correctly.
https://hg.python.org/cpython/rev/3005fc6cff8a

New changeset f56adb9800fa by Steve Dower in branch 'default':
Issue #27469: Adds a shell extension to the launcher so that drag and drop 
works correctly.
https://hg.python.org/cpython/rev/f56adb9800fa

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[STINNER Victor]

Changes by STINNER Victor :


--
nosy:  -haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Decorater]

Decorater added the comment:

I actually like py2exe's memory loader.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Steve Dower]

Steve Dower added the comment:

Sure, added a new patch with all of your suggestions. I also added 
pyshellext.vcxproj to pcbuild.sln.

(And my final word on the magic used to load DLLs from memory is that I'm not 
willing to maintain that code within CPython itself, so it won't be going into 
the core unless someone steps up with a 10+ year promise to look after it.)

--
stage:  -> patch review
Added file: http://bugs.python.org/file43771/27469_2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Eryk Sun]

Eryk Sun added the comment:

Steve, will you be uploading a new patch? The current patch doesn't include 
"pyshellext.vcxproj" in the build, since that was accidentally committed and 
then removed. 

When you call ShellExecute, I suggest passing NULL for lpOperation, to use the 
default verb. If no default is defined the shell uses "open" anyway.

Decorater, there's a memory importer for extension modules in py2exe, based on 
Joachim Bauch's MemoryModule.c:

https://sourceforge.net/p/py2exe/svn/HEAD/tree/trunk/py2exe-3/source

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Steve Dower]

Steve Dower added the comment:

A custom icon for .pyd files doesn't require a shell extension - it only really 
requires the icon.

I don't want to use the same as .pyc, since that has the implication that it 
can be deleted safely, or .py, since that can be double-clicked. So we'd need 
something new.

The rest of the request requires exposing new public APIs in Windows so that we 
can load executables from arbitrary memory locations. That's not something I 
can fix here :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Decorater]

Decorater added the comment:

Oh and when the shell extension is done could you make a custom icon for pyd's 
and have the desciption of "Python C Compiled DLL" to easy selcting them when 
you have no icons for any *.lib, *.exp and *.pdb's Would ease up comping pyd 
files as well.

On further note it seems that python (when told to read from python36.zip seems 
to not read any pyd's that are in the zip as well) I would like for it to be 
able too if someone wants to embed PyNacl with _sodium.cp36-win32.pyd or 
_sodium.cp36-win_amd64.pyd files in it.

(Also readign pyd's in the zips could also simplify things like a ImportError 
on stuff that is actually there.

--
nosy: +Decorater

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Steve Dower]

Steve Dower added the comment:

Thanks, I've merged that change into my patch.

I doubt anyone else is going to build and test this, but just in case I'll let 
it set for a couple of days before merging. Perhaps someone will at least look 
at the code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Eryk Sun]

Eryk Sun added the comment:

Thanks, Steve. I manually added this shell extension as the drop handler for 
Python.File. It's working with non-ANSI filenames, e.g. "αβψδ εφγη ιξκλ μνοπ 
ρστθ ωχυζ.txt" in a Western locale. Also, I was able to drop 939 files from the 
System32 directory, with a total command-line length of 32766 characters. 

However, I ran into heap corruption that crashed Explorer when dropping files 
from "C:\Program Files (x86)\Windows Kits\10\Include\10.0.10586.0\um". I used 
gflags to enable the page heap for Explorer and tracked the problem down to 
FilenameListCchLength[W|A]. It wasn't allocating space for the double quotes 
for paths with spaces in them. So I made the following changes:

FilenameListCchLengthW:

length += oneLength + (wcschr(pszSource, L' ') ? 3 : 1);

FilenameListCchLengthA:

length += oneLength + (strchr(pszSource, ' ') ? 3 : 1);

After this I was able to drop 421 files from the above-mentioned Include 
directory, with a total command-line length of 32737 characters.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Steve Dower]

Steve Dower added the comment:

Attached my first pass at the shell extension.

As this has the potential to crash Windows Explorer, I want to be _really_ 
thorough, so all reviews and feedback welcomed.

--
keywords: +patch
Added file: http://bugs.python.org/file43760/27469_1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Steve Dower]

Steve Dower added the comment:

I don't want Python to get into the business of changing the command line.

I've started looking into building the shell extension so we can pass through 
arguments correctly. Once we have this, there are likely other features we can 
add to it (maybe extract __author__/__version__ and display it in tooltips? 
Certainly some of the requests for IDLE could be accommodated).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Eryk Sun]

Eryk Sun added the comment:

The WSH drop handler was added to resolve issue 1656675. It was chosen because 
it supports long filenames. OTOH, the EXE drop handler probably uses short 
filenames because the command line is limited to 32,768 characters. (Actually, 
it only uses the short name for a path element if it's longer than 12 
characters or contains whitespace, which was why my example still had non-ASCII 
"Ā" in the filename.)

If Python.File switches to the EXE drop handler, then I think PySys_SetArgv 
should automatically expand short filenames. A command-line option could be 
added to disable automatic expansion. It would be nice to implement this change 
in 2.7 as well since the WSH drop handler has been used since 2.6. 

The alternative is to distribute a DLL with the launcher that implements the 
IPersistFile and IDropTarget interfaces [1], with support for both long 
filenames and Unicode. It has to be a shell extension, so AFAIK it can't be 
hosted in a local server process, if that's what Steve meant by embedding it in 
py.exe.

[1]: https://msdn.microsoft.com/en-us/library/cc144165

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Steve Dower]

Steve Dower added the comment:

Got some digging done today and it looks like we're probably best to write our 
own drop handler. It can probably be embedded into the py.exe launcher, which 
will keep all the registration in the one file.

At worst, we should switch to {86C86720-42A0-1069-A2E8-08002B30309D} (same as 
EXE files - converts long/Unicode paths to short filenames).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Steve Dower]

Steve Dower added the comment:

Okay thanks. I'll see if I can track down the right one on Monday.

--
assignee:  -> steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Adam Bartoš]

Adam Bartoš added the comment:

Without a handler the drop feature is disabled.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Steve Dower]

Steve Dower added the comment:

Try removing the handler completely and see what the default behavior is.

Otherwise, I'll do some research and figure out the right one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Eryk Sun]

Eryk Sun added the comment:

Yes, a different drop handler solves the problem. It doesn't have to be the 
exefile handler that's built into shell32.dll. Another handler could be used 
that preserves Unicode filenames and long paths.

I tested in Windows 10.0.10586.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Adam Bartoš]

Adam Bartoš added the comment:

Also, what versions of Windows does this affect? I have 64bit Vista, so maybe 
this is fixed in say Windows 10.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Adam Bartoš]

Adam Bartoš added the comment:

Thank you very much for the analysis. So Python Windows installers may be 
changed to set the other drop handler. If the short paths are problem, they may 
be converted to long ones when initializing `sys.argv`.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Eryk Sun]

Eryk Sun added the comment:

On second thought, it occurred to me that the problem isn't in Explorer or 
shell32, but in the drop handler. It turns out the problem is that the drop 
handler calls the ANSI API DragQueryFileA instead of DragQueryFileW. To see 
this I attached a debugger to Explorer and set a breakpoint on 
shell32!DragQueryFileA:

Breakpoint 0 hit
SHELL32!DragQueryFileA:
7ffd`bed1a5b0 4883ec38sub rsp,38h
0:056> k 3
Child-SP  RetAddr   Call Site
`0ee3e0f8 7ffd`b2c95b7a SHELL32!DragQueryFileA
`0ee3e100 7ffd`bedade2e wshext!CWSHExtension::Drop+0x7a
`0ee3e300 7ffd`bed9d547 
SHELL32!CDVDropTarget::_PerformDrop+0x14a

Note that the shell is deferring to the drop handler for the file type, which 
in our case is implemented by wshext.dll. 

The first DragQueryFileA call is to query the number of files (i.e. iFile == 
0x):

0:056> r rdx
rdx=
0:056> g

The 2nd call gets the ANSI encoded filename:

Breakpoint 0 hit
SHELL32!DragQueryFileA:
7ffd`bed1a5b0 4883ec38sub rsp,38h
0:056> r r8
r8=0ee3e1a0
0:056> pt; da ee3e1a0
`0ee3e1a0  "C:\Temp\A.txt"

The drop handler is set in the Python.File ProgId, which is defined in the 
installer in Tools/msi/launcher/launcher_reg.wxs, which configures the 
following registry entry:

C:\>reg query HKLM\Software\Classes\Python.File\shellex\DropHandler

HKEY_LOCAL_MACHINE\Software\Classes\Python.File\shellex\DropHandler
(Default)REG_SZ{60254CA5-953B-11CF-8C96-00AA00B8708C}

As we previously saw in the stack trace, it's implemented by wshext.dll:

C:\>reg query 
HKLM\Software\Classes\CLSID\{60254CA5-953B-11CF-8C96-00AA00B8708C} /s


HKEY_LOCAL_MACHINE\Software\Classes\CLSID\{60254CA5-953B-11CF-8C96-00AA00B8708C}
(Default)REG_SZShell Extension For Windows Script Host


HKEY_LOCAL_MACHINE\Software\Classes\CLSID\{60254CA5-953B-11CF-8C96-00AA00B8708C}\InProcServer32
(Default)REG_SZC:\Windows\System32\wshext.dll
ThreadingModelREG_SZApartment


I thought I could fix this easily by switching to the drop handler that batfile 
and exefile use:

C:\>reg add HKLM\Software\Classes\Python.File\shellex\DropHandler /ve /d ^
More? {86C86720-42A0-1069-A2E8-08002B30309D}
Value  exists, overwrite(Yes/No)? y
The operation completed successfully.

This gets the Unicode filename right, but when I move "Ā.txt" to a directory 
under "Program Files", I see that it uses short filenames in the path. 

['C:\\PROGRA~1\\Python35\\Ā.txt']

Is that acceptable?

--
resolution: third party -> 
stage: resolved -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Eryk Sun]

Eryk Sun added the comment:

Nothing can be done about this from Python. It's a bug in how Explorer handles 
the dropped filename. 

Note that it's not simply replacing Unicode characters with question marks. 
It's using a best-fit ANSI encoding. For example, codepage 1252 maps "Ā" to 
"A". If there's no defined best-fit mapping, most codepages default to using 
"?" as the replacement character when encoding via WideCharToMultiByte. When 
decoding via MultiByteToWideChar, some codepages (e.g. 932), use katakana 
middle dot (U+30FB) as the default instead of a question mark.

For example, here's the commandline of py.exe when I drop a file named "Ā.txt" 
on a script. Note that "Ā" becomes "A":

0:000> ?? @$peb->ProcessParameters->CommandLine
struct _UNICODE_STRING
 ""C:\Windows\py.exe" "C:\Temp\test.py" C:\Temp\A.txt "
   +0x000 Length   : 0x68
   +0x002 MaximumLength: 0x6a
   +0x004 Buffer   : 0x00771d50  ""C:\Windows\py.exe" 
"C:\Temp\test.py" C:\Temp\A.txt "

It's bizarre that it encodes the filename as ANSI just to decode it later when 
it calls CreateProcess. But Explorer probably still has a lot old code from 
back when it had to run on both Windows NT and DOS-based Windows 9x. This is 
probably a vestige of some workaround.

It isn't a problem if you ShellExecuteEx the Python script. For example, I ran 
"C:\Temp\test.py C:\Temp\Ā.txt" in the command prompt, and here's the resulting 
command line:

0:000> ?? @$peb->ProcessParameters->CommandLine
struct _UNICODE_STRING
 ""C:\Windows\py.exe" "C:\Temp\test.py"  C:\Temp\Ā.txt"
   +0x000 Length   : 0x68
   +0x002 MaximumLength: 0x6a
   +0x004 Buffer   : 0x00981cf8  ""C:\Windows\py.exe" 
"C:\Temp\test.py"  C:\Temp\Ā.txt"

Explorer actually handles drag and drop correctly when dropping the file on 
another window. So as a (clunky) workaround, you can drag the script icon into 
a command prompt or Win+R run dialog, and then drag the target file. The shell 
should add quotes where required.

--
nosy: +eryksun
resolution:  -> third party
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Emanuel Barry]

Changes by Emanuel Barry :


--
nosy: +ebarry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27469] Unicode filename gets crippled on Windows when drag and drop

[Adam Bartoš]

New submission from Adam Bartoš:

When a Python script is run by drag-and-dropping another file on it in Windows 
explorer, the other file's path becomes sys.argv[1]. However, when the path 
contains a Unicode characters (e.g. α), it gets crippled – it is replaced by 
ordinary question mark.

Maybe this is not Python's fault. Calling directly GetCommandLineW already 
contains that question mark. I'm just curious whether the original path even 
can be reconstructed.

--
components: Unicode, Windows
messages: 269991
nosy: Drekin, ezio.melotti, haypo, paul.moore, steve.dower, tim.golden, 
zach.ware
priority: normal
severity: normal
status: open
title: Unicode filename gets crippled on Windows when drag and drop
type: behavior
versions: Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com