[issue30015] Windows also treats full-width spaces as a delimiter when parsing arguments

[LCY]

LCY added the comment:

Thank you for testing and explanation.

My test cases are not enough.

I will close this issue.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30015] Windows also treats full-width spaces as a delimiter when parsing arguments

[Eryk Sun]

Eryk Sun added the comment:

> Windows also treats full-width spaces as a delimiter when parsing 
> command line arguments.

CreateProcess has to parse the beginning of the command-line string if the 
lpApplicationName parameter is omitted. According to the documentation, it 
treats "white space" as a delimiter, but it doesn't actually say which 
characters are in that set. We know for an unquoted name like 
"python{character}spam" that it will try to execute python.exe if "{character}" 
is parsed as a space. Otherwise we expect CreateProcess to fail with 
ERROR_FILE_NOT_FOUND because it's looking for the non-existent file 
"python{character}spam". Here's a test that checks all characters that Unicode 
considers to be whitespace, which includes "ideographic space" (U+3000):

import os
import sys
import subprocess

space_chars = [chr(c) for c in range(sys.maxunicode) if chr(c).isspace()]
assert '\N{IDEOGRAPHIC SPACE}' in space_chars # U+3000

def get_create_delims():
assert not os.path.exists('spam')
filename = 'python{}spam'
basepath = os.path.dirname(sys.executable)
delims = []
for space in space_chars:
path = os.path.join(basepath, filename.format(space))
assert not os.path.exists(path)
try:
subprocess.check_output(path, stderr=subprocess.STDOUT)
except FileNotFoundError:
pass # not a delimiter
except subprocess.CalledProcessError:
delims.append(space)
else:
assert False, 'python.exe should have failed'
return delims


>>> get_create_delims()
['\t', ' ']

CreateProcess considers only space and horizontal tab as white-space 
delimiters, at least on this Windows 10 system.

Otherwise Windows itself doesn't care about the command line. It's up to each 
application to parse its command line however it wants. subprocess.list2cmdline 
assumes an application uses argv from Microsoft's C runtime. The Windows shell 
function CommandLineToArgvW is supposed to follow the same rules. The following 
calls CommandLineToArgvW on a test command-line string for each character in 
the space_chars set:

import ctypes
from ctypes import wintypes

shell32 = ctypes.WinDLL('shell32', use_last_error=True) 

PLPWSTR = ctypes.POINTER(wintypes.LPWSTR)
shell32.CommandLineToArgvW.restype = PLPWSTR

def cmdline2argv(cmdline):
argc = ctypes.c_int()
pargv = shell32.CommandLineToArgvW(cmdline, ctypes.byref(argc))
if not pargv:
raise ctypes.WinError(ctypes.get_last_error())
return pargv[:argc.value]

def get_argv_delims():
cmdline = 'test{}space'
delims = []
for space in space_chars:
if len(cmdline2argv(cmdline.format(space))) > 1:
delims.append(space)
return delims


>>> get_argv_delims()
['\t', '\n', '\x0b', '\x0c', '\r', '\x1c', '\x1d', '\x1e', '\x1f', ' ']

In addition to space and horizontal tab, CommandLineToArgvW also considers line 
feed, vertical tab, form feed, carriage return, file separator, group 
separator, record separator, and unit separator to be white-space delimiters. 
This disagrees with [1], which says it should be limited to space and 
horizontal tab, like CreateProcess. Let's test this as well:

def get_msvc_argv_delims():
template = '"{}" -c "import sys;print(len(sys.argv))" test{}space'
delims = []
for space in space_chars:
cmdline = template.format(sys.executable, space)
out = subprocess.check_output(cmdline)
argc = int(out)
if argc > 2:
delims.append(space)
return delims


>>> get_msvc_argv_delims()
['\t', ' ']

Apparently CommandLineToArgvW is inconsistent with the C runtime in this case.

On my Windows 10 system, ideographic space (U+3000) is not generally a 
command-line delimiter. That's not to say that some applications (and maybe 
localized CRTs?) don't use it that way. But I don't think it's the place of the 
subprocess module to handle it.

[1]: https://msdn.microsoft.com/en-us/library/17w5ykft

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30015] Windows also treats full-width spaces as a delimiter when parsing arguments

[Roundup Robot]

Changes by Roundup Robot :


--
pull_requests: +1186

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30015] Windows also treats full-width spaces as a delimiter when parsing arguments

[林建佑]

New submission from 林建佑:

Windows also treats full-width spaces as a delimiter when parsing command line 
arguments.

Therefore, subprocess.run() and subprocess.Popen() also need to quote the arg 
in the sequence of arguments if there is any full-width spaces in it.

Example:

>> subprocess.run(['foo', 'half-width space', 'full-width　space'])

should be executed as 
>> foo "half-width space" "full-width　space"
Windows will treat it as 3 arguments

but now it is incorrectly executed as 
>> foo "half-width space" full-width　space
Windows will treat it as 4 arguments

--
components: Library (Lib), Windows
messages: 291262
nosy: paul.moore, steve.dower, tim.golden, zach.ware, 林建佑
priority: normal
severity: normal
status: open
title: Windows also treats full-width spaces as a delimiter when parsing 
arguments
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com