[issue13617] Reject embedded null characters in wchar* strings

[STINNER Victor]

STINNER Victor added the comment:

Thank you very much Serhiy of taking care of this bug!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Backporting this to 2.7 requires too much work taking to account that 
PyArg_Parse and other argument parsing functions don't check for null 
characters in 2.7. The most serious security issue is fixed in issue30730, 
other cases unlikely can be used for attacks.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset 54ba940abc2fabb94fede46dfad80f8ac15632a3 by Serhiy Storchaka in 
branch '3.5':
[3.5] bpo-13617: Reject embedded null characters in wchar* strings. (GH-2302) 
(#2463)
https://github.com/python/cpython/commit/54ba940abc2fabb94fede46dfad80f8ac15632a3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset ccdc09ed1ebea7d7c6b41548132aa08bd797bfe8 by Serhiy Storchaka in 
branch 'master':
Fix compiler warnings on Windows introduced in bpo-13617. (#2464)
https://github.com/python/cpython/commit/ccdc09ed1ebea7d7c6b41548132aa08bd797bfe8


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset 0834905d9b61291b1fc5e05a1ffbc69de9c9379f by Serhiy Storchaka in 
branch '3.6':
[3.6] bpo-13617: Reject embedded null characters in wchar* strings. (GH-2302) 
(#2462)
https://github.com/python/cpython/commit/0834905d9b61291b1fc5e05a1ffbc69de9c9379f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +2519

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +2518

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +2517

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset f7eae0adfcd4c50034281b2c69f461b43b68db84 by Serhiy Storchaka in 
branch 'master':
[security] bpo-13617: Reject embedded null characters in wchar* strings. (#2302)
https://github.com/python/cpython/commit/f7eae0adfcd4c50034281b2c69f461b43b68db84


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[STINNER Victor]

STINNER Victor added the comment:

Wow, it's nice to see activity on this issue that I opened 6 years ago :-)

Sorry Serhiy, I don't have the bandwidth right now to review your change :-( In 
lack of review, I suggest you to just push it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

While working on this issue I found a way to inject environment variables for a 
subprocess on Windows. Reclassified this issue as a security issue. PR 2302 
fixes this. May be there are other security vulnerabilities fixed by it.

--
type: behavior -> security

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

PR 2302 doesn't fix all issues with PyUnicode_AsWideCharString(). Issue30708 
should fix them.

--
dependencies: +Ensure that the result of PyUnicode_AsWideCharString() doesn't 
contain null characters if size is not returned

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +2349

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
assignee:  -> serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[STINNER Victor]

STINNER Victor added the comment:

Sorry, I lost track of this issue. Feel free to update and complete my patch :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Mark Lawrence]

Changes by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
type:  -> behavior
versions: +Python 3.6, Python 3.7 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Could you update your patch Victor?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Ben Hoyt]

Ben Hoyt added the comment:

Note that this (or a very similar issue) also affects os.listdir() on Windows: 
os.listdir(bytes_path_with_nul) raises ValueError as expected, but 
os.listdir(unicode_path_with_nul) does not. Test case:

 import os
 os.mkdir('foo')
 os.listdir(b'foo\x00zzz')
Traceback (most recent call last):
  File stdin, line 1, in module
ValueError: listdir: embedded null character in path
 os.listdir('foo\x00zzz')
[]

However, this is not the case on Linux, as there both calls raise an 
appropriate ValueError.

This needs to be fixed in posixmodule.c's path_converter() function.

I'm in the middle of implementing PEP 471 (os.scandir), so don't want to create 
a proper patch right now, but the fix is to add these lines in posixmodule.c 
path_converter() after the if (length  32767) {...} block:

if ((size_t)length != wcslen(wide)) {
FORMAT_EXCEPTION(PyExc_ValueError, embedded null character in %s);
Py_DECREF(unicode);
return 0;
}

We should also add test to test_os.py like the following:

def test_listdir_nul_in_path(self):
self.assertRaises(ValueError, os.listdir, 'y\x00z')
self.assertRaises(ValueError, os.listdir, b'y\x00z')

--
nosy: +benhoyt

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13617
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Could you please answer my comments Victor?

--
versions: +Python 3.4, Python 3.5 -Python 3.2, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13617
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Mark Lawrence]

Mark Lawrence added the comment:

@Victor can you pick this up again please.

--
nosy: +BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13617
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


--
stage:  - needs patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13617
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I added some comments in Rietveld.

I see other instances of the use of non-checked PyUnicode_AsWideCharString() 
and PyUnicode_AsUnicode().

--
nosy: +serhiy.storchaka

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13617
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Arnaud Calmettes]

Arnaud Calmettes calmettes.arn...@gmail.com added the comment:

Here is a patch for the documentation. 

I added warnings for, PyUnicode_AsWideChar*, PyUnicode_EncodeFSDefault and 
PyUnicode_AsUnicode*, since they're all concerned by this issue.

--
nosy: +arnaudc
Added file: http://bugs.python.org/file24034/doc_unicode.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13617
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Arnaud Calmettes]

Arnaud Calmettes calmettes.arn...@gmail.com added the comment:

I removed the hints using wcslen on the result of PyUnicode_AsWideChar*, 
since the resulting wchar_t strings may not be null-terminated

--
Added file: http://bugs.python.org/file24037/doc_unicode-2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13617
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Roundup Robot]

Roundup Robot devn...@psf.upfronthosting.co.za added the comment:

New changeset fa5c8cf29963 by Victor Stinner in branch '3.2':
Issue #13617: Document that the result of the conversion of a Unicode object to
http://hg.python.org/cpython/rev/fa5c8cf29963

New changeset f30ac7729f2b by Victor Stinner in branch 'default':
Issue #13617: Document that the result of the conversion of a Unicode object to
http://hg.python.org/cpython/rev/f30ac7729f2b

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13617
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[Roundup Robot]

Roundup Robot devn...@psf.upfronthosting.co.za added the comment:

New changeset 1c4d9534263e by Victor Stinner in branch '2.7':
Issue #13617: Document that the result PyUnicode_AsUnicode() and
http://hg.python.org/cpython/rev/1c4d9534263e

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13617
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

embedded_nul-2.patch: a more complete patch check also null byte in functions 
calling PyUnicode_EncodeFSDefault().

--
Added file: http://bugs.python.org/file24041/embedded_nul-2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13617
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[STINNER Victor]

New submission from STINNER Victor victor.stin...@haypocalc.com:

The curses module (only since Python 3.3), locale.strcoll(), locale.strxfrm(), 
time.strftime() and imp.NullImporter() (only on Windows) accept embedded null 
characters, whereas they convert the Unicode string to a wide character 
(wchar_t*) string.

The problem is that the null character truncates the string. Example:


 locale.strxfrm('a')
'a'
 locale.strxfrm('a\0b')
'a'

Attached patch fixes these functions. I wrote the patch for Python 3.3.

--
components: Library (Lib), Unicode
files: embedded_nul.patch
keywords: patch
messages: 149656
nosy: ezio.melotti, haypo
priority: normal
severity: normal
status: open
title: Reject embedded null characters in wchar* strings
versions: Python 3.2, Python 3.3
Added file: http://bugs.python.org/file23984/embedded_nul.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13617
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13617] Reject embedded null characters in wchar* strings

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

PyUnicode_AsWideCharString() documentation should also warn about this issue.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13617
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com