[issue29509] redundant interning in PyObject_GetAttrString

[INADA Naoki]

New submission from INADA Naoki:

PyObject_GetAttrString creates temporary unicode object and intern it.
It totally redundant.

--
files: getattrstring-nointern.patch
keywords: patch
messages: 287386
nosy: inada.naoki
priority: normal
severity: normal
stage: patch review
status: open
title: redundant interning in PyObject_GetAttrString
type: performance
versions: Python 3.7
Added file: http://bugs.python.org/file46602/getattrstring-nointern.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26873] xmlrpclib raises when trying to convert an int to string when unicode is available

[Zsolt Endreffy]

Zsolt Endreffy added the comment:

I think this patch breaks compatibility between python 2.7 versions. Our rpc 
server has 2.7.10 Python version, and sends back tuples as responses (first 
value is a boolean, second is a string). If we connect with a computer, which 
has 2.7.11 or earlier Python, then it is working flawlessly. When we connect 
with Python 2.7.12 or later, then it sends this error:
ResponseError: ResponseError("unknown tag u'tuple'",)

As far as I understand, the xmlrpclib should be able to marshal through tuples.

--
nosy: +Zsolt Endreffy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29508] Cannot install python-3.2.5 on Mac 10.11.6

[Ned Deily]

Ned Deily added the comment:

The python.org macOS installers provided on .dmg files for older releases used 
a legacy installer package format that is no longer supported by the OS X 
Installer app on the most recent maxOS releases.  The newer .pkg format 
installer files should be OK.  As Zach notes, 3.2.5 in very out-of-date and its 
use should be avoided now.

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29508] Cannot install python-3.2.5 on Mac 10.11.6

[Zachary Ware]

Zachary Ware added the comment:

Python 3.2 is out of support, so unfortunately there's nothing we can do to 
help here.  I would recommend using Python 3.6 as it is the latest version, but 
if you absolutely need 3.2 for some reason your best bet is probably to build 
it yourself.  It is a fairly straightforward procedure, and a guide to get 
started can be found in the CPython devguide.

--
nosy: +zach.ware
resolution:  -> out of date
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[Jim DeLaHunt]

Jim DeLaHunt added the comment:

Marco:

> To have a wide view, usually it is a good idea to start from the beginning:
> 
> https://groups.google.com/forum/#!msg/comp.lang.python/DfzH5Nrt05E/Yyd3s7fPVxwJ

Thank you, that is a very interesting thread. Clearly the creator of doctests, 
Tim Peters, is a well-regarded person in the Python world. 

On the other hand, I was surprised to see some of the examples used in the 
present-day documentation in Tim's original 1999 message. I find it remarkable 
that in 18 years since then, no rewrite or editing has replaced those examples 
with better ones.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29508] Cannot install python-3.2.5 on Mac 10.11.6

[Michael Rieck]

New submission from Michael Rieck:

python-3.2.5-macosx10.6.dmg starts installation on my Mac 10.11.6 but 
terminates at end with following message: 

The Installation Failed.
The installer could not install the software because there was no software to 
install. 

I cold booted my computer, tried again, and got the same results.

--
components: Installation
messages: 287381
nosy: rieck
priority: normal
severity: normal
status: open
title: Cannot install python-3.2.5 on Mac 10.11.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23402] dynload_shlib does not close ldl handles when the interpreter is shut down

[Super Bo]

Super Bo added the comment:

I think  dynload_unload_all() should be called in PyImport_Cleanup() function.

--
nosy: +Super Bo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22594] Add a link to the regex module in re documentation

[Ma Lin]

Ma Lin added the comment:

How about this? Is it too long?
@Matthew Barnett, I'm animalize on your repository.

.. seealso::

Third-party module `regex `_,

is a substitute for re module, it offers some mainstream features 
which re module lacks (Atomic grouping, Possessive quantifiers,
Variable-length lookbehind, Recursive matching). It also has further
Unicode support, some advanced grammars appears in other regular
expression engines, and many facilitations.

--
nosy: +Ma Lin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29463] Add `docstring` attribute to AST nodes

[INADA Naoki]

INADA Naoki added the comment:

Now I doubt about this patch is really good.

docstring of Module and Class generates two bytecode.
So it's a real, executed statement.

  LOAD_CONST  0 ("docstring")
  STORE_NAME  0 ("__doc__")

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29465] Add _PyObject_FastCall() to reduce stack consumption

[Roundup Robot]

Roundup Robot added the comment:


New changeset e9807e307f1f561a2dfe3e9f97a38444528dba86 by Victor Stinner in 
branch 'master':
Fix PyCFunction_Call() performance issue
https://github.com/python/cpython/commit/e9807e307f1f561a2dfe3e9f97a38444528dba86


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29259] Add tp_fastcall to PyTypeObject: support FASTCALL calling convention for all callable objects

[Roundup Robot]

Roundup Robot added the comment:


New changeset e9807e307f1f561a2dfe3e9f97a38444528dba86 by Victor Stinner in 
branch 'master':
Fix PyCFunction_Call() performance issue
https://github.com/python/cpython/commit/e9807e307f1f561a2dfe3e9f97a38444528dba86


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29506] Incorrect documentation for the copy module

[Steven D'Aprano]

Steven D'Aprano added the comment:

What about "administrative data structures" that should be copied?

I think that "administrative data structures" is a red herring: there could be 
data that needs copying, and data that needs sharing and shouldn't be copied, 
independent of what the data is used for.

"Because deepcopy copies everything it may copy too much, such as data which is 
intended to be shared between copies."

--
nosy: +steven.daprano

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29465] Add _PyObject_FastCall() to reduce stack consumption

[Roundup Robot]

Roundup Robot added the comment:

New changeset 31342913fb1e by Victor Stinner in branch 'default':
Fix PyCFunction_Call() performance issue
https://hg.python.org/cpython/rev/31342913fb1e

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29259] Add tp_fastcall to PyTypeObject: support FASTCALL calling convention for all callable objects

[Roundup Robot]

Roundup Robot added the comment:

New changeset 31342913fb1e by Victor Stinner in branch 'default':
Fix PyCFunction_Call() performance issue
https://hg.python.org/cpython/rev/31342913fb1e

--
status: pending -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

[STINNER Victor]

STINNER Victor added the comment:

Maybe PyObject_Call(), _PyObject_FastCallDict(), etc. can also be modified to 
get the following fast-path:

+if (Py_TYPE(func) == _Type) {
+result = _PyMethod_FastCall(func, args, nargs);
+}

But I don't know how common it is to get a PyMethod_Type object in these 
functions, nor the code of the additional if.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

[STINNER Victor]

New submission from STINNER Victor:

Subset of the (almost) rejected issue #29259 (tp_fastcall), attached patch adds 
_PyMethod_FastCall() and uses it in call_method() of typeobject.c. The change 
avoids the creation of a temporary tuple for Python functions and METH_FASTCALL 
C functions.

Currently, call_method() calls method_call() which calls 
_PyObject_Call_Prepend(), and calling method_call() requires a tuple for 
positional arguments.

Example of benchmark on __getitem__(): 1.3x faster (-22%).

$ ./python -m perf timeit -s 'class C:' -s ' def __getitem__(self, index): 
return index' -s 'c=C()' 'c[0]' 

Median +- std dev: 130 ns +- 1 ns => 102 ns +- 1 ns


See also the issue #29263 "Implement LOAD_METHOD/CALL_METHOD for C functions".

--
components: Interpreter Core
files: method_fastcall.patch
keywords: patch
messages: 287371
nosy: haypo, inada.naoki, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Use FASTCALL in call_method() to avoid temporary tuple
type: performance
versions: Python 3.7
Added file: http://bugs.python.org/file46601/method_fastcall.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28686] py.exe ignored PATH when using python3 shebang

[Eryk Sun]

Eryk Sun added the comment:

If the system doesn't have a "python3.exe" on PATH, then "env python3" will run 
a registered version, exactly like it currently does. IMO, this is slightly 
less confusing than the current behavior, which skips searching PATH in this 
case. We can take it a step further, however. If the launcher can't find the 
versioned name, then search for "python[w].exe" and determine its version using 
the approach I outlined above. If it doesn't match the required version, as 
always the launcher will default to running a registered version.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26388] Disabling changing sys.argv[0] with runpy.run_module(...alter_sys=True)

[Piotr Dobrogost]

Changes by Piotr Dobrogost :


--
nosy: +piotr.dobrogost

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29506] Incorrect documentation for the copy module

[Mariatta Wijaya]

Mariatta Wijaya added the comment:

How about

```
Because deep copy copies everything, it may copy too much, including the 
administrative data structures.
```

?

--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29506] Incorrect documentation for the copy module

[Zachary Ware]

Zachary Ware added the comment:

That line is correct, the point is that deep copy will duplicate even the 
things that you may *want* to be shared.

See also #27416.  I'm not sure what we can do to further clarify that line, but 
it is certainly easy to misread.

--
nosy: +zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29506] Incorrect documentation for the copy module

[Mariatta Wijaya]

Changes by Mariatta Wijaya :


--
stage:  -> needs patch
versions:  -Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29506] Incorrect documentation for the copy module

[Pedro]

New submission from Pedro:

The docs for the copy module contain a bullet that says the following:

"Because deep copy copies everything it may copy too much, e.g., even 
administrative data structures that should be shared even between copies."

There should be a "not" between "should" and "be shared." I think the second 
"even" can be dropped, too.

--
assignee: docs@python
components: Documentation
messages: 287367
nosy: docs@python, pgacv2
priority: normal
severity: normal
status: open
title: Incorrect documentation for the copy module
versions: Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29463] Add `docstring` attribute to AST nodes

[INADA Naoki]

INADA Naoki added the comment:

So what's new entry may be:

+* ``Module``, ``FunctionDef``, ``AsyncFunctionDef``, and
+  ``ClassDef`` AST nodes now have a new ``docstring`` attribute.
+  The first statement in their body is not considered as a docstring anymore.
+  This affects ``co_firstlineno`` and ``co_lnotab`` attribute of code object
+  for module and class.
+  (Contributed by Eugene Toder and INADA Naoki in :issue:`29463`.)
+

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29463] Add `docstring` attribute to AST nodes

[INADA Naoki]

INADA Naoki added the comment:

This patch affects firstlineno and lnotab of module and class, but not 
functions.

module:
- at 0x7f053a8f8b70, file "Lib/importlib/_bootstrap.py", 
line 8>
+ at 0x7fdefbf10340, file "Lib/importlib/_bootstrap.py", 
line 25>
 filename Lib/importlib/_bootstrap.py
-firstlineno: 8
+firstlineno: 25
-lnotab: 
b'\x04\x11\x04\x02\x08\x08\x08\x07\x04\x02\x04\x03\x10\x04\x0eD\x0e\x15\x0e\x13\x08\x13\x08\x13\x08\x0b\x0e\x08\x08\x0b\x08\x0c\x08\x10\x08$\x0e\x1b\x0ee\x10\x1a\x06\x03\n-\x0e<\x08\x11\x08\x11\x08\x19\x08\x1d\x08\x17\x08\x10\x0eI\x0eM\x0e\r\x08\t\x08\t\n/\x08\x14\x04\x01\x08\x02\x08\x1b\x08\x06\n\x19\x08\x1f\x08\x1b\x12#\x08\x07\x08/'
-  8   0 LOAD_CONST   0 ('Core implementation of 
import.\n\nThis module is NOT meant to be directly imported! It has been 
designed such\nthat it can be bootstrapped into Python as the implementation of 
import. As\nsuch it requires the injection of specific modules and attributes 
in order to\nwork. One should use importlib as the public-facing version of 
this module.\n\n')
+lnotab: 
b'\x04\x00\x04\x02\x08\x08\x08\x07\x04\x02\x04\x03\x10\x04\x0eD\x0e\x15\x0e\x13\x08\x13\x08\x13\x08\x0b\x0e\x08\x08\x0b\x08\x0c\x08\x10\x08$\x0e\x1b\x0ee\x10\x1a\x06\x03\n-\x0e<\x08\x11\x08\x11\x08\x19\x08\x1d\x08\x17\x08\x10\x0eI\x0eM\x0e\r\x08\t\x08\t\n/\x08\x14\x04\x01\x08\x02\x08\x1b\x08\x06\n\x19\x08\x1f\x08\x1b\x12#\x08\x07\x08/'
+ 25   0 LOAD_CONST   0 ('Core implementation of 
import.\n\nThis module is NOT meant to be directly imported! It has been 
designed such\nthat it can be bootstrapped into Python as the implementation of 
import. As\nsuch it requires the injection of specific modules and attributes 
in order to\nwork. One should use importlib as the public-facing version of 
this module.\n\n')
   2 STORE_NAME   0 (__doc__)
-
- 25   4 LOAD_CONST   1 (None)
+  4 LOAD_CONST   1 (None)
   6 STORE_GLOBAL 1 (_bootstrap_external)
 

class:
-
+
 filename Lib/importlib/_bootstrap.py
 firstlineno: 51
-lnotab: b'\x08\x04\x04\x02\x08\x08\x08\x0c\x08\x19\x08\r'
+lnotab: b'\x0c\x06\x08\x08\x08\x0c\x08\x19\x08\r'
  51   0 LOAD_NAME0 (__name__)
   2 STORE_NAME   1 (__module__)
   4 LOAD_CONST   0 ('_ModuleLock')
   6 STORE_NAME   2 (__qualname__)
-
- 55   8 LOAD_CONST   1 ('A recursive lock implementation 
which is able to detect deadlocks\n(e.g. thread 1 trying to take locks A 
then B, and thread 2 trying to\ntake locks B then A).\n')
+  8 LOAD_CONST   1 ('A recursive lock implementation 
which is able to detect deadlocks\n(e.g. thread 1 trying to take locks A 
then B, and thread 2 trying to\ntake locks B then A).\n')
  10 STORE_NAME   3 (__doc__)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29476] Simplify set_add_entry()

[INADA Naoki]

INADA Naoki added the comment:

I think it violates O(1).

"s.add(x); s.discard(x);" loop creates dummy chain.
Average chain length is determined by size of the set.  So it is O(len(s)) 
rather than amortized O(1).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29505] Submit the re, json, & csv modules to oss-fuzz testing

[Gregory P. Smith]

New submission from Gregory P. Smith:

For reference, read https://github.com/google/oss-fuzz.

We should investigate creating fuzz targets for the Python re module (_sre.c) 
at a minimum.  There are probably other good targets as well such as _json.c 
and _csv.c.

pickle and marshal are not intended to be secure so ignore those.

--
messages: 287363
nosy: gregory.p.smith
priority: normal
severity: normal
stage: test needed
status: open
title: Submit the re, json, & csv modules to oss-fuzz testing
type: security
versions: Python 2.7, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22594] Add a link to the regex module in re documentation

[Matthew Barnett]

Matthew Barnett added the comment:

Ah, well, if it hasn't changed after this many years, it never will. Expect one 
or two changes to the text. :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29476] Simplify set_add_entry()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I don't have good real-world example. Actually dict is used more often than set 
in such cases.

I agree with your arguments Raymond, in particular about your ranging of cases. 
But even if the bad case is hundreds times more rare than the good case, the 
total effect can be negative if the negative effect of the bad case is hundreds 
times larger that the positive effect of the good case. I checked timing just 
to be sure and was unpleasantly surprised that the negative effect is so large.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29465] Add _PyObject_FastCall() to reduce stack consumption

[STINNER Victor]

STINNER Victor added the comment:

Crap,  pyobject_fastcall-5.patch was not rebased correctly :-/ Please see 
pyobject_fastcall-6.patch instead.

--
Added file: http://bugs.python.org/file46600/pyobject_fastcall-6.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29465] Add _PyObject_FastCall() to reduce stack consumption

[STINNER Victor]

STINNER Victor added the comment:

I splitted my big patch into smaller changes, see my pull request:
https://github.com/python/cpython/pull/74

If you prefer Rietveld ([Review] button), I also attached the squashed change: 
pyobject_fastcall-5.patch.

--
Added file: http://bugs.python.org/file46599/pyobject_fastcall-5.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29503] Make embedded-Python detectable

[Paul Moore]

Paul Moore added the comment:

OK, well I certainly wouldn't bother supporting users trying to provide the 
path to an embedded distribution of Python. That's not what the distribution is 
for, and as the author of Bazel you'd be perfectly OK (IMO) to say you don't 
support that.

If you want to supply an embedded distribution with Bazel, and deploy it with 
the application, that's fine - in that case you can modify the _pth file when 
you copy the distribution to deploy it.

If you want to share the embeddable distribution that is installed with Bazel 
among multiple user scripts that get deployed, then I'd suggest deploying a 
wrapper script with the user's code. That wrapper could set sys.path then run 
the user's script (using the stdlib runpy module, maybe). You mention that 
option, but say it's "not as nice as having a direct command-line switch" - but 
while that may be true (I disagree, but it's a matter of opinion) surely the 
fact that there *is* no such command line switch makes that consideration 
irrelevant?

Also, I'm not sure how your original question (how you can distinguish an 
embeddable distribution from a normal one) would help here anyway. FWIW, if you 
really had to, you could do that by looking to see if there was a _pth file in 
os.listdir(sys.prefix)). That would be a plausible heuristic, but certainly not 
something I'd recommend as a robust solution.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29465] Add _PyObject_FastCall() to reduce stack consumption

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +25

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29504] blake2: compile error with -march=bdver2

[Mike Gilbert]

Mike Gilbert added the comment:

Downstream bug report: https://bugs.gentoo.org/show_bug.cgi?id=608586

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29504] blake2: compile error with -march=bdver2

[Mike Gilbert]

New submission from Mike Gilbert:

When compiling python-3.6.0 with -march=bdver2, the blake2 module fails to 
build.

In file included from 
/var/tmp/portage/dev-lang/python-3.6.0/work/Python-3.6.0/Modules/_blake2/impl/blake2s-round.h:70:0,
 from 
/var/tmp/portage/dev-lang/python-3.6.0/work/Python-3.6.0/Modules/_blake2/impl/blake2s.c:40,
 from 
/var/tmp/portage/dev-lang/python-3.6.0/work/Python-3.6.0/Modules/_blake2/blake2s_impl.c:32:
/var/tmp/portage/dev-lang/python-3.6.0/work/Python-3.6.0/Modules/_blake2/impl/blake2s-load-xop.h:29:4:
 error: expected identifier or '(' before 'return'
return _mm_blendv_epi8(t0, s1, mask);
^
/var/tmp/portage/dev-lang/python-3.6.0/work/Python-3.6.0/Modules/_blake2/impl/blake2s-load-xop.h:30:1:
 error: expected identifier or '(' before '}' token
 }*/
 ^
/var/tmp/portage/dev-lang/python-3.6.0/work/Python-3.6.0/Modules/_blake2/impl/blake2s-load-xop.h:30:3:
 error: expected identifier or '(' before '/' token
 }*/
   ^
In file included from 
/var/tmp/portage/dev-lang/python-3.6.0/work/Python-3.6.0/Modules/_blake2/blake2s_impl.c:32:0:
/var/tmp/portage/dev-lang/python-3.6.0/work/Python-3.6.0/Modules/_blake2/impl/blake2s.c:
 In function 'blake2s_init0':
/var/tmp/portage/dev-lang/python-3.6.0/work/Python-3.6.0/Modules/_blake2/impl/blake2s.c:167:38:
 error: 'blake2s_IV' undeclared (first use in this function)
   for( i = 0; i < 8; ++i ) S->h[i] = blake2s_IV[i];
  ^
/var/tmp/portage/dev-lang/python-3.6.0/work/Python-3.6.0/Modules/_blake2/impl/blake2s.c:167:38:
 note: each undeclared identifier is reported only once for each function it 
appears in
/var/tmp/portage/dev-lang/python-3.6.0/work/Python-3.6.0/Modules/_blake2/impl/blake2s.c:
 In function 'PyBlake2_blake2s_init_param':
/var/tmp/portage/dev-lang/python-3.6.0/work/Python-3.6.0/Modules/_blake2/impl/blake2s.c:176:44:
 error: 'blake2s_IV' undeclared (first use in this function)
   const uint8_t * v = ( const uint8_t * )( blake2s_IV );
^

--
components: Extension Modules
files: build.log
messages: 287356
nosy: floppymaster
priority: normal
severity: normal
status: open
title: blake2: compile error with -march=bdver2
versions: Python 3.6
Added file: http://bugs.python.org/file46598/build.log

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29503] Make embedded-Python detectable

[Andi Bergmeier]

Andi Bergmeier added the comment:

Gladly.

So imagine you have the following files:

- foobar
  - pyfoo
- foo.py
- src
  - test
- bar.py

Since Bazel is a build system, you declare (in directory foobar):

py_library(
  name = "foo", # abstract name
  imports = ".",
  srcs = ["pyfoo/foo.py"], # Which source files are in that library
)

py_test(
  name = "bartest",
  srcs = ["pyfoo/src/test/bar.py"], # The test script
  deps = ["foo"], # "Will import foo" -> import of foo's parent directory is 
added when executing test
)

Bazel does not ship with a specific Python version. So you (the user) start 
Bazel and provide it with a path to a Python on your system. This means that 
the Python version MAY be different on every execution of Bazel.

Then you can use Bazel to execute the bartest.
Bazel will create a temporary directory (as a bit of sandboxing) and copy all 
declared files (+ directories) into it. Bazel will execute the provided Python 
binary and try set PYTHONPATH so that a import foo does work (notice the 
imports declaration above).
This is the part where I think a ._pth alongside the script would be 
beneficial, because for every test invocation the paths will be different. And 
multiple tests may be executed in parallel (so a "global" ._pth does not cut 
it). Using a ._pth one could get around setting an environment variable.


Now to simplify deployment I want to put an embeddable Python alongside Bazel 
and always tell it to use this one. With Embeddable, the only way I have to 
modify sys.path is to execute Python with -c or write an intermediate script. 
Both can then modify sys.path and then load bar.py. Obviously this is not as 
nice as having a direct command-line switch or ._pth mechanism available.

I hope I explained enough. If not - don't hesitate to ask.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28686] py.exe ignored PATH when using python3 shebang

[Riccardo Polignieri]

Riccardo Polignieri added the comment:

Paul: 
> When inside a venv:
- If you want to execute a script, use a shebang of #!/usr/bin/env python and 
then use `py myscript.py`

Yes, I'm totally on board with this - that is, as far as I run my own scripts. 
I just wonder, what if one day I pip install someone else's work, and then 
bang!, 'py' is just working differently now, for a non-local reason, and 
without alerting me. 

It's true that all major libraries use the "right" kind of shebang these 
days... Still, I find it a bit disturbing nevertheless. 

But thanks for the clarifications, anyways - now 'py' expected behavior is much 
more clear to me. 


Eryk:
> it's far simpler to just support versioned executable names

Keeping my fingers crossed about this.

> Even if we don't install links/copies with these names, I don't see the harm 
> in allowing the launcher to look for them. Users can create the links manually

True, but It would allow for yet another possible source of confusion I'm 
afraid. No, if py were to look for versioned executables, then versioned 
executable should be supplied imo.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29503] Make embedded-Python detectable

[Paul Moore]

Paul Moore added the comment:

I'm still not clear what you're doing here - why does it matter where you have 
the _pth file?

Could you explain how your application directory is laid out, and what is the 
main executable for the application? I'm assuming it's a Windows executable, 
myapp.exe, but the fact that you seem to be implying taht you don't have a C 
compiler makes me wonder if that's the case.

You could also explain (in the light of the above details) precisely what 
directories you want to add to sys.path.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29503] Make embedded-Python detectable

[Andi Bergmeier]

Andi Bergmeier added the comment:

Using the Python API would of course be the preferred way to go.
Sadly that also means that you at minimum have a full compiler and that IMO 
seems overkill when you only want to build a Python package or a Python test 
(from the end-user point of view).

The ._pth would be perfect if I could place it alongside the .py script I want 
to execute.

BTW: Can you please remove PYTHONPATH from the help message and so forth for 
the embeddable variant. Can be misleading.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29503] Make embedded-Python detectable

[Paul Moore]

Paul Moore added the comment:

With the embedded distribution, you should probably be calling the Python API 
rather than running python.exe. And if you do it that way, you can set sys.path 
via the API before calling user code.

Alternatively, you can set up a site.py within your copy of the embedded 
distribution to configure the PATH you want.

Or you can just modify the _pth file that's in the embedded distribution. 
That's pretty much what it's for...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29503] Make embedded-Python detectable

[Andi Bergmeier]

Andi Bergmeier added the comment:

As a result either

a, have a way to modify sys.path and add support for embeddable
b, have no way to modify sys.path and fail very soon should someone try to use 
Bazel with a Python embeddable.

Either way the minimum is to have a way to detect which variant it is.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29503] Make embedded-Python detectable

[Andi Bergmeier]

Andi Bergmeier added the comment:

I would like to ship Python hermetically with Bazel (embedding it).
For that the Python embedded seems perfect.

A problem arises since currently Bazel is controlling the search path of Python 
via setting PYTHONPATH (which actually worked with Python 3.5.2-embedded for 
non prefixed paths).

Now I do not know whether there is any other way of modifying sys.path when 
calling python.exe.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29503] Make embedded-Python detectable

[R. David Murray]

R. David Murray added the comment:

Why should a library behave differently if the python using it is embedded in 
an application?  Can you explain your use case in more detail?

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29503] Make embedded-Python detectable

[Paul Moore]

Paul Moore added the comment:

The embedded distribution is meant to be for just that - embedded applications. 
I'm not quite sure what you mean by support it, or when you would write code 
that needed to know it was being run from an embedded application.

Can you clarify precisely why you need to be able to get at this information?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29476] Simplify set_add_entry()

[Raymond Hettinger]

Raymond Hettinger added the comment:

Serhiy, can you post your recurse() code with some sample data so that we can 
run some timings and get a sense of the effect on the extreme case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29476] Simplify set_add_entry()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

For the worst case the drawback is significant:

$ ./python -m perf timeit -s "s = set('a%s' % i for i in range(100))" -- 
"s.add('test'); s.discard('test')"
Unpatched:  Median +- std dev: 861 ns +- 82 ns
Patched:Median +- std dev: 2.81 us +- 0.18 us

How large the benefit in the best case? I can't get any significant difference.

$ ./python -m perf timeit -s "a = ['a%s' % i for i in range(1000)]" -- "set(a)"
Unpatched:  Median +- std dev: 130 us +- 6 us
Patched:Median +- std dev: 127 us +- 8 us

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29503] Make embedded-Python detectable

[Andi Bergmeier]

New submission from Andi Bergmeier:

While it is nice to have embeddable Python for Windows, currently there seems 
to be no way of distinguishing it from a "normal" Python, which reacts quite 
differently to its Environment.

Would like to support Embeddable Python (and ._pth), but really first need a 
way of identifying whether it is embeddable or not.

--
components: Windows
messages: 287344
nosy: Andi Bergmeier, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Make embedded-Python detectable

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29500] AddressSanitizer: heap-buffer-overflow on address 0x61600004a982

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29500] AddressSanitizer: heap-buffer-overflow on address 0x61600004a982

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29502] Should PyObject_Call() call the profiler on C functions, use C_TRACE() macro?

[STINNER Victor]

New submission from STINNER Victor:

call_function() and do_call_core() functions of Python/ceval.c use C_TRACE() 
macro to call the profiler, but PyObject_Call() and similar functions like 
_PyObject_FastCallKeywords() don't.

Is it a bug or a deliberate choice for performance?

Since PyObject_Call() and similar functions have now fast paths, I don't think 
that it's still needed to have these fast paths in ceval.c too. But I kept fast 
paths in ceval.c because of C_TRACE().

--
components: Interpreter Core
messages: 287343
nosy: haypo, inada.naoki, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Should PyObject_Call() call the profiler on C functions, use C_TRACE() 
macro?
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29474] Grammatical errors in weakref.WeakValueDictionary docs

[Marco Buttu]

Marco Buttu added the comment:

Thanks Mariatta, now it is OK to me

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29494] AddressSanitizer: SEGV on unknown address 0x00009fff8001

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29496] AddressSanitizer: SEGV on unknown address 0x01ffe96de071

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29482] AddressSanitizer: attempting double-free on 0x60b000007050

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29495] AddressSanitizer: SEGV on unknown address 0x02007ea947c3

[BeginVuln]

New submission from BeginVuln:

OS Version : Ubuntu 16.04 LTS
Python download link : 
https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz

Python version : 3.6.0

Normal build cmd : 
./configure 
make

Asan build cmd:
export CC="/usr/bin/clang -fsanitize=address
export CXX="/usr/bin/clang++ -fsanitize=address
./confiugre
make

GDB with exploitable:

To enable execution of this file add
add-auto-load-safe-path /home/test/check/PythonGDB/python-gdb.py
line to your configuration file "/home/test/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/test/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
info "(gdb)Auto-loading safe path"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
strlen () at ../sysdeps/x86_64/strlen.S:106
Description: Access violation near NULL on source operand
Short description: SourceAvNearNull (16/22)
Hash: 524fc888253e60855a72647740103ec8.0dd959fe8965dda124a3c8d6b55807e3
Exploitability Classification: PROBABLY_NOT_EXPLOITABLE
Explanation: The target crashed on an access violation at an address matching 
the source operand of the current instruction. This likely indicates a read 
access violation, which may mean the application crashed on a simple NULL 
dereference to data structure that has no immediate effect on control of the 
processor.
Other tags: AccessViolation (21/22)


ASAN:

ASAN:DEADLYSIGNAL
=
==18706==ERROR: AddressSanitizer: SEGV on unknown address 0x02007ea947c3 (pc 
0x0044ffe7 bp 0x7fffaa71f040 sp 0x7fffaa71e7e0 T0)
#0 0x44ffe6 in __interceptor_strlen.part.45 asan_interceptors.cc.o:?
#1 0x44ffe6 in ?? ??:0
#2 0x7f2309ccc95b in my_strdup 
/home/test/check/PythonASAN/Modules/_ctypes/_ctypes_test.c:169 (discriminator 1)
#3 0x7f2309ccc95b in ?? ??:0
#4 0x7f2309f17e3f in ffi_call_unix64 ??:?
#5 0x7f2309f17e3f in ?? ??:0
#6 0x7f2309f178aa in ffi_call ??:?
#7 0x7f2309f178aa in ?? ??:0
#8 0x7f230a145311 in _call_function_pointer 
/home/test/check/PythonASAN/Modules/_ctypes/callproc.c:809
#9 0x7f230a145311 in _ctypes_callproc 
/home/test/check/PythonASAN/Modules/_ctypes/callproc.c:1147
#10 0x7f230a145311 in ?? ??:0
#11 0x7f230a134199 in PyCFuncPtr_call 
/home/test/check/PythonASAN/Modules/_ctypes/_ctypes.c:3870
#12 0x7f230a134199 in ?? ??:0
#13 0x5745f0 in _PyObject_FastCallDict 
/home/test/check/PythonASAN/Objects/abstract.c:2316
#14 0x5745f0 in ?? ??:0
#15 0x7a7429 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4812
#16 0x7a7429 in ?? ??:0
#17 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#18 0x7995cc in ?? ??:0
#19 0x7ab4cb in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#20 0x7ab4cb in _PyFunction_FastCall 
/home/test/check/PythonASAN/Python/ceval.c:4870
#21 0x7ab4cb in fast_function 
/home/test/check/PythonASAN/Python/ceval.c:4905
#22 0x7ab4cb in ?? ??:0
#23 0x7a76f2 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4809
#24 0x7a76f2 in ?? ??:0
#25 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#26 0x7995cc in ?? ??:0
#27 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#28 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#29 0x7a9847 in ?? ??:0
#30 0x7ac2ea in _PyFunction_FastCallDict 
/home/test/check/PythonASAN/Python/ceval.c:5021
#31 0x7ac2ea in ?? ??:0
#32 0x574668 in _PyObject_FastCallDict 
/home/test/check/PythonASAN/Objects/abstract.c:2295
#33 0x574668 in ?? ??:0
#34 0x5749fa in _PyObject_Call_Prepend 
/home/test/check/PythonASAN/Objects/abstract.c:2358
#35 0x5749fa in ?? ??:0
#36 0x573e9b in PyObject_Call 
/home/test/check/PythonASAN/Objects/abstract.c:2246
#37 0x573e9b in ?? ??:0
#38 0x793369 in do_call_core /home/test/check/PythonASAN/Python/ceval.c:5057
#39 0x793369 in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3357
#40 0x793369 in ?? ??:0
#41 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#42 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#43 0x7a9847 in ?? ??:0
#44 0x7ac2ea in _PyFunction_FastCallDict 
/home/test/check/PythonASAN/Python/ceval.c:5021
#45 0x7ac2ea in ?? ??:0
#46 0x574668 in _PyObject_FastCallDict 
/home/test/check/PythonASAN/Objects/abstract.c:2295
#47 0x574668 in ?? ??:0
#48 0x5749fa in _PyObject_Call_Prepend 
/home/test/check/PythonASAN/Objects/abstract.c:2358
#49 

[issue29496] AddressSanitizer: SEGV on unknown address 0x01ffe96de071

[BeginVuln]

New submission from BeginVuln:

OS Version : Ubuntu 16.04 LTS
Python download link : 
https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz

Python version : 3.6.0

Normal build cmd : 
./configure 
make

Asan build cmd:
export CC="/usr/bin/clang -fsanitize=address
export CXX="/usr/bin/clang++ -fsanitize=address
./confiugre
make

GDB with exploitable:

To enable execution of this file add
add-auto-load-safe-path /home/test/check/PythonGDB/python-gdb.py
line to your configuration file "/home/test/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/test/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
info "(gdb)Auto-loading safe path"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Inferior 1 (process 19429) exited with code 01]


ASAN:

ASAN:DEADLYSIGNAL
=
==18136==ERROR: AddressSanitizer: SEGV on unknown address 0x01ffe96de071 (pc 
0x005e9305 bp 0x7ffc234b3300 sp 0x7ffc234b3240 T0)
#0 0x5e9304 in PyLong_AsLongAndOverflow 
/home/test/check/PythonASAN/Objects/longobject.c:408
#1 0x5e9304 in ?? ??:0
#2 0x5e9658 in PyLong_AsLong 
/home/test/check/PythonASAN/Objects/longobject.c:474 (discriminator 1)
#3 0x5e9658 in ?? ??:0
#4 0x7fda5a8bfe3f in ffi_call_unix64 ??:?
#5 0x7fda5a8bfe3f in ?? ??:0
#6 0x7fda5a8bf8aa in ffi_call ??:?
#7 0x7fda5a8bf8aa in ?? ??:0
#8 0x7fda5aaed311 in _call_function_pointer 
/home/test/check/PythonASAN/Modules/_ctypes/callproc.c:809
#9 0x7fda5aaed311 in _ctypes_callproc 
/home/test/check/PythonASAN/Modules/_ctypes/callproc.c:1147
#10 0x7fda5aaed311 in ?? ??:0
#11 0x7fda5aadc199 in PyCFuncPtr_call 
/home/test/check/PythonASAN/Modules/_ctypes/_ctypes.c:3870
#12 0x7fda5aadc199 in ?? ??:0
#13 0x5745f0 in _PyObject_FastCallDict 
/home/test/check/PythonASAN/Objects/abstract.c:2316
#14 0x5745f0 in ?? ??:0
#15 0x7a7429 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4812
#16 0x7a7429 in ?? ??:0
#17 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#18 0x7995cc in ?? ??:0
#19 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#20 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#21 0x7a9847 in ?? ??:0
#22 0x78e15d in PyEval_EvalCodeEx 
/home/test/check/PythonASAN/Python/ceval.c:4140
#23 0x78e15d in ?? ??:0
#24 0x784f1a in builtin___build_class__ 
/home/test/check/PythonASAN/Python/bltinmodule.c:170
#25 0x784f1a in ?? ??:0
#26 0x631f93 in _PyCFunction_FastCallDict 
/home/test/check/PythonASAN/Objects/methodobject.c:231
#27 0x631f93 in ?? ??:0
#28 0x7a7751 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4788 (discriminator 17)
#29 0x7a7751 in ?? ??:0
#30 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#31 0x7995cc in ?? ??:0
#32 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#33 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#34 0x7a9847 in ?? ??:0
#35 0x78e0df in PyEval_EvalCodeEx 
/home/test/check/PythonASAN/Python/ceval.c:4140
#36 0x78e0df in PyEval_EvalCode 
/home/test/check/PythonASAN/Python/ceval.c:695
#37 0x78e0df in ?? ??:0
#38 0x5142f5 in run_mod /home/test/check/PythonASAN/Python/pythonrun.c:980
#39 0x5142f5 in PyRun_FileExFlags 
/home/test/check/PythonASAN/Python/pythonrun.c:933
#40 0x5142f5 in ?? ??:0
#41 0x512afa in PyRun_SimpleFileExFlags 
/home/test/check/PythonASAN/Python/pythonrun.c:396
#42 0x512afa in ?? ??:0
#43 0x53eefd in run_file /home/test/check/PythonASAN/Modules/main.c:320
#44 0x53eefd in Py_Main /home/test/check/PythonASAN/Modules/main.c:780
#45 0x53eefd in ?? ??:0
#46 0x503d16 in main /home/test/check/PythonASAN/./Programs/python.c:69
#47 0x503d16 in ?? ??:0
#48 0x7fda5deaf82f in __libc_start_main 
/build/glibc-GKVZIf/glibc-2.23/csu/../csu/libc-start.c:291
#49 0x7fda5deaf82f in ?? ??:0
#50 0x432548 in _start ??:?
#51 0x432548 in ?? ??:0

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/home/test/check/PythonASAN/python+0x5e9304)
==18136==ABORTING

--
components: Interpreter Core
files: longobj_408
messages: 287335
nosy: beginvuln
priority: normal
severity: normal
status: open
title: AddressSanitizer: SEGV on unknown address 0x01ffe96de071
type: security
versions: Python 3.6
Added file: http://bugs.python.org/file46591/longobj_408

___
Python tracker 

[issue29501] AddressSanitizer: SEGV on unknown address 0x0000000028cb

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29498] AddressSanitizer: SEGV on unknown address 0x0005ffff800d

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29488] AddressSanitizer: SEGV on unknown address 0x0001a5525c1b

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29493] AddressSanitizer: SEGV on unknown address 0x000cffff800d

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29499] AddressSanitizer: SEGV on unknown address 0x000ebfff800d

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29495] AddressSanitizer: SEGV on unknown address 0x02007ea947c3

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29492] AddressSanitizer: SEGV on unknown address 0x0000a0013639

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29474] Grammatical errors in weakref.WeakValueDictionary docs

[Mariatta Wijaya]

Mariatta Wijaya added the comment:

Thanks for reviewing, Marco :)
Updated the patch.

--
Added file: http://bugs.python.org/file46597/issue29474py3-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29496] AddressSanitizer: SEGV on unknown address 0x01ffe96de071

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29486] AddressSanitizer: SEGV on unknown address 0x7f16f88e3560

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29497] AddressSanitizer: SEGV on unknown address 0x000000000008

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29500] AddressSanitizer: heap-buffer-overflow on address 0x61600004a982

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29482] AddressSanitizer: attempting double-free on 0x60b000007050

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29487] AddressSanitizer: heap-buffer-overflow on address 0x60200000e734

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29501] AddressSanitizer: SEGV on unknown address 0x0000000028cb

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29498] AddressSanitizer: SEGV on unknown address 0x0005ffff800d

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29483] AddressSanitizer: heap-buffer-overflow on address 0x60200000e731

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29500] AddressSanitizer: heap-buffer-overflow on address 0x61600004a982

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29499] AddressSanitizer: SEGV on unknown address 0x000ebfff800d

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29501] AddressSanitizer: SEGV on unknown address 0x0000000028cb

[BeginVuln]

New submission from BeginVuln:

OS Version : Ubuntu 16.04 LTS
Python download link : 
https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz

Python version : 3.6.0

Normal build cmd : 
./configure 
make

Asan build cmd:
export CC="/usr/bin/clang -fsanitize=address
export CXX="/usr/bin/clang++ -fsanitize=address
./confiugre
make

GDB with exploitable:

To enable execution of this file add
add-auto-load-safe-path /home/test/check/PythonGDB/python-gdb.py
line to your configuration file "/home/test/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/test/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
info "(gdb)Auto-loading safe path"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
strlen () at ../sysdeps/x86_64/strlen.S:106
Description: Access violation near NULL on source operand
Short description: SourceAvNearNull (16/22)
Hash: 887855ab5f56908afba8d62b6a25a6db.02c83d5748e9f8196679750a04737f93
Exploitability Classification: PROBABLY_NOT_EXPLOITABLE
Explanation: The target crashed on an access violation at an address matching 
the source operand of the current instruction. This likely indicates a read 
access violation, which may mean the application crashed on a simple NULL 
dereference to data structure that has no immediate effect on control of the 
processor.
Other tags: AccessViolation (21/22)


ASAN:

sEASAN:DEADLYSIGNAL
=
==18621==ERROR: AddressSanitizer: SEGV on unknown address 0x28cb (pc 
0x7f1572e57d16 bp 0x7ffeaf5703d0 sp 0x7ffeaf56fb68 T0)
#0 0x7f1572e57d15 in strlen 
/build/glibc-GKVZIf/glibc-2.23/string/../sysdeps/x86_64/strlen.S:76
#1 0x7f1572e57d15 in ?? ??:0
#2 0x44ffac in __interceptor_strlen.part.45 asan_interceptors.cc.o:?
#3 0x44ffac in ?? ??:0
#4 0x7f156c4cdf5c in string_at 
/home/test/check/PythonASAN/Modules/_ctypes/_ctypes.c:5226
#5 0x7f156c4cdf5c in ?? ??:0
#6 0x7f156c2ade3f in ffi_call_unix64 ??:?
#7 0x7f156c2ade3f in ?? ??:0
#8 0x7f156c2ad8aa in ffi_call ??:?
#9 0x7f156c2ad8aa in ?? ??:0
#10 0x7f156c4db311 in _call_function_pointer 
/home/test/check/PythonASAN/Modules/_ctypes/callproc.c:809
#11 0x7f156c4db311 in _ctypes_callproc 
/home/test/check/PythonASAN/Modules/_ctypes/callproc.c:1147
#12 0x7f156c4db311 in ?? ??:0
#13 0x7f156c4ca199 in PyCFuncPtr_call 
/home/test/check/PythonASAN/Modules/_ctypes/_ctypes.c:3870
#14 0x7f156c4ca199 in ?? ??:0
#15 0x5745f0 in _PyObject_FastCallDict 
/home/test/check/PythonASAN/Objects/abstract.c:2316
#16 0x5745f0 in ?? ??:0
#17 0x7a7429 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4812
#18 0x7a7429 in ?? ??:0
#19 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#20 0x7995cc in ?? ??:0
#21 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#22 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#23 0x7a9847 in ?? ??:0
#24 0x7ab648 in fast_function 
/home/test/check/PythonASAN/Python/ceval.c:4929 (discriminator 1)
#25 0x7ab648 in ?? ??:0
#26 0x7a76f2 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4809
#27 0x7a76f2 in ?? ??:0
#28 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#29 0x7995cc in ?? ??:0
#30 0x7ab4cb in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#31 0x7ab4cb in _PyFunction_FastCall 
/home/test/check/PythonASAN/Python/ceval.c:4870
#32 0x7ab4cb in fast_function 
/home/test/check/PythonASAN/Python/ceval.c:4905
#33 0x7ab4cb in ?? ??:0
#34 0x7a76f2 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4809
#35 0x7a76f2 in ?? ??:0
#36 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#37 0x7995cc in ?? ??:0
#38 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#39 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#40 0x7a9847 in ?? ??:0
#41 0x7ac2ea in _PyFunction_FastCallDict 
/home/test/check/PythonASAN/Python/ceval.c:5021
#42 0x7ac2ea in ?? ??:0
#43 0x574668 in _PyObject_FastCallDict 
/home/test/check/PythonASAN/Objects/abstract.c:2295
#44 0x574668 in ?? ??:0
#45 0x5749fa in _PyObject_Call_Prepend 
/home/test/check/PythonASAN/Objects/abstract.c:2358
#46 0x5749fa in ?? ??:0
#47 0x573e9b in PyObject_Call 
/home/test/check/PythonASAN/Objects/abstract.c:2246
#48 0x573e9b in ?? ??:0
#49 0x793369 in do_call_core 

[issue29487] AddressSanitizer: heap-buffer-overflow on address 0x60200000e734

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29497] AddressSanitizer: SEGV on unknown address 0x000000000008

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29500] AddressSanitizer: heap-buffer-overflow on address 0x61600004a982

[BeginVuln]

New submission from BeginVuln:

OS Version : Ubuntu 16.04 LTS
Python download link : 
https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz

Python version : 3.6.0

Normal build cmd : 
./configure 
make

Asan build cmd:
export CC="/usr/bin/clang -fsanitize=address
export CXX="/usr/bin/clang++ -fsanitize=address
./confiugre
make

GDB with exploitable:

To enable execution of this file add
add-auto-load-safe-path /home/test/check/PythonGDB/python-gdb.py
line to your configuration file "/home/test/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/test/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
info "(gdb)Auto-loading safe path"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Inferior 1 (process 19456) exited normally]


ASAN:

=
==18010==ERROR: AddressSanitizer: heap-buffer-overflow on address 
0x6164a982 at pc 0x00830a11 bp 0x7fff6131b9b0 sp 0x7fff6131b9a8
READ of size 2 at 0x6164a982 thread T0
#0 0x830a10 in find_op /home/test/check/PythonASAN/Python/peephole.c:101 
(discriminator 1)
#1 0x830a10 in PyCode_Optimize 
/home/test/check/PythonASAN/Python/peephole.c:712 (discriminator 1)
#2 0x830a10 in ?? ??:0
#3 0x7ccf6c in makecode /home/test/check/PythonASAN/Python/compile.c:5249
#4 0x7ccf6c in assemble /home/test/check/PythonASAN/Python/compile.c:5367
#5 0x7ccf6c in ?? ??:0
#6 0x7d0a09 in compiler_function 
/home/test/check/PythonASAN/Python/compile.c:1886
#7 0x7d0a09 in ?? ??:0
#8 0x7b0923 in compiler_body 
/home/test/check/PythonASAN/Python/compile.c:1463
#9 0x7b0923 in ?? ??:0
#10 0x7ae107 in compiler_mod 
/home/test/check/PythonASAN/Python/compile.c:1483
#11 0x7ae107 in PyAST_CompileObject 
/home/test/check/PythonASAN/Python/compile.c:341
#12 0x7ae107 in ?? ??:0
#13 0x5142d8 in run_mod /home/test/check/PythonASAN/Python/pythonrun.c:977
#14 0x5142d8 in PyRun_FileExFlags 
/home/test/check/PythonASAN/Python/pythonrun.c:933
#15 0x5142d8 in ?? ??:0
#16 0x512afa in PyRun_SimpleFileExFlags 
/home/test/check/PythonASAN/Python/pythonrun.c:396
#17 0x512afa in ?? ??:0
#18 0x53eefd in run_file /home/test/check/PythonASAN/Modules/main.c:320
#19 0x53eefd in Py_Main /home/test/check/PythonASAN/Modules/main.c:780
#20 0x53eefd in ?? ??:0
#21 0x503d16 in main /home/test/check/PythonASAN/./Programs/python.c:69
#22 0x503d16 in ?? ??:0
#23 0x7f5554ba782f in __libc_start_main 
/build/glibc-GKVZIf/glibc-2.23/csu/../csu/libc-start.c:291
#24 0x7f5554ba782f in ?? ??:0
#25 0x432548 in _start ??:?
#26 0x432548 in ?? ??:0

0x6164a982 is located 0 bytes to the right of 514-byte region 
[0x6164a780,0x6164a982)
allocated by thread T0 here:
#0 0x4d2678 in malloc ??:?
#1 0x4d2678 in ?? ??:0
#2 0x508c35 in PyMem_RawMalloc 
/home/test/check/PythonASAN/Objects/obmalloc.c:386
#3 0x508c35 in _PyObject_Alloc 
/home/test/check/PythonASAN/Objects/obmalloc.c:1427
#4 0x508c35 in ?? ??:0

SUMMARY: AddressSanitizer: heap-buffer-overflow 
(/home/test/check/PythonASAN/python+0x830a10)
Shadow bytes around the buggy address:
  0x0c2c800014e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c800014f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c80001500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c80001510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c80001520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c2c80001530:[02]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c80001540: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c80001550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c80001560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c80001570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c80001580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:   00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:   fa
  Heap right redzone:  fb
  Freed heap region:   fd
  Stack left redzone:  f1
  Stack mid redzone:   f2
  Stack right redzone: f3
  Stack partial redzone:   f4
  Stack after return:  f5
  Stack use after scope:   f8
  Global redzone:  f9
  Global init order:   f6
  Poisoned by user:f7
  Container overflow:  fc
  Array cookie:ac
  Intra object redzone:bb
  ASan internal:   fe
  Left alloca redzone: ca
  Right alloca redzone:cb
==18010==ABORTING

--
components: Interpreter Core
files: peephole_101
messages: 287339
nosy: beginvuln
priority: normal
severity: normal
status: 

[issue29489] AddressSanitizer: SEGV on unknown address 0x7f4a36c604d0

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29491] AddressSanitizer: heap-buffer-overflow on address 0x60200000e734

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29486] AddressSanitizer: SEGV on unknown address 0x7f16f88e3560

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29490] AddressSanitizer: heap-buffer-overflow on address 0x60200000e72f

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29484] AddressSanitizer: heap-buffer-overflow on address 0x60200000e738

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29492] AddressSanitizer: SEGV on unknown address 0x0000a0013639

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29499] AddressSanitizer: SEGV on unknown address 0x000ebfff800d

[BeginVuln]

New submission from BeginVuln:

OS Version : Ubuntu 16.04 LTS
Python download link : 
https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz

Python version : 3.6.0

Normal build cmd : 
./configure 
make

Asan build cmd:
export CC="/usr/bin/clang -fsanitize=address
export CXX="/usr/bin/clang++ -fsanitize=address
./confiugre
make

GDB with exploitable:

To enable execution of this file add
add-auto-load-safe-path /home/test/check/PythonGDB/python-gdb.py
line to your configuration file "/home/test/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/test/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
info "(gdb)Auto-loading safe path"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
_PyObject_Alloc (ctx=0x0, elsize=136, nelem=1, use_calloc=0) at 
Objects/obmalloc.c:1258
1258if ((pool->freeblock = *(block **)bp) != NULL) {
Description: Access violation on source operand
Short description: SourceAv (19/22)
Hash: 931f1ff7977aaf47bb64eec6d074074f.3e2cbb794853bcf6a077da4bfa99ade4
Exploitability Classification: UNKNOWN
Explanation: The target crashed on an access violation at an address matching 
the source operand of the current instruction. This likely indicates a read 
access violation.
Other tags: AccessViolation (21/22)


ASAN:

EsEASAN:DEADLYSIGNAL
=
==18115==ERROR: AddressSanitizer: SEGV on unknown address 0x000ebfff800d (pc 
0x005082ed bp 0x0072006f sp 0x7fffe2536f60 T0)
#0 0x5082ec in _PyObject_Alloc 
/home/test/check/PythonASAN/Objects/obmalloc.c:1258
#1 0x5082ec in ?? ??:0
#2 0x54318c in _PyObject_GC_Alloc 
/home/test/check/PythonASAN/Modules/gcmodule.c:1714
#3 0x54318c in ?? ??:0
#4 0x543391 in _PyObject_GC_Malloc 
/home/test/check/PythonASAN/Modules/gcmodule.c:1736
#5 0x543391 in _PyObject_GC_New 
/home/test/check/PythonASAN/Modules/gcmodule.c:1748
#6 0x543391 in ?? ??:0
#7 0x5d5516 in PyFunction_NewWithQualName 
/home/test/check/PythonASAN/Objects/funcobject.c:21
#8 0x5d5516 in ?? ??:0
#9 0x796ecf in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3373
#10 0x796ecf in ?? ??:0
#11 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#12 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#13 0x7a9847 in ?? ??:0
#14 0x7ab648 in fast_function 
/home/test/check/PythonASAN/Python/ceval.c:4929 (discriminator 1)
#15 0x7ab648 in ?? ??:0
#16 0x7a76f2 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4809
#17 0x7a76f2 in ?? ??:0
#18 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#19 0x7995cc in ?? ??:0
#20 0x7ab4cb in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#21 0x7ab4cb in _PyFunction_FastCall 
/home/test/check/PythonASAN/Python/ceval.c:4870
#22 0x7ab4cb in fast_function 
/home/test/check/PythonASAN/Python/ceval.c:4905
#23 0x7ab4cb in ?? ??:0
#24 0x7a76f2 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4809
#25 0x7a76f2 in ?? ??:0
#26 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#27 0x7995cc in ?? ??:0
#28 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#29 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#30 0x7a9847 in ?? ??:0
#31 0x7ab648 in fast_function 
/home/test/check/PythonASAN/Python/ceval.c:4929 (discriminator 1)
#32 0x7ab648 in ?? ??:0
#33 0x7a76f2 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4809
#34 0x7a76f2 in ?? ??:0
#35 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#36 0x7995cc in ?? ??:0
#37 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#38 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#39 0x7a9847 in ?? ??:0
#40 0x7ab648 in fast_function 
/home/test/check/PythonASAN/Python/ceval.c:4929 (discriminator 1)
#41 0x7ab648 in ?? ??:0
#42 0x7a76f2 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4809
#43 0x7a76f2 in ?? ??:0
#44 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#45 0x7995cc in ?? ??:0
#46 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#47 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#48 0x7a9847 in ?? ??:0
#49 0x7ab648 in fast_function 

[issue29494] AddressSanitizer: SEGV on unknown address 0x00009fff8001

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29495] AddressSanitizer: SEGV on unknown address 0x02007ea947c3

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29493] AddressSanitizer: SEGV on unknown address 0x000cffff800d

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29491] AddressSanitizer: heap-buffer-overflow on address 0x60200000e734

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29484] AddressSanitizer: heap-buffer-overflow on address 0x60200000e738

[Christian Heimes]

Changes by Christian Heimes :


--
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29490] AddressSanitizer: heap-buffer-overflow on address 0x60200000e72f

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29483] AddressSanitizer: heap-buffer-overflow on address 0x60200000e731

[Christian Heimes]

Changes by Christian Heimes :


--
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29488] AddressSanitizer: SEGV on unknown address 0x0001a5525c1b

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29489] AddressSanitizer: SEGV on unknown address 0x7f4a36c604d0

[Christian Heimes]

Changes by Christian Heimes :


--
components: +Extension Modules -Interpreter Core
priority: normal -> low
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29498] AddressSanitizer: SEGV on unknown address 0x0005ffff800d

[BeginVuln]

New submission from BeginVuln:

OS Version : Ubuntu 16.04 LTS
Python download link : 
https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz

Python version : 3.6.0

Normal build cmd : 
./configure 
make

Asan build cmd:
export CC="/usr/bin/clang -fsanitize=address
export CXX="/usr/bin/clang++ -fsanitize=address
./confiugre
make

GDB with exploitable:

To enable execution of this file add
add-auto-load-safe-path /home/test/check/PythonGDB/python-gdb.py
line to your configuration file "/home/test/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/test/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
info "(gdb)Auto-loading safe path"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
_PyObject_GenericGetAttrWithDict (dict=0x2c006f, name=0x77eed3b0, 
obj=0x7628ebf8) at Objects/object.c:1088
1088Py_INCREF(dict);
Description: Access violation on destination operand
Short description: DestAv (8/22)
Hash: 5fba3f64e0a5cd874121e05187de0b92.c7630c31a2ff26cdc6fb85881fa40252
Exploitability Classification: EXPLOITABLE
Explanation: The target crashed on an access violation at an address matching 
the destination operand of the instruction. This likely indicates a write 
access violation, which means the attacker may control the write address and/or 
value.
Other tags: AccessViolation (21/22)


ASAN:

EsEASAN:DEADLYSIGNAL
=
==18600==ERROR: AddressSanitizer: SEGV on unknown address 0x0005800d (pc 
0x0063acfe bp 0x7f86cde063b0 sp 0x7fffa5d9ea90 T0)
#0 0x63acfd in _PyObject_GenericGetAttrWithDict 
/home/test/check/PythonASAN/Objects/object.c:1088
#1 0x63acfd in ?? ??:0
#2 0x7966cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:2815 (discriminator 1)
#3 0x7966cc in ?? ??:0
#4 0x7ab4cb in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#5 0x7ab4cb in _PyFunction_FastCall 
/home/test/check/PythonASAN/Python/ceval.c:4870
#6 0x7ab4cb in fast_function /home/test/check/PythonASAN/Python/ceval.c:4905
#7 0x7ab4cb in ?? ??:0
#8 0x7a76f2 in call_function /home/test/check/PythonASAN/Python/ceval.c:4809
#9 0x7a76f2 in ?? ??:0
#10 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#11 0x7995cc in ?? ??:0
#12 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#13 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#14 0x7a9847 in ?? ??:0
#15 0x7ac2ea in _PyFunction_FastCallDict 
/home/test/check/PythonASAN/Python/ceval.c:5021
#16 0x7ac2ea in ?? ??:0
#17 0x574668 in _PyObject_FastCallDict 
/home/test/check/PythonASAN/Objects/abstract.c:2295
#18 0x574668 in ?? ??:0
#19 0x5749fa in _PyObject_Call_Prepend 
/home/test/check/PythonASAN/Objects/abstract.c:2358
#20 0x5749fa in ?? ??:0
#21 0x573e9b in PyObject_Call 
/home/test/check/PythonASAN/Objects/abstract.c:2246
#22 0x573e9b in ?? ??:0
#23 0x793369 in do_call_core /home/test/check/PythonASAN/Python/ceval.c:5057
#24 0x793369 in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3357
#25 0x793369 in ?? ??:0
#26 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#27 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#28 0x7a9847 in ?? ??:0
#29 0x7ac2ea in _PyFunction_FastCallDict 
/home/test/check/PythonASAN/Python/ceval.c:5021
#30 0x7ac2ea in ?? ??:0
#31 0x574668 in _PyObject_FastCallDict 
/home/test/check/PythonASAN/Objects/abstract.c:2295
#32 0x574668 in ?? ??:0
#33 0x5749fa in _PyObject_Call_Prepend 
/home/test/check/PythonASAN/Objects/abstract.c:2358
#34 0x5749fa in ?? ??:0
#35 0x573e9b in PyObject_Call 
/home/test/check/PythonASAN/Objects/abstract.c:2246
#36 0x573e9b in ?? ??:0
#37 0x66efe4 in slot_tp_call 
/home/test/check/PythonASAN/Objects/typeobject.c:6167
#38 0x66efe4 in ?? ??:0
#39 0x5745f0 in _PyObject_FastCallDict 
/home/test/check/PythonASAN/Objects/abstract.c:2316
#40 0x5745f0 in ?? ??:0
#41 0x7a7429 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4812
#42 0x7a7429 in ?? ??:0
#43 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#44 0x7995cc in ?? ??:0
#45 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#46 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#47 0x7a9847 in ?? ??:0
#48 0x7ac2ea in _PyFunction_FastCallDict 

[issue29487] AddressSanitizer: heap-buffer-overflow on address 0x60200000e734

[Christian Heimes]

Changes by Christian Heimes :


--
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29497] AddressSanitizer: SEGV on unknown address 0x000000000008

[BeginVuln]

New submission from BeginVuln:

OS Version : Ubuntu 16.04 LTS
Python download link : 
https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz

Python version : 3.6.0

Normal build cmd : 
./configure 
make

Asan build cmd:
export CC="/usr/bin/clang -fsanitize=address
export CXX="/usr/bin/clang++ -fsanitize=address
./confiugre
make

GDB with exploitable:

To enable execution of this file add
add-auto-load-safe-path /home/test/check/PythonGDB/python-gdb.py
line to your configuration file "/home/test/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/test/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
info "(gdb)Auto-loading safe path"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGABRT, Aborted.
0x77116418 in __GI_raise (sig=sig@entry=6) at 
../sysdeps/unix/sysv/linux/raise.c:54
Description: Heap error
Short description: HeapError (10/22)
Hash: 2aa3ac417e1aa62c7fe1524ebca9f7a3.8d7f0ad1f2db61942ed3977c83757030
Exploitability Classification: EXPLOITABLE
Explanation: The target's backtrace indicates that libc has detected a heap 
error or that the target was executing a heap function when it stopped. This 
could be due to heap corruption, passing a bad pointer to a heap function such 
as free(), etc. Since heap errors might include buffer overflows, 
use-after-free situations, etc. they are generally considered exploitable.
Other tags: AbortSignal (20/22)


ASAN:

ASAN:DEADLYSIGNAL
=
==18277==ERROR: AddressSanitizer: SEGV on unknown address 0x0008 (pc 
0x7f65f421d380 bp 0x7f65f4560b20 sp 0x7ffe10375320 T0)
#0 0x7f65f421d37f in _int_free 
/build/glibc-GKVZIf/glibc-2.23/malloc/malloc.c:4057
#1 0x7f65f421d37f in ?? ??:0
#2 0x7f65f4220abb in __GI___libc_free 
/build/glibc-GKVZIf/glibc-2.23/malloc/malloc.c:2969 (discriminator 4)
#3 0x7f65f4220abb in ?? ??:0
#4 0x7f65f0640e3f in ffi_call_unix64 ??:?
#5 0x7f65f0640e3f in ?? ??:0
#6 0x7f65f06408aa in ffi_call ??:?
#7 0x7f65f06408aa in ?? ??:0
#8 0x7f65f0885311 in _call_function_pointer 
/home/test/check/PythonASAN/Modules/_ctypes/callproc.c:809
#9 0x7f65f0885311 in _ctypes_callproc 
/home/test/check/PythonASAN/Modules/_ctypes/callproc.c:1147
#10 0x7f65f0885311 in ?? ??:0
#11 0x7f65f0874199 in PyCFuncPtr_call 
/home/test/check/PythonASAN/Modules/_ctypes/_ctypes.c:3870
#12 0x7f65f0874199 in ?? ??:0
#13 0x5745f0 in _PyObject_FastCallDict 
/home/test/check/PythonASAN/Objects/abstract.c:2316
#14 0x5745f0 in ?? ??:0
#15 0x7a7429 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4812
#16 0x7a7429 in ?? ??:0
#17 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#18 0x7995cc in ?? ??:0
#19 0x7ab4cb in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#20 0x7ab4cb in _PyFunction_FastCall 
/home/test/check/PythonASAN/Python/ceval.c:4870
#21 0x7ab4cb in fast_function 
/home/test/check/PythonASAN/Python/ceval.c:4905
#22 0x7ab4cb in ?? ??:0
#23 0x7a76f2 in call_function 
/home/test/check/PythonASAN/Python/ceval.c:4809
#24 0x7a76f2 in ?? ??:0
#25 0x7995cc in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3275
#26 0x7995cc in ?? ??:0
#27 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#28 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#29 0x7a9847 in ?? ??:0
#30 0x7ac2ea in _PyFunction_FastCallDict 
/home/test/check/PythonASAN/Python/ceval.c:5021
#31 0x7ac2ea in ?? ??:0
#32 0x574668 in _PyObject_FastCallDict 
/home/test/check/PythonASAN/Objects/abstract.c:2295
#33 0x574668 in ?? ??:0
#34 0x5749fa in _PyObject_Call_Prepend 
/home/test/check/PythonASAN/Objects/abstract.c:2358
#35 0x5749fa in ?? ??:0
#36 0x573e9b in PyObject_Call 
/home/test/check/PythonASAN/Objects/abstract.c:2246
#37 0x573e9b in ?? ??:0
#38 0x793369 in do_call_core /home/test/check/PythonASAN/Python/ceval.c:5057
#39 0x793369 in _PyEval_EvalFrameDefault 
/home/test/check/PythonASAN/Python/ceval.c:3357
#40 0x793369 in ?? ??:0
#41 0x7a9847 in PyEval_EvalFrameEx 
/home/test/check/PythonASAN/Python/ceval.c:718
#42 0x7a9847 in _PyEval_EvalCodeWithName 
/home/test/check/PythonASAN/Python/ceval.c:4119
#43 0x7a9847 in ?? ??:0
#44 0x7ac2ea in _PyFunction_FastCallDict 
/home/test/check/PythonASAN/Python/ceval.c:5021
#45 0x7ac2ea in ?? ??:0
#46 0x574668 in _PyObject_FastCallDict 
/home/test/check/PythonASAN/Objects/abstract.c:2295
#47 0x574668 in ?? ??:0
#48 0x5749fa in _PyObject_Call_Prepend 

[issue29486] AddressSanitizer: SEGV on unknown address 0x7f16f88e3560

[Christian Heimes]

Changes by Christian Heimes :


--
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29482] AddressSanitizer: attempting double-free on 0x60b000007050

[Christian Heimes]

Changes by Christian Heimes :


--
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29476] Simplify set_add_entry()

[Raymond Hettinger]

Raymond Hettinger added the comment:

> How this change affects this case?

I would expect to see a tiny net benefit.  The set.add() would be 
microscopically faster (because of less work finding and going back to a free 
slot).  The set.discard() would do the same work (locating a value and marking 
it as a dummy).  The resize step would run at the same speed (it just skips 
over dummies and reinserts only active values).  The resize would run a little 
earlier (because we're not reusing a small proportion of the dummy entries) but 
it would clean out 100% of the dummies, making the table more sparse sooner.

> Sets often are used in following pattern

Not really.  This is one use pattern out of many and is one of the least 
common.  The two dominant use cases for sets are uniquification (deduping) and 
fast membership testing.  The third most common case is data analysis using 
fast set-to-set operations (union, intersection, and difference).  Then comes 
cases with individual element membership tests followed by an individual 
element set.add (i.e. the "if x not in seen: {seen.add(x); work_on(x);}" case). 
 Dead last are the affected cases that the bounce around with a mix of 
set.add() and set.discard() or set.remove().

The comment in dictobject.c suggests that the latter case is uncommon by a 
factor of hundreds.  Hence, I think work should be taken out of the inner loop 
for the common cases and instead deferred to the high-speed resize step which 
cleans out 100% of the dummy entries all at once.

The common cases all benefit (none of those have dummies, so there is no reason 
at all to check for dummies in set_add_entry).   The uncommon case (the mix of 
individual adds and discards) is about neutral or slightly positive (as 
explained above).  

I've tried to think of a case that would be negatively affected and all I can 
think of is repeatedly adding and removing exactly the *same* element or small 
group of elements.  In that case, the freeslot would be reused 100% of the time 
and the would never need a resize.  I've not seen such a case and if I had, I 
would still care about the common cases more.

Also, I like the simplification of set_add_entry() and the separation of 
concerns (dummy reclamation occurs in exactly one place and that one place 
eliminates 100% of the dummies in a single pass).

FWIW, there is a loose analogy between this idea and the trade-off between 
reference counting and GC.  Reference counting reuses memory quicker than 
waiting for GC to reclaim memory in one pass later, but it entails encumbering 
all of the setting and deleting code.  GC-only systems make the rest of the 
code much cleaner and faster, but they have to wait to reclaim memory all at 
once.  Where the analogy fails though is that use of reuse of dummies in sets 
is by far the least common case, that early freeslot checks only recapture a 
small fraction of the deletions (only the ones that happen to have exactly the 
same hash slot), and that early freeslot checks are completely wasted in all of 
the common cases (which typically have no dummies at all).

--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com