Re: [Python-Dev] Changes to configure.ac, auto-detection and related build issues

[Eitan Adler]

On Monday, 14 May 2018, Victor Stinner  wrote:

> Hi Eitan,
>
> 2018-05-15 0:01 GMT-04:00 Eitan Adler :
> > I am working on updating, fixing, or otherwise changing python's
> > configure.ac. This work is complex, (...)
>
> Is your work public? Is there an open issue on bugs.python.org or an
> open pull request?


I'm opening bugs and PRs as I Go. Some examples are:

https://github.com/python/cpython/commit/98929b545e86e7c7296c912d8f34e8e8d3fd6439
https://github.com/python/cpython/pull/6845
https://github.com/python/cpython/pull/6848
https://github.com/python/cpython/pull/6849
https://bugs.python.org/issue33485

And so on


>
> If not, would you mind to at least describe the changes that you plan to
> do?
>
> > Please feel free to tag me in
> > related PRs or bugs or emails over the next few weeks.
>
> Hopefully, we only rarely need to modify configure.ac


I'm primarily worried about breaking arcane platforms I don't have direct
access to.

>
>

Victor
>


-- 
Sent from my Turing Machine
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Changes to configure.ac, auto-detection and related build issues

[Victor Stinner]

Hi Eitan,

2018-05-15 0:01 GMT-04:00 Eitan Adler :
> I am working on updating, fixing, or otherwise changing python's
> configure.ac. This work is complex, (...)

Is your work public? Is there an open issue on bugs.python.org or an
open pull request?

If not, would you mind to at least describe the changes that you plan to do?

> Please feel free to tag me in
> related PRs or bugs or emails over the next few weeks.

Hopefully, we only rarely need to modify configure.ac.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Changes to configure.ac, auto-detection and related build issues

[Eitan Adler]

Hi all,

Hope this is an appropriate list to send this message to; if not,
please redirect me.

I am working on updating, fixing, or otherwise changing python's
configure.ac. This work is complex, lacks dedicated unit tests, and is
easy to miss corner cases. As these changes make it into master I'll
be watching the various build bots and be on the look out for related
failures. That said, I will miss things. Please feel free to tag me in
related PRs or bugs or emails over the next few weeks.

Thanks

-- 
Eitan Adler
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (Looking for) A Retrospective on the Move to Python 3

[Chris Barker - NOAA Federal via Python-Dev]

> between 3.0 and 3.6 (.5?) -- py3 grew a lot of minor features that made it
>> easier to write py2/py3 compatible code.
>> u"string", b'bytes %i' % something -- and when where the various
>> __future__ imports made available?
>>
>
> You'll need to be more specific.  __future__ has been around for a
> long time.


I meant the various ones that support py2/3 compatibility — I know division
predates py3, not sure about the others.

But it was a rhetorical question anyway :-)
-CHB



https://github.com/python/cpython/blame/master/Lib/__future__.py


>
> --
> ~Ethan~
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.
> turner%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/05/18 04:55, Jeroen Demeyer wrote:

Hello all,

I have updated PEP 575 in response to some posts on this mailing list 
and to some discussions in person with the core Cython developers.

See https://www.python.org/dev/peps/pep-0575/

The main differences with respect to the previous version are:

* "builtin_function" was renamed to "cfunction". Since we are changing 
the name anyway, "cfunction" looked like a better choice because the 
word "built-in" typically refers to things from the builtins module.


* defined_function now only defines an API (it must support all 
attributes that a Python function has) without specifying the 
implementation.


* The "Two-phase Implementation" proposal for better backwards 
compatibility has been expanded and now offers 100% backwards 
compatibility for the classes and for the inspect functions.


The PEP says:


User flags: METH_CUSTOM and METH_USRx
These flags are meant for applications that want to use tp_methods for an 
extension type or m_methods for a module but that do not want the default 
built-in functions to be created. Those applications would set METH_CUSTOM. The 
application is also free to use METH_USR0, ..., METH_USR7 for its own purposes, 
for example to customize the creation of special function instances.

There is no immediate concrete use case, but we expect that tools which 
auto-generate functions or extension types may want to define custom flags. 
Given that it costs essentially nothing to have these flags, it seems like a 
good idea to allow it.


Why are these flags added?
They aren't free – the space of available flags is not infinite. If 
something (Cython?) needs eight of them, it would be nice to mention the 
use case, at least as an example.


What should Python do with a m_methods entry that has METH_CUSTOM set? 
Again it would be nice to have an example or use case.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[M.-A. Lemburg]

On 14.05.2018 18:26, Chris Barker via Python-Dev wrote:
> 
> 
> On Fri, May 11, 2018 at 11:05 AM, Ryan Gonzalez  > wrote:
> 
>  https://refi64.com/uprocd/ 
> 
> 
> very cool -- but *nix only, of course :-(
> 
> But it seems that there is a demand for this sort of thing, and a few
> major projects are rolling their own. So maybe it makes sense to put
> something into the standard library that everyone could contribute to
> and use.
> 
> With regard to forking -- is there another way? I don't have the
> expertise to have any idea if this is possible, but:
> 
> start up python
> 
> capture the entire runtime image as a single binary blob.
> 
> could that blob be simply loaded into memory and run?
> 
> (hmm -- probably not -- memory addresses would be hard-coded then, yes?)
> or is memory virtualized enough these days?

You might want to look into combining this with PyRun:

https://www.egenix.com/products/python/PyRun/

which takes care of mmap'ing the byte code of the stdlib into
memory.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-33257: seeking advice & approval on the course of action

[Ivan Pozdeev via Python-Dev]

On 14.05.2018 22:05, Ivan Pozdeev wrote:

On 14.05.2018 21:58, Terry Reedy wrote:

On 5/14/2018 12:20 PM, Chris Barker via Python-Dev wrote:
On Wed, May 2, 2018 at 8:21 PM, Terry Reedy > wrote:


    On 5/2/2018 4:38 PM, Ivan Pozdeev via Python-Dev wrote:

    The bottom line is: Tkinter is currently broken


    This is way over-stated.  Many modules have bugs, somethings in
    features more central to their main purpose.

I'll suggest a re-statement:

tkinter is not thread safe,


Still over-stated.  If one uses tcl/tk compiled with thread support, 
tkinter *is* thread-safe.  This is 'as far as I know' from running 
posted 'failing' examples (possible with bug fixes) with 3.5+ on 
Windows, which is installed with tcl/tk 8.6, which defaults to 
thread-safe.


This means that you didn't (yet) read the letter that I attached to 
https://bugs.python.org/issue33479 .

Reciting the relevant section:

===
The reality is that with neither flavor of Tcl is Tkinter completely 
thread-safe, but with threaded flavor, it's more so:


* with nonthreaded Tcl, making concurrent Tcl calls leads to crashes 
due to incorrect management of the "Tcl lock" as per 
https://bugs.python.org/issue33257
* with threaded Tcl, the only issue that I found so far is that a few 
APIs must be called from the interpreter's thread 
(https://bugs.python.org/issue33412#msg316152; so far, I know 
`mainloop()` and `destroy()` to be this) -- while most can be called 
from anywhere. Whether the exceptions are justified is a matter of 
discussion (e.g. at first glance, `destroy()` can be fixed).
And another undocumented limitation for threaded Tcl: when calling 
anything from outside the interpreter thread, `mainloop()` must be 
running in the interpreter threads, or the call will either raise or 
hang (dunno any more details atm).

===


Tkinter was intended to also be thread-safe when using tcl/tk without 
thread support, which was the default for tcl/tk 8.5 and before. The 
posted examples can fail on 2.x on Windows, which comes with tcl/tk 
8.5 or before. _tkinter.c has some different #ifdefs for the two 
situations.



and yet it is documented as being thread safe


True in https://docs.python.org/3/library/tk.html
Unspecified in https://docs.python.org/3/library/tkinter.html


This is either a bug(s) in the implementation or the docs.


Both


So what are the solutions?

1) fix the docs -- unless tkInter is made thread safe really soon, 
and fixes are back-ported, this seems like a no brainer -- at least 
temporarily.


https://bugs.python.org/issue33479 'Document tkinter and threads'


2) fix the issues that make tkInter not thread safe


with non-thread tcl/tk.

https://bugs.python.org/issue33257 has a patch that might improve the 
situation for one type of call.  Fixing everything might not be 
possible.  AFAIK, there are currently no tests of thread safety.






--
Regards,
Ivan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-33257: seeking advice & approval on the course of action

[Ivan Pozdeev via Python-Dev]

On 14.05.2018 21:58, Terry Reedy wrote:

On 5/14/2018 12:20 PM, Chris Barker via Python-Dev wrote:
On Wed, May 2, 2018 at 8:21 PM, Terry Reedy > wrote:


    On 5/2/2018 4:38 PM, Ivan Pozdeev via Python-Dev wrote:

    The bottom line is: Tkinter is currently broken


    This is way over-stated.  Many modules have bugs, somethings in
    features more central to their main purpose.

I'll suggest a re-statement:

tkinter is not thread safe,


Still over-stated.  If one uses tcl/tk compiled with thread support, 
tkinter *is* thread-safe.  This is 'as far as I know' from running 
posted 'failing' examples (possible with bug fixes) with 3.5+ on 
Windows, which is installed with tcl/tk 8.6, which defaults to 
thread-safe.


This means that you didn't (yet) read the letter that I attached to 
https://bugs.python.org/issue33479 .

Reciting the relevant section:

===
The reality is that with neither flavor of Tcl is Tkinter completely 
thread-safe, but with threaded flavor, it's more so:


* with nonthreaded Tcl, making concurrent Tcl calls leads to crashes due 
to incorrect management of the "Tcl lock" as per 
https://bugs.python.org/issue33257
* with threaded Tcl, the only issue that I found so far is that a few 
APIs must be called from the interpreter's thread 
(https://bugs.python.org/issue33412#msg316152; so far, I know 
`mainloop()` and `destroy()` to be this) -- while most can be called 
from anywhere. Whether the exceptions are justified is a matter of 
discussion (e.g. at first glance, `destroy()` can be fixed).

===
Tkinter was intended to also be thread-safe when using tcl/tk without 
thread support, which was the default for tcl/tk 8.5 and before. The 
posted examples can fail on 2.x on Windows, which comes with tcl/tk 
8.5 or before. _tkinter.c has some different #ifdefs for the two 
situations.



and yet it is documented as being thread safe


True in https://docs.python.org/3/library/tk.html
Unspecified in https://docs.python.org/3/library/tkinter.html


This is either a bug(s) in the implementation or the docs.


Both


So what are the solutions?

1) fix the docs -- unless tkInter is made thread safe really soon, 
and fixes are back-ported, this seems like a no brainer -- at least 
temporarily.


https://bugs.python.org/issue33479 'Document tkinter and threads'


2) fix the issues that make tkInter not thread safe


with non-thread tcl/tk.

https://bugs.python.org/issue33257 has a patch that might improve the 
situation for one type of call.  Fixing everything might not be 
possible.  AFAIK, there are currently no tests of thread safety.




--
Regards,
Ivan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-33257: seeking advice & approval on the course of action

[Terry Reedy]

On 5/14/2018 12:20 PM, Chris Barker via Python-Dev wrote:
On Wed, May 2, 2018 at 8:21 PM, Terry Reedy > wrote:


On 5/2/2018 4:38 PM, Ivan Pozdeev via Python-Dev wrote:

The bottom line is: Tkinter is currently broken


This is way over-stated.  Many modules have bugs, somethings in
features more central to their main purpose.

I'll suggest a re-statement:

tkinter is not thread safe,


Still over-stated.  If one uses tcl/tk compiled with thread support, 
tkinter *is* thread-safe.  This is 'as far as I know' from running 
posted 'failing' examples (possible with bug fixes) with 3.5+ on 
Windows, which is installed with tcl/tk 8.6, which defaults to thread-safe.


Tkinter was intended to also be thread-safe when using tcl/tk without 
thread support, which was the default for tcl/tk 8.5 and before.  The 
posted examples can fail on 2.x on Windows, which comes with tcl/tk 8.5 
or before. _tkinter.c has some different #ifdefs for the two situations.



and yet it is documented as being thread safe


True in https://docs.python.org/3/library/tk.html
Unspecified in https://docs.python.org/3/library/tkinter.html


This is either a bug(s) in the implementation or the docs.


Both


So what are the solutions?

1) fix the docs -- unless tkInter is made thread safe really soon, and 
fixes are back-ported, this seems like a no brainer -- at least temporarily.


https://bugs.python.org/issue33479 'Document tkinter and threads'


2) fix the issues that make tkInter not thread safe


with non-thread tcl/tk.

https://bugs.python.org/issue33257 has a patch that might improve the 
situation for one type of call.  Fixing everything might not be 
possible.  AFAIK, there are currently no tests of thread safety.


--
Terry Jan Reedy


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-28055: Fix unaligned accesses in siphash24(). (GH-6123)

[Rob Boehne]

On 5/13/18, 12:44 PM, "Python-Dev on behalf of Christian Heimes" 
 wrote:

On 2018-05-13 06:57, Serhiy Storchaka wrote:
> 
https://github.com/python/cpython/commit/1e2ec8a996daec65d8d5a3d43b66a9909c6d0653
> commit: 1e2ec8a996daec65d8d5a3d43b66a9909c6d0653
> branch: master
> author: Rolf Eike Beer 
> committer: Serhiy Storchaka 
> date: 2018-05-13T13:57:31+03:00
> summary:
> 
> bpo-28055: Fix unaligned accesses in siphash24(). (GH-6123)
> 
> The hash implementation casts the input pointer to uint64_t* and directly 
reads
> from this, which may cause unaligned accesses. Use memcpy() instead so 
this code
> will not crash with SIGBUS on sparc.
> 
> https://bugs.gentoo.org/show_bug.cgi?id=636400
> 
> files:
> A Misc/NEWS.d/next/Core and 
Builtins/2018-04-25-20-44-42.bpo-28055.f49kfC.rst
> M Python/pyhash.c

Hi Serhiy,

I was against the approach a good reason. The PR adds additional CPU
instructions and changes memory access pattern in a  critical path of
CPython. There is no reason to add additional overhead for the majority
of users or X86 and X86_64 architectures. The memcpy() call should only
be used on architectures that do not support unaligned memory access.
See comment https://bugs.python.org/issue28055#msg276257

X86 won't *directly* write misaligned data either, it will intrinsically copy 
it out to a properly aligned location.  In C this is also "undefined behavior", 
so technically the C implementation can do whatever it wants - like raise an 
exception - which is will on SPARC.  While X86 users may not notice any 
problems, depending on undefined behavior working in any particular way has 
many drawbacks.  Often C compilers will optimize code in ways that assume there 
is no undefined behavior in ways that breaks code that does.

At least for latest GCC, the change seems to be fine. GCC emits the same
assembly code for X86_64 before and after your change. Did you check the
output on other CPU architectures as well as clang and MSVC, too?

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/robb%40datalogics.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/05/18 04:55, Jeroen Demeyer wrote:

Hello all,

I have updated PEP 575 in response to some posts on this mailing list 
and to some discussions in person with the core Cython developers.

See https://www.python.org/dev/peps/pep-0575/

The main differences with respect to the previous version are:

* "builtin_function" was renamed to "cfunction". Since we are changing 
the name anyway, "cfunction" looked like a better choice because the 
word "built-in" typically refers to things from the builtins module.


* defined_function now only defines an API (it must support all 
attributes that a Python function has) without specifying the 
implementation.


* The "Two-phase Implementation" proposal for better backwards 
compatibility has been expanded and now offers 100% backwards 
compatibility for the classes and for the inspect functions.



Hi,
I'm reading the PEP thoroughly, trying to "swap it into my brain" for 
the next few days.
It does quite a lot of things, and the changes are all intertwined, 
which will make it hard to get reviewed and accepted.
Are there parts that can be left to a subsequent PEP, to simplify the 
document (and implementation)?
It seems to me that the current complexity is (partly) due to the fact 
that how functions are *called* is tied to how they are *introspected*. 
Perhaps starting to separate that is a better way to untangle things 
than arranging a class hierarchy?



Can the problem of allowing introspection ("It is currently not possible 
to implement a function efficiently in C (only built-in functions can do 
that) while still allowing introspection like inspect.signature or 
inspect.getsourcefile (only Python functions can do that)") be solved in 
a better way?


Maybe we can change `inspect` to use duck-typing instead of isinstance? 
Then, if built-in functions were subclassable, Cython functions could 
need to provide appropriate __code__/__defaults__/__kwdefaults__ 
attributes that inspect would pick up.
Maybe we could eve add more attributes (__isgenerator__?) to separate 
how a function is called from how it should be introspected -- e.g. make 
inspect not consult co_flags.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Oleg Broytman]

On Mon, May 14, 2018 at 12:26:19PM -0400, Chris Barker via Python-Dev 
 wrote:
> With regard to forking -- is there another way? I don't have the expertise
> to have any idea if this is possible, but:
> 
> start up python
> 
> capture the entire runtime image as a single binary blob.
> could that blob be simply loaded into memory and run?

   Like emacs unexec? https://www.google.com/search?q=emacs+unexec

> -CHB
> 
> -- 
> 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
> 
> chris.bar...@noaa.gov

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (Looking for) A Retrospective on the Move to Python 3

[Wes Turner]

On Monday, May 14, 2018, Ethan Furman  wrote:

> On 05/14/2018 09:34 AM, Chris Barker via Python-Dev wrote:
>
> between 3.0 and 3.6 (.5?) -- py3 grew a lot of minor features that made it
>> easier to write py2/py3 compatible code.
>> u"string", b'bytes %i' % something -- and when where the various
>> __future__ imports made available?
>>
>> If these had been in place in 3.0, the whole process would have been
>> easier :-(
>>
>
> You'll need to be more specific.  __future__ has been around for a
> long time.


https://github.com/python/cpython/blame/master/Lib/__future__.py


>
> --
> ~Ethan~
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.
> turner%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[INADA Naoki]

2018年5月15日(火) 2:17 Antoine Pitrou :

>
> Le 14/05/2018 à 19:12, INADA Naoki a écrit :
> > I'm sorry, the word *will* may be stronger than I thought.
> >
> > I meant if memory image dumped on disk is used casually,
> > it may make easier to make security hole.
> >
> > For example, if `hg` memory image is reused, and it can be leaked in some
> > way,
> > hg serve will be hashdos weak.
>
> This discussion subthread is not about having a memory image dumped on
> disk, but a daemon utility that preloads a new Python process when you
> first start up your CLI application.  Each time a new process is
> preloaded, it will by construction use a new hash seed.
>

My reply was to:

> capture the entire runtime image as a single binary blob.
> could that blob be simply loaded into memory and run?

So I thought about reusing memory image undeterministic times.

Of course, prefork is much safer because hash initial vector is only in
process ram.

Regards,
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Antoine Pitrou]

Le 14/05/2018 à 19:12, INADA Naoki a écrit :
> I'm sorry, the word *will* may be stronger than I thought.
> 
> I meant if memory image dumped on disk is used casually,
> it may make easier to make security hole.
> 
> For example, if `hg` memory image is reused, and it can be leaked in some
> way,
> hg serve will be hashdos weak.

This discussion subthread is not about having a memory image dumped on
disk, but a daemon utility that preloads a new Python process when you
first start up your CLI application.  Each time a new process is
preloaded, it will by construction use a new hash seed.

(by contrast, the Node.js CVE issue you linked to is about having the
same hash seed accross a Node.js version; that's disastrous)

Also you add a reuse limit to ensure that the hash seed is rotated (e.g.
every 100 invocations).

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[INADA Naoki]

I'm sorry, the word *will* may be stronger than I thought.

I meant if memory image dumped on disk is used casually,
it may make easier to make security hole.

For example, if `hg` memory image is reused, and it can be leaked in some
way,
hg serve will be hashdos weak.

I don't deny that it's useful and safe when it's used carefully.

Regards,

On Tue, May 15, 2018 at 1:58 AM Antoine Pitrou  wrote:

> On Tue, 15 May 2018 01:33:18 +0900
> INADA Naoki  wrote:
> >
> > It will broke hash randomization.
> >
> > See also: https://www.cvedetails.com/cve/CVE-2017-11499/

> I don't know why it would.  The mechanism of pre-initializing a process
> which is re-used accross many requests is how most server applications
> of Python already work (you don't want to bear the cost of spawning
> a new interpreter for each request, as antiquated CGI does). I have not
> heard that it breaks hash randomization, so a similar mechanism on the
> CLI side shouldn't break it either.

> Regards

> Antoine.


> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com



-- 
-- 
INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-28055: Fix unaligned accesses in siphash24(). (GH-6123)

[Serhiy Storchaka]

13.05.18 20:42, Christian Heimes пише:

I was against the approach a good reason. The PR adds additional CPU
instructions and changes memory access pattern in a  critical path of
CPython. There is no reason to add additional overhead for the majority
of users or X86 and X86_64 architectures. The memcpy() call should only
be used on architectures that do not support unaligned memory access.
See comment https://bugs.python.org/issue28055#msg276257

At least for latest GCC, the change seems to be fine. GCC emits the same
assembly code for X86_64 before and after your change. Did you check the
output on other CPU architectures as well as clang and MSVC, too?


For the initial implementation of pyhash.c [1] I proposed a patch that 
use memcpy() conditionally to avoid an overhead on Windows:


+#ifdef _MSC_VER
+block.value = *(const Py_uhash_t*)p;
+#else
+memcpy(block.bytes, p, SIZEOF_PY_UHASH_T);
+#endif

(and similar code for FNV).

But many developers confirmed that all modern compilers including latest 
versions of MS VS optimize memcpy() with a constant size into a single 
CPU instruction. Seems avoiding to use memcpy() no longer needed.


If using memcpy() adds an overhead on some platforms we can return to 
using a compiler/platform depending code.


[1] https://bugs.python.org/issue19183

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Antoine Pitrou]

On Tue, 15 May 2018 01:33:18 +0900
INADA Naoki  wrote:
> 
> It will broke hash randomization.
> 
> See also: https://www.cvedetails.com/cve/CVE-2017-11499/

I don't know why it would.  The mechanism of pre-initializing a process
which is re-used accross many requests is how most server applications
of Python already work (you don't want to bear the cost of spawning
a new interpreter for each request, as antiquated CGI does). I have not
heard that it breaks hash randomization, so a similar mechanism on the
CLI side shouldn't break it either.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (Looking for) A Retrospective on the Move to Python 3

[Ethan Furman]

On 05/14/2018 09:34 AM, Chris Barker via Python-Dev wrote:


between 3.0 and 3.6 (.5?) -- py3 grew a lot of minor features that made it 
easier to write py2/py3 compatible code.
u"string", b'bytes %i' % something -- and when where the various __future__ 
imports made available?

If these had been in place in 3.0, the whole process would have been easier :-(


You'll need to be more specific.  __future__ has been around for a long 
time.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (Looking for) A Retrospective on the Move to Python 3

[Chris Barker via Python-Dev]

On Sat, May 12, 2018 at 8:14 AM, Skip Montanaro 
wrote:

> > I have found 2to3 conversion to be remarkably easy and painless.
>
> > And the whole Unicode thing is much easier.
>

Another point here:

between 3.0 and 3.6 (.5?) -- py3 grew a lot of minor features that made it
easier to write py2/py3 compatible code. u"string", b'bytes %i' % something
-- and when where the various __future__ imports made available?

If these had been in place in 3.0, the whole process would have been easier
:-(

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Chris Barker via Python-Dev]

On Mon, May 14, 2018 at 12:33 PM, INADA Naoki 
wrote:

> It will broke hash randomization.
>
> See also: https://www.cvedetails.com/cve/CVE-2017-11499/


I'm not enough of a security expert to know how much that matters in this
case, but I suppose one could do a bit of post-proccessing on the image to
randomize the hashes? or is that just insane?

Also -- I wasn't thinking it would be a pre-build binary blob that everyone
used -- but one built on the fly on an individual system, maybe once per
reboot, or once per shell instance even. So if you are running, e.g. hg a
bunch of times in a shell, does it matter that the instances are all
identical?

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[INADA Naoki]

On Tue, May 15, 2018 at 1:29 AM Chris Barker via Python-Dev <
python-dev@python.org> wrote:


> On Fri, May 11, 2018 at 11:05 AM, Ryan Gonzalez  wrote:

>>  https://refi64.com/uprocd/ 


> very cool -- but *nix only, of course :-(

> But it seems that there is a demand for this sort of thing, and a few
major projects are rolling their own. So maybe it makes sense to put
something into the standard library that everyone could contribute to and
use.

> With regard to forking -- is there another way? I don't have the
expertise to have any idea if this is possible, but:

> start up python

> capture the entire runtime image as a single binary blob.

> could that blob be simply loaded into memory and run?

> (hmm -- probably not -- memory addresses would be hard-coded then, yes?)
or is memory virtualized enough these days?

> -CHB


It will broke hash randomization.

See also: https://www.cvedetails.com/cve/CVE-2017-11499/

Regards,

-- 
Inada Naoki
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Chris Barker via Python-Dev]

On Fri, May 11, 2018 at 11:05 AM, Ryan Gonzalez  wrote:

>  https://refi64.com/uprocd/ 


very cool -- but *nix only, of course :-(

But it seems that there is a demand for this sort of thing, and a few major
projects are rolling their own. So maybe it makes sense to put something
into the standard library that everyone could contribute to and use.

With regard to forking -- is there another way? I don't have the expertise
to have any idea if this is possible, but:

start up python

capture the entire runtime image as a single binary blob.

could that blob be simply loaded into memory and run?

(hmm -- probably not -- memory addresses would be hard-coded then, yes?) or
is memory virtualized enough these days?

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bpo-33257: seeking advice & approval on the course of action

[Chris Barker via Python-Dev]

On Wed, May 2, 2018 at 8:21 PM, Terry Reedy  wrote:

> On 5/2/2018 4:38 PM, Ivan Pozdeev via Python-Dev wrote:
>
>> The bottom line is: Tkinter is currently broken
>>
>
> This is way over-stated.  Many modules have bugs, somethings in features
> more central to their main purpose.


I'll suggest a re-statement:

tkinter is not thread safe, and yet it is documented as being thread safe
(or at least not documented as NOT being thread safe)

This is either a bug(s) in the implementation or the docs.

So what are the solutions?

1) fix the docs -- unless tkInter is made thread safe really soon, and
fixes are back-ported, this seems like a no brainer -- at least temporarily.

2) fix the issues that make tkInter not thread safe -- apparently there is
a thread safe tcl/tk, so it should be possible, though I have to say I'm
really surprised that that's the case for an old C code base -- but great!

The problem here is that we'll need qualified people to submit and review
the code, and it really should get some extensive testing -- that's a lot
of work.

And it's going to take time, so see (1) above.

Another issue:

Many GUI toolkits are not thread safe (I have personal experience with
wxPython), so it's not so bad to simply say "don't do that" for tkInter --
that is, don't make tkInter calls from more than one thread.

However, wxPython (for example) makes this not-too-bad for multi-threaded
programs by providing thread-safe ways to put events on the event queue --
whether with wx.PostEvent, or the utilities wx.CallAfter() and
wx.CallLater(). This makes it pretty easy to keep the GUI in one thread
while taking advantage of threads for long running tasks, etc.

IIUC, tkinter does not have such an easy way to interact with the GUI from
another thread -- so maybe adding that would be a good first step.

I neither use tkinter, nor have the expertise to contribute this -- but
clearly Ivan does -- and maybe others would want to help.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Looking for examples: proof that a list comp is a function

[Nick Coghlan]

On 14 May 2018 at 04:05, Tim Peters  wrote:

> I say "pretty much" because, for whatever reason(s), it seems to be
> trying hard _not_ to use the word "function".  But I can't guess what
> "then passed as an argument to the implicitly nested scope" could
> possibly mean otherwise (it doesn't make literal sense to "pass an
> argument" to "a scope").
>

I think my motivation was to avoid promising *exact* equivalence with a
regular nested function, since the define-and-call may allow us
opportunities for optimization that don't exist when those two are
separated (e.g. Guido's point in another thread that we actually avoid
calling "iter" twice even though the nominal expansion implies that we
should). However, you're right that just calling it a function may be
clearer than relying on the ill-defined phrase "implicitly nested scope".

For Chris's actual question, this is part of why I think adding
"parentlocal" would actually make the scoping proposal easier to explain,
as it means the name binding semantics aren't a uniquely magical property
of binding expressions (however spelled), they're just a new form of target
scope declaration that the compiler understands, and the binding expression
form implies. Note: eas*ier*, not easy ;)

It also occurs to me that we could do something pretty neat for class
scopes: have parent local declarations in methods target the implicit
lexical scope where __class__ lives (to support zero-arg super), *not* the
class body. That would entail adding a "classlocal" declaration to target
that implied scope, though.

That would give the following definition for "lexical scopes that parent
local scoping can target":

- module globals (parentlocal -> global)
- function locals, including lambda expression locals (parentlocal ->
nonlocal)
- implicit class closure, where __class__ lives (parentlocal -> nonlocal in
current scope, classlocal in class scope)

Most notably, in the synthetic functions created for generator expressions
and comprehensions, a parentlocal declaration in a child scope would imply
a parentlocal declaration in the synthetic function as well, propagating
back up the chain of nested lexical scopes until it terminated in one of
the above three permitted targets.

Using the explicit forms would then look like:

from __future import parent_scopes # Enable the explicit declaration
forms

class C:
classlocal _n # Declares _n as a cell akin to __class__ rather than
a class attribute
_n = []
@staticmethod
def get_count():
return len(_n)

assert not hasattr(C, "_n")
assert C.get_count() == 0

def  _writes_to_parent_scope():
parentlocal outer_name
outer_name = 42

   assert outer_name == 42

I'm still doubtful the complexity of actually doing that is warranted, but
I'm now satisfied the semantics can be well specified in a way that allows
us to retain the explanation of generator expressions and comprehensions in
terms of their statement level counterparts (with the added bonus of making
"__class__" a little less of a magically unique snowflake along the way).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Looking for examples: proof that a list comp is a function

[Tim Peters]

[Chris Angelico  ...
> With current semantics, you can easily prove that a list comp is
> implemented with a function by looking at how it interacts with other
> scopes (mainly class scope), but Tim's proposal may change that.

Absolutely not.  I haven't considered for a nanosecond that anything
_essential_ would change in the current implementation.  In effect, my
proposal to

bind assignment statement targets
that appear in a listcomp or genexp
in the blocks that immediately contain their synthetic functions

"merely" sprinkles in some `nonlocal` and/or `global` declarations to
change the targets' scopes.  Indeed, it _relies_ on that they're
implemented as (potentially nested) synthetic functions today.  And if
you haven't read my proposed changes to the reference manual, they
explicitly state that they're talking about the synthetic functions
created to implement genexps and listcomps.


> So I'm looking for examples that prove that a list comp is executed
> inside an implicit function. Ideally, examples that are supported by
> language guarantees, but something that's "CPython has done it this
> way since 3.0" is important too.

I don't believe you'll find that - but, of course, may be wrong about that.

> I'm aware of just two: the name lookup interaction that may be
> changing, and the fact that there's an extra line in a traceback. And
> the latter, as far as I know, is not guaranteed (and I doubt anyone
> would care if it changed). Are there any other provable points?

Nick pointed me to these future docs that _will_ pretty much imply it:

https://docs.python.org/dev/reference/expressions.html#displays-for-lists-sets-and-dictionaries

In part:

"""
However, aside from the iterable expression in the leftmost for
clause, the comprehension is executed in a separate implicitly nested
scope. This ensures that names assigned to in the target list don’t
“leak” into the enclosing scope.

The iterable expression in the leftmost for clause is evaluated
directly in the enclosing scope and then passed as an argument to the
implictly nested scope.
"""

I say "pretty much" because, for whatever reason(s), it seems to be
trying hard _not_ to use the word "function".  But I can't guess what
"then passed as an argument to the implicitly nested scope" could
possibly mean otherwise (it doesn't make literal sense to "pass an
argument" to "a scope").
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Looking for examples: proof that a list comp is a function

[Franklin? Lee]

On Mon, May 14, 2018, 03:36 Chris Angelico  wrote:

> Guido has stated that this parallel is desired and important:
>
> result = [f(x) for x in iter if g(x)]
> result = list(f(x) for x in iter if g(x))
>
> Obviously the genexp has to be implemented with a nested function,
> since there's no guarantee that it'll be iterated over in this way.
> With current semantics, you can easily prove that a list comp is
> implemented with a function by looking at how it interacts with other
> scopes (mainly class scope), but Tim's proposal may change that.
>
> So I'm looking for examples that prove that a list comp is executed
> inside an implicit function. Ideally, examples that are supported by
> language guarantees, but something that's "CPython has done it this
> way since 3.0" is important too.
>
> I'm aware of just two: the name lookup interaction that may be
> changing, and the fact that there's an extra line in a traceback. And
> the latter, as far as I know, is not guaranteed (and I doubt anyone
> would care if it changed). Are there any other provable points?
>

Related to the traceback one: the extra stack frame shows up in a debugger,
and a profiler counts the extra frame separately. The first often confuses
me because I don't immediately see which frame I'm in just by seeing the
line of code.

There are odd interactions between `yield`/`yield from` and comprehensions
that was discussed some months ago: "[Python-Dev] Tricky way of of creating
a generator via a comprehension expression". Wait, is this a continuation
of that discussion?

>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Looking for examples: proof that a list comp is a function

[Chris Angelico]

Guido has stated that this parallel is desired and important:

result = [f(x) for x in iter if g(x)]
result = list(f(x) for x in iter if g(x))

Obviously the genexp has to be implemented with a nested function,
since there's no guarantee that it'll be iterated over in this way.
With current semantics, you can easily prove that a list comp is
implemented with a function by looking at how it interacts with other
scopes (mainly class scope), but Tim's proposal may change that.

So I'm looking for examples that prove that a list comp is executed
inside an implicit function. Ideally, examples that are supported by
language guarantees, but something that's "CPython has done it this
way since 3.0" is important too.

I'm aware of just two: the name lookup interaction that may be
changing, and the fact that there's an extra line in a traceback. And
the latter, as far as I know, is not guaranteed (and I doubt anyone
would care if it changed). Are there any other provable points?

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com