[Python-ideas] Re: More efficient list copying

[Gregory P. Smith]

On Sat, Oct 2, 2021, 10:20 PM Christopher Barker 
wrote:

>
> But sure, if we can eliminate inefficiencies in Python standard data
> types, then why not?
>

Because the C implementation becomes hard to maintain.

All of our linear containers could benefit from non-linear implementations
in some scenarios. But these usually have downsides (not trivial O(1)
indexing) in others and greatly add to implementation complexity.

For linear containers, ex: Abseil's absl::cord structure applied to any of
those. https://github.com/abseil/abseil-cpp/blob/master/absl/strings/cord.h

It'd be neat to have something like that.

But having such a thing _replace_ the simple list, str, or bytes, or
bytearray implementations itself presents practical challenges.


> I’ve lost track of the exact use case here, but maybe the proposal a while
> back for sequence views would help?
>

A memoryview-like for sequences? Yeah, that is what I'd start with as a
concept. It'll wind up becoming a sequence-cord though. :P

-gps
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/F5YOP7YXUNA7Y3BFO4R2PIVN5I7IIKCF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Improving sys.executable for embedded Python scenarios

[Gregory P. Smith]

On Sat, May 1, 2021 at 10:49 AM Gregory Szorc 
wrote:

> The way it works today, if you have an application embedding Python, your
> sys.argv[0] is (likely) your main executable and sys.executable is probably
> None or the empty string (per the stdlib docs which say not to set
> sys.executable if there isn't a path to a known `python` executable).
>
> Unfortunately, since sys.executable is a str, the executable it points to
> must behave as `python` does. This means that your application embedding
> and distributing its own Python must provide a `python` or `python`-like
> standalone executable and use it for sys.executable and this executable
> must be independent from your main application because the run-time
> behavior is different. (Yes, you can employ symlink hacks and your
> executable can sniff argv[0] and dispatch to your app or `python`
> accordingly. But symlinks aren't reliable on Windows and this still
> requires multiple files/executables.) **This limitation effectively
> prevents the existence of single file application binaries who also want to
> expose a full `python`-like environment, as there's no standard way to
> advertise a mechanism to invoke `python` that isn't a standalone executable
> with no arguments.**
>

minor nit: I wouldn't use the words "must behave" above... Since using
sys.executable = None at work for the past five years.  The issues we run
into are predominantly in unit tests that try to launch an interpreter via
subprocess of sys.executable.  and the bulk of that is in CPython's own
test suite (which I vote "doesn't really count").

regardless, not needing to tweak even those would be a convenience and it
could open up doors for some more application frameworks that make such
environment assumptions and are thus hard to distribute stand-alone.  ex:
It'd open the door for multiprocessing spawn mode within stand alone
embedded binaries.

While applications embedding Python may not have an explicit `python`
> executable, they do likely have the capability to instantiate a
> `python`-like environment at run-time: they have the interpreter after all,
> they "just" need to provide a mechanism to invoke Py_RunMain() with an
> interpreter config initialized using the "python" profile.
>
> **I'd like to propose a long-term replacement to sys.executable that
> enables applications embedding Python to advertise a mechanism for invoking
> the same executable such that they get a `python` experience.**
>
> The easiest way to do this is to introduce a list[str] variant. Let's call
> it sys.python_interpreter. Here's how it would work.
>
> Say I've produced myapp.exe, a Windows application. If you run `myapp.exe
> python --`, the executable behaves like `python`. e.g. `myapp.exe python --
> -c 'print("hello, world")'` would be equivalent to `python -c
> 'print("hello, world")'`. The app would set `sys.python_interpreter =
> ["myapp.exe", "python", "--"]`. Then Python code wanting to invoke a Python
> interpreter would do something like
> `subprocess.run(sys.python_interpreter)` and automatically dispatch through
> the same executable.
>

yep, that seems reasonable.  unfortunately the command line arguments are a
global namespace, but choosing a unique "launch me as a standalone python
interpreter" arg when building a standalone python executable app that will
never conflict with an application, at build time, is doable.  Nobody's
application wants this specific unique per build ---$(uuid) flag in argv[1]
right? ;) ...

There's still an API challenge to decide on here: people using
sys.executable also expect to pass flags to the python interpreter.  Do we
make an API guarantee that the final flag in sys.python_interpreter is
always a terminator that separates python flags from application flags (--
or otherwise)?

For applications not wanting to expose a `python`-like capability, they
> would simply set sys.python_interpreter to None or [], just like they do
> with sys.executable today.
>

Yep.  Though that should be done at stand alone python application build
time to avoid any command line of the binary possibly launching as a plain
interpreter.  (this isn't security, anyone with access to read the stand
alone executable can figure out how to construct a raw interpreter usable
in their environment from that)


> In fact, I imagine Python's initialization would automatically set
> sys.python_interpreter to [sys.executable] by default and applications
> would have to opt in to a more advanced PyConfig field to make
> sys.python_interpreter different. This would make sys.python_interpreter
> behaviorally backwards compatible, so code bases could use
> sys.python_interpreter as a modern substitute for sys.executable, if
> available, without that much risk.
>

+1

-gps


>
> Some applications may want more advanced mechanisms than command line
> arguments to dispatch off of. For example, maybe you want to key off an
> environment variable to activate "Python mode."  This scenario is a 

[Python-ideas] Re: Amend PEP-8 to require clear, understandable comments instead of Strunk & White Standard English comments

[Gregory P. Smith]

Given I've never even heard of Strunk & White (my own privilege i'd
assume)... yeah.  I don't actually know what the existing "When writing
English, follow Strunk and White." text in PEP-8 even means.

It doesn't hyperlink to an online source for English style probably because
this was written so long ago, but I wouldn't want it to anyways.

People shouldn't be arguing about "proper" English within comments
anyways.  Is it readable to all involved and to the next folks involved
(future selves included) who will read it several years later?  Good enough.

Perhaps just send a PR removing that sentence to simplify things?

-gps

On Fri, Jun 26, 2020 at 2:29 PM Keara Berlin  wrote:

> Hi all, this is a very small change, but I thought I would field it here
> to see if anyone has suggestions or ideas. Instead of requiring that
> comments be written in Strunk & White Standard English, PEP-8 should
> require instead that English-language comments be clear and easily
> understandable by other English speakers. This accomplishes the same goal
> without alienating or putting up barriers for people (especially people of
> color) whose native dialect of English is not Standard English. This change
> is a simple way to correct that while maintaining the original intent of
> the requirement. This change may even make the requirement more clear to
> people who are not familiar with Strunk & White, since for programmers, the
> main relevant aspect of that standard is "be clear and concise;" simply
> saying that instead of referencing Strunk & White may communicate this more
> effectively.
> Here is the current line in PEP-8: "When writing English, follow Strunk
> and White."
> I propose changing this line to "When writing English, ensure that your
> comments are clear and easily understandable to other English speakers."
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/AE2M7KOIQR37K3XSQW7FSV5KO4LMYHWX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/OHI2I6U3TRCOK7DVD46FRMR3JEEK223T/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Bringing the print statement back

[Gregory P. Smith]

On Tue, Jun 9, 2020 at 5:49 PM Naomi Ceder  wrote:

> FWIW...
>
> As someone who has spent at least the last 5 years trying (unsuccessfully)
> to reprogram 15 years of muscle memory of print as a statement, I vote +1
> to print without parens.
>
> As someone who teaches Python and groans at explaining exceptions, I'm -0
> on print without parens and -1 on other calls without parens.
>

-1 overall for me...

even though I *liked* the Python 0-2 print statement syntax for being
"different" and readable without the need for shift-key-awkwardness of ()s
including the shell like use of >>sys.stderr up front to redirect the
output.  This "difference" excused it from auto-inserting magic spaces
between everything and repurposing >> - it was "special".  Today's function
though, it just fits in better with the overall language.  It is
consistent.  It is obviously a function the likes of which anyone could
define.

As soon as callables can magically be called using a space instead of ()s a
whole new world of typos and misunderstandings of what is what and what
belongs with what comes up.  Modifications to code that need to change
calling formats require more complicated edits, etc.  I realize that many
other high level languages feature this kind of "treat a callable as a
command" syntax.  But I've never felt it was consistent as it was purely a
style choice which syntax to use and when, so why cause conflict and
require that decision?  Useful for small scripting mentality, but not while
programming and engineering reusable maintainable code.  In large Perl code
bases of yore, the style norm was to not use this style of call or if used,
limit it solely to built-ins.

We've got one way of doing things that people understand and works, why do
we need yet another?

We aren't Perl, Ruby, Rexx, Lua, Tcl, ...

I love that the new parser allows us to even explore these possibilities.
We should be very cautious about what syntax changes we actually adopt.

-gps
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/7FQDT3KTVRWP7SQ5FMB77R6OXLO5SXHL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Gregory P. Smith]

On Wed, Apr 8, 2020, 10:37 AM Antoine Pitrou  wrote:

> On Wed, 8 Apr 2020 10:18:47 -0700
> Guido van Rossum  wrote:
> >
> > > But when I leave "large" temp objects hanging and
> > > give a rip, I already stick in "del" statements anyway.  Very rarely,
> > > but it happens.
> > >
> >
> > I recall that in the development of asyncio there were a few places where
> > we had to insert del statements, not so much to free a chunk of memory,
> but
> > to cause some destructor or finalizer to run early enough. (IIRC not
> right
> > at that moment, but at some later moment even if some Futures are still
> > alive.) Those issues took considerable effort to find, and would
> > conceivably have been prevented by this proposal.
>
> If code relies on life variable analysis for correctness, it implies
> other Python implementations must implement it with exactly the same
> results.
>

As far as I know they all do? The existence of locals() as an API cements
this behavior. If you want something to disappear from locals it requires
an explicit del.  (explicit is better than implicit and all...)

I'd actually accept this optimization in something like micropython where
bending rules to fit in mere kilobytes makes sense. But in CPython I want
to see serious demonstrated practical benefit before changing this behavior
in any file by default.  (it could be implemented per file based on a
declaration; this would be a bytecode optimization pass)

-gps


> Regards
>
> Antoine.
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/V6AGHX43FB2EF6RZGQQJTYGQTRVX3W2F/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/NQU7KKRO2C7AUQ6NDVZV4SITRZMV3DP5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Live variable analysis -> earlier release?

[Gregory P. Smith]

On Wed, Apr 8, 2020 at 10:21 AM Guido van Rossum  wrote:

> On Wed, Apr 8, 2020 at 10:05 AM Alex Hall  wrote:
>
>> This would break uses of locals(), e.g.
>>
>
> Hm, okay, so suppose the code analysis was good enough to recognize most
> un-obfuscated uses of locals(), exec() and eval() (and presumably
> sys._getframe() -- IIUC there's already a Python implementation that
> generates less optimal code for functions where it detects usage of
> sys._getframe(), maybe IronPython).
>
>
>> Plus if the calculation raises an exception and I'm looking at the report
>> on Sentry, I'd like to see the values of all variables. In particular I
>> might have expected the function to return early and I want to see what `x`
>> was.
>>
>
> That's a very valid objection. For simpletons like myself who just use pdb
> it could also be problematic. So at the very least there would have to be a
> way to turn it off, and probably it should have to be requested explicitly
> (maybe just with -O).
>

Even though it seems like a pedantically correct behavior to toss something
after no future direct references to it are detected, it would just break
so much existing code that the upgrade to an interpreter doing this would
be a nightmare.  Yes, people should in theory be using a context manager or
explicit reference to things that need to live, but what _is_ an explicit
reference if not the local scope?  In practice code all over the place
assumes "still in local scope" means the reference lives on.  Including
wrapped C/C++ code where the destructor frees underlying resources that are
needed by other things outside of CPython's view.

Different behavior when debugging or adding a debug print vs when running
normally is bad.  Seeing surprising behavior changes due to the addition or
removal of code later in a scope that inadvertently changes when
something's destructor is called is action at a distance and hard to
debug.  In my experience, people understand scope-based lifetime.  It is
effectively the same as modern C++ STL based pointer management semantics;
destruction only happens after going out of scope or when explicitly called
for.

Making it optional?  It'd need to be at least at a per-file level (from
__future__ style... but called something else as this isn't likely to be
our future default).  A behavior change of this magnitude globally for a
program with -O would just reinforce the existing practice of nobody
practically using -O.  (does anyone actually run their tests under -O let
alone deploy with -O?  I expect that to be a single digit % or lower
minority)

-gps



>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/CNON3MYEASWG6WLW5C2QWRSTMQVZDDCX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/R7EAK4Q2MVED44EZQMKBQASIN24UIYFX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Really long ints

[Gregory P. Smith]

On Tue, Feb 4, 2020 at 3:29 AM Chris Angelico  wrote:

>
> Another one to throw into the mix: Trailing underscores, but only if
> the expression is incomplete. So in simple cases like this, that means
> parenthesizing the number:
>
> P = (29674495668685510550154174642905332730771991_
> 79985304335099507553127683875317177019959423_
> 8596428121188033664754218345562493168782883)
>
>
FWIW, if a multi-line int literal syntax is deemed worthy of having, this
syntax really makes me smile as the most obvious about its intent.  I do
not think anybody unaware of specific Python syntaxes would misread it.

The requirement of the ()s fits with the general recommendation made to
avoid \ by enclosing in ()s.  The question that remains is if the () around
every such integer are required, or if this occurring within any existing
()s is sufficient.  ex:

method.call(123_
  456,
  786_
  9)

*could* be semiconfusing.  Though , and _ are visually distinct enough that
I think it would stand out.  And disallowing a final trailing _ prevents
"_," accidents.  Requiring additional ()s in this case would be fine, but
probably isn't worth it.  I expect anyone entering a multi-line super long
literal to not be inlining them in common practice and always be assigning
them to a useful name for readability's sake.

-gps


> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/TTEDIQXY7E5CSL26CTZKCL4KBJJ35ORU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/35DMDTRXXXFHCUQO34SNDWJCDC2VIRYI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Allow Path object instances in subprocess.Popen

[Gregory P. Smith]

On Sun, Nov 3, 2019 at 12:33 AM Brendan Barnwell 
wrote:

> On 2019-11-03 00:02, Eric V. Smith wrote:
> > On 11/2/2019 10:02 PM, Jonathan Goble wrote:
> >> So there's no reason in that regard to not allow it, and I'm +1 on the
> idea.
> >
> > I disagree. I don't think there's precedent in Python for stringifying
> > parameters. By your logic, shouldn't we also call str() on the
> > parameters for str.join()? Or how about ord()? (Oops, I accidentally
> > called len: "s='3'; ord(len(n))".) Or the first parameter of open()? (Do
> > we really want "open(fn_that_might_occasionally_return_none(), 'w')" to
> > succeed?)
>
> For the record, I have always thought that str.join should call
> str on
> all its arguments.  The fact that it doesn't is really annoying.
>
> But to the larger point that Popen "can't know" that the path
> objects
> represent filenames. . . why not?  Why can't Popen have code that checks
> if something in the arglist is a Path object, and if so, converts it to
> a string?  That wouldn't be any different than various existing
> functions that accept, say, something or a tuple of them (e.g.,
> isintance) or alternative formats (e.g., functions in the re module that
> take either a pattern object or a string).  Popen is in the stdlib and
> is changeable if we want to change it, and I don't see what's so strange
> about updating it to handle new developments like Path objects.
>
>
We *started* down this path in https://bugs.python.org/issue31961 but had
to revert part of that before 3.7 as the implementation was incomplete
making it inconsistent across platforms.
https://github.com/python/cpython/pull/4329.

Specifically accepting PathLike objects within the args list hasn't been
ruled out for the future but nobody has done the relevant work.  Tracking
it and the api issues to answer in https://bugs.python.org/issue38670.

-gpshead (subprocess owner)
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GF7L4XTU6EOC4UFY7ALWACARFCNC6LMZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Make itertools recipes license easier

[Gregory P. Smith]

On Thu, Oct 24, 2019 at 3:21 PM Guido van Rossum  wrote:

> OK, let's see if we can do CC0. Todd, do you want to read through the link
> Steven gave and find out how we should apply this, either to just the
> itertools examples, or possibly to all examples in the docs?
>

Obviously all examples in the Python docs are intended to be copied by
anyone at will for use anywhere.  Lets not limit it.

Lets also make sure we don't need to do any manual work polluting the .rst
and docstrings with individual license markers.  This should be automated
in the doc generation and/or perhaps a common footnote somewhere.

-gps


> On Thu, Oct 24, 2019 at 2:01 PM Steven D'Aprano 
> wrote:
>
>> On Thu, Oct 24, 2019 at 12:36:40PM -0700, Christopher Barker wrote:
>> > IANAL, but if we could put a clause in the docs that all recipes are in
>> the
>> > public domain, that would be great.
>>
>> The public domain is *exceedingly* problematic. Many juristictions do
>> not have any legal process for putting things into the public domain
>> before the standard copyright expiry, or even prohibit the attempt to
>> surrender such rights.
>>
>> That's why the Creative Commons folks have their CC0 licence, which
>> seems to be the most appropriate for this case.
>>
>> https://creativecommons.org/share-your-work/public-domain/cc0/
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/U3SMXGIWEXUKPN5QW2WR4NL2RPXACEVG/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/XP5GF5G754APG64JG3HG2K5666LNRA5A/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/2IVANLB674JGH6WCGWZJ2VP4EQXGCZAU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add __slots__ to dataclasses to use less than half as much RAM

[Gregory P. Smith]

On Fri, Sep 27, 2019 at 5:21 PM Guido van Rossum  wrote:

> On Fri, Sep 27, 2019 at 11:18 AM Serhiy Storchaka 
> wrote:
>
>> I think it needs an explicit support in the type creation machinery (as
>> __slots__ itself has) to support descriptors and slots with the same name.
>>
>
> That would be tough, since __slots__ is currently implemented by creating
> a separate descriptor for each slot.
>
> I do think that it would be nice to have a way to automatically create
> __slots__ from annotations. It would be even nicer if that could be done
> without copying the class object (as the current state of the art requires:
> https://github.com/ericvsmith/dataclasses/blob/master/dataclass_tools.py#L23
> ).
>
> Thinking aloud, perhaps this could be done by setting __slots__ to a
> magical value, e.g.
>
> class Point:
> __slots__ = "__auto__"
> x: float
> y: float
>
> This would be independent from the @dataclass decorator (though the
> decorator may have to be aware of the magic value).
>
> If that's too wacky, we could also use a class keyword argument:
>
> class Point(slots=True):
> x: float
> y: float
>
> (Though arguably that's just as wacky. :-)
>

An argument is less wacky than assigning a magic value to a magic dunder.

-gps
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/BX25SVXD3Z5QFUNBYF54MSTNG2U33DT3/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improve os.times() resolution

[Gregory P. Smith]

On Sun, Mar 24, 2019 at 5:16 AM Giampaolo Rodola' 
wrote:

> It turns out we could use resource.getrusage() which provides micro
> seconds (tested on Linux and macOS):
>
> import os, resource
> for x in range(1000):  # warm up
> pass
> for x in range(5):
> a = os.times()
> b = resource.getrusage(resource.RUSAGE_SELF)
> print(a.user, a.system)
> print(b.ru_utime, b.ru_stime)
>
> ...it prints:
>
> 0.39 0.01
> 0.394841 0.011963
> 0.39 0.01
> 0.394899 0.011966
> 0.39 0.01
> 0.394908 0.011966
> 0.39 0.01
> 0.394936 0.011967
> 0.39 0.01
> 0.394963 0.011968
>
> getrusage(RUSAGE_CHILDREN) can be used to calculate "children_user"
> and "children_system". I see 2 possibilities here:
>
> 1) doc fix, mentioning that resource.getrusage provides a better resolution
> 2) if available (it should always be as it's a POSIX standard), just
> use getrusage in Modules/posixmodule.c. It seems we can check
> availability by reusing HAVE_SYS_RESOURCE_H and HAVE_SYS_TIME_H
> definitions which are already in place.
>
> I'm not sure what's best to do as os.* functions usually expose the
> original C function with the same name, but given that "elapsed" field
> is not part of times(2) struct and that on Windows "elapsed",
> "children_user" and "children_system" are set to 0 it appears there
> may be some space for flexibility here.
>
> Thoughts?
>

I'd just document that resource.getrusage() provides better data.  That is
what man pages for times(3) have done as well.  It is good to keep the os
module close to the underlying library/system calls and leave it to higher
level code to abstract and choose as deemed appropriate.

-gps
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add subprocess.Popen suspend() and resume()

[Gregory P. Smith]

I don't think this belongs in subprocess.  It isn't related to processes
creation.

A module on PyPI with the Windows code would make more sense.

On Wed, Mar 20, 2019 at 3:19 PM eryk sun  wrote:

> On 3/18/19, Giampaolo Rodola'  wrote:
> >
> > I've been having these 2 implemented in psutil for a long time. On
> > POSIX these are convenience functions using os.kill() + SIGSTOP /
> > SIGCONT (the same as CTRL+Z  / "fg"). On Windows they use
> > undocumented NtSuspendProcess and NtResumeProcess Windows
> > APIs available since XP.
>
> Currently, Windows Python only calls documented C runtime-library and
> Windows API functions. It doesn't directly call NT runtime-library and
> system functions. Maybe it could in the case of documented functions,
> but calling undocumented functions in the standard library should be
> avoided. Unfortunately, without NtSuspendProcess and NtResumeProcess,
> I don't see a way to reliably implement this feature for Windows. I'm
> CC'ing Steve Dower. He might say it's okay in this case, or know of
> another approach.
>
> DebugActiveProcess, the other simple approach mentioned in the linked
> SO answer [1], is unreliable and has the wrong semantics.  A process
> only has a single debug port, so DebugActiveProcess will fail the PID
> as an invalid parameter if another debugger is already attached to the
> process. (The underlying NT call, DbgUiDebugActiveProcess, fails with
> STATUS_PORT_ALREADY_SET.) Additionally, the semantics that I expect
> here, at least for Windows, is that each call to suspend() will
> require a corresponding call to resume(), since it's incrementing the
> suspend count on the threads; however, a debugger can't reattach to
> the same process. Also, if the Python process exits while it's
> attached as a debugger, the system will terminate the debugee as well,
> unless we call DebugSetProcessKillOnExit(0), but that interferes with
> the Python process acting as a debugger normally, as does this entire
> wonky idea. Also, the debugging system creates a thread in the debugee
> that calls NT DbgUiRemoteBreakin, which executes a breakpoint. This
> thread is waiting, but it's not suspended, so the process will never
> actually appear as suspended in Task Manager or Process Explorer.
>
> That leaves enumerating threads in a snapshot and calling OpenThread
> and SuspendThread on each thread that's associated with the process.
> In comparison, let's take an abridged look at the guts of
> NtSuspendProcess.
>
> nt!NtSuspendProcess:
> ...
> mov r8,qword ptr [nt!PsProcessType]
> ...
> callnt!ObpReferenceObjectByHandleWithTag
> ...
> callnt!PsSuspendProcess
> ...
> mov ebx,eax
> callnt!ObfDereferenceObjectWithTag
> mov eax,ebx
> ...
> ret
>
> nt!PsSuspendProcess:
> ...
> callnt!ExAcquireRundownProtection
> cmp al,1
> jne nt!PsSuspendProcess+0x74
> ...
> callnt!PsGetNextProcessThread
> xor ebx,ebx
> jmp nt!PsSuspendProcess+0x62
>
> nt!PsSuspendProcess+0x4d:
> ...
> callnt!PsSuspendThread
> ...
> callnt!PsGetNextProcessThread
>
> nt!PsSuspendProcess+0x62:
> ...
> testrax,rax
> jne nt!PsSuspendProcess+0x4d
> ...
> callnt!ExReleaseRundownProtection
> jmp nt!PsSuspendProcess+0x79
>
> nt!PsSuspendProcess+0x74:
> mov ebx,0C10Ah (STATUS_PROCESS_IS_TERMINATING)
>
> nt!PsSuspendProcess+0x79:
> ...
> mov eax,ebx
> ...
> ret
>
> This code repeatedly calls PsGetNextProcessThread to walk the
> non-terminated threads of the process in creation order (based on a
> linked list in the process object) and suspends each thread via
> PsSuspendThread. In contrast, a Tool-Help thread snapshot is
> unreliable since it won't include threads created after the snapshot
> is created. The alternative is to use a different undocumented system
> call, NtGetNextThread [2], which is implemented via
> PsGetNextProcessThread. But that's slightly worse than calling
> NtSuspendProcess.
>
> [1]: https://stackoverflow.com/a/11010508
> [2]:
> https://github.com/processhacker/processhacker/blob/v2.39/phnt/include/ntpsapi.h#L848
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Enabling / disabling optional type hinting

[Gregory P. Smith]

On Sat, Mar 23, 2019 at 11:00 AM Stefan Krah  wrote:

> On Sat, Mar 23, 2019 at 10:37:43AM -0700, Gregory P. Smith wrote:
> > A useless statement like that isn't likely to be typed.  I've never seen
> > anyone do that.
>
> Unlikely yes, but ideally type annotations should not alter program
> behavior:
>
> >>> d = {}
> >>> try: d["x"]
> ... except KeyError: print("KeyError")
> ...
> KeyError
> >>>
> >>> d = {}
> >>> try: d["x"] : int
> ... except KeyError: print("KeyError")
> ...
>

Unfortunately that isn't what PEP 526 said:
https://www.python.org/dev/peps/pep-0526/#annotating-expressions

-gps
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Enabling / disabling optional type hinting

[Gregory P. Smith]

On Sat, Mar 23, 2019 at 7:37 AM Andre Roberge 
wrote:

> Consider the following example [1]:
>
>  Python 3.7.0 (v3.7.0:1bf9cc5093...
>  >>> d = {
>  ...   "injury": "flesh wound"
>  ... }
>  >>> d["answer"]: 42
>  >>> if "answer" in d:
>  ... print("Don't panic!")
>  ... else:
>  ... print("Sorry, I can't help you.")
>  ...
>  Sorry, I can't help you.
>
> = =
> No SyntaxError raised (which would have been the case before version 3.5?)
> and yet what could be a very unexpected result occurred.
>
> Of course, the problem is with the line
>
> d["answer"]: 42
>
> which is not an assignment but is an "optional" type hint.
>

A useless statement like that isn't likely to be typed.  I've never seen
anyone do that.

Sure, someone is going to typo and omit the = from a := assignment in 3.8
but the walrus is unlikely to be used outside of an conditional or loop
test context so this seems like a made up problem.  (if anything, this
possibility encourages people not to use the walrus in unintended places).
Someone might also typo it by meaning to use a ; for multiple statements
but enter : instead.  Again, very rare.  Because ; is frowned upon.

A linter (running live as part of an editor/ide these days) can flag most
meaningless annotations quite easily.

I think it would be very useful to have a way to turn off completely type
> hinting and flag any use of code like the above as a SyntaxError.
>
> My preference would be if type hinting would be something that is enabled
> per file with a top-level comment pragma, something like
>
> # type: enable
>
> Failing that, having a to-level comment pragma like
>
> # type: disable
>

Too late.  Requiring something like that would break existing code.

might be acceptable as it would not require any change to any existing
> file. However, the other option would be more inline with type hinting
> being "optional" and something that should not trip beginners who are not
> aware of its existence.
>

What evidence is there that this frequently trips up beginners?

Enhancing the dev environments beginners use to automatically lint would be
better and would automate some learning, handling cases way beyond this one.

-gps
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why operators are useful

[Gregory P. Smith]

On Sat, Mar 16, 2019 at 5:02 AM Gustavo Carneiro 
wrote:

> On Sat, 16 Mar 2019 at 10:33, Steven D'Aprano  wrote:
>
>> On Fri, Mar 15, 2019 at 10:53:31PM +, MRAB wrote:
>>
>> > There was also the suggestion of having both << and >>.
>> >
>> > Actually, now that dicts are ordered, that would provide a use-case,
>> > because you would then be able to choose which values were overwritten
>> > whilst maintaining the order of the dict on the LHS.
>>
>> Is that common enough that it needs to be built-in to dict itself?
>>
>> If it is uncommon, then the conventional solution is to subclass dict,
>> overriding the merge operator to use first-seen semantics.
>>
>> The question this PEP is trying to answer is not "can we support every
>> use-case imaginable for a merge operator?" but "can we support the most
>> typical use-case?", which I believe is a version of:
>>
>> new = a.copy()
>> new.update(b)
>> # do something with new
>>
>
> Already been said, but might have been forgotten, but the new proposed
> syntax:
>
> new = a + b
>
> has to compete with the already existing syntax:
>
> new = {**a, **b}
>
> The existing syntax is not exactly an operator in the mathematical sense
> (or is it?...), but my intuition is that it already triggers the visual
> processing part of the brain, similarly to operators.
>
> The only argument for "a + b" in detriment of "{**a, **b}" is that "a + b"
> is more easy to discover, while not many programmers are familiar with
> "{**a, **b}".
>
> I wonder if this is only a matter of time, and over time programmers will
> become more accustomed to "{**a, **b}", thereby reducing the relative
> benefit of  "a + b"?  Especially as more and more developers migrate code
> bases from Python 2 to Python 3...
>

FWIW, even as a core developer I had forgotten that the {**a, **b} syntax
existed, thanks for the reminder! :)  But that's more likely because I
rarely write code that needs to update and merge a dict or when i do it's
still 2and3 compatible.

Antoine said:

> If "+" is added to dicts, then we're overloading an already heavily used
operator.  It makes reading code more difficult.

This really resonated with me.  Reading code gives you a feel for what
possible types something could be.  The set of possibilities for + is
admittedly already quite large in Python.  But making an existing core type
start supporting + *reduces the information given to the reader* by that
one line of code.  They now have more possibilities to consider and must
seek hints from more surrounding code.

For type inferencers such as us humans 
or tools like pytype , it means we need
to consider which version Python's dict the code may be running under in
order to infer what it may mean from the code's context.  For tooling,
that's just a flag and a matter of conditionally changing code defining
dict, but for humans they need to carry the possibility of that flag with
them everywhere.

We should just promote the use of {**d1, **d2} syntax for anyone who wants
an inline updated copy.

Why?

(1) It already exists.  (insert zen of python quote here)

(2) Copying via the + operator encourages inefficient code (already true
for bytes/str/list).  A single + is fine.  But the natural human extension
to that when people want to merge a bunch of things is to use a string of
multiple operators, because that is how we're taught math.  No matter what
type we're talking about, in Python this is an efficiency antipattern.

 z = a + b + c + d + e

That's four __add__ calls.  Each of which is a copy+update/extend/append
operation for a dict/list/str respectively.

We already tell people not to do that with bytes and lists, instead using
b''.join(a,b,c,d) or z = []; z.extend(X)... calls or something from
itertools.

Given dict addition, it'd always be more efficient to join the "don't use
tons of  + operators" club (a good lint warning) and write that as
 z = {**a, **b, **c, **d, **e}.

Unless the copy+update concept is an extremely common operation, having
more than one way to do it feels like it'll cause more cognitive harm than
good.

Now (2) could *also* be used as an argument that Python should detect
chains of operators and allow those to be optimized.  That'd be a PEP of
its own and is complicated to do; technically a semantic change given how
dynamic Python is, as we do name lookups at time of use and each __add__
call could potentially have side effects changing the results of future
name lookups (the a+b could change the meaning of c).  Yes, that is
horrible and people writing code that does that deserve very bad things,
but those are the semantics we'd be breaking if we tried to detect and
support a mythical new construct like __chained_add__ being invoked when
the types of all elements being added sequentially are identical (how
identical? do subtypes count? see, complicated).

a + b + c:
   0 LOAD_GLOBAL  0 (a)
   

Re: [Python-ideas] PEP 8 and long string literals

[Gregory P. Smith]

On Mon, Feb 25, 2019 at 2:05 AM INADA Naoki  wrote:

> I think long URL in comment or docstring is good reason to ignore
> line length limit.
>

Yep, that's what we do in the yapf autoformatter.  There's good reason too,
source viewers and editors linkify URLs, if you break them across strings
to fit within a line length limit, you get a broken partial url.  It also
makes searching a codebase for references to specific identifiers much more
difficult if you've broken up their value at arbitrary points.

There are other examples for this such as allowing trailing line comments
for pylint, type checkers, etc. to go beyond 80 columns.

See the Exceptions in
https://google.github.io/styleguide/pyguide.html#32-line-length for what
make good practical examples.

I'm not convinced that codifying this in PEP-8 makes sense though unless we
see this happening within the Python standard library code itself (the
purpose of pep-8).  PEP-8 is not trying to be a style guide for all Python
users, just a stdlib style guide that happens to serve a base from which to
build upon.

-gps
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add support for external annotations in the typing module

[Gregory P. Smith]

On Thu, Jan 17, 2019 at 2:34 PM Till  wrote:

> We started a discussion in https://github.com/python/typing/issues/600
> about adding support for extra annotations in the typing module.
>
> Since this is probably going to turn into a PEP I'm transferring the
> discussion here to have more visibility.
>
> The document below has been modified a bit from the one in GH to reflect
> the feedback I got:
>
>  + Added a small blurb about how ``Annotated`` should support being used
> as an alias
>
> Things that were raised but are not reflected in this document:
>
>  + The dataclass example is confusing. I kept it for now because
> dataclasses often come up in conversations about why we might want to
> support annotations in the typing module. Maybe I should rework the
> section.
>
>  + `...` as a valid parameter for the first argument (if you want to add
> an annotation but use the type inferred by your type checker). This is an
> interesting idea, it's probably worth adding support for it if and only if
> we decide to support in other places. (c.f.:
> https://github.com/python/typing/issues/276)
>
> Thanks,
>
> Add support for external annotations in the typing module
> ==
>
> We propose adding an ``Annotated`` type to the typing module to decorate
> existing types with context-specific metadata. Specifically, a type ``T``
> can be annotated with metadata ``x`` via the typehint ``Annotated[T, x]``.
> This metadata can be used for either static analysis or at runtime. If a
> library (or tool) encounters a typehint ``Annotated[T, x]`` and has no
> special logic for metadata ``x``, it should ignore it and simply treat the
> type as ``T``. Unlike the `no_type_check` functionality that current exists
> in the ``typing`` module which completely disables typechecking annotations
> on a function or a class, the ``Annotated`` type allows for both static
> typechecking of ``T`` (e.g., via MyPy or Pyre, which can safely ignore
> ``x``)  together with runtime access to ``x`` within a specific
> application. We believe that the introduction of this type would address a
> diverse set of use cases of interest to the broader Python community.
>
> Motivating examples:
> 
>
> reading binary data
> +++
>
> The ``struct`` module provides a way to read and write C structs directly
> from their byte representation. It currently relies on a string
> representation of the C type to read in values::
>
>   record = b'raymond   \x32\x12\x08\x01\x08'
>   name, serialnum, school, gradelevel = unpack('<10sHHb', record)
>
> The struct documentation [struct-examples]_ suggests using a named tuple
> to unpack the values and make this a bit more tractable::
>
>   from collections import namedtuple
>   Student = namedtuple('Student', 'name serialnum school gradelevel')
>   Student._make(unpack('<10sHHb', record))
>   # Student(name=b'raymond   ', serialnum=4658, school=264, gradelevel=8)
>
>
> However, this recommendation is somewhat problematic; as we add more
> fields, it's going to get increasingly tedious to match the properties in
> the named tuple with the arguments in ``unpack``.
>
> Instead, annotations can provide better interoperability with a type
> checker or an IDE without adding any special logic outside of the
> ``struct`` module::
>
>   from typing import NamedTuple
>   UnsignedShort = Annotated[int, struct.ctype('H')]
>   SignedChar = Annotated[int, struct.ctype('b')]
>
>   @struct.packed
>   class Student(NamedTuple):
> # MyPy typechecks 'name' field as 'str'
> name: Annotated[str, struct.ctype("<10s")]
> serialnum: UnsignedShort
> school: SignedChar
> gradelevel: SignedChar
>
>   # 'unpack' only uses the metadata within the type annotations
>   Student.unpack(record))
>   # Student(name=b'raymond   ', serialnum=4658, school=264, gradelevel=8)
>
>
>
> dataclasses
> 
>
> Here's an example with dataclasses [dataclass]_ that is a problematic from
> the typechecking standpoint::
>
>   from dataclasses import dataclass, field
>
>   @dataclass
>   class C:
> myint: int = 0
> # the field tells the @dataclass decorator that the default action in
> the
> # constructor of this class is to set "self.mylist = list()"
> mylist: List[int] = field(default_factory=list)
>
> Even though one might expect that ``mylist`` is a class attribute
> accessible via ``C.mylist`` (like ``C.myint`` is) due to the assignment
> syntax, that is not the case. Instead, the ``@dataclass`` decorator strips
> out the assignment to this attribute, leading to an ``AttributeError`` upon
> access::
>
>   C.myint  # Ok: 0
>   C.mylist  # AttributeError: type object 'C' has no attribute 'mylist'
>
>
> This can lead to confusion for newcomers to the library who may not expect
> this behavior. Furthermore, the typechecker needs to understand the
> semantics of dataclasses and know to not treat the above example as an
> assignment 

Re: [Python-ideas] Using sha512 instead of md5 on python.org/downloads

[Gregory P. Smith]

On Fri, Dec 7, 2018 at 3:38 PM Steven D'Aprano  wrote:

> On Fri, Dec 07, 2018 at 01:25:19PM -0800, Nathaniel Smith wrote:
>
> > For this specific purpose, md5 is just as good as a proper hash. But all
> > else being equal, it would still be better to use a proper hash, just so
> > people don't have to go through the whole security analysis to check
> that.
>
> I don't understand what you are trying to say here about "the whole
> security analysis" to check "that". What security analysis, and
> what is "that"?
>
> It seems to me that moving to a cryptographically-secure hash would give
> many people a false sense of security, that just because the hash
> matched, the download was not only not corrupted, but not compromised as
> well. For those two purposes:
>
> - testing for accidental corruption;
> - testing for deliberate compromise;
>
> md5 and sha512 are precisely equivalent: both are sufficient for the
> first, and useless for the second. But a crypto-hash can give a false
> sense of security. The original post in this thread is evidence of that.
>
> As such, I don't think we should move to anything stronger than md5.
>

If we switched to sha2+ or listed 8 different hashes at once in the
announcement text so that nobody can find the actual link content, we'd
stop having people pipe up and complain that we used md5 for something.
Less mailing list threads like this one seems like a benefit. :P

Debian provides all of the popular FIPS hashes, in side files, so people
can use whatever floats their boat for a content integrity check:
 https://cdimage.debian.org/debian-cd/current/ppc64el/iso-cd/

>From a semi-security perspective without verifying gpg signatures, listing
a currently collision-resistant hash (sha2 onwards today) in widely
disseminated release announcement that goes on mailing lists and gets
forwarded and reposted in many places is still useful.  Being not hosted in
a single central place, if the downloads and hashes on the main servers
change *after* their computation, publishing, and announcement - it serves
as a widely distributed question mark.  A pointless one, as the gpg
signature also exists, but it is one none the less.

As to windows and mac providing hashing functions on the command line,
nope.  assume nothing is provided.  On linux my fingers would use "openssl
hashname" rather than *sum commands.  But none of those are ever required
to be installed by anything.  The only people who ever check hashes are
those who already know what tools to use and how.  Some could ironically
install the downloaded python and use it to check its own hash. None of
that is our problem.

-gps
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] anyone need a frozenset or bytearray literal?

[Gregory P. Smith]

On Wed, Jul 11, 2018 at 6:13 PM Guido van Rossum  wrote:

> I know of many use cases for frozenset({...}) and I think a hack along
> those lines is fine -- but what's the common use for bytearray(b"...") or
> bytearray((...))? Almost invariably a bytearray is created empty or as a
> given number of zeros. I certainly wouldn't want to burden the tuple type
> with a to_bytearray() method, the types are unrelated (unlike set and
> frozenset).
>

Agreed, bytearray(b'...') should be way less common.  I don't immediately
have a use for that beyond merely than disliking the copy from temporary
bytes object and gc behind the scenes.  I could find a practical use for it
in micropython where ram is extremely limited, but that VM could already
implement such a compile time optimization on its own.  The concept of the
optimization that'd be required just seemed similar to that of frozenset to
me.

frozenset is the one that led me down this train of thought as I was
looking at code declaring a bunch on constants.

-gps


> On Wed, Jul 11, 2018 at 6:03 PM, Robert Vanden Eynde <
> robertvandeney...@hotmail.com> wrote:
>
>> {1,2,7}.freeze() or {1,2,7}.to_frozenset() seem very natural and if this
>> can be optimized to avoid the copy, it's perfect.
>> For bytearray, one use case would be to optimise bytearray([1,2,7,2]) in
>> something like [1,2,7,2].to_byterray().
>> About bytes, one could have (1,2,7,2).to_bytes() instead of
>> bytes((1,2,7,2)) because b'\x01\x02\x07\x02' is long and boring.
>> What about variables in the values {1,2,x}.freeze() should work too ?
>> bytes((1,2,7,x)) is not writable as a b string and creates a copy.
>>
>>
>> Le jeu. 12 juil. 2018 à 02:24, Chris Angelico  a
>> écrit :
>>
>>> On Thu, Jul 12, 2018 at 10:13 AM, Gregory P. Smith 
>>> wrote:
>>> >
>>> > On Wed, Jul 11, 2018 at 4:45 PM Jelle Zijlstra <
>>> jelle.zijls...@gmail.com>
>>> > wrote:
>>> >> This could be done safely and without too much craziness if .freeze()
>>> on a
>>> >> set returned a new frozenset. The compiler could then safely optimize
>>> {a,
>>> >> set, literal}.freeze() into a frozenset literal, because methods on
>>> builtin
>>> >> types cannot be monkeypatched. There's been talk of a similar
>>> optimization
>>> >> on calls to .format() on string literals (not sure whether it's been
>>> >> implemented).
>>> >>
>>> >> Whether implementing that is a good use of anyone's time is a
>>> different
>>> >> question.
>>> >
>>> >
>>> > Neat optimization.  I hadn't considered that.  We do know for sure it
>>> is a
>>> > builtin type at that point.
>>> >
>>> > If that were implemented, bytes objects could gain a to_bytearray()
>>> (along
>>> > the lines of the int.to_bytes() API) method that could be optimized
>>> away in
>>> > literal circumstances.
>>>
>>> Be careful: a bytearray is mutable, so this isn't open to very many
>>> optimizations. A .freeze() method on sets would allow a set display to
>>> become a frozenset "literal", stored as a constant on the
>>> corresponding function object, the way a tuple is; but that's safe
>>> because the frozenset doesn't need to concern itself with identity,
>>> only value. Example:
>>>
>>> def f(x):
>>> a = (1, 2, 3) # can be optimized
>>> b = (x, 4, 5) # cannot
>>> c = [6, 7, 8] # cannot
>>>
>>> Disassemble this or look at f.__code__.co_consts and you'll see (1, 2,
>>> 3) as a single constant; but the others have to be built.
>>>
>>> +1 on set.freeze(); +0 on bytes.to_bytearray().
>>>
>>> ChrisA
>>> ___
>>> Python-ideas mailing list
>>> Python-ideas@python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] anyone need a frozenset or bytearray literal?

[Gregory P. Smith]

On Wed, Jul 11, 2018 at 4:45 PM Jelle Zijlstra 
wrote:

> 2018-07-11 16:25 GMT-07:00 Gregory P. Smith :
>
>> Completely obvious what it does, but it irritates my aesthetic
>> sensibilities every time I see:
>>   frozenset({spam, eggs})
>>
>> Why? Because I assume under the hood that creates a set of spam and eggs
>> before calling frozenset to copy it into a new frozenset object before the
>> original set is garbage collected.  Wasteful.  This is in fact what happens
>> in CPython 3.7 today.
>>
>> I'd love to avoid this.  I have no rational measured reason to believe it
>> even matters (thus seeding this on python-ideas and not elsewhere), even
>> though it would technically speed up frozenset creation.
>>
>> (a) detecting frozenset({}) as syntax to encode a frozenset in the python
>> bytecode would be somewhat magical.  it could break the person unfortunate
>> enough to monkeypatch out the frozenset builtin (really? don't do that!).
>>
>> (b) abusing the concept of letter prefixes as we already have for strings
>> on {} syntax would be possible but not at all obvious to a reader:
>>
>>   f{} or c{} or r{} perhaps.  but then someone would want a frozendict.
>>
>> (c) adding a .freeze() method to sets which would raise an exception if
>> the set's refcount were > 1 and would mutate the type of the set object
>> into a frozenset object in place.  refcount assertions are bad, not all VMs
>> need refcounts.  The concept of a method that can mutate the type of the
>> underlying object in place is... unpythonic.  even though technically
>> possible to implement within CPython.
>>
>> This could be done safely and without too much craziness if .freeze() on
> a set returned a new frozenset. The compiler could then safely optimize {a,
> set, literal}.freeze() into a frozenset literal, because methods on builtin
> types cannot be monkeypatched. There's been talk of a similar optimization
> on calls to .format() on string literals (not sure whether it's been
> implemented).
>
> Whether implementing that is a good use of anyone's time is a different
> question.
>

Neat optimization.  I hadn't considered that.  We do know for sure it is a
builtin type at that point.

If that were implemented, bytes objects could gain a to_bytearray() (along
the lines of the int.to_bytes() API) method that could be optimized away in
literal circumstances.

-gps


>
>> I'm -1 on all of my ideas above.  But figured I'd toss them out there as
>> food for thought for the concept.
>>
>> We lived for years without even having a set literal in the first place.
>> So this isn't a big deal.
>>
>> frozenset is not the only base type that lacks a literals leading to
>> loading values into these types involving creation of an intermediate
>> throwaway object: bytearray.  bytearray(b'short lived bytes object')
>>
>> I was going to suggest complex was in this boat as well, but upon
>> investigation we appear to do constant folding (or amazing parsingon that
>> so 0.3+0.6j does turn into a single LOAD_CONST instead of two consts and an
>> addition.  Nice!  Not that I expect practical code to use complex numbers.
>>
>> -gps
>>
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] anyone need a frozenset or bytearray literal?

[Gregory P. Smith]

Completely obvious what it does, but it irritates my aesthetic
sensibilities every time I see:
  frozenset({spam, eggs})

Why? Because I assume under the hood that creates a set of spam and eggs
before calling frozenset to copy it into a new frozenset object before the
original set is garbage collected.  Wasteful.  This is in fact what happens
in CPython 3.7 today.

I'd love to avoid this.  I have no rational measured reason to believe it
even matters (thus seeding this on python-ideas and not elsewhere), even
though it would technically speed up frozenset creation.

(a) detecting frozenset({}) as syntax to encode a frozenset in the python
bytecode would be somewhat magical.  it could break the person unfortunate
enough to monkeypatch out the frozenset builtin (really? don't do that!).

(b) abusing the concept of letter prefixes as we already have for strings
on {} syntax would be possible but not at all obvious to a reader:

  f{} or c{} or r{} perhaps.  but then someone would want a frozendict.

(c) adding a .freeze() method to sets which would raise an exception if the
set's refcount were > 1 and would mutate the type of the set object into a
frozenset object in place.  refcount assertions are bad, not all VMs need
refcounts.  The concept of a method that can mutate the type of the
underlying object in place is... unpythonic.  even though technically
possible to implement within CPython.

I'm -1 on all of my ideas above.  But figured I'd toss them out there as
food for thought for the concept.

We lived for years without even having a set literal in the first place. So
this isn't a big deal.

frozenset is not the only base type that lacks a literals leading to
loading values into these types involving creation of an intermediate
throwaway object: bytearray.  bytearray(b'short lived bytes object')

I was going to suggest complex was in this boat as well, but upon
investigation we appear to do constant folding (or amazing parsingon that
so 0.3+0.6j does turn into a single LOAD_CONST instead of two consts and an
addition.  Nice!  Not that I expect practical code to use complex numbers.

-gps
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Secure string disposal (maybe other inmutable seq types too?)

[Gregory P. Smith]

On Sat, Jun 23, 2018 at 12:57 PM Christian Heimes 
wrote:

>
> If you need to protect sensitive data like private keys, then don't load
> them into memory of your current process. It's that simple. :) Bugs like
> heartbleed were an issue, because private key were in the same process
> space as the TLS/SSL code. Solutions like gpg-agent, ssh-agent, TPM,
> HSM, Linux's keyring and AF_ALG socket all aim to offload operations
> with private key material into a secure subprocess, Kernel space or
> special hardware.
>

+10

It is fundamentally impossible for a Python VM (certainly CPython) to
implement any sort of guaranteed erasure of data and/or control over data
to prevent copying that is ever stored in a Python object.  This is not
unique to Python.  All interpreted and jitted VMs share this trait, as do
most languages with garbage collection.  ex: Java, Ruby, Go, etc.

Trying to pretend we could offer tracking and wiping of sensitive data
in-process is harmful at best as it cannot be guaranteed and thus gives the
wrong impression and will lead to misuse by people who ignore that.

-gps
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Spelling of Assignment Expressions PEP 572 (was post #4)

[Gregory P. Smith]

On Mon, Apr 16, 2018 at 11:11 AM Ned Batchelder 
wrote:

> On 4/16/18 1:42 PM, Chris Angelico wrote:
> > 3) "expr -> name" ==> The information went data way.
> >
> > So either you take a parallel from elsewhere in Python syntax, or you
> > take a hopefully-intuitive dataflow mnemonic symbol. Take your pick.
>
> My problem with the "->" option is that function annotations already use
> "->" to indicate the return type of a function.  This is an unfortunate
> parallel from elsewhere in Python syntax, since the meaning is
> completely different.
>
> ":=" is at least new syntax.
>
> "as" is nice in that it's already used for assignment, but seems to be
> causing too much difficulty in parsing, whether by compilers or people.
>
>
FWIW - We used "as" in our Python C++ binding interface description
language in CLIF to denote renaming from the original C++ name to a new
name in Python - effectively an assignment syntax.
   https://github.com/google/clif/blob/master/clif/python/primer.md

I currently have a "-0" opinion on the entire PEP 572 as I don't buy that
assignments within expressions are even a good thing to have in the
language.  #complexity - Think of people learning the language.

-gps
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Move optional data out of pyc files

[Gregory P. Smith]

On Tue, Apr 10, 2018 at 12:51 PM Eric V. Smith  wrote:

>
> >> 3. Annotations. They are used mainly by third party tools that
> >> statically analyze sources. They are rarely used at runtime.
> >
> > Even less used than docstrings probably.
>
> typing.NamedTuple and dataclasses use annotations at runtime.
>
> Eric
>

Yep. Everything accessible in any way at runtime is used by something at
runtime. It's a public API, we can't just get rid of it.

Several libraries rely on docstrings being available (additional case in
point beyond the already linked to cli tool: ply
)

Most of the world never appears to use -O and -OO.  If they do, they don't
use these libraries or jump through special hoops to prevent pyo
compliation of any sources that need them.  (unlikely)

-gps
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Disallow importing the same module under multiple names

[Gregory P. Smith]

On Thu, Mar 15, 2018 at 3:26 AM Nick Coghlan  wrote:

> On 14 March 2018 at 15:20, Chris Billington 
> wrote:
>>
>> I wonder if there's any reason something like this shouldn't be built
>> into Python's default import system.
>>
>
> There are two main challenges with enforcing such a check, one affecting
> end users in general, one affecting standard library maintainers in
> particular:
>
> * the user facing problem is a backwards compatibility one: while
> double-imports usually aren't what people wanted, they're also typically
> fairly harmless. As a result, elevating them from "sometimes a source of
> obscure bugs" to "categorically prohibited" risks breaking currently
> working code. While a conventional deprecation cycle should be possible, it
> isn't clear whether or not the problem occurs frequently enough to warrant
> that effort.
>

I call all such code "working"... It is a bad design.

They are not harmless for any module that initializes global state or
defines exceptions or types and relies on catching those exceptions or
types via isinstance checks when things created in one part of the program
wind up used in another that refers to a different multiply imported copy
of the module. This is even more painful for extension modules. Debugging
these issues is hard.

A cache by os.path.abspath would be a good thing.

I'd personally prefer it to be an ImportError with a message explaining the
problem. Including a pointer to the original successful import when the
import under another sys.modules name is attempted. It'd lead to better
code health. But others likely disagree and prefer to silently return the
existing module.


> * the maintainer level problem is that we actually do this on purpose in
> the standard library's test suite in order to test both pure Python and C
> accelerated variants of various modules. That could be handled by being
> careful about exactly where the reverse lookup cache from filenames back to
> module names is checked and updated, but it does make the problem a bit
> trickier than just "maintain a reverse lookup table from filesystem paths
> to module names and complain if an import gets a hit in that table"
>
> I'm definitely sympathetic to the idea, though.
>
> If we did head in this direction, then we'd also need to accept &
> implement PEP 499 [1] (which proposes aliasing __main__ as
> __main__.__spec__.name in sys.modules when executed with "-m") to avoid
> causing problems.
>

I don't doubt that doing this would require a lot of code cleanup. :)


>
> Cheers,
> Nick.
>
> [1] https://www.python.org/dev/peps/pep-0499/
>
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 572: Statement-Local Name Bindings

[Gregory P. Smith]

On Tue, Feb 27, 2018 at 2:35 PM Chris Angelico  wrote:

> This is a suggestion that comes up periodically here or on python-dev.
> This proposal introduces a way to bind a temporary name to the value
> of an expression, which can then be used elsewhere in the current
> statement.
>
> The nicely-rendered version will be visible here shortly:
>
> https://www.python.org/dev/peps/pep-0572/
>
> ChrisA
>
> PEP: 572
> Title: Syntax for Statement-Local Name Bindings
> Author: Chris Angelico 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 28-Feb-2018
> Python-Version: 3.8
> Post-History: 28-Feb-2018
>
>
> Abstract
> 
>
> Programming is all about reusing code rather than duplicating it.  When
> an expression needs to be used twice in quick succession but never again,
> it is convenient to assign it to a temporary name with very small scope.
> By permitting name bindings to exist within a single statement only, we
> make this both convenient and safe against collisions.
>
>
> Rationale
> =
>
> When an expression is used multiple times in a list comprehension, there
> are currently several suboptimal ways to spell this, and no truly good
> ways. A statement-local name allows any expression to be temporarily
> captured and then used multiple times.
>
>
> Syntax and semantics
> 
>
> In any context where arbitrary Python expressions can be used, a named
> expression can appear. This must be parenthesized for clarity, and is of
> the form `(expr as NAME)` where `expr` is any valid Python expression,
> and `NAME` is a simple name.
>
> The value of such a named expression is the same as the incorporated
> expression, with the additional side-effect that NAME is bound to that
> value for the remainder of the current statement.
>
> Just as function-local names shadow global names for the scope of the
> function, statement-local names shadow other names for that statement.
> They can also shadow each other, though actually doing this should be
> strongly discouraged in style guides.
>
>
> Example usage
> =
>
> These list comprehensions are all approximately equivalent::
>
> # Calling the function twice
> stuff = [[f(x), f(x)] for x in range(5)]
>
> # Helper function
> def pair(value): return [value, value]
> stuff = [pair(f(x)) for x in range(5)]
>
> # Inline helper function
> stuff = [(lambda v: [v,v])(f(x)) for x in range(5)]
>
> # Extra 'for' loop - see also Serhiy's optimization
> stuff = [[y, y] for x in range(5) for y in [f(x)]]
>
> # Expanding the comprehension into a loop
> stuff = []
> for x in range(5):
> y = f(x)
> stuff.append([y, y])
>
> # Using a statement-local name
> stuff = [[(f(x) as y), y] for x in range(5)]
>
> If calling `f(x)` is expensive or has side effects, the clean operation of
> the list comprehension gets muddled. Using a short-duration name binding
> retains the simplicity; while the extra `for` loop does achieve this, it
> does so at the cost of dividing the expression visually, putting the named
> part at the end of the comprehension instead of the beginning.
>
> Statement-local name bindings can be used in any context, but should be
> avoided where regular assignment can be used, just as `lambda` should be
> avoided when `def` is an option.
>
>
> Open questions
> ==
>
> 1. What happens if the name has already been used? `(x, (1 as x), x)`
>Currently, prior usage functions as if the named expression did not
>exist (following the usual lookup rules); the new name binding will
>shadow the other name from the point where it is evaluated until the
>end of the statement.  Is this acceptable?  Should it raise a syntax
>error or warning?
>
> 2. The current implementation [1] implements statement-local names using
>a special (and mostly-invisible) name mangling.  This works perfectly
>inside functions (including list comprehensions), but not at top
>level.  Is this a serious limitation?  Is it confusing?
>
> 3. The interaction with locals() is currently[1] slightly buggy.  Should
>statement-local names appear in locals() while they are active (and
>shadow any other names from the same function), or should they simply
>not appear?
>
> 4. Syntactic confusion in `except` statements.  While technically
>unambiguous, it is potentially confusing to humans.  In Python 3.7,
>parenthesizing `except (Exception as e):` is illegal, and there is no
>reason to capture the exception type (as opposed to the exception
>instance, as is done by the regular syntax).  Should this be made
>outright illegal, to prevent confusion?  Can it be left to linters?
>
> 5. Similar confusion in `with` statements, with the difference that there
>is good reason to capture the result of an expression, and it is also
>very common for `__enter__` methods to return `self`.  In many 

Re: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas

[Gregory P. Smith]

fwiw, we're going to need the tool name in any pragma anyways so the
existing thing that should be common is:

# tool-name: meaningfultoken

It seems like the only convention that makes sense to me.

When I saw your flake8 example of "# noqa: F401" I wanted to rip my eyes
out. Because it didn't mention the tool name *and* it used a numeric code.
Tool authors: use descriptive names!  Otherwise it is meaningless to anyone
reading it.  ex:

import rlcompleter  # pylint: disable=unused-import

Nobody will ever question _what_ that means.

-gps

On Mon, Nov 13, 2017 at 2:19 PM Brett Cannon  wrote:

>
>
> On Mon, Nov 13, 2017, 13:59 Paul Moore,  wrote:
>
>> On 13 November 2017 at 20:43, MRAB  wrote:
>> > On 2017-11-13 19:10, Barry Warsaw wrote:
>> >> The specifics aren't as important as the general use case: multiple
>> >> tools competing for the same valuable real-estate.
>> >>
>> >> I have no ideas how to improve the situation, and of course any
>> solution
>> >> would involve some coordination between all of these tools, but it's
>> >> beginning to feel like a losing battle.  Is there a better way?
>> >>
>> > I suppose that you could have suggest to them that they follow a
>> convention
>> > such as:
>> >
>> > 1. There can be multiple pragmas in a comment, separated by semicolons:
>> if
>> > you don't recognise it, skip past the semicolon.
>> >
>> > 2. A pragma can be prefixed with the name of the tool, e.g. "#
>> flake8.noqa:
>> > F401": if there's a prefix, but it's not yours, skip past the semicolon.
>>
>> An informational PEP defining a common convention for pragma-style
>> comments could standardise things. I'd suggest starting a discussion
>> (somewhere?) with the development teams for the relevant projects
>> (flake8, mypy, coverage...) with the intention of developing such a
>> PEP that they could all support.
>>
>
> And possibly the easiest way to reach them is on the pyqa-dev mailing list.
>
> -brett
>
>
>> Paul
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Please consider adding context manager versions of setUp/tearDown to unittest.TestCase

[Gregory P. Smith]

Neil, you might also bring this up on the
http://lists.idyll.org/listinfo/testing-in-python list as I suspect people
there have opinions on this topic.

-gps

On Mon, Aug 21, 2017 at 9:07 AM Ned Batchelder 
wrote:

> On 8/20/17 9:32 PM, Neil Girdhar wrote:
>
> This question describes an example of the problem:
> https://stackoverflow.com/questions/8416208/in-python-is-there-a-good-idiom-for-using-context-managers-in-setup-teardown.
>  You want to invoke a context manager in your setup/tearing-down, but the
> easiest way to do that is to override run, which seems ugly.
>
> Why not add two methods to unittest.TestCase whose default implementations
> are given below:
>
> class TestCase:
>
> @contextmanager
> def method_context(self):
> self.setUp()
> try:
> yield
> finally:
> self.tearDown()
>
> @contextmanager
> def class_context(self):
> self.setUpClass()
> try:
> yield
> finally:
> self.tearDown()
>
>
> Then, if for example someone wants to use a context manager in setUp, they
> can do so:
>
> class SomeTest(TestCase):
>
> @contextmanager
> def method_context(self):
> with np.errstate(all='raise'):
> with super().method_context():
> yield
>
> Best,
>
> Neil
>
>
> ___
> Python-ideas mailing 
> listPython-ideas@python.orghttps://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> I've achieved a similar effect with this:
>
> def setup_with_context_manager(testcase, cm):
> """Use a contextmanager to setUp a test case.
>
> If you have a context manager you like::
>
> with ctxmgr(a, b, c) as v:
> # do something with v
>
> and you want to have that effect for a test case, call this function
> from
> your setUp, and it will start the context manager for your test, and
> end it
> when the test is done::
>
> def setUp(self):
> self.v = setup_with_context_manager(self, ctxmgr(a, b, c))
>
> def test_foo(self):
> # do something with self.v
>
> """
> val = cm.__enter__()
> testcase.addCleanup(cm.__exit__, None, None, None)
> return val
>
>
> I think the use is easier than yours, which needs too much super and
> @contextmanager boilerplate.
>
>
> --Ned.
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Data Classes (was: Re: JavaScript-Style Object Creation in Python (using a constructor function instead of a class to create objects))

[Gregory P. Smith]

On Thu, May 18, 2017 at 6:38 PM Eric V. Smith  wrote:

> On 5/18/17 2:26 PM, Sven R. Kunze wrote:
> > On 17.05.2017 23:29, Ivan Levkivskyi wrote:
> >> the idea is to write it into a PEP and consider API/corner
> >> cases/implementation/etc.
> >
> > Who's writing it?
>
> Guido, Hynek, and I met today. I'm writing up our notes, and hopefully
> that will eventually become a PEP. I'm going to propose calling this
> feature "Data Classes" as a placeholder until we come up with something
> better.
>
> Once I have something readable, I'll open it up for discussion.
>

Did anything PEP-ish ever come out of this?

-gps
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Security: remove "." from sys.path?

[Gregory P. Smith]

On Mon, Jun 5, 2017 at 10:52 AM Chris Barker  wrote:

> On Mon, Jun 5, 2017 at 3:55 AM, Victor Stinner 
> wrote:
>
>> A minimum change would be to add the (empty string) at the end of
>> sys.path in Python 3.7 rather than adding it at the start.
>>
>> It would increase Python usability since it avoids the "random has no
>> randint() function" caused by a random.py file in the script directory. In
>> my experience, this bug hits every developers starting to learn Python and
>> it can be very strange when you get the error when trying to run IDLE.
>>
>
> But it would add the "why won't python import my file?!??!" problem, which
> newbies also struggle with.
>
> Which leaves me with no suggestion for a solution...
>

We already got rid of implicit relative imports within packages in Python
3. The primary value in continuing to treat the __main__ module differently
is simplicity for learners.

If the problem we're trying to solve (really - that needs to be nailed
down) is that of overriding standard library modules with your own .py
files being an issue (it is! it comes up time and time again)...  This
simple move from beginning to end is the right way to do it.  It does not
remove implicit relative imports for the main module but does remove
implicit stdlib shadowing. which is something nobody ever wants.  and when
they do want that unwantable thing, they should be explicit about it
instead of relying on magic that depends on code being executed as __main__
vs imported as a module from elsewhere.

+0.667 on moving the empty string to the end of sys.path in 3.7 from me.

-gps
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fused multiply-add (FMA)

[Gregory P. Smith]

Makes sense, thanks!  math.fma() it is. :)

On Tue, Jan 17, 2017, 7:48 AM Stephan Houben  wrote:

> Hi Xavier,
>
> In this bright age of IEEE-754 compatible CPUs,
> it is certainly possible to achieve reproducible FP.
> I worked for a company whose software produced bit-identical
> results on various CPUs (x86, Sparc, Itanium) and OSes (Linux, Solaris,
> Windows).
>
> The trick is to closely RTFM for your CPU and compiler, in particular all
> those nice
> appendices related to "FPU control words" and "FP consistency models".
>
> For example, if the author of that article had done so, he might have
> learned
> about the "precision control" field of the x87 status register, which you
> can set
> so that all intermediate operations are always represented as 64-bits
> doubles.
> So no double roundings from double-extended precision.
>
> (Incidentally, the x87-internal double-extended precision is another fine
> example where
> being "more precise on occasion"  usually does not help.)
>
> Frankly not very impressed with that article.
> I could go in detail but that's off-topic, and I will try to fight
> the "somebody is *wrong* on the Internet" urge.
>
> Stephan
>
> 2017-01-17 16:04 GMT+01:00 Xavier Combelle :
>
>
> Generally speaking, there are two reasons why people may *not* want an FMA
> operation.
> 1. They need their results to be reproducible across compilers/platforms.
> (the most common reason)
>
> The reproducibility of floating point calculation is very hard to reach  a
> good survey of the problem is
> https://randomascii.wordpress.com/2013/07/16/floating-point-determinism/
> it mention the fma problem but it only a part of a biggest picture
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fused multiply-add (FMA)

[Gregory P. Smith]

Is there a good reason not to detect single expression multiply adds and
just emit a new FMA bytecode?

Is our goal for floats to strictly match the result of the same operations
coded in unoptimized C using doubles?

Or can we be more precise on occasion?

I guess a similar question may be asked of all C compilers as they too
could emit an FMA instruction on such expressions... If they don't do it by
default, that suggests we match them and not do it either.

Regardless +1 on adding math.fma() either way as it is an expression of
precise intent.

-gps

On Mon, Jan 16, 2017, 10:44 AM David Mertz  wrote:

> My understanding is that NumPy does NOT currently support a direct FMA
> operation "natively."  However, higher-level routines like
> `numpy.linalg.solve` that are linked to MKL or BLAS DO take advantage of
> FMA within the underlying libraries.
>
> On Mon, Jan 16, 2017 at 10:06 AM, Guido van Rossum 
> wrote:
>
> Does numpy support this?
>
> --Guido (mobile)
>
> On Jan 16, 2017 7:27 AM, "Stephan Houben"  wrote:
>
> Hi Steve,
>
> Very good!
> Here is a version which also handles the nan's, infinities,
> negative zeros properly.
>
> ===
> import math
> from fractions import Fraction
>
> def fma2(x, y, z):
> if math.isfinite(x) and math.isfinite(y) and math.isfinite(z):
> result = float(Fraction(x)*Fraction(y) + Fraction(z))
> if not result and not z:
> result = math.copysign(result, x*y+z)
> else:
> result = x * y + z
> assert not math.isfinite(result)
> return result
> ===
>
> Stephan
>
>
> 2017-01-16 12:04 GMT+01:00 Steven D'Aprano :
>
> On Mon, Jan 16, 2017 at 11:01:23AM +0100, Stephan Houben wrote:
>
> [...]
> > So the following would not be a valid FMA fallback
> >
> > double bad_fma(double x, double y, double z) {
> >   return x*y + z;
> > }
> [...]
> > Upshot: if we want to provide a software fallback in the Python code, we
> > need to do something slow and complicated like musl does.
>
> I don't know about complicated. I think this is pretty simple:
>
> from fractions import Fraction
>
> def fma(x, y, z):
> # Return x*y + z with only a single rounding.
> return float(Fraction(x)*Fraction(y) + Fraction(z))
>
>
> When speed is not the number one priority and accuracy is important,
> its hard to beat the fractions module.
>
>
> --
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] api suggestions for the cProfile module

[Gregory P. Smith]

At a glance, all of these sound like good modernizing enhancements for
cprofile. It just takes someone to contribute the work. :)

On Tue, Jan 10, 2017, 8:57 AM Ethan Furman  wrote:

> On 01/10/2017 08:36 AM, Thane Brimhall wrote:
>
> > Does anyone have thoughts on this topic? I assume the silence is because
> >  this suggestion is too trivial to matter.
>
> Sometimes it's just a matter of timing.  :)
>
> > I use cProfile a lot, and would like to suggest three
> backwards-compatible
> >  improvements to the API.
> >
> > 1: When using cProfile on a specific piece of code I often use the
> >  enable() and disable() methods. It occurred to me that this would
> >  be an obvious place to use a context manager.
>
> Absolutely.
>
> > 2: Enhance the `print_stats` method on Profile to accept more options
> >  currently available only through the pstats.Stats class. For example,
> >  strip_dirs could be a boolean argument, and limit could accept an int.
> >  This would reduce the number of cases you'd need to use the more complex
> >  API.
>
> I don't have much experience with cProfile, but this seems reasonable.
>
> > 3: I often forget which string keys are available for sorting. It would
> >  be nice to add an enum for these so a user could have their linter and
> >  IDE check that value pre-runtime. Since it would subclass `str` and
> >  `Enum` it would still work with all currently existing code.
>
> Absolutely!  :)
>
> > The current documentation contains the following code:
> >
> > import cProfile, pstats, io
> > pr = cProfile.Profile()
> > pr.enable()
> > # ... do something ...
> > pr.disable()
> > s = io.StringIO()
> > sortby = 'cumulative'
> > ps = pstats.Stats(pr, stream=s).sort_stats(sortby)
> > ps.print_stats()
> > print(s.getvalue())
> >
> > While the code below doesn't exactly match the functionality above (eg.
> not
> >  using StringIO), I envision the context manager working like this, along
> >  with some adjustments on how to get the stats from the profiler:
> >
> > import cProfile, pstats
> > with cProfile.Profile() as pr:
> >  # ... do something ...
> >  pr.print_stats(sort=pstats.Sort.cumulative, limit=10,
> strip_dirs=True)
> >
> > As you can see, the code is shorter and somewhat more self-documenting.
> The
> >  best thing about these suggestions is that as far as I can tell they
> would
> >  be backwards-compatible API additions.
>
> The `pr.print_stats... line should not be inside the `with` block unless
> you want to profile that part as well.
>
> These suggestions seem fairly uncontroversial.  Have you opened an issue
> on the issue tracker?
>
> The fun part of the patch will be the C code, but a Python
> proof-of-concept would be useful.
>
> --
> ~Ethan~
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/