[Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc.

[Steve Barnes]

In a lot of uses of os.walk it is desirable to skip version control 
directories, (which are usually hidden directories), to the point that 
almost all of the examples given look like:

import os
for root, dirs, files in os.walk(some_dir):
 if 'CVS' in dirs:
 dirs.remove('CVS')  # or .svn or .hg etc.
 # do something...

But of course there are many version control systems to the point that 
much of my personal code looks like, (note that I have to use a 
multitude of version control systems due to project requirements):


import os
vcs_dirs = ['.hg', '.svn', 'CSV', '.git', '.bz']  # Version control 
directory names I know


for root, dirs, files in os.walk(some_dir):
 for dirname in vcs_dirs:
 dirs.remove(dirname)

I am sure that I am missing many other version control systems but the 
one thing that all of the ones that I am familiar with default to 
creating their files in hidden directories. I know that the above 
sometimes hits problems on Windows if someone manually created a 
directory and you end up with abortions such as Csv\ or .SVN 

Since it could be argued that hidden directories are possibly more 
common than simlinks, (especially in the Windows world of course), and 
that hidden directories have normally been hidden by someone for a 
reason it seems to make sense to me to normally ignore them in directory 
traversal.

Obviously there are also occasions when it makes sense to include VCS, 
or other hidden, directories files, (e.g. "Where did all of my disk 
space go?" or "delete recursively"), so I would like to suggest 
including in the os.walk family of functions an additional parameter to 
control skipping all hidden directories - either positively or negatively.

Names that spring to mind include:
  * nohidden
  * nohidden_dirs
  * hidden
  * hidden_dirs

This change could be made with no impact on current behaviour by 
defaulting to hidden=True (or nohidden=False) which would just about 
ensure that no existing code is broken or quite a few bugs in existing 
code could be quietly fixed, (and some new ones introduced), by 
defaulting to this behaviour.

Since the implementation of os.walk has changed to use os.scandir which 
exposes the returned file statuses in the os.DirEntry.stat() the 
overhead should be minimal.

An alternative would be to add another new function, say os.vwalk(), to 
only walk visible entries.

Note that a decision would have to be made on whether to include such 
filtering when topdown is False, personally I am tempted to include the 
filtering so as to maintain consistency but ignoring the filter when 
topdown is False, (or if topdown is False and the hidden behaviour is 
unspecified), might make sense if the skipping of hidden directories 
becomes the new default (then recursively removing files & directories 
would still include processing hidden items by default).

If this receives a positive response I would be happy to undertake the 
effort involved in producing a PR.
-- 
Steve (Gadget) Barnes
Any opinions in this message are my personal opinions and do not reflect 
those of my employer.

---
This email has been checked for viruses by AVG.
http://www.avg.com

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] __dir__ in which folder is this py file

[Barry Warsaw]

Yuval Greenfield wrote:


I often need to reference a script's current directory. I end up writing:

import os
SRC_DIR = os.path.dirname(__file__)


The question I have is, why do you want to reference the script's 
current directory?


If the answer is to access other files in that directory, then please 
consider using importlib.resources (for Python 3.7) and 
importlib_resources (for Python 2.7, 3.4-3.6).


__file__ simply isn't safe, and pkg_resources can be a performance 
killer.  The problem of course is that if you're writing an application 
and *any* of your dependencies use either technique, you are going to 
pay for it.  This is exactly why Brett and I wrote importlib.resources. 
We wanted a consistent API, that allows custom loaders to play along, 
and which is about as efficient as possible, uses Python's import 
machinery, and is safe for uses like zipapps.


now-you-don't-have-to-attend-my-pycon-talk-ly y'rs,
-Barry


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Tim Peters]

[Tim]
> ... The ":= is treated specially in comprehensions" idea is aimed
> more at them than at people who think invoking a synthetic
> anonymous lambda "is obvious".

It occurs to me that, while suggestive, this is an unhelpful way to
express it.  It's not at all that the semantics of ":=" change inside
a listcomp/genexp, it's that the latter's idea of intended _scopes_
for names is made more nuanced (inside a synthetic function created to
implement a listcomp/genexp, names bound by "=" are local; names bound
by ":=" are nonlocal; names bound by both are "who cares?"-
compiler-time error would be fine by me, or the first person to show a
real use case wins).

Regardless, the runtime implementation of ":=" remains the same everywhere.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A cute Python implementation of itertools.tee

[Tim Peters]

I posted this several weeks ago, just for fun - an obscure but
surprisingly brief Python implementation of itertools.tee, sharing a
single stream (as a singly linked list) among all the returned
iterables.

Didn't think about it again until today, when recent discussions of
lexical scoping made me wonder "hmm - when's the last time I even used
nonlocal?".  Well, this was.  And shortly thereafter, "but there's no
need to keep track of `last` at all!"  I have no idea where that
thought came from :-)

> def mytee(xs, n):
> last = [None, None]
>
> def gen(it, mylast):
> nonlocal last
> while True:
> mylast = mylast[1]
> if not mylast:
> mylast = last[1] = last = [next(it), None]
> yield mylast[0]
>
> it = iter(xs)
> return tuple(gen(it, last) for _ in range(n))

So, in fact, the inner function there can be replaced by the even briefer:

def gen(it, mylast):
while True:
if mylast[1] is None:
mylast[1] = [next(it), None]
mylast = mylast[1]
yield mylast[0]

To make it more relevant to current discussions, collapsing the last
two lines using a binding expression:

yield (mylast := mylast[1])[0]

isn't even slightly tempting.  Most of the cases where it would be
_possible_ to use binding expressions don't strike me as being
_sensible_ places to use them - slamming together conceptually
different tasks just because it's possible to do so.  But because name
bindings are so very common, that still leaves plenty where the
transformation leaves the code clearer to my eyes (usually just a
little, rarely a lot).  There are no such cases in the code above.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Steven D'Aprano]

On Mon, May 07, 2018 at 10:38:09AM -0700, Guido van Rossum wrote:

> The only solution that makes sense to me is Steven's (2). (1) is what the
> PEP currently says and what Tim doesn't want; (3) has no precedent
> (function defaults don't really work like this) and just gets my hackles
> all up. (I can't even tell if Steven meant it as a serious proposal.)

It doesn't get my hackles up as much as you, but its not really what I 
want. It's just a compromise between what I *don't* want (1), which 
fails to solve the original motivating example that started this 
discussion, and what Chris was pushing back against (2).


> There's one corner case (again) -- class scopes. If the containing scope is
> a function, everything's fine, we can use the existing closure machinery.
> If the containing scope is global, everything's fine too, we can treat it
> as a global. But if the containing scope is a class, we can't easily extend
> the current machinery. But this breakage is similar to the existing
> breakage with comprehensions in class scope that reference class variables:
[...]
> I propose to punt on this case. If we want to fix it we can fix it in a
> number of ways and the fix can easily apply to both getting and setting --
> but this is a separate fix (and we should take it out of PEP 572).

+1

Whether the current class behaviour is "broken" or desirable or 
somewhere in between, it is what we have now and its okay if binding 
expressions have the same behaviour.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter

[Danilo J. S. Bellini]

(1)

On 7 May 2018 at 01:07, Raymond Hettinger 
wrote:

> He could've written:
>
> get = rpartial(getattr, 'foo', None)
> return get(args) or get(config) or get(env)
>

That's somewhat hybrid, it would behave like:

lambda obj: getattr(obj, "foo", None),

but one might expect a "reversed partial" to have its arguments reversed as
well, like:

lambda obj: getattr(obj, None, "foo").

(2)

Why getattr can't accept keyword arguments?

>>> getattr_zero = partial(getattr, default=0)
>>> getattr_zero({}, "blah")
Traceback (most recent call last):
  File "", line 1, in 
TypeError: getattr() takes no keyword arguments

Since partial/partialmethod can apply keyword arguments, allowing the
"default=" keyword argument in getattr would be an alternative to the
"rpartial".

(3)

At one time, lambda was the one obvious way. [...]
>

I really like lambdas, but there are some "closure gotchas" like:

>>> for idx in indices:
... do_something(lambda seq: seq[idx])

If the lambda is stored somewhere to be called after the loop ends (or
after its iteration), the seq[idx] would load the item with the last
iterated index. This:

>>> for idx in indices:
... do_something(itemgetter(idx))

would behave more like this instead:

>>> for idx in indices:
... do_something((lambda i: (lambda seq: seq[i]))(idx))

Which I wouldn't call "the one obvious way".

(4)

If itemgetter and attrgetter only did a single lookup, a default might make
> sense.  However, that doesn't fit well with multiple and/or chained lookups
> where are number of options are possible. (See https://bugs.python.org/
> issue14384#msg316222 for examples and alternatives).

As attrgetter/itemgetter might get heterogeneus data, I would expect a
per-get default, not [just] a global default.
Perhaps something like:

>>> itemgetter(-1, 0, -2,
...default0="first argument default",
...default1=["second", "argument", "default"],
...default={"global": "default"},
... )

-- 
Danilo J. S. Bellini
---
"*It is not our business to set up prohibitions, but to arrive at
conventions.*" (R. Carnap)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Tim Peters]

[Terry Reedy ]
> ...
> If I am understanding correctly, this would also let one *intentionally
> 'leak' (export) the last value of the loop variable when wanted.
>
> [math.log(xlast:=x) for x in it if x > 0]
> print(xlast)

Yup!  You can do that today by emulating a nonlocal "cell" reference,
but I don't recommend it ;-)

xlast_cell = [None]
[math.log(x) for x in it if x > 0
 for xlast_cell[0] in [x]]
print(xlast_cell[0])
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Terry Reedy]

On 5/7/2018 1:38 PM, Guido van Rossum wrote:
I am convinced by Tim's motivation. I hadn't thought of this use case 
before -- I had mostly thought "local scope in a comprehension or 
generator expression is the locals of the synthetic function". But Tim's 
reasoning feels right.


The only solution that makes sense to me is Steven's (2). (1) is what 
the PEP currently says and what Tim doesn't want; (3) has no precedent 
(function defaults don't really work like this) and just gets my hackles 
all up. (I can't even tell if Steven meant it as a serious proposal.)


So let's see if we can change PEP 572 so that := inside a comprehension 
or generator expression al ways assigns to a variable in the containing 
scope.


It may be inconsistent with the scope of the loop control variable, but 
it's consistent with uses of := outside comprehensions:


   [x := 0]
   [x := i for i in range(1)]

both have the side effect of setting x to zero. I like that.


If I am understanding correctly, this would also let one *intentionally 
'leak' (export) the last value of the loop variable when wanted.


[math.log(xlast:=x) for x in it if x > 0]
print(xlast)

There's one corner case (again) -- class scopes. If the containing scope 
is a function, everything's fine, we can use the existing closure 
machinery. If the containing scope is global, everything's fine too, we 
can treat it as a global. But if the containing scope is a class, we 
can't easily extend the current machinery. But this breakage is similar 
to the existing breakage with comprehensions in class scope that 
reference class variables:


   class C:
   hosts = ['boring', 'haring', 'tering']
   full_hosts = [host + suffix for suffix in ('.cwi.nl 
', '.com') for host in hosts]


Traceback (most recent call last):
   File "", line 1, in 
   File "", line 3, in C
   File "", line 3, in 
NameError: name 'hosts' is not defined


This is a special case of the fact that no function called in class 
scope can access class variables, even if defined in the class scope.


>>> class C:
x = 0
def f():
return x
z = f()

Traceback (most recent call last):
  File "", line 1, in 
class C:
  File "", line 5, in C
z = f()
  File "", line 4, in f
return x
NameError: name 'x' is not defined

I would find it strange if only functions defined by a comprehension 
were given new class scope access.


PS1. The various proposals that add random extra keywords to the syntax 
(like 'for nonlocal i') don't appeal to me at all.


PS2. IIRC the reason we gave loop control variables their own scope was 
the poor imagination of many people when it comes to choosing loop 
control variable names. We had seen just too many examples of


   for x in something:
   ...lots of code using x...
   blah blah [x+1 for x in something else]
   ...some more code using x, broken...

It's true that this can also happen with a for-loop statement nested 
inside the outer loop (and it does) but the case of a comprehension was 
easier to miss. I've never looked back.


PS3. Comprehensions and generator expressions should be interchangeable. 
They just look too similar to have different semantics (and the possibly 
delayed execution of generator expressions is not an issue -- it's rare, 
and closure semantics make it work just fine).


To me, this is the prime justification for the 3.0 comprehension change. 
 I currently see a comprehension as a specialized generator expression. 
 A generator expression generalizes math set builder notation. If left 
'raw', the implied function yields the values generated (what else could 
it do?).  If a collection type is indicated by the fences and expression 
form, values are instead added to an anonymous instance thereof.


--
Terry Jan Reedy


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] string method count()

[Neil Girdhar]

Regular expressions are not just "an order of magnitude better"—they're 
asymptotically faster.  
See https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm 
for a non-regular-expression algorithm.

On Thursday, April 26, 2018 at 5:45:20 AM UTC-4, Jacco van Dorp wrote:
>
> or build it yourself...
>
> def str_count(string, sub):
>   c = 0
>   for c in range(len(string)-len(sub)):
> if string[c:].startswith(sub):
>   c += 1
>   return c
>
> (probably some optimizations possible...)
>
> Or in one line with a generator expression:
> def str_count(string, sub):
>   return sum(string[c:].startswith(sub) for c in 
> range(len(string)-len(sub)))
>
> regular expressions would probably be at least an order of magnitude
> better in speed, if it's a bottleneck to you. But pure python
> implementation for this is a lot easier than it would be for the
> current string.count().
>
> 2018-04-26 8:57 GMT+02:00 Wes Turner :
> >
> >
> > On Wednesday, April 25, 2018, Steven D'Aprano  > wrote:
> >>
> >> On Wed, Apr 25, 2018 at 11:22:24AM -0700, Julia Kim wrote:
> >> > Hi,
> >> >
> >> > There’s an error with the string method count().
> >> >
> >> > x = ‘AAA’
> >> > y = ‘AA’
> >> > print(x.count(y))
> >> >
> >> > The output is 1, instead of 2.
> >>
> >> Are you proposing that there ought to be a version of count that looks
> >> for *overlapping* substrings?
> >>
> >> When will this be useful?
> >
> >
> > "Finding a motif in DNA"
> > http://rosalind.info/problems/subs/
> >
> > This is possible with re.find, re.finditer, re.findall, regex.findall(,
> > overlapped=True), sliding window
> > 
> https://stackoverflow.com/questions/2970520/string-count-with-overlapping-occurrences
> >
> > n-grams can be by indices or by value.
> > count = len(indices)
> > https://en.wikipedia.org/wiki/N-gram#Examples
> >
> > 
> https://en.wikipedia.org/wiki/String_(computer_science)#String_processing_algorithms
> >
> > https://en.wikipedia.org/wiki/Sequential_pattern_mining
> >
> >>
> >>
> >> --
> >> Steve
> >> ___
> >> Python-ideas mailing list
> >> python...@python.org 
> >> https://mail.python.org/mailman/listinfo/python-ideas
> >> Code of Conduct: http://python.org/psf/codeofconduct/
> >
> >
> > ___
> > Python-ideas mailing list
> > python...@python.org 
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
> ___
> Python-ideas mailing list
> python...@python.org 
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Allow mutable builtin types (optionally)

[Petr Viktorin]

On 05/07/18 11:37, Eloi Gaudry wrote:

Hi,

I'd like to bring back this discussion (from 2005, by Greg):
https://bugs.python.org/issue1229239

Briefly, non-heap types cannot have their
attributes changed by Python code. This makes sense for python builtin
types, but not for the types defined in extension/modules.

As we have been using this patch for the very same reasons and for more
than 10 years, I think it might make sense to reconsider the discussion
that Greg started.

The main question at that time was "why not using a heap type instead
?" (because heap-type do not have this limitation).

But I think that the right question could have been "why imposing such
a restriction on non-heap types as defined in (non Python core)
extensions ?".

I mean, to my knowledge, there is no reason why a type should be
allocated on the heap (https://docs.python.org/2/c-api/typeobj.html) to
be able to change its attributes at Python level.


One reason is sub-interpreter support: you can have multiple 
interpreters per process, and those shouldn't influence each other.

(see https://docs.python.org/3/c-api/init.html#sub-interpreter-support)

With heap types, each sub-interpreter can have its own copy of the type 
object. But with builtins, changes done in one interpreter would be 
visible in all the others.




I'm not saying that all non-heap types should be able to do so, just
that it would make sense to allow this behavior, as an option (bit
flag).

At the implementation level, the changes needed are really limited
(about a few lines):
- Include/object.h
- Objects/typeobject.c:

Eloi

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Tim Peters]

[Tim]
> While I don't have real use cases beyond that, given that much,
> "consistency" kicks in to suggest that:
>
> def f():
>  [x := 42 for x in range(1)]
>
> makes `x` local to `f` despite that x wasn't bound elsewhere in f's body.
>
> def f():
> global x
>  [x := 42 for x in range(1)]
>
> binds the global `x`.
>
> def f():
> nonlocal x
>  [x := 42 for x in range(1)]
>
> binds `x` in the closest-containing scope in which `x` is local.  The
> last two act as if the declaration of `x` in `f` were duplicated at
> the start of the synthetic function.
>
> More succinctly, that `x := RHS` in a synthetic function "act the
> same" as `x = RHS` appearing in the scope directly containing the
> synthetic function.

Oh, fudge - I wasn't trying to make a silly subtle point by reusing
`x` as the `for` variable too.  Pretend those all said "for i in
range(1)" instead.  Of course what happens if `x` is used in both
places needs to be defined, but that's entirely beside the intended
point _here_.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Tim Peters]

[Tim]
>> In a different thread I noted that I sometimes want to write code like
>> this:
>>
>>  while any(n % p == 0 for p in small_primes):
>>  # divide p out - but what's p?
>>
>> But generator expressions hide the value of `p` that succeeded, so I
>> can't.  `any()` and `all()` can't address this themselves - they
>> merely work with an iterable of objects to evaluate for truthiness,
>> and know nothing about how they're computed.  If you want to identify
>> a witness (for `any()` succeeding) or a counterexample (for `all()`
>> failing), you need to write a workalike loop by hand.

[Brendan Barnwell ]
 > I agree that is a limitation, and I see from a later message in the
> thread that Guido finds it compelling, but personally I don't find that that
> particular case such a showstopper that it would tip the scales for me
> either way.  If you have to write the workalike look that iterates and
> returns the missing value, so be it.  That's not a big deal.

Guido didn't find it compelling:  for that specific example to show
`p` would require that for-loop targets "leak", and he remains opposed
to that.  I don't want that changed either.

The issue instead is what the brand-new proposed ":=" should do, which
isn't used in that example at all.  Whether that specific example can
be written in 500 other ways (of course it can) isn't really relevant.

One of the ironies already noted is that PEP 572 gives an example of
something that _won't_ work ("progressive_sums") which happens to be
the very use case that started the current debate about assignment
expressions to begin with.  That raises the very same issue about ":="
that "the obvious" rewrite of my example at the top raises.  Which
suggests to me (& apparently to Guido too) that there may be a real
issue here worth addressing.

There are many use cases for binding expressions outside of
synthetically generated functions.  For PEP 572, it's the totality
that will be judged, not just how they might work inside list
comprehensions and generator expressions (the only topics in _this_
particular thread), let alone how they work in one specific example.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Tim Peters]

[Guido]
> I am convinced by Tim's motivation. I hadn't thought of this use case before
> -- I had mostly thought "local scope in a comprehension or generator
> expression is the locals of the synthetic function". But Tim's reasoning
> feels right.

I'm trying very hard _not_ to reason.  That is, I'm looking at code
and trying to figure out what would actually work well, logic &
consistency & preconceptions be damned.  "Reasons" can be made up
after the fact - which is how the world actually works regardless ;-)


> The only solution that makes sense to me is Steven's (2). (1) is what the
> PEP currently says and what Tim doesn't want; (3) has no precedent (function
> defaults don't really work like this) and just gets my hackles all up. (I
> can't even tell if Steven meant it as a serious proposal.)

He doesn't want (3) either.  I can channel him on that.


> So let's see if we can change PEP 572 so that := inside a comprehension or
> generator expression always assigns to a variable in the containing scope.

While I don't have real use cases beyond that, given that much,
"consistency" kicks in to suggest that:

def f():
 [x := 42 for x in range(1)]

makes `x` local to `f` despite that x wasn't bound elsewhere in f's body.

def f():
global x
 [x := 42 for x in range(1)]

binds the global `x`.

def f():
nonlocal x
 [x := 42 for x in range(1)]

binds `x` in the closest-containing scope in which `x` is local.  The
last two act as if the declaration of `x` in `f` were duplicated at
the start of the synthetic function.

More succinctly, that `x := RHS` in a synthetic function "act the
same" as `x = RHS` appearing in the scope directly containing the
synthetic function.

Does that generalize to class scope too?  I don't know.  I never use
fancy stuff in class scopes, and have no idea how they work anymore.
So long as "def method(self, ...)" continues to work, I'm happy ;-)


> It may be inconsistent with the scope of the loop control variable, but it's
> consistent with uses of := outside comprehensions:
>
>   [x := 0]
>   [x := i for i in range(1)]
>
> both have the side effect of setting x to zero. I like that.

And is what anyone would expect if they didn't think too much about it.

... [snipping stuff about class scope - nothing to add] ...


> PS1. The various proposals that add random extra keywords to the syntax
> (like 'for nonlocal i') don't appeal to me at all.

They're appealing to the extent that "explicit is better than
implicit" for people who actually understand how all this stuff is
implemented.  I don't know what percentage of Python programmers that
is, though.  I've certainly, e.g., seen many on Stackoverflow who can
make a list comprehension work who couldn't distinguish a lexical
scope from an avocado.  The ":= is treated specially in
comprehensions" idea is aimed more at them than at people who think
invoking a synthetic anonymous lambda "is obvious".


> PS2. IIRC the reason we gave loop control variables their own scope was the
> poor imagination of many people when it comes to choosing loop control
> variable names. We had seen just too many examples of
>
>   for x in something:
>   ...lots of code using x...
>   blah blah [x+1 for x in something else]
>   ...some more code using x, broken...
>
> It's true that this can also happen with a for-loop statement nested inside
> the outer loop (and it does) but the case of a comprehension was easier to
> miss. I've never looked back.

I don't want to change anything about any of that - already believe
Python struck the best balance possible.


> PS3. Comprehensions and generator expressions should be interchangeable.
> They just look too similar to have different semantics (and the possibly
> delayed execution of generator expressions is not an issue -- it's rare, and
> closure semantics make it work just fine).

Wholly agreed.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter

[Vincent Maillol]

Hi everybody,

> Have you encountered situations yourself where this would make a difference ?
I need to get multiple objects in several dicts using the same set of keys.

I wanted to use itemgetter to reduce the number of lines of code but I
have mixed up getattr/dict.get
that have default parameter with attrgetter/itemgetter.

> At some point, we're really better off just using a lambda.
I kept severals line with `.get('key', value)`, I find that is more readable.

> Also, I'm concerned that about increasing the complexity of itemgetter() API 
> to serve an occasional exotic use case rather that being easy to learn and 
> remember for the common cases.
I understand you, each additional parameter increases the cost of
maintenance and update operator module
will take a lot of work we should update c and python module, and
ensure compatibility with pickle.

I did it just to try it with itemgetter

https://github.com/Maillol/cpython/compare/master...Add-default-parameter-to-operator-itemgetter

I don't know if we add parameter `default` to itemgetter, getitem,
attrgetter the API will become
more complexe or more consistency, but I agree with you, it is an
occasional use case and we can always
use `my_dict.get`.

2018-05-07 6:07 GMT+02:00 Raymond Hettinger :
>
>> On May 6, 2018, at 6:00 AM, Steven D'Aprano  wrote:
>>
>> On Thu, May 03, 2018 at 04:32:09PM +1000, Steven D'Aprano wrote:
>>
>>> Maybe I'm slow today, but I'm having trouble seeing how to write this as
>>> a lambda.
>>
>> Yes, I was definitely having a "cannot brain, I have the dumb" day,
>> because it is not that hard to write using lambda. See discussion here:
>>
>> https://mail.python.org/pipermail/python-list/2018-May/732795.html
>>
>> If anything, the problem is a plethora of choices, where it isn't clear
>> which if any is the best way, or the One Obvious Way
>
> At one time, lambda was the one obvious way.  Later, partial, itemgetter, 
> attrgetter, and methodcaller were added to express common patterns for 
> key-functions and map().  If needed, the zoo of lambda alternatives could be 
> further extended to add a rpartial() function that partials from the right.  
> That would have helped with Miki's example.  Instead of:
>
> get = attrgetter('foo', None)
> return get(args) or get(config) or get(env)
>
> He could've written:
>
> get = rpartial(getattr, 'foo', None)
> return get(args) or get(config) or get(env)
>
> If itemgetter and attrgetter only did a single lookup, a default might make 
> sense.  However, that doesn't fit well with multiple and/or chained lookups 
> where are number of options are possible. (See 
> https://bugs.python.org/issue14384#msg316222 for examples and alternatives).
>
>
> Raymond
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Brendan Barnwell]

On 2018-05-06 18:32, Tim Peters wrote:

In a different thread I noted that I sometimes want to write code like this:

 while any(n % p == 0 for p in small_primes):
 # divide p out - but what's p?

But generator expressions hide the value of `p` that succeeded, so I
can't.  `any()` and `all()` can't address this themselves - they
merely work with an iterable of objects to evaluate for truthiness,
and know nothing about how they're computed.  If you want to identify
a witness (for `any()` succeeding) or a counterexample (for `all()`
failing), you need to write a workalike loop by hand.


	I agree that is a limitation, and I see from a later message in the 
thread that Guido finds it compelling, but personally I don't find that 
that particular case such a showstopper that it would tip the scales for 
me either way.  If you have to write the workalike look that iterates 
and returns the missing value, so be it.  That's not a big deal.


--
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."

   --author unknown
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] __dir__ in which folder is this py file

[Brendan Barnwell]

On 2018-05-07 09:17, Steven D'Aprano wrote:

I'm arguing that for some people, your preferred syntax*is*  more
distracting and hard to comprehend than the more self-descriptive
version with named functions. And its not just a matter of*learning*
the API, it is a matter of using it so often that it ceases to look
weird and looks natural.[1]



But if you think it isn't distracting, I think you are mistaken, and I
think we ought to show caution in making it a built-in or an offical
part of the module API.


	As an aside, this has some parallels with the recent thread about 
"objectively quantifying readability".  Saying things like "you are 
mistaken" implies that there is an objective ground truth about what is 
distracting and what is not.  And personally I agree that there is such 
an objective ground truth, and that it is based on facts about human 
pyschology (although I don't think I agree with you about this 
particular case).  Of course, there may be differences in how 
individuals react to things, but there is a real sense in which 
different syntaxes, constructs, etc., have something like a "mean level 
of confusion" which represents how easy to deal with people in general 
find them on average, and by which they can be meaningfully compared. 
I'm not sure how to proceed to uncover this (unless the PSF starts 
funding psychological experiments!), but I do think it would be good if 
we could find ways to get at something like hard evidence for claims 
about whether things "really are" distracting, readable, unreadable, 
intuitive, etc.


--
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."

   --author unknown
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Inline assignments using "given" clauses

[Brett Cannon]

On Fri, 4 May 2018 at 05:07 Nick Coghlan  wrote:

> (Note: Guido's already told me off-list that he doesn't like the way this
> spelling reads, but I wanted to share it anyway since it addresses one of
> the recurring requests in the PEP 572 discussions for a more targeted
> proposal that focused specifically on the use cases that folks had agreed
> were reasonable potential use cases for inline assignment expressions.
>
> I'll also note that another potential concern with this specific proposal
> is that even though "given" wasn't used as a term in any easily discovered
> Python APIs back when I first wrote PEP 3150, it's now part of the
> Hypothesis testing API, so adopting it as a keyword now would be markedly
> more disruptive than it might have been historically)
>
> Recapping the use cases where the inline assignment capability received
> the most agreement regarding being potentially more readable than the
> status quo:
>
> 1. Making an "exactly one branch is executed" construct clearer than is
> the case for nested if statements:
>
> if m := pattern.search(data):
> ...
> elif m := other_pattern.search(data):
> ...
> else:
> ...
>
> 2. Replacing a loop-and-a-half construct:
>
> while m := pattern.search(remaining_data):
> ...
>
> 3. Sharing values between filtering clauses and result expressions in
> comprehensions:
>
> result = [(x, y, x/y) for x in data if (y := f(x))]
>
> The essence of the given clause concept would be to modify *these specific
> cases* (at least initially) to allow the condition expression to be
> followed by an inline assignment, of the form "given TARGET = EXPR". (Note:
> being able to implement such a syntactic constraint is a general
> consequence of using a ternary notation rather than a binary one, since it
> allows the construct to start with an arbitrary expression, without
> requiring that expression to be both the result of the operation *and* the
> value bound to a name - it isn't unique to the "given" keyword specifically)
>
> While the leading keyword would allow TARGET to be an arbitrary assignment
> target without much chance for confusion, it could also be restricted to
> simple names instead (as has been done for PEP 572.
>
> With that spelling, the three examples above would become:
>
> # Exactly one branch is executed here
> if m given m = pattern.search(data):
> ...
> elif m given m = other_pattern.search(data)):
> ...
> else:
> ...
>
> # This name is rebound on each trip around the loop
> while m given m = pattern.search(remaining_data):
> ...
>
> # "f(x)" is only evaluated once on each iteration
> result = [(x, y, x/y) for x in data if y given y = f(x)]
>

My brain wants to drop the variable name in front of 'given':

if given m = pattern.search(data):

while given m = pattern.search(remaining_data):

Maybe it's because the examples use such a short variable name?

if match given match = pattern.search(data):
vs.
if given match = pattern.search(data);

Nope, I still like mine more. ;)

-Brett


>
> Constraining the syntax that way (at least initially) would avoid poking
> into any dark corners of Python's current scoping and expression execution
> ordering semantics, while still leaving the door open to later making
> "result given NAME = expr" a general purpose ternary operator that returns
> the LHS, while binding the RHS to the given name as a side effect.
>
> Using a new keyword (rather than a symbol) would make the new construct
> easier to identify and search for, but also comes with all the downsides of
> introducing a new keyword. (Hence the not-entirely-uncommon suggestion of
> using "with" for a purpose along these lines, which runs into a different
> set of problems related to trying to use "with" for two distinct and
> entirely unrelated purposes).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Guido van Rossum]

On Mon, May 7, 2018 at 6:24 AM, Serhiy Storchaka 
wrote:

> I just don't understand why you need a new keyword for writing runtime
> checks.
>

Oh, that's pretty clear. The OP wants to be able to turn these checks off
with some flag he can set/clear at runtime, and when it's off he doesn't
want to incur the overhead of evaluating the check. The assert statement
has the latter property, but you have to use -O to turn it off. He
basically wants a macro so that

  runtime_assert()

expands to

  if  and ():
  raise AssertionError

In Lisp this would be easy. :-)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Guido van Rossum]

I am convinced by Tim's motivation. I hadn't thought of this use case
before -- I had mostly thought "local scope in a comprehension or generator
expression is the locals of the synthetic function". But Tim's reasoning
feels right.

The only solution that makes sense to me is Steven's (2). (1) is what the
PEP currently says and what Tim doesn't want; (3) has no precedent
(function defaults don't really work like this) and just gets my hackles
all up. (I can't even tell if Steven meant it as a serious proposal.)

So let's see if we can change PEP 572 so that := inside a comprehension or
generator expression al ways assigns to a variable in the containing scope.

It may be inconsistent with the scope of the loop control variable, but
it's consistent with uses of := outside comprehensions:

  [x := 0]
  [x := i for i in range(1)]

both have the side effect of setting x to zero. I like that.

There's one corner case (again) -- class scopes. If the containing scope is
a function, everything's fine, we can use the existing closure machinery.
If the containing scope is global, everything's fine too, we can treat it
as a global. But if the containing scope is a class, we can't easily extend
the current machinery. But this breakage is similar to the existing
breakage with comprehensions in class scope that reference class variables:

  class C:
  hosts = ['boring', 'haring', 'tering']
  full_hosts = [host + suffix for suffix in ('.cwi.nl', '.com') for
host in hosts]

Traceback (most recent call last):
  File "", line 1, in 
  File "", line 3, in C
  File "", line 3, in 
NameError: name 'hosts' is not defined

I propose to punt on this case. If we want to fix it we can fix it in a
number of ways and the fix can easily apply to both getting and setting --
but this is a separate fix (and we should take it out of PEP 572).

PS1. The various proposals that add random extra keywords to the syntax
(like 'for nonlocal i') don't appeal to me at all.

PS2. IIRC the reason we gave loop control variables their own scope was the
poor imagination of many people when it comes to choosing loop control
variable names. We had seen just too many examples of

  for x in something:
  ...lots of code using x...
  blah blah [x+1 for x in something else]
  ...some more code using x, broken...

It's true that this can also happen with a for-loop statement nested inside
the outer loop (and it does) but the case of a comprehension was easier to
miss. I've never looked back.

PS3. Comprehensions and generator expressions should be interchangeable.
They just look too similar to have different semantics (and the possibly
delayed execution of generator expressions is not an issue -- it's rare,
and closure semantics make it work just fine).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Brett Cannon]

On Mon, 7 May 2018 at 06:28 Serhiy Storchaka  wrote:

> 07.05.18 12:58, Eloi Gaudry пише:
> > I didn't mean to replace the current (debug) assert but I wanted to add
> > another one that would allow to be switch on or off on production builds.
> >
> > The need for a new keyword (not syntax) comes from this difference.
> >
> >
> >   I cannot think of another example that would convince you of the
> > benefit of having a specific keyword for such a runtime assert.  I do
> > believe that having such a feature in non-debug build is more than
> > interesting but indeed needed.
>
> I just don't understand why you need a new keyword for writing runtime
> checks.
>

My question is how is this different to running with -O which leaves the
assert statement out of the bytecode and so you avoid any run-time cost of
the statement entirely?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] __dir__ in which folder is this py file

[Eric Fahlgren]

On Sun, May 6, 2018 at 9:30 PM, Mike Miller 
wrote:

> On 2018-05-06 19:13, Nick Coghlan wrote:
>
>> Specifically, the ones I'd have in mind would be:
>>
>> - dirname (aka os.path.dirname)
>> - joinpath (aka os.path.join)
>> - abspath (aka os.path.abspath)
>>
> Yes, I end up importing those in most scripts currently.  Just "join" has
> worked fine, although I could imagine someone getting confused about it.


Our homebuilt pre-pathlib package has an 'abs_path' parameter in join, so
that could easily eliminate the abspath function itself:

>>> joinpath('.', abs_path=True)

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] __dir__ in which folder is this py file

[Brett Cannon]

On Mon, 7 May 2018 at 08:17 Serhiy Storchaka  wrote:

> 07.05.18 17:42, Eric Snow пише:
> > I'm not necessarily saying we should add ModuleSpec.dirname(), but it
> > (or something like it) is what I'd advocate for *if* we were to add a
> > convenient shortcut to the directory a module is in.  FWIW, I'd
> > probably use it.
>
> The question is *why* you need the absolute path to the directory the
> module is in? Taking into account the availability of
> importlib.resources etc.
>

And just "why", and "how often"? I'm sure we have all done it before, but
it isn't something that comes up *constantly*. And duplicating part of the
details what __spec__.location contains just to save an import and a line
to strip off the file seems unnecessary. Plus, this doesn't take into
consideration the fact that not every module is going to exist in a
directory (e.g. what if I loaded from a sqlite database?).

IOW I'm -1 on this addition to modules as I don't think it's difficult
enough or used enough to warrant adding the overhead of providing it.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Chris Angelico]

On Tue, May 8, 2018 at 12:26 AM, Steven D'Aprano  wrote:
> Here's a sketch of how I think locals are currently handled:
>
> 1. When a function is compiled, the compiler does a pass over the
>source and determines which locals are needed.
>
> 2. The compiler builds an array of slots, one for each local,
>and sets the initial value of the slot to "empty" (undefined).
>
> 3. When the function is called, if it tries reading from a local's
>slot which is still empty, it raises UnboundLocalError.
>
> (am I close?)

Yeah, I'm pretty sure that's all correct.

> Here's the change I would suggest:
>
> 2. The compiler builds an array of slots, one for each local:
>
> 2a. For locals that are the target of binding-expression only:
>
> - look up the target in the current scope (that is, not the
>   comprehension's scope, but the scope that the comprehension
>   is inside) using the normal lookup rules, as if you were
>   compiling "lambda x=x: None" and needed the value of x;
>
> - if the target is undefined, then swallow the error and leave
>   the slot as empty;
>
> - otherwise store a reference to that value in the slot.
>
> 2b. For all other locals, proceed as normal.

It's easy when you're not implementing things. I'm going to just say
"sure, go for it", and also not implement it. Have fun, whoever goes
in and tries to actually do the work...

I don't think there's any other construct in Python that can replicate
"a thing or the absence of a thing" in that way. For instance, there's
no way to copy a value from one dict into another, or delete from the
target dict if it isn't in the source (other than a long-hand). I've
no idea how it would be implemented, and don't feel like finding out.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] __dir__ in which folder is this py file

[Steven D'Aprano]

On Mon, May 07, 2018 at 11:42:00AM +, Nathaniel Smith wrote:
> On Mon, May 7, 2018, 03:45 Steven D'Aprano  wrote:

[...]
> > So yes, its very distracting.
> 
> Well, yes, you do have to know the API to use it, and if you happen to have
> learned the os.path API but not the pathlib API then of course the os.path
> API will look more familiar. I'm not sure what this is supposed to prove.

Apologies for not being more clear.

I'm arguing that for some people, your preferred syntax *is* more 
distracting and hard to comprehend than the more self-descriptive 
version with named functions. And its not just a matter of *learning* 
the API, it is a matter of using it so often that it ceases to look 
weird and looks natural.[1]

There's a school of thought that says that operator overloading is a bad 
idea, that operators should never be overridden to do something aside 
from their common meaning (e.g. + should always mean plus, / should 
always mean numeric division, etc).

>From that perspective, using / to mean something kinda-sorta like string 
concatenation, only path separator aware, is precisely the sort of thing 
that makes some people dislike operator overloading.

http://cafe.elharo.com/programming/operator-overloading-considered-harmful/

https://blog.jooq.org/2014/02/10/why-everyone-hates-operator-overloading/

I am not going to go quite that far. I think operator overloading has 
its uses. I'm not going to argue that pathlib's use of / was "bad" or a 
mistake or harmful. I called it *weird* and that's as far as I'll go. I 
use lots of weird things, and I even like some of them.

But if you think it isn't distracting, I think you are mistaken, and I 
think we ought to show caution in making it a built-in or an offical 
part of the module API.

Your earlier comment (which I redacted):

Hmm, the feedback I've heard from at least some folks teaching
intro-python-for-scientists is like, "pathlib is so great for
scripting that it justifies upgrading to python 3".

felt to me awfully close to

"pathlib! it's the future!"

I know that's not what you said, or even meant, but I felt it was 
important to remind people that not everyone knows pathlib or finds its 
API clearer than the explicitly named functions of os.path.

joinpath() may be longer than / but it is self-descriptive and easier to 
look up. help(joinpath) will tell you exactly what it does. help("/") is 
almost surely going to talk about numeric division, and it probably 
won't even mention strings or path objects at all.

I say that because we've had + for string concatenation since way back
in Python 1.5 or older, and yet as late as 3.6 help("+") still doesn't
say a thing about string, list or tuple concatenation.

As a Linux user, I'm used to paths containing slashes:

$HOMEDIR/spam/eggs

but putting quotes around the components looks unusual and is a hint
that something usual is going on (namely a shell escape). But writing
something like:

HOMEDIR / "spam" / "eggs"

doesn't even look like a path, just looks *wrong*. It looks like I'm 
escaping the wrong parts of the path: instead of escaping the spaces, 
I've escaped the parts with no spaces.

It looks wrong as a Linux path, it looks wrong as a Windows path, and it 
looks wrong as division. So, yes, it is distracting.

I'm not saying that everyone will feel the same way, or that I cannot or 
will not learn to accept / as I've learned to accept % for string 
interpolation despite it looking like percentage. But I'm saying it's 
not a slam-dunk useability win to move to pathlib.


> > First I have to work out what __filepath__ is, then I have to remember
> > the differences between all the various flavours of pathlib.Path
> > and suffer a moment or two of existential dread as I try to work out
> > whether or not *this* specific flavour is the one I need. This might not
> > matter for heavy users of pathlib, but for casual users, it's a big,
> > intimidating API with:
> >
> > - an important conceptual difference between pure paths and
> >   concrete paths;
> > - at least six classes;
> >
> 
> The docs could perhaps be more beginner friendly. For casual users, the
> answer is always "you want pathlib.Path".

That might be what I want, but it isn't what I get:

py> p = pathlib.Path('/')
py> p
PosixPath('/')

I know what a PosixPath is. But the point is, even beginners have to 
deal with the complexity of the pathlib API the moment they print a path 
object in the interactive interpreter.



[1] I've been using % for string interpolation for two decades now, and 
it still looks like a misplaced percentage sign every single time.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Allow mutable builtin types (optionally)

[Eloi Gaudry]

Hi,

I'd like to bring back this discussion (from 2005, by Greg):
https://bugs.python.org/issue1229239

Briefly, non-heap types cannot have their
attributes changed by Python code. This makes sense for python builtin
types, but not for the types defined in extension/modules.

As we have been using this patch for the very same reasons and for more
than 10 years, I think it might make sense to reconsider the discussion
that Greg started.

The main question at that time was "why not using a heap type instead
?" (because heap-type do not have this limitation).

But I think that the right question could have been "why imposing such
a restriction on non-heap types as defined in (non Python core)
extensions ?".

I mean, to my knowledge, there is no reason why a type should be
allocated on the heap (https://docs.python.org/2/c-api/typeobj.html) to
be able to change its attributes at Python level.

I'm not saying that all non-heap types should be able to do so, just
that it would make sense to allow this behavior, as an option (bit
flag).

At the implementation level, the changes needed are really limited
(about a few lines):
- Include/object.h
- Objects/typeobject.c:

Eloi
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] __dir__ in which folder is this py file

[Serhiy Storchaka]

07.05.18 17:42, Eric Snow пише:

I'm not necessarily saying we should add ModuleSpec.dirname(), but it
(or something like it) is what I'd advocate for *if* we were to add a
convenient shortcut to the directory a module is in.  FWIW, I'd
probably use it.


The question is *why* you need the absolute path to the directory the 
module is in? Taking into account the availability of 
importlib.resources etc.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] __dir__ in which folder is this py file

[Eric Snow]

On Mon, May 7, 2018 at 7:14 AM, Serhiy Storchaka  wrote:
> * Additional burden on maintainers of import machinery. It is already too
> complex, and __file__ is set in multiple places. Don't forgot about
> third-party implementations.
>
> See also issue33277: "Deprecate __loader__, __package__, __file__, and
> __cached__ on modules" (https://bugs.python.org/issue33277).

Thanks for mentioning all this, Serhiy. :)

That said, it *may* be worth considering a method on ModuleSpec (aka
"dir()['__spec__']").  One (rough) possibility:

def dirname(self):
"""Return the absolute path to the directory the module is in.

This will return None for modules that do not have __file__ or
where "directory" does not make sense (e.g. extension modules).
"""
if self.origin is None:  # XXX ...or self.origin isn't a filename.
return None
import os.path  # This "lazy" import is necessary in this case.
filename = os.path.abspath(self.origin)
return os.path.dirname(filename)

Putting this on the module spec has several advantages:

1. __spec__ is a single source of truth (though tied to how a module
was "found" rather than to anything that happened when "loaded")
2. encourages folks to rely on __spec__ (where we'd like to head, as
demonstrated by the issue Serhiy referenced above)
3. does not add any overhead to import performance (i.e. cost only
incurred when needed)
4. does not add complexity to any other part of the import machinery

I'm not necessarily saying we should add ModuleSpec.dirname(), but it
(or something like it) is what I'd advocate for *if* we were to add a
convenient shortcut to the directory a module is in.  FWIW, I'd
probably use it.

-eric
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Steven D'Aprano]

On Mon, May 07, 2018 at 10:38:51PM +1000, Chris Angelico wrote:
> On Mon, May 7, 2018 at 9:42 PM, Steven D'Aprano  wrote:
[...]
> > (3) A compromise: binding assignments are scoped local to the
> > comprehension, but they are initialised from their surrounding scope.
[...]
> Does it HAVE to be initialised from the surrounding scope? What if the
> surrounding scope doesn't have that variable?

No. Then it's just an uninitialised local, and it is a NameError to try 
to evaluate it before something gets bound to it.


> stuff = [spam for x in items if (spam := f(x)) < 0]
> 
> Is this going to NameError if you haven't defined spam?

It shouldn't be an error, because by the time the comprehension looks up 
the value of "spam", it has been bound by the binding-expression.


> Or is the
> compiler to somehow figure out whether or not to pull in a value?
> Never mind about implementation - what are the semantics?

Variable "spam" is not defined in any surrounding scope, so these ought 
to all be NameError (or UnboundLocalError):

[spam for a in it]  # obviously
[(spam := spam + a) for a in it]
[spam if True else (spam := a) for a in it]
[spam for a in it if True or (spam := a)]

They are errors because the name "spam" is unbound when you do a lookup. 
This is not an error, because the name is never looked up:

[True or spam for a in it if True or (spam := a)]

Although "spam" never gets bound, neither does it get looked up, so no 
error. The next one is also okay, because "spam" gets bound before the 
first lookup:

[(spam := spam+1) for a in it if (spam := a*2) > 0]


Here's a sketch of how I think locals are currently handled:

1. When a function is compiled, the compiler does a pass over the
   source and determines which locals are needed.

2. The compiler builds an array of slots, one for each local, 
   and sets the initial value of the slot to "empty" (undefined).

3. When the function is called, if it tries reading from a local's
   slot which is still empty, it raises UnboundLocalError.

(am I close?)

Here's the change I would suggest:

2. The compiler builds an array of slots, one for each local:

2a. For locals that are the target of binding-expression only:

- look up the target in the current scope (that is, not the
  comprehension's scope, but the scope that the comprehension
  is inside) using the normal lookup rules, as if you were
  compiling "lambda x=x: None" and needed the value of x;

- if the target is undefined, then swallow the error and leave
  the slot as empty;

- otherwise store a reference to that value in the slot.

2b. For all other locals, proceed as normal.




> >> Part of it is just that people seem to be fighting for the sake of
> >> fighting.
> >
> > Them's fightin' words! *wink*
> >
> > Honestly Chris, I know this must be frustrating, but I'm not fighting
> > for the sake of it, and I doubt Tim is either. I'm arguing because there
> > are real use-cases which remain unmet if binding-variables inside
> > comprehensions are confined to the comprehension.
> 
> I don't think you are and I don't think Tim is. But do you honestly
> want to say that about EVERY person in these threads?

I'm going to assume good faith, no matter the evidence :-)


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Serhiy Storchaka]

07.05.18 12:58, Eloi Gaudry пише:
I didn't mean to replace the current (debug) assert but I wanted to add 
another one that would allow to be switch on or off on production builds.


The need for a new keyword (not syntax) comes from this difference.


  I cannot think of another example that would convince you of the 
benefit of having a specific keyword for such a runtime assert.  I do 
believe that having such a feature in non-debug build is more than 
interesting but indeed needed.


I just don't understand why you need a new keyword for writing runtime 
checks.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

On Sun, 2018-05-06 at 01:30 +1000, Steven D'Aprano wrote:
> On Sat, May 05, 2018 at 08:04:45AM +, Eloi Gaudry wrote:
> 
> > Hi folks,
> > I intend to add a runtime assertion feature in python.
> 
> I'm very interested in this idea, but I'm afraid your draft PEP
> isn't 
> very clear about this feature. Comments below.
Thanks, I'll try to clarify this here.

> > Rationale
> > There is no runtime assert relying on python grammar available.
> > For 
> > diagnostics and/or debugging reasons, it would be valuable to add
> > such 
> > a feature.
> 
> Why not this?
> 
> if DEBUG_ENABLED:
> run_check(expression)
> 
> where run_check is an ordinary function. If DEBUG_ENABLED is false,
> the 
> only runtime overhead is the name lookup and bool test, the
> expression 
> is not evaluated.
for the sake of clarity across any Python application/extension for
instance, and because having run_check defined as a statement seems
actually the good thing to do.
> 
> What features does your suggestion offer over this simple technique?
Code readiness I think (kiss) : simpler and shorter.
The simple technique would still be used for other purpose (assert that
are context based for instance).


> 
> [...]
> > A brief example why avoiding evaluating the expression is needed
> > to 
> > avoid any overhead in case the runtime assert should be ignored.
> > ::
> > runtime_assert( 999 in { i:None for i in range( 1000 ) } )
> 
> You say that you have been using this feature in production for two 
> years. You should give real-life examples, not fake examples like
> the 
> above. That example would be better written as:
> 
> runtime_assert(True)
> 
> since no part of the test actually depends on either the enormous
> dict 
> or range objects that you build only to throw away.
Real life examples would be more application-based I think.

In my case, I used this for switching off some heaving computing
expression evaluation at runtime. Those expression would have been
responsible for sorting vectors and communicating them through MPI
using some method from our framework, in order to finally return a
boolean.


> > Usage
> > ::
> > 
> > runtime_assert( expr )
> > 
> > #would result in 
> > if expr and runtime_assert_active:
> > print RuntimeAssertionError()
> 
> Surely the test should be the other way around?
> 
> if runtime_assert_active and expr:
> print(RuntimeAssertionError())
otherwise the expression is evaluated regardless of whether the
runtime 
> assertions are active or not.
You are absolutely right :)

> Please use Python 3 syntax for the PEP, as Python 2 is in
> maintenance 
> mode and will absolutely not get any new language features.
Sure

> 
> Some more observations:
> 
> I like the idea of more control over runtime checking, but your PEP 
> should justify why this is a good idea and why "just write more unit 
> tests" isn't the solution.
My main reasons is the one I gave in the context description of the
PEP. Assert remains an efficient way for controlling your application
behavior when running in production (as opposed to a debug build).

Writing more unit tests is always a good practice, but :
- it does not necessarily cover all possible paths that are runtime
dependent (an application relies on various app-parameters and
architecture-parameters for instance)
- it does not offer a solution for your extension/application consumers
(that might be using your application in a very specific set of
paths/set of decision/etc.). 

> What happens if the caller has defined their own function or
> variable 
> called runtime_assert?
making runtime_assert a statement (as assert is already) would forbid
such a definition, thus it would break backward compatibility (in those
specific cases) for sure.


> Can the caller customise the assertion error message?
At this stage, no, but I don't know if this would be something
difficult to add.

> If you are changing the grammar, why not make runtime_assert a
> statement?
I'm not really changing the grammar, just making 'runtime_assert' a
statement as 'assert' already is (and IMHO could have been named as
debug_assert). There is no significant difference between both assert
in term of implementation.

> Can the caller set an application-wide global across all modules, or
> do 
> they have to set
> 
> runtime_assert_active = True
> 
> in every module they want to use runtime_assert?
I would vote in favor of having two different ways to activate or not
the runtime assertion evaluation:
- the first one would consist of controlling a global variable at the
C-level (the current recipe in cPython) and would be used by python
extension. This is the solution I am using personnaly.
- the second would use a set method available in pure python (not
implemented, to be done at C-level too I think) to easily (de)activate
all the  runtime asserts.
___
Python-ideas mailing list
Python-ideas@python.org

Re: [Python-ideas] __dir__ in which folder is this py file

[Serhiy Storchaka]

06.05.18 09:53, Yuval Greenfield пише:

I often need to reference a script's current directory. I end up writing:

import os
SRC_DIR = os.path.dirname(__file__)


But I would prefer to have a new dunder for that. I propose: "__dir__". 
I was wondering if others would find it convenient to include such a 
shortcut.


Here are some examples of dirname(__file__) in prominent projects.

https://github.com/tensorflow/models/search?l=Python=dirname=
https://github.com/django/django/search?l=Python=dirname=
https://github.com/nose-devs/nose/search?l=Python=dirname=

Reasons not to add __dir__:
* There already is one way to do it and it's clear and fairly short..
* Avoid the bikeshed discussion of __dir__, __folder__, and other 
candidates.


* Additional burden on maintainers of import machinery. It is already 
too complex, and __file__ is set in multiple places. Don't forgot about 
third-party implementations.


See also issue33277: "Deprecate __loader__, __package__, __file__, and 
__cached__ on modules" (https://bugs.python.org/issue33277).


* More complex user code, because you have to handle different cases:

  - __file__ is set, but __dir__ is not set.
  - __file__ and __dir__ are set, but are not consistent.


Reasons to add it:


Are you aware of importlib.resources?

https://docs.python.org/3.7/whatsnew/3.7.html#importlib-resources

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] __dir__ in which folder is this py file

[Chris Angelico]

On Mon, May 7, 2018 at 8:44 PM, Steven D'Aprano  wrote:
> On Sun, May 06, 2018 at 09:33:03PM -0700, Nathaniel Smith wrote:
>
>> How is
>>
>> data_path = __filepath__.parent / "foo.txt"
>>
>> more distracting than
>>
>> data_path = joinpath(dirname(__file__), "foo.txt")
>
>
> Why are you dividing by a string? That's weird.
>
> [looks up the pathlib docs]
>
> Oh, that's why. It's still weird.
>
> So yes, its very distracting.

Isn't it strange how we can divide a path by a string, and that works,
and we can take the remainder after you divide a string by a string,
and that works as long as there's exactly one "%s" in the string, but
nobody's interested in having "foo bar spam ham"/" " ==>
["foo","bar","spam","ham"] ?

Just sayin', it ain't all that strange.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Chris Angelico]

On Mon, May 7, 2018 at 9:42 PM, Steven D'Aprano  wrote:
> On Mon, May 07, 2018 at 12:48:53PM +1000, Chris Angelico wrote:
>> On Mon, May 7, 2018 at 12:34 PM, Tim Peters  wrote:
>
>> > There's a difference, though:  if `y` "leaks", BFD.  Who cares? ;-)
>> > If `y` remains inaccessible, there's no way around that.
>>
>> That's Steve D'Aprano's view - why not just let them ALL leak? I don't
>> like it though.
>
> I know popular opinion is against me, and backward compatibility and all
> that, but I wish that generator expressions and comprehensions ran
> in their surrounding scope, like regular for statements.
>
> (Yes, I know that makes generator expressions tricky to implement. As
> the guy who doesn't have to implement it, I don't have to care :-)

Yeah, it's really easy when you don't have to worry about how on earth
you can implement the concept of "unexecuted block of code that can be
executed later even after the surrounding context has returned, but
which isn't a function". :)

> (3) A compromise: binding assignments are scoped local to the
> comprehension, but they are initialised from their surrounding scope.
>
> This would be similar to the way Lua works, as well as function
> parameter defaults.
>
> I have some vague ideas about implementation, but there's no point
> discussing that unless people actually are interested in this option.
>
> This will *half* satisfy the running-total example:
>
> total = 0
> running_totals = [(total := total + x) for x in [98, 99]]
> assert total == 0
>
>
> Guaranteed to generate at least two Stackoverflow posts a month
> complaining about it, but better than nothing :-)

Does it HAVE to be initialised from the surrounding scope? What if the
surrounding scope doesn't have that variable?

stuff = [spam for x in items if (spam := f(x)) < 0]

Is this going to NameError if you haven't defined spam? Or is the
compiler to somehow figure out whether or not to pull in a value?
Never mind about implementation - what are the semantics?

>> Having
>> the iteration variable NOT leak means it's a self-contained unit that
>> simply says "that thing we're iterating over".
>
> Assuming there are no side-effects to any of the operations inside the
> comprehension.

Which is often the case.

>> Part of it is just that people seem to be fighting for the sake of
>> fighting.
>
> Them's fightin' words! *wink*
>
> Honestly Chris, I know this must be frustrating, but I'm not fighting
> for the sake of it, and I doubt Tim is either. I'm arguing because there
> are real use-cases which remain unmet if binding-variables inside
> comprehensions are confined to the comprehension.

I don't think you are and I don't think Tim is. But do you honestly
want to say that about EVERY person in these threads?

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] __dir__ in which folder is this py file

[Nick Coghlan]

On 7 May 2018 at 21:42, Nathaniel Smith  wrote:

> On Mon, May 7, 2018, 03:45 Steven D'Aprano  wrote:
>
>> On Sun, May 06, 2018 at 09:33:03PM -0700, Nathaniel Smith wrote:
>>
>> > How is
>> >
>> > data_path = __filepath__.parent / "foo.txt"
>> >
>> > more distracting than
>> >
>> > data_path = joinpath(dirname(__file__), "foo.txt")
>>
>>
>> Why are you dividing by a string? That's weird.
>>
>> [looks up the pathlib docs]
>>
>> Oh, that's why. It's still weird.
>>
>> So yes, its very distracting.
>>
>
> Well, yes, you do have to know the API to use it, and if you happen to
> have learned the os.path API but not the pathlib API then of course the
> os.path API will look more familiar. I'm not sure what this is supposed to
> prove.
>

I think it strongly suggests that *magically* introducing a path object
into a module's namespace would be a bad idea, since it harms readability
(since merely having `path` in the name isn't a strong enough hint that the
object in question is a `pathlib.Path` instance).

Your original point is still valid though: given the boilerplate reduction
already available via "from pathlib import Path; _this_dir =
Path(__file__).parent", it's the pathlib version that needs to be taken as
the baseline for how verbose the status quo really is, not the lower level
os.path API (no matter how accustomed some of us may still be to using the
latter).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] __dir__ in which folder is this py file

[Nick Coghlan]

On 7 May 2018 at 20:44, Steven D'Aprano  wrote:

> First I have to work out what __filepath__ is, then I have to remember
> the differences between all the various flavours of pathlib.Path
> and suffer a moment or two of existential dread as I try to work out
> whether or not *this* specific flavour is the one I need. This might not
> matter for heavy users of pathlib, but for casual users, it's a big,
> intimidating API with:
>
> - an important conceptual difference between pure paths and
>   concrete paths;
> - at least six classes;
> - about 50 or so methods and properties
>

Right, but that's why I think this may primarily be a docs issue, as for
simple use cases, only one pathlib class matters, and that's "pathlib.Path"
(which is the appropriate concrete path type for the running platform),
together with its alternate constructors "Path.cwd()" and "Path.home()".

So if you spell out the OP's original example with pathlib instead of
os.path, you get:

from pathlib import Path
SRC_DIR = Path(__file__).parent


And then SRC_DIR is a rich path object that will mostly let you avoid
importing any of:

- os
- os.path
- stat
- glob
- fnmatch


> As far as performance goes, I don't think it matters that we could
> technically make pathlib imported lazily. Many people put all their
> pathname manipulations at the beginning of their script, so lazy or not,
> the pathlib module is going to be loaded *just after* startup, .
>

It's the fnmatch and re module imports *inside* pathlib that may be worth
making lazy, as those currently account for a reasonable chunk of the
import time but are only used to implement PurePath.match and
_WildcardSelector. That means making them lazy may allow folks to avoid
those imports if they don't use any of the wildcard matching features.


> For many scripts, this isn't going to matter, but for those who want to
> avoid the overhead of pathlib, making it lazy doesn't help. That just
> delays the overhead, it doesn't remove it.
>

That's fine - it's not uncommon for folks looking to minimise startup
overhead to have to opt in to using a lower level API for exactly that
reason.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Steven D'Aprano]

On Mon, May 07, 2018 at 12:48:53PM +1000, Chris Angelico wrote:
> On Mon, May 7, 2018 at 12:34 PM, Tim Peters  wrote:

> > There's a difference, though:  if `y` "leaks", BFD.  Who cares? ;-)
> > If `y` remains inaccessible, there's no way around that.
> 
> That's Steve D'Aprano's view - why not just let them ALL leak? I don't
> like it though.

I know popular opinion is against me, and backward compatibility and all 
that, but I wish that generator expressions and comprehensions ran
in their surrounding scope, like regular for statements.

(Yes, I know that makes generator expressions tricky to implement. As 
the guy who doesn't have to implement it, I don't have to care :-)

Calling it a "leak" assumes that it is a bad thing. I don't think it is 
a bad thing. It's not often that I want to check the value of a 
comprehension loop, but when I do, I have to tear the comprehension 
apart into a for-loop. Even if it is only temporarily, for debugging, 
then put the comprehension back together.

The only time I can see it is a bad thing is if I blindly copy and paste 
a comprehension out of one piece of code and dump it into another piece 
of code without checking to see that it slots in nicely without blowing 
away existing variables.

But if you're in the habit of blindly and carelessly pasting into your 
code base without looking it over, this is probably the least of your 
worries... *wink*

But what's done is done, and while there are plenty of windmills I am 
willing to tilt at, reversing the comprehensions scope decision is not 
one of them.


[...]
> Sorry, I meant "local to the comprehension's scope". We can't know the
> user's intention. We have to create semantics before the user's
> intention even exists.

Surely that's backwards? We ought to find out what people want before 
telling them that they can't have it :-)


> > But the point above remains:  if they don't leak, contexts that want
> > them to leak have no recourse.  If they do leak, then the other uses
> > would still work fine, but they'd possibly be annoyed by a leak they
> > didn't want.

Indeed.


> Then let's revert the Py3 change that put comprehensions into
> functions, and put them back to the vanilla transformation:

You know we can't do that. But we do have a choice with binding 
expressions. *Either way*, whatever we do, we're going to upset 
somebody, so we simply have to decide who that will be.

Actually we have at least three choices:


(1) Consistency Über Alles (whether foolish or not)

Now that comprehensions are their own scope, be consistent about it. 
Binding expressions inside the comprehension will be contained to the 
comprehension. I'll hate it, but at least it is consistent and easy to 
remember: the entities which create a new scope are modules, classes, 
functions, plus comprehensions.

That's going to cut out at least one motivating example though. See 
below.


(2) Binding assignments are *defined* as "leaking", or as I prefer, 
defined as existing in the lexical scope that contains the 
comprehension. Hence:

# module level
[(x := a) for a in [98, 99]]
assert x == 99

# class level
class X:
[(x := a) for a in [98, 99]]

assert X.x == 99

# function level
def func():
[(x := a) for a in [98, 99]]
assert x == 99

Note that in *all* of these cases, the variable a does not "leak".

This will helpfully support the "running total" use-case that began this 
whole saga:

total = 0
running_totals = [(total := total + x) for x in [98, 99]]
assert total == 197

(I have to say, given that this was THE motivating use-case that began 
this discussion, I think it is ironic and not a little sad that the PEP 
has evolved in a direction that leaves this use-case unsatisfied.)


(3) A compromise: binding assignments are scoped local to the 
comprehension, but they are initialised from their surrounding scope.

This would be similar to the way Lua works, as well as function 
parameter defaults.

I have some vague ideas about implementation, but there's no point 
discussing that unless people actually are interested in this option.

This will *half* satisfy the running-total example:

total = 0
running_totals = [(total := total + x) for x in [98, 99]]
assert total == 0


Guaranteed to generate at least two Stackoverflow posts a month 
complaining about it, but better than nothing :-)



> Having
> the iteration variable NOT leak means it's a self-contained unit that
> simply says "that thing we're iterating over".

Assuming there are no side-effects to any of the operations inside the 
comprehension.



> Part of it is just that people seem to be fighting for the sake of
> fighting.

Them's fightin' words! *wink*

Honestly Chris, I know this must be frustrating, but I'm not fighting 
for the sake of it, and I doubt Tim is either. I'm arguing because there 
are real use-cases which remain unmet if binding-variables inside 
comprehensions are confined to the 

Re: [Python-ideas] __dir__ in which folder is this py file

[Nathaniel Smith]

On Mon, May 7, 2018, 03:45 Steven D'Aprano  wrote:

> On Sun, May 06, 2018 at 09:33:03PM -0700, Nathaniel Smith wrote:
>
> > How is
> >
> > data_path = __filepath__.parent / "foo.txt"
> >
> > more distracting than
> >
> > data_path = joinpath(dirname(__file__), "foo.txt")
>
>
> Why are you dividing by a string? That's weird.
>
> [looks up the pathlib docs]
>
> Oh, that's why. It's still weird.
>
> So yes, its very distracting.
>

Well, yes, you do have to know the API to use it, and if you happen to have
learned the os.path API but not the pathlib API then of course the os.path
API will look more familiar. I'm not sure what this is supposed to prove.


> First I have to work out what __filepath__ is, then I have to remember
> the differences between all the various flavours of pathlib.Path
> and suffer a moment or two of existential dread as I try to work out
> whether or not *this* specific flavour is the one I need. This might not
> matter for heavy users of pathlib, but for casual users, it's a big,
> intimidating API with:
>
> - an important conceptual difference between pure paths and
>   concrete paths;
> - at least six classes;
>

The docs could perhaps be more beginner friendly. For casual users, the
answer is always "you want pathlib.Path".

- about 50 or so methods and properties
>

Yeah, filesystems have lots of operations. That's why before pathlib users
had to learn about os and os.path and shutil and glob and maybe some more
I'm forgetting.


> As far as performance goes, I don't think it matters that we could
> technically make pathlib imported lazily. Many people put all their
> pathname manipulations at the beginning of their script, so lazy or not,
> the pathlib module is going to be loaded *just after* startup, .
>
> For many scripts, this isn't going to matter, but for those who want to
> avoid the overhead of pathlib, making it lazy doesn't help. That just
> delays the overhead, it doesn't remove it.
>

AFAIK were two situations where laziness has been mentioned in this thread:

- my suggestion that we delay loading pathlib until someone accesses
__filepath__. I don't actually know how to implement this so it was mostly
intended to try to spur new ideas, but if we could do it, the point of the
laziness would be so that scripts that didn't use __filepath__ wouldn't pay
for it.

- Nick's observation that pathlib could load faster if it loaded fnmatch
lazily. Since this is only used for a few methods, this would benefit any
script that didn't use those methods. (And for scripts that do need
fnmatch's functionality, without pathlib they'd just be importing it
directly, so pathlib importing it isn't really an extra cost.)

It's true that laziness isn't a silver bullet, though, yeah. We should also
look for ways to speed things up.

-n
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] __dir__ in which folder is this py file

[Steven D'Aprano]

On Sun, May 06, 2018 at 09:33:03PM -0700, Nathaniel Smith wrote:

> How is
> 
> data_path = __filepath__.parent / "foo.txt"
> 
> more distracting than
> 
> data_path = joinpath(dirname(__file__), "foo.txt")


Why are you dividing by a string? That's weird.

[looks up the pathlib docs]

Oh, that's why. It's still weird.

So yes, its very distracting.

First I have to work out what __filepath__ is, then I have to remember 
the differences between all the various flavours of pathlib.Path 
and suffer a moment or two of existential dread as I try to work out 
whether or not *this* specific flavour is the one I need. This might not 
matter for heavy users of pathlib, but for casual users, it's a big, 
intimidating API with:

- an important conceptual difference between pure paths and
  concrete paths;
- at least six classes;
- about 50 or so methods and properties

As far as performance goes, I don't think it matters that we could 
technically make pathlib imported lazily. Many people put all their 
pathname manipulations at the beginning of their script, so lazy or not, 
the pathlib module is going to be loaded *just after* startup, .

For many scripts, this isn't going to matter, but for those who want to 
avoid the overhead of pathlib, making it lazy doesn't help. That just 
delays the overhead, it doesn't remove it.




-- 
Steve

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Runtime assertion with no overhead when not active

[Eloi Gaudry]

I didn't mean to replace the current (debug) assert but I wanted to add another 
one that would allow to be switch on or off on production builds.

The need for a new keyword (not syntax) comes from this difference.


 I cannot think of another example that would convince you of the benefit of 
having a specific keyword for such a runtime assert.  I do believe that having 
such a feature in non-debug build is more than interesting but indeed needed.


At some point, there should be a trade-off between to relying on a simple 
keyword/mechanism (for a runtime assert) versus the development cost of adding 
such a feature and maintaining it. Anyway, thanks for your feedback Serhiy, it 
helps.





From: Python-ideas  on 
behalf of Serhiy Storchaka 
Sent: Saturday, May 5, 2018 17:44
To: python-ideas@python.org
Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active

05.05.18 18:04, Eloi Gaudry пише:
> By 'self-contained', I meant that using the assert keyword and its
> expression is sufficient. An inline assertive expression as the one you
> describe does not fulfill this assert requirement.

Sufficient for what? And why writing with using the existing syntax is
not sufficient?

> My proposal is simply to extend the current assert to non-debug builds
> and allow to switch it off/on at runtime.

The main property of the assert statement is that has a zero overhead in
non-debug run. If you remove this property, it will be not the assert
statement, and you will not need a special syntax support for writing
this runtime check.

> The syntax example I gave illustrated what I meant by syntax aware.

It doesn't illustrate why a new syntax is necessary. Or I can't
understand this illustration.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] A comprehension scope issue in PEP 572

[Jacco van Dorp]

For what it's worth, i'm totally +1 on inline uses of global and nonlocal.

As a related improvement, i'd also like it if "global x = 5" would be
a legal statement. As a noob learning python, I was suprised to find
out I couldn't and had to split it on two lines.(aside from a 9-hour
course of C and some labview (which I totally hate), python was my
first language and still the one im by far most proficient with.)

2018-05-07 6:04 GMT+02:00 Tim Peters :
> [Nick Coghlan ]
>> The issue is that because name binding expressions are just ordinary
>> expressions, they can't be defined as "in comprehension scope they do X, in
>> other scopes they do Y" - they have to have consistent scoping semantics
>> regardless of where they appear.
>
> While I'm not generally a fan of arguments, I have to concede that's a
> really good argument :-)
>
> Of course their definition _could_ be context-dependent, but even I'll
> agree they shouldn't be.  Never mind!
>
>
>> However, it occurs to me that a nonlocal declaration clause could be allowed
>> in comprehension syntax, regardless of how any nested name bindings are
>> spelt:
>>
>> p = rem = None
>> while any((rem := n % p) for p in small_primes nonlocal (p, rem)):
>> # p and rem were declared as nonlocal in the nested scope, so our
>> rem and p point to the last bound value
>>
>> I don't really like that though, since it doesn't read as nicely as being
>> able to put the nonlocal declaration inline.
>
> If the idea gets traction, I'm sure we'll see 100 other syntax ideas
> by the time I wake up again.
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/