Re: [Python-ideas] pathlib.Path should handle Pythonpath or package root

[Nick Coghlan]

On 19 July 2017 at 06:40, George Fischhof  wrote:
> I think yes ;-)
> I would like to use (or I think it would be good to use) something like
> pathlib.Path(package_root)
> so I could use
>
> importlib.import_module(pathlib.Path(package_root) / plugins / plugin_name)

No, as that's fundamentally incompatible with how Python's import
system works - the filesystem is *a* way of representing package
namespacing, but far from the only way. Managing the import state also
has nothing whatsoever to do with pathlib.

That said, the idea of better encapsulating the import state so we can
more readily have constrained "import engines" *is* a reasonable one,
it just runs into significant practical problems related to the
handling of transitive imports in both Python modules and (especially)
extension modules.

The last serious attempt at pursuing something like that is documented
in PEP 406, "Improved Encapsulation of Import State":
https://www.python.org/dev/peps/pep-0406/

Unfortunately, the main outcome of Greg Slodkowicz's GSoC work on the
idea was to conclude that that particular approach wasn't viable due
to the fact that import system plugins at that time were pretty much
required to directly manipulate global state, which ran directly
counter to the goal of encapsulation.

However, we also haven't had anyone seriously revisit the idea since
the updated import plugin API was defined in PEP 451 - that
deliberately moved a lot of the global state management out of the
plugins and into the import system, so it should be more amenable to
an "import engine" style approach to state encapsulation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] between block and function [was: Custom Code Folding: Standardized Rules and Syntax?]

[Jim J. Jewett]

There have been times when I wanted to group portions of a module,
class, or function.  Giving names to each of those groupings would be
useful, and would be appropriate for python-level changes.

That said, it would probably need more proof in the wild first, so the
first step would be getting support in an
editor (such as PyCharm), and the second would be
an informational PEP on how to standardize the
practice, like PEP 257 does for docstrings.
https://www.python.org/dev/peps/pep-0257/

For these steps, you would probably have to use a comment or string
convention, such as #{{{ or ":(group X ", though I suppose an external
file (similar to the stub files for typing) is also possible.

If these take off enough that the line noise gets annoying, that will
prove the need, and getting support from python itself will be a lot
easier.

Example code

:group C
class C1:
:group emulateFoo
...

:group manageBar
def xx(self, bar, name, val=None):
:group setupBar
...
:group actualXX

class C2:
...

:group D
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Alternative Unicode implementations (NSString/NSMutableString)

[Victor Stinner]

Supporting a new kind of string storage would require a lot of efforts.
There are a lot of C code specialized for each Unicode kind

Victor

Le 19 juil. 2017 12:43 AM, "Jim J. Jewett"  a écrit :

> Ronald Oussoren came up with a concrete use case for wanting the
> interpreter to consider something a string, even if it isn't
> implemented with the default datastructure.
>
> In https://mail.python.org/pipermail/python-ideas/2017-July/046407.html
> he writes:
>
>The reason I need to subclass str: in PyObjC I use
>a subclass of str to represent Objective-C strings
>(NSString/NSMutableString), and I need to keep track
>of the original value; mostly because there are some
>Objective-C APIs that use object identity. The worst
>part is that fully initialising the PyUnicodeObject fields
>often isn’t necessary as a lot of Objective-C strings
>aren’t used as strings in Python code.
>
> The PyUnicodeObject (via its leading PyASCIIObject member) currently
> uses 7 flag bits including 2 for kind.  Would it be worth adding an
> 8th big to indicate that string is a virtual subclass, and that the
> internals should not be touched directly?  (This would require
> changing some of the macros; at the time of PEP 393 it Martin ruled
> YAGNI ... but is this something that might reasonably be reconsidered,
> if someone did the work.  Which I am considering, but not committing
> to.)
>
> -jJ
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] a new namedtuple

[Guido van Rossum]

On Tue, Jul 18, 2017 at 3:18 PM, Greg Ewing 
wrote:

> Ethan Furman wrote:
>
>> I certainly don't expect the signature to change, but why is using a
>> metaclass out?  The use (or not) of a metaclass /is/ an implementation
>> detail.
>>
>
> For me, the main benefit of using a metaclass would be that
> it enables using normal class declaration syntax to define a
> namedtuple. That's not just an implementation detail!


Newer versions of the typing module do this:
https://docs.python.org/3/library/typing.html#typing.NamedTuple (and indeed
it's done with a metaclass).

-- 
--Guido van Rossum (python.org/~guido )
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] namedtuple with ordereddict

[Greg Ewing]

Jim J. Jewett wrote:

is there any reason not to simply define it as a view on a dict, or at
least as a limited proxy to one?


Some valuable characteristics of namedtuples as they are now:

* Instances are very lightweight
* Access by index is fast
* Can be used as a dict key

All of those would be lost if namedtuple became a dict view.

--
Greg
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Enabling Event.set to notify all waiters with an exception

[Nathaniel Smith]

On Tue, Jul 18, 2017 at 2:33 PM, Pau Freixes  wrote:
> Morning guys,

(Not everyone here is a guy.)

> I came across with that idea trying to resolve a typical dogpile
> pattern [1], having many DNS calls to the same domain because of a
> miss in a DNS cache.
>
> The usage of the the set either to notify that the waiters could be
> awake and get the result from the cache or use it to notify that
> something was wrong helped me to reduce the complexity of the code.
> Just as an example:
>
>  ```
> if key in throttle_dns_events:
> yield from throttle_dns_events[key].wait()
> else:
> throttle_dns_events[key] = Event(loop=loop)
> try:
> addrs = yield from \
> resolver.resolve(host, port, family=family)
> cached_hosts.add(key, addrs)
> throttle_dns_events[key].set()
> except Exception as e:
> # any DNS exception, independently of the implementation
> # is set for the waiters to raise the same exception.
> throttle_dns_events[key].set(exc=e)
> raise
> finally:
> throttle_dns_events.pop(key)
> ```
>
> Any error caught by the locker will be broadcasted to the waiters. For
> example, a invalid hostname.
>
> I tried to open a PR to the CPython implementation, and they claim
> that the current interface of all of the locks objects behind the
> asyncio.locks [2] module try to keep the same interface as the
> threading one [3]. Therefore, to modify the asyncio implementation
> would need first a change in the threading interface.
>
> I was determined to justify that change, but after a bit research, I
> didn't find any example in other languages such as Java [4], C# [5] or
> C++ [6]  allowing you to send an exception as a signal value to wake
> up the sleeping threads.

'Event' is designed as a lowish-level primitive: the idea is that it
purely provides the operation of "waiting for something", and then you
can compose it with other data structures to build whatever
higher-level semantics you need. From this point of view, it doesn't
make much sense to add features like exception throwing -- that would
make it more useful for some particular cases, but add overhead that
others don't want or need.

In this case, don't you want to cache an error return as well, anyway?

It sounds like you're reinventing the idea of a Future, which is
intended as a multi-reader eventually-arriving value-or-error --
exactly what you want here. So it seems like you could just write:

# Conceptually correct, but subtly broken due to asyncio quirks
if key not in cache:
cache[key] = asyncio.ensure_future(resolver.resolve(...))
return await cache[key]

BUT, unfortunately, this turns out to be really broken when combined
with asyncio's cancellation feature, so you shouldn't do this :-(.
When using asyncio, you basically need to make sure to never await any
given Future more than once.

Maybe adding a shield() inside the await is the right solution? The
downside is that you actually do want to propagate the cancellation
into the resolution Task, just... only if *all* the callers are
cancelled *and* only if you can make sure that the cancellation is not
cached. It's quite tricky actually!

But I don't think adding exception-throwing functionality to Event()
is the right solution :-)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] a new namedtuple

[Ethan Furman]

On 07/18/2017 09:09 AM, Guido van Rossum wrote:

On Tue, Jul 18, 2017 at 8:56 AM, Ethan Furman wrote:



I certainly don't expect the signature to change, but why is using a metaclass 
out?  The use (or not) of a metaclass
/is/ an implementation detail.


It is until you try to subclass with another metaclass -- then you have a 
metaclass conflict. If the namedtuple had no
metaclass this would not be a conflict. (This is one reason to love class 
decorators.)


Ah, so metaclasses are leaky implementation details.  Makes sense.

--
~Ethan~

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Enabling Event.set to notify all waiters with an exception

[Pau Freixes]

Morning guys,

I came across with that idea trying to resolve a typical dogpile
pattern [1], having many DNS calls to the same domain because of a
miss in a DNS cache.

The usage of the the set either to notify that the waiters could be
awake and get the result from the cache or use it to notify that
something was wrong helped me to reduce the complexity of the code.
Just as an example:

 ```
if key in throttle_dns_events:
yield from throttle_dns_events[key].wait()
else:
throttle_dns_events[key] = Event(loop=loop)
try:
addrs = yield from \
resolver.resolve(host, port, family=family)
cached_hosts.add(key, addrs)
throttle_dns_events[key].set()
except Exception as e:
# any DNS exception, independently of the implementation
# is set for the waiters to raise the same exception.
throttle_dns_events[key].set(exc=e)
raise
finally:
throttle_dns_events.pop(key)
```

Any error caught by the locker will be broadcasted to the waiters. For
example, a invalid hostname.

I tried to open a PR to the CPython implementation, and they claim
that the current interface of all of the locks objects behind the
asyncio.locks [2] module try to keep the same interface as the
threading one [3]. Therefore, to modify the asyncio implementation
would need first a change in the threading interface.

I was determined to justify that change, but after a bit research, I
didn't find any example in other languages such as Java [4], C# [5] or
C++ [6]  allowing you to send an exception as a signal value to wake
up the sleeping threads.

is that enough to give up? I'm still reticent, I believe that this
simple change in the interface can help reducing the complexity to
handle errors in some scenarios.

I would like to gather more ideas, thoughts, and comments from you
about how can I still justify this change ...

Thanks,

[1] 
https://github.com/pfreixes/aiohttp/blob/throttle_dns_requests/aiohttp/connector.py#L678
[2] https://docs.python.org/3/library/asyncio-sync.html
[3] https://docs.python.org/3/library/threading.html
[4] https://docs.oracle.com/javase/7/docs/api/java/lang/Object.html#notifyAll()
[5] 
https://msdn.microsoft.com/en-us/library/system.threading.monitor.pulseall.aspx
[6] http://en.cppreference.com/w/cpp/thread/condition_variable/notify_all

-- 
--pau
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] a new namedtuple

[Joao S. O. Bueno]

In the other thread, I had mentioned my "extradict" implementation - it
does have quite a few differences as it did not try to match namedtuple
API, but it works nicely for all common use cases - these are the timeit
timings:

(env) [gwidion@caylus ]$ python3 -m timeit --setup "from collections import
namedtuple" "K = namedtuple('K', 'a b c')"
1000 loops, best of 3: 362 usec per loop

(env) [gwidion@caylus ]$ python3 -m timeit --setup "from extradict import
namedtuple" "K = namedtuple('K', 'a b c')"
1 loops, best of 3: 20 usec per loop

(env) [gwidion@caylus ]$ python3 -m timeit --setup "from extradict import
fastnamedtuple as namedtuple" "K = namedtuple('K', 'a b
c')"
1 loops, best of 3: 21 usec per loop


Source at:
https://github.com/jsbueno/extradict/blob/master/extradict/extratuple.py

On 17 July 2017 at 22:34, Steven D'Aprano  wrote:

> On Mon, Jul 17, 2017 at 05:01:58PM -0700, Ethan Furman wrote:
> > Guido has decreed that namedtuple shall be reimplemented with speed in
> mind.
> >
> > I haven't timed it (I'm hoping somebody will volunteer to be the bench
> mark
> > guru), I'll offer my NamedTuple implementation from my aenum [1] library.
>
> With respect Ethan, if you're going to offer up NamedTuple as a faster
> version of namedtuple, you should at least do a quick proof of
> concept to demonstrate that it actually *is* faster. Full bench marking
> can wait, but you should be able to do at least something like:
>
>
> python3 -m timeit --setup "from collections import namedtuple" \
> "K = namedtuple('K', 'a b c')"
>
> versus
>
> python3 -m timeit --setup "from aenum import NamedTuple" \
> "K = NamedTuple('K', 'a b c')"
>
> (or whatever the interface is). If there's only a trivial speed up, or
> if its slower, then there's no point even considing it unless you speed
> it up first.
>
>
> --
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] a new namedtuple

[Guido van Rossum]

On Tue, Jul 18, 2017 at 8:56 AM, Ethan Furman  wrote:

> I certainly don't expect the signature to change, but why is using a
> metaclass out?  The use (or not) of a metaclass /is/ an implementation
> detail.
>

It is until you try to subclass with another metaclass -- then you have a
metaclass conflict. If the namedtuple had no metaclass this would not be a
conflict. (This is one reason to love class decorators.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] a new namedtuple

[Ethan Furman]

On 07/17/2017 06:25 PM, Eric Snow wrote:

On Mon, Jul 17, 2017 at 6:01 PM, Ethan Furman wrote:



Guido has decreed that namedtuple shall be reimplemented with speed in mind.


FWIW, I'm sure that any changes to namedtuple will be kept as minimal
as possible.  Changes would be limited to the underlying
implementation, and would not include the namedtuple() signature, or
using metaclasses, etc.  However, I don't presume to speak for Guido
or Raymond. :)


I certainly don't expect the signature to change, but why is using a metaclass out?  The use (or not) of a metaclass 
/is/ an implementation detail.


--
~Ethan~
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] pathlib.Path should handle Pythonpath or package root

[Nick Coghlan]

On 18 July 2017 at 22:08, George Fischhof  wrote:
> Hi there,
>
> I created a program which uses plugins (import them). I started to test it,
> and found that I need two types of paths: one for file system and another
> one which is package relative.
>
> So I thing this is a good idea, to enhance pathlib to handle package roots.

Is there a specific behaviour you're looking for that isn't already
addressed by "pathlib.Path(module.__file__).parent"?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] a new namedtuple

[Nick Coghlan]

On 18 July 2017 at 14:31, Guido van Rossum  wrote:
> On Mon, Jul 17, 2017 at 6:25 PM, Eric Snow 
> wrote:
>>
>> On Mon, Jul 17, 2017 at 6:01 PM, Ethan Furman  wrote:
>> > Guido has decreed that namedtuple shall be reimplemented with speed in
>> > mind.
>>
>> FWIW, I'm sure that any changes to namedtuple will be kept as minimal
>> as possible.  Changes would be limited to the underlying
>> implementation, and would not include the namedtuple() signature, or
>> using metaclasses, etc.  However, I don't presume to speak for Guido
>> or Raymond. :)
>
>
> Indeed. I referred people here for discussion of ideas like this:
>
 a = (x=1, y=0)

In that vein, something I'll note that *wasn't* historically possible
due to the lack of keyword argument order preservation is an
implementation that implicitly defines anonymous named tuple types
based on the received keyword arguments.

Given Python 3.6+ though, this works:

from collections import namedtuple

def _make_named_tuple(*fields):
cls_name = "_ntuple_" + "_".join(fields)
# Use the module globals as a cache for pickle compatibility
namespace = globals()
try:
return namespace[cls_name]
except KeyError:
cls = namedtuple(cls_name, fields)
return namespace.setdefault(cls_name, cls)

def ntuple(**items):
cls = _make_named_tuple(*items)
return cls(*items.values())

>>> p1 = ntuple(x=1, y=2)
>>> p2 = ntuple(x=4, y=5)
>>> type(p1) is type(p2)
True
>>> type(p1)


That particular approach isn't *entirely* pickle friendly (since
unpickling will still fail if a suitable type hasn't been defined in
the destination process yet), but you can fix that by way of playing
games with setting cls.__qualname__ to refer to an instance of a
custom class that splits "_ntuple_*" back out into the component field
names in __getattr__ and then calls _make_named_tuple, rather than
relying solely on a factory function as I have done here.

However, it also isn't all that hard to imagine a literal syntax
instead using a dedicated builtin type factory (perhaps based on
structseq) that implicitly produced types that knew to rely on the
appropriate builtin to handle instance creation on unpickling - the
hardest part of the problem (preserving the keyword argument order)
was already addressed in 3.6.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] tempfile.TemporaryDirectory() should be able to create temporary directory at a given arbitrary place

[Steven D'Aprano]

On Tue, Jul 18, 2017 at 01:55:15PM +0200, George Fischhof wrote:
> Hi there,
> 
> I used tempfile.TemporaryDirectory(). On first usage it was good, but on
> second one there was a need to create tempopray directory and files in it a
> given place. (It needed for a test).
> 
> And I found that TemporaryDirectory() is not able to do this. So my idea is
> to implement this behaviour with an addittional path parameter in it.

Guido's Time Machine strikes again. TemporaryDirectory takes a dir 
argument to set the location where the temporary directory is created.

https://docs.python.org/3/library/tempfile.html#tempfile.TemporaryDirectory

As far as I can tell, this functionality has existed as far back as 
Python 2.3:

https://docs.python.org/2/library/tempfile.html#tempfile.mkdtemp



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] tempfile.TemporaryDirectory() should be able to create temporary directory at a given arbitrary place

[George Fischhof]

2017-07-18 14:06 GMT+02:00 Serhiy Storchaka :

> 18.07.17 14:55, George Fischhof пише:
>
>> I used tempfile.TemporaryDirectory(). On first usage it was good, but on
>> second one there was a need to create tempopray directory and files in it a
>> given place. (It needed for a test).
>>
>> And I found that TemporaryDirectory() is not able to do this. So my idea
>> is to implement this behaviour with an addittional path parameter in it.
>>
>
> You can pass the dir argument to TemporaryDirectory().
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>


Hi Serhiy,

thank you very much. ;-) I was lost in documentation... and did not find
this.

BR,
George
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] pathlib.Path should handle Pythonpath or package root

[George Fischhof]

Hi there,

I created a program which uses plugins (import them). I started to test it,
and found that I need two types of paths: one for file system and another
one which is package relative.

So I thing this is a good idea, to enhance pathlib to handle package roots.

(I know about sys.path, environment variables, importlib etc, but I think
it should be in pathlib.)

BR,
George
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] tempfile.TemporaryDirectory() should be able to create temporary directory at a given arbitrary place

[Serhiy Storchaka]

18.07.17 14:55, George Fischhof пише:
I used tempfile.TemporaryDirectory(). On first usage it was good, but on 
second one there was a need to create tempopray directory and files in 
it a given place. (It needed for a test).


And I found that TemporaryDirectory() is not able to do this. So my idea 
is to implement this behaviour with an addittional path parameter in it.


You can pass the dir argument to TemporaryDirectory().

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] tempfile.TemporaryDirectory() should be able to create temporary directory at a given arbitrary place

[George Fischhof]

Hi there,

I used tempfile.TemporaryDirectory(). On first usage it was good, but on
second one there was a need to create tempopray directory and files in it a
given place. (It needed for a test).

And I found that TemporaryDirectory() is not able to do this. So my idea is
to implement this behaviour with an addittional path parameter in it.


BR,
George
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/