Re: [Python-Dev] Decision of having a deprecation period or not for changing csv.DictReader returning type.

[尚辉]

Since regular dicts are ordered in 3.7, it might be cleaner to
returning regular dict instead of OrderedDict?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Usefulness of binary compatibility accross Python versions?

[Random832]

On Sat, Dec 16, 2017, at 08:22, Antoine Pitrou wrote:
> Typically, when adding a tp_XXX slot, you also need to add a
> Py_TPFLAGS_HAVE_XXX type flag to signal those static type structures
> that have been compiled against a recent enough PyTypeObject
> definition.  This way, extensions compiled against Python N-1 are
> supposed to "still work": as they don't have Py_TPFLAGS_HAVE_XXX set,
> the core Python runtime won't try to access the (non-existing) tp_XXX
> member.

Is there any practical for of having the flag off for one slot and on
for another slot that's been added later?

Could this be replaced (that is, a slot for such a thing added before
it's too late) with a simple counter that goes up with each version, and
any "unused" slot should have NULL or some other sentinel value? If it
really is important to have the flags themselves, just add another set
of flags - Py_TPFLAGS_HAVE_MORE_FLAGS.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Usefulness of binary compatibility accross Python versions?

[Nathaniel Smith]

On Dec 16, 2017 11:44 AM, "Guido van Rossum"  wrote:

On Sat, Dec 16, 2017 at 11:14 AM, Antoine Pitrou 
wrote:

> On Sat, 16 Dec 2017 19:37:54 +0100
> Antoine Pitrou  wrote:
> >
> > Currently, you can pass a `module_api_version` to PyModule_Create2(),
> > but that function is for specialists only :-)
> >
> > ("""Most uses of this function should be using PyModule_Create()
> > instead; only use this if you are sure you need it.""")
>
> Ah, it turns out I misunderstood that piece of documentation and also
> what PEP 3121 really did w.r.t the module API check.
>
> PyModule_Create() is actually a *macro* calling PyModule_Create2() with
> the version number is was compiled against!
>
> #ifdef Py_LIMITED_API
> #define PyModule_Create(module) \
> PyModule_Create2(module, PYTHON_ABI_VERSION)
> #else
> #define PyModule_Create(module) \
> PyModule_Create2(module, PYTHON_API_VERSION)
> #endif
>
> And there's already a check for that version number in moduleobject.c:
> https://github.com/python/cpython/blob/master/Objects/moduleobject.c#L114
>
> That check is always invoked when calling PyModule_Create() and
> PyModule_Create2().  Currently it merely invokes a warning, but we can
> easily turn that into an error.
>
> (with apologies to Martin von Löwis for not fully understanding what he
> did at the time :-))
>

If it's only a warning, I worry that if we stop checking the flag bits it
can cause wild pointer following. This sounds like it would be a potential
security issue (load a module, ignore the warning, try to use a certain API
on a class it defines, boom). Also, could there still be 3rd party modules
out there that haven't been recompiled in a really long time and use some
older backwards compatible module initialization API? (I guess we could
stop supporting that and let them fail hard.)


I think there's a pretty simple way to avoid this kind of problem.

Since PEP 3149 (Python 3.2), the import system has (IIUC) checked for:

foo.cpython-XYm.so
foo.abi3.so
foo.so

If we drop foo.so from this list, then we're pretty much guaranteed not to
load anything into a python that it wasn't intended for.

How disruptive would this be? AFAICT there hasn't been any standard way to
build python extensions named like 'foo.so' since 3.2 was released, so
we're talking about modules from 3.1 and earlier (or else people who are
manually hacking around the compatibility checking system, who can
presumably take care of themselves). We've at a minimum been issuing
warnings about these modules for 5 versions now (based on Antoine's
analysis above), and I'd be really surprised if a module built for 3.1
works on 3.7 anyway. So this change seems pretty reasonable to me.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 -- Context Variables

[Yury Selivanov]

Hi Ben,

On Sun, Dec 17, 2017 at 10:38 AM, Ben Darnell  wrote:
> On Tue, Dec 12, 2017 at 12:34 PM Yury Selivanov 
> wrote:
>>
>> Hi,
>>
>> This is a new proposal to implement context storage in Python.
>>
>> It's a successor of PEP 550 and builds on some of its API ideas and
>> datastructures.  Contrary to PEP 550 though, this proposal only focuses
>> on adding new APIs and implementing support for it in asyncio.  There
>> are no changes to the interpreter or to the behaviour of generator or
>> coroutine objects.
>
>
> I like this proposal. Tornado has a more general implementation of a similar
> idea
> (https://github.com/tornadoweb/tornado/blob/branch4.5/tornado/stack_context.py),
> but it also tried to solve the problem of exception handling of
> callback-based code so it had a significant performance cost (to interpose
> try/except blocks all over the place). Limiting the interface to
> coroutine-local variables should keep the performance impact minimal.

Thank you, Ben!

Yes, task local API of PEP 567 has no impact on generators/coroutines.
Impact on asyncio should be well within 1-2% slowdown, visible only in
microbenchmarks (and asyncio will be 3-6% faster in 3.7 at least due
to some asyncio.Task C optimizations).

[..]
> One caveat based on Tornado's experience with stack_context: There are times
> when the automatic propagation of contexts won't do the right thing (for
> example, a database client with a connection pool may end up hanging on to
> the context from the request that created the connection instead of picking
> up a new context for each query).

I can see two scenarios that could lead to that:

1. The connection pool explicitly captures the context with 'get_context()' at
the point where it was created. It later schedules all of its code within the
captured context with Context.run().

2. The connection pool calls ContextVar.get() once and _caches_ it.

Both (1) and (2) are anti-patterns.  The documentation of asyncio and
contextvars
module will explain that users are supposed to simply call
ContextVar.get() whenever
they need to get a context value (e.g. there's no need to
cache/persist it) and that
 they should not manage the context manually (just trust asyncio to do
that for you).

Thank you,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decision of having a deprecation period or not for changing csv.DictReader returning type.

[Antoine Pitrou]

On Mon, 18 Dec 2017 03:24:05 +0900
INADA Naoki  wrote:

> On Mon, Dec 18, 2017 at 12:46 AM, Guido van Rossum  wrote:
> > My gut suggests me not to do this (neither here nor in other similar cases).
> > I doubt there's much of a performance benefit anyway.  
> 
> OrderedDict uses 2x memory than dict.
> So it affects memory usage of applications loading large CSV with DictReader.
> 
> While I think application should use tuple when memory consumption is
> matter, there is significant benefit.

Or they should use a dict-of-lists instead of a list-of-dicts.
Or they should simply switch to Pandas.

Simply put, I doubt using DictReader is a sensible decision if you're
writing performance-sensitive code.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decision of having a deprecation period or not for changing csv.DictReader returning type.

[INADA Naoki]

On Mon, Dec 18, 2017 at 12:46 AM, Guido van Rossum  wrote:
> My gut suggests me not to do this (neither here nor in other similar cases).
> I doubt there's much of a performance benefit anyway.

OrderedDict uses 2x memory than dict.
So it affects memory usage of applications loading large CSV with DictReader.

While I think application should use tuple when memory consumption is
matter, there is significant benefit.

INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[David Mertz]

On Sun, Dec 17, 2017 at 8:22 AM, Guido van Rossum  wrote:

> On Sun, Dec 17, 2017 at 2:11 AM, Julien Salort  wrote:
>
>> Naive question from a lurker: does it mean that it works also if one
>> annotates with something that is not a type, e.g. a comment,
>>
>> @dataclass
>> class C:
>> a: "This represents the amplitude" = 0.0
>> b: "This is an offset" = 0.0
>
>
> I would personally not use the notation for this, but it is legal code.
> However static type checkers like mypy won't be happy with this.
>

Mypy definitely won't like that use of annotation, but documentation
systems might.  For example, in a hover tooltip in an IDE/editor, it's
probably more helpful to see the descriptive message than "int" or "float"
for the attribute.

What about data that isn't built-in scalars? Does this look right to people
(and will mypy be happy with it)?

@dataclass
class C:
a:numpy.ndarray = numpy.random.random((3,3))
b:MyCustomClass = MyCustomClass("foo", 37.2, 1+2j)

I don't think those look terrible, but I think this looks better:

@dataclass
class C:
a:Infer = np.random.random((3,3))
b:Infer = MyCustomClass("foo", 37.2, 1+2j)

Where the name 'Infer' (or some other spelling) was a name defined in the
`dataclasses` module.  In this case, I don't want to use `typing.Any` since
I really do want "the type of thing the default value has."

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 -- Context Variables

[Ben Darnell]

On Tue, Dec 12, 2017 at 12:34 PM Yury Selivanov 
wrote:

> Hi,
>
> This is a new proposal to implement context storage in Python.
>
> It's a successor of PEP 550 and builds on some of its API ideas and
> datastructures.  Contrary to PEP 550 though, this proposal only focuses
> on adding new APIs and implementing support for it in asyncio.  There
> are no changes to the interpreter or to the behaviour of generator or
> coroutine objects.
>

I like this proposal. Tornado has a more general implementation of a
similar idea (
https://github.com/tornadoweb/tornado/blob/branch4.5/tornado/stack_context.py),
but it also tried to solve the problem of exception handling of
callback-based code so it had a significant performance cost (to interpose
try/except blocks all over the place). Limiting the interface to
coroutine-local variables should keep the performance impact minimal.

If the contextvars package were published on pypi (and backported to older
pythons), I'd deprecate Tornado's stack_context and use it instead (even if
there's not an official backport, I'll probably move towards whatever
interface is defined in this PEP if it is accepted).

One caveat based on Tornado's experience with stack_context: There are
times when the automatic propagation of contexts won't do the right thing
(for example, a database client with a connection pool may end up hanging
on to the context from the request that created the connection instead of
picking up a new context for each query). Compatibility with this feature
will require testing and possible fixes with many libraries in the asyncio
ecosystem before it can be relied upon.

-Ben


>
>
> PEP: 567
> Title: Context Variables
> Version: $Revision$
> Last-Modified: $Date$
> Author: Yury Selivanov 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 12-Dec-2017
> Python-Version: 3.7
> Post-History: 12-Dec-2017
>
>
> Abstract
> 
>
> This PEP proposes the new ``contextvars`` module and a set of new
> CPython C APIs to support context variables.  This concept is
> similar to thread-local variables but, unlike TLS, it allows
> correctly keeping track of values per asynchronous task, e.g.
> ``asyncio.Task``.
>
> This proposal builds directly upon concepts originally introduced
> in :pep:`550`.  The key difference is that this PEP is only concerned
> with solving the case for asynchronous tasks, and not generators.
> There are no proposed modifications to any built-in types or to the
> interpreter.
>
>
> Rationale
> =
>
> Thread-local variables are insufficient for asynchronous tasks which
> execute concurrently in the same OS thread.  Any context manager that
> needs to save and restore a context value and uses
> ``threading.local()``, will have its context values bleed to other
> code unexpectedly when used in async/await code.
>
> A few examples where having a working context local storage for
> asynchronous code is desired:
>
> * Context managers like decimal contexts and ``numpy.errstate``.
>
> * Request-related data, such as security tokens and request
>   data in web applications, language context for ``gettext`` etc.
>
> * Profiling, tracing, and logging in large code bases.
>
>
> Introduction
> 
>
> The PEP proposes a new mechanism for managing context variables.
> The key classes involved in this mechanism are ``contextvars.Context``
> and ``contextvars.ContextVar``.  The PEP also proposes some policies
> for using the mechanism around asynchronous tasks.
>
> The proposed mechanism for accessing context variables uses the
> ``ContextVar`` class.  A module (such as decimal) that wishes to
> store a context variable should:
>
> * declare a module-global variable holding a ``ContextVar`` to
>   serve as a "key";
>
> * access the current value via the ``get()`` method on the
>   key variable;
>
> * modify the current value via the ``set()`` method on the
>   key variable.
>
> The notion of "current value" deserves special consideration:
> different asynchronous tasks that exist and execute concurrently
> may have different values.  This idea is well-known from thread-local
> storage but in this case the locality of the value is not always
> necessarily to a thread.  Instead, there is the notion of the
> "current ``Context``" which is stored in thread-local storage, and
> is accessed via ``contextvars.get_context()`` function.
> Manipulation of the current ``Context`` is the responsibility of the
> task framework, e.g. asyncio.
>
> A ``Context`` is conceptually a mapping, implemented using an
> immutable dictionary.  The ``ContextVar.get()`` method does a
> lookup in the current ``Context`` with ``self`` as a key, raising a
> ``LookupError``  or returning a default value specified in
> the constructor.
>
> The ``ContextVar.set(value)`` method clones the current ``Context``,
> assigns the ``value`` to it with ``self`` as a key, and sets the
> new ``Context`` as a new current.  

Re: [Python-Dev] Is static typing still optional?

[Guido van Rossum]

On Sun, Dec 17, 2017 at 2:11 AM, Julien Salort  wrote:

> Le 15/12/2017 à 22:14, Paul Moore a écrit :
>
> Annotations and the annotation syntax are fundamental to the design.
>> But that's core Python syntax. But I wouldn't describe types as being
>> that significant to the design, it's more "if you supply them we'll
>> make use of them".
>>
> Naive question from a lurker: does it mean that it works also if one
> annotates with something that is not a type, e.g. a comment,
>
> @dataclass
> class C:
> a: "This represents the amplitude" = 0.0
> b: "This is an offset" = 0.0


I would personally not use the notation for this, but it is legal code.
However static type checkers like mypy won't be happy with this.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decision of having a deprecation period or not for changing csv.DictReader returning type.

[Guido van Rossum]

My gut suggests me not to do this (neither here nor in other similar
cases). I doubt there's much of a performance benefit anyway.

On Sat, Dec 16, 2017 at 10:04 PM, 尚辉  wrote:

> Hi, guys
>
> In https://github.com/python/cpython/pull/4904, I made csv.DictReader
> returning regular dict instead of OrderedDict. But this code could break
> existing code that relied on methods like move_to_end() which are present
> in OrderedDict() but not in dict().
>
> As rhettinger suggested, such code is either unlikely or rare, so it would
> be net less disruptive for users to just go forward with this patch.
>
> So, would we just go forward with this patch or having a deprecation
> period before this patch?
>
> Any help is big thanks.
>
> --
>
>
> Best Regards .
>
> shangdahao
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Julien Salort]

Le 15/12/2017 à 22:14, Paul Moore a écrit :


Annotations and the annotation syntax are fundamental to the design.
But that's core Python syntax. But I wouldn't describe types as being
that significant to the design, it's more "if you supply them we'll
make use of them".

Naive question from a lurker: does it mean that it works also if one
annotates with something that is not a type, e.g. a comment,

@dataclass
class C:
    a: "This represents the amplitude" = 0.0
    b: "This is an offset" = 0.0
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decision of having a deprecation period or not for changing csv.DictReader returning type.

[Antoine Pitrou]

On Sun, 17 Dec 2017 14:04:52 +0800
尚辉  wrote:
> Hi, guys
> 
> In https://github.com/python/cpython/pull/4904, I made csv.DictReader
> returning regular dict instead of OrderedDict. But this code could break
> existing code that relied on methods like move_to_end() which are present
> in OrderedDict() but not in dict().

What is the motivation to return a regular dict?  Is it actually
faster on some benchmark?

PS: apparently the decision to return an OrderedDict was made in...
Python 3.6.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com