Re: [Python-Dev] To reduce Python "application" startup time

[Brett Cannon]

I should mention that I have a prototype design for improving importlib's
lazy loading to be easier to turn on and use. See
https://notebooks.azure.com/Brett/libraries/di2Btqj7zSI/html/Lazy%20importing.ipynb
for my current notes. Part of it includes an explicit lazy_import()
function which would negate needing to hide imports in functions to delay
their importation.

On Wed, 6 Sep 2017 at 20:50 INADA Naoki  wrote:

> > I’m not sure however whether burying imports inside functions (as a kind
> of poor man’s lazy import) is ultimately going to be satisfying.  First,
> it’s not natural, it generally violates coding standards (e.g. PEP 8), and
> can make linters complain.
>
> Of course,  I tried to move imports only when (1) it's only used one
> or two of many functions in the module,
> (2) it's relatively heavy, (3) it's rerely imported from other modules.
>
>
> >   Second, I think you’ll end up chasing imports all over the stdlib and
> third party modules in any sufficiently complicated application.
>
> Agree.  I won't use much time to optimization by moving import from
> top to inner function in stdlib.
>
> I think my import-profiler patch can be polished and committed in Python
> to help
> library maintainers to know import time easily.  (Maybe, `python -X
> import-profile`)
>
>
> > Third, I’m not sure that the gains you’ll get won’t just be overwhelmed
> by lots of other things going on, such as pkg_resources entry point
> processing, pth file processing, site.py effects, command line processing
> libraries such as click, and implicitly added distribution exception hooks
> (e.g. Ubuntu’s apport).
>
> Yes.  I noticed some of them while profiling imports.
> For example, old-style namespace package imports types module for
> types.Module.
> Types module imports functools, and functools imports collections.
> So some efforts in CPython (Avoid importing collections and functools
> from site) is not
> worth enough when there are at least one old-style namespace package
> is installed.
>
>
> >
> > Many of these can’t be blamed on Python itself, but all can contribute
> significantly to Python’s apparent start up time.  It’s definitely worth
> investigating the details of Python import, and a few of us at the core
> sprint have looked at those numbers and thrown around ideas for
> improvement, but we’ll need to look at the effects up and down the stack to
> improve the start up performance for the average Python application.
> >
>
> Yes. I totally agree with you.  That's why I use import-profile.patch
> for some 3rd party libraries.
>
> Currently, I have these ideas to optimize application startup time.
>
> * Faster, or lazily compiling regular expression. (pkg_resources
> imports pyparsing, which has lot regex)
> * More usable lazy import. (which can be solved "PEP 549: Instance
> Properties (aka: module properties)")
> * Optimize enum creation.
> * Faster namedtuple (There is pull request already)
> * Faster ABC
> * Breaking large import tree in stdlib.  (PEP 549 may help this too)
>
> Regards,
>
> INADA Naoki  
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)

[Brett Cannon]

On Thu, 14 Sep 2017 at 12:09 Ivan Levkivskyi  wrote:

> On 14 September 2017 at 01:13, Guido van Rossum  wrote:
>
>>
>> That last sentence is a key observation. Do we even know whether there
>> are (non-toy) things that you can do *in principle* with __class__
>> assignment but which are too slow *in practice* to bother? And if yes, is
>> __getattr__ fast enough? @property?
>>
>>
> I myself have never implemented deprecation warnings management nor lazy
> loading,
> so it is hard to say if __class__ assignment is fast enough.
>

For lazy loading it is, but that's because after the lazy load,
module.__class__ is set back to its original value, so any performance
penalty is paid only once upon triggering __getattribute__ on the lazy
module.

IOW no one has complained and I know plenty of places using the lazy
loading mechanism in importlib at scale. :)

-Brett


> For me it is more combination
> of three factors:
>
> * modest performance improvement
> * some people might find __getattr__ clearer than __class__ assignment
> * this would be consistent with how stubs work
>
>
>> IMO we're still looking for applications.
>>
>>
> How about this
>
> def allow_forward_references(*allowed):
> caller_globals = sys._getframe().__globals__
> def typing_getattr(name):
> if name in allowed:
> return name
> raise AttributeError(...)
> caller_globals.__getattr__ = typing_getattr
>
> from typing_extensions import allow_forward_references
> allow_forward_references('Vertex', 'Edge')
>
> T = TypeVar('T', bound=Edge)
>
> class Vertex(List[Edge]):
> def copy(self: T) -> T:
> ...
>
> class Edge:
> ends: Tuple[Vertex, Vertex]
> ...
>
> Look mum, no quotes! :-)
>
> --
> Ivan
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 552: deterministic pycs

[Brett Cannon]

On Fri, 8 Sep 2017 at 10:53 Benjamin Peterson  wrote:

> Thank you all for the feedback. I've now updated the PEP to specify a
> 4-word pyc header with a bit field in every case.
>
> On Fri, Sep 8, 2017, at 09:43, Nick Coghlan wrote:
> > On 8 September 2017 at 07:55, Antoine Pitrou 
> wrote:
> > > On Fri, 8 Sep 2017 07:49:46 -0700
> > > Nick Coghlan  wrote:
> > >> > I'd rather a single magic number and a separate bitfield that tells
> > >> > what the header encodes exactly.  We don't *have* to fight for a
> tiny
> > >> > size reduction of pyc files.
> > >>
> > >> One of Benjamin's goals was for the existing timestamp-based pyc
> > >> format to remain completely unchanged, so we need some kind of marker
> > >> in the magic number to indicate whether the file is using the new
> > >> format or nor.
> > >
> > > I don't think that's a useful goal, as long as we bump the magic
> number.
> >
> > Yeah, we (me, Benjamin, Greg) discussed that here, and we agree -
> > there isn't actually any benefit to keeping the timestamp based pyc's
> > using the same layout, since the magic number is already going to
> > change anyway.
> >
> > Given that, I think your suggested 16 byte header layout would be a
> > good one: 4 byte magic number, 4 bytes reserved for format flags, 8
> > bytes with an interpretation that depends on the format flags.
> >
>

+1 from me!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pythonhosted.org doc upload no longer works

[Brett Cannon]

This is actually a question for distutils-sig since they manage PyPI.

On Mon, 4 Sep 2017 at 03:50 Giampaolo Rodola'  wrote:

> I know pythonhosted.org was deprecated long ago in favor of readthedocs
> but I kept postponing it and my doc for psutil is still hosted on
> https://pythonhosted.org/psutil/.
>
> http://pythonhosted.org/ web page still recommends this:
>
> < http://pypi.python.org/pypi?%3Aaction=pkg_edit=yourpackage), and
> fill out the form at the bottom of the page.>>
>
> I took a look at:
> https://pypi.python.org/pypi?:action=pkg_edit=psutil
> ...but the form to upload the doc in zip format is gone.
>
> The only available option is the "destroy documentation" button.
> I would like to at least upload a static HTML page redirecting to the new
> doc which I will soon move on readthedocs. Is that possible?
>
> Thanks in advance.
>
> --
> Giampaolo - http://grodola.blogspot.com
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Does Cygwin still have broken slot initialization?

[Stefan Krah]

Hi,

The docs have this rule for slot initialization for the benefit of Cygwin:

   
https://github.com/python/cpython/commit/db6a569de7ae595ada53b618fce6bbbd1c98d350

Synopsis


  -PyType_GenericNew, /* tp_new */
  +noddy_NoddyType.tp_new = PyType_GenericNew;
  +if (PyType_Ready(_NoddyType) < 0)
  +return;



This is absolutely not required by C99 (and probably never was).

'PyType_GenericNew' is an address constant, and MSVC supports it just
fine -- at least since VS 2008.


Does anyone know if Cygwin still misbehaves? I would like to get rid
of this arcane rule.

   https://bugs.python.org/issue31443


Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com