Re: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug.
On Fri, May 24, 2019 at 3:49 AM Gregory P. Smith wrote: > > I suggest filing a bug to track this... > I created the issue: https://bugs.python.org/issue37029 -- Inada Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 594: update 1
Steve Dower wrote: We need to make it more clear somehow that Python uses series.major.minor.micro versioning, not SemVer. We could also do with deciding what the "series" part is really supposed to be for. At the moment it seems to be "we change it when the major number gets up to 9, or we feel like it, whichever comes first." -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] we will probably be having an difficult discussion about the stdlib after PEP 594 is done (was: PEP 594: Removing dead batteries from the standard library)
On May 23, 2019, at 14:17, Brett Cannon wrote: > I'm personally viewing it as a first step in addressing the maintenance > burden we have with such a large stdlib. Christian started this work over a > year ago and I think it's worth seeing through. After that we should probably > have a discussion as a team about how we view the stdlib long-term and how > that ties into maintaining it so that people's opinion of the stdlib's > quality goes up rather than viewing the quality of it as varying > module-to-module. +1 :) -Barry signature.asc Description: Message signed with OpenPGP ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 594: update 1
On Thu, May 23, 2019 at 02:06:13PM -0700, Brett Cannon wrote: > On Wed, May 22, 2019 at 1:23 PM Sean Wallitsch < > sean.wallit...@dreamworks.com> wrote: > > > My apologies for that oversight. My understanding is that many of the > > methods present in aifc depend heavily on audioop for reading and writing. > > > > But are people using audioop directly? Yes they are. https://old.reddit.com/r/Python/comments/brgl8v/pep_594_removing_dead_batteries_from_the_standard/eodvexl/ Shouldn't the PEP be responsable for establishing (as well as any negative can be proven) that these modules aren't used, rather than merely assuming that they aren't? Of course it is hard to establish that a module isn't being used. Not all code can be searched for on the Internet, there's huge amounts of non-open source code written by users that never leaves their computer, or behind corporate firewalls. The difficulty of proving this negative requires that we be especially conservative about removal. At the very least, we should demonstrate that the package is *an active burden*. Its not enough to say "nobody has touched this module for ages" since stable, working code doesn't rot unless we change the language (or the std lib!) and break it. So far, the PEPs record hasn't been too good: out of the 31 modules listed in the original draft, the PEP marks four as "keep", leaving 27 to be deprecated. So far, we've found: - the sound libraries are in heavy use by hobbyists and the professional audio-visual industry; aifc in particular is a mature, stable library that isn't break and doesn't need much attention; - cgi and cgitb are used by people who don't need a heavyweight HTML solution (see Reddit); - colorsys, fileinput and nntplib (at least) have been added to the "keep" list; - the removal of spwd (and crypt?) has been controversial. So far, nearly 30% of the so-called "dead batteries" turn out to be not so dead after all. I may have missed some. Nor have I even talked much about modules which I personally use occasionally, like binhex, because it's not about *me* its about the community. As much as I would prefer binhex to remain, if it is removed I will be disappointed but I'll make do. Speaking of binhex, I tried to get a sense of how much of a burden it actually is. There was a comment by Guido in *2007*: https://github.com/python/cpython/commit/34a042d301d6ab88645046a6dfa6c38265ca4b39 "This is the last time I fix binhex. If it breaks again it goes in the dustbin" which of course is Guido's perogative to say Won't Fix. Its now 2019 and it hasn't broken again, or at least it hasn't broken again sufficiently to make any difference. Serhey found one issue: https://bugs.python.org/issue29566 but I'm not sure if that's actually a bug or not. There was a documentation bug (in the source code) in 2017: https://bugs.python.org/issue29557 and one enhancement request to clean up the code: https://bugs.python.org/issue34063 Other than that, Serhey touched binhex.py as part of a mass patch to almost the whole stdlib to modernise file handling to use "with". So by my count, in 12 years since Guido's frustrated comment about binhex, it has seen: - removal of an obsolete and ambiguous comment from the source code; - addition of a few with blocks to modernise file handling; - one enhancement request still outstanding; - one possibly a bug (or maybe not) still outstanding. I may have missed some, but we're talking about one issue per three years. How is this a maintenance burden? -- Steven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] we will probably be having an difficult discussion about the stdlib after PEP 594 is done (was: PEP 594: Removing dead batteries from the standard library)
On Thu, May 23, 2019 at 11:03 AM Barry Warsaw wrote: > On May 20, 2019, at 13:15, Christian Heimes wrote: > > > > here is the first version of my PEP 594 to deprecate and eventually > remove modules from the standard library. The PEP started last year with > talk during the Python Language Summit 2018, > https://lwn.net/Articles/755229/. > > > > The PEP can be confirmed in two stages. I'm not planning any code > changes for 3.8. Instead I only like to document a bunch of modules as > deprecated. Active deprecation is planned for 3.9 and removal for 3.10. The > long deprecation phase gives us 3 years to change our minds or handle edge > cases, too. > > Hi Christian, > > First, I appreciate the work you’ve done on this, and the general > sentiment. But I do feel like this is a partial solution to a problem the > Python distribution has had for a very long time. Maybe it’s a good step > forward, but in a sense it is incomplete and only chips away at the problem. > Yep, but I will take this chipping away over nothing. :) > > We have two competing pressures, one to provide a rich standard library > with lots of useful features that come right out of the box. Let’s not > underestimate the value that this has for our users, and the contribution > such a stdlib has made to making Python as popular as it is. > > But it’s also true that lots of the stdlib don’t get the love they need to > stay relevant, and a curated path to keeping only the most useful and > modern libraries. I wonder how much the long development cycle and > relatively big overhead for contributing to stdlib maintenance causes a big > part of our headaches with the stdlib. Current stdlib development > processes also incur burden for alternative implementations. > > We’ve had many ideas over the years, such as stripping the CPython repo > stdlib to its bare minimum and providing some way of *distributing* a sumo > tarball. But none have made it far enough to be adopted. I don’t have any > new bright ideas for how to make this work, but I think finding a holistic > approach to these competing pressures is in the best long term interest of > Python. > > That all said, if PEP 594 can be viewed as part of the bigger picture, > then maybe it’s a good idea. Perhaps the approaches for maintaining such > deprecated libraries can be used as an experiment for finding a more > modern, useful, and vibrant approach to stdlib maintenance. > I'm personally viewing it as a first step in addressing the maintenance burden we have with such a large stdlib. Christian started this work over a year ago and I think it's worth seeing through. After that we should probably have a discussion as a team about how we view the stdlib long-term and how that ties into maintaining it so that people's opinion of the stdlib's quality goes up rather than viewing the quality of it as varying module-to-module. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 594: update 1
On Wed, May 22, 2019 at 1:23 PM Sean Wallitsch < sean.wallit...@dreamworks.com> wrote: > My apologies for that oversight. My understanding is that many of the > methods present in aifc depend heavily on audioop for reading and writing. > But are people using audioop directly? This shifts whether we still publicly deprecate auidoop and make it an implementation detail of aifc versus a free-standing module whose API we need to continue to support. -Brett > > On Wed, May 22, 2019 at 12:35 PM Nathaniel Smith wrote: > >> On Wed, May 22, 2019, 12:14 Sean Wallitsch >> wrote: >> >>> Dear python-dev, >>> >>> I'm writing to provide some feedback on PEP-594, primarily the proposed >>> deprecation and reason for the removal of the aifc and audioop libraries. >>> >>> The post production film industry continues to make heavy use of AIFFs, >>> as completely uncompressed audio is preferred. Support for the consumer >>> alternatives (ALAC, FLAC) is virtually non-existent, with no movement >>> towards adoption of those formats. Even Apple's own professional editing >>> tool Final Cut Pro does not support ALAC. Many of the applications also >>> support WAV, but not all. >>> >>> Removal of this module from the standard library is complicated by the >>> fact that a large number of film industry facilities have extremely limited >>> internet access for security reasons. This does not make it impossible to >>> get a library from pypi, but speaking to those devs has made me aware of >>> what a painful process that is for them. They have benefited greatly from >>> aifc's inclusion in the standard library. >>> >> >> That's really helpful data, thank you! >> >> Is audioop also used? You mention both aifc and audioop at the beginning >> and end of your message, but all the details in the middle focus on just >> aifc. >> >> -n >> > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug.
It seems pretty clear now that the primary cause is keeping arenas sorted by number of free pools. As deallocation goes on, the number of distinct "# of free pools" values decreases, leaving large numbers of arenas sharing the same value. Then, e.g., if there are 10,000 arenas with 30 free pools remaining, and another pool is freed in one of them, its free pool count jumps to 31, and on average we have to crawl over 5,000 of its former peers to locate its new position in the list. Which is also consistent with what the OP saw, and I did too but in a much smaller case: the longer deallocation goes on, the slower it gets (fewer and fewer Nodes freed per unit time). Which suggests a different - albeit related - approach: it's not really the order of arenas that matters, but the partitioning of arenas by number of free pools. So instead of a singly doubly linked list of arenas, have a vector of doubly linked lists, one list per possible number of free pools. That's a fixed and relatively small number. For example, even if all arena bytes could be used for pools (they can't be - there's bookkeeping info at the start of an arena), the maximum number of free pools in an arena would be 2**18 / 2**12 == 2**6 == 64. When a pool is freed, no big deal: unlink the arena from its current list, and stick it in the list for the next higher number of free pools. This, of course, is constant-time. Of course there are edge cases (e.g., if the area is entirely free, it should be given back to C instead), but they seem pretty obvious, and I haven't thought of a real problem. Tedious, though ;-) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 595: Improving bugs.python.org
Hello, Berker and I have been working on a PEP that suggests we keep using and improving bugs.python.org and Roundup instead of switching to GitHub Issues as proposed by PEP 581. The PEP covers: * What are the advantages of Roundup over GitHub issues; * What features are missing in Roundup and how can we add them; * Issues with PEP 581; * Issues with the migration plan proposed by PEP 588; The rendered version of PEP 595 is available at https://www.python.org/dev/peps/pep-0595/ For reference, you can consult PEP 581 and 588 at https://www.python.org/dev/peps/pep-0581/ and https://www.python.org/dev/peps/pep-0588/ The full text of the PEP is include below. We are planning to update the PEP to include the feedback we receive and to update the status of features as we implement them (we also have a Google Summer of Code students working on it). Best Regards, Ezio Melotti PEP: 595 Title: Improving bugs.python.org Author: Ezio Melotti , Berker Peksag Status: Draft Type: Process Content-Type: text/x-rst Created: 12-May-2019 Abstract This PEP proposes a list of improvements to make bugs.python.org more usable for contributors and core developers. This PEP also discusses why remaining on Roundup should be preferred over switching to GitHub Issues, as proposed by :pep:`581`. Motivation == On May 14th, 2019 :pep:`581` has been accepted [#]_ without much public discussion and without a clear consensus [#]_. The PEP contains factual errors and doesn't address some of the issues that the migration to GitHub Issues might present. Given the scope of the migration, the amount of work required, and how it will negatively affect the workflow during the transition phase, this decision should be re-evaluated. Roundup advantages over GitHub Issues = This section discusses reasons why Roundup should be preferred over GitHub Issues and Roundup features that are not available on GitHub Issues. * **Roundup is the status quo.** Roundup has been an integral part of the CPython workflow for years. It is a stable product that has been tested and customized to adapt to our needs as the workflow evolved. It is possible to gradually improve it and avoid the disruption that a switch to a different system would inevitabily bring to the workflow. * **Open-source and Python powered.** Roundup is an open-source project and is written in Python. By using it and supporting it, we also support the Python ecosystem. Several features developed for bpo have also been ported to upstream Roundup over the years. * **Fully customizable.** Roundup can be (and has been) fully customized to fit our needs. * **Finer-grained access control.** Roundup allows the creation of different roles with different permissions (e.g. create, view, edit, etc.) for each individual property, and users can have multiple roles. * **Flexible UI.** While Roundup UI might look dated, it is convenient and flexible. For example, on the issue page, each field (e.g. title, type, versions, status, linked files and PRs, etc.) have appropriate UI elements (input boxes, dropdowns, tables, etc.) that are easy to set and also provide a convenient way to get info about the issue at a glance. The number of fields, their values, and the UI element they use is also fully customizable. GitHub only provides labels. The issue list page presents the issues in a compact and easy to read table with separate columns for different fields. For comparison, Roundup lists 50 issues in a screen, whereas GitHub takes two screens to shows 25 issues. * **Advanced search.** Roundup provides an accurate way to search and filter by using any combination of issue fields. It is also possible to customize the number of results and the fields displayed in the table, and the sorting and grouping (up to two levels). bpo also provides predefined summaries (e.g. "Created by you", "Assigned to you", etc.) and allows the creation of custom search queries that can be conveniently accessed from the sidebar. * **Nosy list autocomplete.** The nosy list has an autocomplete feature that suggests maintainers and experts. The suggestions are automatically updated when the experts index [#]_ changes. * **Dependencies and Superseders.** Roundup allows to specify dependencies that must be addressed before the current issues can be closed and a superseder issue to easily mark duplicates [#]_. The list of dependencies can also be used to create meta-issues that references several other sub-issues [#]_. Improving Roundup = This section lists some of the issues mentioned by :pep:`581` and other desired features and discusses how they can be implemented by improving Roundup and/or our instance. * **REST API support.** A REST API will make integration with other services and the development of new tools and applications
Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library
What would _really_ help is getting the groups that maintain each dead parrot to collaborate on a "Python Legacy release" that adds back anything with a maintainer to the stdlib of the current Python version. Even that will demand large resources, and if it's to be organised in a way that doesn't involve lots of dev time then perhaps the PSF could be involved? I presume the Steering Committee are the people to consider directions like this. Steve Holden On Thu, May 23, 2019 at 7:03 PM Barry Warsaw wrote: > On May 20, 2019, at 13:15, Christian Heimes wrote: > > > > here is the first version of my PEP 594 to deprecate and eventually > remove modules from the standard library. The PEP started last year with > talk during the Python Language Summit 2018, > https://lwn.net/Articles/755229/. > > > > The PEP can be confirmed in two stages. I'm not planning any code > changes for 3.8. Instead I only like to document a bunch of modules as > deprecated. Active deprecation is planned for 3.9 and removal for 3.10. The > long deprecation phase gives us 3 years to change our minds or handle edge > cases, too. > > Hi Christian, > > First, I appreciate the work you’ve done on this, and the general > sentiment. But I do feel like this is a partial solution to a problem the > Python distribution has had for a very long time. Maybe it’s a good step > forward, but in a sense it is incomplete and only chips away at the problem. > > We have two competing pressures, one to provide a rich standard library > with lots of useful features that come right out of the box. Let’s not > underestimate the value that this has for our users, and the contribution > such a stdlib has made to making Python as popular as it is. > > But it’s also true that lots of the stdlib don’t get the love they need to > stay relevant, and a curated path to keeping only the most useful and > modern libraries. I wonder how much the long development cycle and > relatively big overhead for contributing to stdlib maintenance causes a big > part of our headaches with the stdlib. Current stdlib development > processes also incur burden for alternative implementations. > > We’ve had many ideas over the years, such as stripping the CPython repo > stdlib to its bare minimum and providing some way of *distributing* a sumo > tarball. But none have made it far enough to be adopted. I don’t have any > new bright ideas for how to make this work, but I think finding a holistic > approach to these competing pressures is in the best long term interest of > Python. > > That all said, if PEP 594 can be viewed as part of the bigger picture, > then maybe it’s a good idea. Perhaps the approaches for maintaining such > deprecated libraries can be used as an experiment for finding a more > modern, useful, and vibrant approach to stdlib maintenance. > > Cheers, > -Barry > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library
It might also serve to identify those with an interest in maintaining the non-core packages, which might even be given some special status on PyPI. On Thu, May 23, 2019 at 9:01 AM Alex Walters wrote: > I've watched the entire thread and its taken me a few days to put a finger > on what bothers me about it. > > In my opinion, this shouldn't be a pep describing the list of modules that > need to go as "dead batteries", but should be a process pep describing how > dead batteries should be removed, and the individual modules should be > given > their own pep. I think reactions to individual module peps will give a > better indication of if it's a used module or not. > > > -Original Message- > > From: Python-Dev > list=sdamon@python.org> On Behalf Of Christian Heimes > > Sent: Monday, May 20, 2019 4:15 PM > > To: Python Dev > > Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard > > library > > > > Hi, > > > > here is the first version of my PEP 594 to deprecate and eventually > remove > > modules from the standard library. The PEP started last year with talk > during > > the Python Language Summit 2018, https://lwn.net/Articles/755229/. > > > > The PEP can be confirmed in two stages. I'm not planning any code changes > > for 3.8. Instead I only like to document a bunch of modules as > deprecated. > > Active deprecation is planned for 3.9 and removal for 3.10. The long > > deprecation phase gives us 3 years to change our minds or handle edge > > cases, too. > > > > Regards, > > Christian > > > > > > > > > > PEP: 594 > > Title: Removing dead batteries from the standard library > > Author: Christian Heimes > > Status: Active > > Type: Process > > Content-Type: text/x-rst > > Created: 20-May-2019 > > Post-History: > > > > > > Abstract > > > > > > This PEP proposed a list of standard library modules to be removed from > the > > standard library. The modules are mostly historic data formats and APIs > that > > have been superseded a long time ago, e.g. Mac OS 9 and Commodore. > > > > > > Rationale > > = > > > > Back in the early days of Python, the interpreter came with a large set > of > > useful modules. This was often refrained to as "batteries included" > > philosophy and was one of the corner stones to Python's success story. > > Users didn't have to figure out how to download and install separate > > packages in order to write a simple web server or parse email. > > > > Times have changed. The introduction of the cheese shop (PyPI), > setuptools, > > and later pip, it became simple and straight forward to download and > install > > packages. Nowadays Python has a rich and vibrant ecosystem of third party > > packages. It's pretty much standard to either install packages from PyPI > or > > use one of the many Python or Linux distributions. > > > > On the other hand, Python's standard library is piling up cruft, > unnecessary > > duplication of functionality, and dispensable features. This is > undesirable > > for several reasons. > > > > * Any additional module increases the maintenance cost for the Python > core > > development team. The team has limited resources, reduced maintenance > > cost > > frees development time for other improvements. > > * Modules in the standard library are generally favored and seen as the > > de-facto solution for a problem. A majority of users only pick 3rd > party > > modules to replace a stdlib module, when they have a compelling reason, > > e.g. > > lxml instead of `xml`. The removal of an unmaintained stdlib module > > increases the chances of a community contributed module to become > > widely > > used. > > * A lean and mean standard library benefits platforms with limited > resources > > like devices with just a few hundred kilobyte of storage (e.g. BBC > > Micro:bit). Python on mobile platforms like BeeWare or WebAssembly > > (e.g. pyodide) also benefit from reduced download size. > > > > The modules in the PEP have been selected for deprecation because their > > removal is either least controversial or most beneficial. For example > > least controversial are 30 years old multimedia formats like ``sunau`` > > audio format, which was used on SPARC and NeXT workstations in the late > > 1980ties. The ``crypt`` module has fundamental flaws that are better > solved > > outside the standard library. > > > > This PEP also designates some modules as not scheduled for removal. Some > > modules have been deprecated for several releases or seem unnecessary at > > first glance. However it is beneficial to keep the modules in the > standard > > library, mostly for environments where installing a package from PyPI is > not > > an option. This can be cooperate environments or class rooms where > > external > > code is not permitted without legal approval. > > > > * The usage of FTP is declining, but some files are still provided over
Re: [Python-Dev] PEP 594: update 1
Besides which, it would be lovely to have a major release that didn't involve any pain at all for the majority of users! Our erstwhile BDFL always eschewed two-digit version identifiers- due to the possibility for confusion about collating sequence, I beleive.. We should honour his preferences by going from 3.9 to 4.0. On Thu, May 23, 2019 at 5:49 PM Anders Munch wrote: > Fra: Paul Moore [mailto:p.f.mo...@gmail.com]: > > > A major version change serves as a heads up that something is going on > and you need to check the consequences before upgrading. > > Python's backward compatibility policy allows breaking changes between > versions X.Y and X.Y+1 (with a suitable deprecation period). This proposal > is no different. > > Except perhaps in scale. The same people that upgrade from 3.x to 3.x+1 > without giving it a second thought, just to be on the latest version, will > hesitate to go from 3.x to 4.y, because the major version change is a hint > that they should be more careful. That means they're ready for it when > they get the ModuleNotFoundError exception, instead of feeling ambushed. > > OK, it may be that this is not enough to warrant a 4.0 release, but I do > think python-dev should get over its fear of major versions sometime. And > that transitioning to a leaner standard library with some things moved to > PyPI would not be a bad program statement for a Python 4.0. > > regards, Anders > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug.
I suggest filing a bug to track this... On Thu, May 23, 2019 at 10:13 AM Tim Peters wrote: > [Inada Naoki ] > > ... > > 2. This loop is cleary hot: > > > https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L1816-L1819 > > Which is 3 lines of code plus a closing brace. The OP didn't build > their own Python, and the source from which it was compiled wasn't > available to them. But they did say that when they got into gdb and > single-stepped, it was looping through the same three lines of code, > in obmalloc.c, over & over. Which is 100% consistent with the loop > you identified as being "the hot spot". > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library
On May 20, 2019, at 13:15, Christian Heimes wrote: > > here is the first version of my PEP 594 to deprecate and eventually remove > modules from the standard library. The PEP started last year with talk during > the Python Language Summit 2018, https://lwn.net/Articles/755229/. > > The PEP can be confirmed in two stages. I'm not planning any code changes for > 3.8. Instead I only like to document a bunch of modules as deprecated. Active > deprecation is planned for 3.9 and removal for 3.10. The long deprecation > phase gives us 3 years to change our minds or handle edge cases, too. Hi Christian, First, I appreciate the work you’ve done on this, and the general sentiment. But I do feel like this is a partial solution to a problem the Python distribution has had for a very long time. Maybe it’s a good step forward, but in a sense it is incomplete and only chips away at the problem. We have two competing pressures, one to provide a rich standard library with lots of useful features that come right out of the box. Let’s not underestimate the value that this has for our users, and the contribution such a stdlib has made to making Python as popular as it is. But it’s also true that lots of the stdlib don’t get the love they need to stay relevant, and a curated path to keeping only the most useful and modern libraries. I wonder how much the long development cycle and relatively big overhead for contributing to stdlib maintenance causes a big part of our headaches with the stdlib. Current stdlib development processes also incur burden for alternative implementations. We’ve had many ideas over the years, such as stripping the CPython repo stdlib to its bare minimum and providing some way of *distributing* a sumo tarball. But none have made it far enough to be adopted. I don’t have any new bright ideas for how to make this work, but I think finding a holistic approach to these competing pressures is in the best long term interest of Python. That all said, if PEP 594 can be viewed as part of the bigger picture, then maybe it’s a good idea. Perhaps the approaches for maintaining such deprecated libraries can be used as an experiment for finding a more modern, useful, and vibrant approach to stdlib maintenance. Cheers, -Barry signature.asc Description: Message signed with OpenPGP ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug.
[Inada Naoki ] > ... > 2. This loop is cleary hot: > https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L1816-L1819 Which is 3 lines of code plus a closing brace. The OP didn't build their own Python, and the source from which it was compiled wasn't available to them. But they did say that when they got into gdb and single-stepped, it was looping through the same three lines of code, in obmalloc.c, over & over. Which is 100% consistent with the loop you identified as being "the hot spot". ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 594: update 1
On 23May2019 0947, Anders Munch wrote: Fra: Paul Moore [mailto:p.f.mo...@gmail.com]: A major version change serves as a heads up that something is going on and you need to check the consequences before upgrading. Python's backward compatibility policy allows breaking changes between versions X.Y and X.Y+1 (with a suitable deprecation period). This proposal is no different. Except perhaps in scale. The same people that upgrade from 3.x to 3.x+1 without giving it a second thought, just to be on the latest version, will hesitate to go from 3.x to 4.y, because the major version change is a hint that they should be more careful. That means they're ready for it when they get the ModuleNotFoundError exception, instead of feeling ambushed. OK, it may be that this is not enough to warrant a 4.0 release, but I do think python-dev should get over its fear of major versions sometime. And that transitioning to a leaner standard library with some things moved to PyPI would not be a bad program statement for a Python 4.0. We need to make it more clear somehow that Python uses series.major.minor.micro versioning, not SemVer. I see this confusion happen all the time. Think of it as Python3 7.x moving to Python3 8.x, and evaluate how much effort you should put into migration :) That said, I'm totally in favour of Python 4.0 being the point where we change expectations about what will be in the standard library (and also C API, language/implementation semantics, etc.). But until we have a clear vision statement that everyone (or enough) is behind, there's nothing gained by prematurely causing that much disruption. Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug.
On Fri, 24 May 2019 00:59:08 +0900 Inada Naoki wrote: > > > > It's relatively easy to test replacing our custom allocators with the > > system ones, yes? Can we try those to see whether they have the same > > characteristic? > > > > Yes. > > PYTHONMALLOC=malloc LD_PRELOAD=/path/to/jemalloc.so python script.py > > I will try it tomorrow. Can you simply test with the system allocator rather than jemalloc? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 594: update 1
Fra: Paul Moore [mailto:p.f.mo...@gmail.com]: > > A major version change serves as a heads up that something is going on and > > you need to check the consequences before upgrading. > Python's backward compatibility policy allows breaking changes between > versions X.Y and X.Y+1 (with a suitable deprecation period). This proposal is > no different. Except perhaps in scale. The same people that upgrade from 3.x to 3.x+1 without giving it a second thought, just to be on the latest version, will hesitate to go from 3.x to 4.y, because the major version change is a hint that they should be more careful. That means they're ready for it when they get the ModuleNotFoundError exception, instead of feeling ambushed. OK, it may be that this is not enough to warrant a 4.0 release, but I do think python-dev should get over its fear of major versions sometime. And that transitioning to a leaner standard library with some things moved to PyPI would not be a bad program statement for a Python 4.0. regards, Anders ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library
+1 On 5/23/19 1:00 AM, Alex Walters wrote: I've watched the entire thread and its taken me a few days to put a finger on what bothers me about it. In my opinion, this shouldn't be a pep describing the list of modules that need to go as "dead batteries", but should be a process pep describing how dead batteries should be removed, and the individual modules should be given their own pep. I think reactions to individual module peps will give a better indication of if it's a used module or not. -Original Message- From: Python-Dev On Behalf Of Christian Heimes Sent: Monday, May 20, 2019 4:15 PM To: Python Dev Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library Hi, here is the first version of my PEP 594 to deprecate and eventually remove modules from the standard library. The PEP started last year with talk during the Python Language Summit 2018, https://lwn.net/Articles/755229/. The PEP can be confirmed in two stages. I'm not planning any code changes for 3.8. Instead I only like to document a bunch of modules as deprecated. Active deprecation is planned for 3.9 and removal for 3.10. The long deprecation phase gives us 3 years to change our minds or handle edge cases, too. Regards, Christian PEP: 594 Title: Removing dead batteries from the standard library Author: Christian Heimes Status: Active Type: Process Content-Type: text/x-rst Created: 20-May-2019 Post-History: Abstract This PEP proposed a list of standard library modules to be removed from the standard library. The modules are mostly historic data formats and APIs that have been superseded a long time ago, e.g. Mac OS 9 and Commodore. Rationale = Back in the early days of Python, the interpreter came with a large set of useful modules. This was often refrained to as "batteries included" philosophy and was one of the corner stones to Python's success story. Users didn't have to figure out how to download and install separate packages in order to write a simple web server or parse email. Times have changed. The introduction of the cheese shop (PyPI), setuptools, and later pip, it became simple and straight forward to download and install packages. Nowadays Python has a rich and vibrant ecosystem of third party packages. It's pretty much standard to either install packages from PyPI or use one of the many Python or Linux distributions. On the other hand, Python's standard library is piling up cruft, unnecessary duplication of functionality, and dispensable features. This is undesirable for several reasons. * Any additional module increases the maintenance cost for the Python core development team. The team has limited resources, reduced maintenance cost frees development time for other improvements. * Modules in the standard library are generally favored and seen as the de-facto solution for a problem. A majority of users only pick 3rd party modules to replace a stdlib module, when they have a compelling reason, e.g. lxml instead of `xml`. The removal of an unmaintained stdlib module increases the chances of a community contributed module to become widely used. * A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size. The modules in the PEP have been selected for deprecation because their removal is either least controversial or most beneficial. For example least controversial are 30 years old multimedia formats like ``sunau`` audio format, which was used on SPARC and NeXT workstations in the late 1980ties. The ``crypt`` module has fundamental flaws that are better solved outside the standard library. This PEP also designates some modules as not scheduled for removal. Some modules have been deprecated for several releases or seem unnecessary at first glance. However it is beneficial to keep the modules in the standard library, mostly for environments where installing a package from PyPI is not an option. This can be cooperate environments or class rooms where external code is not permitted without legal approval. * The usage of FTP is declining, but some files are still provided over the FTP protocol or hosters offer FTP to upload content. Therefore ``ftplib`` is going to stay. * The ``optparse`` and ``getopt`` module are widely used. They are mature modules with very low maintenance overhead. * According to David Beazley [5]_ the ``wave`` module is easy to teach to kids and can make crazy sounds. Making a computer generate crazy sounds is powerful and highly motivating exercise for a 9yo aspiring developer. It's a fun battery to keep. Deprecation schedule 3.8 --- This PEP targets Python 3.8.
Re: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug.
> > It's relatively easy to test replacing our custom allocators with the > system ones, yes? Can we try those to see whether they have the same > characteristic? > Yes. PYTHONMALLOC=malloc LD_PRELOAD=/path/to/jemalloc.so python script.py I will try it tomorrow. > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 558: Defined semantics for locals()
On 23May2019 0636, Nick Coghlan wrote: However, I think the PR does show that the proposed technique can be implemented without too much additional code complexity, and will hopefully be adaptable for all implementations that emulate the frame API at all. Much excitement! One of the things I like best about this change is that it actually makes it *easier* for alternative implementations to use a simpler frame object without having to emulate CPython semantics (I'd love to get to a place where bug-for-bug compatibility wasn't required, but this is where we are right now so *shrug*). Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug.
On 23May2019 0542, Inada Naoki wrote: 1. perf shows 95% of CPU time is eaten by _PyObject_Free, not kernel space. 2. This loop is cleary hot: https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L1816-L1819 I can attach the process by gdb and I confirmed many arenas have same nfreepools. It's relatively easy to test replacing our custom allocators with the system ones, yes? Can we try those to see whether they have the same characteristic? Given the relative amount of investment over the last 19 years [1], I wouldn't be surprised if most system ones are at least as good for our needs now. Certainly Windows HeapAlloc has had serious improvements in that time to help with fragmentation and small allocations. Cheers, Steve [1]: https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L769 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 558: Defined semantics for locals()
On Wed, 22 May 2019 at 00:51, Nick Coghlan wrote: > P.S. I'm away this weekend, so I expect the reference implementation > to be done late next week, and I'd be submitting the PEP to Nathaniel > for formal pronouncement at that point. However, I'm posting this > thread now so that there's more time for discussion prior to the 3.8b1 > deadline. I found some time the other day to finish up the core of the reference implementation and resolve some bugs in the new test cases, so the essential changes can now be seen in https://github.com/python/cpython/pull/3640/files There are still some code level TODO items to cover off a few minor points in the PEP, as well as to make the fast locals proxy a better behaved mutable mapping. The proposed documentation changes also still need to be brought over from the PEP, and at least a quick scan done through the code comments and the rest of the documentation for now outdated references to the legacy trace function namespace handling. However, I think the PR does show that the proposed technique can be implemented without too much additional code complexity, and will hopefully be adaptable for all implementations that emulate the frame API at all. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 594: update 1
On Thu, 23 May 2019 at 14:25, Anders Munch wrote: > A major version change serves as a heads up that something is going on and > you need to check the consequences before upgrading. Python's backward compatibility policy allows breaking changes between versions X.Y and X.Y+1 (with a suitable deprecation period). This proposal is no different. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 594: update 1
Fra: Python-Dev [mailto:python-dev-bounces+ajm=flonidan...@python.org] På vegne af Terry Reedy: >> Deprecation schedule >> > > I think it worth adding that some modules were deprecated years ago but kept > until after 2.7 end-of-life to ease 3.x transition. Removing modules is not > a sudden new idea. > > Also, the proposed schedule (with one exception) is a slower alternative than > proposals to add a deprecation warning (if not one already) in 3.8 and remove > in 3.9. About that schedule: I think the removal version should be called 4.0. These are working modules that to varying degrees are still in use that are being removed. It's perfectly legitimate for python-dev to decide to stop maintaining them, but it's a breaking change none the less. For me, it's a couple of CGI scripts that will break. If I hadn't read this thread, I would at some point be confronted with a ModuleNotFoundError exception, with no guidance on how to fix that. That's not a very good user experience, even if a fix can be found after some googling. A major version change serves as a heads up that something is going on and you need to check the consequences before upgrading. regards, Anders ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug.
1. perf shows 95% of CPU time is eaten by _PyObject_Free, not kernel space. 2. This loop is cleary hot: https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L1816-L1819 I can attach the process by gdb and I confirmed many arenas have same nfreepools. I believe this is not jitter caused from NUMA or something else in cloud. -- Inada Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug.
On Thu, May 23, 2019 at 11:52 AM Inada Naoki wrote: > I have only 32GB mem, but AWS has larger memory machine! > Be aware that on larger cloud vendor instances, the backing vCPUs and memory you get allocated might or might not be spread opaquely across multiple sockets and/or NUMA nodes of the backing hardware. Some of them have options where you can allocate raw hardware as well so you can opt to lock your process to run within just one NUMA node and ensure hardware locality for your performance debugging. I'm pointing this out in case you experience any mystery jitters in your benchmark results. -- Joni Orponen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug.
I have only 32GB mem, but AWS has larger memory machine! Linux perf shows here is bottleneck: https://github.com/python/cpython/blob/master/Objects/obmalloc.c#L1784-L1819 obmalloc sorts arenas by number of free pools. If there are many arenas, and memory block is freed by random order, this sorting become O(N^2). That's too bad. I'm trying address order instead. Regards, -- Inada Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python in next Windows 10 update
On Thu, 23 May 2019 00:23:39 +0200 Ray Donnelly wrote: > On Thu, May 23, 2019, 12:17 AM Ivan Pozdeev via Python-Dev < > python-dev@python.org> wrote: > > > On 22.05.2019 23:52, Steve Dower wrote: > > > On 22May2019 1309, Ivan Pozdeev via Python-Dev wrote: > > >> As someone whose job is to diagnose and fix problems with running > > software: > > >> Are there patches in your release? Do you provide corresponding sources > > and debug symbols for it? > > > > > > You can find the sources at https://github.com/python/cpython :) > > > > > For Anaconda, this is not so, they apply private patches. So I had to make > > sure. > > > > There's nothing 'private' about them. > https://github.com/AnacondaRecipes/python-feedstock/tree/master/recipe Note that some of those patches could probably be contributed back. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library
I've watched the entire thread and its taken me a few days to put a finger on what bothers me about it. In my opinion, this shouldn't be a pep describing the list of modules that need to go as "dead batteries", but should be a process pep describing how dead batteries should be removed, and the individual modules should be given their own pep. I think reactions to individual module peps will give a better indication of if it's a used module or not. > -Original Message- > From: Python-Dev list=sdamon@python.org> On Behalf Of Christian Heimes > Sent: Monday, May 20, 2019 4:15 PM > To: Python Dev > Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard > library > > Hi, > > here is the first version of my PEP 594 to deprecate and eventually remove > modules from the standard library. The PEP started last year with talk during > the Python Language Summit 2018, https://lwn.net/Articles/755229/. > > The PEP can be confirmed in two stages. I'm not planning any code changes > for 3.8. Instead I only like to document a bunch of modules as deprecated. > Active deprecation is planned for 3.9 and removal for 3.10. The long > deprecation phase gives us 3 years to change our minds or handle edge > cases, too. > > Regards, > Christian > > > > PEP: 594 > Title: Removing dead batteries from the standard library > Author: Christian Heimes > Status: Active > Type: Process > Content-Type: text/x-rst > Created: 20-May-2019 > Post-History: > > > Abstract > > > This PEP proposed a list of standard library modules to be removed from the > standard library. The modules are mostly historic data formats and APIs that > have been superseded a long time ago, e.g. Mac OS 9 and Commodore. > > > Rationale > = > > Back in the early days of Python, the interpreter came with a large set of > useful modules. This was often refrained to as "batteries included" > philosophy and was one of the corner stones to Python's success story. > Users didn't have to figure out how to download and install separate > packages in order to write a simple web server or parse email. > > Times have changed. The introduction of the cheese shop (PyPI), setuptools, > and later pip, it became simple and straight forward to download and install > packages. Nowadays Python has a rich and vibrant ecosystem of third party > packages. It's pretty much standard to either install packages from PyPI or > use one of the many Python or Linux distributions. > > On the other hand, Python's standard library is piling up cruft, unnecessary > duplication of functionality, and dispensable features. This is undesirable > for several reasons. > > * Any additional module increases the maintenance cost for the Python core > development team. The team has limited resources, reduced maintenance > cost > frees development time for other improvements. > * Modules in the standard library are generally favored and seen as the > de-facto solution for a problem. A majority of users only pick 3rd party > modules to replace a stdlib module, when they have a compelling reason, > e.g. > lxml instead of `xml`. The removal of an unmaintained stdlib module > increases the chances of a community contributed module to become > widely > used. > * A lean and mean standard library benefits platforms with limited resources > like devices with just a few hundred kilobyte of storage (e.g. BBC > Micro:bit). Python on mobile platforms like BeeWare or WebAssembly > (e.g. pyodide) also benefit from reduced download size. > > The modules in the PEP have been selected for deprecation because their > removal is either least controversial or most beneficial. For example > least controversial are 30 years old multimedia formats like ``sunau`` > audio format, which was used on SPARC and NeXT workstations in the late > 1980ties. The ``crypt`` module has fundamental flaws that are better solved > outside the standard library. > > This PEP also designates some modules as not scheduled for removal. Some > modules have been deprecated for several releases or seem unnecessary at > first glance. However it is beneficial to keep the modules in the standard > library, mostly for environments where installing a package from PyPI is not > an option. This can be cooperate environments or class rooms where > external > code is not permitted without legal approval. > > * The usage of FTP is declining, but some files are still provided over > the FTP protocol or hosters offer FTP to upload content. Therefore > ``ftplib`` is going to stay. > * The ``optparse`` and ``getopt`` module are widely used. They are mature > modules with very low maintenance overhead. > * According to David Beazley [5]_ the ``wave`` module is easy to teach to > kids and can make crazy sounds. Making a computer generate crazy sounds > is > powerful and highly motivating exercise for
