[Distutils]Re: Introducing XAR - SquashFS based mountable executables - Calling OS/Distro Maintainers

[Barry Warsaw]

>> * Is there any practical operational or performance limits on the number of 
>> mounted XARs you can have?  E.g. what's the impact of deploying say a few 
>> thousand XARs on any particular machine (generally, of Linux and macOS - I'm 
>> not as concerned about Windows :)
> We have tiers (think of that as a collection of servers for a particular 
> service) that have 100s mounted at times. Have not hit 1000s - But have not 
> seen any huge problems with 100s. What do your servers do that would need 
> 1000s of XARs running at once? Seems like that could be optimized.

I’m not thinking about just our server deployed applications, but our command 
line tools too.  Combined, that’s roughly in the ballpark of 1000 or so Python 
applications (more if we count other languages).

> 
>> 
>> * Do you just put the XAR mountpoint's bin on everyone's $PATH or do you 
>> symlink those bins into a standard $PATH location?
> Standard PATH location via RPM installs.

Cool, thanks!

-Barry



signature.asc
Description: Message signed with OpenPGP
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/MKJYW7PI42BW5E2UYYP6Y4YS366KSJTP/

[Distutils]Re: Introducing XAR - SquashFS based mountable executables - Calling OS/Distro Maintainers

[Barry Warsaw]

On Jul 16, 2018, at 18:14, Nick Terrell  wrote:
> 
> I collected the XAR benchmark numbers. I spent some time today
> investigating what
> exactly is causing the difference between native and XAR start times. The 
> native
> installation I was benchmarking against used 
> `pkg_resources.load_entry_point()`
> to run the script, while XAR called the entry point directly.

> black: 0.171 s (vs 0.208 for XAR)
> jupyter-nbextension: 0.165 s (vs 0.179 s for XAR)
> 
> Without the pkg_resources difference we have a small overhead over
> native installs.

Thanks Nick, that definitely jives with our analysis.  pkg_resources can be a 
hidden source of significant overhead, imposed at import time.  That’s the main 
reason why importlib.resources was born.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/WSKHUA75PKFKF3VKNU6ZF6OFLHR4FML2/

[Distutils]Re: Introducing XAR - SquashFS based mountable executables - Calling OS/Distro Maintainers

[Barry Warsaw]

Cooper Ry Lees wrote on 7/13/18 13:51:


Today Facebook Open Sourced a competitor to PEX and other zip based 
distribution methods for Python (and potentially other languages). Basically 
it's claim to fame is start up time for large modules being similar to regular 
on file system modules due to extracting on read via SquashFS mounted 
executables. For more information read our blog post:
- 
https://code.fb.com/data-infrastructure/xars-a-more-efficient-open-source-system-for-self-contained-executables/
 



This is really interesting!  Thanks for releasing it and announcing it. 
I should mention that LinkedIn (my day job) recently released 'shiv' as 
a more modern, and in our case much faster, zipapp container similar to pex.


http://shiv.readthedocs.io/en/latest/

I have some quick questions about XAR:

* How do you achieve faster hot start times with XAR over native file 
system?  That's a bit unexpected, although based on our shiv work, I can 
imagine some things about how you start Python or lay out the code that 
might provide better hot start times (e.g. fewer entries on sys.path and 
a fanatical avoidance of pkg_resources).  OTOH, I'd think that relying 
on FUSE would impose some additional overhead over native file systems.


* Is there any practical operational or performance limits on the number 
of mounted XARs you can have?  E.g. what's the impact of deploying say a 
few thousand XARs on any particular machine (generally, of Linux and 
macOS - I'm not as concerned about Windows :)


* Do you just put the XAR mountpoint's bin on everyone's $PATH or do you 
symlink those bins into a standard $PATH location?


Cheers,
-Barry
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/7CD3HDYFNIAFWSD4FSSTJQRNGJ7M3TDP/

Re: [Distutils] Renaming this list to python-packaging

[Barry Warsaw]

Donald Stufft wrote:

I don’t think we can rename a mailing list easily, we can create a new one and 
archive the old one but that has a lot of churn and breaks people’s filters, 
etc.


Although it should be easier to rename if the list is migrated to 
Mailman 3 first.  Please contact postmaster@p.o if you want to go that 
route.


-Barry


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating/Removing OpenID/Google login support for PyPI

[Barry Warsaw]

Donald Stufft wrote:
> 
> * Very few people actually are using OpenID or Google logins as it is. In one 
> month we had ~15k logins using the web form, ~5k using basic auth, and 62 
> using Google and 7 using OpenID. This is a several orders of magnitude 
> difference.
> * Regardless of how you log into PyPI (Password or Google/OpenID) you’re 
> required to have a password added to your account to actually upload anything 
> to PyPI. This negates much of the benefit of a federated authentication for 
> PyPI as it stands.
> * Keeping these requires ongoing maintenance to deal with any changes in the 
> specification or to update as Google deprecates/changes things.
> * Adding support for them to Warehouse requires additional work that could 
> better be used elsewhere, where it would have a higher impact.

I'm one of those 7, but I really can't argue for you to keep supporting
it just for *me* :).  Have you considered allowing developers to use
their GitHub, GitLab, Bitbucket logins?  Those three probably cover a
large majority of package authors on PyPI.  I don't know how hard that
would be to support though.

-Barry


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] library development patterns

[Barry Warsaw]

Nick Coghlan wrote:
> The tox model is the one we decided to natively support in Fedora as
> well - while there's only ever one "full" Python 3 stack in the main
> repos (with all the distro API bindings, etc), there are also
> interpreter-only packages for other still supported and/or still
> popular Python X.Y branches, and "dnf install tox" will bring in all
> of them as weak dependencies.
> 
> Hence my preference for where I think it would make sense to take
> pipenv in this regard: better *enable* the tox model, without
> *duplicating* the tox model.

I'm a big fan of the tox model.  It works great on Debian/Ubuntu where
you can have multiple Python 3 interpreters (with some shared
infrastructure) during transitions, and macOS development where you
might have multiple versions of Python installed from brew/fink/macports
and from-source installations, including the current Python development
versions.  It also works well for things like
https://gitlab.com/python-devs/ci-images/tree/master

tox provides a nice, easy to invoke and remember CLI, good separation of
concerns (e.g. runtime deps in setup.py, test deps in tox.ini), and
convenient management of venvs.

Cheers,
-Barry


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] library development patterns

[Barry Warsaw]

Chris Withers wrote:
> For me, I use travis-ci coupled with a few local virtualenvs for canary
> versions. Some people like tox here, I never got on with it.

For me, tox is transformative.  While there are a couple of usability
issues that my clone army seems to be remiss in fixing, for the most
part tox is awesome.  In fact, I usually don't put any of my test
dependencies in setup.py any more; these all go in my tox.ini.  It
integrates well with flake8, and lets me definite all the various
testing matrixes I care about (e.g. all the Python versions w/ and w/o
coverage, "qa" environments for flake8, and a "docs" environment for
ensuring the docs build (and building them as a nice side benefit).  tox
of course manages all the venvs nicely, so it's easy to drop into them
when necessary.

tox doesn't yet play as nicely with flit/pyproject.toml, and I wish tox
supported environment groups and null generated environments, but I've
mostly found ways around those minor problems.

> Then it's "what testrunner do you use?", I'm gradually moving to pytest,
> but I still have a lot of nose 1.

I have a ton of nose2, which I really like, but I'm gradually moving to
pytest too.  I'm getting over my cosmetic complaints about pytest, since
it really is a very robust framework and does seem to have the most
momentum.

> As far as build and deployment, again, travis's tag-based deployment
> model that pushes artifacts to pypi, coupled with readthedocs
> pull-and-publish works for the things I care about. Then I guess you
> could talk about issue trackers and, indeed, community discussion
> channels ;-)

Yep, RTD is pretty darn awesome; I'm a fan of GitLab for various reasons
I suppose is off topic for this thread.

> I wonder how much of this makes sense to put in a how-to for library
> developers and where it branches out into areas where there are multiple
> legitimate choices?

Variation and competition is good; it keeps pushing the state of the art
even higher.  Some day we'll all be amazing that we used setup.py files
for a couple of decades in Python's early history. .  I think that
means it's worth at least describing in blog posts and such, but there's
not going to be a once-size-fits-all solution.  I do love hearing about
how others do it, because it gives me the ootz I need to try other
alternatives I might like better.

-Barry


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 345, 566, 508 version specifiers and OR clauses

[Barry Warsaw]

On Dec 13, 2017, at 19:39, Dustin Ingram  wrote:
> 
> It seems like a small amount of convenience in exchange for a
> significant increase in complexity to me.

I can’t speak to the complexity, but it’s very definitely an important use 
case.  Imagine when Python 3.10 is out; do you really want to have to write:

requires-python = '>=2.7,!=3.0,!=3.1,!=3.2,!=3.3,!=3.4,!=3.5,!=3.6,!=3.7’

and so on?  Plus, I don’t think it’s uncommon to have disjointed version 
support like this (well, maybe until 2020 when Python 2 is dead and gone 
).  And you have to get requires-python right, otherwise tools like tox 
can’t do the right thing when they install your library into its virtual 
environments.

> FYI, some previous discussion on the topic can be found here:

Thanks for the links.

And thanks Donald for your vote of support.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PEP 345, 566, 508 version specifiers and OR clauses

[Barry Warsaw]

I'm about to release a new version of importlib_resources, so I want to
get my flit.ini's require-python clause right.  We support Python 2.7,
and 3.4 and beyond.  This makes me sad:

requires-python = '>=2.7,!=3.0,!=3.1,!=3.2,!=3.3'

Of course, I'd like to write this like:

requires-python = '(>=2.7 and <3) or >= 3.4'

I understand that OR clauses aren't supported under any syntax
currently, but as PEPs 566 and 508 are still open/active, wouldn't it be
reasonable to support something like this explicitly?

It seems like wanting to support 2.7 and some versions of Python 3 (but
not all) is a fairly common need.

Cheers,
-Barry

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Multiple package authors

[Barry Warsaw]

I think I implicitly knew this, but as I've just released a package (to
be announced soon) that actually has multiple authors, I found out first
hand that PyPI rejects uploads where the author-email field isn't a
completely valid email address, and that there is no support for
multiple author emails.

As it turns out, you can kludge this into your pyproject.toml or
setup.py file.  flit for example separates multiple emails with a
newline, but you could also separate them with commas.  You don't notice
the problem until PyPI rejects the upload (with a 400 IIRC).

I filed this issue with flit: https://github.com/takluyver/flit/issues/153

It looks like Thomas agrees that at least flit will eventually validate
its fields so you error early.  It was a bit of a PITA to do my upload
because I didn't notice the problem until after I'd tagged the repo.

Multiple package authors doesn't seem like that fringe of a use case;
are there any plans, documents, PEPs, musings, grumbles about supporting
multiple package authors explicitly?

Cheers,
-Barry

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Provide separate development and documentation URLs in PyPI metadata?

[Barry Warsaw]

On Jun 25, 2017, at 11:33 AM, Nick Coghlan wrote:

>I'd favour "Participate" over any variant of "Contribute", as without
>context, "Contribute" makes me think of financial support in the
>crowdfunding/tip jar sense.

"Participate" may mean two different things.

* Here's the development home, with a repo and issue tracker, contributions
  welcome!

* Here's the mailing list or other forum where we discuss the future of
  Guido's Magical Mystery Time Machine.

It would be nice to be able to capture both meanings.

-Barry


pgpdT1COV2v52.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Provide separate development and documentation URLs in PyPI metadata?

[Barry Warsaw]

On Jun 24, 2017, at 05:34 PM, Brett Cannon wrote:

>So my question/idea is if it would make sense to have separate, explicit
>development and documentation URLs in the PyPI metadata?

Wholehearted +1

I already use PyPI as the first stop for finding out about a library or Python
project.  I have a nice Chrome search shortcut to do a search, and once I've
found the PyPI page I have a much greater chance of finding a link to
upstream.  Regardless of whether that points to the development site or the
documentation site, I can usually find a way back to the other one.

Still, I'd love for it to be much more explicit.

I also think pythonhosted should be deprecated if it's not already.  It's kind
of more of a pain than useful these days.

Cheers,
-Barry


pgpapLXPVEy56.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] RFC: PEP 541 - Package Index Name Retention

[Barry Warsaw]

On Jan 13, 2017, at 10:08 AM, Lukasz Langa wrote:

>PEP: 541
>Title: Package Index Name Retention

Overall, +1

I agree with Steve that some short term name squatting may be appropriate.
I'm not sure how you would word that in the PEP, but it will probably
effectively work itself out anyway.  When I come up with a name for a new
project, one of the first things I do is check PyPI and reserve it before the
first upload.  But that first upload will usually happen pretty quickly.

>Name conflict resolution for active projects
>
>
>The maintainers of the Package Index are not arbiters in disputes
>around *active* projects.  There are many possible scenarios here,
>a non-exclusive list describing some real-world examples is presented
>below. None of the following qualify for package name ownership
>transfer:

This is another opportunity to encourage cooperation among the parties to
resolve naming conflicts.  I think that's what's called for by the CoC, and
the hope is that the vast majority of disputes (in practice which I think are
pretty rare anyway) can be amicably resolved.

It does bring to mind the ability or lack thereof for renaming projects.  IIRC
that's not possible in current PyPI software but it may be possible in
Warehouse.

As for trademark owners having priority over names, I suppose there's no way
to prevent that, I'm mostly glad that this PEP doesn't speak to that.   I'm
always edgy about the imbalance of power inherent in the issue, allowing a
trademark owner to abuse their power to take over long-established names.
OTOH, trademark owners do often have legitimate concerns against squatters.
So, I say leave all that out of the PEP.

Cheers,
-Barry


pgpQu7rVwaCrW.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: TLSv1.2 will become mandatory in the future

[Barry Warsaw]

On Jan 10, 2017, at 08:24 AM, Donald Stufft wrote:

>python3 -c "import urllib.request,json; 
> print(json.loads(urllib.request.urlopen('https://www.howsmyssl.com/a/check').read())['tls_version'])"

For Python 3.5:

python3 -c "import urllib.request,json; 
print(json.loads(urllib.request.urlopen('https://www.howsmyssl.com/a/check').read().decode('utf-8'))['tls_version'])"

Cheers,
-Barry


pgpGqT0dWZFyc.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Barry Warsaw]

On Dec 01, 2016, at 10:45 AM, Freddy Rietdijk wrote:

>Having a compatible set of packages is not only interesting for developers,
>but also for downstream distributions. All distributions try to find a set
>of packages that are working together and release them. This is a lot of
>work, and I think it would be in everyone's benefit if we try to solve this
>issue together.

It's an interesting but difficult problem at the level of PyPI.

It's difficult because the definition of compatibility is highly dependent on
the consumer's environment.  For example, C extension compatibility will
depend on the version of libraries available on the platform versions you care
about.  There are also dependents on Python libraries that you can't capture
only in PyPI, e.g. some distro-only package may depend on a particular Python
package API.  PyPI can't test any of this.  And some distros (e.g. Ubuntu) may
have multiple dimensions of consumability, e.g. the classic apt packages
vs. snaps, primary devel archive vs. stable distro versions vs. backports, etc.

Ubuntu has an elaborate automated system for testing some dimension of
compatibility issues between packages, not just Python packages.  Debian has
the same system but isn't gated on the results.  Individual distro packages
can include a set of tests that are run against the built version of the
package (as opposed to tests run at package build time).  When a new version
of that package is uploaded it ends up in a "proposed" pocket that is
generally not installed by users.  The proposed new version has its tests run,
and any package that depends on that package also has its tests run.  Only
when all those tests pass is the package automatically promoted to the release
pocket, and thus is installable by the general user population.  This does a
great job within the context of the distro of raising the quality of the
archive because obvious incompatibilities (i.e. those for which tests exist)
block such promotion.

(FWIW, the difference is that Debian doesn't block promotion of packages
failing their automated tests, so 1) it provides less value to Debian; 2) we
end up inheriting and seeing these problems in Ubuntu and so have to expend
downstream effort to fix the failures.)

All of this happens automatically within the context of the distro, on
multiple architectures, so it provides a lot of value, but I'm not sure how
useful it would be higher up the food chain, since those contexts will be
different enough to cause both false positives and false negatives.  And it
does often take quite a bit of focused engineering effort to monitor packages
which don't promote (something we want to automate), to actually fix the
problems wherever is most appropriate (and as far upstream as possible), and
to create meaningful tests of compatibility in the first place (except for
default tests such as installability).

Still, there may be value in inter-Python package compatibility tests, but
it'll take serious engineering effort (i.e. $ and time), ongoing maintenance,
ongoing effort to fix problems, and tooling to gate installability of failing
packages (with overrides for downstreams which don't care or already expend
such effort).

Cheers,
-Barry


pgp1avNbf02_T.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] continuous integration options (was Re: Travis-CI is not open source, except in fact it *is* open source)

[Barry Warsaw]

On Nov 03, 2016, at 11:08 AM, Glyph Lefkowitz wrote:

>I think phrasing this in terms of "perfect" and "good enough" presents a
>highly misleading framing.  Examined in this fashion, of course we may
>reluctantly use the "good enough" option, but don't we want the best option?

What are the criteria for "best"?

I'm not saying don't use Travis, I'm just trying to express that there are
technical limitations, which is almost definitely true of any CI
infrastructure.  If those limitations don't affect your project, great, go for
it!

Cheers,
-Barry


pgpnNUUCsLakA.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Travis-CI is not open source. Was: Current Python packaging status (from my point of view)

[Barry Warsaw]

On Nov 03, 2016, at 12:54 AM, Nick Coghlan wrote:

>This is also an area where I'm fine with recommending freemium
>solutions if they're the lowest barrier to entry option for new users,
>and "Use GitHub + Travis CI" qualifies on that front.

I won't rehash the GitHub/GitLab debate, but in some of my projects (hosted on
GH) I've had to ditch Travis because of limitations on that platform.
Specifically, I needed to run various tests on an exact specification of
various Ubuntu platforms, e.g. does X run on an up-to-date Ubuntu Y.Z?

I originally used Docker for this, but our projects had additional
constraints, such as needing to bind-mount, which aren't supported on the
Travis+Docker platform.  So we ended up ditching the Travis integration and
hooking our test suite into the Ubuntu autopkgtest system (which is nearly
identical to the Debian autopkgtest system but runs on Ubuntu infrastructure).

Python may not be affected by similar constraints, but it is worth keeping in
mind.  Travis isn't a perfect technical solution for all projects, but it may
be good enough for Python.

Cheers,
-Barry


pgpOSaL3aAK0M.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] When can we kill Python 2.6 support?

[Barry Warsaw]

On Sep 02, 2016, at 05:05 PM, Donald Stufft wrote:

>Do we think that a ~3% usage of Python 2.6 and being end-of-life'd for ~3
>years is enough to start deprecating and dropping 2.6?

Yes!

Cheers,
-Barry


pgpCalDLuc3vp.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] greater compression on pypi?

[Barry Warsaw]

On Aug 22, 2016, at 02:29 PM, Daniel Holth wrote:

>1. If it's a zip, nested zips like so,
>
>setup.py
>README
>(metadata)
>data.zip

That would be much more disruptive to downstream consumers of sdists.

Cheers,
-Barry


pgpcWwl_cnoP1.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Barry Warsaw]

On Aug 22, 2016, at 09:14 PM, Nick Coghlan wrote:

>In that case, perhaps the way to go for sdists (at least for now)
>would be to continue allowing both .tar.gz and .zip, but disallow
>uploading both of them for any given release?

I'd prefer allowing uploading of both formats for now, with mild encouragement
to at least include zips.

I'm fairly certain the Debian Python ecosystem in general can consume zip
sdists without problem, although there may be some individual packages that
need some adjustment.  I don't have any sense of large an impact it would be
to switch sdists from tar.gz to zip, but allowing for some overlap with
"gentle nudges" toward the preferred format would allow us to do the analysis
and give us some time to plan a transition.

Personally, I don't much care if the sdist format is tar.gz or zip, and it
does make sense to promote one archive format for both source and binary
distributions.  I mildly prefer tar.gz but for pretty shallow reasons
(i.e. not good enough :): one tool to rule them all (tar(1) both packs and
unpacks, while zip needs two commands), and I've been using tar for way longer
than zip so I'm a little more comfortable with it.

What I'd want most of all is a clearly documented transition plan, i.e. a PEP
that I can point other people to and say "we need to make sure all our tooling
is in place by date X because that's when .tar.gz is going away".

Cheers,
-Barry


pgpM_wi_53rBa.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Barry Warsaw]

On Aug 17, 2016, at 03:21 PM, Nick Coghlan wrote:

>This means that the situation we've managed to get to now is that we
>have wheels as a satisfactory replacement for the "binary
>distribution" use case, but we haven't even *tried* to replace eggs
>for the "standalone path entry" use case.

I'm very sure I don't understand something, because I thought wheels were just
fine for the import-from-sys.path use case.  I mean, pip does this and in
Debian, we have a program (dirtbike) that turns installed package's file
system layouts back into wheels so they can be put on sys.path and imported
from.

Cheers,
-Barry


pgpYxVUnFBslR.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Barry Warsaw]

On Aug 15, 2016, at 05:39 PM, Donald Stufft wrote:

>so narrowing it down to only .tar.gz and .tgz is pretty safe, but practically
>nobody uses .tgz anyways so why bother?

I agree with all of your reasoning.  FWIW, I personally use .tgz all the time
when making backups of things locally, but rarely use it for publicly
consumable artifacts.  And since the sdists setupdistutilstools generates will
only be .tar.gz, I think it's fine to disallow .tgz.

Cheers,
-Barry


pgpdf_YU0aTD9.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Barry Warsaw]

On Aug 15, 2016, at 03:09 PM, Donald Stufft wrote:

>Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
>and bdist_dumb. I also believe that there is a strong case for removing
>bdist_msi and bdist_wininst. I also think we should consider removing
>bdist_egg.

+1 for all of these, or maybe with +0 for bdist_egg in the short term.  My
only concern is that the deprecation period has been long enough, but I do
agree that they should eventually be prohibited.

>On a related but different note, I'd also like to restrict the acceptable
>extensions for sdists. Currently distutils allows people to generate .tar,
>.tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly
>others. This is a bit problematic because each of those types requires a
>different set of optional libraries (which may or may not exist depending on
>Python version) so you can't really use a lot of them (for example, while
>.tar.xz might give you better compression, it's also not usable before Python
>3).
>
>Looking at numbers again, the current number of projects for each file ext
>are:
>
>* .tar.gz: 444,338
>* .zip: 58,774
>* .tar.bz2: 3,265
>* .tgz: 217
>* Everything Else: 0

I'm a little less certain about this.  It's probably okay, although since the
.tgz format is the same as .tar.gz, the tools argument doesn't hold for that
extension.  I know compression nerds like to debate the merits of gz, bz2, and
xz, but I doubt with the size of most of these files, it's going to make any
difference.

FWIW, restricting the extensions would simplify Debian's template for the
PyPI debian/watch file (e.g. the pattern we use to download packages):

https://pypi.debian.net/nose2/nose2-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

(Ignore the pypi.debian.net redirector.)

That would be a mild win.

Cheers,
-Barry


pgpW5bBvhyE9X.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Contributing money to package authors/maintainers via PyPI

[Barry Warsaw]

On Jul 26, 2016, at 01:24 PM, Nick Coghlan wrote:

>The PSF has considered this, but there's not a lot of value we could
>provide above and beyond other organisations that already do this for
>open source projects in general. For example:
>
>- Software Freedom Conservancy
>- Software in the Public Interest
>- Outercurve Foundation

The Free Software Foundation also runs a donation program, and GNU Mailman has
benefited from this to some extent.  They do take a healthy cut, which is okay
with us because it helps fund their own programs and administrative costs.

One thing to keep in mind is that not all developers can or want to accept
donations.  It can cause tax and employment headaches.  But in GNU Mailman's
case, having donations go to a project-centric fund has allowed us to help pay
for travel and Pycon attendance for our GSoC students, at least one of which
has since become a core developer.  At our size, that's been way more valuable
to the project than paying individual developers.

Cheers,
-Barry


pgpVhN0ltbX0I.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI and GPG Signatures

[Barry Warsaw]

On May 12, 2016, at 04:34 PM, Donald Stufft wrote:

>So my response to this is, let's pretend for a minute that we have the
>greatest and most amazing setup for verifying that the key 0x6E3CBCE93372DCFA
>belongs to me. What's your next step? How do you verify that I'm allowed to
>release for pip?

I'd hope that the project home page would say that.  I sheepishly admit that
we don't have that information on the Mailman home page, but you *could*
follow the link from me (described as the lead developer) to my own home page
and then grab the key from there, verified from keybase.io.

>What happens if tomorrow I decide I'm no longer going to use key
>0x6E3CBCE93372DCFA because it got compromised (remembering that key
>revocation is hilariously broken [1]).  What if we add a new signing key
>because I'm tired of releasing pip and someone else is going to take over,
>what path is Debian going to take for verifying that some new key is allowed
>to sign for it that doesn't put "Whatever PyPI says" in the path of trust?

uscan would complain and then I'd have to try to figure out the new signing
credentials.

It's not wonderful, but for platform and package maintainers who care, I think
it does provide value, and the signing credentials likely don't change that
often.

Cheers,
-Barry


pgp05KLOmMvV6.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI and GPG Signatures

[Barry Warsaw]

On May 12, 2016, at 07:41 AM, Donald Stufft wrote:

>I am aware of a single tool anywhere that actively supports verifying the
>signatures that people upload to PyPI, and that is Debian's uscan
>program. Even in that case the people writing the Debian watch file have to
>hardcode in a signing key into it and in my experience, when faced with a
>validation error it's not unusual for Debian to simply disable signature
>checking for that project and/or just blindly update the key to whatever the
>new key is.

I like that uscan provides this feature, but I don't know how many packages
actually use it, either within the Debian Python teams, or in the larger
Debian community.  I'd like to use it more often on packages I maintain but
it's kind of difficult to find your way back to an authoritative signing key.
For my own packages that I also maintain in Debian, it's of course trivial, so
I have that enabled for them.  I sign all my package uploads to PyPI, and I
mostly trust myself .

If it's possible to get signing keys from PyPI, I really have no idea how to
do that.  The web ui doesn't at all make it obvious (to me, at least).

I understand the implementation dilemma for Warehouse, but rather than ditch
this feature, I'd rather see it improve by making the signing keys more
discoverable and verifiable.  I wonder if keybase.io could be used somehow.
Or perhaps a prominent link in the package metadata pointing to a pubkey
location.  Then it would be up to projects to utilize these mechanisms to make
their signing keys obvious, and tools like uscan can increase their usage of
such features.

Cheers,
-Barry


pgp3Wicvdnew4.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Barry Warsaw]

On May 09, 2016, at 08:30 PM, Donald Stufft wrote:

>How hard is it to bundle it with pip by copying the source files into
>pip._vendor.*

Every time another package is vendored, a kitten falls off a unicorn. ;)

Cheers,
-Barry


pgpetwbrYXBeb.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Barry Warsaw]

On May 08, 2016, at 09:05 AM, Robert Collins wrote:

>E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name,
>as pybuild is a thing in the debian space, and this will confuse the
>heck out of folk). https://wiki.debian.org/Python/Pybuild

Yes, please don't call it pybuild ;)

Also, I think it makes a lot of sense to go with YAML even if it isn't the
best most readable option.  It's much more common than TOML so the learning
curve will be lessened.

Cheers,
-Barry


pgpcWJAr9wuKi.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wheel including files it shouldn't

[Barry Warsaw]

On Apr 27, 2016, at 10:00 PM, Alex Grönholm wrote:

>The sdist should include all the source files, including tests and
>documentation. In binary distributions, however, they are just dead
>weight. Do you want the full documentation and test suites to be installed
>for every single dependency when you deploy your application? I sure don't.

Shouldn't that be left to the package author to decide?

-Barry


pgpYtPzVV03fY.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Barry Warsaw]

On Feb 16, 2016, at 06:01 PM, Matthias Klose wrote:

>I know we had such an issue where pip accidentally modified system installed
>files, maybe Barry still remembers the details ...

I don't, but I hope that should not be a problem these days, with modern pip
on modern Debian/Ubuntu systems.  We prevent pip from installing in system
locations in various ways, including making --user the default.

I don't think we're doing anything that isn't at least within the pip
roadmap.  E.g. --user will probably someday be the default in stock pip, but
because of reasons it hasn't happened yet.

IIRC, Donald was also talking about better ways to detect system owned files,
so pip would also refuse to overwrite them.

Cheers,
-Barry


pgpvUxg7OmTnO.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

[Barry Warsaw]

On Feb 11, 2016, at 01:58 PM, Nick Coghlan wrote:

>Hmm, I got the py2dsc reference from https://wiki.debian.org/Python/Packaging
>but the newer https://wiki.debian.org/Python/LibraryStyleGuide doesn't appear
>to mention any particular way of generating the initial packaging skeleton
>from the upstream project.

I'm not sure what the state of py2dsc is, and I personally don't use it.
Mostly I cargo-cult other packages I know to be good ;).  I do think other
people still use py2dsc and python-stdeb though, and if Piotr is maintaining
it, it'll be good (he's also the maintainer of pybuild and dh_python{2,3}, the
recommended and most popular build tools for Python packages in Debian).

And while initial packaging is important, it's a comparatively rare event in
contrast to ongoing maintenance (e.g. updating to new upstreams).  py2dsc
won't help you there, but there are a stack of tools that do, some of which
are tied to various team workflows.  E.g. the LibraryStyleGuide you mention
(and the current git-based workflow[1]) are standards for the Debian Python
Modules Team (DPMT) which maintains the majority, but definitely not all,
Python packages in the archive.  Lots (even pure-Python ones) are maintained
elsewhere such as the OpenStack team, or just by individual maintainers using
whatever workflows they want.

Some maintainers want to do new upstream releases from (signed?) git tags but
the consensus for DPMT, and I think most Python package maintainers in Debian,
is to use a tarball-based workflow.

>Anyway, the core point is wanting to ensure we can automate not only
>"direct to binary" installation with Python specific tools, but also
>the "convert to alternate source archive format and build from there"
>workflows needed by redistributor ecosystems like Linux distros,
>conda, Canopy, PyPM, Nix, etc.

Cool.

Note that not even all Debian-based distros are equal here.  For example, in
Debian, especially for architecture independent (i.e. pure-Python) packages,
the source package to binary package step happens on the maintainer's local
system, while in Ubuntu we upload the source package and let the centrally
maintained build daemons produce the resulting binary packages.

Anyway...
-Barry


[1] https://wiki.debian.org/Python/GitPackaging


pgp_ySmkM4p1F.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

[Barry Warsaw]

On Feb 10, 2016, at 10:08 AM, Paul Moore wrote:

>But those people will then find that distributing their sources isn't
>something that flit covers, so they'll make up their own approach (if it were
>me, I'd probably just point people at the project's github account).
>
>Once people get set up with a workflow that goes like this (build
>wheels and point people to github for source) it'll be harder to
>encourage them later to switch *back* to a process of uploading
>sources to PyPI.
>
>And that I do think is bad - that we end up pushing people who would
>otherwise happily use PyPI for source and binary hosting, to end up
>with a solution where they host binaries only on PyPI and make the
>source available via another (non-standardised) means.

That worries me a lot.  Think of the downstream consumers who aren't end
users, e.g. Linux distro developers.  Some distros have strict requirements on
the availability of the source, reproducibility of builds, and so on, along
with stacks of tooling that are built on downloading tarballs from PyPI.

It's not impossible to migrate to something else, but it's impractical to
migrate to dozens of something elses.  Right now, if we can count on PyPI
having the source in an easily consumable lowest common denominator format,
the friction of providing those packages to *our* end users, and updating them
in a timely manner, is often minimal.  Changing that ecosystem upstream of us,
either deliberately or otherwise, will likely result in more out of date
packages in the distros.

Cheers,
-Barry


pgpI7uxU9PrYr.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Barry Warsaw]

On Nov 05, 2015, at 04:08 PM, Donald Stufft wrote:

>One benefit of the third option is that we can remove the need to directly
>copy the bundled libraries into the pip source code and we can install just
>bundle it inside the built zip file.

This shouldn't be a problem from Debian's p.o.v. if we can adjust how packages
end up in the zip file.  If for example the zip contains wheels, and we finish
the dirtbike (or similiar) project to rewheel distro packages, we'd use those
pip-build-time generated wheels in the zip and be all good.  Of course, we'd
probably have to rebuild pip when we change any of those dependencies, but I
think that's tractable.

I think the proposed solution is a better user experience that either of
pipX.Y or `pythonX.Y -m pip`, so I'm glad you're experimenting with this
approach.

Cheers,
-Barry


pgpGEI2LPVpSf.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Please don't impose additional barriers to participation (was: build system abstraction PEP)

[Barry Warsaw]

On Oct 29, 2015, at 10:52 PM, Marcus Smith wrote:

>> If python-dev ends up adopting GitLab for the main PEPs repo, then we
>> should be able to move the whole process there, rather than needing to
>> maintain a separate copy.
>>
>will that be as open as pypa/interoperability-peps?  if it's closed off such
>that only python devs can log PRs against PEPs once they're in the system,
>then that's no fun.  If so, I'd still want to keep pypa/interoperability-peps
>as the front end tool for the actual change management.

The way I believe it would work via GitLab is that anybody can fork the repo,
push branches to their fork, and file PRs against the PEPs repo.  Pushing to
the PEPs repo would be gated via privileged members of the repo's owner,
either via git push or button push (i.e. "hey website, I'm chillin' on the
beach with my tablet so do the merge for me!")

Cheers,
-Barry


pgpkmAGBbce9s.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Uploading stdeb built debs ?

[Barry Warsaw]

On Oct 10, 2015, at 06:48 AM, Stuart Axon via Distutils-SIG wrote:

>Hi,    I built a package of using stdeb, any idea if these are acceptable to
>debian + ubuntu, and where to start / who to contact to get a package
>included in those distros ? S++

You can contact debian-pyt...@lists.debian.org

Cheers,
-Barry


pgpRf0MdkwlID.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Barry Warsaw]

On Oct 07, 2015, at 09:46 AM, Ben Finney wrote:

>So “I'm a big fan of putting tests inside the [Python] package
>[directory]” can't be motivated by “Having the tests there in the
>installed package”. The two aren't related, AFAICT.

It makes it easier for sure.  When the tests are inside the package, nothing
special has to be done; you just install the package and the tests
subdirectories come along for the ride.  If the tests are outside the package
then you first have to figure out where they're going to go when they're
installed, and then do something special to get them there.

Cheers,
-Barry


pgpVco0pIiE1x.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Barry Warsaw]

On Oct 07, 2015, at 08:18 AM, Donald Stufft wrote:

>tox and setup.py test are not really equivalent. There’s no way (to my
>knowledge) to test the item outside of a virtual environment. This is
>important for downstreams who want to test that the package build and the
>tests successfully are executed in their environment, not within some virtual
>environment.

I usually do not use tox to test a package when building it for Debian.  It's
pretty easy to extract the actual command used to run the test suit from the
tox.ini and that's what I put in the debian/rules file.  It can make things
build a little more reliably, but also eliminates a build dependency on tox.

Cheers,
-Barry


pgpzLAJQoCUdy.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Barry Warsaw]

On Oct 07, 2015, at 08:35 AM, Marius Gedminas wrote:

>I have hopes for 'tox.ini' becoming the standard way to test a Python
>project.

As do I, modulo outliers of course.  I'd like to see 90% of PyPI packages have
a tox.ini.

Cheers,
-Barry


pgpxJ0ovIWbuL.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Barry Warsaw]

On Oct 06, 2015, at 05:54 AM, Donald Stufft wrote:

>I dislike putting tests inside the package.

I'm a big fan of putting the tests inside the package.  I've often looked at a
package's tests to get a better understanding of something that was unclear
for the documentation, or didn't work the way I expected.  Having the tests
there in the installed package makes it easier to refer to.  I also find that
with tox+nose2 (my preferred one-two punch for testing), it makes it quite
easy to find and run the full test suite or individual tests based on a regexp
pattern.  I also like the symmetry of having a docs/ directory for doctests
and a tests/ directory for unittests.

For complex packages with lots of subpackages, I have lots of tests/
directories, so that the unitests are near to the code they test.  This way
the source tree gets organized for free without additional complexity in an
outside-the-package tests tree.

YMMV,
-Barry


pgpOG4ufsOJOQ.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Barry Warsaw]

On Oct 06, 2015, at 06:21 AM, Donald Stufft wrote:

>FreeBSD relies on ``python setup.py test`` as it's preferred test invocation,
>so it apparently doesn't find it useful either.

Oh how I wish there was a standard way to *declare* how to run the test suite,
such that all our automated tools (or the humans :) didn't have to guess.

At least in Debuntu though, we can pretty much make any of the usual ways work
during package build.

Cheers,
-Barry


pgpyCxpDDqkWN.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Barry Warsaw]

On Oct 06, 2015, at 11:33 AM, David Cournapeau wrote:

>I would like to hear their rationale before guessing. It is hard for me to
>imagine they would not rather test the binaries rather than from sources.
>Something as simple as making sure you have not forgotten runtime
>dependencies becomes much easier this way.

In Debian we can do both.  It's usually good practice to run the package's
test suite during the build process, on the unbuilt source tree.  That doesn't
work for all packages though (tox comes to mind as a recent example), so we
*also* have a way to run the test suite on a built-and-installed version of
the Debian binary package.  I usually try to at least do an import test in
this phase, so for some like tox, I'll do a more extensive test.

In Ubuntu, failing the built-and-installed test (a.k.a. autopkgtest or DEP-8)
will prevent a package from getting promoted from the -proposed channel to the
release channel, which usually shields end users from seeing broken packages.
Debian doesn't have this gateway in place yet.  There is a way to run those on
a local test build, so that's pretty nice.

Cheers,
-Barry


pgpkayj8nhw7u.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Barry Warsaw]

On Oct 06, 2015, at 08:18 PM, Ben Finney wrote:

>Yes, this is a sensible approach:
>
>* The source package contains all the source files a developer would use
>  to make further changes and test them.
>
>* The package for installation contains only those files useful run-time
>  users, plus metadata (e.g. copyright information).

I generally don't both stripping out in-package tests when building binary
packages for Debian.  First because it's more work for (IMHO) dubious value
and second because I think installed test files are actually useful sometimes.

Cheers,
-Barry


pgpokHlqtXi9z.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Barry Warsaw]

On Oct 06, 2015, at 05:41 PM, Donald Stufft wrote:

>I’m not sure I understand what you’re advocating here, it sounds like you
>want your tests at something like mycoolproject/tests so that they are
>importable from mycoolproject.tests… but then you talk about symmetry with
>docs/ and tests/ which sounds more like you have top level directories for
>tests/ docs/ and then mycoolproject/.

Ah, sorry for being unclear.

I put tests in myproj/tests so yes they are importable via myproj.tests.  I
also put __init__.py's in my docs/ directory so the directory, but not the
individual .rst doc files are also importable via myproj.docs.  I do this
because of the handy nose2 plugin I cargo cult around that allows me to run
individual tests or doctests with a command line switch.

If I have subpackages, the pattern repeats, so e.g.

myproj/subpkg1/tests -> import myproj.subpkg1.tests
myproj/subpkg2/docs  -> import myproj.subpkg2.docs

Cheers,
-Barry


pgpBn0Vyt4Ejs.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Barry Warsaw]

On Oct 07, 2015, at 08:54 AM, Ben Finney wrote:

>Barry Warsaw <ba...@python.org> writes:
>
>> I'm a big fan of putting the tests inside the package. I've often
>> looked at a package's tests to get a better understanding of something
>> that was unclear for the documentation, or didn't work the way I
>> expected. Having the tests there in the installed package makes it
>> easier to refer to.
>
>That doesn't follow, or I'm not understanding you.
>
>If you have the tests in the source package, as is being advocated, you
>have the tests available for reference. So the *relative location* of
>the tests, within that source tree, doesn't argue for what you're
>saying.

Since I'm not sure I follow that, I'll answer the question you asked:

>Are you arguing the separate point of whether tests should be
>*installed* with the package?

Yes.  We've had this conversation before in the context of Debian package
sponsorship.  I know and respect that you disagree.

Cheers,
-Barry


pgpV4RTrP7GIM.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Barry Warsaw]

On Oct 06, 2015, at 10:44 AM, Antoine Pitrou wrote:

>Doesn't / didn't setuptools have something called test_requires?

Since I pretty much use tox everywhere these days, I've taken to defining test
requirements in my tox.ini rather than my setup.py.

Cheers,
-Barry


pgpDfbZCf3GwZ.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Barry Warsaw]

On Oct 07, 2015, at 08:51 AM, Ben Finney wrote:

>I think the above describes the standard way of declaring the test
>runner: The ‘setup.py test’ command.
>
>Now, I lament that more Python projects don't *conform to* that
>standard, but at least it exists.

It's *a* standard but not *the* standard, just from pure observation.

Cheers,
-Barry


pgp4AaoIRNNQa.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Barry Warsaw]

On Oct 06, 2015, at 09:51 AM, Antoine Pitrou wrote:

>The PyP"A" should definitely fix its sample project to reflect good
>practices.

Here's my own sample project.  There are actually two git branches, one with
an extension module and one with pure Python.

https://gitlab.com/warsaw/stupid

Cheers,
-Barry


pgpZbpXfL7zg2.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI and Uploading Documentation

[Barry Warsaw]

On May 15, 2015, at 09:48 AM, Donald Stufft wrote:

One of the things that this brought to light was that the documentation
upload ability in PyPI is something that is not often used* however it
represents something which is one of our slowest routes.

I use it for all my packages, mostly because it's easy for my upload
workflow: `python setup.py upload_docs`.

That said, with the rise of RTD, I have wondered about the usefulness of
pythonhosted documentation.   And because twine supports secure uploads of
code, but not documentation, that unease has grown.

So even while I use it, I agree it's time to consider retiring the service.

One thing I definitely want to retain though is the link to Package
Documentation from the project's PyPI page.  Please do give us a way to
specify that link.

The PSF is a supporter of RTD, but let's all make sure they stay in business!
https://readthedocs.org/sustainability/#about

Cheers,
-Barry


pgpE3C9e_rzyR.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Barry Warsaw]

On Apr 23, 2015, at 07:08 AM, Robert Collins wrote:

PEP-420 and legacy packages being mixed for one namespace doesn't work at all
today - in principle fixable via changes to both setuptools and importlib -
but it was about here that the other openstack folk said 'ok wow, lets just
stop using namespace packages :)

I think this is actually by design, despite what PEP 420 says.  If you have a
portion that contains an __init__.py, that basically overrides any namespace
portions found on sys.path.  It's all or nothing.

Now, were this comes up for me is in bilingual code.  I have some namespace
packages (e.g. flufl.*) which should live in the same parent package even if
the portions live in distinct directories.  Each package has an __init__.py
that stitches things together for Python 2 (using the various common hack
recipes), but of course installing this package in Python 3 means they aren't
namespace packages, and this makes me sad.

I wish there was some kind of exception or marker I could put in the
flufl/__init__.py files that signaled PEP 420-aware Python 3s to treat it as
if the __init__.py doesn't exist.

In the Debian ecosystem, we solve this with package builder help.  The
standard helpers will actually not include the top-level __init__.py file for
the Python 3 binary package version, so they'll be stitched-together
namespace-ish for Python 3 and straight up PEP 420 namespace packages for
Python 3.

Cheers,
-Barry


pgpoJv7iFNd3J.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Get dependencies of a package without full download

[Barry Warsaw]

On Apr 01, 2015, at 04:14 PM, Thomas Güttler wrote:

Is it possible to get the dependencies of a package without full download
from pypi?

It would be kind of nice if you could get the package's metadata (e.g
egg-info/entry_points.txt) out of its PyPI JSON blob:

https://pypi.python.org/pypi/flufl.i18n/json

Cheers,
-Barry


pgp5LjyrJtXLE.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] it's happened - wheels without sdists (flit)

[Barry Warsaw]

On Mar 30, 2015, at 11:56 AM, Donald Stufft wrote:

Honestly, I don’t think that setup.py as a development interface is that
bad.

Especially when you cargo cult most of a new project's basic infrastructure[*]
from one that's already working.  Sweat out the first one, then reuse. ;)

Cheers,
-Barry

[*] For me, not just setup.py but also tox.ini, coverage.ini, MANIFEST.in,
other helpers, readmes, license files, source templates, etc.


pgp6UWM2grPTR.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] it's happened - wheels without sdists (flit)

[Barry Warsaw]

On Mar 31, 2015, at 09:14 AM, Nick Coghlan wrote:

We need to build from source not just to ensure our binaries match the
published source code, but also because our build systems are designed to
let us *patch* the packages before we build them. This is what lets us
backport security updates, bug fixes, and sometimes even entire features
without needing to rebase a package on a new upstream release of a project.

For Debian, all this, plus it's required by the Debian Social Contract and
Debian Free Software Guidelines.

(AFAIK Debian still favours invoking setup.py directly, hopefully Barry
will correct me if I'm wrong).

Correct.  For any setup.py/distutils/setuptools-based project, we have one
preferred, recommended, and non-deprecated build system, and though it's
fairly flexible, it mostly relies on a working setup.py.  As Debian Jessie is
frozen right now, I don't expect that to change for now.

We have sort of a love-hate relationship with pip, but it would be interesting
to sit down at Pycon and discuss how we might like to see downstream distro
support improve or change as it relates to common upstream packaging
standards.

Cheers,
-Barry


pgp4mKza0LaGV.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Google Auth is broken for PyPI

[Barry Warsaw]

On Feb 10, 2015, at 02:27 PM, Brett Cannon wrote:

Does Launchpad support OpenID Connect? If so then migrating to that would
solve this.

No, I don't believe so.  I've just filed this bug:

https://bugs.launchpad.net/launchpad/+bug/1420348

Otherwise it may be time to have a serious look at our federated login
support and consider adding GitHub and Bitbucket for wider reach as OpenID
1.0/2.0 is extremely niche at this point.

I'd prefer not to *remove* OpenID support if it's still working, which it is.
And while I think GH and BB support might be useful, it's important to
remember that those are both closed systems, so I would put them on par with
Google, Facebook, or Yahoo! SSO.

Launchpad at least has the advantage of being free software, so anyone with
sufficient motivation and resources could contribution the needed support.

Cheers,
-Barry


pgpdeArtpYBY9.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Google Auth is broken for PyPI

[Barry Warsaw]

On Feb 08, 2015, at 11:37 PM, Richard Jones wrote:

Google has discontinued support for OpenID, so we're not going to be
putting any effort into debugging this issue.

I hope you'll continue to support other OpenID providers, e.g. Launchpad.

Cheers,
-Barry



pgpVJAH8ZQaH8.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Python module for use in ‘setup.py’ but not to install

[Barry Warsaw]

On Jan 19, 2015, at 07:23 PM, Ben Finney wrote:

My understanding, based on an answer received elsewhere [0], is that
omitting the file from ‘setup.py’ but adding it to ‘MANIFEST.in’ causes
it to be included in the sdist but omitted from install targets.

I haven't read the whole thread (no time right now) but this is a pretty
standard solution for packaging problems I've had with Debian.   If a file
(like MANIFEST.in itself) is not included in the sdist, then building the
binary package will often fail.  I forget the exact symptoms, but it's
probably easily reproducible if you care.

Cheers,
-Barry


pgpvbQ0uUeIuk.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Bilingual namespace package conundrum

[Barry Warsaw]

On Jan 13, 2015, at 02:39 PM, Robert Collins wrote:

Welcome to hell :)

Well, purgatory maybe.  I patched the lazr packages I cared about. :)

So I dug down ridiculously deep into this when investigating a very
similar issue in the openstack space. I have an alternative solution
to the ones mentioned already - use importlib2 and patch the importer
(e.g. in a local site.py) to allow genuine namespaces to work. (Its
nearly there - just needs a little more love I think).

I like the idea in theory, but I'm not sure how general of a solution this
will be.  Not everyone can or will want to install the custom importer just to
get real namespace packages in Python 2.

Cheers,
-Barry


pgpa5LfbzNW1u.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Bilingual namespace package conundrum

[Barry Warsaw]

I hope the following makes sense; I've been a little under the weather. ;)
Apologies in advance for the long data-dump, but I wanted to provide as
complete information as possible.

I have ported Mailman 3 to Python 3.4[*].  While working on various
development tasks, I noticed something rather strange regarding
(pseudo-)namespace packages which MM3 is dependent on.  These libraries work
for both Python 2 and 3, but of course namespace packages work completely
differently in those two versions (as of Python 3.3, i.e. PEP 420).  I've been
thinking about how such bilingual packages can play better with PEP 420 when
running under a compatible Python 3 version, and still *seem* to work well in
Python 2.  More on that later.

It turns out this isn't actually my problem.  Here's what's going wrong.

My py3 branch has a fairly straightforward tox.ini.  When I run `tox -e py34`
it builds the hidden .tox/py34 virtualenv, installs all the dependencies into
this, runs the test suite, and all is happy.  Since I do most of my
development this way, I never noticed any problems.

But then I wanted to work with the code in a slightly different way, so I
created a virtual environment, activated it, and ran `python setup.py develop`
which appeared to work fine.  The problem though is that some packages are not
importable by the venv's python.  Let's concentrate on three of these
dependencies, lazr.config, lazr.delegates, and lazr.smtptest.  All three give
ImportErrors when trying to import them from the interactive prompt of the
venv's python.

I tried something different to see if I could narrow down the problem, so I
created another venv, activated it, and ran `pip install lazr.config
lazr.delegates lazr.smtptest`.  Firing up the venv's interactive python, I can
now import all three of them just fine.

So this tells me that `python setup.py develop` is broken, or at the very
least behaves differently that the `pip install` route.  Actually, `python
setup.py install` exhibits the same problem.

Just to be sure Debian's Python 3.4 package isn't introducing any weirdnesses,
all of this work was done with an up-to-date build of Python from hg, using
the 3.4 branch, and the virtualenvs were created with `./python -m venv`

Let's look more closely at part of the site-packages directories of the two
virtualenvs.  The one that fails looks like this:

  drwxrwxr-x  [...] lazr.config-2.0.1-py3.4.egg
  drwxrwxr-x  [...] lazr.delegates-2.0.1-py3.4.egg
  drwxrwxr-x  [...] lazr.smtptest-2.0.2-py3.4.egg

All three directories exhibit the same pattern:

lazr.config-2.0.1-py3.4.egg
  EGG-INFO
  namespace_packages.txt (contains one line saying lazr)
  lazr
  config
  __init__.py  (contains the pseudo-namespace boilerplate code[+])
  __pycache__

[+] that boilerplate code looks like:

-snip snip-
# This is a namespace package, however under = Python 3.3, let it be a true
# namespace package (i.e. this cruft isn't necessary).
import sys

if sys.hexversion  0x30300f0:
try:
import pkg_resources
pkg_resources.declare_namespace(__name__)
except ImportError:
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
-snip snip-

but of course, this doesn't really play nicely with PEP 420 because it's the
mere existence of the __init__.py in the namespace package that prevents PEP
420 from getting invoked.  As an aside, Debian's packaging tools will actually
*remove* this __init__.py for declared namespace packages in Python 3, so
under Debian-land, things do work properly.  The upstream tools have no such
provision, so it's clear why we're getting the ImportErrors.  These three
packages do not share a PEP 420-style namespace, and the sys.hexversion guard
prevents the old-style pseudo-namespace hack from working.

This could potentially be fixable in the upstream packages, but again, more on
that later.

Now let's look at the working venvs:

  drwxrwxr-x  [...] lazr
  drwxrwxr-x  [...] lazr.config-2.0.1-py3.4.egg-info
  -rw-rw-r--  [...] lazr.config-2.0.1-py3.4-nspkg.pth
  drwxrwxr-x  [...] lazr.delegates-2.0.1-py3.4.egg-info
  -rw-rw-r--  [...] lazr.delegates-2.0.1-py3.4-nspkg.pth
  drwxrwxr-x  [...] lazr.smtptest-2.0.2-py3.4.egg-info
  -rw-rw-r--  [...] lazr.smtptest-2.0.2-py3.4-nspkg.pth


This looks quite different, and digging into the 'lazr' directory shows:

lazr
   config
   delegates
   smtptest

No __init__.py in 'lazr', and the sub-package directories contain just the
code for the appropriate library.  The egg-info directories look similar, but
now we also have -nspkg.pth files which contain something like the following:

-snip snip-
import sys, types, os;p = os.path.join(sys._getframe(1).f_locals['sitedir'], 
*('lazr',));ie = os.path.exists(os.path.join(p,'__init__.py'));m = not ie and 
sys.modules.setdefault('lazr', types.ModuleType('lazr'));mp = (m or []) and 
m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
-snip snip-

Re: [Distutils] Bilingual namespace package conundrum

[Barry Warsaw]

On Jan 01, 2015, at 09:14 PM, Donald Stufft wrote:

Why are you puzzled by the notion that something designed to work with a
one mechanism for a particular feature probably does not work with a
newer, different mechanism for a particular feature? My assumption is that
setuptools is ensuring that the __init__.py files are being written (or the
nspkg or whatever, I forget which files are used in which cases) which is
breaking the PEP420 “implicit” namespace packages.

Note that in neither case are actual PEP 420 namespace packages being used.
It would be nice if both pip and setuptools could handle namespace packages,
but neither do.  In fact I think if they did, there wouldn't be a problem.
The problem comes about because setuptools is installing them as separate eggs
which are *not* PEP 420 portions.  pip succeeds because it happens to install
the portions under the same top-level directory, so the namespace's
__init__.py just happens to get overwritten and thus works.

Cheers,
-Barry


pgp0ZRJOihdvm.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Bilingual namespace package conundrum

[Barry Warsaw]

On Jan 01, 2015, at 03:20 PM, Tres Seaver wrote:

That sounds right to me.  I never really understood the motivation for
PEP 420, but if the presence of that file disables it, then it should
ensure the old behavior regardless.

The motivation is described in the PEP, but briefly, on (many, most, all?)
Linux distros any single file can be owned by exactly one system package.  So
which package would own e.g. zope/__init__.py?

Before PEP 420, the answer is well, it's complicated.  After PEP 420, the
answer is no package, because that file doesn't exist.

This is why the Debian tools delete any stray zope/__init__.py files when they
are creating any zope.* binary package for Python 3.

I think you're right that removing the version guard will make it work, but
they won't be PEP 420 packages.  I don't think it's possible to make a
bilingual package a pseudo-namespace package under Python 2, but a PEP 420
namespace package under Python = 3.3.

Cheers,
-Barry


pgp5VR5IonTtK.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] stdeb and python dependencies

[Barry Warsaw]

On Oct 22, 2014, at 05:00 PM, Kelly Goedert wrote:

I am new to python. I created a simple python cli application that depends
on jsonpickle and plumbum.

Using this command python setup.py --command-packages=stdeb.command
bdist_deb I was able to create a .deb package. But when I try to install
the package with dpkg -i package_name I get

 package_name depends on python-plumbum; however:
  Package python-plumbum is not installed.
 package_name depends on python-jsonpickle; however:
 Package python-jsonpickle

In my setup.py I have this:

dependencies=['plumbum','jsonpickle']
install_requires=dependencies

Is it possible to make the debian package install this python dependencies?
Or is this the wrong approach? If so, what would be the recommend way?

`dpkg -i` doesn't install dependencies.  After you get this, run
`sudo apt-get install -f` to get your missing dependencies.

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Barry Warsaw]

On Sep 30, 2014, at 11:06 AM, M.-A. Lemburg wrote:

Installers and PyPI would then regard 3.1.4-1 as belonging to
release 3.1.4, but being a more current build as a distribution
file carrying 3.1.4 in its file name.

Please don't literally use 3.1.4-1.  That will cause all kinds of havoc with
the Debian ecosystem.  There we use a dash to separate upstream version
numbers from Debian version numbers.  Thus 3.1.4-1 means the first Debian
upload of upstream's 3.1.4.  3.1.4-2 is the second, etc.

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Barry Warsaw]

On Sep 30, 2014, at 02:34 PM, Jeremy Stanley wrote:

I'm becoming less and less convinced it actually *is* a source
distribution any more. My constant interaction with downstream Linux
distro packagers shows a growing disinterest in consuming release
tarballs of software, that they would generally prefer to pull
releases directly from tags in the project's revision control
systems instead.

This is not a universally held consensus.

We had a discussion about this at the recently concluded Debian conference.
There are folks who only want to use git tags as the consumption point for
Debian packages, but this opinion was not the majority opinion.  There's no
guarantee that what you get from a tagged upstream source revision will match
what comes in the sdist tarball.  Plus, the greater Debian ecosystem is firmly
camped in the tarball world, so even if you do checkout from a tag, you still
have to build a tarball for uploads, *and* you have to do it in a binary exact
copy reproducible way.

Thus, in the Debian Python team our policy is that if upstream produces
tarballs (as is the case for the vast majority of our packages, which are
sourced from PyPI), then we want the Debian package to use tarballs.  There
can be exceptions to the rule, but still today they are exceptions.

I don't think the tarball format is dead yet.

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Barry Warsaw]

On Sep 30, 2014, at 05:40 PM, Wichert Akkerman wrote:

Debian does allow 3.1.4-1-1. I forgot the exact rules, but I seem to remember
the package version is considered to start after the last dash. Debian will
also sort 3.1.4a after 3.1.4 unlike Python rules, so version “massaging”
might be necessary in other situations as well.

Yeah, but it's maddeningly confusing.  The havoc I mention is human based[*]
if not tools based. ;)

-Barry

[*] at least to *this* human.


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Barry Warsaw]

On Sep 30, 2014, at 04:25 PM, Jeremy Stanley wrote:

Good to know. The Debian Developer packaging the majority of the
projects I work on must be in that minority.

IIRC, the OpenStack packagers were probably the most prominent proponent of
release-from-tag.

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Barry Warsaw]

On Sep 28, 2014, at 07:31 PM, Donald Stufft wrote:

I'd like to discuss the idea of moving PyPI to having immutable files. This
would mean that once you publish a particular file you can never reupload
that file again with different contents. This would still allow deleting the
file or reuploading it if the checksums match what was there prior.

Although I have abused this in the past, as others have pointed out, because
once uploaded I realize there is a bug in the package.  There's a certain
class of such bugs that prompt a quick re-upload rather than a version rev,
such as some display problem on PyPI (because of package metadata), or some
follow on packaging bug, such as a missing MANIFEST.in causing Debian package
build to fail.  Yes, the latter is more easily checked before upload, but
sometimes you feel optimistic. ;)

This won't make your lives easier, but I'd like to propose some support for
embargoed uploads.  These would be normal uploads except that they wouldn't
be publicly available until a 'publish' button were pushed.  Such embargoed
uploads wouldn't be subject to the checksum limitation, and we'd have to
figure out exactly how such packages would be available (certainly to a logged
in owner of the project via the web, but perhaps through an authenticated
scriptable interface).

Even if you decide against supporting something like this, I'd still be okay
with the checksum restriction.  You never run out of version numbers.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Barry Warsaw]

On Sep 29, 2014, at 09:40 AM, Ian Cordasco wrote:

That's essentially what I see as the chief use-case for
testpypi.python.org. I don't think pypi.python.org needs to support
this as well. Simple is better than complex after all :)

Can we then make it easy to upload to testpypi via the cli?  Maybe it already
is or I'm using the wrong tools.  (I'm just a dumb `setup.py upload` monkey.)

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Create formal process for claiming 'abandoned' packages

[Barry Warsaw]

On Sep 20, 2014, at 06:10 PM, Toshio Kuratomi wrote:

All distros I can think of have some sort of self-governance whereas pypi is
more akin to a bunch of customers making use of a service.  Some of the
distro policies don't apply very well in this space.  Some do, however, so I
hope other people who are familiar with their distros will also filter the
relevant policy ideas from their realms and put them forward.

Debian and Ubuntu have very interesting differences, especially given that
one is derived from the other.

Debian has a very strong personal maintainership culture, where often one
person maintains a package alone.  Debian Developers have upload rights for
any package, and they can do non-maintainer uploads.  Debian also has various
policies related to orphaning and adopting packages, but those are mostly for
cooperative package ownership transfers.  When a maintainer cannot be
contacted, there is a missing-in-action process that can be used to wrest
ownership for a non-responsive maintainer.

Many packages are team maintained, and I personally find these much more
productive and enjoyable to work with.  A team maintained package doesn't have
to worry about ownership transfers because any team member with general upload
rights can upload the package, and even non-uploading team members can do
everything short of that.  Primarily that means prepare the package's vcs so
that it's ready to be sponsored by an uploading developer.

Ubuntu is different in that no package is maintained by a single person.
Essentially they are all team maintained.  Rights to upload packages are
conferred on the basis of pockets and package-sets.  So for example, if
someone wants to be involved in Zope, they could join the ~ubuntu-zope-dev
team and once approved, they'd have upload rights to any package in the Zope
package set.  There are also pockets such as universe (packages which are
available in Ubuntu but without security and other distro-level guarantees),
and there is a MOTU (masters of the universe) team that can upload there.  At
the top of the ladder, core-devs can upload anything.

In the context of PyPI, I tend to think that teams can be an answer to a lot
of the problem.  I'm looking for example at one of the lazr.* packages I
co-maintain on PyPI.  The package has a number of individual owner roles, but
I know that there are probably only two of those people who still care enough
about the package to maintain it upstream, or would ever likely upload new
versions to PyPI.  Handling roles in this way is pretty inconvenient because
there might be dozens of packages that some combination of those group of
people would be responsible for.  If I could create a LAZR team and manage
roles within that team, and then assign ownership of a package to that team,
it would be easier to administer.

That doesn't solve the problem where individuals have a strong preference for
personal ownership of PyPI entries, but given that upstreams often are a team
effort, I think it would go a long way toward helping easy transition efforts
for PyPI ownership.

It might even allow for better handling of transitions.  For example, if a
package owner is not reachable for some period of time, and someone steps up
to take it over, you could create a team and put both people in it, then
transfer the ownership to that team.

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wheels or system packages for pip on ubuntu

[Barry Warsaw]

On Sep 03, 2014, at 12:24 PM, Reinout van Rees wrote:

All of them have ubuntu packages, but especially for gdal we sometimes need a
newer version. A PPA can help here, but I thought a wheel could be nice,
too.

In many cases, it mostly takes an interested person to get Ubuntu and Debian
packages updated.  Generally it's best to get the latest versions in Debian
first (and that may be easy or hard depending on the maintainership of the
package), and then sync or merge the newer Debian versions into Ubuntu.
Looking quickly at gdal, I can see that Ubuntu carries some deltas over
Debian, but I haven't looked at the details.  It would take some evaluation
as to whether the Ubuntu deltas can be dropped or need to be merged.

#debian-python on OTFC or the debian-python mailing lists are good places to
start about getting some help with the packages you care about.

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wheels or system packages for pip on ubuntu

[Barry Warsaw]

On Sep 04, 2014, at 10:39 AM, Marcus Smith wrote:

wouldn't that only update it for the *next* release of debian/ubuntu?

Generally yes.  There's also backports, but that's more effort.

https://help.ubuntu.com/community/UbuntuBackports
https://wiki.debian.org/Backports

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Other ideas from today's packaging meetup at EuroPython

[Barry Warsaw]

On Jul 25, 2014, at 08:46 AM, Donald Stufft wrote:

Yea, I’m not sure whether I like it or not. Probably once we get a for real
build farm for PyPI setup that will be a pretty reasonable sized carrot for
people to upload sources.

That's really the right long-term approach, IMO.  I'd like to some day see
source-only uploads, with wheel builds for various supported platforms
appearing automatically via the build farm.

But ultimately you're right.  If we don't have source available from
*somewhere* (with PyPI being the most obvious and easiest location for
downstreams), then oh well, you won't get your package into some Linux
distros, and your users will have to roll their own.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Other ideas from today's packaging meetup at EuroPython

[Barry Warsaw]

On Jul 24, 2014, at 01:28 PM, Richard Jones wrote:

1. reject wheel uploads in the absence of an sdist in the index (the linux
guys were really happy about that as a proposal ;)

+1 :)

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP440 Local versions idea in rpm/deb?

[Barry Warsaw]

On May 02, 2014, at 12:01 PM, Marcus Smith wrote:

PEP440 has the local version idea to distinguish locally patched projects
from upstream versions.
http://legacy.python.org/dev/peps/pep-0440/#local-version-identifiers

e.g.  Django-1.6.4-1  for a locally patched Django-1.6.4 to place on a
local index.

Although it doesn't relate directly to this list, I know Nick (the PEP440
author) works with Redhat, so for understanding, what's the parallel in rpm
(or deb)?  is there a documented concept for this, because I can't seem to
find anything other than post releases.

Debian (and thus of course also Ubuntu) packages usually start with the
upstream version number, and add additional qualifiers on the end.  So for
example, upstream Django 1.6.1 might be packaged in Debian as 1.6.1-2
(meaning, the 2nd Debian-specific revision of upstream's 1.6.1).  When a
Debian developer modifies the package, they'll typically bump the number after
the dash.

Ideally Ubuntu would just inherit the Debian version, but when we need to make
additional deltas to the Debian packages, we'll have a more specific
qualifier, such as 1.6.1-2ubuntu3.  That tells you that the package is
upstream 1.6.1, with Ubuntu rev 3 over Debian rev 2.  A package that's only in
Ubuntu might look like 1.6.1-0ubuntu3 (the 0 meaning there's no Debian
equivalent yet of 1.6.1).

There are plenty of variations, including many that don't strictly follow this
scheme, e.g. if a version is packaged from a vcs branch.  But this should give
you a taste of the most common version numbers you'll see on Debian and
Ubuntu.

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pycon

[Barry Warsaw]

On Mar 28, 2014, at 03:06 PM, Daniel Holth wrote:

Who is going to pycon? I will be there.

I'll be there, for the duration (language summit through sprints).  It would
be great to hav an OpenSpace or BoF for discussing the intersection of Python
packaging issues and distros.

-Barry



signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] nspkg.pth files break $PYTHONPATH overrides

[Barry Warsaw]

On Mar 25, 2014, at 03:35 PM, PJ Eby wrote:

I think the correct fix would be to change the nspkg.pth magic to check for
PEP 420 support, but unfortunately it seems we may have to use version
checking: on rereading PEP 420 I see there's no 'sys.namespace_packages' or
similar object that can be directly checked for feature support.  :-(

There is.  It's *pronounced* sys.namespace_packages, but it's spelled
importlib._bootstrap._NamespaceLoader ;)

http://youtu.be/ehKGlT2EW1Q?t=44s

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] nspkg.pth files break $PYTHONPATH overrides

[Barry Warsaw]

Apologies for cross-posting, but this intersects setuptools and the import
system, and I wanted to be sure it reached the right audience.

A colleague asked me why a seemingly innocent and common use case for
developing local versions of system installed packages wasn't working, and I
was quite perplexed.  As I dug into the problem, more questions than answers
came up.  I finally (think! I) figured out what is happening, but not so much
as to why, or what can/should be done about it.

This person had a local checkout of a package's source, where the package was
also installed into the system Python.  He wanted to be able to set
$PYTHONPATH so that the local package wins when he tries to import it.  E.g.:

% PYTHONPATH=`pwd`/src python3

but this didn't work because despite the setting of PYTHONPATH, the system
version of the package was always found first.  The package in question is
lazr.uri, although other packages with similar layouts will also suffer the
same problem, which prevents an easy local development of a newer version of
the package, aside from being a complete head-scratcher.

The lazr.uri package is intended to be a submodule of the lazr namespace
package.  As such, the lazr/__init__.py has the old style way of declaring a
namespace package:

try:
import pkg_resources
pkg_resources.declare_namespace(__name__)
except ImportError:
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)

and its setup.py declares a namespace package:

setup(
name='lazr.uri',
version=__version__,
namespace_packages=['lazr'],
...

One of the things that the Debian helper program does when it builds a
package for the archive is call `$python setup.py install_egg_info`.  It's
this command that breaks $PYTHONPATH overriding.

install_egg_info looks at the lazr.uri.egg-info/namespace_packages.txt file,
in which it finds the string 'lazr', and it proceeds to write a
lazr-uri-1.0.3-py3.4-nspkg.pth file.  This causes other strange and unexpected
things to happen:

% python3
Python 3.4.0 (default, Mar 22 2014, 22:51:25) 
[GCC 4.8.2] on linux
Type help, copyright, credits or license for more information.
 import sys
 sys.modules['lazr']  
module 'lazr'
 sys.modules['lazr'].__path__  
['/usr/lib/python3/dist-packages/lazr']

It's completely weird that sys.modules would contain a key for 'lazr' when
that package was never explicitly imported.  Even stranger, because a fake
module object is stuffed into sys.modules via the .pth file, tracing imports
with -v gives you no clue as to what's happening.  And while
sys.modules['lazr'] has an __path__, it has no other attributes.

I really don't understand what the purpose of the nspkg.pth file is,
especially for Python 3 namespace packages.

Here's what the nspkg.pth file contains:

import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], 
*('lazr',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and 
sys.modules.setdefault('lazr',types.ModuleType('lazr')); mp = (m or []) and 
m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)

The __path__ value is important here because even though you've never
explicitly imported 'lazr', when you *do* explicitly import 'lazr.uri', the
existing lazr module object's __path__ takes over, and thus the system
lazr.uri package is found even though both lazr/ and lazr/uri/ should have
been found earlier on sys.path (yes, sys.path looks exactly as expected).

So the presence of the nspkg.pth file breaks $PYTHONPATH overriding.  That
seems bad. ;)

If you delete the nspkg.path file, then things work as expected, but even this
is a little misleading!

I think the Debian helper is running install_egg_info as a way to determine
what namespace packages are defined, so that it can actually *remove* the
parent's __init__.py file and use PEP 420 style namespace packages.  In fact,
in the Debian python3-lazr.uri binary package, you find no system
lazr/__init__.py file.  This is why removing the nspkg.pth file works.

So I thought, why not conditionally define setup(..., namespace_packages) only
for Python 2?  This doesn't work because the Debian helper will see that no
namespace packages are defined, and thus it will leave the original
lazr/__init__.py file in place.  This then breaks $PYTHONPATH overriding too
because of __path__ extension of the pre-PEP 420 code only *appends* the local
development path.  IOW, the system import path is the first element of a
2-element list on lazr.__path__.  While the local import path is the second
element, in this case too the local import fails.

It seems like what you want for Python 3 (and we're talking = 3.2 here) is
for there to be neither a nspkg.pth file, nor the lazr/__init__.py file, and
let PEP 420 do it's thing.  In fact if you set things up this way, $PYTHONPATH
overriding works exactly as expected.

Because I don't know why install_egg_info is installing the nspkg.pth file, I
don't know which component needs to be 

Re: [Distutils] nspkg.pth files break $PYTHONPATH overrides

[Barry Warsaw]

On Mar 24, 2014, at 05:53 PM, Donald Stufft wrote:

See also https://github.com/pypa/pip/issues/3

Hah, yeah.  I didn't realize --single-version-externally-managed is implied by
--root, and in fact our Debian build scripts often (either explicitly or
implicitly) define --root, as is the case in lazr.uri.

Basically prior to PEP420 namespace packages were bad and using them results
in pain sooner or later :( I’m not sure if a good solution yet, perhaps we
can backport PEP420 to PyPI and have namespace packages depend on that?

Or maybe do a version check before installing this file?  Python 3.3 and newer
will have PEP 420 support, and this file makes no sense in that case.

(Backporting PEP 420 support to Python 3.2 might be useful, though a pain
without importlib.  There's no sense in backporting to Python 3.1, IMHO.)

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python))

[Barry Warsaw]

On Feb 05, 2014, at 01:05 AM, Nick Coghlan wrote:

That only works if the system site packages is configured to be
visible inside the virtual environment - it usually isn't these days.

Really?  I do this all the time.  It prevents downloading gobs of stuff from
PyPI that my system already provides in reasonably up-to-date versions.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Warehouse and XMLRPC

[Barry Warsaw]

On Nov 15, 2013, at 10:12 PM, Donald Stufft wrote:

The bulk of XMLRPC has been implemented in Warehouse. It would be great if
people who are using the XMLRPC could try it out using the
https://preview-pypi.python.org/pypi url.

Currently the search API does not work.

Are you planning on putting a REST API on Warehouse?

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] URL Structure of Packages URLs

[Barry Warsaw]

On Oct 12, 2013, at 06:15 PM, Donald Stufft wrote:

Just to be clear, I don't fault folks for using the /packages/ urls. I was
just trying to get some sort of idea if anyone actually used that url
structure or not. I also don't plan on breaking anything for people who do.

That being said, do you know if the Debian case requires something to be
served at /packages/source/D/Django/ in order to locate a file at
/packages/source/D/Django/Django-*.tar.gz?

It does.

uscan(1) is the tool that parses and acts on debian/watch files.  My Perl-fu
is pretty rusty so I'm not sure I'm reading it correctly, but I think it's
evident what the tool does (modulo tons of corner cases that mostly aren't
relevant for PyPI urls).

Here for example is tox's debian/watch file:

-snip snip-
version=3
http://pypi.python.org/packages/source/t/tox/tox-(.*).tar.gz
-snip snip-

And verbose output:

% uscan --verbose
-- Scanning for watchfiles in .
-- Found watchfile in ./debian
-- In debian/watch, processing watchfile line:
   http://pypi.python.org/packages/source/t/tox/tox-(.*).tar.gz
-- Found the following matching hrefs:
 tox-0.5.tar.gz (0.5)
 tox-0.5.tar.gz (0.5)
 tox-0.6.tar.gz (0.6)
 tox-0.6.tar.gz (0.6)
 tox-0.7.tar.gz (0.7)
 tox-0.7.tar.gz (0.7)
 tox-0.8.tar.gz (0.8)
 tox-0.8.tar.gz (0.8)
 tox-1.4.3.tar.gz (1.4.3)
 tox-1.4.3.tar.gz (1.4.3)
 tox-1.5.0.tar.gz (1.5.0)
 tox-1.5.0.tar.gz (1.5.0)
 tox-1.6.0.tar.gz (1.6.0)
 tox-1.6.0.tar.gz (1.6.0)
 tox-1.6.1.tar.gz (1.6.1)
 tox-1.6.1.tar.gz (1.6.1)
Newest version on remote site is 1.6.1, local version is 1.6.0
 = Newer version available from
http://pypi.python.org/packages/source/t/tox/tox-1.6.1.tar.gz
-- Downloading updated package tox-1.6.1.tar.gz
-- Successfully downloaded updated package tox-1.6.1.tar.gz
and symlinked tox_1.6.1.orig.tar.gz to it
-- Scan finished


And a directory listing of the URL leading up to the pattern (edited for
readability):

Index of /raw-packages/source/t/tox/

../
tox-0.5.tar.gz  12-Jul-2010 11:30   40853
tox-0.6.tar.gz  12-Jul-2010 18:21   41424
tox-0.7.tar.gz  14-Jul-2010 20:48   45934
tox-0.8.tar.gz  30-Jul-2010 23:41   41828
tox-0.9.zip 25-Nov-2010 20:05   57001
tox-1.0.zip 28-May-2011 14:59   61367
tox-1.1.zip 09-Jul-2011 10:09   99297
tox-1.2.zip 10-Nov-2011 18:40   66284
tox-1.3.zip 21-Dec-2011 08:40   67862
tox-1.4.1.zip   03-Jul-2012 09:34   76920
tox-1.4.2.zip   20-Jul-2012 11:05   77565
tox-1.4.3.tar.gz01-Mar-2013 09:44   71661
tox-1.4.zip 13-Jun-2012 15:01   76147
tox-1.5.0.tar.gz22-Jun-2013 13:05   73866
tox-1.6.0.tar.gz15-Aug-2013 13:39  130734
tox-1.6.1.tar.gz04-Sep-2013 14:17  131654

So yes there needs to be some url that uscan can get a list of to determine
the highest available version matching a wildcarded pattern.

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] URL Structure of Packages URLs

[Barry Warsaw]

On Oct 12, 2013, at 08:50 PM, Floris Bruynooghe wrote:

Sounds like this could be addressed with a lintian warning and a long
transition time (~2 years) so leaving proper redirects around would be
advisable.

Sure, but my point was that people will use the most easily discoverable
pattern available, which currently is the download link from the package's
PyPI page.  If we want to make some other link more commonly used, it should
be more easily discoverable.

-Barry



signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] URL Structure of Packages URLs

[Barry Warsaw]

On Oct 08, 2013, at 10:44 AM, Donald Stufft wrote:

Hrm, I'm assuming these require a file listing at

/packages/source/D/ too.

Of course these files should probably be using the simple API and
not the packages url directly :/

Maybe that URL should be given on the PyPI page for the package?

The current watch patterns are used overwhelmingly probably because they are
easily discovered.

E.g. Copy the URL from the PyPI page, and patternize it.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Ubuntu says Virtualenv, Pip and Setuptools untrusted

[Barry Warsaw]

On Jul 27, 2013, at 10:12 AM, Erik Bernoth wrote:

did you know that Ubuntu 13.10 (or maybe Debian?) declares those
packages as untrusted and asks you twice, if you really want to
install them? Is there anything that can be done about that?

Um, what?

Please provide details.  What commands are you running and what do you see?

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Shebang lines, /usr/bin/python, and PEP394

[Barry Warsaw]

On Jul 26, 2013, at 09:58 PM, Nick Coghlan wrote:

Not everybody uses generated script wrappers, though - if there is a
hardcoded /usr/bin/env python or /usr/bin/python in a shebang
line, the Python build tools won't touch it. There's also a whole lot
of software that isn't packaged at all - it's sitting around in user
and admin home directories, or maybe in a shared directory on a file
server or even in a private source control repo.

I actually think installing a script via setuptools should rewrite shebangs
even in the case of /usr/bin/env.

I love `#!/usr/bin/env python` *for development* but I really think its a bad
thing to have for installed scripts.  Certainly, for distro installed scripts,
it's (usually) terrible.  I think virtualenv installs are generally in the
same boat though - if you're installing a script into a virtualenv, it's
better to rewrite the shebang to use the executable that was used to install
it.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Shebang lines, /usr/bin/python, and PEP394

[Barry Warsaw]

On Jul 26, 2013, at 08:38 PM, Paul Moore wrote:

There are cases where it's useful and appropriate

Sure, I don't disagree.  Just that I think the general rule should be:

* Use /usr/bin/env in your source tree
* Use /usr/bin/$python when installed

I think those rules cover the majority of cases.  Any automatic shebang
rewriting must be selectable.  I'd argue for default-rewrite with an option to
disable.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Shebang lines, /usr/bin/python, and PEP394

[Barry Warsaw]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On Jul 25, 2013, at 03:37 PM, Philip Jenvey wrote:

What's the value of sys.executable in the brave new world of distributions
that follow PEP 394?

One use case is locating other scripts/executables that might get installed
next to sys.executable in a virtualenv.

- -Barry
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.20 (GNU/Linux)

iQIcBAEBCAAGBQJR8cZfAAoJEBJutWOnSwa/lnMQALE6VLkSNElQfJnIgBraS/PR
/C7m2kCNbeBdHfTXjRJYeHoecFPO6z5mSurO2ikbFvwvxmWi1MjxX6/gKR6jaNJ6
PJcyoO1KaJJ9CKP0hlAcuLYGyj86Nhmw1cK20yeejV+YJzqpFjqXxHyH9GlaGQKc
aDKG/xN/SX139V4/KfgaAcXKhbsCkhL6cAugHFGl8qTytdvaOOzRs2/jHsTcPDp3
Dfb2nz69PteQAYSPHi7Na9bAe2+lGy9nz93zfw7T6FLB4ya3trATnbxD1Ojmg2mw
lnuvTBg6GiRv/BZgDB6Fl8OTMj6BvyfKKG9Nb6rCCCYyPR8Q8t1TkxODCAkHi6kY
omwMOR+Fc4Btm/rtRcY4rXKPY6FXVgnzysOT/z4GcpjWotp2E6ZiSruwsm7okwM3
Pbl9BEAD4WdgotSJDrb17BmcLl48qqenODOmrHKvfIu67iagSAr8D9Qp2B0G1V0B
GRtmCVrcvGheMKw+5VAYAmfyVnNQmi1MVY1fNN9pya9Wi38mYiWFxXhD/wIhU6Ih
/vs38dt+PSu6AMfomvf7tzOuZKE+JKkS//e7+OlEwwaDl4am5awXh0Ci/6z8cJHg
ywH003hYZ4SWzuMHWerkIxydXW3D8GUvRDcpR3OZaoknEKQYkGoMVY0KimKG2ykZ
KcGciNDFFXpxaq6+4cc1
=Dy/O
-END PGP SIGNATURE-
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Q about best practices now (or near future)

[Barry Warsaw]

On Jul 17, 2013, at 11:46 AM, Daniel Holth wrote:

I'd like to see an ambitious person begin uploading wheels that have
no traditional sdist.

You're not getting rid of sdists are you?

Please note that without source distributions (preferably .tar.gz) your
package will never get distributed on a Linux distro.

Maybe the keyword here is traditional though.  In that case, keep in mind
that at least in Debian and its derivatives, we have a lot of tools that make
it pretty trivial to package something setup.py based from PyPI.  If/when that
goes away, it will be more difficult to get new package updates, until the
distro's supporting tools catch up.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Q about best practices now (or near future)

[Barry Warsaw]

On Jul 17, 2013, at 07:56 PM, Paul Moore wrote:

The long-term intent is to remove executable setup.py. When this happens,
definitely consumers (end users, Python tools like pip, and distro
packaging systems) will have some migration work to do. Keeping that
manageable will definitely be important. But doing nothing and staying
where we are isn't really an option, so we'll have to accept and manage the
pain.

Definitely.  And if that leads to a declarative equivalent that we can reason
about without executing, all the better.

the-setup.cfg-is-dead,-long-live-the-setup.cfg-ly y'rs,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Q about best practices now (or near future)

[Barry Warsaw]

On Jul 17, 2013, at 08:34 PM, Oscar Benjamin wrote:


I imagined that distro packaging tools would end up using the wheel as
an intermediate format when building a deb from a source deb.

Do you mean, the distro would download the wheel or that it would build it
during the build step for the archive?  Probably not the former, as any binary
blobs in a wheel would both violate policy and likely be inappropriate for all
the platforms we build for.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 439 and pip bootstrap updated

[Barry Warsaw]

On Jul 10, 2013, at 02:43 PM, Paul Moore wrote:

I would find it distinctly irritating if in Python 3.4 I have to type pip3
bootstrap to bootstrap pip - and even worse if *after* the bootstrap the
command I use is still pip. (And no, there is currently no pip3 command
installed by pip, and even if there were, I would not want to use it, I'm
happy with the unsuffixed version).

I have a lot of sympathy for this, and the general issue has come up in a
number of different contexts, e.g. nostests/nosetests3 and so on.  On a distro
like Debian, this just adds more gunk to /usr/bin, especially since some
scripts are also minor-version dependent.

One approach is to use `$python -m nose` or in this case `$python -m pip`
which cuts down on the number of scripts, is unambiguous, but is far from
convenient and may not work in all cases, e.g. for older Python's that don't
support -m or don't support it for packages.

I think there was a thread on python-ideas about this, but in the back of my
mind, I have this horrible idea for a version-aware relauncher you could use
in your shebang line.  Something like:

#! /usr/bin/pylaunch

So that you could do something like:

$ nosetests -3
$ nosetests -2
$ nosetests -3.3
$ nosetests -2.7

and it would relaunch itself using the correct Python version, consuming the
version argument so the actual script wouldn't see it.

I'm not sure if the convenience is worth it, and I'm sorry for making you
throw up a little in your mouth there.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 439 and pip bootstrap updated

[Barry Warsaw]

On Jul 10, 2013, at 09:50 PM, Paul Moore wrote:

I think python -m pip should be the canonical form (used in
documentation, examples, etc). The unittest module has taken this route, as
has timeit. Traditionally, python-dev have been lukewarm about the -m
interface, but its key advantage is that it bypasses all the issues around
versioned executables, cross-platform issues, the general dreadfulness of
script wrappers on Windows, etc, in one fell swoop.

+1

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] option #1 plus download_url scraping

[Barry Warsaw]

On Jun 04, 2013, at 03:23 PM, Noah Kantrowitz wrote:

Do you mean you just don't want to update the version number in setup.py
before you release?

Correct, although on further reflection, if the alternative download site has
predictable URLs, then it wouldn't be too difficult to calculate the new url
for setup.py.  I just don't want to have to change the version number in more
than one place (which for my packages, is *not* the setup.py).

I'm a bit unsure of the reason for this. The goal is very specifically the
hosting outside of PyPI is no longer encouraged. The reliability and
performance of PyPI have enough of a track record now that I want it on my
own site just in case no longer holds enough water to be worth the
substantial downsides.

Fair enough.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] option #1 plus download_url scraping

[Barry Warsaw]

On Jun 04, 2013, at 04:46 PM, Carl Meyer wrote:

On 06/04/2013 04:30 PM, Donald Stufft wrote:

 I'm interested in the use case. How are generating a release without
 running setup.py sdist?

I think you misunderstood (the way Barry described it wasn't completely
clear). If I'm reading it correctly, I don't want to have to change
setup.py every time I make a release means I don't want to have to
change the Download-URL metadata in setup.py to point to a specific new
sdist tarball URL every time I make a release; I want it to always point
to the same index URL and have that index page scraped for new release
tarball URLs.

Correct, sorry for not being clear.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] option #1 plus download_url scraping

[Barry Warsaw]

On Jun 05, 2013, at 12:16 PM, Donald Stufft wrote:

Where are you updating the version information at? And how are you generating
a tarball so that it's name has the correct version in it?

It depends on the package, but let's say it's in a version.txt file.  Your
implication is correct though - if setup.py is parsing that file to calculate
the version key, it can also do the same and calculate the download_url value.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] option #1 plus download_url scraping

[Barry Warsaw]

On Jun 05, 2013, at 02:47 PM, Donald Stufft wrote:

I'm really just trying to get a sense of your workflow to see if I can make
any changes to improve the process for it.

One of the big problems with download_url is that the data in setup.py is
used in (and influences the content of) the final dist file. This means that
inside of a setup.py you won't know what the hash of the final file is. So
it's difficult for a setup.py based workflow with external urls to provide
md5 sums for the files which means that pip and friends can't verify that no
body modified the download in transit.

Let me explain what I (used to) do, and I'll let you decide whether anything
needs to change. ;)

When I've finally got my vcs into a releasable state, I'll generally do:

$ python setup.py sdist upload -s

As you know,  this will create the tarball and signature file in dist, and
upload everything nicely to the Cheeseshop.  At this point, I go to my
project's Launchpad page and push the big I made a release button.  This
fiddles some state on my project page, and it allows me to upload files
attached to that particular release.  The nice thing is that I can just upload
the dist/*.tar.gz and dist/*.asc  to add the tarball and signature to the
Launchpad download page.  E.g.

https://launchpad.net/flufl.enum

and

https://launchpad.net/flufl.enum/+download

The url is predictable (which is good because it also has to play nicely with
Debian watch files), with option #3, I just added the index page to
download_url and let clients scrape it.  You'll see that it contains links to
the md5 checksum and the locally generated signature.

There must be some value to also allowing folks to download from Launchpad, as
shown by the 1055 downloads of flufl.enum.  Where are the PyPI download stats?

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] option #1 plus download_url scraping

[Barry Warsaw]

Like many of you, I got Donald's message about the changes to URLs for
Cheeseshop packages.  My question is about the three options; I think I want a
middle ground, but I'm interested to see why you will discourage me from that
wink.

IIUC, option #1 is fine for packages hosted on PyPI.  But what if our packages
are *also* hosted elsewhere, say for redundancy purposes, and that external
location needs to be scraped?

Specifically, say I have a download_url in my setup.py.  I *want* that url to
be essentially a wildcard or index page because I don't want to have to change
setup.py every time I make a release (unless of course `setup.py sdist` did it
for me).  I also can't add this url to the Additional File URLs page for my
package because again I'd have to change it every time I do a release.

So the middle ground I think I want is: option #1 plus scraping from
download_url, but only download_url.

Am I a horrible person for wanting this?  Is there a better way.

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Merge catalog-sig and distutils-sig

[Barry Warsaw]

On Mar 28, 2013, at 02:22 PM, Donald Stufft wrote:

Is there much point in keeping catalog-sig and distutils-sig separate?

Without yet reading the whole thread, I'll just mention that it's probably
easier to just retire one or the other mailing lists and divert all discussion
to the other one.  Of course, the archives for the retired list would be
retained for historical purposes.  In fact, sigs are *supposed* to be
periodically reviewed for renewal or retirement, though I think practically
speaking we haven't done that in a very long time.

If there's consensus on what you want to do, please contact postmaster@ and
let them know.  Let's say you just want to retire catalog-sig: we can set up
forwards to distutils-sig and let the former be an acceptable alias to the
latter so postings will be accepted when addressed to either.  Of course,
folks on the defunct list should manually subscribe to the good list
(i.e. opt-in).

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Catalog-sig] Merge catalog-sig and distutils-sig

[Barry Warsaw]

On Mar 28, 2013, at 03:42 PM, Donald Stufft wrote:

Don't care how it's done. I don't know Mailman enough to know what is
possible or how easy things are. I thought packaging-sig sounded nice but if
you can't rename + redirect or merge or something in mailman I'm down for
whatever.

Renaming can be done, but it's a bit of a pain.  Of course, we can keep the
archives for any retired list, so urls don't need to break.  OTOH, it's
definitely easier just to keep distutils-sig and retire catalog-sig.

-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip merges wheel

[Barry Warsaw]

On Mar 18, 2013, at 02:16 PM, Lennart Regebro wrote:

I still think it is unfortunate that we are starting to extend pip to
be a tool for developers to create distributions. It would be better
of pip was kept as an install tool, and we added the utilities for
creating distributions separate.

+1.  Doesn't this violate Nick's mission for killing `setup.py install` which
IIUC is motivated by wanting to separate building and installing?

I really want pip to just be about installing and use other tools to build
source and binary distributions.

-Barry
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip merges wheel

[Barry Warsaw]

On Mar 18, 2013, at 04:13 PM, Nick Coghlan wrote:

Eventually I expect pip will grow a --wheel-only option to run it in
strict installer only mode, but the ecosystem is a long way from
supporting that being a useful option (especially since there are some
cases which will still require falling back to the build from source
model).

If that's the end goal, then it should be the default now.

-Barry
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Library instability on PyPI and impact on OpenStack

[Barry Warsaw]

On Mar 04, 2013, at 06:11 PM, Nick Coghlan wrote:

My point of view is that the system Python is there primarily to run
system utilities and user scripts, rather than arbitrary Python
applications. Users can install alternate versions of software into
their user site directories, or into virtual environments. Projects
are, of course, also free to include part of their version number in
the project name.

I have the same opinion, but it doesn't make developers very happy at all.

Cheers,
-Barry
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig