Re: [Distutils] [Python-ideas] Pypi private repo's

[Jannis Gebauer]

What if there was some kind of “blessed” entity that runs these services and 
puts the majority of the revenue into a fund that funds development on PyPi 
(maybe trough the PSF)? 

Jannis

> On 4. Apr 2018, at 23:24, Dustin Ingram  wrote:
> 
> This was recently discussed on the Packaging-WG mailing list. To
> summarize, there are a few key reasons why this would be challenging:
> 
> 1) The PSF is a non-profit. Taking on work generally in the domain of
> for-profit enterprises might jeopardize our tax-exempt status.
> 
> 2) PyPI relies heavily (~$1M/yr) on donated services and
> infrastructure. If we start trying to make money, our sponsors may not
> appreciate it.
> 
> 3) If PyPI is in the business of hosting private packages, it may
> de-incentivize us from helping to make sure "competing" private
> indices (devpi, Artifactory, gemfury, etc) are functional.
> 
> 4) With the exception of the current MOSS grant, PyPI is supported
> entirely by unpaid volunteers. Is it fair to ask volunteers to
> continue contributing their time to a for-profit enterprise?
> 
> Not to say that this would be impossible -- PyCon is quite similar
> (turns a profit, has sponsors, competes with other conferences, uses
> volunteer support) has addressed (and is addressing) many of these
> challenges, but it remains that the transition would be challenging.
> 
> D.
> 
> On Wed, Apr 4, 2018 at 3:55 PM, Alex Walters  wrote:
>> I am fairly sure if you give the PyPA that suggestion, they will just
>> deflate at the thought of the workload.  Besides, we already offer private
>> repos for free, several ways ranging from devpi to python -m
>> SimpleHTTPServer in a specially created directory.
>> 
>> 
>> 
>> 
>> 
>> From: Python-ideas 
>> On Behalf Of Nick Humrich
>> Sent: Wednesday, April 4, 2018 12:26 PM
>> To: python-id...@python.org
>> Subject: [Python-ideas] Pypi private repo's
>> 
>> 
>> 
>> I am sure this has been discussed before, and this might not even be the
>> best place for this discussion, but I just wanted to make sure this has been
>> thought about.
>> 
>> What if pypi.org supported private repos at a cost, similar to npm?
>> 
>> This would be able to help support the cost of pypi, and hopefully make it
>> better/more reliable, thus in turn improving the python community.
>> 
>> If this discussion should happen somewhere else, let me know.
>> 
>> Nick
>> 
>> 
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>> 
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Building a Python package build service for warehouse

[Jannis Gebauer]

Hi!

I’m currently working on a package build server. My goal is to produce useful 
additional meta data for all packages available on PyPi.

This includes:

- Transitive dependencies
- Is the package installable under Python 3?
- Various automated “code quality” tests like pylint, pyflakes, pep8, mccabe 
etc.
- Automated security tests
- (possibly changelogs, commit logs)
- Licenses!

The main idea is to run the build process in a restricted “sandbox” docker 
container that pulls the package from PyPi, installs it and runs a couple of 
tools on it. Code is still pretty rough, nothing to look at at the moment I’m 
afraid.

Is there any interest in working on this together? Maybe even with the goal to 
make it an open API that can be consumed by warehouse et al.?

Interested in any thoughts on this!

Cheers,

Jannis

P.S: I’m currently crunching trough the data on a 96 CPU cluster. There’s an 
API available, but it’s sitting behind HTTP Basic Auth as it is basically an 
endpoint for remote code execution (and throws lots of 500s :D). Send me a mail 
to j...@pyup.io  if you want to play around with it.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Jannis Gebauer]

Thanks for the heads-up, Paul.

I’m currently using `pip.get_installed_distributions` and as far as I can see 
that has moved into `_internal`, too: 
https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L333 


Any recommendations?

Thanks,

Jannis

> On 20. Oct 2017, at 15:22, Paul Moore  wrote:
> 
> We're in the process of starting to plan for a release of pip (the
> long-awaited pip 10). We're likely still a month or two away from a
> release, but now is the time for people to start ensuring that
> everything works for them. One key change in the new version will be
> that all of the internal APIs of pip will no longer be available, so
> any code that currently calls functions in the "pip" namespace will
> break. Calling pip's internal APIs has never been supported, and
> always carried a risk of such breakage, so projects doing so should,
> in theory, be prepared for such things. However, reality is not always
> that simple, and we are aware that people will need time to deal with
> the implications.
> 
> Just in case it's not clear, simply finding where the internal APIs
> have moved to and calling them under the new names is *not* what
> people should do. We can't stop people calling the internal APIs,
> obviously, but the idea of this change is to give people the incentive
> to find a supported approach, not just to annoy people who are doing
> things we don't want them to ;-)
> 
> So please - if you're calling pip's internals in your code, take the
> opportunity *now* to check out the in-development version of pip, and
> ensure your project will still work when pip 10 is released.
> 
> And many thanks to anyone else who helps by testing out the new
> version, as well :-)
> 
> Thanks,
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting dependecies of package from PyPiJSON

[Jannis Gebauer]

Unfortunately, yes there is no public API for a Python packages’ dependencies 
available.

> On 20. Jul 2017, at 15:43, Krzysiek Płachno  wrote:
> 
> None of them sends back required dependencies for package. 
> Does it mean - this information is not obtainable via JSON Api?
> 
> Best,
> Krzysztof
> 
> 2017-07-20 15:04 GMT+02:00 Jannis Gebauer  <mailto:ja@me.com>>:
> Hi Krzysztof,
> 
> Two options, e.g. for the “graphene” package:
> 
> - https://pypi.python.org/pypi/graphene/json 
> <https://pypi.python.org/pypi/graphene/json>
> - https://pypi.org/pypi/graphene/json <https://pypi.org/pypi/graphene/json>
> 
> Best,
> 
> Jannis
> 
>> On 20. Jul 2017, at 14:55, Krzysiek Płachno > <mailto:krzysiekplac...@gmail.com>> wrote:
>> 
>> Hi!
>> 
>> I'm contributing to XWiki open source project http://www.xwiki.org/ 
>> <http://www.xwiki.org/>. XWiki platform is written in Java and apart from 
>> many wonderful features allows for scripting in python on its pages. 
>> 
>> My current task is to enable installing packages form PyPi repository so 
>> that they can be later used in scripts. The installment is done in custom 
>> way making python packages available for Java classloader.
>> 
>> To make downloaded package working it's needed to install also dependencies. 
>> Is it possible to get dependencies information directly from PyPiJSON API? 
>> (e.g. by adding some request parameter or header in GET request) 
>> Or is it possible to this data in any other way (apart from downloading 
>> package)?
>> 
>> Thanks for help in advance!
>> 
>> Best,
>> Krzysztof
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org 
>> <mailto:Distutils-SIG@python.org>
>> https://mail.python.org/mailman/listinfo/distutils-sig 
>> <https://mail.python.org/mailman/listinfo/distutils-sig>
> 
> 

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Getting dependecies of package from PyPiJSON

[Jannis Gebauer]

Hi Krzysztof,

Two options, e.g. for the “graphene” package:

- https://pypi.python.org/pypi/graphene/json 

- https://pypi.org/pypi/graphene/json 

Best,

Jannis

> On 20. Jul 2017, at 14:55, Krzysiek Płachno  wrote:
> 
> Hi!
> 
> I'm contributing to XWiki open source project http://www.xwiki.org/ 
> . XWiki platform is written in Java and apart from 
> many wonderful features allows for scripting in python on its pages. 
> 
> My current task is to enable installing packages form PyPi repository so that 
> they can be later used in scripts. The installment is done in custom way 
> making python packages available for Java classloader.
> 
> To make downloaded package working it's needed to install also dependencies. 
> Is it possible to get dependencies information directly from PyPiJSON API? 
> (e.g. by adding some request parameter or header in GET request) 
> Or is it possible to this data in any other way (apart from downloading 
> package)?
> 
> Thanks for help in advance!
> 
> Best,
> Krzysztof
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Malicious packages on PyPI

[Jannis Gebauer]

> This makes me remember 
> https://hackernoon.com/building-a-botnet-on-pypi-be1ad280b8d6 
>  on a related 
> note.


Yep, that’s basically the same thing. Instead of using package names of 
builtins, the attacker is using a combination of popular apt/yum packages with 
a mix of package names with typos.

During development, it’s not uncommon to make mistakes like:

pip install requirements.txt (forgot the -r)
pip install requestd (typo)
pip install tkinter (not registered)

Or to use the wrong package manager (apt-get install python-dev vs. pip install 
python-dev). 

I wonder if it would make sense to build some kind of blacklist for this. 
According to the blog post there were close to 10k installs over a period of 
just three days. I believe Debian is running some kind of popularity contest 
for their packages which could be used to identify problematic packages. This 
will be a lot of manual work, but I’d work on a list like this.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] The sad and insecure state of commercial private package indexes

[Jannis Gebauer]

I did some research on commercial private package indexes, namely Gemfury and 
packagecloud.

Both of them recommend to use `--extra-index-url` as a parameter to point to 
their own index servers hosting the private package. This is blatantly insecure.

Using `--extra-index-url` tells pip to use the server as an _extra_ index url 
(huge surprise). This basically means that, during pip install, PyPi and the 
private server share the same namespace. Pip queries both servers for available 
releases for a given package. On unpinned packages, the server with the latest 
release seems to win.

This means, If I’m using one of these private package indexes, an attacker is 
able to run arbitrary Python code (through setup.py during installation) simply 
by guessing my private package names and uploading them to PyPi.

I’ve contacted both Gemfury and packagecloud. Gemfury didn’t respond. 
Packagecloud basically said works as intended, wontfix. They could, of course, 
fix this very easily by running their own PyPi mirrors. 

I couldn’t care less about these companies, but I care about Python packaging 
in general. I talked to a couple of Python developers regarding this. All of 
them use pip and PyPi regularly but have no idea about the internals. This was 
a huge surprise to them. 

My problem with this is that PyPi and pip will look bad if this is ever going 
to be abused.

What are your thoughts on this?

—

Jannis Gebauer
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Data on requirement files on GitHub

[Jannis Gebauer]

Hi,

I ran a couple of queries against GitHubs public big query dataset [0] last 
week. I’m interested in requirement files in particular, so I ran a query 
extracting all available requirement files.

Since queries against this dataset are rather expensive ($7 on all repos), I 
thought I’d share the raw data here [1]. The data contains the repo name, the 
requirements file path and the contents of the file. Every line represents a 
JSON blob, read it with:

with open('data.json') as f:
for line in f.readlines():
data = json.loads(line)

Maybe that’s of interest to some of you.

If you have any ideas on what to do with the data, please let me know.

—

Jannis Gebauer



[0]: https://cloud.google.com/bigquery/public-data/github 
<https://cloud.google.com/bigquery/public-data/github>
[1]: https://github.com/jayfk/requirements-dataset 
<https://github.com/jayfk/requirements-dataset>___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig