-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 05/13/2015 08:21 AM, Ashby, Jason (IMS) wrote: > I’m looking to set up a pypi mirror with pulp. I’m currently > using Bandersnatch for this, but it’d be nice to drop it and use > pulp instead. Per the docs*, I see that you can sync specific > packages from pypi, e.g. > > pulp-admin python repo create --repo-id pypi --feed > https://pypi.python.org/ --package-names numpy,scipy > > but I can’t seem to sync ALL packages. I tried leaving off the > --package-names option, but a sync downloads 0 packages. Should > I submit an issue/feature request at > https://pulp.plan.io/projects/pulp_python/issues?
Hi Jason! The problem is that PyPI does not have one single manifest file for the available package versions, but rather one manifest per package name. Due to this, in order to sync all packages from PyPI it would be necessary to make around 45-50,000 web requests just to find out what would need to be downloaded, and then of course we would need to perform the actual package downloads. That said, we are working on a plan to have Pulp be able to lazy fetch packages as they are requested. This plan will take a long time to implement (so don't expect it in any of our close releases) but I think it will solve this problem in a performant way. Another possible solution may be Warehouse[0]. I've been talking to the PyPA developers about this problem, and they are aware that it needs to be solved. They may fix it there, in which case we can get the Python importer to be aware of all the packages. I have also considered just doing the 50k requests anyway. I suspect that PyPI won't like if we do that, but it is technically possible as well. I say go ahead and file an RFE. I'll think some more about how we might be able to get it working. Thanks for the note, and I hope you enjoy the plugin otherwise! [0] https://warehouse.python.org/ - -- Randy Barlow -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCgAGBQJVU0rsAAoJEIyFaKUJtmpiaUMP/1YfE0nNIum6EQSobZ0UPUDn Ji9OiDLjsWDras0JEFdDOg8IS6Go2IZ3LThIOKIHXMNvMk3OAIC93dfbukRusvP3 BoyYlpZIyPS+24hbP7VPeLVcRzhC6NAf36rCdDjLtrGZGdfs2xK6h3YoPsdigqzK 9jhFFP6Det9yhqyL4IgLyCps7BrmpnQ9zHR9jbGjl0afIRhgAEjZaglBUlhXql0W RIji9tsW3a2Cj8y/N472fY9YL9RKiossCMTw2pMY3+87zt7ihHHRuzxye4ZZixRf PDDKfWLzDawxZyWbvKEvTdTYNeoYQ0YvHQ7D7q/x00TYm2wT/e2xsKfQcIxkzp6s 7eN3E76D7qlGrKjU2T9Jemxj6A+qidTc6AUcVyWxGb3dseRJ+vuhfzvpNuWGLSVn N16GnEgesUn2Dl1aY763CHntlBW2mrmCrRjWAXp6s+Y62f2DNo7yN8mRcPfeSn6J ypzo3CWv0C6PmK61RKZ7zkvTyYOi9jD+gyTAO3e9UsQNHdUDpsMW1ZGu2uaPzBLc j578fcuspLsVYML0WKpg7wBsRSOy9O3++5XPQgPm1TUP6gm9jRaM+xNlvrZ0B0hf mt+z9+Lf5A/I7j8/kcJ+3oTpsLovb7CQRRjagbM4W5Al3WQdVueH1gcy2YAv0zbf XtYScIpLWhZxHHuaGgyw =OD1J -----END PGP SIGNATURE----- _______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list