Re: [Distutils] Good news everyone, PyPI is behind a CDN
On 28 May 2013 02:53, Donald Stufft don...@stufft.io wrote: Figured it out. Use HTTPS. Can I suggest that if the new CDN means that there are additional restrictions on what is supported (I've used the XMLRPC API without https in one-off scripts in the past) then the officially supported API should be properly documented once and for all in a PEP, including some sort of what's new or rationale section describing the various changes that have occurred recently and their impact on user code? I'm purely a casual user of the PyPI API and the discussion of these changes haa mostly gone over my head. The one thing I've taken away from it is that I may get problems if I just google for sample code to use. For example, the above comment implies that http://wiki.python.org/moin/PyPIXmlRpc (AIUI, the nearest to formal documentation that the XMLRPC API has) is wrong (as it uses http). I do appreciate all the work that is going on to improve the PyPI infrastructure. I'm not saying the changes should be reverted, just that the consequences should be clearly explained. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Good news everyone, PyPI is behind a CDN
On Tue, May 28, 2013 at 07:42 +0100, Paul Moore wrote: On 28 May 2013 02:53, Donald Stufft don...@stufft.io wrote: Figured it out. Use HTTPS. Can I suggest that if the new CDN means that there are additional restrictions on what is supported (I've used the XMLRPC API without https in one-off scripts in the past) then the officially supported API should be properly documented once and for all in a PEP, including some sort of what's new or rationale section describing the various changes that have occurred recently and their impact on user code? I second this. I am building tools that interact with PyPI and people and customers are using them. I don't want to find a switch announced which breaks them and then hear sorry, that's the future now without this future being documented and discussed before the fact. The PyPI infrastructure and its supported tool interactions today are as important as evolving the language itself so PEPs are warranted. As with PEP438 i am willing to help this process. I'm purely a casual user of the PyPI API and the discussion of these changes haa mostly gone over my head. The one thing I've taken away from it is that I may get problems if I just google for sample code to use. For example, the above comment implies that http://wiki.python.org/moin/PyPIXmlRpc (AIUI, the nearest to formal documentation that the XMLRPC API has) is wrong (as it uses http). I do appreciate all the work that is going on to improve the PyPI infrastructure. I'm not saying the changes should be reverted, just that the consequences should be clearly explained. I also appreciate Noah's and Donald's CDN work here, up to the point where it breaks things for unclear reasons. Reasons which might very well be valid, nevertheless! best, holger ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Good news everyone, PyPI is behind a CDN
On May 28, 2013, at 2:42 AM, Paul Moore p.f.mo...@gmail.com wrote: On 28 May 2013 02:53, Donald Stufft don...@stufft.io wrote: Figured it out. Use HTTPS. Can I suggest that if the new CDN means that there are additional restrictions on what is supported (I've used the XMLRPC API without https in one-off scripts in the past) then the officially supported API should be properly documented once and for all in a PEP, including some sort of what's new or rationale section describing the various changes that have occurred recently and their impact on user code? I'm purely a casual user of the PyPI API and the discussion of these changes haa mostly gone over my head. The one thing I've taken away from it is that I may get problems if I just google for sample code to use. For example, the above comment implies that http://wiki.python.org/moin/PyPIXmlRpc (AIUI, the nearest to formal documentation that the XMLRPC API has) is wrong (as it uses http). I do appreciate all the work that is going on to improve the PyPI infrastructure. I'm not saying the changes should be reverted, just that the consequences should be clearly explained. Paul. To be quite honest the HTTP 1.0 + HTTP issue simply wasn't discovered in testing. The http url works fine on Python 2.7 (which I'm assuming uses HTTP 1.1). I'm not completely happy that HTTP is broken in Python2.6 (and I'm assuming earlier) and have it on my list to see if there's anything that can be done. THat being said the most future compatible way will be to use the HTTPS url for any interaction (and ideally verify the SSL, but the built in XMLRPC library doesn't do that). My Use HTTPS was more to speak how to solve the issue *right now*. Documentation should be updated to point to HTTPS though. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Good news everyone, PyPI is behind a CDN
On May 28, 2013, at 2:57 AM, holger krekel hol...@merlinux.eu wrote: On Tue, May 28, 2013 at 07:42 +0100, Paul Moore wrote: On 28 May 2013 02:53, Donald Stufft don...@stufft.io wrote: Figured it out. Use HTTPS. Can I suggest that if the new CDN means that there are additional restrictions on what is supported (I've used the XMLRPC API without https in one-off scripts in the past) then the officially supported API should be properly documented once and for all in a PEP, including some sort of what's new or rationale section describing the various changes that have occurred recently and their impact on user code? I second this. I am building tools that interact with PyPI and people and customers are using them. I don't want to find a switch announced which breaks them and then hear sorry, that's the future now without this future being documented and discussed before the fact. The PyPI infrastructure and its supported tool interactions today are as important as evolving the language itself so PEPs are warranted. As with PEP438 i am willing to help this process. I'm purely a casual user of the PyPI API and the discussion of these changes haa mostly gone over my head. The one thing I've taken away from it is that I may get problems if I just google for sample code to use. For example, the above comment implies that http://wiki.python.org/moin/PyPIXmlRpc (AIUI, the nearest to formal documentation that the XMLRPC API has) is wrong (as it uses http). I do appreciate all the work that is going on to improve the PyPI infrastructure. I'm not saying the changes should be reverted, just that the consequences should be clearly explained. I also appreciate Noah's and Donald's CDN work here, up to the point where it breaks things for unclear reasons. Reasons which might very well be valid, nevertheless! best, holger Moving to a CDN has been discussed before on either catalog-sig or distutils-sig (Can't recall which offhand). Weekly status updates were posted to the infrastructure list as well as the communication between us and Fastly as we ironed out SSL issues. The mirroring issue pre-invalidation was quickly corrected. We now invalidate and we are looking at a window that is at most a few seconds large. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Good news everyone, PyPI is behind a CDN
On Tue, May 28, 2013 at 10:07 PM, Donald Stufft don...@stufft.io wrote: Moving to a CDN has been discussed before on either catalog-sig or distutils-sig (Can't recall which offhand). Weekly status updates were posted to the infrastructure list as well as the communication between us and Fastly as we ironed out SSL issues. The mirroring issue pre-invalidation was quickly corrected. We now invalidate and we are looking at a window that is at most a few seconds large. One of the things I (successfully) advocated for at PyCon US was to open up the PEP process to cover things where python-dev aren't directly involved, but we need an official avenue for publication of significant changes in the Python ecosystem (with my main aim being to empower distutils-sig as a place where we could actually making final decisions about the evolution of the packaging ecosystem). Given that, an Informational PEP with Discussion-To set to infrastructure-sig and Noah as BDFL-Delegate would be an eminently suitable way of keeping PyPI users and mirror operators that *aren't* following infrastructure-sig informed of upcoming changes that may impact the operation of PyPI clients. infrastructure-sig has historically just been for backend hosting details, without significant impact to *client* facing behaviour - while I think it's fine to change that, it's also understandable that most developers of PyPI clients wouldn't be aware of upcoming changes that have only been discussed in detail on that list. So, as Holger said, great work and thanks for your efforts, but good communication does matter with these things. People don't like surprises, even well intentioned ones :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Good news everyone, PyPI is behind a CDN
On Mon, May 27, 2013 at 1:19 AM, Lennart Regebro rege...@gmail.com wrote: On Sun, May 26, 2013 at 7:34 PM, Noah Kantrowitz n...@coderanger.net wrote: /farnsworth but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. This is going to spell disaster for the coffee industry, as you no longer have to take a coffee break when re-running a buildout. Thanks! I always test pip installation from PyPI just in case after uploading a new package, so the new cache delay still leaves some time for a coffee break (until Daniel gets the cache invalidation integrated :/). But yes, so many hoorays for this \o/ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Good news everyone, PyPI is behind a CDN
Noah Kantrowitz n...@coderanger.net writes: /farnsworth but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. --Noah the xmlrpc api is broken when using http 1.0. the second call to curl uses http/1.0 and returns an empty response: $ cat body.txt EOF ?xml version='1.0'? methodCall methodNamepackage_releases/methodName params param valuestringe/string/value /param /params /methodCall EOF $ curl -X POST -d @body.txt http://pypi.python.org/pypi --header Content-Type:text/xml ?xml version='1.0'? methodResponse params param valuearraydata valuestring1.4.5/string/value /data/array/value /param /params /methodResponse $ curl -0 -X POST -d @body.txt http://pypi.python.org/pypi --header Content-Type:text/xml $ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Good news everyone, PyPI is behind a CDN
On May 27, 2013, at 2:21 PM, Ralf Schmitt r...@systemexit.de wrote: Noah Kantrowitz n...@coderanger.net writes: /farnsworth but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. --Noah the xmlrpc api is broken when using http 1.0. the second call to curl uses http/1.0 and returns an empty response: $ cat body.txt EOF ?xml version='1.0'? methodCall methodNamepackage_releases/methodName params param valuestringe/string/value /param /params /methodCall EOF $ curl -X POST -d @body.txt http://pypi.python.org/pypi --header Content-Type:text/xml ?xml version='1.0'? methodResponse params param valuearraydata valuestring1.4.5/string/value /data/array/value /param /params /methodResponse $ curl -0 -X POST -d @body.txt http://pypi.python.org/pypi --header Content-Type:text/xml $ We have not supported HTTP 1.0 for quite some time. Even before the CDN move, we used the Host header to route between different HAProxy server blocks on the load balancers. I'm unaware of any reason people would be using HTTP 1.0 clients at this point, HTTP 1.1 has been a standard for 14 years now. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Good news everyone, PyPI is behind a CDN
Noah Kantrowitz n...@coderanger.net writes: the xmlrpc api is broken when using http 1.0. the second call to curl uses http/1.0 and returns an empty response: $ cat body.txt EOF ?xml version='1.0'? methodCall methodNamepackage_releases/methodName params param valuestringe/string/value /param /params /methodCall EOF $ curl -X POST -d @body.txt http://pypi.python.org/pypi --header Content-Type:text/xml ?xml version='1.0'? methodResponse params param valuearraydata valuestring1.4.5/string/value /data/array/value /param /params /methodResponse $ curl -0 -X POST -d @body.txt http://pypi.python.org/pypi --header Content-Type:text/xml $ We have not supported HTTP 1.0 for quite some time. well, it has been working until at least last friday. Even before the CDN move, we used the Host header to route between different HAProxy server blocks on the load balancers. I'm unaware of any reason people would be using HTTP 1.0 clients at this point, HTTP 1.1 has been a standard for 14 years now. the fact that you are unaware of any reason people would be using HTTP 1.0 is not a good reason not to support it. python 2.6 xmlrpclib uses HTTP/1.0! -- Cheers Ralf ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Good news everyone, PyPI is behind a CDN
On May 27, 2013, at 6:02 PM, Ralf Schmitt r...@systemexit.de wrote: Noah Kantrowitz n...@coderanger.net writes: the xmlrpc api is broken when using http 1.0. the second call to curl uses http/1.0 and returns an empty response: $ cat body.txt EOF ?xml version='1.0'? methodCall methodNamepackage_releases/methodName params param valuestringe/string/value /param /params /methodCall EOF $ curl -X POST -d @body.txt http://pypi.python.org/pypi --header Content-Type:text/xml ?xml version='1.0'? methodResponse params param valuearraydata valuestring1.4.5/string/value /data/array/value /param /params /methodResponse $ curl -0 -X POST -d @body.txt http://pypi.python.org/pypi --header Content-Type:text/xml $ We have not supported HTTP 1.0 for quite some time. well, it has been working until at least last friday. Even before the CDN move, we used the Host header to route between different HAProxy server blocks on the load balancers. I'm unaware of any reason people would be using HTTP 1.0 clients at this point, HTTP 1.1 has been a standard for 14 years now. the fact that you are unaware of any reason people would be using HTTP 1.0 is not a good reason not to support it. python 2.6 xmlrpclib uses HTTP/1.0! Maybe I misunderstand, but xmlrpclib on Python2.6 and the new CDN work fine together. $ python2.6 (pypi) Python 2.6.7 (r267:88850, Oct 11 2012, 20:15:00) [GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin Type help, copyright, credits or license for more information. import xmlrpclib s = xmlrpclib.ServerProxy(https://pypi.python.org/pypi;) s ServerProxy for pypi.python.org/pypi s.user_packages(dstufft) [...] # Just incase we aren't getting the CDN on pypi.python.org s = xmlrpclib.ServerProxy(https://pypi.a.ssl.fastly.net/pypi;) s.user_packages(dstufft) [...] -- Cheers Ralf ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Good news everyone, PyPI is behind a CDN
On May 27, 2013, at 9:33 PM, Donald Stufft don...@stufft.io wrote: On May 27, 2013, at 6:02 PM, Ralf Schmitt r...@systemexit.de wrote: Noah Kantrowitz n...@coderanger.net writes: the xmlrpc api is broken when using http 1.0. the second call to curl uses http/1.0 and returns an empty response: $ cat body.txt EOF ?xml version='1.0'? methodCall methodNamepackage_releases/methodName params param valuestringe/string/value /param /params /methodCall EOF $ curl -X POST -d @body.txt http://pypi.python.org/pypi --header Content-Type:text/xml ?xml version='1.0'? methodResponse params param valuearraydata valuestring1.4.5/string/value /data/array/value /param /params /methodResponse $ curl -0 -X POST -d @body.txt http://pypi.python.org/pypi --header Content-Type:text/xml $ We have not supported HTTP 1.0 for quite some time. well, it has been working until at least last friday. Even before the CDN move, we used the Host header to route between different HAProxy server blocks on the load balancers. I'm unaware of any reason people would be using HTTP 1.0 clients at this point, HTTP 1.1 has been a standard for 14 years now. the fact that you are unaware of any reason people would be using HTTP 1.0 is not a good reason not to support it. python 2.6 xmlrpclib uses HTTP/1.0! Maybe I misunderstand, but xmlrpclib on Python2.6 and the new CDN work fine together. $ python2.6 (pypi) Python 2.6.7 (r267:88850, Oct 11 2012, 20:15:00) [GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin Type help, copyright, credits or license for more information. import xmlrpclib s = xmlrpclib.ServerProxy(https://pypi.python.org/pypi;) s ServerProxy for pypi.python.org/pypi s.user_packages(dstufft) [...] # Just incase we aren't getting the CDN on pypi.python.org s = xmlrpclib.ServerProxy(https://pypi.a.ssl.fastly.net/pypi;) s.user_packages(dstufft) [...] -- Cheers Ralf ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig Figured it out. Use HTTPS. $ curl -0 -X POST -d @body.txt https://pypi.python.org/pypi --header Content-Type:text/xml ?xml version='1.0'? methodResponse params param valuearraydata valuestring1.4.5/string/value /data/array/value /param /params /methodResponse - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Good news everyone, PyPI is behind a CDN
/farnsworth but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Good news everyone, PyPI is behind a CDN
On Sun, May 26, 2013 at 7:34 PM, Noah Kantrowitz n...@coderanger.net wrote: /farnsworth but seriously, at long last today it was my honor to throw the DNS switch to move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for doing much of the heavy lifting on the PyPI side, and to Fastly for graciously offering to host us. What does this mean for everyone? Well the biggest change is PyPI should get a whole lot faster. There are two major downsides however. There will now be a delay of several minutes in some cases between updating a package and having it be installable, and download counts will now be even more incorrect than they were before. The PyPI admins are discussing what to do about download counts long-term, but for now we all feel that the performance and availability benefits outweigh the loss. If anyone has any questions, or hears anything about issues with PyPI please don't hesitate to contact me. This is going to spell disaster for the coffee industry, as you no longer have to take a coffee break when re-running a buildout. Thanks! //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
