Re: Stopping nightly releases to Pypi
As we have many users in China, I'm considering the accessibility of S3. For pip, we can mirrors. On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard wrote: > I would like to remind everyone that lazy consensus is assumed if no > objections > are raised before 2019-12-05 at 05:42 UTC. There has been some discussion > about > the proposal, but to my understanding no objections were raised. > > If the proposal is accepted, MXNet releases would be installed via > >pip install mxnet > > And release candidates via > > pip install --pre mxnet > > (or with the respective cuda version specifier appended etc.) > > To obtain releases built automatically from the master branch, users would > need > to specify something like "-f > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to pip. > > Best regards > Leonard > > On Mon, 2019-12-02 at 05:42 +, Lausen, Leonard wrote: > > Hi MXNet Community, > > > > since more than 2 months our binary Python nightly releases published on > Pypi > > are broken. The problem is that our binaries exceed Pypi's size limit. > > Decreasing the binary size by adding compression breaks third-party > libraries > > loading libmxnet.so > https://github.com/apache/incubator-mxnet/issues/16193 > > > > Sheng requested Pypi to increase their size limit: > > https://github.com/pypa/pypi-support/issues/50 > > > > Currently "the biggest cost for PyPI from [the many MXNet binaries with > > nightly > > release to Pypi] is the bandwidth consumed when several hundred mirrors > > attempt > > to mirror each release immediately after it's published". So Pypi is not > > inclined to allow us to upload even larger binaries on a nightly > schedule. > > Their compromise is to allow it on a weekly cadence. > > > > However, I would like the community to revisit the necessity of > releasing pre- > > release binaries to Pypi on a nightly (or weekly) cadence. Instead, we > can > > release nightly binaries ONLY to a public S3 bucket and instruct users to > > install from there. On our side, we only need to prepare a html document > that > > contains links to all released nightly binaries. > > Finally users will install the nightly releases via > > > > pip install --pre mxnet-cu101 -f > http://mxnet.s3.amazonaws.com/mxnet-cu101/ > > nightly.html > > > > Instead of > > > > pip install --pre mxnet-cu101 > > > > Of course proper releases and release candidates should still be made > > available > > via Pypi. Thus releases would be installed via > > > > pip install mxnet-cu101 > > > > And release candidates via > > > > pip install --pre mxnet-cu101 > > > > This will substantially reduce the costs of the Pypi project and in fact > > matches > > the installation experience provided by PyTorch. I don't think the > benefit of > > not including "-f http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html > " > > matches the costs we currently externalize to the Pypi team. > > > > This suggestion seems uncontroversial to me. Thus I would like to start > lazy > > consensus. If there are no objections, I will assume lazy consensus on > > stopping > > nightly releases to Pypi in 72hrs. > > > > Best regards > > Leonard >
Re: Stopping nightly releases to Pypi
* For pypi, we can use mirrors. On Tue, Dec 3, 2019 at 9:28 PM Tao Lv wrote: > As we have many users in China, I'm considering the accessibility of S3. > For pip, we can mirrors. > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > wrote: > >> I would like to remind everyone that lazy consensus is assumed if no >> objections >> are raised before 2019-12-05 at 05:42 UTC. There has been some discussion >> about >> the proposal, but to my understanding no objections were raised. >> >> If the proposal is accepted, MXNet releases would be installed via >> >>pip install mxnet >> >> And release candidates via >> >> pip install --pre mxnet >> >> (or with the respective cuda version specifier appended etc.) >> >> To obtain releases built automatically from the master branch, users >> would need >> to specify something like "-f >> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to pip. >> >> Best regards >> Leonard >> >> On Mon, 2019-12-02 at 05:42 +, Lausen, Leonard wrote: >> > Hi MXNet Community, >> > >> > since more than 2 months our binary Python nightly releases published >> on Pypi >> > are broken. The problem is that our binaries exceed Pypi's size limit. >> > Decreasing the binary size by adding compression breaks third-party >> libraries >> > loading libmxnet.so >> https://github.com/apache/incubator-mxnet/issues/16193 >> > >> > Sheng requested Pypi to increase their size limit: >> > https://github.com/pypa/pypi-support/issues/50 >> > >> > Currently "the biggest cost for PyPI from [the many MXNet binaries with >> > nightly >> > release to Pypi] is the bandwidth consumed when several hundred mirrors >> > attempt >> > to mirror each release immediately after it's published". So Pypi is not >> > inclined to allow us to upload even larger binaries on a nightly >> schedule. >> > Their compromise is to allow it on a weekly cadence. >> > >> > However, I would like the community to revisit the necessity of >> releasing pre- >> > release binaries to Pypi on a nightly (or weekly) cadence. Instead, we >> can >> > release nightly binaries ONLY to a public S3 bucket and instruct users >> to >> > install from there. On our side, we only need to prepare a html >> document that >> > contains links to all released nightly binaries. >> > Finally users will install the nightly releases via >> > >> > pip install --pre mxnet-cu101 -f >> http://mxnet.s3.amazonaws.com/mxnet-cu101/ >> > nightly.html >> > >> > Instead of >> > >> > pip install --pre mxnet-cu101 >> > >> > Of course proper releases and release candidates should still be made >> > available >> > via Pypi. Thus releases would be installed via >> > >> > pip install mxnet-cu101 >> > >> > And release candidates via >> > >> > pip install --pre mxnet-cu101 >> > >> > This will substantially reduce the costs of the Pypi project and in fact >> > matches >> > the installation experience provided by PyTorch. I don't think the >> benefit of >> > not including "-f >> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"; >> > matches the costs we currently externalize to the Pypi team. >> > >> > This suggestion seems uncontroversial to me. Thus I would like to start >> lazy >> > consensus. If there are no objections, I will assume lazy consensus on >> > stopping >> > nightly releases to Pypi in 72hrs. >> > >> > Best regards >> > Leonard >> >
Re: CI Update
Hi MXNet community. We are in the process of updating the base AMIs for CI with an updated CUDA driver to fix the CI blockage. We would need help from the community to diagnose some of the build errors which don't seem related to the infrastructure. I have observed this build failure with tvm when not installing the cuda driver in the container: https://pastebin.com/bQA0W2U4 centos gpu builds and tests seem to run with the updated AMI and changes to the container. Thanks. On Mon, Dec 2, 2019 at 12:11 PM Pedro Larroy wrote: > Small update about CI, which is blocked. > > Seems there's a nvidia driver compatibility problem in the base AMI that > is running in GPU instances and the nvidia docker images that we use for > building and testing. > > We are working on providing a fix by updating the base images as doesn't > seem to be easy to fix by just changing the container. > > Thanks. > > Pedro. >
Re: CI Update
Also please take note that there's a stage building TVM which is executing compilation serially and takes a lot of time which impacts CI turnaround time: https://github.com/apache/incubator-mxnet/issues/16962 Pedro On Tue, Dec 3, 2019 at 9:49 AM Pedro Larroy wrote: > Hi MXNet community. We are in the process of updating the base AMIs for CI > with an updated CUDA driver to fix the CI blockage. > > We would need help from the community to diagnose some of the build errors > which don't seem related to the infrastructure. > > I have observed this build failure with tvm when not installing the cuda > driver in the container: > > > https://pastebin.com/bQA0W2U4 > > centos gpu builds and tests seem to run with the updated AMI and changes > to the container. > > > Thanks. > > > On Mon, Dec 2, 2019 at 12:11 PM Pedro Larroy > wrote: > >> Small update about CI, which is blocked. >> >> Seems there's a nvidia driver compatibility problem in the base AMI that >> is running in GPU instances and the nvidia docker images that we use for >> building and testing. >> >> We are working on providing a fix by updating the base images as doesn't >> seem to be easy to fix by just changing the container. >> >> Thanks. >> >> Pedro. >> >
Re: Stopping nightly releases to Pypi
Isn't there an s3 endpoint in Beijing? It seems like this topic still warrants some discussion and thus I'd prefer if we don't move forward with lazy consensus. -Marco Tao Lv schrieb am Di., 3. Dez. 2019, 14:31: > * For pypi, we can use mirrors. > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv wrote: > > > As we have many users in China, I'm considering the accessibility of S3. > > For pip, we can mirrors. > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > > > wrote: > > > >> I would like to remind everyone that lazy consensus is assumed if no > >> objections > >> are raised before 2019-12-05 at 05:42 UTC. There has been some > discussion > >> about > >> the proposal, but to my understanding no objections were raised. > >> > >> If the proposal is accepted, MXNet releases would be installed via > >> > >>pip install mxnet > >> > >> And release candidates via > >> > >> pip install --pre mxnet > >> > >> (or with the respective cuda version specifier appended etc.) > >> > >> To obtain releases built automatically from the master branch, users > >> would need > >> to specify something like "-f > >> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to pip. > >> > >> Best regards > >> Leonard > >> > >> On Mon, 2019-12-02 at 05:42 +, Lausen, Leonard wrote: > >> > Hi MXNet Community, > >> > > >> > since more than 2 months our binary Python nightly releases published > >> on Pypi > >> > are broken. The problem is that our binaries exceed Pypi's size limit. > >> > Decreasing the binary size by adding compression breaks third-party > >> libraries > >> > loading libmxnet.so > >> https://github.com/apache/incubator-mxnet/issues/16193 > >> > > >> > Sheng requested Pypi to increase their size limit: > >> > https://github.com/pypa/pypi-support/issues/50 > >> > > >> > Currently "the biggest cost for PyPI from [the many MXNet binaries > with > >> > nightly > >> > release to Pypi] is the bandwidth consumed when several hundred > mirrors > >> > attempt > >> > to mirror each release immediately after it's published". So Pypi is > not > >> > inclined to allow us to upload even larger binaries on a nightly > >> schedule. > >> > Their compromise is to allow it on a weekly cadence. > >> > > >> > However, I would like the community to revisit the necessity of > >> releasing pre- > >> > release binaries to Pypi on a nightly (or weekly) cadence. Instead, we > >> can > >> > release nightly binaries ONLY to a public S3 bucket and instruct users > >> to > >> > install from there. On our side, we only need to prepare a html > >> document that > >> > contains links to all released nightly binaries. > >> > Finally users will install the nightly releases via > >> > > >> > pip install --pre mxnet-cu101 -f > >> http://mxnet.s3.amazonaws.com/mxnet-cu101/ > >> > nightly.html > >> > > >> > Instead of > >> > > >> > pip install --pre mxnet-cu101 > >> > > >> > Of course proper releases and release candidates should still be made > >> > available > >> > via Pypi. Thus releases would be installed via > >> > > >> > pip install mxnet-cu101 > >> > > >> > And release candidates via > >> > > >> > pip install --pre mxnet-cu101 > >> > > >> > This will substantially reduce the costs of the Pypi project and in > fact > >> > matches > >> > the installation experience provided by PyTorch. I don't think the > >> benefit of > >> > not including "-f > >> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"; > >> > matches the costs we currently externalize to the Pypi team. > >> > > >> > This suggestion seems uncontroversial to me. Thus I would like to > start > >> lazy > >> > consensus. If there are no objections, I will assume lazy consensus on > >> > stopping > >> > nightly releases to Pypi in 72hrs. > >> > > >> > Best regards > >> > Leonard > >> > > >
Re: Stopping nightly releases to Pypi
Yes, there is. We can also make it easier to access by using a geo-location based DNS server so that China users are directed to that local mirror. The rest of the world is already covered by the global cloudfront. -sz On 2019/12/03 18:22:22, Marco de Abreu wrote: > Isn't there an s3 endpoint in Beijing? > > It seems like this topic still warrants some discussion and thus I'd prefer > if we don't move forward with lazy consensus. > > -Marco > > Tao Lv schrieb am Di., 3. Dez. 2019, 14:31: > > > * For pypi, we can use mirrors. > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv wrote: > > > > > As we have many users in China, I'm considering the accessibility of S3. > > > For pip, we can mirrors. > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > > > > > wrote: > > > > > >> I would like to remind everyone that lazy consensus is assumed if no > > >> objections > > >> are raised before 2019-12-05 at 05:42 UTC. There has been some > > discussion > > >> about > > >> the proposal, but to my understanding no objections were raised. > > >> > > >> If the proposal is accepted, MXNet releases would be installed via > > >> > > >>pip install mxnet > > >> > > >> And release candidates via > > >> > > >> pip install --pre mxnet > > >> > > >> (or with the respective cuda version specifier appended etc.) > > >> > > >> To obtain releases built automatically from the master branch, users > > >> would need > > >> to specify something like "-f > > >> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to pip. > > >> > > >> Best regards > > >> Leonard > > >> > > >> On Mon, 2019-12-02 at 05:42 +, Lausen, Leonard wrote: > > >> > Hi MXNet Community, > > >> > > > >> > since more than 2 months our binary Python nightly releases published > > >> on Pypi > > >> > are broken. The problem is that our binaries exceed Pypi's size limit. > > >> > Decreasing the binary size by adding compression breaks third-party > > >> libraries > > >> > loading libmxnet.so > > >> https://github.com/apache/incubator-mxnet/issues/16193 > > >> > > > >> > Sheng requested Pypi to increase their size limit: > > >> > https://github.com/pypa/pypi-support/issues/50 > > >> > > > >> > Currently "the biggest cost for PyPI from [the many MXNet binaries > > with > > >> > nightly > > >> > release to Pypi] is the bandwidth consumed when several hundred > > mirrors > > >> > attempt > > >> > to mirror each release immediately after it's published". So Pypi is > > not > > >> > inclined to allow us to upload even larger binaries on a nightly > > >> schedule. > > >> > Their compromise is to allow it on a weekly cadence. > > >> > > > >> > However, I would like the community to revisit the necessity of > > >> releasing pre- > > >> > release binaries to Pypi on a nightly (or weekly) cadence. Instead, we > > >> can > > >> > release nightly binaries ONLY to a public S3 bucket and instruct users > > >> to > > >> > install from there. On our side, we only need to prepare a html > > >> document that > > >> > contains links to all released nightly binaries. > > >> > Finally users will install the nightly releases via > > >> > > > >> > pip install --pre mxnet-cu101 -f > > >> http://mxnet.s3.amazonaws.com/mxnet-cu101/ > > >> > nightly.html > > >> > > > >> > Instead of > > >> > > > >> > pip install --pre mxnet-cu101 > > >> > > > >> > Of course proper releases and release candidates should still be made > > >> > available > > >> > via Pypi. Thus releases would be installed via > > >> > > > >> > pip install mxnet-cu101 > > >> > > > >> > And release candidates via > > >> > > > >> > pip install --pre mxnet-cu101 > > >> > > > >> > This will substantially reduce the costs of the Pypi project and in > > fact > > >> > matches > > >> > the installation experience provided by PyTorch. I don't think the > > >> benefit of > > >> > not including "-f > > >> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"; > > >> > matches the costs we currently externalize to the Pypi team. > > >> > > > >> > This suggestion seems uncontroversial to me. Thus I would like to > > start > > >> lazy > > >> > consensus. If there are no objections, I will assume lazy consensus on > > >> > stopping > > >> > nightly releases to Pypi in 72hrs. > > >> > > > >> > Best regards > > >> > Leonard > > >> > > > > > >
Re: Stopping nightly releases to Pypi
Excellent! Could we maybe come up with a POC and a quick writeup and then start a proper vote after everyone verified that it covers their use-cases? -Marco Sheng Zha schrieb am Di., 3. Dez. 2019, 19:24: > Yes, there is. We can also make it easier to access by using a > geo-location based DNS server so that China users are directed to that > local mirror. The rest of the world is already covered by the global > cloudfront. > > -sz > > On 2019/12/03 18:22:22, Marco de Abreu wrote: > > Isn't there an s3 endpoint in Beijing? > > > > It seems like this topic still warrants some discussion and thus I'd > prefer > > if we don't move forward with lazy consensus. > > > > -Marco > > > > Tao Lv schrieb am Di., 3. Dez. 2019, 14:31: > > > > > * For pypi, we can use mirrors. > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv wrote: > > > > > > > As we have many users in China, I'm considering the accessibility of > S3. > > > > For pip, we can mirrors. > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > > > > > > > > wrote: > > > > > > > >> I would like to remind everyone that lazy consensus is assumed if no > > > >> objections > > > >> are raised before 2019-12-05 at 05:42 UTC. There has been some > > > discussion > > > >> about > > > >> the proposal, but to my understanding no objections were raised. > > > >> > > > >> If the proposal is accepted, MXNet releases would be installed via > > > >> > > > >>pip install mxnet > > > >> > > > >> And release candidates via > > > >> > > > >> pip install --pre mxnet > > > >> > > > >> (or with the respective cuda version specifier appended etc.) > > > >> > > > >> To obtain releases built automatically from the master branch, users > > > >> would need > > > >> to specify something like "-f > > > >> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to pip. > > > >> > > > >> Best regards > > > >> Leonard > > > >> > > > >> On Mon, 2019-12-02 at 05:42 +, Lausen, Leonard wrote: > > > >> > Hi MXNet Community, > > > >> > > > > >> > since more than 2 months our binary Python nightly releases > published > > > >> on Pypi > > > >> > are broken. The problem is that our binaries exceed Pypi's size > limit. > > > >> > Decreasing the binary size by adding compression breaks > third-party > > > >> libraries > > > >> > loading libmxnet.so > > > >> https://github.com/apache/incubator-mxnet/issues/16193 > > > >> > > > > >> > Sheng requested Pypi to increase their size limit: > > > >> > https://github.com/pypa/pypi-support/issues/50 > > > >> > > > > >> > Currently "the biggest cost for PyPI from [the many MXNet binaries > > > with > > > >> > nightly > > > >> > release to Pypi] is the bandwidth consumed when several hundred > > > mirrors > > > >> > attempt > > > >> > to mirror each release immediately after it's published". So Pypi > is > > > not > > > >> > inclined to allow us to upload even larger binaries on a nightly > > > >> schedule. > > > >> > Their compromise is to allow it on a weekly cadence. > > > >> > > > > >> > However, I would like the community to revisit the necessity of > > > >> releasing pre- > > > >> > release binaries to Pypi on a nightly (or weekly) cadence. > Instead, we > > > >> can > > > >> > release nightly binaries ONLY to a public S3 bucket and instruct > users > > > >> to > > > >> > install from there. On our side, we only need to prepare a html > > > >> document that > > > >> > contains links to all released nightly binaries. > > > >> > Finally users will install the nightly releases via > > > >> > > > > >> > pip install --pre mxnet-cu101 -f > > > >> http://mxnet.s3.amazonaws.com/mxnet-cu101/ > > > >> > nightly.html > > > >> > > > > >> > Instead of > > > >> > > > > >> > pip install --pre mxnet-cu101 > > > >> > > > > >> > Of course proper releases and release candidates should still be > made > > > >> > available > > > >> > via Pypi. Thus releases would be installed via > > > >> > > > > >> > pip install mxnet-cu101 > > > >> > > > > >> > And release candidates via > > > >> > > > > >> > pip install --pre mxnet-cu101 > > > >> > > > > >> > This will substantially reduce the costs of the Pypi project and > in > > > fact > > > >> > matches > > > >> > the installation experience provided by PyTorch. I don't think the > > > >> benefit of > > > >> > not including "-f > > > >> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"; > > > >> > matches the costs we currently externalize to the Pypi team. > > > >> > > > > >> > This suggestion seems uncontroversial to me. Thus I would like to > > > start > > > >> lazy > > > >> > consensus. If there are no objections, I will assume lazy > consensus on > > > >> > stopping > > > >> > nightly releases to Pypi in 72hrs. > > > >> > > > > >> > Best regards > > > >> > Leonard > > > >> > > > > > > > > > >
Re: CI Update
Some PRs were experiencing build timeouts in the past. I have diagnosed this to be a saturation of the EFS volume holding the compilation cache. Once CI is back online this problem is very likely to be solved and you should not see any more build timeout issues. On Tue, Dec 3, 2019 at 10:18 AM Pedro Larroy wrote: > Also please take note that there's a stage building TVM which is executing > compilation serially and takes a lot of time which impacts CI turnaround > time: > > https://github.com/apache/incubator-mxnet/issues/16962 > > Pedro > > On Tue, Dec 3, 2019 at 9:49 AM Pedro Larroy > wrote: > >> Hi MXNet community. We are in the process of updating the base AMIs for >> CI with an updated CUDA driver to fix the CI blockage. >> >> We would need help from the community to diagnose some of the build >> errors which don't seem related to the infrastructure. >> >> I have observed this build failure with tvm when not installing the cuda >> driver in the container: >> >> >> https://pastebin.com/bQA0W2U4 >> >> centos gpu builds and tests seem to run with the updated AMI and changes >> to the container. >> >> >> Thanks. >> >> >> On Mon, Dec 2, 2019 at 12:11 PM Pedro Larroy < >> pedro.larroy.li...@gmail.com> wrote: >> >>> Small update about CI, which is blocked. >>> >>> Seems there's a nvidia driver compatibility problem in the base AMI that >>> is running in GPU instances and the nvidia docker images that we use for >>> building and testing. >>> >>> We are working on providing a fix by updating the base images as doesn't >>> seem to be easy to fix by just changing the container. >>> >>> Thanks. >>> >>> Pedro. >>> >>
Re: Stopping nightly releases to Pypi
This is certainly one way to do it. However, the binary size limits our ability to publish pypi. So assuming that we want to have our binary on pypi still, we'd have to convince pypa to raise our limits. Thus, it seems to me that this hypothetical vote with respect to stopping nightly publish to pypi would likely only have one acceptable outcome. This is more of an emergency situation as an essential distribution channel is currently broken so I'm focusing on the POC for now. -sz On 2019/12/03 18:28:44, Marco de Abreu wrote: > Excellent! Could we maybe come up with a POC and a quick writeup and then > start a proper vote after everyone verified that it covers their use-cases? > > -Marco > > Sheng Zha schrieb am Di., 3. Dez. 2019, 19:24: > > > Yes, there is. We can also make it easier to access by using a > > geo-location based DNS server so that China users are directed to that > > local mirror. The rest of the world is already covered by the global > > cloudfront. > > > > -sz > > > > On 2019/12/03 18:22:22, Marco de Abreu wrote: > > > Isn't there an s3 endpoint in Beijing? > > > > > > It seems like this topic still warrants some discussion and thus I'd > > prefer > > > if we don't move forward with lazy consensus. > > > > > > -Marco > > > > > > Tao Lv schrieb am Di., 3. Dez. 2019, 14:31: > > > > > > > * For pypi, we can use mirrors. > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv wrote: > > > > > > > > > As we have many users in China, I'm considering the accessibility of > > S3. > > > > > For pip, we can mirrors. > > > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > > > > > > > > > > > wrote: > > > > > > > > > >> I would like to remind everyone that lazy consensus is assumed if no > > > > >> objections > > > > >> are raised before 2019-12-05 at 05:42 UTC. There has been some > > > > discussion > > > > >> about > > > > >> the proposal, but to my understanding no objections were raised. > > > > >> > > > > >> If the proposal is accepted, MXNet releases would be installed via > > > > >> > > > > >>pip install mxnet > > > > >> > > > > >> And release candidates via > > > > >> > > > > >> pip install --pre mxnet > > > > >> > > > > >> (or with the respective cuda version specifier appended etc.) > > > > >> > > > > >> To obtain releases built automatically from the master branch, users > > > > >> would need > > > > >> to specify something like "-f > > > > >> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to pip. > > > > >> > > > > >> Best regards > > > > >> Leonard > > > > >> > > > > >> On Mon, 2019-12-02 at 05:42 +, Lausen, Leonard wrote: > > > > >> > Hi MXNet Community, > > > > >> > > > > > >> > since more than 2 months our binary Python nightly releases > > published > > > > >> on Pypi > > > > >> > are broken. The problem is that our binaries exceed Pypi's size > > limit. > > > > >> > Decreasing the binary size by adding compression breaks > > third-party > > > > >> libraries > > > > >> > loading libmxnet.so > > > > >> https://github.com/apache/incubator-mxnet/issues/16193 > > > > >> > > > > > >> > Sheng requested Pypi to increase their size limit: > > > > >> > https://github.com/pypa/pypi-support/issues/50 > > > > >> > > > > > >> > Currently "the biggest cost for PyPI from [the many MXNet binaries > > > > with > > > > >> > nightly > > > > >> > release to Pypi] is the bandwidth consumed when several hundred > > > > mirrors > > > > >> > attempt > > > > >> > to mirror each release immediately after it's published". So Pypi > > is > > > > not > > > > >> > inclined to allow us to upload even larger binaries on a nightly > > > > >> schedule. > > > > >> > Their compromise is to allow it on a weekly cadence. > > > > >> > > > > > >> > However, I would like the community to revisit the necessity of > > > > >> releasing pre- > > > > >> > release binaries to Pypi on a nightly (or weekly) cadence. > > Instead, we > > > > >> can > > > > >> > release nightly binaries ONLY to a public S3 bucket and instruct > > users > > > > >> to > > > > >> > install from there. On our side, we only need to prepare a html > > > > >> document that > > > > >> > contains links to all released nightly binaries. > > > > >> > Finally users will install the nightly releases via > > > > >> > > > > > >> > pip install --pre mxnet-cu101 -f > > > > >> http://mxnet.s3.amazonaws.com/mxnet-cu101/ > > > > >> > nightly.html > > > > >> > > > > > >> > Instead of > > > > >> > > > > > >> > pip install --pre mxnet-cu101 > > > > >> > > > > > >> > Of course proper releases and release candidates should still be > > made > > > > >> > available > > > > >> > via Pypi. Thus releases would be installed via > > > > >> > > > > > >> > pip install mxnet-cu101 > > > > >> > > > > > >> > And release candidates via > > > > >> > > > > > >> > pip install --pre mxnet-cu101 > > > > >> > > > > > >> > This will substantially reduce the costs of the Pypi project and > > in > > > > fact > > > > >> > matches > > > > >>
Re: Stopping nightly releases to Pypi
What about cutting down on SMs as recommended by Kellen? Sheng Zha schrieb am Di., 3. Dez. 2019, 20:15: > This is certainly one way to do it. However, the binary size limits our > ability to publish pypi. So assuming that we want to have our binary on > pypi still, we'd have to convince pypa to raise our limits. Thus, it seems > to me that this hypothetical vote with respect to stopping nightly publish > to pypi would likely only have one acceptable outcome. > > This is more of an emergency situation as an essential distribution > channel is currently broken so I'm focusing on the POC for now. > > -sz > > On 2019/12/03 18:28:44, Marco de Abreu wrote: > > Excellent! Could we maybe come up with a POC and a quick writeup and then > > start a proper vote after everyone verified that it covers their > use-cases? > > > > -Marco > > > > Sheng Zha schrieb am Di., 3. Dez. 2019, 19:24: > > > > > Yes, there is. We can also make it easier to access by using a > > > geo-location based DNS server so that China users are directed to that > > > local mirror. The rest of the world is already covered by the global > > > cloudfront. > > > > > > -sz > > > > > > On 2019/12/03 18:22:22, Marco de Abreu > wrote: > > > > Isn't there an s3 endpoint in Beijing? > > > > > > > > It seems like this topic still warrants some discussion and thus I'd > > > prefer > > > > if we don't move forward with lazy consensus. > > > > > > > > -Marco > > > > > > > > Tao Lv schrieb am Di., 3. Dez. 2019, 14:31: > > > > > > > > > * For pypi, we can use mirrors. > > > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv wrote: > > > > > > > > > > > As we have many users in China, I'm considering the > accessibility of > > > S3. > > > > > > For pip, we can mirrors. > > > > > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> I would like to remind everyone that lazy consensus is assumed > if no > > > > > >> objections > > > > > >> are raised before 2019-12-05 at 05:42 UTC. There has been some > > > > > discussion > > > > > >> about > > > > > >> the proposal, but to my understanding no objections were raised. > > > > > >> > > > > > >> If the proposal is accepted, MXNet releases would be installed > via > > > > > >> > > > > > >>pip install mxnet > > > > > >> > > > > > >> And release candidates via > > > > > >> > > > > > >> pip install --pre mxnet > > > > > >> > > > > > >> (or with the respective cuda version specifier appended etc.) > > > > > >> > > > > > >> To obtain releases built automatically from the master branch, > users > > > > > >> would need > > > > > >> to specify something like "-f > > > > > >> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to > pip. > > > > > >> > > > > > >> Best regards > > > > > >> Leonard > > > > > >> > > > > > >> On Mon, 2019-12-02 at 05:42 +, Lausen, Leonard wrote: > > > > > >> > Hi MXNet Community, > > > > > >> > > > > > > >> > since more than 2 months our binary Python nightly releases > > > published > > > > > >> on Pypi > > > > > >> > are broken. The problem is that our binaries exceed Pypi's > size > > > limit. > > > > > >> > Decreasing the binary size by adding compression breaks > > > third-party > > > > > >> libraries > > > > > >> > loading libmxnet.so > > > > > >> https://github.com/apache/incubator-mxnet/issues/16193 > > > > > >> > > > > > > >> > Sheng requested Pypi to increase their size limit: > > > > > >> > https://github.com/pypa/pypi-support/issues/50 > > > > > >> > > > > > > >> > Currently "the biggest cost for PyPI from [the many MXNet > binaries > > > > > with > > > > > >> > nightly > > > > > >> > release to Pypi] is the bandwidth consumed when several > hundred > > > > > mirrors > > > > > >> > attempt > > > > > >> > to mirror each release immediately after it's published". So > Pypi > > > is > > > > > not > > > > > >> > inclined to allow us to upload even larger binaries on a > nightly > > > > > >> schedule. > > > > > >> > Their compromise is to allow it on a weekly cadence. > > > > > >> > > > > > > >> > However, I would like the community to revisit the necessity > of > > > > > >> releasing pre- > > > > > >> > release binaries to Pypi on a nightly (or weekly) cadence. > > > Instead, we > > > > > >> can > > > > > >> > release nightly binaries ONLY to a public S3 bucket and > instruct > > > users > > > > > >> to > > > > > >> > install from there. On our side, we only need to prepare a > html > > > > > >> document that > > > > > >> > contains links to all released nightly binaries. > > > > > >> > Finally users will install the nightly releases via > > > > > >> > > > > > > >> > pip install --pre mxnet-cu101 -f > > > > > >> http://mxnet.s3.amazonaws.com/mxnet-cu101/ > > > > > >> > nightly.html > > > > > >> > > > > > > >> > Instead of > > > > > >> > > > > > > >> > pip install --pre mxnet-cu101 > > > > > >> > > > > > > >> > Of course proper releases and release candidates should still > be > > > made > > >
Re: Stopping nightly releases to Pypi
This has the following effect: - the unfortunate group of user whose GPUs' SMs were removed would be sacrificed so that import of mxnet on their machine would take quite some time on the order of hours. we don't have the usage information to guide which SMs to drop. - it would cut down our binary size by only a corresponding proportion (currently there is 11 SMs total), and if we offer it as the only substitute for continuation of nightly releases on pypi it likely won't help us get the size limit increase. We chose to publish nightly to pypi out of convenience instead of necessity. -sz On 2019/12/03 19:52:42, Marco de Abreu wrote: > What about cutting down on SMs as recommended by Kellen? > > Sheng Zha schrieb am Di., 3. Dez. 2019, 20:15: > > > This is certainly one way to do it. However, the binary size limits our > > ability to publish pypi. So assuming that we want to have our binary on > > pypi still, we'd have to convince pypa to raise our limits. Thus, it seems > > to me that this hypothetical vote with respect to stopping nightly publish > > to pypi would likely only have one acceptable outcome. > > > > This is more of an emergency situation as an essential distribution > > channel is currently broken so I'm focusing on the POC for now. > > > > -sz > > > > On 2019/12/03 18:28:44, Marco de Abreu wrote: > > > Excellent! Could we maybe come up with a POC and a quick writeup and then > > > start a proper vote after everyone verified that it covers their > > use-cases? > > > > > > -Marco > > > > > > Sheng Zha schrieb am Di., 3. Dez. 2019, 19:24: > > > > > > > Yes, there is. We can also make it easier to access by using a > > > > geo-location based DNS server so that China users are directed to that > > > > local mirror. The rest of the world is already covered by the global > > > > cloudfront. > > > > > > > > -sz > > > > > > > > On 2019/12/03 18:22:22, Marco de Abreu > > wrote: > > > > > Isn't there an s3 endpoint in Beijing? > > > > > > > > > > It seems like this topic still warrants some discussion and thus I'd > > > > prefer > > > > > if we don't move forward with lazy consensus. > > > > > > > > > > -Marco > > > > > > > > > > Tao Lv schrieb am Di., 3. Dez. 2019, 14:31: > > > > > > > > > > > * For pypi, we can use mirrors. > > > > > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv wrote: > > > > > > > > > > > > > As we have many users in China, I'm considering the > > accessibility of > > > > S3. > > > > > > > For pip, we can mirrors. > > > > > > > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> I would like to remind everyone that lazy consensus is assumed > > if no > > > > > > >> objections > > > > > > >> are raised before 2019-12-05 at 05:42 UTC. There has been some > > > > > > discussion > > > > > > >> about > > > > > > >> the proposal, but to my understanding no objections were raised. > > > > > > >> > > > > > > >> If the proposal is accepted, MXNet releases would be installed > > via > > > > > > >> > > > > > > >>pip install mxnet > > > > > > >> > > > > > > >> And release candidates via > > > > > > >> > > > > > > >> pip install --pre mxnet > > > > > > >> > > > > > > >> (or with the respective cuda version specifier appended etc.) > > > > > > >> > > > > > > >> To obtain releases built automatically from the master branch, > > users > > > > > > >> would need > > > > > > >> to specify something like "-f > > > > > > >> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to > > pip. > > > > > > >> > > > > > > >> Best regards > > > > > > >> Leonard > > > > > > >> > > > > > > >> On Mon, 2019-12-02 at 05:42 +, Lausen, Leonard wrote: > > > > > > >> > Hi MXNet Community, > > > > > > >> > > > > > > > >> > since more than 2 months our binary Python nightly releases > > > > published > > > > > > >> on Pypi > > > > > > >> > are broken. The problem is that our binaries exceed Pypi's > > size > > > > limit. > > > > > > >> > Decreasing the binary size by adding compression breaks > > > > third-party > > > > > > >> libraries > > > > > > >> > loading libmxnet.so > > > > > > >> https://github.com/apache/incubator-mxnet/issues/16193 > > > > > > >> > > > > > > > >> > Sheng requested Pypi to increase their size limit: > > > > > > >> > https://github.com/pypa/pypi-support/issues/50 > > > > > > >> > > > > > > > >> > Currently "the biggest cost for PyPI from [the many MXNet > > binaries > > > > > > with > > > > > > >> > nightly > > > > > > >> > release to Pypi] is the bandwidth consumed when several > > hundred > > > > > > mirrors > > > > > > >> > attempt > > > > > > >> > to mirror each release immediately after it's published". So > > Pypi > > > > is > > > > > > not > > > > > > >> > inclined to allow us to upload even larger binaries on a > > nightly > > > > > > >> schedule. > > > > > > >> > Their compromise is to allow it on a weekly cadence. > > > > > > >> > > > > > > > >> > However, I would like
Re: Stopping nightly releases to Pypi
CloudFront launched endpoints in Hong Kong during 2019. Based on my tests, they have excellent connectivity to mainland China [1]. Thus we don't need to roll our own geo-location based DNS server solution at this moment (though the great firewall policy may change and we need to adapt). Best regards Leonard [1]: For anyone in China, the Hong Kong endpoints can be used to build a WebSocket based tunnel to get around the great firewall's deep packet inspection based throttling and blocking of foreign services and VPNs. It has excellent performance in my experience, which means it's not throttled / blocked / recognized by the great firewall. https://v2ray.com On Tue, 2019-12-03 at 18:24 +, Sheng Zha wrote: > Yes, there is. We can also make it easier to access by using a geo-location > based DNS server so that China users are directed to that local mirror. The > rest of the world is already covered by the global cloudfront. > > -sz > > On 2019/12/03 18:22:22, Marco de Abreu wrote: > > Isn't there an s3 endpoint in Beijing? > > > > It seems like this topic still warrants some discussion and thus I'd prefer > > if we don't move forward with lazy consensus. > > > > -Marco > > > > Tao Lv schrieb am Di., 3. Dez. 2019, 14:31: > > > > > * For pypi, we can use mirrors. > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv wrote: > > > > > > > As we have many users in China, I'm considering the accessibility of S3. > > > > For pip, we can mirrors. > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard < > > > > lau...@amazon.com.invalid > > > > > > > > wrote: > > > > > > > > > I would like to remind everyone that lazy consensus is assumed if no > > > > > objections > > > > > are raised before 2019-12-05 at 05:42 UTC. There has been some > > > > > > discussion > > > > > about > > > > > the proposal, but to my understanding no objections were raised. > > > > > > > > > > If the proposal is accepted, MXNet releases would be installed via > > > > > > > > > >pip install mxnet > > > > > > > > > > And release candidates via > > > > > > > > > > pip install --pre mxnet > > > > > > > > > > (or with the respective cuda version specifier appended etc.) > > > > > > > > > > To obtain releases built automatically from the master branch, users > > > > > would need > > > > > to specify something like "-f > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to pip. > > > > > > > > > > Best regards > > > > > Leonard > > > > > > > > > > On Mon, 2019-12-02 at 05:42 +, Lausen, Leonard wrote: > > > > > > Hi MXNet Community, > > > > > > > > > > > > since more than 2 months our binary Python nightly releases > > > > > > published > > > > > > > > > > on Pypi > > > > > > are broken. The problem is that our binaries exceed Pypi's size > > > > > > limit. > > > > > > Decreasing the binary size by adding compression breaks third-party > > > > > > > > > > libraries > > > > > > loading libmxnet.so > > > > > > > > > > https://github.com/apache/incubator-mxnet/issues/16193 > > > > > > > > > > > > Sheng requested Pypi to increase their size limit: > > > > > > https://github.com/pypa/pypi-support/issues/50 > > > > > > > > > > > > Currently "the biggest cost for PyPI from [the many MXNet binaries > > > > > > with > > > > > > nightly > > > > > > release to Pypi] is the bandwidth consumed when several hundred > > > > > > mirrors > > > > > > attempt > > > > > > to mirror each release immediately after it's published". So Pypi is > > > > > > not > > > > > > inclined to allow us to upload even larger binaries on a nightly > > > > > > > > > > schedule. > > > > > > Their compromise is to allow it on a weekly cadence. > > > > > > > > > > > > However, I would like the community to revisit the necessity of > > > > > > > > > > releasing pre- > > > > > > release binaries to Pypi on a nightly (or weekly) cadence. Instead, > > > > > > we > > > > > > > > > > can > > > > > > release nightly binaries ONLY to a public S3 bucket and instruct > > > > > > users > > > > > > > > > > to > > > > > > install from there. On our side, we only need to prepare a html > > > > > > > > > > document that > > > > > > contains links to all released nightly binaries. > > > > > > Finally users will install the nightly releases via > > > > > > > > > > > > pip install --pre mxnet-cu101 -f > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/ > > > > > > nightly.html > > > > > > > > > > > > Instead of > > > > > > > > > > > > pip install --pre mxnet-cu101 > > > > > > > > > > > > Of course proper releases and release candidates should still be > > > > > > made > > > > > > available > > > > > > via Pypi. Thus releases would be installed via > > > > > > > > > > > > pip install mxnet-cu101 > > > > > > > > > > > > And release candidates via > > > > > > > > > > > > pip install --pre mxnet-cu101 > > > > > > > > > > > > This will substantially reduce the costs of the Pypi project and in > > > > > > fact
Re: Stopping nightly releases to Pypi
As a simple POC to test distribution, you can try installing MXNet based on these 3 URLs: pip install --no-cache-dir https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl pip install --no-cache-dir https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl pip install --no-cache-dir https://d19zq12jzu4w95.cloudfront.net/ mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl where --no-cache-dir prevents caching the downloaded file, for the purpose of testing. (cu101 chosen based on large size) The first URL uses standard S3 bucket in US. The second uses S3 Accelerate based on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the third URL, as S3 Accelerate may or may not use all new CloudFront endpoints yet. Regarding voting: Uploading to Pypi is currently impossible, which is a reality (so there is no option to continue as we do currently). Pypi folks indicated they will unblock our uploads to Pypi once we stop uploading nightly releases and taking up 20% of their ressources [1]. If there are any shortcomings or problems identified with uploading to S3, we can work to address them. But for now, status quo is broken and this seems the only solution addressing Pypi's problem. I don't mind if you state that you object to lazy consensus and start a vote. If your "maybe [...] start a proper vote" was supposed to be an objection to lazy consensus, please state so clearly (I'm not sure if "maybe" qualifies as objection). Though I think it only makes sense with at least 2 options to vote on. Status quo is not a meaningful option, as it is already broken. Best regards Leonard [1]: https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706 On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote: > Excellent! Could we maybe come up with a POC and a quick writeup and then > start a proper vote after everyone verified that it covers their use-cases? > > -Marco > > Sheng Zha schrieb am Di., 3. Dez. 2019, 19:24: > > > Yes, there is. We can also make it easier to access by using a > > geo-location based DNS server so that China users are directed to that > > local mirror. The rest of the world is already covered by the global > > cloudfront. > > > > -sz > > > > On 2019/12/03 18:22:22, Marco de Abreu wrote: > > > Isn't there an s3 endpoint in Beijing? > > > > > > It seems like this topic still warrants some discussion and thus I'd > > > > prefer > > > if we don't move forward with lazy consensus. > > > > > > -Marco > > > > > > Tao Lv schrieb am Di., 3. Dez. 2019, 14:31: > > > > > > > * For pypi, we can use mirrors. > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv wrote: > > > > > > > > > As we have many users in China, I'm considering the accessibility of > > > > S3. > > > > > For pip, we can mirrors. > > > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > > > > > > > > > > > > > wrote: > > > > > > > > > > > I would like to remind everyone that lazy consensus is assumed if no > > > > > > objections > > > > > > are raised before 2019-12-05 at 05:42 UTC. There has been some > > > > > > > > discussion > > > > > > about > > > > > > the proposal, but to my understanding no objections were raised. > > > > > > > > > > > > If the proposal is accepted, MXNet releases would be installed via > > > > > > > > > > > >pip install mxnet > > > > > > > > > > > > And release candidates via > > > > > > > > > > > > pip install --pre mxnet > > > > > > > > > > > > (or with the respective cuda version specifier appended etc.) > > > > > > > > > > > > To obtain releases built automatically from the master branch, users > > > > > > would need > > > > > > to specify something like "-f > > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to pip. > > > > > > > > > > > > Best regards > > > > > > Leonard > > > > > > > > > > > > On Mon, 2019-12-02 at 05:42 +, Lausen, Leonard wrote: > > > > > > > Hi MXNet Community, > > > > > > > > > > > > > > since more than 2 months our binary Python nightly releases > > > > published > > > > > > on Pypi > > > > > > > are broken. The problem is that our binaries exceed Pypi's size > > > > limit. > > > > > > > Decreasing the binary size by adding compression breaks > > > > third-party > > > > > > libraries > > > > > > > loading libmxnet.so > > > > > > > > > > > > https://github.com/apache/incubator-mxnet/issues/16193 > > > > > > > > > > > > > > Sheng requested Pypi to increase their size limit: > > > > > > > https://github.com/pypa/pypi-support/issues/50 > > > > > > > > > > > > > > Currently "the biggest cost for PyPI from [the many MXNet binaries > > > > > > > > with > > > > > > > nightly > > > > > > > release to Pypi] is the bandwidth consumed when several hundred > > > > > > > > mirrors > > > > > > > attempt > > > > > > > to mirror each release immediately after it's published". So Pypi > > > > is > > > > not
