For the conda environment, just pasting the URL to the whl works fine. For example, replacing
- mxnet-cu92mkl==1.6.0b20190910 with - https://lllausen-data.s3.amazonaws.com/mxnet-1.6.0-py3-none-any.whl in the conda file would use the pre-release build of the 1.6 release which is kept at that URL. On Mon, 2019-12-02 at 11:17 -0800, Joshua Z. Zhang wrote: > Separating the nightly wheels from PYPI certainly can reduce our turnaround > time for processing new packages, overall I am in favor of this proposal. > > However, one question is that would external private server causing problems > when you are trying to pin a nightly version of MXNet in Conda pip > environment? > > For example, GluonCV heavily rely on nightly builds of mxnet in CI ( > https://github.com/dmlc/gluon-cv/blob/master/docs/build.yml#L12 < > https://github.com/dmlc/gluon-cv/blob/master/docs/build.yml#L12>;) and I know > conda has bad reputation in response to pip installation options: > https://github.com/conda/conda/issues/6805 < > https://github.com/conda/conda/issues/6805> > > Thanks, > Zhi > > > On Dec 1, 2019, at 11:14 PM, Lausen, Leonard <lau...@amazon.com.INVALID> > > wrote: > > > > > For the d2l.ai case, bugs within MXNet are fine, so long as users are sent > > > to > > > a pinned version that corresponds to the notebooks that are created for > > > it. > > > > I'm not sure if this approach provides the best user-experience. Ideally we > > would like d2l users to continue using MXNet after going through the book. > > But > > such adoption could be limited if readers run into bugs due to usage of an > > old > > (pinned) nightly version when they begin to use more advanced features (not > > included in / tested by the book). > > > > It seems preferable to have novice users to use a carefully vetted version > > of > > MXNet. Such version could be easily obtained by backporting the new > > operators to > > the latest stable release, and tagging a release candidate for a new stable > > minor release. We can set up a pipeline to automatically build such release > > candidates to Pypi? > > > > But if that's not an option, including the link could be fine as long as > > users > > can copy-and-paste it. It seems we may expect copy-and-paste ability for the > > online version of the book. What do you think? Arguably any printed version > > should not rely on nightly releases. > > > > Best regards > > Leonard > > > > On Mon, 2019-12-02 at 06:13 +0000, Chung, Alex wrote: > > > For the d2l.ai case, bugs within MXNet are fine, so long as users are sent > > > to > > > a pinned version that corresponds to the notebooks that are created for > > > it. I > > > suppose we can once again provide the whole link, but getting directly > > > from > > > pip is the familiar experience for most devs. > > > > > > Yes, 1.6 is the target release, but I don't see a world where the team can > > > create new operators, and then get it pushed out to stable fast enough for > > > the > > > book writers. > > > > > > Sincerely, > > > > > > Alex Chung > > > > > > ________________________________________ > > > From: Lausen, Leonard <lau...@amazon.com.INVALID> > > > Sent: Sunday, December 1, 2019 10:08 PM > > > To: dev@mxnet.incubator.apache.org > > > Cc: Kamakoti, Balaji > > > Subject: Re: Stopping nightly releases to Pypi > > > > > > If we decide to do weekly pre-release builds to Pypi, what's the benefit? > > > To > > > catch bugs and pinpoint when they were introduced, having weekly builds > > > may be > > > too coarse. So people would likely prefer the nightly releases and install > > > them > > > from S3 via: pip install --pre mxnet-cu101 -f > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html > > > > > > For any future project that relies on nightly builds of MXNet 1.7 (or > > > later), > > > we > > > can include above line. (Similar to the Install instructions on > > > pytorch.org if > > > you select "Nightly" and "Pip"). > > > > > > For existing projects, which taught users to use "pip install --pre mxnet- > > > cu101" > > > to get the MXNet 1.6 pre-release: There is no problem, because we will > > > have a > > > stable release of 1.6 in the near future. > > > Based on my understanding, d2l.ai currently targets the MXNet 1.6 release. > > > > > > On Mon, 2019-12-02 at 05:51 +0000, Chung, Alex wrote: > > > > Hi Leonard, > > > > > > > > Is there any reason why we shouldn't take both options? Ie we do weekly > > > > build > > > > on PyPi and provide the s3 option. I would be inclined to make sure we > > > > provide > > > > as many avenues as possible to reduce friction for developers. The > > > > d2l.ai > > > > book > > > > by Alex Smola is attracting a community that so far has been relying on > > > > the > > > > nightly builds in advance of stable. > > > > > > > > Sincerely, > > > > > > > > Alex Chung > > > > > > > > ________________________________________ > > > > From: Lausen, Leonard <lau...@amazon.com.INVALID> > > > > Sent: Sunday, December 1, 2019 9:42 PM > > > > To: d...@mxnet.apache.org > > > > Subject: Stopping nightly releases to Pypi > > > > > > > > Hi MXNet Community, > > > > > > > > since more than 2 months our binary Python nightly releases published on > > > > Pypi > > > > are broken. The problem is that our binaries exceed Pypi's size limit. > > > > Decreasing the binary size by adding compression breaks third-party > > > > libraries > > > > loading libmxnet.so > > > > https://github.com/apache/incubator-mxnet/issues/16193 > > > > > > > > Sheng requested Pypi to increase their size limit: > > > > https://github.com/pypa/pypi-support/issues/50 > > > > > > > > Currently "the biggest cost for PyPI from [the many MXNet binaries with > > > > nightly > > > > release to Pypi] is the bandwidth consumed when several hundred mirrors > > > > attempt > > > > to mirror each release immediately after it's published". So Pypi is not > > > > inclined to allow us to upload even larger binaries on a nightly > > > > schedule. > > > > Their compromise is to allow it on a weekly cadence. > > > > > > > > However, I would like the community to revisit the necessity of > > > > releasing > > > > pre- > > > > release binaries to Pypi on a nightly (or weekly) cadence. Instead, we > > > > can > > > > release nightly binaries ONLY to a public S3 bucket and instruct users > > > > to > > > > install from there. On our side, we only need to prepare a html document > > > > that > > > > contains links to all released nightly binaries. > > > > Finally users will install the nightly releases via > > > > > > > > pip install --pre mxnet-cu101 -f > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/ > > > > nightly.html > > > > > > > > Instead of > > > > > > > > pip install --pre mxnet-cu101 > > > > > > > > Of course proper releases and release candidates should still be made > > > > available > > > > via Pypi. Thus releases would be installed via > > > > > > > > pip install mxnet-cu101 > > > > > > > > And release candidates via > > > > > > > > pip install --pre mxnet-cu101 > > > > > > > > This will substantially reduce the costs of the Pypi project and in fact > > > > matches > > > > the installation experience provided by PyTorch. I don't think the > > > > benefit > > > > of > > > > not including "-f http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html > > > > " > > > > matches the costs we currently externalize to the Pypi team. > > > > > > > > This suggestion seems uncontroversial to me. Thus I would like to start > > > > lazy > > > > consensus. If there are no objections, I will assume lazy consensus on > > > > stopping > > > > nightly releases to Pypi in 72hrs. > > > > > > > > Best regards > > > > Leonard
