For the conda environment, just pasting the URL to the whl works fine.

For example, replacing

  - mxnet-cu92mkl==1.6.0b20190910

with

  - https://lllausen-data.s3.amazonaws.com/mxnet-1.6.0-py3-none-any.whl

in the conda file would use the pre-release build of the 1.6 release which is
kept at that URL.

On Mon, 2019-12-02 at 11:17 -0800, Joshua Z. Zhang wrote:
> Separating the nightly wheels from PYPI certainly can reduce our turnaround
> time for processing new packages, overall I am in favor of this proposal.
> 
> However, one question is that would external private server causing problems
> when you are trying to pin a nightly version of MXNet in Conda pip
> environment?
> 
> For example, GluonCV heavily rely on nightly builds of mxnet in CI (
> https://github.com/dmlc/gluon-cv/blob/master/docs/build.yml#L12 <
> https://github.com/dmlc/gluon-cv/blob/master/docs/build.yml#L12>;) and I know
> conda has bad reputation in response to pip installation options: 
> https://github.com/conda/conda/issues/6805 <
> https://github.com/conda/conda/issues/6805>
> 
> Thanks,
> Zhi
> 
> > On Dec 1, 2019, at 11:14 PM, Lausen, Leonard <lau...@amazon.com.INVALID>
> > wrote:
> > 
> > > For the d2l.ai case, bugs within MXNet are fine, so long as users are sent
> > > to
> > > a pinned version that corresponds to the notebooks that are created for
> > > it.
> > 
> > I'm not sure if this approach provides the best user-experience. Ideally we
> > would like d2l users to continue using MXNet after going through the book.
> > But
> > such adoption could be limited if readers run into bugs due to usage of an
> > old
> > (pinned) nightly version when they begin to use more advanced features (not
> > included in / tested by the book).
> > 
> > It seems preferable to have novice users to use a carefully vetted version
> > of
> > MXNet. Such version could be easily obtained by backporting the new
> > operators to
> > the latest stable release, and tagging a release candidate for a new stable
> > minor release. We can set up a pipeline to automatically build such release
> > candidates to Pypi?
> > 
> > But if that's not an option, including the link could be fine as long as
> > users
> > can copy-and-paste it. It seems we may expect copy-and-paste ability for the
> > online version of the book. What do you think? Arguably any printed version
> > should not rely on nightly releases.
> > 
> > Best regards
> > Leonard
> > 
> > On Mon, 2019-12-02 at 06:13 +0000, Chung, Alex wrote:
> > > For the d2l.ai case, bugs within MXNet are fine, so long as users are sent
> > > to
> > > a pinned version that corresponds to the notebooks that are created for
> > > it. I
> > > suppose we can once again provide the whole link, but getting directly
> > > from
> > > pip is the familiar experience for most devs.
> > > 
> > > Yes, 1.6 is the target release, but I don't see a world where the team can
> > > create new operators, and then get it pushed out to stable fast enough for
> > > the
> > > book writers.
> > > 
> > > Sincerely,
> > > 
> > > Alex Chung
> > > 
> > > ________________________________________
> > > From: Lausen, Leonard <lau...@amazon.com.INVALID>
> > > Sent: Sunday, December 1, 2019 10:08 PM
> > > To: dev@mxnet.incubator.apache.org
> > > Cc: Kamakoti, Balaji
> > > Subject: Re: Stopping nightly releases to Pypi
> > > 
> > > If we decide to do weekly pre-release builds to Pypi, what's the benefit?
> > > To
> > > catch bugs and pinpoint when they were introduced, having weekly builds
> > > may be
> > > too coarse. So people would likely prefer the nightly releases and install
> > > them
> > > from S3 via: pip install --pre mxnet-cu101 -f
> > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html
> > > 
> > > For any future project that relies on nightly builds of MXNet 1.7 (or
> > > later),
> > > we
> > > can include above line. (Similar to the Install instructions on
> > > pytorch.org if
> > > you select "Nightly" and "Pip").
> > > 
> > > For existing projects, which taught users to use "pip install --pre mxnet-
> > > cu101"
> > > to get the MXNet 1.6 pre-release: There is no problem, because we will
> > > have a
> > > stable release of 1.6 in the near future.
> > > Based on my understanding, d2l.ai currently targets the MXNet 1.6 release.
> > > 
> > > On Mon, 2019-12-02 at 05:51 +0000, Chung, Alex wrote:
> > > > Hi Leonard,
> > > > 
> > > > Is there any reason why we shouldn't take both options? Ie we do weekly
> > > > build
> > > > on PyPi and provide the s3 option. I would be inclined to make sure we
> > > > provide
> > > > as many avenues as possible to reduce friction for developers. The
> > > > d2l.ai
> > > > book
> > > > by Alex Smola is attracting a community that so far has been relying on
> > > > the
> > > > nightly builds in advance of stable.
> > > > 
> > > > Sincerely,
> > > > 
> > > > Alex Chung
> > > > 
> > > > ________________________________________
> > > > From: Lausen, Leonard <lau...@amazon.com.INVALID>
> > > > Sent: Sunday, December 1, 2019 9:42 PM
> > > > To: d...@mxnet.apache.org
> > > > Subject: Stopping nightly releases to Pypi
> > > > 
> > > > Hi MXNet Community,
> > > > 
> > > > since more than 2 months our binary Python nightly releases published on
> > > > Pypi
> > > > are broken. The problem is that our binaries exceed Pypi's size limit.
> > > > Decreasing the binary size by adding compression breaks third-party
> > > > libraries
> > > > loading libmxnet.so 
> > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > 
> > > > Sheng requested Pypi to increase their size limit:
> > > > https://github.com/pypa/pypi-support/issues/50
> > > > 
> > > > Currently "the biggest cost for PyPI from [the many MXNet binaries with
> > > > nightly
> > > > release to Pypi] is the bandwidth consumed when several hundred mirrors
> > > > attempt
> > > > to mirror each release immediately after it's published". So Pypi is not
> > > > inclined to allow us to upload even larger binaries on a nightly
> > > > schedule.
> > > > Their compromise is to allow it on a weekly cadence.
> > > > 
> > > > However, I would like the community to revisit the necessity of
> > > > releasing
> > > > pre-
> > > > release binaries to Pypi on a nightly (or weekly) cadence. Instead, we
> > > > can
> > > > release nightly binaries ONLY to a public S3 bucket and instruct users
> > > > to
> > > > install from there. On our side, we only need to prepare a html document
> > > > that
> > > > contains links to all released nightly binaries.
> > > > Finally users will install the nightly releases via
> > > > 
> > > >  pip install --pre mxnet-cu101 -f 
> > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > nightly.html
> > > > 
> > > > Instead of
> > > > 
> > > >  pip install --pre mxnet-cu101
> > > > 
> > > > Of course proper releases and release candidates should still be made
> > > > available
> > > > via Pypi. Thus releases would be installed via
> > > > 
> > > >  pip install mxnet-cu101
> > > > 
> > > > And release candidates via
> > > > 
> > > >  pip install --pre mxnet-cu101
> > > > 
> > > > This will substantially reduce the costs of the Pypi project and in fact
> > > > matches
> > > > the installation experience provided by PyTorch. I don't think the
> > > > benefit
> > > > of
> > > > not including "-f http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html
> > > > "
> > > > matches the costs we currently externalize to the Pypi team.
> > > > 
> > > > This suggestion seems uncontroversial to me. Thus I would like to start
> > > > lazy
> > > > consensus. If there are no objections, I will assume lazy consensus on
> > > > stopping
> > > > nightly releases to Pypi in 72hrs.
> > > > 
> > > > Best regards
> > > > Leonard

Reply via email to