Separating the nightly wheels from PYPI certainly can reduce our turnaround time for processing new packages, overall I am in favor of this proposal.
However, one question is that would external private server causing problems when you are trying to pin a nightly version of MXNet in Conda pip environment? For example, GluonCV heavily rely on nightly builds of mxnet in CI (https://github.com/dmlc/gluon-cv/blob/master/docs/build.yml#L12 <https://github.com/dmlc/gluon-cv/blob/master/docs/build.yml#L12>) and I know conda has bad reputation in response to pip installation options: https://github.com/conda/conda/issues/6805 <https://github.com/conda/conda/issues/6805> Thanks, Zhi > On Dec 1, 2019, at 11:14 PM, Lausen, Leonard <lau...@amazon.com.INVALID> > wrote: > >> For the d2l.ai case, bugs within MXNet are fine, so long as users are sent to >> a pinned version that corresponds to the notebooks that are created for it. > > I'm not sure if this approach provides the best user-experience. Ideally we > would like d2l users to continue using MXNet after going through the book. But > such adoption could be limited if readers run into bugs due to usage of an old > (pinned) nightly version when they begin to use more advanced features (not > included in / tested by the book). > > It seems preferable to have novice users to use a carefully vetted version of > MXNet. Such version could be easily obtained by backporting the new operators > to > the latest stable release, and tagging a release candidate for a new stable > minor release. We can set up a pipeline to automatically build such release > candidates to Pypi? > > But if that's not an option, including the link could be fine as long as users > can copy-and-paste it. It seems we may expect copy-and-paste ability for the > online version of the book. What do you think? Arguably any printed version > should not rely on nightly releases. > > Best regards > Leonard > > On Mon, 2019-12-02 at 06:13 +0000, Chung, Alex wrote: >> For the d2l.ai case, bugs within MXNet are fine, so long as users are sent to >> a pinned version that corresponds to the notebooks that are created for it. I >> suppose we can once again provide the whole link, but getting directly from >> pip is the familiar experience for most devs. >> >> Yes, 1.6 is the target release, but I don't see a world where the team can >> create new operators, and then get it pushed out to stable fast enough for >> the >> book writers. >> >> Sincerely, >> >> Alex Chung >> >> ________________________________________ >> From: Lausen, Leonard <lau...@amazon.com.INVALID> >> Sent: Sunday, December 1, 2019 10:08 PM >> To: dev@mxnet.incubator.apache.org >> Cc: Kamakoti, Balaji >> Subject: Re: Stopping nightly releases to Pypi >> >> If we decide to do weekly pre-release builds to Pypi, what's the benefit? To >> catch bugs and pinpoint when they were introduced, having weekly builds may >> be >> too coarse. So people would likely prefer the nightly releases and install >> them >> from S3 via: pip install --pre mxnet-cu101 -f >> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html >> >> For any future project that relies on nightly builds of MXNet 1.7 (or later), >> we >> can include above line. (Similar to the Install instructions on pytorch.org >> if >> you select "Nightly" and "Pip"). >> >> For existing projects, which taught users to use "pip install --pre mxnet- >> cu101" >> to get the MXNet 1.6 pre-release: There is no problem, because we will have a >> stable release of 1.6 in the near future. >> Based on my understanding, d2l.ai currently targets the MXNet 1.6 release. >> >> On Mon, 2019-12-02 at 05:51 +0000, Chung, Alex wrote: >>> Hi Leonard, >>> >>> Is there any reason why we shouldn't take both options? Ie we do weekly >>> build >>> on PyPi and provide the s3 option. I would be inclined to make sure we >>> provide >>> as many avenues as possible to reduce friction for developers. The d2l.ai >>> book >>> by Alex Smola is attracting a community that so far has been relying on the >>> nightly builds in advance of stable. >>> >>> Sincerely, >>> >>> Alex Chung >>> >>> ________________________________________ >>> From: Lausen, Leonard <lau...@amazon.com.INVALID> >>> Sent: Sunday, December 1, 2019 9:42 PM >>> To: d...@mxnet.apache.org >>> Subject: Stopping nightly releases to Pypi >>> >>> Hi MXNet Community, >>> >>> since more than 2 months our binary Python nightly releases published on >>> Pypi >>> are broken. The problem is that our binaries exceed Pypi's size limit. >>> Decreasing the binary size by adding compression breaks third-party >>> libraries >>> loading libmxnet.so https://github.com/apache/incubator-mxnet/issues/16193 >>> >>> Sheng requested Pypi to increase their size limit: >>> https://github.com/pypa/pypi-support/issues/50 >>> >>> Currently "the biggest cost for PyPI from [the many MXNet binaries with >>> nightly >>> release to Pypi] is the bandwidth consumed when several hundred mirrors >>> attempt >>> to mirror each release immediately after it's published". So Pypi is not >>> inclined to allow us to upload even larger binaries on a nightly schedule. >>> Their compromise is to allow it on a weekly cadence. >>> >>> However, I would like the community to revisit the necessity of releasing >>> pre- >>> release binaries to Pypi on a nightly (or weekly) cadence. Instead, we can >>> release nightly binaries ONLY to a public S3 bucket and instruct users to >>> install from there. On our side, we only need to prepare a html document >>> that >>> contains links to all released nightly binaries. >>> Finally users will install the nightly releases via >>> >>> pip install --pre mxnet-cu101 -f >>> http://mxnet.s3.amazonaws.com/mxnet-cu101/ >>> nightly.html >>> >>> Instead of >>> >>> pip install --pre mxnet-cu101 >>> >>> Of course proper releases and release candidates should still be made >>> available >>> via Pypi. Thus releases would be installed via >>> >>> pip install mxnet-cu101 >>> >>> And release candidates via >>> >>> pip install --pre mxnet-cu101 >>> >>> This will substantially reduce the costs of the Pypi project and in fact >>> matches >>> the installation experience provided by PyTorch. I don't think the benefit >>> of >>> not including "-f http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"; >>> matches the costs we currently externalize to the Pypi team. >>> >>> This suggestion seems uncontroversial to me. Thus I would like to start lazy >>> consensus. If there are no objections, I will assume lazy consensus on >>> stopping >>> nightly releases to Pypi in 72hrs. >>> >>> Best regards >>> Leonard
