Hi MXNet Community,

since more than 2 months our binary Python nightly releases published on Pypi
are broken. The problem is that our binaries exceed Pypi's size limit.
Decreasing the binary size by adding compression breaks third-party libraries
loading libmxnet.so https://github.com/apache/incubator-mxnet/issues/16193

Sheng requested Pypi to increase their size limit: 
https://github.com/pypa/pypi-support/issues/50

Currently "the biggest cost for PyPI from [the many MXNet binaries with nightly
release to Pypi] is the bandwidth consumed when several hundred mirrors attempt
to mirror each release immediately after it's published". So Pypi is not
inclined to allow us to upload even larger binaries on a nightly schedule.
Their compromise is to allow it on a weekly cadence.

However, I would like the community to revisit the necessity of releasing pre-
release binaries to Pypi on a nightly (or weekly) cadence. Instead, we can
release nightly binaries ONLY to a public S3 bucket and instruct users to
install from there. On our side, we only need to prepare a html document that
contains links to all released nightly binaries.
Finally users will install the nightly releases via

  pip install --pre mxnet-cu101 -f http://mxnet.s3.amazonaws.com/mxnet-cu101/
nightly.html

Instead of

  pip install --pre mxnet-cu101

Of course proper releases and release candidates should still be made available
via Pypi. Thus releases would be installed via

  pip install mxnet-cu101

And release candidates via

  pip install --pre mxnet-cu101

This will substantially reduce the costs of the Pypi project and in fact matches
the installation experience provided by PyTorch. I don't think the benefit of
not including "-f http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html";
matches the costs we currently externalize to the Pypi team.

This suggestion seems uncontroversial to me. Thus I would like to start lazy
consensus. If there are no objections, I will assume lazy consensus on stopping
nightly releases to Pypi in 72hrs.

Best regards
Leonard

Reply via email to