or use PyPICloud.it can storage pack to s3 and compatible to pip.

shiwen hu <yajiedes...@gmail.com> 于2020年1月7日周二 上午10:27写道:

> why not use pypiserver? it is  compatible server for pip.use it can
> install pack like pip install -i http://xxxx/ mxnet --pre
>
> Skalicky, Sam <sska...@amazon.com.invalid> 于2020年1月7日周二 上午2:55写道:
>
>> Thats a good idea Leonard, we can have a static html page in the bucket
>> for this. But keep in mind pip wheels do have a COMMIT_HASH file packaged
>> inside. So we can always figure out which commit/build a user has by
>> dumping this file from the mxnet installation. File name of the pip wheel
>> is not so important.
>>
>> Sam
>>
>> > On Jan 6, 2020, at 10:19 AM, Lausen, Leonard <lau...@amazon.com.INVALID>
>> wrote:
>> >
>> > Consider a user finds a bug in a nightly version but we can't narrow
>> down the
>> > version of mxnet used as the name is constant over time. Or users wan't
>> to
>> > revert back to the previous nightly version installed but don't know
>> which date
>> > it was from due to constant name.
>> >
>> > Instead I suggest we introduce an autogenerated page like
>> > https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
>> >
>> > Then "pip install -f URLTOPAGE mxnet" will install the latest available
>> version.
>> > Maybe the team maintaining the S3 bucket can reconsider creating such
>> page for
>> > the intermediate time until the CD based nighlty build is operating.
>> >
>> > On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
>> >> +1 for a nightly pip with fixed name.
>> >>
>> >> We need this to track mxnet integration with other packages such as
>> Horovod.
>> >>
>> >> Sam, when do you think we can have this nightly build with a fixed
>> name?
>> >>
>> >> Thanks,
>> >>
>> >> Lin
>> >>
>> >> On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
>> <sska...@amazon.com.invalid>
>> >> wrote:
>> >>
>> >>> Hi Tao,
>> >>>
>> >>> We dont have this yet, but we did think about putting the latest
>> wheels in
>> >>> a specific place in the s3 bucket so they are always updated.
>> Initially we
>> >>> decided not to do this since the main MXNet CD should have been
>> fixed. But
>> >>> since its still not fixed yet, we might try and go ahead and do this.
>> >>>
>> >>> Sam
>> >>>
>> >>> On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a...@intel.com<mailto:
>> >>> tao.a...@intel.com>> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> How to install the latest available build of a flavor without
>> specifying
>> >>> the build date? Something like `pip install mxnet --pre`.
>> >>>
>> >>> Thanks,
>> >>> -tao
>> >>>
>> >>> -----Original Message-----
>> >>> From: Skalicky, Sam <sska...@amazon.com.INVALID<mailto:
>> >>> sska...@amazon.com.INVALID>>
>> >>> Sent: Monday, January 6, 2020 2:09 AM
>> >>> To: dev@mxnet.incubator.apache.org<mailto:
>> dev@mxnet.incubator.apache.org>
>> >>> Subject: Re: Stopping nightly releases to Pypi
>> >>>
>> >>> Hi Haibin,
>> >>>
>> >>> You typed the correct URLs, the cu100 build has been failing since
>> >>> December 30th but other builds have succeeded. The wheels are being
>> >>> delivered into a public bucket that anyone with an AWS account can
>> access
>> >>> and go poke around, here’s the URL for web access:
>> >>>
>> >>>
>> >>>
>> https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
>> >>>
>> >>> You will have to log into your AWS account to access it however (which
>> >>> means you’ll need an AWS account).
>> >>>
>> >>> It looks like only the following flavors are available for 2020-01-01:
>> >>> mxnet
>> >>> mxnet-cu92
>> >>> mxnet-cu92mkl
>> >>> mxnet-mkl
>> >>>
>> >>> Sam
>> >>>
>> >>> On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin....@gmail.com
>> <mailto:
>> >>> haibin.lin....@gmail.com><mailto:haibin.lin....@gmail.com>> wrote:
>> >>>
>> >>> I was trying the nightly builds, but none of them is available:
>> >>>
>> >>> pip3 install
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
>> >>> --user
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
>> >>>>
>> >>> pip3 install
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
>> >>> --user
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
>> >>>>
>> >>> pip3 install
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>> >>> --user
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
>> >>>>
>> >>> pip3 install
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
>> >>> --user
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
>> >>>>
>> >>>
>> >>> ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103 from
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>> >>> because of HTTP error 404 Client Error: Not Found for url:
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>> >>> for URL
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>> Please let me know if I typed wrong URLs.
>> >>>
>> >>> 1. The discoverability of available nightly builds needs improvement.
>> If
>> >>> someone can help write a script to list all links that exist, that
>> would be
>> >>> very helpful.
>> >>> 2. If any nightly build is not built successfully, how do the
>> community
>> >>> know the reason of the failure, and potentially offer helps?
>> Currently I
>> >>> don't have much visibility of the nightly build status.
>> >>>
>> >>> Best,
>> >>> Haibin
>> >>>
>> >>>
>> >>> On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <
>> pedro.larroy.li...@gmail.com
>> >>> <mailto:pedro.larroy.li...@gmail.com>>
>> >>> wrote:
>> >>>
>> >>> Just to clarify, the current CI is quite an overhead to maintain for
>> >>> several reasons, this complexity is overkill for CD. Jenkins also has
>> >>> constant plugin upgrades, security vulnerabilities, has to be
>> restarted
>> >>> from time to time as it stops working... and to make binary builds
>> from an
>> >>> environment which runs unsafe code, I don't think is good practice.
>> So for
>> >>> that, having a separate Jenkins, CodeBuild, Drone or using a separate
>> >>> Jenkins node is the right solution. Agree with you that is just a
>> >>> scheduler, but somebody is making efforts to keep it running. If you
>> have
>> >>> the appetite and resources to duplicate it for CD please go ahead.
>> >>>
>> >>> On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <
>> marco.g.ab...@gmail.com
>> >>> <mailto:marco.g.ab...@gmail.com>>
>> >>> wrote:
>> >>>
>> >>> Regarding your point of finding somebody to maintain the solution: At
>> >>> Apache we usually retire things if there's no maintainer, since that
>> >>> indicates that the feature/system is not of enough interest to warrant
>> >>> maintenance - otherwise, someone would step up.
>> >>>
>> >>> While assistance in the form of a fix is always appreciated, the fix
>> still
>> >>> has to conform with the way this project and Apache operates. Next
>> time I'd
>> >>> recommend to contribute time on improving the existing community
>> solution
>> >>> instead of developing an internal system.
>> >>>
>> >>> -Marco
>> >>>
>> >>> Marco de Abreu <marco.g.ab...@gmail.com<mailto:
>> marco.g.ab...@gmail.com>>
>> >>> schrieb am Sa., 4. Jan. 2020,
>> >>> 00:21:
>> >>>
>> >>> Sam, while I understand that this solution was developed out of
>> necessity,
>> >>> my question why a new system has been developed instead of fixing the
>> >>> existing one or adapting the solution. CodeBuild is a scheduler in
>> the same
>> >>> fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
>> without
>> >>> much hassle.
>> >>>
>> >>> I'm not volunteering for this - why should I? The role of a PMC
>> member is
>> >>> to steer the direction of the project. Just because a manager points
>> >>> towards a certain direction, if doesn't mean that they're going to do
>> it.
>> >>>
>> >>> Apparently there was enough time at some point to develop a new
>> solution
>> >>> from scratch. It might have been a solution for your internal team and
>> >>> that's fine, but upgrading it "temporarily" to be the advertised way
>> on the
>> >>> official website is something different.
>> >>>
>> >>> I won't argue about how the veto can be enforced. I think it's in the
>> best
>> >>> interest of the project if we try working on a solution instead of
>> spending
>> >>> time on trying to figure out the power of the PMC.
>> >>>
>> >>> Pedro, that's certainly a step towards the right direction. But
>> committers
>> >>> would also need access to the control plane of the system - to
>> trigger,
>> >>> stop and audit builds. We could go down that road, but i think the
>> fewer
>> >>> systems, the better - also for the sake of maintainability.
>> >>>
>> >>> Best regards,
>> >>> Marco
>> >>>
>> >>>
>> >>>
>> >>> Pedro Larroy <pedro.larroy.li...@gmail.com<mailto:
>> >>> pedro.larroy.li...@gmail.com>> schrieb am Fr., 3. Jan.
>> >>> 2020,
>> >>> 20:55:
>> >>>
>> >>> I'm not involved in such efforts, but one possibility is to have the
>> yaml
>> >>> files that describe the pipelines for CD in the Apache repositories,
>> would
>> >>> that be acceptable from the Apache POV? In the end they should be
>> very thin
>> >>> and calling the scripts that are part of the CD packages.
>> >>>
>> >>> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
>> marco.g.ab...@gmail.com
>> >>> <mailto:marco.g.ab...@gmail.com>>
>> >>> wrote:
>> >>>
>> >>> Agree, but the question how a non Amazonian is able to maintain and
>> access
>> >>> the system is still open. As it stands right now, the community has
>> taken a
>> >>> step back and loses some control if we continue down that road.
>> >>>
>> >>> I personally am disapproving of that approach since committers are no
>> >>> longer in control of that process. So far it seems like my questions
>> were
>> >>> skipped and further actions have been taken. As openness and the
>> community
>> >>> having control are part of our graduation criteria, I'm putting in my
>> veto
>> >>> with a grace period until 15th of January. Please bring the system
>> into a
>> >>> state that aligns with Apache values or revert the changes.
>> >>>
>> >>> -Marco
>> >>>
>> >>> Pedro Larroy <pedro.larroy.li...@gmail.com<mailto:
>> >>> pedro.larroy.li...@gmail.com>> schrieb am Fr., 3. Jan.
>> >>> 2020,
>> >>> 03:33:
>> >>>
>> >>> CD should be separate from CI for security reasons in any case.
>> >>>
>> >>>
>> >>> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
>> marco.g.ab...@gmail.com
>> >>> <mailto:marco.g.ab...@gmail.com>>
>> >>> wrote:
>> >>>
>> >>> Could you elaborate how a non-Amazonian is able to access, maintain
>> and
>> >>> review the CodeBuild pipeline? How come we've diverted from the
>> community
>> >>> agreed-on standard where the public Jenkins serves for the purpose of
>> >>> testing and releasing MXNet? I'd be curious about the issues you're
>> >>> encountering with Jenkins CI that led to a non-standard solution.
>> >>>
>> >>> -Marco
>> >>>
>> >>>
>> >>> Skalicky, Sam <sska...@amazon.com.invalid<mailto:
>> >>> sska...@amazon.com.invalid>> schrieb am Sa., 7.
>> >>> Dez.
>> >>> 2019,
>> >>> 18:39:
>> >>>
>> >>> Hi MXNet Community,
>> >>>
>> >>> We have been working on getting nightly builds fixed and made
>> available
>> >>> again. We’ve made another system using AWS CodeBuild & S3 to work
>> around
>> >>> the problems with Jenkins CI, PyPI, etc. It is currently building all
>> the
>> >>> flavors and publishing to an S3 bucket here:
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
>> >>>
>> >>> There are folders for each set of nightly builds, try out the wheels
>> >>> starting today 2019-12-07. Builds start at 1:30am PT (9:30am
>> >>> GMT)
>> >>> and
>> >>> arrive in the bucket 30min-2hours later. Inside each folder are the
>> wheels
>> >>> for each flavor of MXNet. Currently we’re only building for linux,
>> builds
>> >>> for windows/Mac will come later.
>> >>>
>> >>> If you want to download the wheels easily you can use a URL in the
>> form
>> >>> of:
>> >>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
>> >>> manylinux1_x86_64.whl
>> >>>
>> >>> Heres a set of links for today’s builds
>> >>>
>> >>> (Plain mxnet, no mkl no cuda)
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>> (mxnet-mkl
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>> >>>>
>> >>> <
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>> >>>
>> >>> )
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>> (mxnet-cuXXX
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>> >>>>
>> >>> <
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>> >>>
>> >>> )
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>> (mxnet-cuXXXmkl
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>> >>>>
>> >>> <
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>> >>>
>> >>> )
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>> You can easily install these pip wheels in your system either by
>> >>> downloading them to your machine first and then installing by
>> >>> doing:
>> >>>
>> >>> pip install /path/to/downloaded/wheel.whl
>> >>>
>> >>> Or you can install directly by just giving the link to pip like
>> >>> this:
>> >>>
>> >>> pip install
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>> Credit goes to everyone involved (in no particular order) Rakesh
>> Vasudevan
>> >>> Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya Bapat
>> >>>
>> >>> Thanks!
>> >>> Sam
>> >>>
>> >>>
>> >>> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lau...@amazon.com.INVALID
>> >>> <mailto:lau...@amazon.com.INVALID> <mailto:lau...@amazon.com.INVALID
>> >>
>> >>> wrote:
>> >>>
>> >>> We don't loose pip by hosting on S3. We just don't host nightly
>> releases
>> >>> on Pypi servers and mirror them to several hundred mirrors immediately
>> >>> after each build is published which is very expensive for the Pypi
>> project..
>> >>> People
>> >>> can
>> >>> still
>> >>> install the nightly builds with pip by specifying the -f option.
>> >>>
>> >>> Uploading weekly releases to Pypi will reduce the cost for Pypi by
>> ~75%
>> >>> [1]. It may be acceptable to Pypi, but does it make sense for us? I'm
>> not
>> >>> convinced weekly release on Pypi is a good idea. Consider one release
>> is
>> >>> buggy, users will need to wait for 7 days for a fix. It doesn't
>> provide
>> >>> good user experience.
>> >>> If someone has a stronger conviction about the value of weekly
>> releases on
>> >>> Pypi, that person shall please go ahead and propose it in a separate
>> >>> discussion thread.
>> >>>
>> >>> Currently we don't have generally working nightly builds to Pypi and
>> as a
>> >>> matter of fact we know that we can't have them due to Pypi's policy
>> and our
>> >>> apparent need for large binaries. Given this fact and that no
>> objection was
>> >>> raised by
>> >>> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping
>> >>> upload attempts of nightly builds to Pypi.
>> >>>
>> >>> With consensus established, we can change the CI job to stop trying to
>> >>> upload the nightly builds and then request Pypi to increase the limit.
>> >>> Then
>> >>> we
>> >>> have one
>> >>> less blocker for the 1.6 release.
>> >>>
>> >>> Best regards
>> >>> Leonard
>> >>>
>> >>> [1]: Lower cost due to less releases, but higher cost due to 500MB ->
>> >>> 800MB limit increase. Assuming that the limit increase translates into
>> >>> actually larger binaries.
>> >>>
>> >>>
>> >>> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
>> >>> Are weekly releases an option? It was brought up as concern that we
>> might
>> >>> lose pip as a pretty common distribution channel where people consume
>> >>> nightly builds. I don't feel like that concern has been properly
>> addressed
>> >>> so far.
>> >>>
>> >>> -Marco
>> >>>
>> >>> Lausen, Leonard <lau...@amazon.com.invalid<mailto:
>> >>> lau...@amazon.com.invalid><mailto:
>> >>> lau...@amazon.com.invalid<mailto:lau...@amazon.com.invalid>>>
>> schrieb am
>> >>> Mi., 4. Dez. 2019,
>> >>> 04:09:
>> >>>
>> >>> As a simple POC to test distribution, you can try installing MXNet
>> based
>> >>> on these 3 URLs:
>> >>>
>> >>> pip install --no-cache-dir
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> >>> pip install --no-cache-dir
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> >>> pip install --no-cache-dir
>> >>> https://d19zq12jzu4w95.cloudfront.net/
>> >>> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> >>> <
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>> <
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>> where --no-cache-dir prevents caching the downloaded file, for the
>> purpose
>> >>> of testing. (cu101 chosen based on large size)
>> >>>
>> >>> The first URL uses standard S3 bucket in US. The second uses
>> >>> S3
>> >>> Accelerate
>> >>> based
>> >>> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the
>> third
>> >>> URL, as S3 Accelerate may or may not use all new CloudFront endpoints
>> yet.
>> >>>
>> >>> Regarding voting: Uploading to Pypi is currently impossible, which is
>> a
>> >>> reality (so there is no option to continue as we do currently). Pypi
>> folks
>> >>> indicated they will unblock our uploads to Pypi once we stop uploading
>> >>> nightly releases and taking up 20% of their ressources [1].
>> >>>
>> >>> If there are any shortcomings or problems identified with uploading
>> to S3,
>> >>> we can work to address them. But for now, status quo is broken and
>> this
>> >>> seems the only solution addressing Pypi's problem.
>> >>>
>> >>> I don't mind if you state that you object to lazy consensus and start
>> a
>> >>> vote. If your "maybe [...] start a proper vote" was supposed to be an
>> >>> objection to lazy consensus, please state so clearly (I'm not sure if
>> >>> "maybe"
>> >>> qualifies
>> >>> as
>> >>> objection). Though I think it only makes sense with at least 2
>> options to
>> >>> vote on. Status quo is not a meaningful option, as it is already
>> broken.
>> >>>
>> >>> Best regards
>> >>> Leonard
>> >>>
>> >>> [1]:
>> >>>
>> >>> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
>> >>>
>> >>> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
>> >>> Excellent! Could we maybe come up with a POC and a quick writeup and
>> then
>> >>> start a proper vote after everyone verified that it covers their
>> use-cases?
>> >>> -Marco
>> >>>
>> >>> Sheng Zha <zhash...@apache.org<mailto:zhash...@apache.org>> schrieb
>> am
>> >>> Di., 3. Dez. 2019,
>> >>> 19:24:
>> >>>
>> >>> Yes, there is. We can also make it easier to access by using a
>> >>> geo-location based DNS server so that China users are directed to that
>> >>> local mirror. The rest of the world is already covered by the global
>> >>> cloudfront.
>> >>>
>> >>> -sz
>> >>>
>> >>> On 2019/12/03 18:22:22, Marco de Abreu < marco.g.ab...@gmail.com
>> <mailto:
>> >>> marco.g.ab...@gmail.com>
>> >>>
>> >>> wrote:
>> >>> Isn't there an s3 endpoint in Beijing?
>> >>>
>> >>> It seems like this topic still warrants some discussion and thus I'd
>> >>>
>> >>> prefer
>> >>> if we don't move forward with lazy consensus.
>> >>>
>> >>> -Marco
>> >>>
>> >>> Tao Lv <mutou...@gmail.com<mailto:mutou...@gmail.com>> schrieb am
>> Di., 3.
>> >>> Dez. 2019,
>> >>> 14:31:
>> >>>
>> >>> * For pypi, we can use mirrors.
>> >>>
>> >>> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutou...@gmail.com<mailto:
>> >>> mutou...@gmail.com>>
>> >>> wrote:
>> >>>
>> >>> As we have many users in China, I'm considering the accessibility of
>> S3.
>> >>> For pip, we can mirrors.
>> >>>
>> >>> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
>> >>>
>> >>> <lau...@amazon.com.invalid<mailto:lau...@amazon.com.invalid>
>> >>> wrote:
>> >>>
>> >>> I would like to remind everyone that lazy consensus is assumed if no
>> >>> objections are raised before 2019-12-05 at 05:42 UTC. There has been
>> some
>> >>>
>> >>> discussion
>> >>> about
>> >>> the proposal, but to my understanding no objections were raised.
>> >>> If the proposal is accepted, MXNet releases would be installed via
>> pip
>> >>> install mxnet
>> >>>
>> >>> And release candidates via
>> >>>
>> >>> pip install --pre mxnet
>> >>>
>> >>> (or with the respective cuda version specifier appended etc.)
>> >>>
>> >>> To obtain releases built automatically from the master branch, users
>> would
>> >>> need to specify something like "-f
>> >>> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to pip.
>> >>> Best regards
>> >>> Leonard
>> >>>
>> >>> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
>> >>> Hi MXNet Community,
>> >>>
>> >>> since more than 2 months our binary Python nightly releases
>> >>>
>> >>> published
>> >>> on Pypi
>> >>> are broken. The problem is that our binaries exceed Pypi's size limit.
>> >>> Decreasing the binary size by adding compression breaks
>> >>>
>> >>> third-party
>> >>> libraries
>> >>> loading libmxnet.so
>> >>>
>> >>> https://github.com/apache/incubator-mxnet/issues/16193
>> >>> Sheng requested Pypi to increase their size limit:
>> >>> https://github.com/pypa/pypi-support/issues/50
>> >>>
>> >>> Currently "the biggest cost for PyPI from [the many MXNet binaries
>> with
>> >>> nightly release to Pypi] is the bandwidth consumed when several
>> hundred
>> >>> mirrors attempt to mirror each release immediately after it's
>> published".
>> >>> So Pypi is not inclined to allow us to upload even larger binaries on
>> a
>> >>> nightly schedule.
>> >>> Their compromise is to allow it on a weekly cadence.
>> >>>
>> >>> However, I would like the community to revisit the necessity of
>> releasing
>> >>> pre- release binaries to Pypi on a nightly (or weekly) cadence.
>> >>>
>> >>> Instead, we
>> >>> can
>> >>> release nightly binaries ONLY to a public S3 bucket and instruct
>> users to
>> >>> install from there. On our side, we only need to prepare a html
>> document
>> >>> that contains links to all released nightly binaries.
>> >>> Finally users will install the nightly releases via
>> >>>
>> >>> pip install --pre mxnet-cu101 -f
>> >>>
>> >>> http://mxnet.s3.amazonaws.com/mxnet-cu101/
>> >>> nightly.html
>> >>>
>> >>> Instead of
>> >>>
>> >>> pip install --pre mxnet-cu101
>> >>>
>> >>> Of course proper releases and release candidates should still be made
>> >>> available via Pypi. Thus releases would be installed via
>> >>>
>> >>> pip install mxnet-cu101
>> >>>
>> >>> And release candidates via
>> >>>
>> >>> pip install --pre mxnet-cu101
>> >>>
>> >>> This will substantially reduce the costs of the Pypi project and in
>> fact
>> >>> matches the installation experience provided by PyTorch. I don't
>> think the
>> >>> benefit of not including "-f
>> >>>
>> >>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html";
>> >>> matches the costs we currently externalize to the Pypi team.
>> >>>
>> >>> This suggestion seems uncontroversial to me. Thus I would like to
>> start
>> >>> lazy consensus. If there are no objections, I will assume lazy
>> >>>
>> >>> consensus on
>> >>> stopping
>> >>> nightly releases to Pypi in 72hrs.
>> >>>
>> >>> Best regards
>> >>> Leonard
>> >>>
>> >>>
>>
>>

Reply via email to