I am curious what reasoning went into a non-community entity deploying what is effectively de-facto public builds of an Apache project, "temporary" or not. Was this discussed on dev list? Btw, I don't buy this "temporary" thing -- "temporary" has a bad habit of becoming "permanent". Also, I challenge the logic behind "We built something that violates Apache guidelines because no one else was doing it".
-Chris On Fri, Jan 3, 2020 at 9:42 AM Skalicky, Sam <sska...@amazon.com.invalid> wrote: > Hi Marco, > > I don’t think anyone wants only Amazonians to control access to the > system. However, no one has stepped up to help develop one that the > community can maintain. Sure there has been some work here or there but > nothing consistent. I think what we’re waiting for is someone to volunteer > and commit to actually spending time and writing the code and getting this > done. > > Are you volunteering to do this, or are you willing to find someone who > is? > > There is nothing to veto here. There was a problem with CD, we came up > with a short-term fix, and are waiting for the community to finish the > Jenkins CD so that the community can go back to maintaining the system. > > Sam > > > On Jan 3, 2020, at 6:56 AM, Marco de Abreu <marco.g.ab...@gmail.com> > wrote: > > > > Agree, but the question how a non Amazonian is able to maintain and > access > > the system is still open. As it stands right now, the community has > taken a > > step back and loses some control if we continue down that road. > > > > I personally am disapproving of that approach since committers are no > > longer in control of that process. So far it seems like my questions were > > skipped and further actions have been taken. As openness and the > community > > having control are part of our graduation criteria, I'm putting in my > veto > > with a grace period until 15th of January. Please bring the system into a > > state that aligns with Apache values or revert the changes. > > > > -Marco > > > > Pedro Larroy <pedro.larroy.li...@gmail.com> schrieb am Fr., 3. Jan. > 2020, > > 03:33: > > > >> CD should be separate from CI for security reasons in any case. > >> > >> > >> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <marco.g.ab...@gmail.com > > > >> wrote: > >> > >>> Could you elaborate how a non-Amazonian is able to access, maintain and > >>> review the CodeBuild pipeline? How come we've diverted from the > community > >>> agreed-on standard where the public Jenkins serves for the purpose of > >>> testing and releasing MXNet? I'd be curious about the issues you're > >>> encountering with Jenkins CI that led to a non-standard solution. > >>> > >>> -Marco > >>> > >>> > >>> Skalicky, Sam <sska...@amazon.com.invalid> schrieb am Sa., 7. Dez. > 2019, > >>> 18:39: > >>> > >>>> Hi MXNet Community, > >>>> > >>>> We have been working on getting nightly builds fixed and made > available > >>>> again. We’ve made another system using AWS CodeBuild & S3 to work > >> around > >>>> the problems with Jenkins CI, PyPI, etc. It is currently building all > >> the > >>>> flavors and publishing to an S3 bucket here: > >>>> > >>>> > >>> > >> > https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2 > >>>> > >>>> There are folders for each set of nightly builds, try out the wheels > >>>> starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) and > >>>> arrive in the bucket 30min-2hours later. Inside each folder are the > >>> wheels > >>>> for each flavor of MXNet. Currently we’re only building for linux, > >> builds > >>>> for windows/Mac will come later. > >>>> > >>>> If you want to download the wheels easily you can use a URL in the > form > >>> of: > >>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/ > >>>> > >>> > >> > <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl > >>>> > >>>> Heres a set of links for today’s builds > >>>> > >>>> (Plain mxnet, no mkl no cuda) > >>>> > >>>> > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > >>>> (mxnet-mkl > >>>> < > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl > >>>> > >>>> ) > >>>> > >>>> > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > >>>> (mxnet-cuXXX > >>>> < > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX > >>>> > >>>> ) > >>>> > >>>> > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > >>>> > >>>> > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > >>>> > >>>> > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > >>>> > >>>> > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > >>>> (mxnet-cuXXXmkl > >>>> < > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl > >>>> > >>>> ) > >>>> > >>>> > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > >>>> > >>>> > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > >>>> > >>>> > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > >>>> > >>>> > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > >>>> > >>>> You can easily install these pip wheels in your system either by > >>>> downloading them to your machine first and then installing by doing: > >>>> > >>>> pip install /path/to/downloaded/wheel.whl > >>>> > >>>> Or you can install directly by just giving the link to pip like this: > >>>> > >>>> pip install > >>>> > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > >>>> > >>>> Credit goes to everyone involved (in no particular order) > >>>> Rakesh Vasudevan > >>>> Zach Kimberg > >>>> Manu Seth > >>>> Sheng Zha > >>>> Jun Wu > >>>> Pedro Larroy > >>>> Chaitanya Bapat > >>>> > >>>> Thanks! > >>>> Sam > >>>> > >>>> > >>>> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lau...@amazon.com.INVALID > >>>> <mailto:lau...@amazon.com.INVALID>> wrote: > >>>> > >>>> We don't loose pip by hosting on S3. We just don't host nightly > >> releases > >>>> on Pypi > >>>> servers and mirror them to several hundred mirrors immediately after > >> each > >>>> build > >>>> is published which is very expensive for the Pypi project.. People can > >>>> still > >>>> install the nightly builds with pip by specifying the -f option. > >>>> > >>>> Uploading weekly releases to Pypi will reduce the cost for Pypi by > ~75% > >>>> [1]. It > >>>> may be acceptable to Pypi, but does it make sense for us? I'm not > >>> convinced > >>>> weekly release on Pypi is a good idea. Consider one release is buggy, > >>>> users will > >>>> need to wait for 7 days for a fix. It doesn't provide good user > >>> experience. > >>>> If someone has a stronger conviction about the value of weekly > releases > >>> on > >>>> Pypi, > >>>> that person shall please go ahead and propose it in a separate > >> discussion > >>>> thread. > >>>> > >>>> Currently we don't have generally working nightly builds to Pypi and > >> as a > >>>> matter > >>>> of fact we know that we can't have them due to Pypi's policy and our > >>>> apparent > >>>> need for large binaries. Given this fact and that no objection was > >> raised > >>>> by > >>>> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping > >>>> upload > >>>> attempts of nightly builds to Pypi. > >>>> > >>>> With consensus established, we can change the CI job to stop trying to > >>>> upload > >>>> the nightly builds and then request Pypi to increase the limit. Then > we > >>>> have one > >>>> less blocker for the 1.6 release. > >>>> > >>>> Best regards > >>>> Leonard > >>>> > >>>> [1]: Lower cost due to less releases, but higher cost due to 500MB -> > >>> 800MB > >>>> limit increase. Assuming that the limit increase translates into > >> actually > >>>> larger > >>>> binaries. > >>>> > >>>> > >>>> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote: > >>>> Are weekly releases an option? It was brought up as concern that we > >> might > >>>> lose pip as a pretty common distribution channel where people consume > >>>> nightly builds. I don't feel like that concern has been properly > >>> addressed > >>>> so far. > >>>> > >>>> -Marco > >>>> > >>>> Lausen, Leonard <lau...@amazon.com.invalid<mailto: > >>>> lau...@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019, > >>>> 04:09: > >>>> > >>>> As a simple POC to test distribution, you can try installing MXNet > >> based > >>> on > >>>> these 3 URLs: > >>>> > >>>> pip install --no-cache-dir > >>>> > >>>> > >>>> > >>> > >> > https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > >>>> pip install --no-cache-dir > >>>> > >>>> > >>>> > >>> > >> > https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > >>>> pip install --no-cache-dir https://d19zq12jzu4w95.cloudfront.net/ > >>>> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > >>>> < > >>> > >> > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > >>>> > >>>> < > >>>> > >>>> > >>> > >> > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > >>>> > >>>> > >>>> where --no-cache-dir prevents caching the downloaded file, for the > >>> purpose > >>>> of > >>>> testing. (cu101 chosen based on large size) > >>>> > >>>> The first URL uses standard S3 bucket in US. The second uses S3 > >>> Accelerate > >>>> based > >>>> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the > >>> third > >>>> URL, > >>>> as S3 Accelerate may or may not use all new CloudFront endpoints yet. > >>>> > >>>> Regarding voting: Uploading to Pypi is currently impossible, which is > a > >>>> reality > >>>> (so there is no option to continue as we do currently). Pypi folks > >>>> indicated > >>>> they will unblock our uploads to Pypi once we stop uploading nightly > >>>> releases > >>>> and taking up 20% of their ressources [1]. > >>>> > >>>> If there are any shortcomings or problems identified with uploading to > >>> S3, > >>>> we > >>>> can work to address them. But for now, status quo is broken and this > >>> seems > >>>> the > >>>> only solution addressing Pypi's problem. > >>>> > >>>> I don't mind if you state that you object to lazy consensus and start > a > >>>> vote. If > >>>> your "maybe [...] start a proper vote" was supposed to be an objection > >> to > >>>> lazy > >>>> consensus, please state so clearly (I'm not sure if "maybe" qualifies > >> as > >>>> objection). Though I think it only makes sense with at least 2 options > >> to > >>>> vote > >>>> on. Status quo is not a meaningful option, as it is already broken. > >>>> > >>>> Best regards > >>>> Leonard > >>>> > >>>> [1]: > >>> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706 > >>>> > >>>> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote: > >>>> Excellent! Could we maybe come up with a POC and a quick writeup and > >> then > >>>> start a proper vote after everyone verified that it covers their > >>>> use-cases? > >>>> -Marco > >>>> > >>>> Sheng Zha <zhash...@apache.org> schrieb am Di., 3. Dez. 2019, 19:24: > >>>> > >>>> Yes, there is. We can also make it easier to access by using a > >>>> geo-location based DNS server so that China users are directed to that > >>>> local mirror. The rest of the world is already covered by the global > >>>> cloudfront. > >>>> > >>>> -sz > >>>> > >>>> On 2019/12/03 18:22:22, Marco de Abreu <marco.g.ab...@gmail.com> > >>>> wrote: > >>>> Isn't there an s3 endpoint in Beijing? > >>>> > >>>> It seems like this topic still warrants some discussion and thus I'd > >>>> > >>>> prefer > >>>> if we don't move forward with lazy consensus. > >>>> > >>>> -Marco > >>>> > >>>> Tao Lv <mutou...@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31: > >>>> > >>>> * For pypi, we can use mirrors. > >>>> > >>>> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutou...@gmail.com> wrote: > >>>> > >>>> As we have many users in China, I'm considering the > >>>> accessibility of > >>>> S3. > >>>> For pip, we can mirrors. > >>>> > >>>> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > >>>> > >>>> <lau...@amazon.com.invalid > >>>> wrote: > >>>> > >>>> I would like to remind everyone that lazy consensus is assumed > >>>> if no > >>>> objections > >>>> are raised before 2019-12-05 at 05:42 UTC. There has been some > >>>> > >>>> discussion > >>>> about > >>>> the proposal, but to my understanding no objections were > >>>> raised. > >>>> If the proposal is accepted, MXNet releases would be installed > >>>> via > >>>> pip install mxnet > >>>> > >>>> And release candidates via > >>>> > >>>> pip install --pre mxnet > >>>> > >>>> (or with the respective cuda version specifier appended etc.) > >>>> > >>>> To obtain releases built automatically from the master branch, > >>>> users > >>>> would need > >>>> to specify something like "-f > >>>> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to > >>>> pip. > >>>> Best regards > >>>> Leonard > >>>> > >>>> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote: > >>>> Hi MXNet Community, > >>>> > >>>> since more than 2 months our binary Python nightly releases > >>>> > >>>> published > >>>> on Pypi > >>>> are broken. The problem is that our binaries exceed Pypi's > >>>> size > >>>> limit. > >>>> Decreasing the binary size by adding compression breaks > >>>> > >>>> third-party > >>>> libraries > >>>> loading libmxnet.so > >>>> > >>>> https://github.com/apache/incubator-mxnet/issues/16193 > >>>> Sheng requested Pypi to increase their size limit: > >>>> https://github.com/pypa/pypi-support/issues/50 > >>>> > >>>> Currently "the biggest cost for PyPI from [the many MXNet > >>>> binaries > >>>> with > >>>> nightly > >>>> release to Pypi] is the bandwidth consumed when several > >>>> hundred > >>>> mirrors > >>>> attempt > >>>> to mirror each release immediately after it's published". So > >>>> Pypi > >>>> is > >>>> not > >>>> inclined to allow us to upload even larger binaries on a > >>>> nightly > >>>> schedule. > >>>> Their compromise is to allow it on a weekly cadence. > >>>> > >>>> However, I would like the community to revisit the necessity > >>>> of > >>>> releasing pre- > >>>> release binaries to Pypi on a nightly (or weekly) cadence. > >>>> > >>>> Instead, we > >>>> can > >>>> release nightly binaries ONLY to a public S3 bucket and > >>>> instruct > >>>> users > >>>> to > >>>> install from there. On our side, we only need to prepare a > >>>> html > >>>> document that > >>>> contains links to all released nightly binaries. > >>>> Finally users will install the nightly releases via > >>>> > >>>> pip install --pre mxnet-cu101 -f > >>>> > >>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/ > >>>> nightly.html > >>>> > >>>> Instead of > >>>> > >>>> pip install --pre mxnet-cu101 > >>>> > >>>> Of course proper releases and release candidates should > >>>> still be > >>>> made > >>>> available > >>>> via Pypi. Thus releases would be installed via > >>>> > >>>> pip install mxnet-cu101 > >>>> > >>>> And release candidates via > >>>> > >>>> pip install --pre mxnet-cu101 > >>>> > >>>> This will substantially reduce the costs of the Pypi project > >>>> and > >>>> in > >>>> fact > >>>> matches > >>>> the installation experience provided by PyTorch. I don't > >>>> think the > >>>> benefit of > >>>> not including "-f > >>>> > >>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"; > >>>> matches the costs we currently externalize to the Pypi team. > >>>> > >>>> This suggestion seems uncontroversial to me. Thus I would > >>>> like to > >>>> start > >>>> lazy > >>>> consensus. If there are no objections, I will assume lazy > >>>> > >>>> consensus on > >>>> stopping > >>>> nightly releases to Pypi in 72hrs. > >>>> > >>>> Best regards > >>>> Leonard > >>>> > >>>> > >>> > >> > >
