Re: Stopping nightly releases to Pypi
Agree, but the question how a non Amazonian is able to maintain and access the system is still open. As it stands right now, the community has taken a step back and loses some control if we continue down that road. I personally am disapproving of that approach since committers are no longer in control of that process. So far it seems like my questions were skipped and further actions have been taken. As openness and the community having control are part of our graduation criteria, I'm putting in my veto with a grace period until 15th of January. Please bring the system into a state that aligns with Apache values or revert the changes. -Marco Pedro Larroy schrieb am Fr., 3. Jan. 2020, 03:33: > CD should be separate from CI for security reasons in any case. > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu > wrote: > > > Could you elaborate how a non-Amazonian is able to access, maintain and > > review the CodeBuild pipeline? How come we've diverted from the community > > agreed-on standard where the public Jenkins serves for the purpose of > > testing and releasing MXNet? I'd be curious about the issues you're > > encountering with Jenkins CI that led to a non-standard solution. > > > > -Marco > > > > > > Skalicky, Sam schrieb am Sa., 7. Dez. 2019, > > 18:39: > > > > > Hi MXNet Community, > > > > > > We have been working on getting nightly builds fixed and made available > > > again. We’ve made another system using AWS CodeBuild & S3 to work > around > > > the problems with Jenkins CI, PyPI, etc. It is currently building all > the > > > flavors and publishing to an S3 bucket here: > > > > > > > > > https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2 > > > > > > There are folders for each set of nightly builds, try out the wheels > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) and > > > arrive in the bucket 30min-2hours later. Inside each folder are the > > wheels > > > for each flavor of MXNet. Currently we’re only building for linux, > builds > > > for windows/Mac will come later. > > > > > > If you want to download the wheels easily you can use a URL in the form > > of: > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/ > > > > > > /dist/-1.6.0b-py2.py3-none-manylinux1_x86_64.whl > > > > > > Heres a set of links for today’s builds > > > > > > (Plain mxnet, no mkl no cuda) > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > (mxnet-mkl > > > < > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl > > > > > > ) > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > (mxnet-cuXXX > > > < > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX > > > > > > ) > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > (mxnet-cuXXXmkl > > > < > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl > > > > > > ) > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > You can easily install these pip wheels in your system either by > > > downloading them to your machine first and then installing by doing: > > > > > > pip install /path/to/downloaded/wheel.whl > > > > > > Or you can install directly by just giving the link to pip like this: > > > > > > pip install > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > Credit goes to everyone involved (in no particular order) > > > Rakesh Vasudevan > > > Zach
Re: Stopping nightly releases to Pypi
Hi Marco, I don’t think anyone wants only Amazonians to control access to the system. However, no one has stepped up to help develop one that the community can maintain. Sure there has been some work here or there but nothing consistent. I think what we’re waiting for is someone to volunteer and commit to actually spending time and writing the code and getting this done. Are you volunteering to do this, or are you willing to find someone who is? There is nothing to veto here. There was a problem with CD, we came up with a short-term fix, and are waiting for the community to finish the Jenkins CD so that the community can go back to maintaining the system. Sam > On Jan 3, 2020, at 6:56 AM, Marco de Abreu wrote: > > Agree, but the question how a non Amazonian is able to maintain and access > the system is still open. As it stands right now, the community has taken a > step back and loses some control if we continue down that road. > > I personally am disapproving of that approach since committers are no > longer in control of that process. So far it seems like my questions were > skipped and further actions have been taken. As openness and the community > having control are part of our graduation criteria, I'm putting in my veto > with a grace period until 15th of January. Please bring the system into a > state that aligns with Apache values or revert the changes. > > -Marco > > Pedro Larroy schrieb am Fr., 3. Jan. 2020, > 03:33: > >> CD should be separate from CI for security reasons in any case. >> >> >> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu >> wrote: >> >>> Could you elaborate how a non-Amazonian is able to access, maintain and >>> review the CodeBuild pipeline? How come we've diverted from the community >>> agreed-on standard where the public Jenkins serves for the purpose of >>> testing and releasing MXNet? I'd be curious about the issues you're >>> encountering with Jenkins CI that led to a non-standard solution. >>> >>> -Marco >>> >>> >>> Skalicky, Sam schrieb am Sa., 7. Dez. 2019, >>> 18:39: >>> Hi MXNet Community, We have been working on getting nightly builds fixed and made available again. We’ve made another system using AWS CodeBuild & S3 to work >> around the problems with Jenkins CI, PyPI, etc. It is currently building all >> the flavors and publishing to an S3 bucket here: >>> >> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2 There are folders for each set of nightly builds, try out the wheels starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) and arrive in the bucket 30min-2hours later. Inside each folder are the >>> wheels for each flavor of MXNet. Currently we’re only building for linux, >> builds for windows/Mac will come later. If you want to download the wheels easily you can use a URL in the form >>> of: https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/ >>> >> /dist/-1.6.0b-py2.py3-none-manylinux1_x86_64.whl Heres a set of links for today’s builds (Plain mxnet, no mkl no cuda) >>> >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl (mxnet-mkl < >>> >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl ) >>> >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl (mxnet-cuXXX < >>> >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX ) >>> >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl >>> >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl >>> >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl >>> >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl (mxnet-cuXXXmkl < >>> >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl ) >>> >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl >>> >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl >>> >> https://apache-mxnet.s3-us-west-2.amazonaws.com/
Re: Stopping nightly releases to Pypi
I am curious what reasoning went into a non-community entity deploying what is effectively de-facto public builds of an Apache project, "temporary" or not. Was this discussed on dev list? Btw, I don't buy this "temporary" thing -- "temporary" has a bad habit of becoming "permanent". Also, I challenge the logic behind "We built something that violates Apache guidelines because no one else was doing it". -Chris On Fri, Jan 3, 2020 at 9:42 AM Skalicky, Sam wrote: > Hi Marco, > > I don’t think anyone wants only Amazonians to control access to the > system. However, no one has stepped up to help develop one that the > community can maintain. Sure there has been some work here or there but > nothing consistent. I think what we’re waiting for is someone to volunteer > and commit to actually spending time and writing the code and getting this > done. > > Are you volunteering to do this, or are you willing to find someone who > is? > > There is nothing to veto here. There was a problem with CD, we came up > with a short-term fix, and are waiting for the community to finish the > Jenkins CD so that the community can go back to maintaining the system. > > Sam > > > On Jan 3, 2020, at 6:56 AM, Marco de Abreu > wrote: > > > > Agree, but the question how a non Amazonian is able to maintain and > access > > the system is still open. As it stands right now, the community has > taken a > > step back and loses some control if we continue down that road. > > > > I personally am disapproving of that approach since committers are no > > longer in control of that process. So far it seems like my questions were > > skipped and further actions have been taken. As openness and the > community > > having control are part of our graduation criteria, I'm putting in my > veto > > with a grace period until 15th of January. Please bring the system into a > > state that aligns with Apache values or revert the changes. > > > > -Marco > > > > Pedro Larroy schrieb am Fr., 3. Jan. > 2020, > > 03:33: > > > >> CD should be separate from CI for security reasons in any case. > >> > >> > >> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu > > >> wrote: > >> > >>> Could you elaborate how a non-Amazonian is able to access, maintain and > >>> review the CodeBuild pipeline? How come we've diverted from the > community > >>> agreed-on standard where the public Jenkins serves for the purpose of > >>> testing and releasing MXNet? I'd be curious about the issues you're > >>> encountering with Jenkins CI that led to a non-standard solution. > >>> > >>> -Marco > >>> > >>> > >>> Skalicky, Sam schrieb am Sa., 7. Dez. > 2019, > >>> 18:39: > >>> > Hi MXNet Community, > > We have been working on getting nightly builds fixed and made > available > again. We’ve made another system using AWS CodeBuild & S3 to work > >> around > the problems with Jenkins CI, PyPI, etc. It is currently building all > >> the > flavors and publishing to an S3 bucket here: > > > >>> > >> > https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2 > > There are folders for each set of nightly builds, try out the wheels > starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) and > arrive in the bucket 30min-2hours later. Inside each folder are the > >>> wheels > for each flavor of MXNet. Currently we’re only building for linux, > >> builds > for windows/Mac will come later. > > If you want to download the wheels easily you can use a URL in the > form > >>> of: > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/ > > >>> > >> > /dist/-1.6.0b-py2.py3-none-manylinux1_x86_64.whl > > Heres a set of links for today’s builds > > (Plain mxnet, no mkl no cuda) > > > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > (mxnet-mkl > < > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl > > ) > > > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > (mxnet-cuXXX > < > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX > > ) > > > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > >>> > >> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > >>> > >> >
Re: Stopping nightly releases to Pypi
Hi Chris, It really came down to some developers wanting to use regular nightly pip builds. The CD system wasn’t working (stopped about 10/25/2019) so the nightly builds that were available were stale. We put together a system to build the nightlies quickly for our own use, and started putting them in a publicly accessible bucket. If others want to use them thats great and we’re happy to share. But there is no “replacing the community managed CD system with a non-community entity managed CD”. Rather the community managed CD is broken and no one is fixing it. Thanks, Sam > On Jan 3, 2020, at 10:04 AM, Chris Olivier wrote: > > I am curious what reasoning went into a non-community entity deploying what > is effectively de-facto public builds of an Apache project, "temporary" or > not. Was this discussed on dev list? Btw, I don't buy this "temporary" > thing -- "temporary" has a bad habit of becoming "permanent". Also, I > challenge the logic behind "We built something that violates Apache > guidelines because no one else was doing it". > > -Chris > > > > On Fri, Jan 3, 2020 at 9:42 AM Skalicky, Sam > wrote: > >> Hi Marco, >> >> I don’t think anyone wants only Amazonians to control access to the >> system. However, no one has stepped up to help develop one that the >> community can maintain. Sure there has been some work here or there but >> nothing consistent. I think what we’re waiting for is someone to volunteer >> and commit to actually spending time and writing the code and getting this >> done. >> >> Are you volunteering to do this, or are you willing to find someone who >> is? >> >> There is nothing to veto here. There was a problem with CD, we came up >> with a short-term fix, and are waiting for the community to finish the >> Jenkins CD so that the community can go back to maintaining the system. >> >> Sam >> >>> On Jan 3, 2020, at 6:56 AM, Marco de Abreu >> wrote: >>> >>> Agree, but the question how a non Amazonian is able to maintain and >> access >>> the system is still open. As it stands right now, the community has >> taken a >>> step back and loses some control if we continue down that road. >>> >>> I personally am disapproving of that approach since committers are no >>> longer in control of that process. So far it seems like my questions were >>> skipped and further actions have been taken. As openness and the >> community >>> having control are part of our graduation criteria, I'm putting in my >> veto >>> with a grace period until 15th of January. Please bring the system into a >>> state that aligns with Apache values or revert the changes. >>> >>> -Marco >>> >>> Pedro Larroy schrieb am Fr., 3. Jan. >> 2020, >>> 03:33: >>> CD should be separate from CI for security reasons in any case. On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu >> wrote: > Could you elaborate how a non-Amazonian is able to access, maintain and > review the CodeBuild pipeline? How come we've diverted from the >> community > agreed-on standard where the public Jenkins serves for the purpose of > testing and releasing MXNet? I'd be curious about the issues you're > encountering with Jenkins CI that led to a non-standard solution. > > -Marco > > > Skalicky, Sam schrieb am Sa., 7. Dez. >> 2019, > 18:39: > >> Hi MXNet Community, >> >> We have been working on getting nightly builds fixed and made >> available >> again. We’ve made another system using AWS CodeBuild & S3 to work around >> the problems with Jenkins CI, PyPI, etc. It is currently building all the >> flavors and publishing to an S3 bucket here: >> >> > >> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2 >> >> There are folders for each set of nightly builds, try out the wheels >> starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) and >> arrive in the bucket 30min-2hours later. Inside each folder are the > wheels >> for each flavor of MXNet. Currently we’re only building for linux, builds >> for windows/Mac will come later. >> >> If you want to download the wheels easily you can use a URL in the >> form > of: >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/ >> > >> /dist/-1.6.0b-py2.py3-none-manylinux1_x86_64.whl >> >> Heres a set of links for today’s builds >> >> (Plain mxnet, no mkl no cuda) >> >> > >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl >> (mxnet-mkl >> < > >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl >> >> ) >> >> > >> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1
Re: Stopping nightly releases to Pypi
I'm not involved in such efforts, but one possibility is to have the yaml files that describe the pipelines for CD in the Apache repositories, would that be acceptable from the Apache POV? In the end they should be very thin and calling the scripts that are part of the CD packages. On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu wrote: > Agree, but the question how a non Amazonian is able to maintain and access > the system is still open. As it stands right now, the community has taken a > step back and loses some control if we continue down that road. > > I personally am disapproving of that approach since committers are no > longer in control of that process. So far it seems like my questions were > skipped and further actions have been taken. As openness and the community > having control are part of our graduation criteria, I'm putting in my veto > with a grace period until 15th of January. Please bring the system into a > state that aligns with Apache values or revert the changes. > > -Marco > > Pedro Larroy schrieb am Fr., 3. Jan. 2020, > 03:33: > > > CD should be separate from CI for security reasons in any case. > > > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu > > wrote: > > > > > Could you elaborate how a non-Amazonian is able to access, maintain and > > > review the CodeBuild pipeline? How come we've diverted from the > community > > > agreed-on standard where the public Jenkins serves for the purpose of > > > testing and releasing MXNet? I'd be curious about the issues you're > > > encountering with Jenkins CI that led to a non-standard solution. > > > > > > -Marco > > > > > > > > > Skalicky, Sam schrieb am Sa., 7. Dez. > 2019, > > > 18:39: > > > > > > > Hi MXNet Community, > > > > > > > > We have been working on getting nightly builds fixed and made > available > > > > again. We’ve made another system using AWS CodeBuild & S3 to work > > around > > > > the problems with Jenkins CI, PyPI, etc. It is currently building all > > the > > > > flavors and publishing to an S3 bucket here: > > > > > > > > > > > > > > https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2 > > > > > > > > There are folders for each set of nightly builds, try out the wheels > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) and > > > > arrive in the bucket 30min-2hours later. Inside each folder are the > > > wheels > > > > for each flavor of MXNet. Currently we’re only building for linux, > > builds > > > > for windows/Mac will come later. > > > > > > > > If you want to download the wheels easily you can use a URL in the > form > > > of: > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/ > > > > > > > > > > /dist/-1.6.0b-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > Heres a set of links for today’s builds > > > > > > > > (Plain mxnet, no mkl no cuda) > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > (mxnet-mkl > > > > < > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl > > > > > > > > ) > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > (mxnet-cuXXX > > > > < > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX > > > > > > > > ) > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > (mxnet-cuXXXmkl > > > > < > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl > > > > > > > > ) > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dis
Re: Stopping nightly releases to Pypi
Sam, while I understand that this solution was developed out of necessity, my question why a new system has been developed instead of fixing the existing one or adapting the solution. CodeBuild is a scheduler in the same fashion as Jenkins is. It runs code. So you can adapt it to Jenkins without much hassle. I'm not volunteering for this - why should I? The role of a PMC member is to steer the direction of the project. Just because a manager points towards a certain direction, if doesn't mean that they're going to do it. Apparently there was enough time at some point to develop a new solution from scratch. It might have been a solution for your internal team and that's fine, but upgrading it "temporarily" to be the advertised way on the official website is something different. I won't argue about how the veto can be enforced. I think it's in the best interest of the project if we try working on a solution instead of spending time on trying to figure out the power of the PMC. Pedro, that's certainly a step towards the right direction. But committers would also need access to the control plane of the system - to trigger, stop and audit builds. We could go down that road, but i think the fewer systems, the better - also for the sake of maintainability. Best regards, Marco Pedro Larroy schrieb am Fr., 3. Jan. 2020, 20:55: > I'm not involved in such efforts, but one possibility is to have the yaml > files that describe the pipelines for CD in the Apache repositories, would > that be acceptable from the Apache POV? In the end they should be very thin > and calling the scripts that are part of the CD packages. > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu > wrote: > > > Agree, but the question how a non Amazonian is able to maintain and > access > > the system is still open. As it stands right now, the community has > taken a > > step back and loses some control if we continue down that road. > > > > I personally am disapproving of that approach since committers are no > > longer in control of that process. So far it seems like my questions were > > skipped and further actions have been taken. As openness and the > community > > having control are part of our graduation criteria, I'm putting in my > veto > > with a grace period until 15th of January. Please bring the system into a > > state that aligns with Apache values or revert the changes. > > > > -Marco > > > > Pedro Larroy schrieb am Fr., 3. Jan. > 2020, > > 03:33: > > > > > CD should be separate from CI for security reasons in any case. > > > > > > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu < > marco.g.ab...@gmail.com> > > > wrote: > > > > > > > Could you elaborate how a non-Amazonian is able to access, maintain > and > > > > review the CodeBuild pipeline? How come we've diverted from the > > community > > > > agreed-on standard where the public Jenkins serves for the purpose of > > > > testing and releasing MXNet? I'd be curious about the issues you're > > > > encountering with Jenkins CI that led to a non-standard solution. > > > > > > > > -Marco > > > > > > > > > > > > Skalicky, Sam schrieb am Sa., 7. Dez. > > 2019, > > > > 18:39: > > > > > > > > > Hi MXNet Community, > > > > > > > > > > We have been working on getting nightly builds fixed and made > > available > > > > > again. We’ve made another system using AWS CodeBuild & S3 to work > > > around > > > > > the problems with Jenkins CI, PyPI, etc. It is currently building > all > > > the > > > > > flavors and publishing to an S3 bucket here: > > > > > > > > > > > > > > > > > > > > https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2 > > > > > > > > > > There are folders for each set of nightly builds, try out the > wheels > > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) > and > > > > > arrive in the bucket 30min-2hours later. Inside each folder are the > > > > wheels > > > > > for each flavor of MXNet. Currently we’re only building for linux, > > > builds > > > > > for windows/Mac will come later. > > > > > > > > > > If you want to download the wheels easily you can use a URL in the > > form > > > > of: > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/ > > > > > > > > > > > > > > > /dist/-1.6.0b-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > Heres a set of links for today’s builds > > > > > > > > > > (Plain mxnet, no mkl no cuda) > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > (mxnet-mkl > > > > > < > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl > > > > > > > > > > ) > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > (mxnet-cuXXX > > >
Re: Stopping nightly releases to Pypi
Regarding your point of finding somebody to maintain the solution: At Apache we usually retire things if there's no maintainer, since that indicates that the feature/system is not of enough interest to warrant maintenance - otherwise, someone would step up. While assistance in the form of a fix is always appreciated, the fix still has to conform with the way this project and Apache operates. Next time I'd recommend to contribute time on improving the existing community solution instead of developing an internal system. -Marco Marco de Abreu schrieb am Sa., 4. Jan. 2020, 00:21: > Sam, while I understand that this solution was developed out of necessity, > my question why a new system has been developed instead of fixing the > existing one or adapting the solution. CodeBuild is a scheduler in the same > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins without > much hassle. > > I'm not volunteering for this - why should I? The role of a PMC member is > to steer the direction of the project. Just because a manager points > towards a certain direction, if doesn't mean that they're going to do it. > > Apparently there was enough time at some point to develop a new solution > from scratch. It might have been a solution for your internal team and > that's fine, but upgrading it "temporarily" to be the advertised way on the > official website is something different. > > I won't argue about how the veto can be enforced. I think it's in the best > interest of the project if we try working on a solution instead of spending > time on trying to figure out the power of the PMC. > > Pedro, that's certainly a step towards the right direction. But committers > would also need access to the control plane of the system - to trigger, > stop and audit builds. We could go down that road, but i think the fewer > systems, the better - also for the sake of maintainability. > > Best regards, > Marco > > > > Pedro Larroy schrieb am Fr., 3. Jan. 2020, > 20:55: > >> I'm not involved in such efforts, but one possibility is to have the yaml >> files that describe the pipelines for CD in the Apache repositories, would >> that be acceptable from the Apache POV? In the end they should be very >> thin >> and calling the scripts that are part of the CD packages. >> >> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu >> wrote: >> >> > Agree, but the question how a non Amazonian is able to maintain and >> access >> > the system is still open. As it stands right now, the community has >> taken a >> > step back and loses some control if we continue down that road. >> > >> > I personally am disapproving of that approach since committers are no >> > longer in control of that process. So far it seems like my questions >> were >> > skipped and further actions have been taken. As openness and the >> community >> > having control are part of our graduation criteria, I'm putting in my >> veto >> > with a grace period until 15th of January. Please bring the system into >> a >> > state that aligns with Apache values or revert the changes. >> > >> > -Marco >> > >> > Pedro Larroy schrieb am Fr., 3. Jan. >> 2020, >> > 03:33: >> > >> > > CD should be separate from CI for security reasons in any case. >> > > >> > > >> > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu < >> marco.g.ab...@gmail.com> >> > > wrote: >> > > >> > > > Could you elaborate how a non-Amazonian is able to access, maintain >> and >> > > > review the CodeBuild pipeline? How come we've diverted from the >> > community >> > > > agreed-on standard where the public Jenkins serves for the purpose >> of >> > > > testing and releasing MXNet? I'd be curious about the issues you're >> > > > encountering with Jenkins CI that led to a non-standard solution. >> > > > >> > > > -Marco >> > > > >> > > > >> > > > Skalicky, Sam schrieb am Sa., 7. Dez. >> > 2019, >> > > > 18:39: >> > > > >> > > > > Hi MXNet Community, >> > > > > >> > > > > We have been working on getting nightly builds fixed and made >> > available >> > > > > again. We’ve made another system using AWS CodeBuild & S3 to work >> > > around >> > > > > the problems with Jenkins CI, PyPI, etc. It is currently building >> all >> > > the >> > > > > flavors and publishing to an S3 bucket here: >> > > > > >> > > > > >> > > > >> > > >> > >> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2 >> > > > > >> > > > > There are folders for each set of nightly builds, try out the >> wheels >> > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) >> and >> > > > > arrive in the bucket 30min-2hours later. Inside each folder are >> the >> > > > wheels >> > > > > for each flavor of MXNet. Currently we’re only building for linux, >> > > builds >> > > > > for windows/Mac will come later. >> > > > > >> > > > > If you want to download the wheels easily you can use a URL in the >> > form >> > > > of: >> > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/ >> > > > > >> > > > >
Re: Stopping nightly releases to Pypi
Hey Marco. As far as I have learned from other Apache mailing lists while lurking is that Apache only cares about making source releases, binaries are a courtesy to users that some projects decide to do, but I'm not sure I understand your concerns regarding the PMC and what exactly are you vetoing here, since everyone can compile, build and package our project as per the open source license. I would suggest to have a constructive approach and see how we can make this happen for the best of the project, specially since somebody is volunteering to help with this and dedicate valuable compute resources and people's time. Regarding manual changes I don't see any need to have access to a code build control plane for *anybody*, for several reasons, first is that manual access to production account is a discouraged practice and are best managed through pipeline deployments, second is that Code build is a hosted service which is basically just using a build description file to do the work, there's no need to do any manual fiddling or triggering. If all the CD and description files are in the apache repository you can use your own account or compute resources to do your own build flavor if you so desire. Is your proposal to host this in Apache infrastructure? Maybe I'm missing something on this conversation Pedro. On Fri, Jan 3, 2020 at 3:21 PM Marco de Abreu wrote: > Sam, while I understand that this solution was developed out of necessity, > my question why a new system has been developed instead of fixing the > existing one or adapting the solution. CodeBuild is a scheduler in the same > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins without > much hassle. > > I'm not volunteering for this - why should I? The role of a PMC member is > to steer the direction of the project. Just because a manager points > towards a certain direction, if doesn't mean that they're going to do it. > > Apparently there was enough time at some point to develop a new solution > from scratch. It might have been a solution for your internal team and > that's fine, but upgrading it "temporarily" to be the advertised way on the > official website is something different. > > I won't argue about how the veto can be enforced. I think it's in the best > interest of the project if we try working on a solution instead of spending > time on trying to figure out the power of the PMC. > > Pedro, that's certainly a step towards the right direction. But committers > would also need access to the control plane of the system - to trigger, > stop and audit builds. We could go down that road, but i think the fewer > systems, the better - also for the sake of maintainability. > > Best regards, > Marco > > > > Pedro Larroy schrieb am Fr., 3. Jan. 2020, > 20:55: > > > I'm not involved in such efforts, but one possibility is to have the yaml > > files that describe the pipelines for CD in the Apache repositories, > would > > that be acceptable from the Apache POV? In the end they should be very > thin > > and calling the scripts that are part of the CD packages. > > > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu > > wrote: > > > > > Agree, but the question how a non Amazonian is able to maintain and > > access > > > the system is still open. As it stands right now, the community has > > taken a > > > step back and loses some control if we continue down that road. > > > > > > I personally am disapproving of that approach since committers are no > > > longer in control of that process. So far it seems like my questions > were > > > skipped and further actions have been taken. As openness and the > > community > > > having control are part of our graduation criteria, I'm putting in my > > veto > > > with a grace period until 15th of January. Please bring the system > into a > > > state that aligns with Apache values or revert the changes. > > > > > > -Marco > > > > > > Pedro Larroy schrieb am Fr., 3. Jan. > > 2020, > > > 03:33: > > > > > > > CD should be separate from CI for security reasons in any case. > > > > > > > > > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu < > > marco.g.ab...@gmail.com> > > > > wrote: > > > > > > > > > Could you elaborate how a non-Amazonian is able to access, maintain > > and > > > > > review the CodeBuild pipeline? How come we've diverted from the > > > community > > > > > agreed-on standard where the public Jenkins serves for the purpose > of > > > > > testing and releasing MXNet? I'd be curious about the issues you're > > > > > encountering with Jenkins CI that led to a non-standard solution. > > > > > > > > > > -Marco > > > > > > > > > > > > > > > Skalicky, Sam schrieb am Sa., 7. Dez. > > > 2019, > > > > > 18:39: > > > > > > > > > > > Hi MXNet Community, > > > > > > > > > > > > We have been working on getting nightly builds fixed and made > > > available > > > > > > again. We’ve made another system using AWS CodeBuild & S3 to work > > > > around > > > > > > the problems with Jenkins CI
Re: Stopping nightly releases to Pypi
Just to clarify, the current CI is quite an overhead to maintain for several reasons, this complexity is overkill for CD. Jenkins also has constant plugin upgrades, security vulnerabilities, has to be restarted from time to time as it stops working... and to make binary builds from an environment which runs unsafe code, I don't think is good practice. So for that, having a separate Jenkins, CodeBuild, Drone or using a separate Jenkins node is the right solution. Agree with you that is just a scheduler, but somebody is making efforts to keep it running. If you have the appetite and resources to duplicate it for CD please go ahead. On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu wrote: > Regarding your point of finding somebody to maintain the solution: At > Apache we usually retire things if there's no maintainer, since that > indicates that the feature/system is not of enough interest to warrant > maintenance - otherwise, someone would step up. > > While assistance in the form of a fix is always appreciated, the fix still > has to conform with the way this project and Apache operates. Next time I'd > recommend to contribute time on improving the existing community solution > instead of developing an internal system. > > -Marco > > Marco de Abreu schrieb am Sa., 4. Jan. 2020, > 00:21: > > > Sam, while I understand that this solution was developed out of > necessity, > > my question why a new system has been developed instead of fixing the > > existing one or adapting the solution. CodeBuild is a scheduler in the > same > > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins > without > > much hassle. > > > > I'm not volunteering for this - why should I? The role of a PMC member is > > to steer the direction of the project. Just because a manager points > > towards a certain direction, if doesn't mean that they're going to do it. > > > > Apparently there was enough time at some point to develop a new solution > > from scratch. It might have been a solution for your internal team and > > that's fine, but upgrading it "temporarily" to be the advertised way on > the > > official website is something different. > > > > I won't argue about how the veto can be enforced. I think it's in the > best > > interest of the project if we try working on a solution instead of > spending > > time on trying to figure out the power of the PMC. > > > > Pedro, that's certainly a step towards the right direction. But > committers > > would also need access to the control plane of the system - to trigger, > > stop and audit builds. We could go down that road, but i think the fewer > > systems, the better - also for the sake of maintainability. > > > > Best regards, > > Marco > > > > > > > > Pedro Larroy schrieb am Fr., 3. Jan. > 2020, > > 20:55: > > > >> I'm not involved in such efforts, but one possibility is to have the > yaml > >> files that describe the pipelines for CD in the Apache repositories, > would > >> that be acceptable from the Apache POV? In the end they should be very > >> thin > >> and calling the scripts that are part of the CD packages. > >> > >> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu > >> wrote: > >> > >> > Agree, but the question how a non Amazonian is able to maintain and > >> access > >> > the system is still open. As it stands right now, the community has > >> taken a > >> > step back and loses some control if we continue down that road. > >> > > >> > I personally am disapproving of that approach since committers are no > >> > longer in control of that process. So far it seems like my questions > >> were > >> > skipped and further actions have been taken. As openness and the > >> community > >> > having control are part of our graduation criteria, I'm putting in my > >> veto > >> > with a grace period until 15th of January. Please bring the system > into > >> a > >> > state that aligns with Apache values or revert the changes. > >> > > >> > -Marco > >> > > >> > Pedro Larroy schrieb am Fr., 3. Jan. > >> 2020, > >> > 03:33: > >> > > >> > > CD should be separate from CI for security reasons in any case. > >> > > > >> > > > >> > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu < > >> marco.g.ab...@gmail.com> > >> > > wrote: > >> > > > >> > > > Could you elaborate how a non-Amazonian is able to access, > maintain > >> and > >> > > > review the CodeBuild pipeline? How come we've diverted from the > >> > community > >> > > > agreed-on standard where the public Jenkins serves for the purpose > >> of > >> > > > testing and releasing MXNet? I'd be curious about the issues > you're > >> > > > encountering with Jenkins CI that led to a non-standard solution. > >> > > > > >> > > > -Marco > >> > > > > >> > > > > >> > > > Skalicky, Sam schrieb am Sa., 7. > Dez. > >> > 2019, > >> > > > 18:39: > >> > > > > >> > > > > Hi MXNet Community, > >> > > > > > >> > > > > We have been working on getting nightly builds fixed and made > >> > available > >> > > > > again. We’ve made another system using AWS
