Re: Pylint Undefined variable/name error
Hi, that's a great idea. Unfortunately, we can't enable it in CI first unless all errors have been resolved. Otherwise, your PR is going to fail with the exact same errors and we won't be able to merge it. Best regards, Marco Vandana Kannan schrieb am Fr., 27. Juli 2018, 07:16: > Hi All, > > On enabling the option "undefined-variable" in pylint (in pylintrc) and > executing on the latest code, 52 errors show up (most of them from the > example folder). These could lead to Python NameError at runtime. The > errors are documented in > https://github.com/apache/incubator-mxnet/issues/11904. > > Currently, this Pylint option is disabled in CI and pylint is not executed > on the example folder. > > Would it be a good idea to enable the option in CI to catch these errors, > and in the meantime, fix the backlog? > > Thanks, > Vandana >
Re: Request to join developing PR Best practice guidelines
Hi Naveen, Thanks for your lead on the initiative of improving PRs! Just curious on the actual usage of the "PR best practices for authors", is it going to be a standard the community enforce on all PRs? That is, all PRs must follow those best practices before they could be merged. Or is it simply a bunch of extra tips that we recommend all contributors to follow together with their best judgements? Could you please clarify the purpose of it a bit here? Thanks! Best wishes, Hao Jin On Thu, Jul 26, 2018 at 11:35 AM, Naveen Swamy wrote: > Hi All, > > As a part of my job I am looking for ways to improve PR response time, > participation, and PR quality. On asking a few committers/contributors near > by(Andrea, Hagay, Mu Li, Steffen) and their feedback was that some PRs do > not have enough details and only those who collaborate closely can review > the PR. This could be because developers for various reasons, they might > forget due to time-pressure, other priorities and the current guidelines > might not be(IMO) enough. > I would like to volunteer to enhance the existing PR Checklist created by > the community members, and I request members to join this effort so we can > collaborate and come up with a good set of guidelines and propose here on > the dev@, please reach out to me on Slack or here if you are interested to > contribute. > > Currently I am thinking of splitting this effort into > 1) PR Best Practices for authors. > 2) PR Checklist for reviewers. > 3) Coding Guidelines. > > Thanks, Naveen >
Re: Release blocker: non-determinstic forward in gluon
Hi Tong - thanks for root causing the problem. Sheng - what is 1.2.1.post0? Shouldn't a patch with fix be released as 1.2.2? Steffen On Thu, Jul 26, 2018 at 5:33 PM Sheng Zha wrote: > Dear users and developers of Apache MXNet (Incubating), > > Thanks to Tong's dedication, the root cause for this issue was identified > to be instability in OpenBLAS's latest stable version 0.3.1. For details, > see Tong's comment > < > https://github.com/apache/incubator-mxnet/issues/11853#issuecomment-408272772 > > > . > > Since both the nightly build and the 1.2.1 wheels are affected, we > recommend that we stay on OpenBLAS last known stable version 0.2.20 that > we've been using. I will assume lazy consensus and prepare the fix > (1.2.1.post0). > > -sz > > On Tue, Jul 24, 2018 at 3:35 PM, Tong He wrote: > > > Recently there's an issue regarding the inconsistent result from gluon > > forward: > > > > https://github.com/apache/incubator-mxnet/issues/11853 > > > > Given a constant input image and loaded pretrained parameters, we expect > a > > deterministic output from arbitrary repeats of forwards. However from the > > issue I see that the forwarded result is non-determinstic. It is harmful > as > > it makes the results from experments/benchmarks/inference meaningless. > > > > Therefore I propose to block the 1.3 release before it gets resolved. > > >
Re: Request to join developing PR Best practice guidelines
Hi Hao, It would be a standard(after agreed upon here) and expect reviewers to enforce it. purpose, from my earlier email- "improve PR response time, participation, and PR quality". Since we currently do have any standard or guidelines, currently developers are using their best judgement and in some cases(when there is not enough details) it so happens that only the close collaborators can review the PR. Thanks, Naveen On Fri, Jul 27, 2018 at 2:27 AM, Hao Jin wrote: > Hi Naveen, > Thanks for your lead on the initiative of improving PRs! Just curious on > the actual usage of the "PR best practices for authors", is it going to be > a standard the community enforce on all PRs? That is, all PRs must follow > those best practices before they could be merged. Or is it simply a bunch > of extra tips that we recommend all contributors to follow together with > their best judgements? Could you please clarify the purpose of it a bit > here? Thanks! > Best wishes, > Hao Jin > > On Thu, Jul 26, 2018 at 11:35 AM, Naveen Swamy wrote: > > > Hi All, > > > > As a part of my job I am looking for ways to improve PR response time, > > participation, and PR quality. On asking a few committers/contributors > near > > by(Andrea, Hagay, Mu Li, Steffen) and their feedback was that some PRs do > > not have enough details and only those who collaborate closely can review > > the PR. This could be because developers for various reasons, they might > > forget due to time-pressure, other priorities and the current guidelines > > might not be(IMO) enough. > > I would like to volunteer to enhance the existing PR Checklist created by > > the community members, and I request members to join this effort so we > can > > collaborate and come up with a good set of guidelines and propose here on > > the dev@, please reach out to me on Slack or here if you are interested > to > > contribute. > > > > Currently I am thinking of splitting this effort into > > 1) PR Best Practices for authors. > > 2) PR Checklist for reviewers. > > 3) Coding Guidelines. > > > > Thanks, Naveen > > >
Re: Release blocker: non-determinstic forward in gluon
In this case we can regard it as a release problem, which is usually what post release versions are for. It’s still the same release with different dependency, so there is no code change needed. -sz > On Jul 27, 2018, at 8:31 AM, Steffen Rochel wrote: > > Hi Tong - thanks for root causing the problem. > Sheng - what is 1.2.1.post0? Shouldn't a patch with fix be released as > 1.2.2? > Steffen > >> On Thu, Jul 26, 2018 at 5:33 PM Sheng Zha wrote: >> >> Dear users and developers of Apache MXNet (Incubating), >> >> Thanks to Tong's dedication, the root cause for this issue was identified >> to be instability in OpenBLAS's latest stable version 0.3.1. For details, >> see Tong's comment >> < >> https://github.com/apache/incubator-mxnet/issues/11853#issuecomment-408272772 >>> >> . >> >> Since both the nightly build and the 1.2.1 wheels are affected, we >> recommend that we stay on OpenBLAS last known stable version 0.2.20 that >> we've been using. I will assume lazy consensus and prepare the fix >> (1.2.1.post0). >> >> -sz >> >>> On Tue, Jul 24, 2018 at 3:35 PM, Tong He wrote: >>> >>> Recently there's an issue regarding the inconsistent result from gluon >>> forward: >>> >>> https://github.com/apache/incubator-mxnet/issues/11853 >>> >>> Given a constant input image and loaded pretrained parameters, we expect >> a >>> deterministic output from arbitrary repeats of forwards. However from the >>> issue I see that the forwarded result is non-determinstic. It is harmful >> as >>> it makes the results from experments/benchmarks/inference meaningless. >>> >>> Therefore I propose to block the 1.3 release before it gets resolved. >>> >>
Re: Release blocker: non-determinstic forward in gluon
Forgot to mention, the post release version is a pip package version. -sz > On Jul 27, 2018, at 10:42 AM, Sheng Zha wrote: > > In this case we can regard it as a release problem, which is usually what > post release versions are for. It’s still the same release with different > dependency, so there is no code change needed. > > -sz > > >> On Jul 27, 2018, at 8:31 AM, Steffen Rochel wrote: >> >> Hi Tong - thanks for root causing the problem. >> Sheng - what is 1.2.1.post0? Shouldn't a patch with fix be released as >> 1.2.2? >> Steffen >> >>> On Thu, Jul 26, 2018 at 5:33 PM Sheng Zha wrote: >>> >>> Dear users and developers of Apache MXNet (Incubating), >>> >>> Thanks to Tong's dedication, the root cause for this issue was identified >>> to be instability in OpenBLAS's latest stable version 0.3.1. For details, >>> see Tong's comment >>> < >>> https://github.com/apache/incubator-mxnet/issues/11853#issuecomment-408272772 >>> . >>> >>> Since both the nightly build and the 1.2.1 wheels are affected, we >>> recommend that we stay on OpenBLAS last known stable version 0.2.20 that >>> we've been using. I will assume lazy consensus and prepare the fix >>> (1.2.1.post0). >>> >>> -sz >>> On Tue, Jul 24, 2018 at 3:35 PM, Tong He wrote: Recently there's an issue regarding the inconsistent result from gluon forward: https://github.com/apache/incubator-mxnet/issues/11853 Given a constant input image and loaded pretrained parameters, we expect >>> a deterministic output from arbitrary repeats of forwards. However from the issue I see that the forwarded result is non-determinstic. It is harmful >>> as it makes the results from experments/benchmarks/inference meaningless. Therefore I propose to block the 1.3 release before it gets resolved. >>>
Re: Release blocker: non-determinstic forward in gluon
Hi Sheng, I also opened an issue on OpenBLAS repo: https://github.com/xianyi/OpenBLAS/issues/1700 . As informed that "0.3.2 should be released this weekend", I tested their develope branch as well, and seems the new version has fixed the bug. Since OpenBLAS 0.3.2 could also have performance improvement, therefore I propose to wait for OpenBLAS 0.3.2 for our pip post release. Best regards, Tong He 2018-07-27 10:54 GMT-07:00 Sheng Zha : > Forgot to mention, the post release version is a pip package version. > > -sz > > > On Jul 27, 2018, at 10:42 AM, Sheng Zha wrote: > > > > In this case we can regard it as a release problem, which is usually > what post release versions are for. It’s still the same release with > different dependency, so there is no code change needed. > > > > -sz > > > > > >> On Jul 27, 2018, at 8:31 AM, Steffen Rochel > wrote: > >> > >> Hi Tong - thanks for root causing the problem. > >> Sheng - what is 1.2.1.post0? Shouldn't a patch with fix be released as > >> 1.2.2? > >> Steffen > >> > >>> On Thu, Jul 26, 2018 at 5:33 PM Sheng Zha wrote: > >>> > >>> Dear users and developers of Apache MXNet (Incubating), > >>> > >>> Thanks to Tong's dedication, the root cause for this issue was > identified > >>> to be instability in OpenBLAS's latest stable version 0.3.1. For > details, > >>> see Tong's comment > >>> < > >>> https://github.com/apache/incubator-mxnet/issues/11853# > issuecomment-408272772 > > >>> . > >>> > >>> Since both the nightly build and the 1.2.1 wheels are affected, we > >>> recommend that we stay on OpenBLAS last known stable version 0.2.20 > that > >>> we've been using. I will assume lazy consensus and prepare the fix > >>> (1.2.1.post0). > >>> > >>> -sz > >>> > On Tue, Jul 24, 2018 at 3:35 PM, Tong He wrote: > > Recently there's an issue regarding the inconsistent result from gluon > forward: > > https://github.com/apache/incubator-mxnet/issues/11853 > > Given a constant input image and loaded pretrained parameters, we > expect > >>> a > deterministic output from arbitrary repeats of forwards. However from > the > issue I see that the forwarded result is non-determinstic. It is > harmful > >>> as > it makes the results from experments/benchmarks/inference > meaningless. > > Therefore I propose to block the 1.3 release before it gets resolved. > > >>> >
Re: Release blocker: non-determinstic forward in gluon
Tong, That's great news. I'm glad that OpenBLAS people are responding so quickly. In that case it's probably a better idea to use that version instead. The latest OpenBLAS version brings many optimization for all kinds of hardware. -sz On Fri, Jul 27, 2018 at 11:10 AM, Tong He wrote: > Hi Sheng, > > I also opened an issue on OpenBLAS repo: > https://github.com/xianyi/OpenBLAS/issues/1700 . > > As informed that "0.3.2 should be released this weekend", I tested their > develope branch as well, and seems the new version has fixed the bug. > > Since OpenBLAS 0.3.2 could also have performance improvement, therefore I > propose to wait for OpenBLAS 0.3.2 for our pip post release. > > > Best regards, > > Tong He > > 2018-07-27 10:54 GMT-07:00 Sheng Zha : > > > Forgot to mention, the post release version is a pip package version. > > > > -sz > > > > > On Jul 27, 2018, at 10:42 AM, Sheng Zha wrote: > > > > > > In this case we can regard it as a release problem, which is usually > > what post release versions are for. It’s still the same release with > > different dependency, so there is no code change needed. > > > > > > -sz > > > > > > > > >> On Jul 27, 2018, at 8:31 AM, Steffen Rochel > > wrote: > > >> > > >> Hi Tong - thanks for root causing the problem. > > >> Sheng - what is 1.2.1.post0? Shouldn't a patch with fix be released as > > >> 1.2.2? > > >> Steffen > > >> > > >>> On Thu, Jul 26, 2018 at 5:33 PM Sheng Zha > wrote: > > >>> > > >>> Dear users and developers of Apache MXNet (Incubating), > > >>> > > >>> Thanks to Tong's dedication, the root cause for this issue was > > identified > > >>> to be instability in OpenBLAS's latest stable version 0.3.1. For > > details, > > >>> see Tong's comment > > >>> < > > >>> https://github.com/apache/incubator-mxnet/issues/11853# > > issuecomment-408272772 > > > > >>> . > > >>> > > >>> Since both the nightly build and the 1.2.1 wheels are affected, we > > >>> recommend that we stay on OpenBLAS last known stable version 0.2.20 > > that > > >>> we've been using. I will assume lazy consensus and prepare the fix > > >>> (1.2.1.post0). > > >>> > > >>> -sz > > >>> > > On Tue, Jul 24, 2018 at 3:35 PM, Tong He wrote: > > > > Recently there's an issue regarding the inconsistent result from > gluon > > forward: > > > > https://github.com/apache/incubator-mxnet/issues/11853 > > > > Given a constant input image and loaded pretrained parameters, we > > expect > > >>> a > > deterministic output from arbitrary repeats of forwards. However > from > > the > > issue I see that the forwarded result is non-determinstic. It is > > harmful > > >>> as > > it makes the results from experments/benchmarks/inference > > meaningless. > > > > Therefore I propose to block the 1.3 release before it gets > resolved. > > > > >>> > > >
Re: Pylint Undefined variable/name error
On 2018/07/27 03:05:18, sandeep krishnamurthy wrote: > Thanks Vandana for picking up this issue. I think this is important to be > fixed and enabled in CI. Please let us know effort required to fix these > issues and we all can jump in and help you. > > On Thu, Jul 26, 2018, 7:27 PM Vandana Kannan wrote: > > > Hi All, > > > > On enabling the option "undefined-variable" in pylint (in pylintrc) and > > executing on the latest code, 52 errors show up (most of them from the > > example folder). These could lead to Python NameError at runtime. The > > errors are documented in > > https://github.com/apache/incubator-mxnet/issues/11904. > > > > Currently, this Pylint option is disabled in CI and pylint is not executed > > on the example folder. > > > > It might be better to enable this option in CI to catch these errors early > > on, and also work on fixing the errors. Any thoughts/suggestions? > > > > Thanks, > > Vandana > > > Thanks Sandeep. That would be great! We could start with fixing the backlog in the example folder by checking one example at a time. Since pylint is not enabled for these files by default, some of the examples have lint issues other than undefined-variable too. To execute this for a particular example on your dev machine, 1. Modify the enable section of $(ROOTDIR)/tests/ci_build/pylintrc to include "undefined-variable". Remove “undefined-variable” from the disable section. 2. pylint --rcfile=$(ROOTDIR)/tests/ci_build/pylintrc --ignore-patterns=".*\.so$$,.*\.dll$$,.*\.dylib$$" /*.py Example: pylint --rcfile=$(ROOTDIR)/tests/ci_build/pylintrc --ignore-patterns=".*\.so$$,.*\.dll$$,.*\.dylib$$" $(ROOTDIR)/example/profiler/*.py (ROOTDIR is the path to mxnet source on your machine). This would have to be checked for Python 2 and Python 3. Thanks, Vandana
[DISCUSS] improve MXNet Scala release process
Hi all, Recently contributors on Scala Language development worked together and finally able to publish Scala package on Maven. Now I would like to raise a discussion to automate Scala release process and also discover a standard way to release different packages for MXNet so we won’t ask any individuals to spend a long time to publish the package. There are three blocks that stop this automated process: 1. How to build general hardware-compatible backend dependencies for MXNet (Linux CPU/GPU Mac OSX) 2. How to automate the frontend release process and CI integration 3. How to keep credentials for the release in the pipeline Scala Release process created by Naveen: https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process Thanks, Qing
Re: [DISCUSS] improve MXNet Scala release process
Qing, For 1, why would it be a blocker, given that there were previous releases? Has there been compatibility issues for scala packages? If so, why did we release? There are many maven packages that include binary already, so if we can find the binary for all dependency it's probably best to link to them, and let maven manage these dependencies. For 2, since the current CI is based on Jenkins, I imagine it would be some sort of Jenkins pipeline? Other people who're better versed in Jenkins can chime in. Personally, I'm interested in 3 as well. I have a pipeline for building pip packages that's currently not utilizing the CI, and the item 3 is the blocker too. Once you finish, it would be great to refer to the same solution, so that I can move it into the same CI. -sz On Fri, Jul 27, 2018 at 4:37 PM, Qing Lan wrote: > Hi all, > > Recently contributors on Scala Language development worked together and > finally able to publish Scala package on Maven. Now I would like to raise a > discussion to automate Scala release process and also discover a standard > way to release different packages for MXNet so we won’t ask any individuals > to spend a long time to publish the package. > > There are three blocks that stop this automated process: > > 1. How to build general hardware-compatible backend dependencies for > MXNet (Linux CPU/GPU Mac OSX) > 2. How to automate the frontend release process and CI integration > 3. How to keep credentials for the release in the pipeline > > Scala Release process created by Naveen: https://cwiki.apache.org/ > confluence/display/MXNET/MXNet-Scala+Release+Process > > Thanks, > Qing > >
Re: [DISCUSS] improve MXNet Scala release process
I collaborate with Qing as a part of my day time job, to give you a little more perspective on the proposed work For 1) What we found is that users often run into conflicts when they use a different version of the dependency(CUDA, CUDNN, OpenBLAS, OpenCV, etc,.) and the one we build with MXNet backend and use in the MXNet Scala package. Also it makes its not very straight-forward for users to install these dependencies themselves in order to lower the entry barrier and to make everything work out of the box we are thinking to build MXNet all these dependencies with MXNet (as a static library) and embed them in the MXNet Scala package. This is also inspired by the work you have done for Apache MXNet pip packages, Ideally I would like to reuse some of that work. Maven does not manage the binaries, you still have to build the binary and release them but what dependencies the binaries are built with is what causes the confusion and problems. In the past there were 20+ packages of MXNet and I reduced it to 3 (OSX, Linux-CPU, Linux-GPU -- please see the discussion thread below), with the latest version of the dependencies and we'll build/manage additional packages based on the user demand. Please see the previous discussion about this topic. 1) Scala Maven Package discussion: https://lists.apache.org/thread.html/c3846515fc5560d826e7b6f47e9b8b6728a925e6f8decb9676456144@%3Cdev.mxnet.apache.org%3E 2) Config for Scala Package: https://lists.apache.org/thread.html/0be6beb89cc2a792e7ba861a251f9a9a0b81fa36a5a0cd59d9f2cb6f@%3Cdev.mxnet.apache.org%3E 3) Current Scala Package Release process: https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process Hope that answers On Fri, Jul 27, 2018 at 4:59 PM, Sheng Zha wrote: > Qing, > > For 1, why would it be a blocker, given that there were previous releases? > Has there been compatibility issues for scala packages? If so, why did we > release? > There are many maven packages that include binary already, so if we can > find the binary for all dependency it's probably best to link to them, and > let maven manage these dependencies. > > For 2, since the current CI is based on Jenkins, I imagine it would be some > sort of Jenkins pipeline? Other people who're better versed in Jenkins can > chime in. > > Personally, I'm interested in 3 as well. I have a pipeline for building pip > packages that's currently not utilizing the CI, and the item 3 is the > blocker too. Once you finish, it would be great to refer to the same > solution, so that I can move it into the same CI. > > -sz > > On Fri, Jul 27, 2018 at 4:37 PM, Qing Lan wrote: > > > Hi all, > > > > Recently contributors on Scala Language development worked together and > > finally able to publish Scala package on Maven. Now I would like to > raise a > > discussion to automate Scala release process and also discover a standard > > way to release different packages for MXNet so we won’t ask any > individuals > > to spend a long time to publish the package. > > > > There are three blocks that stop this automated process: > > > > 1. How to build general hardware-compatible backend dependencies for > > MXNet (Linux CPU/GPU Mac OSX) > > 2. How to automate the frontend release process and CI integration > > 3. How to keep credentials for the release in the pipeline > > > > Scala Release process created by Naveen: https://cwiki.apache.org/ > > confluence/display/MXNET/MXNet-Scala+Release+Process > > > > Thanks, > > Qing > > > > >
Re: [DISCUSS] improve MXNet Scala release process
Thanks, Naveen. Once we have clarity on 3), it should be no problem for scala to reuse the same solution. For 1), if this is indeed an issue, it seems that we may have rushed a bit on the scala releases. Are there any user reports? -sz On Fri, Jul 27, 2018 at 5:26 PM, Naveen Swamy wrote: > I collaborate with Qing as a part of my day time job, to give you a little > more perspective on the proposed work > > For 1) > What we found is that users often run into conflicts when they use a > different version of the dependency(CUDA, CUDNN, OpenBLAS, OpenCV, etc,.) > and the one we build with MXNet backend and use in the MXNet Scala package. > Also it makes its not very straight-forward for users to install these > dependencies themselves in order to lower the entry barrier and to make > everything work out of the box we are thinking to build MXNet all these > dependencies with MXNet (as a static library) and embed them in the MXNet > Scala package. This is also inspired by the work you have done for Apache > MXNet pip packages, Ideally I would like to reuse some of that work. > > Maven does not manage the binaries, you still have to build the binary and > release them but what dependencies the binaries are built with is what > causes the confusion and problems. > In the past there were 20+ packages of MXNet and I reduced it to 3 (OSX, > Linux-CPU, Linux-GPU -- please see the discussion thread below), with the > latest version of the dependencies and we'll build/manage additional > packages based on the user demand. > > Please see the previous discussion about this topic. > > 1) Scala Maven Package discussion: > https://lists.apache.org/thread.html/c3846515fc5560d826e7b6f47e9b8b > 6728a925e6f8decb9676456144@%3Cdev.mxnet.apache.org%3E > 2) Config for Scala Package: > https://lists.apache.org/thread.html/0be6beb89cc2a792e7ba861a251f9a > 9a0b81fa36a5a0cd59d9f2cb6f@%3Cdev.mxnet.apache.org%3E > 3) Current Scala Package Release process: > https://cwiki.apache.org/confluence/display/MXNET/ > MXNet-Scala+Release+Process > > > Hope that answers > > > On Fri, Jul 27, 2018 at 4:59 PM, Sheng Zha wrote: > > > Qing, > > > > For 1, why would it be a blocker, given that there were previous > releases? > > Has there been compatibility issues for scala packages? If so, why did we > > release? > > There are many maven packages that include binary already, so if we can > > find the binary for all dependency it's probably best to link to them, > and > > let maven manage these dependencies. > > > > For 2, since the current CI is based on Jenkins, I imagine it would be > some > > sort of Jenkins pipeline? Other people who're better versed in Jenkins > can > > chime in. > > > > Personally, I'm interested in 3 as well. I have a pipeline for building > pip > > packages that's currently not utilizing the CI, and the item 3 is the > > blocker too. Once you finish, it would be great to refer to the same > > solution, so that I can move it into the same CI. > > > > -sz > > > > On Fri, Jul 27, 2018 at 4:37 PM, Qing Lan wrote: > > > > > Hi all, > > > > > > Recently contributors on Scala Language development worked together and > > > finally able to publish Scala package on Maven. Now I would like to > > raise a > > > discussion to automate Scala release process and also discover a > standard > > > way to release different packages for MXNet so we won’t ask any > > individuals > > > to spend a long time to publish the package. > > > > > > There are three blocks that stop this automated process: > > > > > > 1. How to build general hardware-compatible backend dependencies for > > > MXNet (Linux CPU/GPU Mac OSX) > > > 2. How to automate the frontend release process and CI integration > > > 3. How to keep credentials for the release in the pipeline > > > > > > Scala Release process created by Naveen: https://cwiki.apache.org/ > > > confluence/display/MXNET/MXNet-Scala+Release+Process > > > > > > Thanks, > > > Qing > > > > > > > > >
Re: [DISCUSS] improve MXNet Scala release process
Hi all, about the credential management: We already have a solution based on restricted slaves [1] and AWS secrets manager [2] that is generally classified to generate binaries and handle credentials. It was designed with continuous deployment in mind, but we haven't tested it in that field yet. To properly assess the requirements, it would be great if we have this security critical part outlined for each release pipeline. We could then check and see if our existing solution matches all requirements or if further work is necessary. Best regards, Marco [1] https://cwiki.apache.org/confluence/display/MXNET/Restricted+jobs+and+nodes [2] https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html Naveen Swamy schrieb am Sa., 28. Juli 2018, 02:27: > I collaborate with Qing as a part of my day time job, to give you a little > more perspective on the proposed work > > For 1) > What we found is that users often run into conflicts when they use a > different version of the dependency(CUDA, CUDNN, OpenBLAS, OpenCV, etc,.) > and the one we build with MXNet backend and use in the MXNet Scala package. > Also it makes its not very straight-forward for users to install these > dependencies themselves in order to lower the entry barrier and to make > everything work out of the box we are thinking to build MXNet all these > dependencies with MXNet (as a static library) and embed them in the MXNet > Scala package. This is also inspired by the work you have done for Apache > MXNet pip packages, Ideally I would like to reuse some of that work. > > Maven does not manage the binaries, you still have to build the binary and > release them but what dependencies the binaries are built with is what > causes the confusion and problems. > In the past there were 20+ packages of MXNet and I reduced it to 3 (OSX, > Linux-CPU, Linux-GPU -- please see the discussion thread below), with the > latest version of the dependencies and we'll build/manage additional > packages based on the user demand. > > Please see the previous discussion about this topic. > > 1) Scala Maven Package discussion: > > https://lists.apache.org/thread.html/c3846515fc5560d826e7b6f47e9b8b6728a925e6f8decb9676456144@%3Cdev.mxnet.apache.org%3E > 2) Config for Scala Package: > > https://lists.apache.org/thread.html/0be6beb89cc2a792e7ba861a251f9a9a0b81fa36a5a0cd59d9f2cb6f@%3Cdev.mxnet.apache.org%3E > 3) Current Scala Package Release process: > > https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process > > > Hope that answers > > > On Fri, Jul 27, 2018 at 4:59 PM, Sheng Zha wrote: > > > Qing, > > > > For 1, why would it be a blocker, given that there were previous > releases? > > Has there been compatibility issues for scala packages? If so, why did we > > release? > > There are many maven packages that include binary already, so if we can > > find the binary for all dependency it's probably best to link to them, > and > > let maven manage these dependencies. > > > > For 2, since the current CI is based on Jenkins, I imagine it would be > some > > sort of Jenkins pipeline? Other people who're better versed in Jenkins > can > > chime in. > > > > Personally, I'm interested in 3 as well. I have a pipeline for building > pip > > packages that's currently not utilizing the CI, and the item 3 is the > > blocker too. Once you finish, it would be great to refer to the same > > solution, so that I can move it into the same CI. > > > > -sz > > > > On Fri, Jul 27, 2018 at 4:37 PM, Qing Lan wrote: > > > > > Hi all, > > > > > > Recently contributors on Scala Language development worked together and > > > finally able to publish Scala package on Maven. Now I would like to > > raise a > > > discussion to automate Scala release process and also discover a > standard > > > way to release different packages for MXNet so we won’t ask any > > individuals > > > to spend a long time to publish the package. > > > > > > There are three blocks that stop this automated process: > > > > > > 1. How to build general hardware-compatible backend dependencies for > > > MXNet (Linux CPU/GPU Mac OSX) > > > 2. How to automate the frontend release process and CI integration > > > 3. How to keep credentials for the release in the pipeline > > > > > > Scala Release process created by Naveen: https://cwiki.apache.org/ > > > confluence/display/MXNET/MXNet-Scala+Release+Process > > > > > > Thanks, > > > Qing > > > > > > > > >
