Hi all,
I have created a design document on Automated Scala publish in here:
https://cwiki.apache.org/confluence/display/MXNET/Automated+MXNet+Scala+release+design
Please kindly review it and leave any thoughts you may have.
Thanks,
Qing
On 8/3/18, 10:26 AM, "Naveen Swamy" <mnnav...@gmail.com> wrote:
Hi Carin,
The thinking right now is to publish nightly to the Apache Snapshot
repository, building and validating with integration tests. We(Qing, me and
Andrew Ayres) will work on a elaborate document detailing the process and
we'll loop you in as well.
Thanks, Naveen
On Fri, Aug 3, 2018 at 8:35 AM, Carin Meier <carinme...@gmail.com> wrote:
> Hi,
>
> I was thinking about the process for publishing the Clojure jar as well. I
> think, since it will be published to Nexus/Maven and the building of it
> depends on the Scala jar artifact, it might make sense to combine the
> publishing of the Clojure jar at the same time as the Scala Jar. I haven't
> worked out all the details yet, but I'm thinking with that it would be a
> couple command lines in the same script that handles the Scala deployment.
>
> Or if it is a separate process, it might be able to share common parts.
>
> Could you please keep me in the loop of the deployment process so we can
> figure out the best place to work in Clojure as well?
>
> Thanks,
> Carin
>
> On Tue, Jul 31, 2018 at 3:49 PM, Qing Lan <lanking...@live.com> wrote:
>
> > Upon offline discussion with Marco,
> >
> > He proposed a plan that can actually help us conduct 3):
> > 1. This job will not be trigger when PR runs and strictly limit
> > that only committer can run the restricted job.
> > 2. The code being run in there will only covers the code from
the
> > branch you choose to go, it will be committers responsibilities not to
> > merge any trivial credential grabber code.
> > 3. Test this is simple. The restricted job uses a similar
> > architecture with current CI. You can send a PR with dockerfiles,
scripts
> > and configurations on Jenkins to give it a test to run the job with a
> mock
> > credential. Finally please contact people working on CI to give it a
test
> > run and they will do the last step to merge your change to CI.
> > 4. Marco also mentioned the security level of the credentials.
> The
> > credential being used in the AWS Credential services will be assigned
> with
> > an individual IAM role, which only allows to access to the credentials
> that
> > role being assigned to, and used in the restricted job you have set up.
> >
> > I would also like to encourage people in this list to join the
> > https://cwiki.apache.org/confluence/display/MXNET/
> > MXNet+Berlin+Office+Hours as the people who is working on improving the
> > CI are there ready to help.
> >
> > Thanks,
> > Qing
> >
> >
> > On 7/28/18, 11:44 PM, "Qing Lan" <lanking...@live.com> wrote:
> >
> > Thanks Marco, Naveen and Sheng's feedback.
> >
> > About the 1): Scala side will only pack the mxnet binary only and
use
> > dynamic links to all the rest dependencies. So indeed it will require
> users
> > to install all deps as the same as the builder platforms version and
this
> > will make them hard to use. Let's please collaborate and create a (set
> of)
> > general CI script(s) to install the deps and bring static links to the
> > package.
> >
> > About 3): it is indeed a general problems for both Scala and Python
> > publish. If there is a good way we can safely store the credentials, we
> can
> > definitely give automated publish a go. And thanks again for Marco's
> option
> > provided below, I think we can make use of the restricted slaves and
give
> > it a test run. And to Marco:
> > 1. Will this restricted jobs being triggered in every PR runs or
> > it just depends on where you put it (like I put in nightly it will never
> > be trigger in PR)? Will there be a potential risk like a PR attack
> > (create a PR to grab credentials)
> >
> > 2. How do we make sure the coding being run there is under
> control
> > and not be changed by anyone?
> >
> > 3. If I want to test this functionality, where is the best place
> > to create the job and make a test run?
> >
> > Thanks,
> > Qing
> >
> >
> >
> > On 7/27/18, 5:44 PM, "Marco de Abreu" <marco.g.ab...@googlemail.com.
> INVALID>
> > wrote:
> >
> > Hi all,
> >
> > about the credential management: We already have a solution
based
> > on
> > restricted slaves [1] and AWS secrets manager [2] that is
> generally
> > classified to generate binaries and handle credentials. It was
> > designed
> > with continuous deployment in mind, but we haven't tested it in
> > that field
> > yet.
> >
> > To properly assess the requirements, it would be great if we
have
> > this
> > security critical part outlined for each release pipeline. We
> > could then
> > check and see if our existing solution matches all requirements
> or
> > if
> > further work is necessary.
> >
> > Best regards,
> > Marco
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/MXNET/
> > Restricted+jobs+and+nodes
> > [2] https://docs.aws.amazon.com/secretsmanager/latest/
> > userguide/intro.html
> >
> > On 7/27/18, 5:43 PM, "Sheng Zha" <szha....@gmail.com> wrote:
> >
> > Thanks, Naveen. Once we have clarity on 3), it should be no
> > problem for
> > scala to reuse the same solution. For 1), if this is indeed an
> > issue, it
> > seems that we may have rushed a bit on the scala releases. Are
> > there any
> > user reports?
> >
> > -sz
> >
> > On Fri, Jul 27, 2018 at 5:26 PM, Naveen Swamy <
> mnnav...@gmail.com>
> > wrote:
> >
> > > I collaborate with Qing as a part of my day time job, to give
> > you a little
> > > more perspective on the proposed work
> > >
> > > For 1)
> > > What we found is that users often run into conflicts when they
> > use a
> > > different version of the dependency(CUDA, CUDNN, OpenBLAS,
> > OpenCV, etc,.)
> > > and the one we build with MXNet backend and use in the MXNet
> > Scala package.
> > > Also it makes its not very straight-forward for users to
> install
> > these
> > > dependencies themselves in order to lower the entry barrier
and
> > to make
> > > everything work out of the box we are thinking to build MXNet
> > all these
> > > dependencies with MXNet (as a static library) and embed them
in
> > the MXNet
> > > Scala package. This is also inspired by the work you have done
> > for Apache
> > > MXNet pip packages, Ideally I would like to reuse some of that
> > work.
> > >
> > > Maven does not manage the binaries, you still have to build
the
> > binary and
> > > release them but what dependencies the binaries are built with
> > is what
> > > causes the confusion and problems.
> > > In the past there were 20+ packages of MXNet and I reduced it
> to
> > 3 (OSX,
> > > Linux-CPU, Linux-GPU -- please see the discussion thread
> below),
> > with the
> > > latest version of the dependencies and we'll build/manage
> > additional
> > > packages based on the user demand.
> > >
> > > Please see the previous discussion about this topic.
> > >
> > > 1) Scala Maven Package discussion:
> > > https://lists.apache.org/thread.html/
> > c3846515fc5560d826e7b6f47e9b8b
> > > 6728a925e6f8decb9676456144@%3Cdev.mxnet.apache.org%3E
> > > 2) Config for Scala Package:
> > > https://lists.apache.org/thread.html/
> > 0be6beb89cc2a792e7ba861a251f9a
> > > 9a0b81fa36a5a0cd59d9f2cb6f@%3Cdev.mxnet.apache.org%3E
> > > 3) Current Scala Package Release process:
> > > https://cwiki.apache.org/confluence/display/MXNET/
> > > MXNet-Scala+Release+Process
> > >
> > >
> > > Hope that answers
> > >
> > >
> > > On Fri, Jul 27, 2018 at 4:59 PM, Sheng Zha <szha....@gmail.com
> >
> > wrote:
> > >
> > > > Qing,
> > > >
> > > > For 1, why would it be a blocker, given that there were
> > previous
> > > releases?
> > > > Has there been compatibility issues for scala packages? If
> so,
> > why did we
> > > > release?
> > > > There are many maven packages that include binary already,
so
> > if we can
> > > > find the binary for all dependency it's probably best to
link
> > to them,
> > > and
> > > > let maven manage these dependencies.
> > > >
> > > > For 2, since the current CI is based on Jenkins, I imagine
it
> > would be
> > > some
> > > > sort of Jenkins pipeline? Other people who're better versed
> in
> > Jenkins
> > > can
> > > > chime in.
> > > >
> > > > Personally, I'm interested in 3 as well. I have a pipeline
> for
> > building
> > > pip
> > > > packages that's currently not utilizing the CI, and the item
> 3
> > is the
> > > > blocker too. Once you finish, it would be great to refer to
> > the same
> > > > solution, so that I can move it into the same CI.
> > > >
> > > > -sz
> > > >
> > > > On Fri, Jul 27, 2018 at 4:37 PM, Qing Lan <
> lanking...@live.com>
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Recently contributors on Scala Language development worked
> > together and
> > > > > finally able to publish Scala package on Maven. Now I
would
> > like to
> > > > raise a
> > > > > discussion to automate Scala release process and also
> > discover a
> > > standard
> > > > > way to release different packages for MXNet so we won’t
ask
> > any
> > > > individuals
> > > > > to spend a long time to publish the package.
> > > > >
> > > > > There are three blocks that stop this automated process:
> > > > >
> > > > > 1. How to build general hardware-compatible backend
> > dependencies for
> > > > > MXNet (Linux CPU/GPU Mac OSX)
> > > > > 2. How to automate the frontend release process and CI
> > integration
> > > > > 3. How to keep credentials for the release in the
> pipeline
> > > > >
> > > > > Scala Release process created by Naveen:
> > https://cwiki.apache.org/
> > > > > confluence/display/MXNET/MXNet-Scala+Release+Process
> > > > >
> > > > > Thanks,
> > > > > Qing
> > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> >
> >
>
