Thank you Chaitanya and Marco for helping the MXNet community. On Mon, Mar 23, 2020 at 12:56 PM Marco de Abreu <marco.g.ab...@gmail.com> wrote:
> Sure, already done. > > -Marco > > On Mon, Mar 23, 2020 at 8:53 PM Chaitanya Bapat <chai.ba...@gmail.com> > wrote: > > > Hello, > > Update: Apache Infra Ticket for MXNet Bot > > Thanks once again, Marco for opening the ticket. But turns out, Apache > > Infra folks closed it stating: "Security concerns around allowing unknown > > person to submit PR and run our hardware". Furthermore, it goes onto > state > > that bot circumvents the dependence on Jenkins Admins which is like > solving > > a problem that doesn't exist. > > > > I sense there is some confusion in the communication (maybe on my part). > It > > turns out the security concerns aren't actually correct. > > > > 1. Unknown person can submit a PR (before & after bot proposal), and run > > our hardware (pre as well as post bot). > > 2. Code should be reviewed by somebody with an ICLA on file. This doesn't > > change either. Prior to merging a PR, code has to be approved by a > > committer just like before. > > Overall it looks like the job of the bot isn't clear to folks in Apache > > Infra. Bot simply is a means for triggering CI (which could be done > > manually by Log In to Jenkins -> PR -> Job -> Build) and doesn't quite > > tweak with merging procedure. Yes, only addition is now unknown person > (PR > > Author) can trigger CI with a message (but that was possible anyway by > > pushing a commit. Bot just prevents users from pushing empty commits and > > building entire suite). > > > > As can be seen from last 10 open PRs as of Monday 23rd March, 12pm PT > most > > PRs fail on 1/2 jobs. In such a scenario, the proposed MXNet bot would > come > > in handy for just invoking CI on that specific build (instead of a > > non-committer PR Author to push empty commit : hurting on the resource, > > time & cost considerations apart from undesirable dev experience) > > > > @Marco Since I am a non-committer, I guess these 2 clarifications need to > > be conveyed to the Apache Infra by someone with Committer access. > > > > What do you think? > > > > Thanks, > > Chai > > > > On Sat, 21 Mar 2020 at 16:08, Marco de Abreu <marco.g.ab...@gmail.com> > > wrote: > > > > > Hello, > > > > > > the ticket has been created: > > > https://issues.apache.org/jira/browse/INFRA-20005 > > > > > > Best regards, > > > Marco > > > > > > On Thu, Mar 19, 2020 at 11:49 PM Marco de Abreu < > marco.g.ab...@gmail.com > > > > > > wrote: > > > > > > > Sounds like a good plan! > > > > > > > > Please send me the URL (please make sure it's backed by DNS and not > > just > > > > the gateway URL) of the webhook handler, GitHub events you're > > interested > > > in > > > > and the shared secret in a private email to my personal email > address. > > I > > > > will then create the ticket with Apache infra. > > > > > > > > -Marco > > > > > > > > Chaitanya Bapat <chai.ba...@gmail.com> schrieb am Do., 19. März > 2020, > > > > 23:07: > > > > > > > >> @Marco Alright, it makes total sense to test out the Bot feature > > > alongside > > > >> auto-trigger as a transition. > > > >> > > > >> Path Forward: > > > >> 1. Setup MXNet Bot on apache/incubator-mxnet repo (GitHub WebHook > and > > > >> Infra) > > > >> 2. We don't turn off automatic trigger of PR builds for now. > > > >> 3. Hopefully, bot is used by developers to trigger specific jobs > > > >> 4. Later on (say around April 20), let's discuss the possibility of > > > >> switching off auto-trigger (with appropriate data) if it makes > sense. > > > >> Thanks Marco for volunteering to help enable the web hook on > > > >> apache/incubator-mxnet. Let me know if we can sync up on Slack > channel > > > to > > > >> get the ball rolling. > > > >> > > > >> Thanks once again for the entire community to step in and help try > out > > > >> this > > > >> Bot. > > > >> Chai > > > >> > > > >> On Wed, 18 Mar 2020 at 17:07, Marco de Abreu < > marco.g.ab...@gmail.com > > > > > > >> wrote: > > > >> > > > >> > Hi, that's correct. But as stated previously, it's not an option > to > > > >> remove > > > >> > the hook. For now, I'd like to see how the system behaves while > it's > > > >> > optional. Later on, we can talk about revisiting this decision. > But > > to > > > >> me > > > >> > it's not an option to deploy an entirely new system and approach > > > without > > > >> > having a transition or even a timeframe in which we are able to > fall > > > >> back. > > > >> > > > > >> > I'm happy to support the deployment of the bot and add an > additional > > > >> > webhook to enable it's functionality to support selective > triggering > > > by > > > >> PR > > > >> > authors and committers, but I will not support the disabling of > > > >> automatic > > > >> > triggering of branches or PRs. > > > >> > > > > >> > -Marco > > > >> > > > > >> > Chaitanya Bapat <chai.ba...@gmail.com> schrieb am Mi., 18. März > > 2020, > > > >> > 21:00: > > > >> > > > > >> > > Hey Marco, > > > >> > > > > > >> > > I thought currently every commit on PR and master triggers CI > > > >> > > because > > > >> > > a. github webhook points to Jenkins Server > > > >> > > b. GH Webhook events trigger builds on Jenkins for all commits > to > > > any > > > >> > > branch in apache/incubator-mxnet > > > >> > > may it be master/PR/non-PR > > > >> > > Reason: > > > >> > > Because all the 3 types of branches are discovered by Jenkins > > > (non-PR > > > >> > > (including master) and PR) > > > >> > > > > > >> > > Proposal: Remove GitHub WebHook to Jenkins and replace with GH > > > >> Webhook to > > > >> > > Lambda > > > >> > > But after I remove the github webhook that points to Jenkins : > > *N**o > > > >> > commit > > > >> > > will trigger Jenkins build by default* (as Jenkins wont receive > GH > > > >> > events) > > > >> > > Only those that Bot deems fit will be triggered (using Jenkins > API > > > >> > invoked > > > >> > > by Lambda). > > > >> > > Hence its needed to handle that case of master merge. > > > >> > > Am I understanding this correctly? > > > >> > > > > > >> > > On Wed, 18 Mar 2020 at 04:23, Marco de Abreu < > > > marco.g.ab...@gmail.com > > > >> > > > > >> > > wrote: > > > >> > > > > > >> > > > Thanks Chai, sounds good to me. Could you elaborate a bit on > the > > > >> point > > > >> > > > about triggering a CI run after the PR has been merged? We > > already > > > >> to > > > >> > > that > > > >> > > > automatically for the master, so what's the benefit to do it > > > twice? > > > >> > > > > > > >> > > > -Marco > > > >> > > > > > > >> > > > Chaitanya Bapat <chai.ba...@gmail.com> schrieb am Mi., 18. > März > > > >> 2020, > > > >> > > > 09:30: > > > >> > > > > > > >> > > > > Update: > > > >> > > > > > > > >> > > > > > we can ensure that all CI runs ran on the commit that > will > > be > > > >> > merged > > > >> > > > > @Sam Skalicky <samskali...@gmail.com> Branch Protection is > > > added > > > >> to > > > >> > > > public > > > >> > > > > MXNet repo. It ensures that for every PR to be merged, the > CI > > > >> passes. > > > >> > > All > > > >> > > > > the jobs selected "required" jobs will have to be green for > > the > > > >> PR to > > > >> > > be > > > >> > > > > merged. Ofcourse, users with "Adminstrator" access can merge > > > >> without > > > >> > it > > > >> > > > but > > > >> > > > > that's just a backdoor. It is the case now and will continue > > to > > > be > > > >> > the > > > >> > > > case > > > >> > > > > with the inclusion of Bot. > > > >> > > > > > > > >> > > > > > easily verify that the CI has executed all runs on the > > commit > > > >> that > > > >> > > will > > > >> > > > > be merged > > > >> > > > > GitHub UI shows all the jobs and the status corresponding to > > it > > > on > > > >> > > every > > > >> > > > > commit. That should suffice. For the merged commits, Repo -> > > > >> Commits > > > >> > -> > > > >> > > > > Commit ID (Status) can be tracked currently (only way that I > > > know > > > >> > of). > > > >> > > > > Moreover, it is beyond the scope of this project (and > possibly > > > >> out of > > > >> > > our > > > >> > > > > control since this is purely GitHub UI specific use-case). > > > >> > > > > > > > >> > > > > Thanks @przemyslaw for supporting the opt-in. > > > >> > > > > > > > >> > > > > Thanks everyone in the community for sharing concerns, > voicing > > > >> your > > > >> > > > opinion > > > >> > > > > and participating in the discussion. > > > >> > > > > Thanks to those who attended the demo last Friday. > > > >> > > > > > > > >> > > > > Action items from that discussion > > > >> > > > > 1. Handle master merge builds [Done] > > > >> > > > > Bot runs entire CI suite after the PR is merged and comments > > on > > > >> the > > > >> > PR > > > >> > > > > about the same. > > > >> > > > > Design decision : > > > >> > > > > MXNet Bot comment about master merge build on the *merge > > commit > > > vs > > > >> > PR*. > > > >> > > > > After the PR is merged, Bot runs entire CI and comments the > > > >> result of > > > >> > > CI > > > >> > > > > trigger on the PR (because it is easy to track on a PR > rather > > > than > > > >> > > > > commenting inside the merge commit) > > > >> > > > > > > > >> > > > > 2. Idempotent condition > > > >> > > > > In case of already running build, if an attempt is made to > > > >> retrigger > > > >> > > the > > > >> > > > > job then what should be the response > > > >> > > > > a. Not to re-trigger, let the ongoing build continue till > > > >> completion > > > >> > > > > b. End the ongoing build and re-trigger > > > >> > > > > c. Let the ongoing build continue, re-trigger new build > > > >> > > > > > > > >> > > > > From resource saving point of view, *c* looks costly and a > can > > > be > > > >> > > > > avoided/optimized by B. > > > >> > > > > In case when a re-trigger was started "erroneously" then > > killing > > > >> > > ongoing > > > >> > > > > build and re-trigger is a waste. > > > >> > > > > In case when ongoing build failed in one sub-part, then > > > >> re-triggering > > > >> > > is > > > >> > > > > justified. > > > >> > > > > Erroneous re-triggers would be less often while conscious > > > >> re-triggers > > > >> > > to > > > >> > > > > suppress failure is more common use-case. It looks like a > safe > > > >> > > assumption > > > >> > > > > to make given the trade-off. > > > >> > > > > [Open to debate] > > > >> > > > > > > > >> > > > > 3. Add security consideration [Use of secret manager, but > > > without > > > >> > > > > auto-rotation due to Jenkins manual config requirement] > [Done] > > > >> > > > > 4. New PR Instruction message by the Bot [Done] > > > >> > > > > Thanks to the suggestion of Leonard, supported by others. > I've > > > now > > > >> > > added > > > >> > > > > the feature where the Bot comments a help message. [For > > > reference > > > >> - > > > >> > > > > https://github.com/ChaiBapchya/incubator-mxnet/pull/52] > > > >> > > > > > > > >> > > > > Barring the opt-in vs opt-out debate & idempotency, > consensus > > > was > > > >> > > quickly > > > >> > > > > reached for the rest. > > > >> > > > > > > > >> > > > > In the coming days, I hope to roll-out this feature into > Prod > > > >> (public > > > >> > > > > MXNet) for all devs to use. > > > >> > > > > > > > >> > > > > Thanks, > > > >> > > > > Chai > > > >> > > > > > > > >> > > > > On Mon, 16 Mar 2020 at 11:57, Marco de Abreu < > > > >> > marco.g.ab...@gmail.com> > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > > > Well that's generally a problem with a deferred CI > approach > > > (CI > > > >> is > > > >> > > run > > > >> > > > at > > > >> > > > > > commit and not at merge time). This can either be solved > > > through > > > >> > the > > > >> > > > > other > > > >> > > > > > proposal that's currently on dev@, by having a bot which > > does > > > >> > merges > > > >> > > > by > > > >> > > > > > having a global lock and a merge queue or by accepting the > > > >> issue. > > > >> > > > Reality > > > >> > > > > > right now is that we're running that model where two PRs > > which > > > >> are > > > >> > > > merged > > > >> > > > > > in parallel might break one another. One thing to consider > > > >> though > > > >> > is > > > >> > > > that > > > >> > > > > > this breakage would have to be introduced in two separate > > > parts > > > >> > since > > > >> > > > > > otherwise there'd be merge conflicts. I think we had that > > > >> situation > > > >> > > > twice > > > >> > > > > > so far and the result was a quick revert, so I'd say that > > > it's a > > > >> > > > problem > > > >> > > > > > that can happily be accepted. All other solutions > basically > > > >> require > > > >> > > > some > > > >> > > > > > form of single-threaded and globally locked solution which > > > >> limits > > > >> > us > > > >> > > in > > > >> > > > > > scalability. I'd recommend to just accept that risk and > > revert > > > >> a PR > > > >> > > in > > > >> > > > > case > > > >> > > > > > it actually had a conflict. > > > >> > > > > > > > > >> > > > > > -Marco > > > >> > > > > > > > > >> > > > > > On Mon, Mar 16, 2020 at 6:29 PM Skalicky, Sam > > > >> > > > <sska...@amazon.com.invalid > > > >> > > > > > > > > >> > > > > > wrote: > > > >> > > > > > > > > >> > > > > > > We probably need some way to track which CI runs ran for > > > which > > > >> > > commit > > > >> > > > > > too, > > > >> > > > > > > that way we can ensure that all CI runs ran on the > commit > > > that > > > >> > will > > > >> > > > be > > > >> > > > > > > merged. Maybe the bot can comment with the commit hash > > when > > > >> > users > > > >> > > > > > command > > > >> > > > > > > it to do something. Although since users can trigger > > > >> individual > > > >> > CI > > > >> > > > runs > > > >> > > > > > its > > > >> > > > > > > possible to have some commits run some CI runs but not > > > >> others. We > > > >> > > > need > > > >> > > > > > some > > > >> > > > > > > way to easily verify that the CI has executed all runs > on > > > the > > > >> > > commit > > > >> > > > > that > > > >> > > > > > > will be merged. > > > >> > > > > > > > > > >> > > > > > > Sam > > > >> > > > > > > > > > >> > > > > > > > On Mar 13, 2020, at 8:28 PM, Przemysław Trędak < > > > >> > > ptre...@apache.org > > > >> > > > > > > > >> > > > > > > wrote: > > > >> > > > > > > > > > > >> > > > > > > > CAUTION: This email originated from outside of the > > > >> > organization. > > > >> > > Do > > > >> > > > > not > > > >> > > > > > > click links or open attachments unless you can confirm > the > > > >> sender > > > >> > > and > > > >> > > > > > know > > > >> > > > > > > the content is safe. > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > I personally like the idea of opt-in more than > opt-out: > > > >> > > > > > > > - ultimately PR author wants the PR to be merged so > they > > > (or > > > >> > > > > committer > > > >> > > > > > > reviewing the PR) will trigger the CI > > > >> > > > > > > > - if it is easy to trigger the PR via the bot command > > then > > > >> the > > > >> > > > amount > > > >> > > > > > of > > > >> > > > > > > work per PR should be less than with opt-out (since most > > of > > > >> the > > > >> > > > commits > > > >> > > > > > > should then be marked as [skip ci] or something similar > > > >> > > > > > > > > > > >> > > > > > > > +1 to the bot making a comment on each new PR with its > > > >> commands > > > >> > > > (and > > > >> > > > > > > also explaining, or at least giving links to the general > > PR > > > >> > process > > > >> > > > so > > > >> > > > > > new > > > >> > > > > > > PR authors are not lost). Maybe we could make the bot > > > >> recognize > > > >> > if > > > >> > > > the > > > >> > > > > PR > > > >> > > > > > > author is new or existing contributor and offer advice > > based > > > >> on > > > >> > > that? > > > >> > > > > > > > > > > >> > > > > > > > Thanks > > > >> > > > > > > > Przemek > > > >> > > > > > > > > > > >> > > > > > > > On 2020/03/13 22:06:58, Marco de Abreu < > > > >> > marco.g.ab...@gmail.com> > > > >> > > > > > wrote: > > > >> > > > > > > >> Hi, > > > >> > > > > > > >> > > > >> > > > > > > >> since it's no longer necessary to push a new commit > to > > > >> trigger > > > >> > > CI, > > > >> > > > > it > > > >> > > > > > > will > > > >> > > > > > > >> already reduce the costs. But to me, requiring an > > action > > > to > > > >> > > enable > > > >> > > > > CI > > > >> > > > > > > after > > > >> > > > > > > >> a PR has been created initially, is a no go. User can > > opt > > > >> out > > > >> > of > > > >> > > > CI, > > > >> > > > > > but > > > >> > > > > > > >> the default has to be CI being triggered > automatically > > > for > > > >> > every > > > >> > > > > > commit > > > >> > > > > > > >> unless specifically disabled by a participant. I'm > also > > > >> fine > > > >> > > with > > > >> > > > > > > >> triggering certain additional jobs (think about > > running a > > > >> > > nightly > > > >> > > > > job > > > >> > > > > > > upon > > > >> > > > > > > >> request for a PR) to require a manual step, but the > PR > > > >> > > validation > > > >> > > > > > > pipelines > > > >> > > > > > > >> have to run automatically. Every check that is marked > > as > > > >> > > > "Required" > > > >> > > > > in > > > >> > > > > > > >> GitHub has to be automatically kicked off. > > > >> > > > > > > >> > > > >> > > > > > > >> -Marco > > > >> > > > > > > >> > > > >> > > > > > > >> On Fri, Mar 13, 2020 at 9:50 PM Chaitanya Bapat < > > > >> > > > > chai.ba...@gmail.com > > > >> > > > > > > > > > >> > > > > > > >> wrote: > > > >> > > > > > > >> > > > >> > > > > > > >>> Firstly, > > > >> > > > > > > >>> Sorry I missed out on attaching the mail thread that > > was > > > >> sent > > > >> > > on > > > >> > > > > 12th > > > >> > > > > > > >>> February for notifying the community of the upcoming > > > >> changes > > > >> > to > > > >> > > > the > > > >> > > > > > > MXNet > > > >> > > > > > > >>> CI > > > >> > > > > > > >>> For reference : > > > >> > > > > > > >>> > > > >> > > > > > > >>> > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://lists.apache.org/thread.html/r09a6ab2803a996fc80e00fe39ed312fa4865e8805e08df847f1addad%40%3Cdev.mxnet.apache.org%3E > > > >> > > > > > > >>> > > > >> > > > > > > >>> Now to the questions, > > > >> > > > > > > >>>> Is it possible for re-triggering a single job to be > > > >> abused? > > > >> > > > > > > >>> @Tao In the case when a user re-triggers a single > job > > > >> > multiple > > > >> > > > > times, > > > >> > > > > > > that > > > >> > > > > > > >>> will be visible in the PR conversation thread. A > > > >> committer, > > > >> > > even > > > >> > > > > > after > > > >> > > > > > > he > > > >> > > > > > > >>> has approved the PR before, generally takes a look > at > > > the > > > >> > final > > > >> > > > > state > > > >> > > > > > > of > > > >> > > > > > > >>> the PR before merging. Would it be fair to assume > the > > > >> > committer > > > >> > > > > could > > > >> > > > > > > take > > > >> > > > > > > >>> the multiple re-trigger of a single job into account > > > >> before > > > >> > > > > merging? > > > >> > > > > > > The > > > >> > > > > > > >>> committer then has the option to invoke `@mxnet-bot > > run > > > ci > > > >> > > [all] > > > >> > > > ` > > > >> > > > > to > > > >> > > > > > > >>> trigger the entire build pipeline one last to > counter > > > the > > > >> > > abuse. > > > >> > > > > This > > > >> > > > > > > is > > > >> > > > > > > >>> aligned with what @Leonard said. > > > >> > > > > > > >>> > > > >> > > > > > > >>> @Sandeep Thanks a lot for collecting and sharing > > > valuable > > > >> > data. > > > >> > > > I'd > > > >> > > > > > > concur > > > >> > > > > > > >>> with the opinion that given the existing things > > > >> committers & > > > >> > PR > > > >> > > > > > Authors > > > >> > > > > > > >>> take care of, invoking CI shouldn't be that big of > an > > > >> > > additional > > > >> > > > > > > burden. > > > >> > > > > > > >>> > > > >> > > > > > > >>> @Marco With the opt-out, the onus remains on the PR > > > >> Author. > > > >> > It > > > >> > > > > > doesn't > > > >> > > > > > > help > > > >> > > > > > > >>> reduce the resource usage. Hence, it was suggested > to > > > >> switch > > > >> > to > > > >> > > > > > > >>> opt-in. @Leo's suggestion for proactive commenting > on > > > the > > > >> > part > > > >> > > of > > > >> > > > > bot > > > >> > > > > > > makes > > > >> > > > > > > >>> sense and is doable. > > > >> > > > > > > >>> > > > >> > > > > > > >>> Default : opt-out and User initiated opt-in (with > > > >> addressing > > > >> > > > Leo's > > > >> > > > > > fix > > > >> > > > > > > for > > > >> > > > > > > >>> the usability issue you correctly pointed out ) > > > >> > > > > > > >>> @Marco How does this sound to you? > > > >> > > > > > > >>> > > > >> > > > > > > >>> Again, thank you all for chiming in and voicing your > > > >> opinion. > > > >> > > > > > > Appreciate > > > >> > > > > > > >>> it. > > > >> > > > > > > >>> We can take ahead these discussions in today's demo > > > >> meeting. > > > >> > > > > [Design > > > >> > > > > > > Doc > > > >> > > > > > > >>> < > > > >> > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot > > > >> > > > >] > > > >> > > > > > > [Demo > > > >> > > > > > > >>> Video <https://www.youtube.com/watch?v=gfOGwZId8aU > >] > > > >> > > > > > > >>> > > > >> > > > > > > >>> Thanks, > > > >> > > > > > > >>> Chai > > > >> > > > > > > >>> > > > >> > > > > > > >>> On Fri, 13 Mar 2020 at 12:34, Marco de Abreu < > > > >> > > > > > marco.g.ab...@gmail.com> > > > >> > > > > > > >>> wrote: > > > >> > > > > > > >>> > > > >> > > > > > > >>>> I'd recommend that the bot makes an initial comment > > > when > > > >> a > > > >> > PR > > > >> > > > gets > > > >> > > > > > > opened > > > >> > > > > > > >>>> and informs the users of its commands. It then > tells > > > the > > > >> > user > > > >> > > > the > > > >> > > > > > > commend > > > >> > > > > > > >>>> to opt out of CI. > > > >> > > > > > > >>>> > > > >> > > > > > > >>>> -Marco > > > >> > > > > > > >>>> > > > >> > > > > > > >>>> Lausen, Leonard <lau...@amazon.com.invalid> > schrieb > > am > > > >> Fr., > > > >> > > 13. > > > >> > > > > > März > > > >> > > > > > > >>> 2020, > > > >> > > > > > > >>>> 20:27: > > > >> > > > > > > >>>> > > > >> > > > > > > >>>>> On opt-out: People may be unaware of opt-out would > > not > > > >> use > > > >> > > it. > > > >> > > > > > There > > > >> > > > > > > is > > > >> > > > > > > >>>> no > > > >> > > > > > > >>>>> incentive to use opt-out, as the PR author doesn't > > pay > > > >> any > > > >> > > > money > > > >> > > > > > for > > > >> > > > > > > CI > > > >> > > > > > > >>>>> run. > > > >> > > > > > > >>>>> > > > >> > > > > > > >>>>> I agree with Marco though that opt-in alone may > > cause > > > >> > > usability > > > >> > > > > > > issues, > > > >> > > > > > > >>>> as > > > >> > > > > > > >>>>> contributors may not be aware of how to trigger > the > > > CI. > > > >> > > > > > > >>>>> One solution is that the bot proactively comments > on > > > >> the PR > > > >> > > and > > > >> > > > > > > reminds > > > >> > > > > > > >>>> the > > > >> > > > > > > >>>>> author to trigger running CI once the author deems > > the > > > >> PR > > > >> > > > ready. > > > >> > > > > > > >>>>> > > > >> > > > > > > >>>>> But even if we choose opt-out, the bot will still > > add > > > a > > > >> lot > > > >> > > of > > > >> > > > > > value, > > > >> > > > > > > >>> as > > > >> > > > > > > >>>> PR > > > >> > > > > > > >>>>> authors can retrigger single jobs that have failed > > due > > > >> to > > > >> > > > > > flakiness. > > > >> > > > > > > >>>>> > > > >> > > > > > > >>>>>> Is it possible for re-triggering a single job to > be > > > >> > abused? > > > >> > > > For > > > >> > > > > > > >>>> example, > > > >> > > > > > > >>>>>> the author spends two days re-triggering a flaky > > job > > > to > > > >> > make > > > >> > > > it > > > >> > > > > > > pass. > > > >> > > > > > > >>>>> > > > >> > > > > > > >>>>> Yes, this is possible. I suggest the committer who > > > >> likes to > > > >> > > > > merge a > > > >> > > > > > > PR > > > >> > > > > > > >>>>> needs to > > > >> > > > > > > >>>>> make a good judgement here if a PR is abusing the > > > >> feature, > > > >> > > and > > > >> > > > if > > > >> > > > > > so, > > > >> > > > > > > >>>>> retrigger > > > >> > > > > > > >>>>> all CI runs. > > > >> > > > > > > >>>>> > > > >> > > > > > > >>>>> Best regards > > > >> > > > > > > >>>>> Leonard > > > >> > > > > > > >>>>> > > > >> > > > > > > >>>>> On Fri, 2020-03-13 at 08:07 +0100, Marco de Abreu > > > wrote: > > > >> > > > > > > >>>>>> Thanks for the data Sandeep. In these cases it > > sounds > > > >> like > > > >> > > it > > > >> > > > > > would > > > >> > > > > > > >>>> have > > > >> > > > > > > >>>>>> rather been better when people explicitly turned > > off > > > >> CI in > > > >> > > > that > > > >> > > > > > > case. > > > >> > > > > > > >>>>>> What's the argument against an opt-out instead of > > an > > > >> > opt-in? > > > >> > > > > > > >>>>>> > > > >> > > > > > > >>>>>> My intention is that I consider it quite > cumbersome > > > to > > > >> > make > > > >> > > > it a > > > >> > > > > > > >>>>> *required* > > > >> > > > > > > >>>>>> step to always trigger CI manually, even if just > > > >> > submitting > > > >> > > a > > > >> > > > > > small > > > >> > > > > > > >>> PR. > > > >> > > > > > > >>>>> I'd > > > >> > > > > > > >>>>>> rather see people explicitly turning off CI if > they > > > >> > wouldn't > > > >> > > > > like > > > >> > > > > > to > > > >> > > > > > > >>>> use > > > >> > > > > > > >>>>> it > > > >> > > > > > > >>>>>> - and there's also the "draft" stage for a PR > which > > > >> some > > > >> > > > > > > contributors > > > >> > > > > > > >>>> are > > > >> > > > > > > >>>>>> using. > > > >> > > > > > > >>>>>> > > > >> > > > > > > >>>>>> With regards to WIP and do not review: I think > > these > > > >> are > > > >> > use > > > >> > > > > cases > > > >> > > > > > > >>>> where > > > >> > > > > > > >>>>>> you want CI feedback, as otherwise you wouldn't > > have > > > >> > opened > > > >> > > > the > > > >> > > > > > PR. > > > >> > > > > > > >>> If > > > >> > > > > > > >>>>> you > > > >> > > > > > > >>>>>> don't want human feedback and neither machine > > > feedback, > > > >> > why > > > >> > > > open > > > >> > > > > > the > > > >> > > > > > > >>> PR > > > >> > > > > > > >>>>> at > > > >> > > > > > > >>>>>> all? > > > >> > > > > > > >>>>>> > > > >> > > > > > > >>>>>> -Marco > > > >> > > > > > > >>>>>> > > > >> > > > > > > >>>>>> > > > >> > > > > > > >>>>>> sandeep krishnamurthy < > sandeep.krishn...@gmail.com > > > > > > >> > schrieb > > > >> > > > am > > > >> > > > > > Fr., > > > >> > > > > > > >>>> 13. > > > >> > > > > > > >>>>>> März 2020, 05:24: > > > >> > > > > > > >>>>>> > > > >> > > > > > > >>>>>>> I tried to gather some data for us to discuss > this > > > >> topic > > > >> > in > > > >> > > > > this > > > >> > > > > > > >>>>> thread. I > > > >> > > > > > > >>>>>>> tried to count number of un-necessary builds by > > > >> looking > > > >> > at > > > >> > > > most > > > >> > > > > > > >>>> recent > > > >> > > > > > > >>>>> (as > > > >> > > > > > > >>>>>>> of 12, March 9 PM PST) 50 PRs merged to master > and > > > 50 > > > >> > PRs. > > > >> > > > > > > >>>>>>> Identifying un-necessary builds is bit > > subjective. I > > > >> > tried > > > >> > > to > > > >> > > > > be > > > >> > > > > > > >>> more > > > >> > > > > > > >>>>>>> conservative where I didn't count a build as > > > >> un-necessary > > > >> > > if > > > >> > > > I > > > >> > > > > > was > > > >> > > > > > > >>> in > > > >> > > > > > > >>>>>>> doubt. Hence, I was not able to automate, but I > > made > > > >> an > > > >> > > > effort > > > >> > > > > to > > > >> > > > > > > >>> go > > > >> > > > > > > >>>>>>> through PRs manually and use below criteria to > > > >> identify > > > >> > > > > > > >>> un-necessary > > > >> > > > > > > >>>>>>> commits triggering the builds. > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>>>> 1. Explicitly marked as WIP / do not review > PR > > > >> > > > > > > >>>>>>> 2. Incremental WIP commit and finally > > commenting a > > > >> > commit > > > >> > > > > > > >>> “trigger > > > >> > > > > > > >>>>> CI” > > > >> > > > > > > >>>>>>> 3. Multiple commits to address all comments > from > > > >> single > > > >> > > > > review. > > > >> > > > > > > >>>>> This is > > > >> > > > > > > >>>>>>> assuming we see a comment, address them, > commit, > > > >> next > > > >> > the > > > >> > > > > > > >>>> following > > > >> > > > > > > >>>>>>> comment > > > >> > > > > > > >>>>>>> 4. Sequence of documentation only changes > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>>>> I found there were around 42 avoidable builds > from > > > >> most > > > >> > > > recent > > > >> > > > > 50 > > > >> > > > > > > >>>>> merged > > > >> > > > > > > >>>>>>> PRs and around 86 builds from recent 50 open > PRs. > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>>>> I synced up with other contributors (Joe Evans, > > > Chai) > > > >> > from > > > >> > > > > Amazon > > > >> > > > > > > >>> who > > > >> > > > > > > >>>>> is > > > >> > > > > > > >>>>>>> contributing to MXNet CI system. I was told that > > on > > > an > > > >> > > > average > > > >> > > > > it > > > >> > > > > > > >>>> costs > > > >> > > > > > > >>>>>>> around $84 per build and on an average 6 commits > > per > > > >> > merged > > > >> > > > PR > > > >> > > > > > (for > > > >> > > > > > > >>>>> year > > > >> > > > > > > >>>>>>> 2019). Going by that, it is approximately 1/6 > > builds > > > >> are > > > >> > > > > > avoidable. > > > >> > > > > > > >>>>> [100 / > > > >> > > > > > > >>>>>>> 300 + 300 ] > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>>>> Usability should be top most priority. But, > since > > > >> either > > > >> > a > > > >> > > > > > reviewer > > > >> > > > > > > >>>> or > > > >> > > > > > > >>>>> pr > > > >> > > > > > > >>>>>>> author can trigger the bot, is it really a > hurdle > > > for > > > >> pr > > > >> > > > author > > > >> > > > > > or > > > >> > > > > > > >>>>> reviewer > > > >> > > > > > > >>>>>>> to call a bot to trigger CI? Given that PR > author > > > and > > > >> > > > reviewer > > > >> > > > > is > > > >> > > > > > > >>>>> already > > > >> > > > > > > >>>>>>> actively commenting various details such as - PR > > > >> > > description, > > > >> > > > > > > >>> review > > > >> > > > > > > >>>>>>> comments and responses, adding labels etc. > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>>>> Me too curious to know the behavior for Tao's > > above > > > >> use > > > >> > > case. > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>>>> Best, > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>>>> Sandeep > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>>>> On Thu, Mar 12, 2020 at 7:18 PM Tao Lv < > > > >> > mutou...@gmail.com > > > >> > > > > > > >> > > > > > wrote: > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>>>>> Is it possible for re-triggering a single job > to > > be > > > >> > > abused? > > > >> > > > > For > > > >> > > > > > > >>>>> example, > > > >> > > > > > > >>>>>>>> the author spends two days re-triggering a > flaky > > > job > > > >> to > > > >> > > make > > > >> > > > > it > > > >> > > > > > > >>>>> pass. But > > > >> > > > > > > >>>>>>>> other jobs which have passed the validation may > > be > > > >> > broken > > > >> > > by > > > >> > > > > > > >>> other > > > >> > > > > > > >>>>>>> commits > > > >> > > > > > > >>>>>>>> during the two day without being noticed. And > > > finally > > > >> > the > > > >> > > PR > > > >> > > > > is > > > >> > > > > > > >>>>> merged > > > >> > > > > > > >>>>>>> with > > > >> > > > > > > >>>>>>>> underlying problems. > > > >> > > > > > > >>>>>>>> > > > >> > > > > > > >>>>>>>> On Fri, Mar 13, 2020 at 6:19 AM Marco de Abreu > < > > > >> > > > > > > >>>>> marco.g.ab...@gmail.com> > > > >> > > > > > > >>>>>>>> wrote: > > > >> > > > > > > >>>>>>>> > > > >> > > > > > > >>>>>>>>> In the end it only comes down to money, > > > considering > > > >> > that > > > >> > > > the > > > >> > > > > > > >>>>> system is > > > >> > > > > > > >>>>>>>> auto > > > >> > > > > > > >>>>>>>>> scaling, making the execution time constant. > > > >> > > > > > > >>>>>>>>> > > > >> > > > > > > >>>>>>>>> If we're trading money for usability, I > > certainly > > > >> would > > > >> > > > > prefer > > > >> > > > > > > >>>>>>> usability. > > > >> > > > > > > >>>>>>>>> I'd rather recommend to spend time on > > > parallelizing > > > >> > test > > > >> > > > > > > >>>> execution > > > >> > > > > > > >>>>> or > > > >> > > > > > > >>>>>>>>> getting rid of integration tests in the PR > stage > > > >> > instead > > > >> > > > > > > >>> reducing > > > >> > > > > > > >>>>> the > > > >> > > > > > > >>>>>>>> costs > > > >> > > > > > > >>>>>>>>> by making people not use it. But taking a step > > > back > > > >> to > > > >> > > > > > > >>> requiring > > > >> > > > > > > >>>>> people > > > >> > > > > > > >>>>>>>> to > > > >> > > > > > > >>>>>>>>> manually trigger CI again doesn't feel right. > > > >> > > > > > > >>>>>>>>> > > > >> > > > > > > >>>>>>>>> I'm happy to see that bot deployed, but I do > not > > > >> agree > > > >> > > with > > > >> > > > > > > >>>>> removing > > > >> > > > > > > >>>>>>> the > > > >> > > > > > > >>>>>>>>> auto trigger functionality for new commits. > > > >> > > > > > > >>>>>>>>> > > > >> > > > > > > >>>>>>>>> -Marco > > > >> > > > > > > >>>>>>>>> > > > >> > > > > > > >>>>>>>>> Chaitanya Bapat <chai.ba...@gmail.com> > schrieb > > am > > > >> Do., > > > >> > > 12. > > > >> > > > > > > >>> März > > > >> > > > > > > >>>>> 2020, > > > >> > > > > > > >>>>>>>>> 22:47: > > > >> > > > > > > >>>>>>>>> > > > >> > > > > > > >>>>>>>>>> @Marco Thanks for pointing that out. > > > >> > > > > > > >>>>>>>>>> Tomorrow i.e. Friday, March 13, 2020 at 3:00 > > PM - > > > >> 3:30 > > > >> > > PM > > > >> > > > in > > > >> > > > > > > >>>>>>>> (UTC-08:00) > > > >> > > > > > > >>>>>>>>>> Pacific Time (US & Canada). > > > >> > > > > > > >>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>> When do we expect this bot to be deployed? > > > >> > > > > > > >>>>>>>>>> @Lin If all goes well in the next week I can > > > >> deploy it > > > >> > > to > > > >> > > > > > > >>>> public > > > >> > > > > > > >>>>>>> Apache > > > >> > > > > > > >>>>>>>>>> (provided I get permissions from Apache > Infra) > > > >> > > > > > > >>>>>>>>>> > > > >> > > > > > > >>>>>>>>>> @Marco Thanks for your feedback. > > > >> > > > > > > >>>>>>>>>>> CI system has to support the community > without > > > >> > > requiring > > > >> > > > > > > >>>>> people to > > > >> > > > > > > >>>>>>>>>> constantly shepherd every single run > > > >> > > > > > > >>>>>>>>>> We have data for the number of times CI was > > > >> triggered > > > >> > > > > > > >>>>> unnecessarily > > > >> > > > > > > >>>>>>>> which > > > >> > > > > > > >>>>>>>>>> includes > > > >> > > > > > > >>>>>>>>>> - Entire build triggered instead of specific > > > build > > > >> > > > > > > >>>>>>>>>> - CI triggered when PR is still work in > > progress > > > or > > > >> > not > > > >> > > > yet > > > >> > > > > > > >>>> ready > > > >> > > > > > > >>>>>>> (say > > > >> > > > > > > >>>>>>>> - > > > >> > > > > > > >>>>>>>>>> intermediate commits) > > > >> > > > > > > >>>>>>>>>> At the end its a trade-off > > > >> > > > > > > >>>>>>>>>> Money, Resources, Time to build for each and > > > every > > > >> > > commit > > > >> > > > vs > > > >> > > > > > > >>>>> Pain of > > > >> > > > > > > >>>>>>>>>> triggering builds > > > >> > > > > > > >>>>>>>>>> > > > >> > > > > > > >>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>> Scan trigger plugin would poll SCM. Can we > use > > > >> plugin > > > >> > > at > > > >> > > > > > > >>>>> scale? > > > >> > > > > > > >>>>>>>>>> > > > >> > > > > > > >>>>>>>>>> 1. I haven't tested it on scale. But I think > > with > > > >> the > > > >> > > > > current > > > >> > > > > > > >>>>> scale > > > >> > > > > > > >>>>>>> of > > > >> > > > > > > >>>>>>>>>> MXNet repo (191 open PRs i.e. checking for > > > changes > > > >> to > > > >> > > 191 > > > >> > > > > > > >>>>> branches - > > > >> > > > > > > >>>>>>> It > > > >> > > > > > > >>>>>>>>>> should be manageable) > > > >> > > > > > > >>>>>>>>>> 2. What's the purpose of the plugin? tldr; > > Branch > > > >> > > > discovery > > > >> > > > > > > >>> or > > > >> > > > > > > >>>>> branch > > > >> > > > > > > >>>>>>>>>> indexing. > > > >> > > > > > > >>>>>>>>>> Scan trigger plugin comes into the picture > only > > > >> once > > > >> > per > > > >> > > > PR > > > >> > > > > > > >>> per > > > >> > > > > > > >>>>> job > > > >> > > > > > > >>>>>>>>> (i.e. 8 > > > >> > > > > > > >>>>>>>>>> times per PR for 8 jobs). It is basically > done > > > >> when a > > > >> > > new > > > >> > > > PR > > > >> > > > > > > >>> is > > > >> > > > > > > >>>>> made > > > >> > > > > > > >>>>>>>> and > > > >> > > > > > > >>>>>>>>>> the job (say unix-cpu hasn't discovered the > new > > > PR > > > >> > > branch > > > >> > > > > > > >>> yet). > > > >> > > > > > > >>>>>>> That's > > > >> > > > > > > >>>>>>>>> it. > > > >> > > > > > > >>>>>>>>>> So it shouldn't be a problem for public MXNet > > > repo. > > > >> > > > > > > >>>>>>>>>> > > > >> > > > > > > >>>>>>>>>> Thanks, > > > >> > > > > > > >>>>>>>>>> Chai > > > >> > > > > > > >>>>>>>>>> > > > >> > > > > > > >>>>>>>>>> > > > >> > > > > > > >>>>>>>>>> On Thu, 12 Mar 2020 at 14:22, Marco de Abreu > < > > > >> > > > > > > >>>>>>> marco.g.ab...@gmail.com> > > > >> > > > > > > >>>>>>>>>> wrote: > > > >> > > > > > > >>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>> Btw you forgot to set a date and time for > the > > > >> metting > > > >> > > > > > > >>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>> On Thu, Mar 12, 2020 at 10:18 PM Marco de > > Abreu > > > < > > > >> > > > > > > >>>>>>>>> marco.g.ab...@gmail.com > > > >> > > > > > > >>>>>>>>>>> wrote: > > > >> > > > > > > >>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>> Thanks Chai, I generally like the idea of > the > > > >> bot. > > > >> > But > > > >> > > > > > > >>> I'm > > > >> > > > > > > >>>>> not a > > > >> > > > > > > >>>>>>>>>>> supporter > > > >> > > > > > > >>>>>>>>>>>> of the idea to disable any automatic > > triggering > > > >> > > > > > > >>> (disabling > > > >> > > > > > > >>>>> the > > > >> > > > > > > >>>>>>>>> webhook > > > >> > > > > > > >>>>>>>>>> is > > > >> > > > > > > >>>>>>>>>>>> also not an option, considering that this > > will > > > >> > disable > > > >> > > > > > > >>>> master > > > >> > > > > > > >>>>>>>>>> triggers). > > > >> > > > > > > >>>>>>>>>>>> The CI system has to support the community > > > >> without > > > >> > > > > > > >>>> requiring > > > >> > > > > > > >>>>>>> people > > > >> > > > > > > >>>>>>>>> to > > > >> > > > > > > >>>>>>>>>>>> constantly shepherd every single run. > > Disabling > > > >> > > > automatic > > > >> > > > > > > >>>>>>>> triggering > > > >> > > > > > > >>>>>>>>>>> seems > > > >> > > > > > > >>>>>>>>>>>> like a step back to me. > > > >> > > > > > > >>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>> Instead, I'd recommend that CI gets > triggered > > > >> upon > > > >> > > every > > > >> > > > > > > >>>>> commit > > > >> > > > > > > >>>>>>> as > > > >> > > > > > > >>>>>>>>>> usual, > > > >> > > > > > > >>>>>>>>>>>> but people have the possibility to call a > > > >> "command" > > > >> > > > (i.e. > > > >> > > > > > > >>>>> make a > > > >> > > > > > > >>>>>>>>>> message > > > >> > > > > > > >>>>>>>>>>>> which results in the bot setting a label) > to > > > >> disable > > > >> > > CI > > > >> > > > > > > >>>> until > > > >> > > > > > > >>>>>>> they > > > >> > > > > > > >>>>>>>>>> revoke > > > >> > > > > > > >>>>>>>>>>>> it. But the standard should still be that a > > new > > > >> > commit > > > >> > > > > > > >>>>> triggers a > > > >> > > > > > > >>>>>>>> new > > > >> > > > > > > >>>>>>>>>> CI > > > >> > > > > > > >>>>>>>>>>>> run. > > > >> > > > > > > >>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>> > > > >> > > > > > > >>>> > > > >> > https://plugins.jenkins.io/multibranch-scan-webhook-trigger/ > > > >> > > > > > > >>>>>>>> seems > > > >> > > > > > > >>>>>>>>>> like > > > >> > > > > > > >>>>>>>>>>>> this would poll SCM. This will incur high > > quota > > > >> > > > > > > >>>>> restrictions. Are > > > >> > > > > > > >>>>>>>> you > > > >> > > > > > > >>>>>>>>>>> sure > > > >> > > > > > > >>>>>>>>>>>> that you can use that plugin at scale? > > > >> > > > > > > >>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>> -Marco > > > >> > > > > > > >>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>> On Thu, Mar 12, 2020 at 10:04 PM Lin Yuan < > > > >> > > > > > > >>>>> apefor...@gmail.com> > > > >> > > > > > > >>>>>>>>> wrote: > > > >> > > > > > > >>>>>>>>>>>>> Chai, > > > >> > > > > > > >>>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>>> Awesome work. When do we expect this bot > to > > be > > > >> > > > > > > >>> deployed? > > > >> > > > > > > >>>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>>> Best, > > > >> > > > > > > >>>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>>> Lin > > > >> > > > > > > >>>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>>> On Thu, Mar 12, 2020 at 2:00 PM Chaitanya > > > Bapat > > > >> < > > > >> > > > > > > >>>>>>>>> chai.ba...@gmail.com > > > >> > > > > > > >>>>>>>>>>>>> wrote: > > > >> > > > > > > >>>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>>>> Hello MXNet community, > > > >> > > > > > > >>>>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>>>> I have built an MXNet Bot < > > > >> > > > > > > >>>> https://github.com/mxnet-bot> > > > >> > > > > > > >>>>> that > > > >> > > > > > > >>>>>>>>>> allows > > > >> > > > > > > >>>>>>>>>>> PR > > > >> > > > > > > >>>>>>>>>>>>>> Authors, Committers and Jenkins Admins to > > > >> trigger > > > >> > CI > > > >> > > > > > > >>>>> manually. > > > >> > > > > > > >>>>>>>>>>>>>> It handles 2 problems > > > >> > > > > > > >>>>>>>>>>>>>> 1. Manual CI trigger instead of existing > > > >> automated > > > >> > > CI > > > >> > > > > > > >>>>> trigger > > > >> > > > > > > >>>>>>>>>>>>>> 2. Gives permissions to PR Authors (in > > > >> addition to > > > >> > > > > > > >>>> MXNet > > > >> > > > > > > >>>>>>>>> Committers > > > >> > > > > > > >>>>>>>>>>> and > > > >> > > > > > > >>>>>>>>>>>>>> Jenkins Admins) > > > >> > > > > > > >>>>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>>>> Design Doc : > > > >> > > > > > > >>>>>>>>>>>>>> > > > >> > > > > > > >>>>>>> > > > >> > > > > https://cwiki.apache.org/confluence/display/MXNET/MXNet+CI+Bot > > > >> > > > > > > >>>>>>>>>>>>>> I urge you all to attend the > demonstration > > > >> meeting > > > >> > > > > > > >>> and > > > >> > > > > > > >>>>> lend > > > >> > > > > > > >>>>>>> your > > > >> > > > > > > >>>>>>>>>> views > > > >> > > > > > > >>>>>>>>>>>>> on > > > >> > > > > > > >>>>>>>>>>>>>> the same. > > > >> > > > > > > >>>>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>>>> Thank you, > > > >> > > > > > > >>>>>>>>>>>>>> Chai > > > >> > > > > > > >>>>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>>>> *Meeting Details*: > > > >> > > > > > > >>>>>>>>>>>>>> ==============Conference Bridge > > > >> > > > > > > >>>> Information============== > > > >> > > > > > > >>>>>>>>>>>>>> You have been invited to an online > meeting, > > > >> > powered > > > >> > > > > > > >>> by > > > >> > > > > > > >>>>> Amazon > > > >> > > > > > > >>>>>>>>> Chime. > > > >> > > > > > > >>>>>>>>>>>>>> *Chime meeting ID*: *9272158344* > > > >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (manually): Select > > > >> > 'Meetings > > > >> > > > > > > >> > > > > > > >>>>> Join a > > > >> > > > > > > >>>>>>>>>> Meeting', > > > >> > > > > > > >>>>>>>>>>>>> and > > > >> > > > > > > >>>>>>>>>>>>>> enter 9272158344 > > > >> > > > > > > >>>>>>>>>>>>>> Join via Chime clients (auto-call): If > you > > > >> invite > > > >> > > > > > > >>>>> auto-call as > > > >> > > > > > > >>>>>>>>>>> attendee, > > > >> > > > > > > >>>>>>>>>>>>>> Chime will call you when the meeting > > starts, > > > >> > select > > > >> > > > > > > >>>>> 'Answer' > > > >> > > > > > > >>>>>>>>>>>>>> *Join via browser screen share*: > > > >> > > > > > > >>>>> https://chime.aws/9272158344 > > > >> > > > > > > >>>>>>>>>>>>>> *Join via phone* (US): > > > >> > +1-929-432-4463,,,9272158344# > > > >> > > > > > > >>>>>>>>>>>>>> *Join via phone (US toll-free)*: > > > >> > > > > > > >>>>> +1-855-552-4463,,,9272158344# > > > >> > > > > > > >>>>>>>>>>>>>> International dial-in: > > > >> > > > > > > >>>> https://chime.aws/dialinnumbers/ > > > >> > > > > > > >>>>>>>>>>>>>> In-room video system: Ext: 62000, Meeting > > > PIN: > > > >> > > > > > > >>>>> 9272158344# > > > >> > > > > > > >>>>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>>>> -- > > > >> > > > > > > >>>>>>>>>>>>>> *Chaitanya Prakash Bapat* > > > >> > > > > > > >>>>>>>>>>>>>> *+1 (973) 953-6299* > > > >> > > > > > > >>>>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>>>>>> [image: > > > >> https://www.linkedin.com//in/chaibapat25] > > > >> > > > > > > >>>>>>>>>>>>>> <https://github.com/ChaiBapchya>[image: > > > >> > > > > > > >>>>>>>>>>>>> https://www.facebook.com/chaibapat > > > >> > > > > > > >>>>>>>>>>>>>> ] > > > >> > > > > > > >>>>>>>>>>>>>> <https://www.facebook.com/chaibapchya > > > >[image: > > > >> > > > > > > >>>>>>>>>>>>>> https://twitter.com/ChaiBapchya] < > > > >> > > > > > > >>>>>>>> https://twitter.com/ChaiBapchya > > > >> > > > > > > >>>>>>>>>>>>>> [image: > > > >> > > > > > > >>>>>>>>>>>>>> https://www.linkedin.com//in/chaibapat25 > ] > > > >> > > > > > > >>>>>>>>>>>>>> < > https://www.linkedin.com//in/chaibapchya/ > > > > > > >> > > > > > > >>>>>>>>>>>>>> > > > >> > > > > > > >>>>>>>>>> > > > >> > > > > > > >>>>>>>>>> -- > > > >> > > > > > > >>>>>>>>>> *Chaitanya Prakash Bapat* > > > >> > > > > > > >>>>>>>>>> *+1 (973) 953-6299* > > > >> > > > > > > >>>>>>>>>> > > > >> > > > > > > >>>>>>>>>> [image: > > https://www.linkedin.com//in/chaibapat25 > > > ] > > > >> > > > > > > >>>>>>>>>> <https://github.com/ChaiBapchya>[image: > > > >> > > > > > > >>>>>>>>> https://www.facebook.com/chaibapat > > > >> > > > > > > >>>>>>>>>> ] > > > >> > > > > > > >>>>>>>>>> <https://www.facebook.com/chaibapchya > >[image: > > > >> > > > > > > >>>>>>>>>> https://twitter.com/ChaiBapchya] < > > > >> > > > > > > >>>>> https://twitter.com/ChaiBapchya > > > >> > > > > > > >>>>>>>>>> [image: > > > >> > > > > > > >>>>>>>>>> https://www.linkedin.com//in/chaibapat25] > > > >> > > > > > > >>>>>>>>>> <https://www.linkedin.com//in/chaibapchya/> > > > >> > > > > > > >>>>>>>>>> > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>>>> -- > > > >> > > > > > > >>>>>>> Sandeep Krishnamurthy > > > >> > > > > > > >>>>>>> > > > >> > > > > > > >>>>> > > > >> > > > > > > >>>> > > > >> > > > > > > >>> > > > >> > > > > > > >>> > > > >> > > > > > > >>> -- > > > >> > > > > > > >>> *Chaitanya Prakash Bapat* > > > >> > > > > > > >>> *+1 (973) 953-6299* > > > >> > > > > > > >>> > > > >> > > > > > > >>> [image: https://www.linkedin.com//in/chaibapat25] > > > >> > > > > > > >>> <https://github.com/ChaiBapchya>[image: > > > >> > > > > > > https://www.facebook.com/chaibapat > > > >> > > > > > > >>> ] > > > >> > > > > > > >>> <https://www.facebook.com/chaibapchya>[image: > > > >> > > > > > > >>> https://twitter.com/ChaiBapchya] < > > > >> > > > https://twitter.com/ChaiBapchya > > > >> > > > > > > >[image: > > > >> > > > > > > >>> https://www.linkedin.com//in/chaibapat25] > > > >> > > > > > > >>> <https://www.linkedin.com//in/chaibapchya/> > > > >> > > > > > > >>> > > > >> > > > > > > >> > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > -- > > > >> > > > > *Chaitanya Prakash Bapat* > > > >> > > > > *+1 (973) 953-6299* > > > >> > > > > > > > >> > > > > [image: https://www.linkedin.com//in/chaibapat25] > > > >> > > > > <https://github.com/ChaiBapchya>[image: > > > >> > > > https://www.facebook.com/chaibapat > > > >> > > > > ] > > > >> > > > > <https://www.facebook.com/chaibapchya>[image: > > > >> > > > > https://twitter.com/ChaiBapchya] < > > > https://twitter.com/ChaiBapchya > > > >> > > > >[image: > > > >> > > > > https://www.linkedin.com//in/chaibapat25] > > > >> > > > > <https://www.linkedin.com//in/chaibapchya/> > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > -- > > > >> > > *Chaitanya Prakash Bapat* > > > >> > > *+1 (973) 953-6299* > > > >> > > > > > >> > > [image: https://www.linkedin.com//in/chaibapat25] > > > >> > > <https://github.com/ChaiBapchya>[image: > > > >> > https://www.facebook.com/chaibapat > > > >> > > ] > > > >> > > <https://www.facebook.com/chaibapchya>[image: > > > >> > > https://twitter.com/ChaiBapchya] < > https://twitter.com/ChaiBapchya > > > >> > >[image: > > > >> > > https://www.linkedin.com//in/chaibapat25] > > > >> > > <https://www.linkedin.com//in/chaibapchya/> > > > >> > > > > > >> > > > > >> > > > >> > > > >> -- > > > >> *Chaitanya Prakash Bapat* > > > >> *+1 (973) 953-6299* > > > >> > > > >> [image: https://www.linkedin.com//in/chaibapat25] > > > >> <https://github.com/ChaiBapchya>[image: > > > >> https://www.facebook.com/chaibapat] > > > >> <https://www.facebook.com/chaibapchya>[image: > > > >> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya > > > >[image: > > > >> https://www.linkedin.com//in/chaibapat25] > > > >> <https://www.linkedin.com//in/chaibapchya/> > > > >> > > > > > > > > > > > > > -- > > *Chaitanya Prakash Bapat* > > *+1 (973) 953-6299* > > > > [image: https://www.linkedin.com//in/chaibapat25] > > <https://github.com/ChaiBapchya>[image: > https://www.facebook.com/chaibapat > > ] > > <https://www.facebook.com/chaibapchya>[image: > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya > >[image: > > https://www.linkedin.com//in/chaibapat25] > > <https://www.linkedin.com//in/chaibapchya/> > > > -- Sandeep Krishnamurthy
