Re: Github integration for Hadoop

[Colin McCabe]

Thanks, Vinayakumar (and Daniel Gruno.)

C.

On Tue, Apr 5, 2016, at 08:48, Vinayakumar B wrote:
> INFRA-11597 has been fixed. Now all github mails are diverted to
> common-iss...@hadoop.apache.org
> 
> -Vinay
> 
> On Tue, Apr 5, 2016 at 1:05 PM, Vinayakumar B <vinayakumar...@huawei.com>
> wrote:
> 
> > https://issues.apache.org/jira/browse/INFRA-11597 has been filed for this.
> >
> > -Vinay
> >
> > -Original Message-
> > From: Colin McCabe [mailto:co...@cmccabe.xyz]
> > Sent: 05 April 2016 08:07
> > To: common-dev@hadoop.apache.org
> > Subject: Re: Github integration for Hadoop
> >
> > Yes, please.  Let's disable these mails.
> >
> > C.
> >
> > On Mon, Apr 4, 2016, at 06:21, Vinayakumar B wrote:
> > > bq. We don't spam common-dev about every time a new patch attachment
> > > gets posted to an existing JIRA.  We shouldn't do that for github
> > > either.
> > >
> > > Is there any update on this. ?
> > > Any INFRA ticket filed to disable these mails?
> > >
> > > Because, I am sure, people would have got frustrated by seeing mails
> > > generated by my recent PR submissions. And each time, when some
> > > comment gets added to PR.
> > >
> > > -Vinay
> > >
> > > On Tue, Nov 24, 2015 at 3:35 AM, Andrew Wang
> > > <andrew.w...@cloudera.com>
> > > wrote:
> > >
> > > > Hi Eric,
> > > >
> > > > My comment wrt GH permissions is available in context earlier in the
> > > > thread, pasting again here:
> > > >
> > > > ===
> > > >
> > > > Great, glad to hear it. That wasn't mentioned in the email thread or
> > > > on the INFRA ticket, and the INFRA ticket mentions these integrations:
> > > >
> > > > Commit Comment, Create, Issue Comment, Issues, Pull Request, Pull
> > > > Request Comment, and push, Push
> > > >
> > > > Are these the right set of permissions to disable integrating PRs?
> > > > Namely, the "push" permissions look unnecessary. We should also
> > > > disable GH issues since we don't want users filing issues there.
> > > >
> > > >
> > > > On Mon, Nov 23, 2015 at 1:59 PM, Colin P. McCabe
> > > > <cmcc...@apache.org>
> > > > wrote:
> > > >
> > > > > We don't spam common-dev about every time a new patch attachment
> > > > > gets posted to an existing JIRA.  We shouldn't do that for github
> > either.
> > > > >
> > > > > +1 to Andrew and Steve's suggestion of disabling these emails (or
> > > > > +at
> > > > > least sending them to a separate list that people can opt in to).
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Sat, Nov 21, 2015 at 9:19 AM, Eric Yang <eric...@gmail.com>
> > wrote:
> > > > > > Hi Andrew,
> > > > > >
> > > > > > Many Apache projects already have Github integration.  Chukwa
> > > > > > also has Github integration.  Pull request can be integrated into
> > JIRA, i.e.
> > > > > > https://issues.apache.org/jira/browse/CHUKWA-783
> > > > > >
> > > > > > This allows code review process to happen at per line level on
> > > > > > Github,
> > > > or
> > > > > > comment on JIRA for summary of the activities.  Micro comments
> > > > > > are
> > > > squash
> > > > > > away.  The final commit is always happening on Apache git rather
> > > > > > than Github.  Therefore, there is no disabling required for pull
> > request.
> > > > > > Primary activity is still on JIRA, and Github is only a
> > > > > > supplement to
> > > > > make
> > > > > > line by line code review easy.  Overall user experience is
> > > > > > better than
> > > > RB
> > > > > > or Gerrit in my opinion.
> > > > > >
> > > > > > regards,
> > > > > > Eric
> > > > > >
> > > > > > On Thu, Nov 19, 2015 at 11:28 AM, Andrew Wang <
> > > > andrew.w...@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Has there been any progress on loc

Re: Github integration for Hadoop

[Vinayakumar B]

INFRA-11597 has been fixed. Now all github mails are diverted to
common-iss...@hadoop.apache.org

-Vinay

On Tue, Apr 5, 2016 at 1:05 PM, Vinayakumar B <vinayakumar...@huawei.com>
wrote:

> https://issues.apache.org/jira/browse/INFRA-11597 has been filed for this.
>
> -Vinay
>
> -Original Message-
> From: Colin McCabe [mailto:co...@cmccabe.xyz]
> Sent: 05 April 2016 08:07
> To: common-dev@hadoop.apache.org
> Subject: Re: Github integration for Hadoop
>
> Yes, please.  Let's disable these mails.
>
> C.
>
> On Mon, Apr 4, 2016, at 06:21, Vinayakumar B wrote:
> > bq. We don't spam common-dev about every time a new patch attachment
> > gets posted to an existing JIRA.  We shouldn't do that for github
> > either.
> >
> > Is there any update on this. ?
> > Any INFRA ticket filed to disable these mails?
> >
> > Because, I am sure, people would have got frustrated by seeing mails
> > generated by my recent PR submissions. And each time, when some
> > comment gets added to PR.
> >
> > -Vinay
> >
> > On Tue, Nov 24, 2015 at 3:35 AM, Andrew Wang
> > <andrew.w...@cloudera.com>
> > wrote:
> >
> > > Hi Eric,
> > >
> > > My comment wrt GH permissions is available in context earlier in the
> > > thread, pasting again here:
> > >
> > > ===
> > >
> > > Great, glad to hear it. That wasn't mentioned in the email thread or
> > > on the INFRA ticket, and the INFRA ticket mentions these integrations:
> > >
> > > Commit Comment, Create, Issue Comment, Issues, Pull Request, Pull
> > > Request Comment, and push, Push
> > >
> > > Are these the right set of permissions to disable integrating PRs?
> > > Namely, the "push" permissions look unnecessary. We should also
> > > disable GH issues since we don't want users filing issues there.
> > >
> > >
> > > On Mon, Nov 23, 2015 at 1:59 PM, Colin P. McCabe
> > > <cmcc...@apache.org>
> > > wrote:
> > >
> > > > We don't spam common-dev about every time a new patch attachment
> > > > gets posted to an existing JIRA.  We shouldn't do that for github
> either.
> > > >
> > > > +1 to Andrew and Steve's suggestion of disabling these emails (or
> > > > +at
> > > > least sending them to a separate list that people can opt in to).
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Sat, Nov 21, 2015 at 9:19 AM, Eric Yang <eric...@gmail.com>
> wrote:
> > > > > Hi Andrew,
> > > > >
> > > > > Many Apache projects already have Github integration.  Chukwa
> > > > > also has Github integration.  Pull request can be integrated into
> JIRA, i.e.
> > > > > https://issues.apache.org/jira/browse/CHUKWA-783
> > > > >
> > > > > This allows code review process to happen at per line level on
> > > > > Github,
> > > or
> > > > > comment on JIRA for summary of the activities.  Micro comments
> > > > > are
> > > squash
> > > > > away.  The final commit is always happening on Apache git rather
> > > > > than Github.  Therefore, there is no disabling required for pull
> request.
> > > > > Primary activity is still on JIRA, and Github is only a
> > > > > supplement to
> > > > make
> > > > > line by line code review easy.  Overall user experience is
> > > > > better than
> > > RB
> > > > > or Gerrit in my opinion.
> > > > >
> > > > > regards,
> > > > > Eric
> > > > >
> > > > > On Thu, Nov 19, 2015 at 11:28 AM, Andrew Wang <
> > > andrew.w...@cloudera.com>
> > > > > wrote:
> > > > >
> > > > >> Has there been any progress on locking down the Github
> > > > >> permissions
> > > like
> > > > I
> > > > >> asked above? It's been about 3 weeks.
> > > > >>
> > > > >> Steve also just asked on another thread to turn off the emails
> > > > >> to common-dev. Since we're sticking with JIRA as the
> > > > >> source-of-truth, the common-dev emails aren't useful.
> > > > >>
> > > > >> On Fri, Nov 13, 2015 at 6:59 PM, Sean Busbey
> > > > >> <bus...@cloudera.com>
> > > >

RE: Github integration for Hadoop

[Vinayakumar B]

https://issues.apache.org/jira/browse/INFRA-11597 has been filed for this.

-Vinay

-Original Message-
From: Colin McCabe [mailto:co...@cmccabe.xyz] 
Sent: 05 April 2016 08:07
To: common-dev@hadoop.apache.org
Subject: Re: Github integration for Hadoop

Yes, please.  Let's disable these mails.

C.

On Mon, Apr 4, 2016, at 06:21, Vinayakumar B wrote:
> bq. We don't spam common-dev about every time a new patch attachment 
> gets posted to an existing JIRA.  We shouldn't do that for github 
> either.
> 
> Is there any update on this. ?
> Any INFRA ticket filed to disable these mails?
> 
> Because, I am sure, people would have got frustrated by seeing mails 
> generated by my recent PR submissions. And each time, when some 
> comment gets added to PR.
> 
> -Vinay
> 
> On Tue, Nov 24, 2015 at 3:35 AM, Andrew Wang 
> <andrew.w...@cloudera.com>
> wrote:
> 
> > Hi Eric,
> >
> > My comment wrt GH permissions is available in context earlier in the 
> > thread, pasting again here:
> >
> > ===
> >
> > Great, glad to hear it. That wasn't mentioned in the email thread or 
> > on the INFRA ticket, and the INFRA ticket mentions these integrations:
> >
> > Commit Comment, Create, Issue Comment, Issues, Pull Request, Pull 
> > Request Comment, and push, Push
> >
> > Are these the right set of permissions to disable integrating PRs? 
> > Namely, the "push" permissions look unnecessary. We should also 
> > disable GH issues since we don't want users filing issues there.
> >
> >
> > On Mon, Nov 23, 2015 at 1:59 PM, Colin P. McCabe 
> > <cmcc...@apache.org>
> > wrote:
> >
> > > We don't spam common-dev about every time a new patch attachment 
> > > gets posted to an existing JIRA.  We shouldn't do that for github either.
> > >
> > > +1 to Andrew and Steve's suggestion of disabling these emails (or 
> > > +at
> > > least sending them to a separate list that people can opt in to).
> > >
> > > best,
> > > Colin
> > >
> > > On Sat, Nov 21, 2015 at 9:19 AM, Eric Yang <eric...@gmail.com> wrote:
> > > > Hi Andrew,
> > > >
> > > > Many Apache projects already have Github integration.  Chukwa 
> > > > also has Github integration.  Pull request can be integrated into JIRA, 
> > > > i.e.
> > > > https://issues.apache.org/jira/browse/CHUKWA-783
> > > >
> > > > This allows code review process to happen at per line level on 
> > > > Github,
> > or
> > > > comment on JIRA for summary of the activities.  Micro comments 
> > > > are
> > squash
> > > > away.  The final commit is always happening on Apache git rather 
> > > > than Github.  Therefore, there is no disabling required for pull 
> > > > request.
> > > > Primary activity is still on JIRA, and Github is only a 
> > > > supplement to
> > > make
> > > > line by line code review easy.  Overall user experience is 
> > > > better than
> > RB
> > > > or Gerrit in my opinion.
> > > >
> > > > regards,
> > > > Eric
> > > >
> > > > On Thu, Nov 19, 2015 at 11:28 AM, Andrew Wang <
> > andrew.w...@cloudera.com>
> > > > wrote:
> > > >
> > > >> Has there been any progress on locking down the Github 
> > > >> permissions
> > like
> > > I
> > > >> asked above? It's been about 3 weeks.
> > > >>
> > > >> Steve also just asked on another thread to turn off the emails 
> > > >> to common-dev. Since we're sticking with JIRA as the 
> > > >> source-of-truth, the common-dev emails aren't useful.
> > > >>
> > > >> On Fri, Nov 13, 2015 at 6:59 PM, Sean Busbey 
> > > >> <bus...@cloudera.com>
> > > wrote:
> > > >>
> > > >> > Hi Colin!
> > > >> >
> > > >> > If Yetus is working on an issue and can't tell what the 
> > > >> > intended
> > > branch
> > > >> is
> > > >> > it points folks to project specific contribution guides.
> > > >> >
> > > >> > For Hadoop, the patch naming for specific branches should be 
> > > >> > covered
> > > in
> > > >> > this section of Hadoop's contribution guide:
> > > >> >
> > > >>

Re: Github integration for Hadoop

[Colin McCabe]

Yes, please.  Let's disable these mails.

C.

On Mon, Apr 4, 2016, at 06:21, Vinayakumar B wrote:
> bq. We don't spam common-dev about every time a new patch attachment
> gets posted
> to an existing JIRA.  We shouldn't do that for github either.
> 
> Is there any update on this. ?
> Any INFRA ticket filed to disable these mails?
> 
> Because, I am sure, people would have got frustrated by seeing mails
> generated by my recent PR submissions. And each time, when some comment
> gets added to PR.
> 
> -Vinay
> 
> On Tue, Nov 24, 2015 at 3:35 AM, Andrew Wang 
> wrote:
> 
> > Hi Eric,
> >
> > My comment wrt GH permissions is available in context earlier in the
> > thread, pasting again here:
> >
> > ===
> >
> > Great, glad to hear it. That wasn't mentioned in the email thread or on the
> > INFRA ticket, and the INFRA ticket mentions these integrations:
> >
> > Commit Comment, Create, Issue Comment, Issues, Pull Request, Pull Request
> > Comment, and push, Push
> >
> > Are these the right set of permissions to disable integrating PRs? Namely,
> > the "push" permissions look unnecessary. We should also disable GH issues
> > since we don't want users filing issues there.
> >
> >
> > On Mon, Nov 23, 2015 at 1:59 PM, Colin P. McCabe 
> > wrote:
> >
> > > We don't spam common-dev about every time a new patch attachment gets
> > > posted to an existing JIRA.  We shouldn't do that for github either.
> > >
> > > +1 to Andrew and Steve's suggestion of disabling these emails (or at
> > > least sending them to a separate list that people can opt in to).
> > >
> > > best,
> > > Colin
> > >
> > > On Sat, Nov 21, 2015 at 9:19 AM, Eric Yang  wrote:
> > > > Hi Andrew,
> > > >
> > > > Many Apache projects already have Github integration.  Chukwa also has
> > > > Github integration.  Pull request can be integrated into JIRA, i.e.
> > > > https://issues.apache.org/jira/browse/CHUKWA-783
> > > >
> > > > This allows code review process to happen at per line level on Github,
> > or
> > > > comment on JIRA for summary of the activities.  Micro comments are
> > squash
> > > > away.  The final commit is always happening on Apache git rather than
> > > > Github.  Therefore, there is no disabling required for pull request.
> > > > Primary activity is still on JIRA, and Github is only a supplement to
> > > make
> > > > line by line code review easy.  Overall user experience is better than
> > RB
> > > > or Gerrit in my opinion.
> > > >
> > > > regards,
> > > > Eric
> > > >
> > > > On Thu, Nov 19, 2015 at 11:28 AM, Andrew Wang <
> > andrew.w...@cloudera.com>
> > > > wrote:
> > > >
> > > >> Has there been any progress on locking down the Github permissions
> > like
> > > I
> > > >> asked above? It's been about 3 weeks.
> > > >>
> > > >> Steve also just asked on another thread to turn off the emails to
> > > >> common-dev. Since we're sticking with JIRA as the source-of-truth, the
> > > >> common-dev emails aren't useful.
> > > >>
> > > >> On Fri, Nov 13, 2015 at 6:59 PM, Sean Busbey 
> > > wrote:
> > > >>
> > > >> > Hi Colin!
> > > >> >
> > > >> > If Yetus is working on an issue and can't tell what the intended
> > > branch
> > > >> is
> > > >> > it points folks to project specific contribution guides.
> > > >> >
> > > >> > For Hadoop, the patch naming for specific branches should be covered
> > > in
> > > >> > this section of Hadoop's contribution guide:
> > > >> >
> > > >> > http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
> > > >> >
> > > >> > Yetus will actually support a little bit more than that guide
> > > suggests.
> > > >> If
> > > >> > a project doesn't define a URL to point people at for help in naming
> > > >> > patches we default to this guide:
> > > >> >
> > > >> > https://yetus.apache.org/documentation/latest/precommit-patchnames/
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Fri, Nov 13, 2015 at 8:05 PM, Colin P. McCabe <
> > cmcc...@apache.org>
> > > >> > wrote:
> > > >> >
> > > >> > > Thanks, Allen, I wasn't aware that Yetus now supported testing for
> > > >> > > other branches.  Is there documentation about how to name the
> > branch
> > > >> > > so it gets tested?
> > > >> > >
> > > >> > > best,
> > > >> > > Colin
> > > >> > >
> > > >> > > On Fri, Nov 13, 2015 at 7:52 AM, Allen Wittenauer <
> > a...@altiscale.com
> > > >
> > > >> > > wrote:
> > > >> > > >
> > > >> > > >> On Nov 12, 2015, at 10:55 AM, Colin P. McCabe <
> > > cmcc...@apache.org>
> > > >> > > wrote:
> > > >> > > >>
> > > >> > > >> gerrit has a button on the UI to cherry-pick to different
> > > branches.
> > > >> > > >> The button creates separate "gerrit changes" which you can then
> > > >> > > >> commit.  Eventually we could hook those up to Jenkins--
> > something
> > > >> > > >> which we've never been able to do for different branches with
> > the
> > > >> > > >> patch-file-based workflow.
> > > >> > > >
> > > >> > > >
> > > >> > > > If 

Re: Github integration for Hadoop

[Vinayakumar B]

bq. We don't spam common-dev about every time a new patch attachment
gets posted
to an existing JIRA.  We shouldn't do that for github either.

Is there any update on this. ?
Any INFRA ticket filed to disable these mails?

Because, I am sure, people would have got frustrated by seeing mails
generated by my recent PR submissions. And each time, when some comment
gets added to PR.

-Vinay

On Tue, Nov 24, 2015 at 3:35 AM, Andrew Wang 
wrote:

> Hi Eric,
>
> My comment wrt GH permissions is available in context earlier in the
> thread, pasting again here:
>
> ===
>
> Great, glad to hear it. That wasn't mentioned in the email thread or on the
> INFRA ticket, and the INFRA ticket mentions these integrations:
>
> Commit Comment, Create, Issue Comment, Issues, Pull Request, Pull Request
> Comment, and push, Push
>
> Are these the right set of permissions to disable integrating PRs? Namely,
> the "push" permissions look unnecessary. We should also disable GH issues
> since we don't want users filing issues there.
>
>
> On Mon, Nov 23, 2015 at 1:59 PM, Colin P. McCabe 
> wrote:
>
> > We don't spam common-dev about every time a new patch attachment gets
> > posted to an existing JIRA.  We shouldn't do that for github either.
> >
> > +1 to Andrew and Steve's suggestion of disabling these emails (or at
> > least sending them to a separate list that people can opt in to).
> >
> > best,
> > Colin
> >
> > On Sat, Nov 21, 2015 at 9:19 AM, Eric Yang  wrote:
> > > Hi Andrew,
> > >
> > > Many Apache projects already have Github integration.  Chukwa also has
> > > Github integration.  Pull request can be integrated into JIRA, i.e.
> > > https://issues.apache.org/jira/browse/CHUKWA-783
> > >
> > > This allows code review process to happen at per line level on Github,
> or
> > > comment on JIRA for summary of the activities.  Micro comments are
> squash
> > > away.  The final commit is always happening on Apache git rather than
> > > Github.  Therefore, there is no disabling required for pull request.
> > > Primary activity is still on JIRA, and Github is only a supplement to
> > make
> > > line by line code review easy.  Overall user experience is better than
> RB
> > > or Gerrit in my opinion.
> > >
> > > regards,
> > > Eric
> > >
> > > On Thu, Nov 19, 2015 at 11:28 AM, Andrew Wang <
> andrew.w...@cloudera.com>
> > > wrote:
> > >
> > >> Has there been any progress on locking down the Github permissions
> like
> > I
> > >> asked above? It's been about 3 weeks.
> > >>
> > >> Steve also just asked on another thread to turn off the emails to
> > >> common-dev. Since we're sticking with JIRA as the source-of-truth, the
> > >> common-dev emails aren't useful.
> > >>
> > >> On Fri, Nov 13, 2015 at 6:59 PM, Sean Busbey 
> > wrote:
> > >>
> > >> > Hi Colin!
> > >> >
> > >> > If Yetus is working on an issue and can't tell what the intended
> > branch
> > >> is
> > >> > it points folks to project specific contribution guides.
> > >> >
> > >> > For Hadoop, the patch naming for specific branches should be covered
> > in
> > >> > this section of Hadoop's contribution guide:
> > >> >
> > >> > http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
> > >> >
> > >> > Yetus will actually support a little bit more than that guide
> > suggests.
> > >> If
> > >> > a project doesn't define a URL to point people at for help in naming
> > >> > patches we default to this guide:
> > >> >
> > >> > https://yetus.apache.org/documentation/latest/precommit-patchnames/
> > >> >
> > >> >
> > >> >
> > >> > On Fri, Nov 13, 2015 at 8:05 PM, Colin P. McCabe <
> cmcc...@apache.org>
> > >> > wrote:
> > >> >
> > >> > > Thanks, Allen, I wasn't aware that Yetus now supported testing for
> > >> > > other branches.  Is there documentation about how to name the
> branch
> > >> > > so it gets tested?
> > >> > >
> > >> > > best,
> > >> > > Colin
> > >> > >
> > >> > > On Fri, Nov 13, 2015 at 7:52 AM, Allen Wittenauer <
> a...@altiscale.com
> > >
> > >> > > wrote:
> > >> > > >
> > >> > > >> On Nov 12, 2015, at 10:55 AM, Colin P. McCabe <
> > cmcc...@apache.org>
> > >> > > wrote:
> > >> > > >>
> > >> > > >> gerrit has a button on the UI to cherry-pick to different
> > branches.
> > >> > > >> The button creates separate "gerrit changes" which you can then
> > >> > > >> commit.  Eventually we could hook those up to Jenkins--
> something
> > >> > > >> which we've never been able to do for different branches with
> the
> > >> > > >> patch-file-based workflow.
> > >> > > >
> > >> > > >
> > >> > > > If you’re saying what I think you’re saying, people have
> > been
> > >> > > able to submit patches via JIRA patch file attachment to major
> > branches
> > >> > for
> > >> > > a few months now. Yetus closes the loop and supports pretty much
> any
> > >> > branch
> > >> > > or git hash.  (Github PRs also go to their respective branch or
> git
> > >> hash
> > >> > as
> > >> > > 

Re: Github integration for Hadoop

[Andrew Wang]

Hi Eric,

My comment wrt GH permissions is available in context earlier in the
thread, pasting again here:

===

Great, glad to hear it. That wasn't mentioned in the email thread or on the
INFRA ticket, and the INFRA ticket mentions these integrations:

Commit Comment, Create, Issue Comment, Issues, Pull Request, Pull Request
Comment, and push, Push

Are these the right set of permissions to disable integrating PRs? Namely,
the "push" permissions look unnecessary. We should also disable GH issues
since we don't want users filing issues there.


On Mon, Nov 23, 2015 at 1:59 PM, Colin P. McCabe  wrote:

> We don't spam common-dev about every time a new patch attachment gets
> posted to an existing JIRA.  We shouldn't do that for github either.
>
> +1 to Andrew and Steve's suggestion of disabling these emails (or at
> least sending them to a separate list that people can opt in to).
>
> best,
> Colin
>
> On Sat, Nov 21, 2015 at 9:19 AM, Eric Yang  wrote:
> > Hi Andrew,
> >
> > Many Apache projects already have Github integration.  Chukwa also has
> > Github integration.  Pull request can be integrated into JIRA, i.e.
> > https://issues.apache.org/jira/browse/CHUKWA-783
> >
> > This allows code review process to happen at per line level on Github, or
> > comment on JIRA for summary of the activities.  Micro comments are squash
> > away.  The final commit is always happening on Apache git rather than
> > Github.  Therefore, there is no disabling required for pull request.
> > Primary activity is still on JIRA, and Github is only a supplement to
> make
> > line by line code review easy.  Overall user experience is better than RB
> > or Gerrit in my opinion.
> >
> > regards,
> > Eric
> >
> > On Thu, Nov 19, 2015 at 11:28 AM, Andrew Wang 
> > wrote:
> >
> >> Has there been any progress on locking down the Github permissions like
> I
> >> asked above? It's been about 3 weeks.
> >>
> >> Steve also just asked on another thread to turn off the emails to
> >> common-dev. Since we're sticking with JIRA as the source-of-truth, the
> >> common-dev emails aren't useful.
> >>
> >> On Fri, Nov 13, 2015 at 6:59 PM, Sean Busbey 
> wrote:
> >>
> >> > Hi Colin!
> >> >
> >> > If Yetus is working on an issue and can't tell what the intended
> branch
> >> is
> >> > it points folks to project specific contribution guides.
> >> >
> >> > For Hadoop, the patch naming for specific branches should be covered
> in
> >> > this section of Hadoop's contribution guide:
> >> >
> >> > http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
> >> >
> >> > Yetus will actually support a little bit more than that guide
> suggests.
> >> If
> >> > a project doesn't define a URL to point people at for help in naming
> >> > patches we default to this guide:
> >> >
> >> > https://yetus.apache.org/documentation/latest/precommit-patchnames/
> >> >
> >> >
> >> >
> >> > On Fri, Nov 13, 2015 at 8:05 PM, Colin P. McCabe 
> >> > wrote:
> >> >
> >> > > Thanks, Allen, I wasn't aware that Yetus now supported testing for
> >> > > other branches.  Is there documentation about how to name the branch
> >> > > so it gets tested?
> >> > >
> >> > > best,
> >> > > Colin
> >> > >
> >> > > On Fri, Nov 13, 2015 at 7:52 AM, Allen Wittenauer  >
> >> > > wrote:
> >> > > >
> >> > > >> On Nov 12, 2015, at 10:55 AM, Colin P. McCabe <
> cmcc...@apache.org>
> >> > > wrote:
> >> > > >>
> >> > > >> gerrit has a button on the UI to cherry-pick to different
> branches.
> >> > > >> The button creates separate "gerrit changes" which you can then
> >> > > >> commit.  Eventually we could hook those up to Jenkins-- something
> >> > > >> which we've never been able to do for different branches with the
> >> > > >> patch-file-based workflow.
> >> > > >
> >> > > >
> >> > > > If you’re saying what I think you’re saying, people have
> been
> >> > > able to submit patches via JIRA patch file attachment to major
> branches
> >> > for
> >> > > a few months now. Yetus closes the loop and supports pretty much any
> >> > branch
> >> > > or git hash.  (Github PRs also go to their respective branch or git
> >> hash
> >> > as
> >> > > well.)
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Sean
> >> >
> >>
>

Re: Github integration for Hadoop

[Colin P. McCabe]

We don't spam common-dev about every time a new patch attachment gets
posted to an existing JIRA.  We shouldn't do that for github either.

+1 to Andrew and Steve's suggestion of disabling these emails (or at
least sending them to a separate list that people can opt in to).

best,
Colin

On Sat, Nov 21, 2015 at 9:19 AM, Eric Yang  wrote:
> Hi Andrew,
>
> Many Apache projects already have Github integration.  Chukwa also has
> Github integration.  Pull request can be integrated into JIRA, i.e.
> https://issues.apache.org/jira/browse/CHUKWA-783
>
> This allows code review process to happen at per line level on Github, or
> comment on JIRA for summary of the activities.  Micro comments are squash
> away.  The final commit is always happening on Apache git rather than
> Github.  Therefore, there is no disabling required for pull request.
> Primary activity is still on JIRA, and Github is only a supplement to make
> line by line code review easy.  Overall user experience is better than RB
> or Gerrit in my opinion.
>
> regards,
> Eric
>
> On Thu, Nov 19, 2015 at 11:28 AM, Andrew Wang 
> wrote:
>
>> Has there been any progress on locking down the Github permissions like I
>> asked above? It's been about 3 weeks.
>>
>> Steve also just asked on another thread to turn off the emails to
>> common-dev. Since we're sticking with JIRA as the source-of-truth, the
>> common-dev emails aren't useful.
>>
>> On Fri, Nov 13, 2015 at 6:59 PM, Sean Busbey  wrote:
>>
>> > Hi Colin!
>> >
>> > If Yetus is working on an issue and can't tell what the intended branch
>> is
>> > it points folks to project specific contribution guides.
>> >
>> > For Hadoop, the patch naming for specific branches should be covered in
>> > this section of Hadoop's contribution guide:
>> >
>> > http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
>> >
>> > Yetus will actually support a little bit more than that guide suggests.
>> If
>> > a project doesn't define a URL to point people at for help in naming
>> > patches we default to this guide:
>> >
>> > https://yetus.apache.org/documentation/latest/precommit-patchnames/
>> >
>> >
>> >
>> > On Fri, Nov 13, 2015 at 8:05 PM, Colin P. McCabe 
>> > wrote:
>> >
>> > > Thanks, Allen, I wasn't aware that Yetus now supported testing for
>> > > other branches.  Is there documentation about how to name the branch
>> > > so it gets tested?
>> > >
>> > > best,
>> > > Colin
>> > >
>> > > On Fri, Nov 13, 2015 at 7:52 AM, Allen Wittenauer 
>> > > wrote:
>> > > >
>> > > >> On Nov 12, 2015, at 10:55 AM, Colin P. McCabe 
>> > > wrote:
>> > > >>
>> > > >> gerrit has a button on the UI to cherry-pick to different branches.
>> > > >> The button creates separate "gerrit changes" which you can then
>> > > >> commit.  Eventually we could hook those up to Jenkins-- something
>> > > >> which we've never been able to do for different branches with the
>> > > >> patch-file-based workflow.
>> > > >
>> > > >
>> > > > If you’re saying what I think you’re saying, people have been
>> > > able to submit patches via JIRA patch file attachment to major branches
>> > for
>> > > a few months now. Yetus closes the loop and supports pretty much any
>> > branch
>> > > or git hash.  (Github PRs also go to their respective branch or git
>> hash
>> > as
>> > > well.)
>> > >
>> >
>> >
>> >
>> > --
>> > Sean
>> >
>>

Re: Github integration for Hadoop

[Eric Yang]

Hi Andrew,

Many Apache projects already have Github integration.  Chukwa also has
Github integration.  Pull request can be integrated into JIRA, i.e.
https://issues.apache.org/jira/browse/CHUKWA-783

This allows code review process to happen at per line level on Github, or
comment on JIRA for summary of the activities.  Micro comments are squash
away.  The final commit is always happening on Apache git rather than
Github.  Therefore, there is no disabling required for pull request.
Primary activity is still on JIRA, and Github is only a supplement to make
line by line code review easy.  Overall user experience is better than RB
or Gerrit in my opinion.

regards,
Eric

On Thu, Nov 19, 2015 at 11:28 AM, Andrew Wang 
wrote:

> Has there been any progress on locking down the Github permissions like I
> asked above? It's been about 3 weeks.
>
> Steve also just asked on another thread to turn off the emails to
> common-dev. Since we're sticking with JIRA as the source-of-truth, the
> common-dev emails aren't useful.
>
> On Fri, Nov 13, 2015 at 6:59 PM, Sean Busbey  wrote:
>
> > Hi Colin!
> >
> > If Yetus is working on an issue and can't tell what the intended branch
> is
> > it points folks to project specific contribution guides.
> >
> > For Hadoop, the patch naming for specific branches should be covered in
> > this section of Hadoop's contribution guide:
> >
> > http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
> >
> > Yetus will actually support a little bit more than that guide suggests.
> If
> > a project doesn't define a URL to point people at for help in naming
> > patches we default to this guide:
> >
> > https://yetus.apache.org/documentation/latest/precommit-patchnames/
> >
> >
> >
> > On Fri, Nov 13, 2015 at 8:05 PM, Colin P. McCabe 
> > wrote:
> >
> > > Thanks, Allen, I wasn't aware that Yetus now supported testing for
> > > other branches.  Is there documentation about how to name the branch
> > > so it gets tested?
> > >
> > > best,
> > > Colin
> > >
> > > On Fri, Nov 13, 2015 at 7:52 AM, Allen Wittenauer 
> > > wrote:
> > > >
> > > >> On Nov 12, 2015, at 10:55 AM, Colin P. McCabe 
> > > wrote:
> > > >>
> > > >> gerrit has a button on the UI to cherry-pick to different branches.
> > > >> The button creates separate "gerrit changes" which you can then
> > > >> commit.  Eventually we could hook those up to Jenkins-- something
> > > >> which we've never been able to do for different branches with the
> > > >> patch-file-based workflow.
> > > >
> > > >
> > > > If you’re saying what I think you’re saying, people have been
> > > able to submit patches via JIRA patch file attachment to major branches
> > for
> > > a few months now. Yetus closes the loop and supports pretty much any
> > branch
> > > or git hash.  (Github PRs also go to their respective branch or git
> hash
> > as
> > > well.)
> > >
> >
> >
> >
> > --
> > Sean
> >
>

Re: Github integration for Hadoop

[Andrew Wang]

Has there been any progress on locking down the Github permissions like I
asked above? It's been about 3 weeks.

Steve also just asked on another thread to turn off the emails to
common-dev. Since we're sticking with JIRA as the source-of-truth, the
common-dev emails aren't useful.

On Fri, Nov 13, 2015 at 6:59 PM, Sean Busbey  wrote:

> Hi Colin!
>
> If Yetus is working on an issue and can't tell what the intended branch is
> it points folks to project specific contribution guides.
>
> For Hadoop, the patch naming for specific branches should be covered in
> this section of Hadoop's contribution guide:
>
> http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
>
> Yetus will actually support a little bit more than that guide suggests. If
> a project doesn't define a URL to point people at for help in naming
> patches we default to this guide:
>
> https://yetus.apache.org/documentation/latest/precommit-patchnames/
>
>
>
> On Fri, Nov 13, 2015 at 8:05 PM, Colin P. McCabe 
> wrote:
>
> > Thanks, Allen, I wasn't aware that Yetus now supported testing for
> > other branches.  Is there documentation about how to name the branch
> > so it gets tested?
> >
> > best,
> > Colin
> >
> > On Fri, Nov 13, 2015 at 7:52 AM, Allen Wittenauer 
> > wrote:
> > >
> > >> On Nov 12, 2015, at 10:55 AM, Colin P. McCabe 
> > wrote:
> > >>
> > >> gerrit has a button on the UI to cherry-pick to different branches.
> > >> The button creates separate "gerrit changes" which you can then
> > >> commit.  Eventually we could hook those up to Jenkins-- something
> > >> which we've never been able to do for different branches with the
> > >> patch-file-based workflow.
> > >
> > >
> > > If you’re saying what I think you’re saying, people have been
> > able to submit patches via JIRA patch file attachment to major branches
> for
> > a few months now. Yetus closes the loop and supports pretty much any
> branch
> > or git hash.  (Github PRs also go to their respective branch or git hash
> as
> > well.)
> >
>
>
>
> --
> Sean
>

Re: Github integration for Hadoop

[Allen Wittenauer]

> On Nov 12, 2015, at 10:55 AM, Colin P. McCabe  wrote:
> 
> gerrit has a button on the UI to cherry-pick to different branches.
> The button creates separate "gerrit changes" which you can then
> commit.  Eventually we could hook those up to Jenkins-- something
> which we've never been able to do for different branches with the
> patch-file-based workflow.


If you’re saying what I think you’re saying, people have been able to 
submit patches via JIRA patch file attachment to major branches for a few 
months now. Yetus closes the loop and supports pretty much any branch or git 
hash.  (Github PRs also go to their respective branch or git hash as well.)

Re: Github integration for Hadoop

[Sean Busbey]

Hi Colin!

If Yetus is working on an issue and can't tell what the intended branch is
it points folks to project specific contribution guides.

For Hadoop, the patch naming for specific branches should be covered in
this section of Hadoop's contribution guide:

http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch

Yetus will actually support a little bit more than that guide suggests. If
a project doesn't define a URL to point people at for help in naming
patches we default to this guide:

https://yetus.apache.org/documentation/latest/precommit-patchnames/



On Fri, Nov 13, 2015 at 8:05 PM, Colin P. McCabe  wrote:

> Thanks, Allen, I wasn't aware that Yetus now supported testing for
> other branches.  Is there documentation about how to name the branch
> so it gets tested?
>
> best,
> Colin
>
> On Fri, Nov 13, 2015 at 7:52 AM, Allen Wittenauer 
> wrote:
> >
> >> On Nov 12, 2015, at 10:55 AM, Colin P. McCabe 
> wrote:
> >>
> >> gerrit has a button on the UI to cherry-pick to different branches.
> >> The button creates separate "gerrit changes" which you can then
> >> commit.  Eventually we could hook those up to Jenkins-- something
> >> which we've never been able to do for different branches with the
> >> patch-file-based workflow.
> >
> >
> > If you’re saying what I think you’re saying, people have been
> able to submit patches via JIRA patch file attachment to major branches for
> a few months now. Yetus closes the loop and supports pretty much any branch
> or git hash.  (Github PRs also go to their respective branch or git hash as
> well.)
>



-- 
Sean

Re: Github integration for Hadoop

[Colin P. McCabe]

Thanks, Allen, I wasn't aware that Yetus now supported testing for
other branches.  Is there documentation about how to name the branch
so it gets tested?

best,
Colin

On Fri, Nov 13, 2015 at 7:52 AM, Allen Wittenauer  wrote:
>
>> On Nov 12, 2015, at 10:55 AM, Colin P. McCabe  wrote:
>>
>> gerrit has a button on the UI to cherry-pick to different branches.
>> The button creates separate "gerrit changes" which you can then
>> commit.  Eventually we could hook those up to Jenkins-- something
>> which we've never been able to do for different branches with the
>> patch-file-based workflow.
>
>
> If you’re saying what I think you’re saying, people have been able to 
> submit patches via JIRA patch file attachment to major branches for a few 
> months now. Yetus closes the loop and supports pretty much any branch or git 
> hash.  (Github PRs also go to their respective branch or git hash as well.)

Re: Github integration for Hadoop

[Colin P. McCabe]

gerrit has a button on the UI to cherry-pick to different branches.
The button creates separate "gerrit changes" which you can then
commit.  Eventually we could hook those up to Jenkins-- something
which we've never been able to do for different branches with the
patch-file-based workflow.

best,
Colin

On Tue, Nov 10, 2015 at 12:17 PM, Steve Loughran  wrote:
>
>> On 6 Nov 2015, at 15:21, Owen O'Malley  wrote:
>>
>> On Sun, Nov 1, 2015 at 2:07 PM, Chris Douglas  wrote:
>>
>>> Wow, this happened quickly.
>>>
>>> Owen, could you please create a Wiki describing the proposal and
>>> cataloging infra references so others can understand the
>>> implementation in detail? Even after reading this thread, I'm still
>>> confused what changes this proposes and how the integration works. A
>>> document pairing open questions with answers/workarounds would help
>>> this converge.
>>>
>>> Ok, I used Mahout's page as a basis. Take a look:
>>
>> https://wiki.apache.org/hadoop/GithubIntegration
>
> Ok, wen't through it, as well as formatting
>
> 1. used feature/hadoop- for branch names
> 2. added some best practices
>
> I've not covered how to patch/cherry pick across branches; I don't know what 
> the strategy should be there with pull requests involved. With classic git 
> apply patches I prefer to commit to branch-2 & cherry pick forwards, just so 
> that CHANGES.TXT is consistent

Re: Github integration for Hadoop

[Karthik Kambatla]

Owen: Thanks for putting the documentation together. I think I understand
the proposal better now.

I agree that reviewing on github is easier than having to download the
patch, apply locally, and for reviews copy-paste code to the JIRA. This, we
get from RB or any other review tool as well.

The committer process seems rather involved though. I am afraid the
complicated process will adversely affect the amount of time people spend
committing. The review is simpler though. By any chance, are there scripts
available (from Spark etc.) or can we put some together to download a patch
based on PRs? If yes, we could commit it as we have been doing so far.

While admitting to my bias towards gerrit, even objectively, I feel using
gerrit as a contributor/committer is way simpler.



On Fri, Nov 6, 2015 at 7:21 AM, Owen O'Malley  wrote:

> On Sun, Nov 1, 2015 at 2:07 PM, Chris Douglas  wrote:
>
> > Wow, this happened quickly.
> >
> > Owen, could you please create a Wiki describing the proposal and
> > cataloging infra references so others can understand the
> > implementation in detail? Even after reading this thread, I'm still
> > confused what changes this proposes and how the integration works. A
> > document pairing open questions with answers/workarounds would help
> > this converge.
> >
> > Ok, I used Mahout's page as a basis. Take a look:
>
> https://wiki.apache.org/hadoop/GithubIntegration
>

Re: Github integration for Hadoop

[Haohui Mai]

I have to some scripts that are tailored for the current git workflow,
which is available at

https://github.com/haohui/hdcli

It's relatively straightforward to make it support github.

~Haohui

On Tue, Nov 10, 2015 at 9:31 AM, Karthik Kambatla  wrote:
> Owen: Thanks for putting the documentation together. I think I understand
> the proposal better now.
>
> I agree that reviewing on github is easier than having to download the
> patch, apply locally, and for reviews copy-paste code to the JIRA. This, we
> get from RB or any other review tool as well.
>
> The committer process seems rather involved though. I am afraid the
> complicated process will adversely affect the amount of time people spend
> committing. The review is simpler though. By any chance, are there scripts
> available (from Spark etc.) or can we put some together to download a patch
> based on PRs? If yes, we could commit it as we have been doing so far.
>
> While admitting to my bias towards gerrit, even objectively, I feel using
> gerrit as a contributor/committer is way simpler.
>
>
>
> On Fri, Nov 6, 2015 at 7:21 AM, Owen O'Malley  wrote:
>
>> On Sun, Nov 1, 2015 at 2:07 PM, Chris Douglas  wrote:
>>
>> > Wow, this happened quickly.
>> >
>> > Owen, could you please create a Wiki describing the proposal and
>> > cataloging infra references so others can understand the
>> > implementation in detail? Even after reading this thread, I'm still
>> > confused what changes this proposes and how the integration works. A
>> > document pairing open questions with answers/workarounds would help
>> > this converge.
>> >
>> > Ok, I used Mahout's page as a basis. Take a look:
>>
>> https://wiki.apache.org/hadoop/GithubIntegration
>>

Re: Github integration for Hadoop

[Steve Loughran]

> On 6 Nov 2015, at 15:21, Owen O'Malley  wrote:
> 
> On Sun, Nov 1, 2015 at 2:07 PM, Chris Douglas  wrote:
> 
>> Wow, this happened quickly.
>> 
>> Owen, could you please create a Wiki describing the proposal and
>> cataloging infra references so others can understand the
>> implementation in detail? Even after reading this thread, I'm still
>> confused what changes this proposes and how the integration works. A
>> document pairing open questions with answers/workarounds would help
>> this converge.
>> 
>> Ok, I used Mahout's page as a basis. Take a look:
> 
> https://wiki.apache.org/hadoop/GithubIntegration

Ok, wen't through it, as well as formatting

1. used feature/hadoop- for branch names
2. added some best practices

I've not covered how to patch/cherry pick across branches; I don't know what 
the strategy should be there with pull requests involved. With classic git 
apply patches I prefer to commit to branch-2 & cherry pick forwards, just so 
that CHANGES.TXT is consistent

Re: Github integration for Hadoop

[Owen O'Malley]

On Sun, Nov 1, 2015 at 2:07 PM, Chris Douglas  wrote:

> Wow, this happened quickly.
>
> Owen, could you please create a Wiki describing the proposal and
> cataloging infra references so others can understand the
> implementation in detail? Even after reading this thread, I'm still
> confused what changes this proposes and how the integration works. A
> document pairing open questions with answers/workarounds would help
> this converge.
>
> Ok, I used Mahout's page as a basis. Take a look:

https://wiki.apache.org/hadoop/GithubIntegration

Re: Github integration for Hadoop

[Sean Busbey]

On Sun, Nov 1, 2015 at 12:52 PM, Allen Wittenauer  wrote:
>
>> On Nov 1, 2015, at 6:05 AM, Tsuyoshi Ozawa  wrote:
>>
>> Thank you for starting this discussion. It's good for us to rethink
>> our workflow to grow community.
>> However, at the moment, my concern is that we can put more pressure on
>> Yetus community if we move the main workflows into github.
>>
>> Allen and Sean, what do you think? Is it good timing for Yetus community?
>
>
> There’s nothing for Yetus to do here; support for github PRs has been 
> there since August. I regularly use it to pull down my own PRs against my own 
> hadoop branch to test with.
>
> The only work is for someone to modify the Jenkins job that runs 
> test-patch to either support both JIRA and GH or to create another job.  
> That’s on the Hadoop community to do.

Just one additional Yetus point here: no new jenkins job is needed if
folks leave a comment with a link to their PR and set the jira status
to "patch available".

-- 
Sean

Re: Github integration for Hadoop

[Zhe Zhang]

>
> So I think Gerrit/Crucible/whatever are on the table, with some work.
> Does anyone want to take the token for asking other projects and
> assembling a proposal for an alternative they're excited about?


Thanks for the update and the suggestion Chris, I think it's a great idea.
I'd like to volunteer for the above plan.

---
Zhe Zhang

On Sun, Nov 1, 2015 at 2:07 PM, Chris Douglas  wrote:

> Wow, this happened quickly.
>
> Owen, could you please create a Wiki describing the proposal and
> cataloging infra references so others can understand the
> implementation in detail? Even after reading this thread, I'm still
> confused what changes this proposes and how the integration works. A
> document pairing open questions with answers/workarounds would help
> this converge.
>
> CC'd David Nalley, who reached out ~6 months ago to ask about
> Gerrit/Crucible/etc. He said that infra would be open to supporting
> any of these tools, but they'll want to study what's required.
> Further, they don't want to support *all* these tools, so if there are
> projects other than Hadoop that want to be part of an experimental
> program, we should collectively pick one and work with infra to ensure
> the foundation's criteria are satisfied (i.e., clear IP provenance)
> and the maintenance burden is manageable.
>
> So I think Gerrit/Crucible/whatever are on the table, with some work.
> Does anyone want to take the token for asking other projects and
> assembling a proposal for an alternative they're excited about? -C
>
> On Fri, Oct 30, 2015 at 4:55 PM, Owen O'Malley  wrote:
> > On Fri, Oct 30, 2015 at 2:37 PM, Andrew Wang 
> > wrote:
> >
> >>
> >> Strongly agree that we need documentation for all this.
> >
> >
> > Agreed.
> >
> >
> >> Some of my open questions are also still pending.
> >>
> >
> > Which open questions?
> >
> > Owen, are you chasing all of this down? Do we need some JIRAs to track?
> >>
> >
> > Yeah, I'll take a first pass at it. I think jiras are overkill.
> >
> > .. Owen
>

Re: Github integration for Hadoop

[Colin P. McCabe]

+1 for setting up a gerrit instance to try it out.

cheers,
Colin

On Mon, Nov 2, 2015 at 7:13 AM, Zhe Zhang  wrote:
>>
>> So I think Gerrit/Crucible/whatever are on the table, with some work.
>> Does anyone want to take the token for asking other projects and
>> assembling a proposal for an alternative they're excited about?
>
>
> Thanks for the update and the suggestion Chris, I think it's a great idea.
> I'd like to volunteer for the above plan.
>
> ---
> Zhe Zhang
>
> On Sun, Nov 1, 2015 at 2:07 PM, Chris Douglas  wrote:
>
>> Wow, this happened quickly.
>>
>> Owen, could you please create a Wiki describing the proposal and
>> cataloging infra references so others can understand the
>> implementation in detail? Even after reading this thread, I'm still
>> confused what changes this proposes and how the integration works. A
>> document pairing open questions with answers/workarounds would help
>> this converge.
>>
>> CC'd David Nalley, who reached out ~6 months ago to ask about
>> Gerrit/Crucible/etc. He said that infra would be open to supporting
>> any of these tools, but they'll want to study what's required.
>> Further, they don't want to support *all* these tools, so if there are
>> projects other than Hadoop that want to be part of an experimental
>> program, we should collectively pick one and work with infra to ensure
>> the foundation's criteria are satisfied (i.e., clear IP provenance)
>> and the maintenance burden is manageable.
>>
>> So I think Gerrit/Crucible/whatever are on the table, with some work.
>> Does anyone want to take the token for asking other projects and
>> assembling a proposal for an alternative they're excited about? -C
>>
>> On Fri, Oct 30, 2015 at 4:55 PM, Owen O'Malley  wrote:
>> > On Fri, Oct 30, 2015 at 2:37 PM, Andrew Wang 
>> > wrote:
>> >
>> >>
>> >> Strongly agree that we need documentation for all this.
>> >
>> >
>> > Agreed.
>> >
>> >
>> >> Some of my open questions are also still pending.
>> >>
>> >
>> > Which open questions?
>> >
>> > Owen, are you chasing all of this down? Do we need some JIRAs to track?
>> >>
>> >
>> > Yeah, I'll take a first pass at it. I think jiras are overkill.
>> >
>> > .. Owen
>>

Re: Github integration for Hadoop

[Steve Loughran]

> On 31 Oct 2015, at 17:58, Colin P. McCabe  wrote:
> 
> Thanks for your responses here.  It sounds like the proposal here is
> for doing code reviews on GH, but still doing commits in our existing
> way.  Since it wasn't spelled out in the initial proposal, I
> interpreted it as doing both reviews and commits on GH, like Spark
> does-- which I think is problematic for all the reasons we've
> discussed here (the fact that GH introduces merge commits, the
> possibility of bypassing jira, duplicate pull requests with no search
> features to dedup them, etc. etc.)  Nobody has really come up with a
> solution for the problems caused by __committing__ through GH that
> scales to our size of community.
> 
> If there is a general consensus that __code reviews__ through GH would
> be helpful, I will change my -1 to a +0 for that.  But let's make sure
> that we are not __commiting__ through GH.  I view this as kind of an
> experiment to see how much easier things are this way, so I will try
> to keep an open mind.

+1 for that.

in fact, we may want to start a wiki page on reviewing through github, to get 
these policies written down

> 
> In parallel with this experiment, I also think we should set up a
> gerrit instance that supports code reviews and precommit testing.  As
> I said, Cloudera uses gerrit internally and we are very happy with it.
> It is nicer than GH because we can set up our own precommit hooks.
> For example, we can reject gerrit change requests that don't have a
> jira number associated with them.  Gerrit change requests can be
> created entirely from the command line as well.  Gerrit is open
> source, and doesn't create merge commits for everything if you commit
> through it.
> 
> I think we can support multiple solutions in parallel and let people
> gravitate to the most convenient one, as long as we keep our project
> history accessible on JIRA and the mailing lists.  Also, as Andrew
> commented, let's make sure we are not setting up duplicate bug
> trackers or mailing lists on GH-- one of each of those is enough :)
> 
> Colin
> 
> On Sat, Oct 31, 2015 at 4:40 AM, Steve Loughran  
> wrote:
>> 
>>> On 30 Oct 2015, at 17:15, Colin P. McCabe  wrote:
>>> 
>>> I think the Spark guys eventually built some kind of UI on top of
>>> github to help them search through pull requests.  We would probably
>>> also need something like this.
>> 
>> https://spark-prs.appspot.com/users
>> 
>> 
>> They do have to impose naming scheme on those patches to help identify the 
>> area. You can just watch a JIRA and wait for a pull-req to arrive.
>> 
>>> 
>>> Spark uses github partially because it started as a github project, so
>>> everyone was familiar with that.  I haven't seen an answer to Andrew's
>>> question about what the value add is here for Hadoop to move to a new
>>> system.  I have seen a few comments about a better review UI and
>>> one-click patch submission, is that the main goal?
>> 
>> I do think it is good for a very fast cycle time on reviews, though that 
>> depends, of course on reviewers willing to put in the time (credit to your 
>> colleagues here, Colin).
>> 
>> 
> 

Re: Github integration for Hadoop

[Allen Wittenauer]

> On Nov 1, 2015, at 6:05 AM, Tsuyoshi Ozawa  wrote:
> 
> Thank you for starting this discussion. It's good for us to rethink
> our workflow to grow community.
> However, at the moment, my concern is that we can put more pressure on
> Yetus community if we move the main workflows into github.
> 
> Allen and Sean, what do you think? Is it good timing for Yetus community?


There’s nothing for Yetus to do here; support for github PRs has been 
there since August. I regularly use it to pull down my own PRs against my own 
hadoop branch to test with.

The only work is for someone to modify the Jenkins job that runs 
test-patch to either support both JIRA and GH or to create another job.  That’s 
on the Hadoop community to do.

Re: Github integration for Hadoop

[Tsuyoshi Ozawa]

Thank you for starting this discussion. It's good for us to rethink
our workflow to grow community.
However, at the moment, my concern is that we can put more pressure on
Yetus community if we move the main workflows into github.

Allen and Sean, what do you think? Is it good timing for Yetus community?

Best,
- Tsuyoshi

Re: Github integration for Hadoop

[Steve Loughran]

> On 30 Oct 2015, at 17:15, Colin P. McCabe  wrote:
> 
> I think the Spark guys eventually built some kind of UI on top of
> github to help them search through pull requests.  We would probably
> also need something like this.

https://spark-prs.appspot.com/users


They do have to impose naming scheme on those patches to help identify the 
area. You can just watch a JIRA and wait for a pull-req to arrive.

> 
> Spark uses github partially because it started as a github project, so
> everyone was familiar with that.  I haven't seen an answer to Andrew's
> question about what the value add is here for Hadoop to move to a new
> system.  I have seen a few comments about a better review UI and
> one-click patch submission, is that the main goal?

I do think it is good for a very fast cycle time on reviews, though that 
depends, of course on reviewers willing to put in the time (credit to your 
colleagues here, Colin). 

Re: Github integration for Hadoop

[Colin P. McCabe]

Thanks for your responses here.  It sounds like the proposal here is
for doing code reviews on GH, but still doing commits in our existing
way.  Since it wasn't spelled out in the initial proposal, I
interpreted it as doing both reviews and commits on GH, like Spark
does-- which I think is problematic for all the reasons we've
discussed here (the fact that GH introduces merge commits, the
possibility of bypassing jira, duplicate pull requests with no search
features to dedup them, etc. etc.)  Nobody has really come up with a
solution for the problems caused by __committing__ through GH that
scales to our size of community.

If there is a general consensus that __code reviews__ through GH would
be helpful, I will change my -1 to a +0 for that.  But let's make sure
that we are not __commiting__ through GH.  I view this as kind of an
experiment to see how much easier things are this way, so I will try
to keep an open mind.

In parallel with this experiment, I also think we should set up a
gerrit instance that supports code reviews and precommit testing.  As
I said, Cloudera uses gerrit internally and we are very happy with it.
It is nicer than GH because we can set up our own precommit hooks.
For example, we can reject gerrit change requests that don't have a
jira number associated with them.  Gerrit change requests can be
created entirely from the command line as well.  Gerrit is open
source, and doesn't create merge commits for everything if you commit
through it.

I think we can support multiple solutions in parallel and let people
gravitate to the most convenient one, as long as we keep our project
history accessible on JIRA and the mailing lists.  Also, as Andrew
commented, let's make sure we are not setting up duplicate bug
trackers or mailing lists on GH-- one of each of those is enough :)

Colin

On Sat, Oct 31, 2015 at 4:40 AM, Steve Loughran  wrote:
>
>> On 30 Oct 2015, at 17:15, Colin P. McCabe  wrote:
>>
>> I think the Spark guys eventually built some kind of UI on top of
>> github to help them search through pull requests.  We would probably
>> also need something like this.
>
> https://spark-prs.appspot.com/users
>
>
> They do have to impose naming scheme on those patches to help identify the 
> area. You can just watch a JIRA and wait for a pull-req to arrive.
>
>>
>> Spark uses github partially because it started as a github project, so
>> everyone was familiar with that.  I haven't seen an answer to Andrew's
>> question about what the value add is here for Hadoop to move to a new
>> system.  I have seen a few comments about a better review UI and
>> one-click patch submission, is that the main goal?
>
> I do think it is good for a very fast cycle time on reviews, though that 
> depends, of course on reviewers willing to put in the time (credit to your 
> colleagues here, Colin).
>
>

Re: Github integration for Hadoop

[Akira AJISAKA]

+1 if the current review process is kept.

Pros:

* Using PRs is good for increasing developers because PRs are very 
familiar for many developers.

* Better UI. (no need to copy the diff manually for reviewing patches!)

Cons:

* Discussions can be split. However, according to the below blog post, 
any comments and other things on PRs can be recorded on the project's 
mailing list or mirrored to the JIRA ticket, so we can search discussion 
by searching MLs or JIRAs. Therefore, IMO, it's not a big problem.


https://blogs.apache.org/infra/entry/improved_integration_between_apache_and

* github.com is out of Apache infra, so the service can be stopped for 
some reason (such as, acquired by some company or system down). I'm 
thinking we need to keep the current review process if we have switched 
to use GitHub PRs.


Regards,
Akira

On 10/30/15 16:38, Alexander Pivovarov wrote:

Andrew, look at Spark github. They use PRs and I do not see extra merge
commits. Committer can do fast-farward merge using git command line.
PRs are used to leave inline feedbacks for the fix
On Oct 29, 2015 1:34 PM, "Andrew Wang"  wrote:


Has anything changed regarding the github integration since the last time
we discussed this? That blog post is from 2014, and we discussed
alternative review systems earlier in 2015.

Colin specifically was concerned about forking the discussion between JIRA
and other places:

http://search-hadoop.com/m/uOzYtkYxo4qazi=Re+Patch+review+process
http://search-hadoop.com/m/uOzYtSz7z624qazi=Re+Patch+review+process

There are also questions about PRs leading to messy commit history with the
extra merge commits. Spark IIRC has something to linearize it again, which
seems important if we actually want to do this.

Could someone outline the upsides of using github? I don't find the review
UI particularly great compared to Gerrit or even RB, and there's the merge
commit issue. For instance, do we think using Github would lead to more
contributions? Improved developer workflows? Have we re-examined
alternatives like Gerrit or RB as well?

On Thu, Oct 29, 2015 at 12:25 PM, Arpit Agarwal 
wrote:


+1, thanks for proposing it.





On 10/29/15, 10:47 AM, "Owen O'Malley"  wrote:


All,
   For code & patch review, many of the newer projects are using the

Github

pull request integration. You can read about it here:





https://blogs.apache.org/infra/entry/improved_integration_between_apache_and


It basically lets you:
* have mirroring between comments on pull requests and jira
* lets you close pull requests
* have mirroring between pull request comments and the Apache mail lists

Thoughts?
.. Owen

Re: Github integration for Hadoop

[Andrew Wang]

On Fri, Oct 30, 2015 at 1:59 PM, Colin P. McCabe  wrote:

> I am -1 on the current GH stuff, just because I feel like there wasn't
> enough discussion, testing, and documentation of it.  The initial
> proposal had no details and was implemented before any of the people
> who had had misgivings on the previous email thread had a chance to
> even see it.
>
> It's a big change to totally change our commit workflow.  Much bigger
> than something like merging a feature branch or making a release, both
> of which require votes.  This kind of change should require an actual
> proposal and a VOTE thread.
>
> So this was only clarified this morning, but we'd be using GH only for
code review, not for code integration. So committers still need to download
the PR, squash it, update CHANGES, push.

Agree if we go beyond this, it 100% needs to be in a VOTE. Even this
probably should have been in a VOTE with a proposal spelling out all these
details.

I would be +0 on GH if there was a concrete proposal addressing
> * How this going to interact with JIRA.  For example, what is the new
> process once a bug has been identified?  Is a JIRA required?  How do
> we link JIRA and GH?  Do we reject GH requests without a JIRA?
> * What will we update the HowToContribute page to say?
> * How will we deal with the merge commit problem?  Will we have a tool
> to remove them?  Can GH be configured to avoid them?  Will we update
> tools on top (including in-house tools) to handle merge commits?
> * Will we have tooling to search GH pull requests?  How do we prevent
> patches from new committers from falling through the cracks if they
> don't know about JIRA?
>
>
Strongly agree that we need documentation for all this. Some of my open
questions are also still pending. We need to test precommit integration too.

Owen, are you chasing all of this down? Do we need some JIRAs to track?

Re: Github integration for Hadoop

[Colin P. McCabe]

I am -1 on the current GH stuff, just because I feel like there wasn't
enough discussion, testing, and documentation of it.  The initial
proposal had no details and was implemented before any of the people
who had had misgivings on the previous email thread had a chance to
even see it.

It's a big change to totally change our commit workflow.  Much bigger
than something like merging a feature branch or making a release, both
of which require votes.  This kind of change should require an actual
proposal and a VOTE thread.

I would be +0 on GH if there was a concrete proposal addressing
* How this going to interact with JIRA.  For example, what is the new
process once a bug has been identified?  Is a JIRA required?  How do
we link JIRA and GH?  Do we reject GH requests without a JIRA?
* What will we update the HowToContribute page to say?
* How will we deal with the merge commit problem?  Will we have a tool
to remove them?  Can GH be configured to avoid them?  Will we update
tools on top (including in-house tools) to handle merge commits?
* Will we have tooling to search GH pull requests?  How do we prevent
patches from new committers from falling through the cracks if they
don't know about JIRA?

Colin

On Fri, Oct 30, 2015 at 11:57 AM, Sean Busbey  wrote:
> On Fri, Oct 30, 2015 at 1:22 PM, Allen Wittenauer  wrote:
>>
>>> * Have we tried our precommit on PRs yet? Does it work for multiple
>>> branches? Is there a way to enforce rebase+squash vs. merge on the PR,
>>> since, per Allen, Yetus requires one commit to work?
>>
>>
>> I don’t know about the Jenkins-side of things (e.g., how does 
>> Jenkins trigger a build?).  As far as Yetus is concerned, here’s the 
>> functionality that has been written:
>>
>> * Pull patches from Github by PR #
>> * Comment on patches in Github, given credentials
>> * Comment on specific lines in Github, given credentials
>> * Test patches against the branch/repo that the pull request is 
>> against
>> * GH<->JIRA intelligence such that if a PR mentions an issue as the 
>> leading text in the subject line or an issue mentions a PR in the comments, 
>> pull from github and put a comment in both places (again, given credentials)
>
>
> Jenkins builds are all driven off of the "Precommit Admin" job that
> does a JQL search for enabled projects with open patches:
>
> https://builds.apache.org/view/PreCommit%20Builds/job/PreCommit-Admin/
>
> I believe with the current integration, that means we'll find an test
> any github PRs that are mentioned in a jira ticket that is in
> PATCH_AVAILABLE status.
>
> At some point we'll need an exemplar jenkins trigger job that just
> activates on open PRs, at least for ASF projects. But no such job
> exists now.
>
>
> --
> Sean

Re: Github integration for Hadoop

[Owen O'Malley]

It seems like there is overwhelming support for enabling the github
integration, so I went ahead and filed the infra ticket
.

This is explicitly not changing the way that we commit on Hadoop and
commits should be squashed and rebased rather than merged on to the master
branch. If you want to close a pull request with a commit, just add a line
at the end of the commit message that says:

closes apache/hadoop#123

If someone else wants to setup gerrit, we can evaluate it. However, I am
skeptical that it would be so much better than Github that it would be
worth making people learn a new tool.

Thanks,
   Owen

Re: Github integration for Hadoop

[Andrew Wang]

I'm trying to slow this discussion down so we can:

a) Determine the problem(s) we're trying to solve
b) See how the proposed solutions meet this problem

The flood of +1's were made before a full proposal of what "enabling github
integration" means, which has only been really specified in Owen's most
recent email.

Referring to a), the problems mentioned in this thread:

1) Better, easier review tool (Steve)
2) Better attribution of contributors with their GH id (Arpit)
3) The backlog of PRs against our unused GH project (Owen)

Applying "use GH PRs as a review tool" to the above problems:

1) I think it has advantages, but as I said earlier, Github PRs really are
not designed for our rebase+squash workflow, since AFAIK it doesn't support
interdiffs.
2) not solved since the proposal is GH only as a review tool, not for
integration
3) not solved since all issues still need to start as a JIRA, or else the
mirroring won't work.

I'd also ask the following unknown questions:

* Is there a way to *disable* using PRs for integration? i.e. disable the
ability to merge PRs?
* Have we tried our precommit on PRs yet? Does it work for multiple
branches? Is there a way to enforce rebase+squash vs. merge on the PR,
since, per Allen, Yetus requires one commit to work?

This thread is barely 24 hours old, and I don't see why we're trying to
move this fast on the issue. Let's discuss some alternatives (!) and settle
on the right solution. We also haven't even broached review alternatives
like RB, Crucible, etc which were in the running last time this topic came
up.

Best,
Andrew


On Fri, Oct 30, 2015 at 9:19 AM, Owen O'Malley  wrote:

> It seems like there is overwhelming support for enabling the github
> integration, so I went ahead and filed the infra ticket
>  >.
>
> This is explicitly not changing the way that we commit on Hadoop and
> commits should be squashed and rebased rather than merged on to the master
> branch. If you want to close a pull request with a commit, just add a line
> at the end of the commit message that says:
>
> closes apache/hadoop#123
>
> If someone else wants to setup gerrit, we can evaluate it. However, I am
> skeptical that it would be so much better than Github that it would be
> worth making people learn a new tool.
>
> Thanks,
>Owen
>

Re: Github integration for Hadoop

[Colin P. McCabe]

I think we should take more than 24 hours of discussion to make a big,
fundamental change to our code review process.

I used github while working on the Spark project, and frankly I didn't
like it.  I didn't like the way it split the discussion between JIRA
and github.  Also, often people would file a jira about something
where the was already a patch up on github for a different jira (or
sometimes no jira at all).  This led to a lot of frustration where
several people would post patches and only some of them would actually
get looked at.  github doesn't have the powerful searching or bug
tracking tools of JIRA.

Basically, imagine jira with no way to search for related issues, no
JQL, no "assignee", no "target version", "fix version", or
"component".  No "in progress" versus "open".  All you would have is a
"list of pull requests" and perhaps a username next to each item.
This system was very hard on newbies because their patches seldom got
looked at.  It is hard to find anything in that pile.

I think the Spark guys eventually built some kind of UI on top of
github to help them search through pull requests.  We would probably
also need something like this.

Spark uses github partially because it started as a github project, so
everyone was familiar with that.  I haven't seen an answer to Andrew's
question about what the value add is here for Hadoop to move to a new
system.  I have seen a few comments about a better review UI and
one-click patch submission, is that the main goal?

Colin

On Fri, Oct 30, 2015 at 9:19 AM, Owen O'Malley  wrote:
> It seems like there is overwhelming support for enabling the github
> integration, so I went ahead and filed the infra ticket
> .
>
> This is explicitly not changing the way that we commit on Hadoop and
> commits should be squashed and rebased rather than merged on to the master
> branch. If you want to close a pull request with a commit, just add a line
> at the end of the commit message that says:
>
> closes apache/hadoop#123
>
> If someone else wants to setup gerrit, we can evaluate it. However, I am
> skeptical that it would be so much better than Github that it would be
> worth making people learn a new tool.
>
> Thanks,
>Owen

Re: Github integration for Hadoop

[Sean Busbey]

On Fri, Oct 30, 2015 at 1:22 PM, Allen Wittenauer  wrote:
>
>> * Have we tried our precommit on PRs yet? Does it work for multiple
>> branches? Is there a way to enforce rebase+squash vs. merge on the PR,
>> since, per Allen, Yetus requires one commit to work?
>
>
> I don’t know about the Jenkins-side of things (e.g., how does Jenkins 
> trigger a build?).  As far as Yetus is concerned, here’s the functionality 
> that has been written:
>
> * Pull patches from Github by PR #
> * Comment on patches in Github, given credentials
> * Comment on specific lines in Github, given credentials
> * Test patches against the branch/repo that the pull request is 
> against
> * GH<->JIRA intelligence such that if a PR mentions an issue as the 
> leading text in the subject line or an issue mentions a PR in the comments, 
> pull from github and put a comment in both places (again, given credentials)


Jenkins builds are all driven off of the "Precommit Admin" job that
does a JQL search for enabled projects with open patches:

https://builds.apache.org/view/PreCommit%20Builds/job/PreCommit-Admin/

I believe with the current integration, that means we'll find an test
any github PRs that are mentioned in a jira ticket that is in
PATCH_AVAILABLE status.

At some point we'll need an exemplar jenkins trigger job that just
activates on open PRs, at least for ASF projects. But no such job
exists now.


-- 
Sean

Re: Github integration for Hadoop

[Andrew Wang]

>
>
> >>
> >> > If that's the case, we should also look at review alternatives
> >> > like RB and Crucible.
> >>
> >> Okay by me if the community consensus supports one of them. The fact
> that
> >> they exist but no one uses them is not a ringing endorsement.
> >>
> >> HBase uses reviewboard, as do I'm sure other Apache projects.
> >reviews.apache.org existed before we had github integration. I've used
> RB a
> >fair bit, and don't mind it.
>
> I could not get RB working with the Hadoop sub-projects. Would you be
> willing to try it out on a Hadoop/HDFS Jira since you have experience with
> it?
>

Here's a RB I made with a test patch, the trick is to use "git diff
--full-index":

https://reviews.apache.org/r/39820/

I'm not clear on if there's mirroring between RB and JIRA, something to
investigate.

Re: Github integration for Hadoop

[Allen Wittenauer]

> * Have we tried our precommit on PRs yet? Does it work for multiple
> branches? Is there a way to enforce rebase+squash vs. merge on the PR,
> since, per Allen, Yetus requires one commit to work?


I don’t know about the Jenkins-side of things (e.g., how does Jenkins 
trigger a build?).  As far as Yetus is concerned, here’s the functionality that 
has been written:

* Pull patches from Github by PR #
* Comment on patches in Github, given credentials
* Comment on specific lines in Github, given credentials
* Test patches against the branch/repo that the pull request is against
* GH<->JIRA intelligence such that if a PR mentions an issue as the 
leading text in the subject line or an issue mentions a PR in the comments, 
pull from github and put a comment in both places (again, given credentials)

Re: Github integration for Hadoop

[Andrew Wang]

>
>
> > 2) Better attribution of contributors with their GH id (Arpit)
> >
>
> This doesn't happen very naturally other than the pull requests are
> typically shown in their fork of the apache repo.
>
> It happens if we used PRs for integration, but yes I agree that it does
not if GH is just used as a review tool.

>
> > 3) The backlog of PRs against our unused GH project (Owen)
> >
>
> This happens too.
>
> The backlog is from people who don't want to go through JIRA and just want
to fork and PR on github.

Enabling GH-as-code-review-tool is not going to help this community of
developers.

>
> >
> > Applying "use GH PRs as a review tool" to the above problems:
> >
> > 1) I think it has advantages, but as I said earlier, Github PRs really
> are
> > not designed for our rebase+squash workflow, since AFAIK it doesn't
> support
> > interdiffs.
> >
>
> Yes, it does. Pushing to the branch that the pull request was created from
> updates the pull request.
>

That's not interdiff support, which was my main point about differences in
workflow.

>
>
> > 3) not solved since all issues still need to start as a JIRA, or else the
> > mirroring won't work.
> >
>
> That isn't true. If you push an empty commit that tells the integration to
> close pull requests, they are closed. We could clean up the current pull
> requests now that the integration is there.
>
> Cleaning them up is good, but as I said above it doesn't help this
community of users who filed these in the first place. That is, users who
want to use GitHub but don't like JIRA.

I also searched "disable pull requests" and found this, which could help
with this issue:

https://nopullrequests.appspot.com/

>
> >
> > I'd also ask the following unknown questions:
> >
> > * Is there a way to *disable* using PRs for integration? i.e. disable the
> > ability to merge PRs?
> >
>
> That is already disabled.
>
> Great, glad to hear it. That wasn't mentioned in the email thread or on
the INFRA ticket, and the INFRA ticket mentions these integrations:

Commit Comment, Create, Issue Comment, Issues, Pull Request, Pull Request
Comment, and push, Push

Are these the right set of permissions to disable integrating PRs? Namely,
the "push" permissions look unnecessary. We should also disable GH issues
since we don't want users filing issues there.

>
> > * Have we tried our precommit on PRs yet? Does it work for multiple
> > branches? Is there a way to enforce rebase+squash vs. merge on the PR,
> > since, per Allen, Yetus requires one commit to work?
> >
>
> Allen said that it was either working or easy to make work.
>
> Great to hear, this was not mentioned on this email thread besides Allen
saying multi-commit PRs are problematic.

>
> >
> > This thread is barely 24 hours old, and I don't see why we're trying to
> > move this fast on the issue. Let's discuss some alternatives (!) and
> settle
> > on the right solution. We also haven't even broached review alternatives
> > like RB, Crucible, etc which were in the running last time this topic
> came
> > up.
> >
>
> I'm sorry that I moved too fast. There are clearly a lot of people who want
> to use it as a review tool and getting github integration enabled is easy.
> Furthermore, it isn't exclusive. There will still be people uploading
> patches on jira. This is about giving the contributors and committers the
> ability to use a popular review tool
>
> I'm onboard with improved code review tools. It was not clear this
proposal was for github *only* as a code review tool until about an hour
ago. It also wasn't clear to a lot of the people who were +1ing based on
the long-form comments.

My one sticking point is that GH remains only a code review tool until a
later discussion. So, no merging of PRs directly, and JIRAs come before
PRs. If people want to revisit this, let's discuss in a thread that does
not begin and end in 24 hours, and do a proper proposal and vote so there
aren't ambiguities or open questions before things happen.

Best,
Andrew

Github integration for Hadoop

[Owen O'Malley]

All,
   For code & patch review, many of the newer projects are using the Github
pull request integration. You can read about it here:

https://blogs.apache.org/infra/entry/improved_integration_between_apache_and

It basically lets you:
* have mirroring between comments on pull requests and jira
* lets you close pull requests
* have mirroring between pull request comments and the Apache mail lists

Thoughts?
.. Owen

Re: Github integration for Hadoop

[Hitesh Shah]

+1 on supporting patch contributions through github pull requests.

— Hitesh

On Oct 29, 2015, at 10:47 AM, Owen O'Malley  wrote:

> All,
>   For code & patch review, many of the newer projects are using the Github
> pull request integration. You can read about it here:
> 
> https://blogs.apache.org/infra/entry/improved_integration_between_apache_and
> 
> It basically lets you:
> * have mirroring between comments on pull requests and jira
> * lets you close pull requests
> * have mirroring between pull request comments and the Apache mail lists
> 
> Thoughts?
> .. Owen

Re: Github integration for Hadoop

[Haohui Mai]

+1

On Thu, Oct 29, 2015 at 10:55 AM, Hitesh Shah  wrote:
> +1 on supporting patch contributions through github pull requests.
>
> — Hitesh
>
> On Oct 29, 2015, at 10:47 AM, Owen O'Malley  wrote:
>
>> All,
>>   For code & patch review, many of the newer projects are using the Github
>> pull request integration. You can read about it here:
>>
>> https://blogs.apache.org/infra/entry/improved_integration_between_apache_and
>>
>> It basically lets you:
>> * have mirroring between comments on pull requests and jira
>> * lets you close pull requests
>> * have mirroring between pull request comments and the Apache mail lists
>>
>> Thoughts?
>> .. Owen
>

Re: Github integration for Hadoop

[Ashish]

+1

On Thu, Oct 29, 2015 at 11:51 AM, Mingliang Liu  wrote:
> +1 (non-binding)
>
> Mingliang Liu
> Member of Technical Staff - HDFS,
> Hortonworks Inc.
> m...@hortonworks.com
>
>
>
>> On Oct 29, 2015, at 10:55 AM, Hitesh Shah  wrote:
>>
>> +1 on supporting patch contributions through github pull requests.
>>
>> — Hitesh
>>
>> On Oct 29, 2015, at 10:47 AM, Owen O'Malley  wrote:
>>
>>> All,
>>>  For code & patch review, many of the newer projects are using the Github
>>> pull request integration. You can read about it here:
>>>
>>> https://blogs.apache.org/infra/entry/improved_integration_between_apache_and
>>>
>>> It basically lets you:
>>> * have mirroring between comments on pull requests and jira
>>> * lets you close pull requests
>>> * have mirroring between pull request comments and the Apache mail lists
>>>
>>> Thoughts?
>>> .. Owen
>>
>>
>



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Re: Github integration for Hadoop

[Sean Busbey]

It looks like there's pretty good consensus. Why do we need a VOTE thread?

Perhaps better for someone to submit a patch with proposed text for
hte contribution guide[1]?

-Sean

[1]: http://wiki.apache.org/hadoop/HowToContribute

On Thu, Oct 29, 2015 at 2:01 PM, Xiaoyu Yao  wrote:
> +1, should we start a vote on this?
>
>
>
>
> On 10/29/15, 11:54 AM, "Ashish"  wrote:
>
>>+1
>>
>>On Thu, Oct 29, 2015 at 11:51 AM, Mingliang Liu  wrote:
>>> +1 (non-binding)
>>>
>>> Mingliang Liu
>>> Member of Technical Staff - HDFS,
>>> Hortonworks Inc.
>>> m...@hortonworks.com
>>>
>>>
>>>
 On Oct 29, 2015, at 10:55 AM, Hitesh Shah  wrote:

 +1 on supporting patch contributions through github pull requests.

 — Hitesh

 On Oct 29, 2015, at 10:47 AM, Owen O'Malley  wrote:

> All,
>  For code & patch review, many of the newer projects are using the Github
> pull request integration. You can read about it here:
>
> https://blogs.apache.org/infra/entry/improved_integration_between_apache_and
>
> It basically lets you:
> * have mirroring between comments on pull requests and jira
> * lets you close pull requests
> * have mirroring between pull request comments and the Apache mail lists
>
> Thoughts?
> .. Owen


>>>
>>
>>
>>
>>--
>>thanks
>>ashish
>>
>>Blog: http://www.ashishpaliwal.com/blog
>>My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>



-- 
Sean

Re: Github integration for Hadoop

[Chang Li]

+1 (non-binding)

On Thu, Oct 29, 2015 at 1:54 PM, Ashish  wrote:

> +1
>
> On Thu, Oct 29, 2015 at 11:51 AM, Mingliang Liu 
> wrote:
> > +1 (non-binding)
> >
> > Mingliang Liu
> > Member of Technical Staff - HDFS,
> > Hortonworks Inc.
> > m...@hortonworks.com
> >
> >
> >
> >> On Oct 29, 2015, at 10:55 AM, Hitesh Shah  wrote:
> >>
> >> +1 on supporting patch contributions through github pull requests.
> >>
> >> — Hitesh
> >>
> >> On Oct 29, 2015, at 10:47 AM, Owen O'Malley  wrote:
> >>
> >>> All,
> >>>  For code & patch review, many of the newer projects are using the
> Github
> >>> pull request integration. You can read about it here:
> >>>
> >>>
> https://blogs.apache.org/infra/entry/improved_integration_between_apache_and
> >>>
> >>> It basically lets you:
> >>> * have mirroring between comments on pull requests and jira
> >>> * lets you close pull requests
> >>> * have mirroring between pull request comments and the Apache mail
> lists
> >>>
> >>> Thoughts?
> >>> .. Owen
> >>
> >>
> >
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>

Re: Github integration for Hadoop

[Xiaoyu Yao]

+1, should we start a vote on this?




On 10/29/15, 11:54 AM, "Ashish"  wrote:

>+1
>
>On Thu, Oct 29, 2015 at 11:51 AM, Mingliang Liu  wrote:
>> +1 (non-binding)
>>
>> Mingliang Liu
>> Member of Technical Staff - HDFS,
>> Hortonworks Inc.
>> m...@hortonworks.com
>>
>>
>>
>>> On Oct 29, 2015, at 10:55 AM, Hitesh Shah  wrote:
>>>
>>> +1 on supporting patch contributions through github pull requests.
>>>
>>> — Hitesh
>>>
>>> On Oct 29, 2015, at 10:47 AM, Owen O'Malley  wrote:
>>>
 All,
  For code & patch review, many of the newer projects are using the Github
 pull request integration. You can read about it here:

 https://blogs.apache.org/infra/entry/improved_integration_between_apache_and

 It basically lets you:
 * have mirroring between comments on pull requests and jira
 * lets you close pull requests
 * have mirroring between pull request comments and the Apache mail lists

 Thoughts?
 .. Owen
>>>
>>>
>>
>
>
>
>-- 
>thanks
>ashish
>
>Blog: http://www.ashishpaliwal.com/blog
>My Photo Galleries: http://www.pbase.com/ashishpaliwal
>

Re: Github integration for Hadoop

[Vinod Vavilapalli]

Don’t think we need a vote. If someone can demonstrate how this works 
end-to-end, and enough folks find it useful, we can start using it. There is no 
need for a mandate.

+Vinod

> On Oct 29, 2015, at 12:01 PM, Xiaoyu Yao  wrote:
> 
> +1, should we start a vote on this?
> 
> 
> 
> 
> On 10/29/15, 11:54 AM, "Ashish"  wrote:
> 
>> +1
>> 
>> On Thu, Oct 29, 2015 at 11:51 AM, Mingliang Liu  wrote:
>>> +1 (non-binding)
>>> 
>>> Mingliang Liu
>>> Member of Technical Staff - HDFS,
>>> Hortonworks Inc.
>>> m...@hortonworks.com
>>> 
>>> 
>>> 
 On Oct 29, 2015, at 10:55 AM, Hitesh Shah  wrote:
 
 +1 on supporting patch contributions through github pull requests.
 
 — Hitesh
 
 On Oct 29, 2015, at 10:47 AM, Owen O'Malley  wrote:
 
> All,
> For code & patch review, many of the newer projects are using the Github
> pull request integration. You can read about it here:
> 
> https://blogs.apache.org/infra/entry/improved_integration_between_apache_and
> 
> It basically lets you:
> * have mirroring between comments on pull requests and jira
> * lets you close pull requests
> * have mirroring between pull request comments and the Apache mail lists
> 
> Thoughts?
> .. Owen
 
 
>>> 
>> 
>> 
>> 
>> -- 
>> thanks
>> ashish
>> 
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>> 

Re: Github integration for Hadoop

[Arpit Agarwal]

+1, thanks for proposing it.





On 10/29/15, 10:47 AM, "Owen O'Malley"  wrote:

>All,
>   For code & patch review, many of the newer projects are using the Github
>pull request integration. You can read about it here:
>
>https://blogs.apache.org/infra/entry/improved_integration_between_apache_and
>
>It basically lets you:
>* have mirroring between comments on pull requests and jira
>* lets you close pull requests
>* have mirroring between pull request comments and the Apache mail lists
>
>Thoughts?
>.. Owen

Re: Github integration for Hadoop

[Mingliang Liu]

+1 (non-binding)

Mingliang Liu
Member of Technical Staff - HDFS,
Hortonworks Inc.
m...@hortonworks.com



> On Oct 29, 2015, at 10:55 AM, Hitesh Shah  wrote:
> 
> +1 on supporting patch contributions through github pull requests.
> 
> — Hitesh
> 
> On Oct 29, 2015, at 10:47 AM, Owen O'Malley  wrote:
> 
>> All,
>>  For code & patch review, many of the newer projects are using the Github
>> pull request integration. You can read about it here:
>> 
>> https://blogs.apache.org/infra/entry/improved_integration_between_apache_and
>> 
>> It basically lets you:
>> * have mirroring between comments on pull requests and jira
>> * lets you close pull requests
>> * have mirroring between pull request comments and the Apache mail lists
>> 
>> Thoughts?
>> .. Owen
> 
> 

Re: Github integration for Hadoop

[Andrew Wang]

Has anything changed regarding the github integration since the last time
we discussed this? That blog post is from 2014, and we discussed
alternative review systems earlier in 2015.

Colin specifically was concerned about forking the discussion between JIRA
and other places:

http://search-hadoop.com/m/uOzYtkYxo4qazi=Re+Patch+review+process
http://search-hadoop.com/m/uOzYtSz7z624qazi=Re+Patch+review+process

There are also questions about PRs leading to messy commit history with the
extra merge commits. Spark IIRC has something to linearize it again, which
seems important if we actually want to do this.

Could someone outline the upsides of using github? I don't find the review
UI particularly great compared to Gerrit or even RB, and there's the merge
commit issue. For instance, do we think using Github would lead to more
contributions? Improved developer workflows? Have we re-examined
alternatives like Gerrit or RB as well?

On Thu, Oct 29, 2015 at 12:25 PM, Arpit Agarwal 
wrote:

> +1, thanks for proposing it.
>
>
>
>
>
> On 10/29/15, 10:47 AM, "Owen O'Malley"  wrote:
>
> >All,
> >   For code & patch review, many of the newer projects are using the
> Github
> >pull request integration. You can read about it here:
> >
> >
> https://blogs.apache.org/infra/entry/improved_integration_between_apache_and
> >
> >It basically lets you:
> >* have mirroring between comments on pull requests and jira
> >* lets you close pull requests
> >* have mirroring between pull request comments and the Apache mail lists
> >
> >Thoughts?
> >.. Owen
>

Re: Github integration for Hadoop

[Steve Loughran]

> On 29 Oct 2015, at 20:34, Andrew Wang  wrote:
> 
> Has anything changed regarding the github integration since the last time
> we discussed this? That blog post is from 2014, and we discussed
> alternative review systems earlier in 2015.
> 
> Colin specifically was concerned about forking the discussion between JIRA
> and other places:
> 
> http://search-hadoop.com/m/uOzYtkYxo4qazi=Re+Patch+review+process
> http://search-hadoop.com/m/uOzYtSz7z624qazi=Re+Patch+review+process
> 
> There are also questions about PRs leading to messy commit history with the
> extra merge commits. Spark IIRC has something to linearize it again, which
> seems important if we actually want to do this.
> 
> Could someone outline the upsides of using github? I don't find the review
> UI particularly great compared to Gerrit or even RB, and there's the merge
> commit issue. For instance, do we think using Github would lead to more
> contributions? Improved developer workflows? Have we re-examined
> alternatives like Gerrit or RB as well?

I've been using it for some ATS integration work

For simple patches, you can do some good review cycles

https://github.com/apache/spark/pull/9232

For something big, well, it gets big

https://github.com/apache/spark/pull/8744#discussion_r43388336

... you have to switch to the many-file view, which kind of loses some of the 
temporal ordering of comments —instead you get lots of little threads on 
individual issues. Which may scale better

https://github.com/apache/spark/pull/8744/files

I'd need more experience to come to a real conclusion. What I do like is the 
immediate push-to-trigger rebuild process, no need to create patches and submit 
them. That I like.


> On Thu, Oct 29, 2015 at 12:25 PM, Arpit Agarwal 
> wrote:
> 
>> +1, thanks for proposing it.
>> 
>> 
>> 
>> 
>> 
>> On 10/29/15, 10:47 AM, "Owen O'Malley"  wrote:
>> 
>>> All,
>>>  For code & patch review, many of the newer projects are using the
>> Github
>>> pull request integration. You can read about it here:
>>> 
>>> 
>> https://blogs.apache.org/infra/entry/improved_integration_between_apache_and
>>> 
>>> It basically lets you:
>>> * have mirroring between comments on pull requests and jira
>>> * lets you close pull requests
>>> * have mirroring between pull request comments and the Apache mail lists
>>> 
>>> Thoughts?
>>> .. Owen
>> 

Re: Github integration for Hadoop

[Owen O'Malley]

On Thu, Oct 29, 2015 at 1:34 PM, Andrew Wang 
wrote:

> Has anything changed regarding the github integration since the last time
> we discussed this?


There is a lot more experience with it now and opinions change over time.
Clearly, there is overwhelming support for this idea now.


> There are also questions about PRs leading to messy commit history with the
> extra merge commits. Spark IIRC has something to linearize it again, which
> seems important if we actually want to do this.
>

In the vast majority of cases, the pull requests should be squashed to a
single commit. The Github integration doesn't change how patches are
committed or pushed into the repository. It is about using the code review
tools that Github provides.

Note that we already have Github pull requests for Hadoop. We just don't
have any way to manage them and they aren't mirrored to the Apache lists &
jiras. Many people find the Github code review tools useful and Apache has
the tools to integrate them into our mailing lists and jira.

.. Owen

Re: Github integration for Hadoop

[Andrew Wang]

On Thu, Oct 29, 2015 at 3:12 PM, Owen O'Malley  wrote:

> On Thu, Oct 29, 2015 at 1:34 PM, Andrew Wang 
> wrote:
>
> > Has anything changed regarding the github integration since the last time
> > we discussed this?
>
>
> There is a lot more experience with it now and opinions change over time.
> Clearly, there is overwhelming support for this idea now.
>
>
> > There are also questions about PRs leading to messy commit history with
> the
> > extra merge commits. Spark IIRC has something to linearize it again,
> which
> > seems important if we actually want to do this.
> >
>
> In the vast majority of cases, the pull requests should be squashed to a
> single commit. The Github integration doesn't change how patches are
> committed or pushed into the repository. It is about using the code review
> tools that Github provides.
>
> Okay, it wasn't clear to me that this proposal was only to use Github as a
review tool, not for patch integration.

If that's the case, we should also look at review alternatives like RB and
Crucible. RB also has cmdline tools for easily updating a patch.


> Note that we already have Github pull requests for Hadoop. We just don't
> have any way to manage them and they aren't mirrored to the Apache lists &
> jiras. Many people find the Github code review tools useful and Apache has
> the tools to integrate them into our mailing lists and jira.
>

If the issue is PRs that no one looks at, one option is to disable PRs and
tell these contributors to file a JIRA instead.

Re: Github integration for Hadoop

[Andrew Wang]

On Thu, Oct 29, 2015 at 2:29 PM, Steve Loughran 
wrote:

>
> > On 29 Oct 2015, at 20:34, Andrew Wang  wrote:
> >
> > Has anything changed regarding the github integration since the last time
> > we discussed this? That blog post is from 2014, and we discussed
> > alternative review systems earlier in 2015.
> >
> > Colin specifically was concerned about forking the discussion between
> JIRA
> > and other places:
> >
> > http://search-hadoop.com/m/uOzYtkYxo4qazi=Re+Patch+review+process
> > http://search-hadoop.com/m/uOzYtSz7z624qazi=Re+Patch+review+process
> >
> > There are also questions about PRs leading to messy commit history with
> the
> > extra merge commits. Spark IIRC has something to linearize it again,
> which
> > seems important if we actually want to do this.
> >
> > Could someone outline the upsides of using github? I don't find the
> review
> > UI particularly great compared to Gerrit or even RB, and there's the
> merge
> > commit issue. For instance, do we think using Github would lead to more
> > contributions? Improved developer workflows? Have we re-examined
> > alternatives like Gerrit or RB as well?
>
> I've been using it for some ATS integration work
>
> For simple patches, you can do some good review cycles
>
> https://github.com/apache/spark/pull/9232
>
> For something big, well, it gets big
>
> https://github.com/apache/spark/pull/8744#discussion_r43388336
>
> ... you have to switch to the many-file view, which kind of loses some of
> the temporal ordering of comments —instead you get lots of little threads
> on individual issues. Which may scale better
>
> https://github.com/apache/spark/pull/8744/files
>
> I'd need more experience to come to a real conclusion. What I do like is
> the immediate push-to-trigger rebuild process, no need to create patches
> and submit them. That I like.
>
> I like this streamlining a lot too, but it's also something Gerrit
provides. I also prefer Gerrit's review interface to Github. Gerrit also
has nice interdiff support, which AFAICT Github does not support at all.
You can push patch revs on top of a PR, but it doesn't show interdiffs when
you force push. PRs seem designed for a merge rather than rebase workflow,
which is at odds with our existing dev workflow.

If I had my druthers, we'd focus on Gerrit rather than Github as a
successor review system. Better review, better fits our dev workflow. The
only question is JIRA integration.

Best,
Andrew

Re: Github integration for Hadoop

[Owen O'Malley]

On Thu, Oct 29, 2015 at 2:42 PM, Andrew Wang 
wrote:

>
> If I had my druthers, we'd focus on Gerrit rather than Github as a
> successor review system. Better review, better fits our dev workflow. The
> only question is JIRA integration.
>

Last time I asked, Apache Infra wasn't willing to support Gerrit and they
obviously do successfully support Github. I don't know if Gerrit is better
or worse than Github, but it really isn't an option at this point.

.. Owen

Re: Github integration for Hadoop

[Andrew Wang]

On Thu, Oct 29, 2015 at 3:29 PM, Owen O'Malley  wrote:

> On Thu, Oct 29, 2015 at 2:42 PM, Andrew Wang 
> wrote:
>
> >
> > If I had my druthers, we'd focus on Gerrit rather than Github as a
> > successor review system. Better review, better fits our dev workflow. The
> > only question is JIRA integration.
> >
>
> Last time I asked, Apache Infra wasn't willing to support Gerrit and they
> obviously do successfully support Github. I don't know if Gerrit is better
> or worse than Github, but it really isn't an option at this point.
>
> I asked INFRA about this before too, and the response was more "Later"
than "Won't Fix", to use JIRA terminology. If we pushed for it, I think it
could happen. So let's not rule Gerrit out from the get-go.

Andrew

Re: Github integration for Hadoop

[Arpit Agarwal]

On 10/29/15, 3:48 PM, "Andrew Wang"  wrote:


> If we pushed for it, I think it could happen. 

Gerrit support is a complete unknown. The community response to date supports 
Github integration. Github will appeal to new contributors as their Github 
profiles will reflect their work. I'd be interested in hearing more contributor 
opinions.



> If that's the case, we should also look at review alternatives
> like RB and Crucible.

Okay by me if the community consensus supports one of them. The fact that they 
exist but no one uses them is not a ringing endorsement.

Re: Github integration for Hadoop

[Andrew Wang]

On Thu, Oct 29, 2015 at 4:58 PM, Arpit Agarwal 
wrote:

> On 10/29/15, 3:48 PM, "Andrew Wang"  wrote:
>
>
> > If we pushed for it, I think it could happen.
>
> Gerrit support is a complete unknown. The community response to date
> supports Github integration. Github will appeal to new contributors as
> their Github profiles will reflect their work. I'd be interested in hearing
> more contributor opinions.
>
> Owen said above that he was proposing using github as a review tool, not
for code integration. So contributors wouldn't have anything showing up on
their github profiles, since we aren't directly taking PRs.

However, if we were to use GH for integration, it would be with the
auto-squash to avoid the merge commit. Would this preserve the correct
attribution?

>
> > If that's the case, we should also look at review alternatives
> > like RB and Crucible.
>
> Okay by me if the community consensus supports one of them. The fact that
> they exist but no one uses them is not a ringing endorsement.
>
> HBase uses reviewboard, as do I'm sure other Apache projects.
reviews.apache.org existed before we had github integration. I've used RB a
fair bit, and don't mind it.

Re: Github integration for Hadoop

[Allen Wittenauer]

> On Oct 29, 2015, at 5:14 PM, Andrew Wang  wrote:
> 
> However, if we were to use GH for integration, it would be with the
> auto-squash to avoid the merge commit. Would this preserve the correct
> attribution?

FWIW, Yetus *really really really* wants a single commit when it comes 
to directly pulling github PRs.  There is a very high risk of Yetus not being 
able to apply a patch generated from github if there are multiple, 
intertwinedcommits to the same tree due to how it functions.

Re: Github integration for Hadoop

[Arpit Agarwal]

On 10/29/15, 5:14 PM, "Andrew Wang"  wrote:



>On Thu, Oct 29, 2015 at 4:58 PM, Arpit Agarwal 
>wrote:
>
>> On 10/29/15, 3:48 PM, "Andrew Wang"  wrote:
>>
>>
>> > If we pushed for it, I think it could happen.
>>
>> Gerrit support is a complete unknown. The community response to date
>> supports Github integration. Github will appeal to new contributors as
>> their Github profiles will reflect their work. I'd be interested in hearing
>> more contributor opinions.
>>
>> Owen said above that he was proposing using github as a review tool, not
>for code integration. So contributors wouldn't have anything showing up on
>their github profiles, since we aren't directly taking PRs.
>
>However, if we were to use GH for integration, it would be with the
>auto-squash to avoid the merge commit. Would this preserve the correct
>attribution?

The original mail said pull request integration. Unless infra is planning on 
Gerrit integration soon it's not a practical alternative.


>>
>> > If that's the case, we should also look at review alternatives
>> > like RB and Crucible.
>>
>> Okay by me if the community consensus supports one of them. The fact that
>> they exist but no one uses them is not a ringing endorsement.
>>
>> HBase uses reviewboard, as do I'm sure other Apache projects.
>reviews.apache.org existed before we had github integration. I've used RB a
>fair bit, and don't mind it.

I could not get RB working with the Hadoop sub-projects. Would you be willing 
to try it out on a Hadoop/HDFS Jira since you have experience with it?