Re: [Gluster-infra] Jenkins and Gerrit issues today

2017-07-17 Thread Nigel Babu 
We just started using the Jenkins pipeline and its associated plugins:
https://build.gluster.org/job/nightly-master/

On Mon, Jul 17, 2017 at 6:08 PM, Michael Scherer 
wrote:

> Le vendredi 14 juillet 2017 à 13:00 +0530, Nigel Babu a écrit :
> > Hello,
> >
> > ## Highlights
> > * If you pushed a patch today or did "recheck centos", please do a
> recheck.
> > Those jobs were not triggered.
> > * Please actually verify that the jobs for your patches have started. You
> > can do that by visiting https://build.gluster.org/job/smoke/ (for
> smoke) or
> > https://build.gluster.org/job/centos6-regression/ (for regression) and
> > searching for your review. Verify that the patchset is correct.
> >
> > ## The Details
> >
> > This morning I installed critical security updates for Jenkins that
> needed
> > a restart of Jenkins.
>
> So I did look at the list of issues, and none did seems to affect
> Jenkins plugins we used, so I didn't do the upgrade. I was not sure
> however of which one did we used, so can you details a bit ?
>
> > After this restart, it appears that the Gerrit plugin
> > failed to load because of an XML error in the config file. As far as I
> > know, this error has always existed, but the newer version of the plugin
> > became more strict in xml parsing. I noticed this only about an hour so
> ago
> > and I've fixed it. Please let me know if there are further problems. Due
> to
> > this any jobs that should have been triggered since about 8:30 am this
> > morning were not triggered. Please manually do a recheck for your
> patches.
> >
> > Additionally, Ravi and Nithya pointed me to a problem where Gerrit wasn't
> > responding. We've noticed this quite often because we've configured
> Gerrit
> > to not drop idle connections. This forces us to restart Gerrit when there
> > are too many long-running idle connections. I've put a timeout of 10 mins
> > for idle connections. This issue should be sorted.
> >
> > However, Jenkins does an SSH connection with Gerrit by running `ssh
> > jenk...@review.gluster.org stream-events`. I'm not sure if this Gerrit
> > config change will cause a conflict with Jenkins, but we'll see in the
> next
> > few hours. None of the documentation explicitly points to a problem.
> >
> > ___
> > Gluster-infra mailing list
> > Gluster-infra@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-infra
>
> --
> Michael Scherer
> Sysadmin, Community Infrastructure and Platform, OSAS
>
>
>
> ___
> Gluster-infra mailing list
> Gluster-infra@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-infra
>



-- 
nigelb
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra

Re: [Gluster-infra] Jenkins and Gerrit issues today

2017-07-17 Thread Michael Scherer 
Le vendredi 14 juillet 2017 à 13:00 +0530, Nigel Babu a écrit :
> Hello,
> 
> ## Highlights
> * If you pushed a patch today or did "recheck centos", please do a recheck.
> Those jobs were not triggered.
> * Please actually verify that the jobs for your patches have started. You
> can do that by visiting https://build.gluster.org/job/smoke/ (for smoke) or
> https://build.gluster.org/job/centos6-regression/ (for regression) and
> searching for your review. Verify that the patchset is correct.
> 
> ## The Details
> 
> This morning I installed critical security updates for Jenkins that needed
> a restart of Jenkins. 

So I did look at the list of issues, and none did seems to affect
Jenkins plugins we used, so I didn't do the upgrade. I was not sure
however of which one did we used, so can you details a bit ?

> After this restart, it appears that the Gerrit plugin
> failed to load because of an XML error in the config file. As far as I
> know, this error has always existed, but the newer version of the plugin
> became more strict in xml parsing. I noticed this only about an hour so ago
> and I've fixed it. Please let me know if there are further problems. Due to
> this any jobs that should have been triggered since about 8:30 am this
> morning were not triggered. Please manually do a recheck for your patches.
> 
> Additionally, Ravi and Nithya pointed me to a problem where Gerrit wasn't
> responding. We've noticed this quite often because we've configured Gerrit
> to not drop idle connections. This forces us to restart Gerrit when there
> are too many long-running idle connections. I've put a timeout of 10 mins
> for idle connections. This issue should be sorted.
> 
> However, Jenkins does an SSH connection with Gerrit by running `ssh
> jenk...@review.gluster.org stream-events`. I'm not sure if this Gerrit
> config change will cause a conflict with Jenkins, but we'll see in the next
> few hours. None of the documentation explicitly points to a problem.
> 
> ___
> Gluster-infra mailing list
> Gluster-infra@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-infra

-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS




signature.asc
Description: This is a digitally signed message part
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra

Re: [Gluster-infra] Jenkins and Gerrit issues today

2017-07-14 Thread Nigel Babu 
Note: I've retriggered smoke/regression where appropriate for all patches
posted since the issue started.

On Fri, Jul 14, 2017 at 1:00 PM, Nigel Babu  wrote:

> Hello,
>
> ## Highlights
> * If you pushed a patch today or did "recheck centos", please do a
> recheck. Those jobs were not triggered.
> * Please actually verify that the jobs for your patches have started. You
> can do that by visiting https://build.gluster.org/job/smoke/ (for smoke)
> or https://build.gluster.org/job/centos6-regression/ (for regression) and
> searching for your review. Verify that the patchset is correct.
>
> ## The Details
>
> This morning I installed critical security updates for Jenkins that needed
> a restart of Jenkins. After this restart, it appears that the Gerrit plugin
> failed to load because of an XML error in the config file. As far as I
> know, this error has always existed, but the newer version of the plugin
> became more strict in xml parsing. I noticed this only about an hour so ago
> and I've fixed it. Please let me know if there are further problems. Due to
> this any jobs that should have been triggered since about 8:30 am this
> morning were not triggered. Please manually do a recheck for your patches.
>
> Additionally, Ravi and Nithya pointed me to a problem where Gerrit wasn't
> responding. We've noticed this quite often because we've configured Gerrit
> to not drop idle connections. This forces us to restart Gerrit when there
> are too many long-running idle connections. I've put a timeout of 10 mins
> for idle connections. This issue should be sorted.
>
> However, Jenkins does an SSH connection with Gerrit by running `ssh
> jenk...@review.gluster.org stream-events`. I'm not sure if this Gerrit
> config change will cause a conflict with Jenkins, but we'll see in the next
> few hours. None of the documentation explicitly points to a problem.
>
> --
> nigelb
>



-- 
nigelb
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra

[Gluster-infra] Jenkins and Gerrit issues today

2017-07-14 Thread Nigel Babu 
Hello,

## Highlights
* If you pushed a patch today or did "recheck centos", please do a recheck.
Those jobs were not triggered.
* Please actually verify that the jobs for your patches have started. You
can do that by visiting https://build.gluster.org/job/smoke/ (for smoke) or
https://build.gluster.org/job/centos6-regression/ (for regression) and
searching for your review. Verify that the patchset is correct.

## The Details

This morning I installed critical security updates for Jenkins that needed
a restart of Jenkins. After this restart, it appears that the Gerrit plugin
failed to load because of an XML error in the config file. As far as I
know, this error has always existed, but the newer version of the plugin
became more strict in xml parsing. I noticed this only about an hour so ago
and I've fixed it. Please let me know if there are further problems. Due to
this any jobs that should have been triggered since about 8:30 am this
morning were not triggered. Please manually do a recheck for your patches.

Additionally, Ravi and Nithya pointed me to a problem where Gerrit wasn't
responding. We've noticed this quite often because we've configured Gerrit
to not drop idle connections. This forces us to restart Gerrit when there
are too many long-running idle connections. I've put a timeout of 10 mins
for idle connections. This issue should be sorted.

However, Jenkins does an SSH connection with Gerrit by running `ssh
jenk...@review.gluster.org stream-events`. I'm not sure if this Gerrit
config change will cause a conflict with Jenkins, but we'll see in the next
few hours. None of the documentation explicitly points to a problem.

-- 
nigelb
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-infra