Re: Jenkins pre/postcommit increased from 35m to 60m+ on Friday

2016-12-13 Thread Dan Halperin
status.apache.org is an excellent resource:
http://status.apache.org/#wplaceholder_86

It does indeed look like Jenkins has been having issues for nearly a week.

On Mon, Dec 12, 2016 at 6:32 PM, Kenneth Knowles 
wrote:

> Great. That means the timestamp change I made for Travis, ported to
> Jenkins, should reveal more.
>
> Meanwhile - any known issues with Jenkins or Maven Central? Status
> dashboard for Maven Central doesn't look unhappy.
>
> On Mon, Dec 12, 2016 at 6:25 PM, Dan Halperin  >
> wrote:
>
> > From the "bad run", the Maven part took 35 minutes and presumably the
> rest
> > is Jenkins / Maven / downloading overhead.
> >
> > [INFO] 
> > 
> > [INFO] Reactor Summary:
> > [INFO]
> > [INFO] Apache Beam :: Parent .. SUCCESS [
> > 37.558 s]
> > [INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [
> > 8.172 s]
> > [INFO] Apache Beam :: SDKs  SUCCESS [
> > 11.521 s]
> > [INFO] Apache Beam :: SDKs :: Java  SUCCESS [
> > 10.801 s]
> > [INFO] Apache Beam :: SDKs :: Java :: Core  SUCCESS
> [04:04
> > min]
> > [INFO] Apache Beam :: Runners . SUCCESS [
> > 14.945 s]
> > [INFO] Apache Beam :: Runners :: Core Java  SUCCESS [
> > 44.654 s]
> > [INFO] Apache Beam :: Runners :: Direct Java .. SUCCESS
> [02:06
> > min]
> > [INFO] Apache Beam :: Runners :: Google Cloud Dataflow  SUCCESS [
> > 33.245 s]
> > [INFO] Apache Beam :: SDKs :: Java :: IO .. SUCCESS [
> > 4.047 s]
> > [INFO] Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform
> > SUCCESS [04:32 min]
> > [INFO] Apache Beam :: SDKs :: Java :: IO :: HDFS .. SUCCESS [
> > 32.009 s]
> > [INFO] Apache Beam :: SDKs :: Java :: IO :: JMS ... SUCCESS [
> > 19.006 s]
> > [INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka . SUCCESS [
> > 22.021 s]
> > [INFO] Apache Beam :: SDKs :: Java :: IO :: Kinesis ... SUCCESS [
> > 22.817 s]
> > [INFO] Apache Beam :: SDKs :: Java :: IO :: MongoDB ... SUCCESS [
> > 27.276 s]
> > [INFO] Apache Beam :: SDKs :: Java :: IO :: JDBC .. SUCCESS [
> > 23.662 s]
> > [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes  SUCCESS [
> > 3.115 s]
> > [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter
> > SUCCESS [ 17.818 s]
> > [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples
> > SUCCESS [ 15.111 s]
> > [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples -
> > Java 8 SUCCESS [ 24.477 s]
> > [INFO] Apache Beam :: SDKs :: Java :: Extensions .. SUCCESS [
> > 5.759 s]
> > [INFO] Apache Beam :: SDKs :: Java :: Extensions :: Join library
> > SUCCESS [ 22.293 s]
> > [INFO] Apache Beam :: SDKs :: Java :: Extensions :: Sorter  SUCCESS [
> > 31.145 s]
> > [INFO] Apache Beam :: SDKs :: Java :: Java 8 Tests  SUCCESS [
> > 8.033 s]
> > [INFO] Apache Beam :: Runners :: Flink  SUCCESS [
> > 6.503 s]
> > [INFO] Apache Beam :: Runners :: Flink :: Core  SUCCESS [
> > 43.593 s]
> > [INFO] Apache Beam :: Runners :: Flink :: Examples  SUCCESS [
> > 20.006 s]
> > [INFO] Apache Beam :: Runners :: Spark  SUCCESS
> [03:10
> > min]
> > [INFO] Apache Beam :: Runners :: Apex . SUCCESS
> [01:03
> > min]
> > [INFO] Apache Beam :: Examples  SUCCESS [
> > 8.124 s]
> > [INFO] Apache Beam :: Examples :: Java  SUCCESS
> [05:29
> > min]
> > [INFO] Apache Beam :: Examples :: Java 8 .. SUCCESS [
> > 19.005 s]
> > [INFO] 
> > 
> > [INFO] BUILD SUCCESS
> > [INFO] 
> > 
> > [INFO] Total time: 34:30 min
> > [INFO] Finished at: 2016-12-09T18:50:49+00:00
> > [INFO] Final Memory: 196M/1051M
> > [INFO] 
> > 
> >
> >
> >
> > On Mon, Dec 12, 2016 at 5:36 PM, Kenneth Knowles  >
> > wrote:
> >
> > > Hi all,
> > >
> > > We have a huge Jenkins backlog, surely exacerbated by the fact that our
> > > test time (precommit and postcommit mvn install) has roughly doubled in
> > the
> > > last few days.
> > >
> > > Here's the quick link to the trend:
> > > https://builds.apache.org/view/Beam/job/beam_PostCommit_
> > Java_MavenInstall/
> > > buildTimeTrend
> > >
> > > Good 33m build at 2016-12-09 02:42:
> > > https://builds.apache.org/view/Beam/job/beam_PostCommit_
> > > Java_MavenInstall/2041/
> > >
> > > Bad 59m build at  2016-12-09 18:00 (trigger by timer):
> > > https://builds.apache.org/view/Beam/job/beam_PostCommit_
> > > Java_MavenInstall/2048/
> > >
> > > There are a couple middling runs in between that I can't place
> > immediately
> > > into either bucket. I'm still l

Re: Jenkins pre/postcommit increased from 35m to 60m+ on Friday

2016-12-12 Thread Kenneth Knowles
Great. That means the timestamp change I made for Travis, ported to
Jenkins, should reveal more.

Meanwhile - any known issues with Jenkins or Maven Central? Status
dashboard for Maven Central doesn't look unhappy.

On Mon, Dec 12, 2016 at 6:25 PM, Dan Halperin 
wrote:

> From the "bad run", the Maven part took 35 minutes and presumably the rest
> is Jenkins / Maven / downloading overhead.
>
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Beam :: Parent .. SUCCESS [
> 37.558 s]
> [INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [
> 8.172 s]
> [INFO] Apache Beam :: SDKs  SUCCESS [
> 11.521 s]
> [INFO] Apache Beam :: SDKs :: Java  SUCCESS [
> 10.801 s]
> [INFO] Apache Beam :: SDKs :: Java :: Core  SUCCESS [04:04
> min]
> [INFO] Apache Beam :: Runners . SUCCESS [
> 14.945 s]
> [INFO] Apache Beam :: Runners :: Core Java  SUCCESS [
> 44.654 s]
> [INFO] Apache Beam :: Runners :: Direct Java .. SUCCESS [02:06
> min]
> [INFO] Apache Beam :: Runners :: Google Cloud Dataflow  SUCCESS [
> 33.245 s]
> [INFO] Apache Beam :: SDKs :: Java :: IO .. SUCCESS [
> 4.047 s]
> [INFO] Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform
> SUCCESS [04:32 min]
> [INFO] Apache Beam :: SDKs :: Java :: IO :: HDFS .. SUCCESS [
> 32.009 s]
> [INFO] Apache Beam :: SDKs :: Java :: IO :: JMS ... SUCCESS [
> 19.006 s]
> [INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka . SUCCESS [
> 22.021 s]
> [INFO] Apache Beam :: SDKs :: Java :: IO :: Kinesis ... SUCCESS [
> 22.817 s]
> [INFO] Apache Beam :: SDKs :: Java :: IO :: MongoDB ... SUCCESS [
> 27.276 s]
> [INFO] Apache Beam :: SDKs :: Java :: IO :: JDBC .. SUCCESS [
> 23.662 s]
> [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes  SUCCESS [
> 3.115 s]
> [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter
> SUCCESS [ 17.818 s]
> [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples
> SUCCESS [ 15.111 s]
> [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples -
> Java 8 SUCCESS [ 24.477 s]
> [INFO] Apache Beam :: SDKs :: Java :: Extensions .. SUCCESS [
> 5.759 s]
> [INFO] Apache Beam :: SDKs :: Java :: Extensions :: Join library
> SUCCESS [ 22.293 s]
> [INFO] Apache Beam :: SDKs :: Java :: Extensions :: Sorter  SUCCESS [
> 31.145 s]
> [INFO] Apache Beam :: SDKs :: Java :: Java 8 Tests  SUCCESS [
> 8.033 s]
> [INFO] Apache Beam :: Runners :: Flink  SUCCESS [
> 6.503 s]
> [INFO] Apache Beam :: Runners :: Flink :: Core  SUCCESS [
> 43.593 s]
> [INFO] Apache Beam :: Runners :: Flink :: Examples  SUCCESS [
> 20.006 s]
> [INFO] Apache Beam :: Runners :: Spark  SUCCESS [03:10
> min]
> [INFO] Apache Beam :: Runners :: Apex . SUCCESS [01:03
> min]
> [INFO] Apache Beam :: Examples  SUCCESS [
> 8.124 s]
> [INFO] Apache Beam :: Examples :: Java  SUCCESS [05:29
> min]
> [INFO] Apache Beam :: Examples :: Java 8 .. SUCCESS [
> 19.005 s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 34:30 min
> [INFO] Finished at: 2016-12-09T18:50:49+00:00
> [INFO] Final Memory: 196M/1051M
> [INFO] 
> 
>
>
>
> On Mon, Dec 12, 2016 at 5:36 PM, Kenneth Knowles 
> wrote:
>
> > Hi all,
> >
> > We have a huge Jenkins backlog, surely exacerbated by the fact that our
> > test time (precommit and postcommit mvn install) has roughly doubled in
> the
> > last few days.
> >
> > Here's the quick link to the trend:
> > https://builds.apache.org/view/Beam/job/beam_PostCommit_
> Java_MavenInstall/
> > buildTimeTrend
> >
> > Good 33m build at 2016-12-09 02:42:
> > https://builds.apache.org/view/Beam/job/beam_PostCommit_
> > Java_MavenInstall/2041/
> >
> > Bad 59m build at  2016-12-09 18:00 (trigger by timer):
> > https://builds.apache.org/view/Beam/job/beam_PostCommit_
> > Java_MavenInstall/2048/
> >
> > There are a couple middling runs in between that I can't place
> immediately
> > into either bucket. I'm still looking into this, but it is a
> clicky-clicky
> > process that hasn't yielded anything yet.
> >
> > If anyone knows something, please share your insight.
> >
> > Kenn
> >
>


Re: Jenkins pre/postcommit increased from 35m to 60m+ on Friday

2016-12-12 Thread Dan Halperin
>From the "bad run", the Maven part took 35 minutes and presumably the rest
is Jenkins / Maven / downloading overhead.

[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Beam :: Parent .. SUCCESS [ 37.558 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  8.172 s]
[INFO] Apache Beam :: SDKs  SUCCESS [ 11.521 s]
[INFO] Apache Beam :: SDKs :: Java  SUCCESS [ 10.801 s]
[INFO] Apache Beam :: SDKs :: Java :: Core  SUCCESS [04:04 min]
[INFO] Apache Beam :: Runners . SUCCESS [ 14.945 s]
[INFO] Apache Beam :: Runners :: Core Java  SUCCESS [ 44.654 s]
[INFO] Apache Beam :: Runners :: Direct Java .. SUCCESS [02:06 min]
[INFO] Apache Beam :: Runners :: Google Cloud Dataflow  SUCCESS [ 33.245 s]
[INFO] Apache Beam :: SDKs :: Java :: IO .. SUCCESS [  4.047 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform
SUCCESS [04:32 min]
[INFO] Apache Beam :: SDKs :: Java :: IO :: HDFS .. SUCCESS [ 32.009 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: JMS ... SUCCESS [ 19.006 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka . SUCCESS [ 22.021 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kinesis ... SUCCESS [ 22.817 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: MongoDB ... SUCCESS [ 27.276 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: JDBC .. SUCCESS [ 23.662 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes  SUCCESS [  3.115 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter
SUCCESS [ 17.818 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples
SUCCESS [ 15.111 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples -
Java 8 SUCCESS [ 24.477 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SUCCESS [  5.759 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Join library
SUCCESS [ 22.293 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Sorter  SUCCESS [ 31.145 s]
[INFO] Apache Beam :: SDKs :: Java :: Java 8 Tests  SUCCESS [  8.033 s]
[INFO] Apache Beam :: Runners :: Flink  SUCCESS [  6.503 s]
[INFO] Apache Beam :: Runners :: Flink :: Core  SUCCESS [ 43.593 s]
[INFO] Apache Beam :: Runners :: Flink :: Examples  SUCCESS [ 20.006 s]
[INFO] Apache Beam :: Runners :: Spark  SUCCESS [03:10 min]
[INFO] Apache Beam :: Runners :: Apex . SUCCESS [01:03 min]
[INFO] Apache Beam :: Examples  SUCCESS [  8.124 s]
[INFO] Apache Beam :: Examples :: Java  SUCCESS [05:29 min]
[INFO] Apache Beam :: Examples :: Java 8 .. SUCCESS [ 19.005 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 34:30 min
[INFO] Finished at: 2016-12-09T18:50:49+00:00
[INFO] Final Memory: 196M/1051M
[INFO] 



On Mon, Dec 12, 2016 at 5:36 PM, Kenneth Knowles 
wrote:

> Hi all,
>
> We have a huge Jenkins backlog, surely exacerbated by the fact that our
> test time (precommit and postcommit mvn install) has roughly doubled in the
> last few days.
>
> Here's the quick link to the trend:
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/
> buildTimeTrend
>
> Good 33m build at 2016-12-09 02:42:
> https://builds.apache.org/view/Beam/job/beam_PostCommit_
> Java_MavenInstall/2041/
>
> Bad 59m build at  2016-12-09 18:00 (trigger by timer):
> https://builds.apache.org/view/Beam/job/beam_PostCommit_
> Java_MavenInstall/2048/
>
> There are a couple middling runs in between that I can't place immediately
> into either bucket. I'm still looking into this, but it is a clicky-clicky
> process that hasn't yielded anything yet.
>
> If anyone knows something, please share your insight.
>
> Kenn
>