Re: Including Apex runner in Beam tutorial at Strata - Singapore

[Sandeep Deshmukh]

Thanks Thomas for the suggestions. I will work on 2nd and 3rd part.

Connecting with  Tyler Akidau offline.

Regards,
Sandeep

On Wed, Nov 16, 2016 at 4:40 AM, Thomas Weise  wrote:

> The runner currently only executes in embedded mode and I'm not sure that
> will change prior to Strata due to dependency on next Apex core release.
>
> I would suggest to aim for the following:
>
>
>- Create a sample wordcount project that produces the packaging that is
>required to launch the pipeline on YARN (I can take this up).
>- Create the instructions for folks to setup a YARN cluster locally. I
>think a good option for the demo would be to use the DataTorrent
> sandbox.
>- Create tutorial documentation that shows how to alter the example,
>launch it using the Apex CLI and also how to observe the application
>through the UI (comes with the sandbox).
>
>
> Thanks,
> Thomas
>
>
>
>
>
> On Tue, Nov 15, 2016 at 7:07 PM, Tyler Akidau 
> wrote:
>
> > Yes, I'll be giving the tutorial in Singapore w/ Dan Halperin. We'd be
> > happy to include Apex in the demos as part of that. Let's sync up offline
> > about what that will entail.
> >
> > -Tyler
> >
> >
> > On Tue, Nov 15, 2016 at 10:02 AM Davor Bonaci 
> > wrote:
> >
> > > Hi Sandeep,
> > > It would be great to include the Apex runner as a part of any tutorial
> > > going forward. I suspect we'll have the 0.4.0-incubating release
> > completed
> > > just before Strata Singapore, which will the first release with the
> Apex
> > > runner, so that aligns quite nicely.
> > >
> > > Are you planning to attend Strata Singapore? If so, I'd encourage you
> to
> > > reach out to Tyler Akidau offline, who's leading the tutorial on this
> > > conference.
> > >
> > > Davor
> > >
> > > On Tue, Nov 15, 2016 at 7:04 AM, Jean-Baptiste Onofré  >
> > > wrote:
> > >
> > > > Hi Sandeep,
> > > >
> > > > Great news !
> > > >
> > > > Yes, you can definitely do a demo using the Apex runner. It's what
> Dan
> > > and
> > > > I are also planning during ApacheCon this week: same Wordcount
> example
> > > > running on different execution engines.
> > > >
> > > > Maybe this blog could help you to prepare the demo:
> > > > http://blog.nanthrax.net/2016/08/apache-beam-in-action-same-
> > > > code-several-execution-engines/
> > > >
> > > > By the way, I will propose a PR to "merge" those blog to Beam
> website.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > >
> > > > On 11/15/2016 04:00 PM, Sandeep Deshmukh wrote:
> > > >
> > > >> Dear Beam Community,
> > > >>
> > > >> There is a Beam tutorial in Strata-Singapore. I would like to
> explore
> > > >> possibility of including the Apex runner as a part of that tutorial.
> > As
> > > >> Apex runner is recently merged into master branch of Beam, it would
> be
> > > of
> > > >> interest to many people.
> > > >>
> > > >> Please let us know if we can do so. I can accordingly work on the
> > same.
> > > >>
> > > >> Regards,
> > > >> Sandeep
> > > >>
> > > >>
> > > > --
> > > > Jean-Baptiste Onofré
> > > > jbono...@apache.org
> > > > http://blog.nanthrax.net
> > > > Talend - http://www.talend.com
> > > >
> > >
> >
>

Re: Beam dinner tonight

[Yonglong Dai]

Cool. Enjoy your time.
Yonglong

2016-11-15 15:23 GMT-08:00 Dan Halperin :

> Hi Yonglong,
>
> First, sorry for spamming the dev list. It may not be the best thing to
> spam the entire community when 10 of us are at a conference ... but since
> Beam is a new project, this conference is the first time we're all meeting
> each other.
>
> Second, no we are not in CA currently -- we are at Apache Big Data in
> Sevilla, Spain
> . But
> keep
> your eyes peeled for Beam-related meetups and similar get-togethers in the
> US soon.
>
> Thanks,
> Dan
>
> On Wed, Nov 16, 2016 at 12:09 AM, Yonglong Dai 
> wrote:
>
> > You guys are enviable. Are you in the CA?
> > Thanks
> >
>

Re: Jenkins skipping PreCommit for a PR caused build failures on master.

[Dan Halperin]

The command is apache-rat:check, not rat:check.
http://creadur.apache.org/rat/apache-rat-plugin/

rat:check is using the (very old) Maven2 plugin for RAT from the now
defunct org.codehaus:
https://mvnrepository.com/artifact/org.codehaus.mojo/rat-maven-plugin/1.0-alpha-3

Dan

On Wed, Nov 16, 2016 at 12:22 AM, Kenneth Knowles 
wrote:

> I honestly do not understand what is going on with our RAT set up. I'd also
> love some docs on it.
>
> Jenkins fails on https://github.com/apache/incubator-beam/pull/1307 saying
> 1 file with unapproved license.
>  - There are no new files or changes to headers. So I assumed it was broken
> on master and went to repro and got curious results.
>
> `mvn rat:check` on master after `git clean -d -f -x` fails with 6
> unapproved licenses in the parent.
>  - But the report says there are 6 Apache Licensed files and 1640 unknown
> licenses
>  - Internally consistent, the thing that occurs 6 times in the report is 6
> AL files, such as .travis.yml and pom.xml, which are all good.
>  - The report lists excluded files as having unknown license or B for
> binary, hence every .git thing, and the report is huge.
>
> `mvn rat:check -pl sdks/java/core` fails saying there are 662 unapproved
> licenses.
>  - But, again, there are 662 good Apache License files and one unknown.
>  - And the unknown is user.avsc, which we explicitly exclude in our base
> configuration.
>
> None of these results is correct. The latter two seem like the plugin
> actually just has a bit flipped and also is not receiving our
> configuration.
>
> On Tue, Nov 15, 2016 at 3:00 PM, Amit Sela  wrote:
>
> > Jenkins had an outage which caused it to skip PreCommit execution for PR
> > #1332 .
> >
> > I reviewed this PR, and merged it.
> > I failed to notice that Jenkins skipped it, and so I merged it with a
> > missing license and checkstyle errors.
> >
> > This spread-out across 3 PRs and a ticket now so I'll summarise here:
> >
> >- Participating PRs: #1362
> > #1364
> > #1332
> >
> >- Jenkins outage skipped (at least 1) PreCommit execution.
> >- A simple mvn install/verify won't execute checkstyle anymore, use:
> >"mvn -Prelease clean verify"
> >- Kenneth Knowles mentions we might have an issue with our
> configuration
> >as "mvn rat:check" fails on master (even after rollback)
> >
> > Let's continue in this thread.
> >
> > Thanks,
> > Amit
> >
>

Re: Beam dinner tonight

[Dan Halperin]

Hi Yonglong,

First, sorry for spamming the dev list. It may not be the best thing to
spam the entire community when 10 of us are at a conference ... but since
Beam is a new project, this conference is the first time we're all meeting
each other.

Second, no we are not in CA currently -- we are at Apache Big Data in
Sevilla, Spain
. But keep
your eyes peeled for Beam-related meetups and similar get-togethers in the
US soon.

Thanks,
Dan

On Wed, Nov 16, 2016 at 12:09 AM, Yonglong Dai  wrote:

> You guys are enviable. Are you in the CA?
> Thanks
>

Re: Including Apex runner in Beam tutorial at Strata - Singapore

[Thomas Weise]

The runner currently only executes in embedded mode and I'm not sure that
will change prior to Strata due to dependency on next Apex core release.

I would suggest to aim for the following:


   - Create a sample wordcount project that produces the packaging that is
   required to launch the pipeline on YARN (I can take this up).
   - Create the instructions for folks to setup a YARN cluster locally. I
   think a good option for the demo would be to use the DataTorrent sandbox.
   - Create tutorial documentation that shows how to alter the example,
   launch it using the Apex CLI and also how to observe the application
   through the UI (comes with the sandbox).


Thanks,
Thomas





On Tue, Nov 15, 2016 at 7:07 PM, Tyler Akidau 
wrote:

> Yes, I'll be giving the tutorial in Singapore w/ Dan Halperin. We'd be
> happy to include Apex in the demos as part of that. Let's sync up offline
> about what that will entail.
>
> -Tyler
>
>
> On Tue, Nov 15, 2016 at 10:02 AM Davor Bonaci 
> wrote:
>
> > Hi Sandeep,
> > It would be great to include the Apex runner as a part of any tutorial
> > going forward. I suspect we'll have the 0.4.0-incubating release
> completed
> > just before Strata Singapore, which will the first release with the Apex
> > runner, so that aligns quite nicely.
> >
> > Are you planning to attend Strata Singapore? If so, I'd encourage you to
> > reach out to Tyler Akidau offline, who's leading the tutorial on this
> > conference.
> >
> > Davor
> >
> > On Tue, Nov 15, 2016 at 7:04 AM, Jean-Baptiste Onofré 
> > wrote:
> >
> > > Hi Sandeep,
> > >
> > > Great news !
> > >
> > > Yes, you can definitely do a demo using the Apex runner. It's what Dan
> > and
> > > I are also planning during ApacheCon this week: same Wordcount example
> > > running on different execution engines.
> > >
> > > Maybe this blog could help you to prepare the demo:
> > > http://blog.nanthrax.net/2016/08/apache-beam-in-action-same-
> > > code-several-execution-engines/
> > >
> > > By the way, I will propose a PR to "merge" those blog to Beam website.
> > >
> > > Regards
> > > JB
> > >
> > >
> > > On 11/15/2016 04:00 PM, Sandeep Deshmukh wrote:
> > >
> > >> Dear Beam Community,
> > >>
> > >> There is a Beam tutorial in Strata-Singapore. I would like to explore
> > >> possibility of including the Apex runner as a part of that tutorial.
> As
> > >> Apex runner is recently merged into master branch of Beam, it would be
> > of
> > >> interest to many people.
> > >>
> > >> Please let us know if we can do so. I can accordingly work on the
> same.
> > >>
> > >> Regards,
> > >> Sandeep
> > >>
> > >>
> > > --
> > > Jean-Baptiste Onofré
> > > jbono...@apache.org
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
>

Re: Beam dinner tonight

[Yonglong Dai]

You guys are enviable. Are you in the CA?
Thanks

Re: Build failed in Jenkins: beam_PostCommit_MavenVerify #1827

[Jason Kuster]

I opened https://issues.apache.org/jira/browse/BEAM-983

On Tue, Nov 15, 2016 at 10:29 AM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See  MavenVerify/1827/changes>
>
> Changes:
>
> [ansela] [BEAM-891] fix build occasionally fails on
> IndexOutOfBoundsException.
>
> --
> [...truncated 5 lines...]
> Building remotely on beam3 (beam) in workspace  job/beam_PostCommit_MavenVerify/ws/>
>  > git rev-parse --is-inside-work-tree # timeout=10
> Fetching changes from the remote Git repository
>  > git config remote.origin.url https://github.com/apache/
> incubator-beam.git # timeout=10
> Fetching upstream changes from https://github.com/apache/
> incubator-beam.git
>  > git --version # timeout=10
>  > git -c core.askpass=true fetch --tags --progress
> https://github.com/apache/incubator-beam.git +refs/heads/*:refs/remotes/
> origin/*
>  > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
>  > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
> Checking out Revision 503f26f448ea9f46fcfcdd46e60cba80e4844e28
> (refs/remotes/origin/master)
>  > git config core.sparsecheckout # timeout=10
>  > git checkout -f 503f26f448ea9f46fcfcdd46e60cba80e4844e28
>  > git rev-list 47646d641848cae9d7adefa6dd537a10cade7b1b # timeout=10
> [EnvInject] - Executing scripts and injecting environment variables after
> the SCM step.
> [EnvInject] - Injecting as environment variables the properties content
> SPARK_LOCAL_IP=127.0.0.1
>
> [EnvInject] - Variables injected successfully.
> Parsing POMs
> Modules changed, recalculating dependency graph
> Established TCP socket on 35235
> maven32-agent.jar already up to date
> maven32-interceptor.jar already up to date
> maven3-interceptor-commons.jar already up to date
> [beam_PostCommit_MavenVerify] $ /home/jenkins/tools/java/latest1.8/bin/java
> -Xmx2g -Xms256m -XX:MaxPermSize=512m -cp /home/jenkins/jenkins-slave/
> maven32-agent.jar:/home/jenkins/tools/maven/apache-
> maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/
> jenkins/tools/maven/apache-maven-3.3.3/conf/logging 
> jenkins.maven3.agent.Maven32Main
> /home/jenkins/tools/maven/apache-maven-3.3.3 
> /home/jenkins/jenkins-slave/slave.jar
> /home/jenkins/jenkins-slave/maven32-interceptor.jar
> /home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 35235
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=512m; support was removed in 8.0
> <===[JENKINS REMOTING CAPACITY]===>   channel started
> Executing Maven:  -B -f  MavenVerify/ws/pom.xml> -Dmaven.repo.local= builds.apache.org/job/beam_PostCommit_MavenVerify/ws/.repository> -B -e
> -P release,dataflow-runner -DrepoToken= clean install coveralls:report
> -DskipITs=false -DintegrationTestPipelineOptions=[ 
> "--project=apache-beam-testing",
> "--tempRoot=gs://temp-storage-for-end-to-end-tests",
> "--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]
> [INFO] Error stacktraces are turned on.
> [INFO] Scanning for projects...
> [INFO] 
> 
> [INFO] Reactor Build Order:
> [INFO]
> [INFO] Apache Beam :: Parent
> [INFO] Apache Beam :: SDKs :: Java :: Build Tools
> [INFO] Apache Beam :: SDKs
> [INFO] Apache Beam :: SDKs :: Java
> [INFO] Apache Beam :: SDKs :: Java :: Core
> [INFO] Apache Beam :: Runners
> [INFO] Apache Beam :: Runners :: Core Java
> [INFO] Apache Beam :: Runners :: Direct Java
> [INFO] Apache Beam :: Runners :: Google Cloud Dataflow
> [INFO] Apache Beam :: SDKs :: Java :: IO
> [INFO] Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform
> [INFO] Apache Beam :: SDKs :: Java :: IO :: HDFS
> [INFO] Apache Beam :: SDKs :: Java :: IO :: JMS
> [INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka
> [INFO] Apache Beam :: SDKs :: Java :: IO :: Kinesis
> [INFO] Apache Beam :: SDKs :: Java :: IO :: MongoDB
> [INFO] Apache Beam :: SDKs :: Java :: IO :: JDBC
> [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes
> [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter
> [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples
> [INFO] Apache Beam :: SDKs :: Java :: Extensions
> [INFO] Apache Beam :: SDKs :: Java :: Extensions :: Join library
> [INFO] Apache Beam :: SDKs :: Java :: Extensions :: Sorter
> [INFO] Apache Beam :: SDKs :: Java :: Microbenchmarks
> [INFO] Apache Beam :: SDKs :: Java :: Java 8 Tests
> [INFO] Apache Beam :: Runners :: Flink
> [INFO] Apache Beam :: Runners :: Flink :: Core
> [INFO] Apache Beam :: Runners :: Flink :: Examples
> [INFO] Apache Beam :: Runners :: Spark
> [INFO] Apache Beam :: Runners :: Apex
> [INFO] Apache Beam :: Examples
> [INFO] Apache Beam :: Examples :: Java
> [INFO] Apache Beam :: Examples :: Java 8
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is
> missing, 

Re: Including Apex runner in Beam tutorial at Strata - Singapore

[Sandeep Deshmukh]

Thanks JB. I will get started on this.

Regards,
Sandeep
On 15-Nov-2016 8:34 PM, "Jean-Baptiste Onofré"  wrote:

> Hi Sandeep,
>
> Great news !
>
> Yes, you can definitely do a demo using the Apex runner. It's what Dan and
> I are also planning during ApacheCon this week: same Wordcount example
> running on different execution engines.
>
> Maybe this blog could help you to prepare the demo:
> http://blog.nanthrax.net/2016/08/apache-beam-in-action-same-
> code-several-execution-engines/
>
> By the way, I will propose a PR to "merge" those blog to Beam website.
>
> Regards
> JB
>
> On 11/15/2016 04:00 PM, Sandeep Deshmukh wrote:
>
>> Dear Beam Community,
>>
>> There is a Beam tutorial in Strata-Singapore. I would like to explore
>> possibility of including the Apex runner as a part of that tutorial. As
>> Apex runner is recently merged into master branch of Beam, it would be of
>> interest to many people.
>>
>> Please let us know if we can do so. I can accordingly work on the same.
>>
>> Regards,
>> Sandeep
>>
>>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Configuring Jenkins

[Robert Bradshaw]

This is great; thanks for doing this!

On Tue, Nov 15, 2016 at 6:43 AM, Dan Halperin
 wrote:
> Seems phenomenal!
>
> Reading between the lines of your email, it sounds like changes to Jenkins
> configuration will not actually be exercised on the PR that makes them. So,
> we still need to work out a process of how we test changes that would
> affect Jenkins config.
>
> (That does not take away from the fact that DSL is a vast improvement!)
> Dan
>
> On Tue, Nov 15, 2016 at 12:52 PM, Aljoscha Krettek 
> wrote:
>
>> +1 I like this a lot!
>>
>> On Tue, 15 Nov 2016 at 10:37 Jean-Baptiste Onofré  wrote:
>>
>> > Fantastic Davor !
>> >
>> > I like this approach, I gonna take a deeper look.
>> >
>> > Thanks !
>> >
>> > Regards
>> > JB
>> >
>> > On 11/15/2016 10:01 AM, Davor Bonaci wrote:
>> > > Hi everybody,
>> > > As I'm sure everybody knows, we use Apache's Jenkins instance for all
>> our
>> > > testing, including pre-commit, post-commit, nightly snapshot, etc.
>> > (Travis
>> > > CI is a backup system and recommended for individual forks only.)
>> > >
>> > > Managing Jenkins projects has been a big pain point so far. Among other
>> > > reasons, only a few of us have access to configure it, way too few of
>> us
>> > > have visibility into what those jobs do, and nobody has any visibility
>> > into
>> > > changes being made or an opportunity to comment on them.
>> > >
>> > > Well, not any more! I was playing a little bit with Jenkins DSL plugin
>> > and
>> > > was able to move our configuration out of Jenkins and into the git
>> > > repository. I've done it as a proof of concept for the website
>> repository
>> > > only [1], but Jason is planning on extending that work to the main
>> > > repository. Look for a PR shortly!
>> > >
>> > > Going forward, anyone can see what our Jenkins jobs are doing, and
>> anyone
>> > > can add new jobs or improve existing ones by simply proposing a pull
>> > > request to change the configuration. Finally, the project maintains a
>> > > history in source repository, instead of direct changes without much
>> > > accountability.
>> > >
>> > > How this works? There's a "seed" job that periodically applies
>> > > configuration specified in the source repository into Jenkins.
>> Currently,
>> > > this happens once per day. If you modify the configuration in the
>> source
>> > > repository, it will be applied within 24 hours. If you, however, modify
>> > the
>> > > configuration in Jenkins directly, it will revert back to whatever is
>> > > specified in the code repository also within 24 hours.
>> > >
>> > > How to understand Jenkins DSL? There are many resources available; I've
>> > > found Jenkins Job DSL API [2] particularly helpful.
>> > >
>> > > I hope you are excited to have this feature available to us! If you
>> have
>> > > any thoughts on improving this further, please comment. Thanks!
>> > >
>> > > Davor
>> > >
>> > > [1] https://github.com/apache/incubator-beam-site/pull/80
>> > > [2] https://jenkinsci.github.io/job-dsl-plugin/
>> > >
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbono...@apache.org
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>> >
>>

Re: Including Apex runner in Beam tutorial at Strata - Singapore

[Jean-Baptiste Onofré]

Hi Sandeep,

Great news !

Yes, you can definitely do a demo using the Apex runner. It's what Dan 
and I are also planning during ApacheCon this week: same Wordcount 
example running on different execution engines.


Maybe this blog could help you to prepare the demo: 
http://blog.nanthrax.net/2016/08/apache-beam-in-action-same-code-several-execution-engines/


By the way, I will propose a PR to "merge" those blog to Beam website.

Regards
JB

On 11/15/2016 04:00 PM, Sandeep Deshmukh wrote:

Dear Beam Community,

There is a Beam tutorial in Strata-Singapore. I would like to explore
possibility of including the Apex runner as a part of that tutorial. As
Apex runner is recently merged into master branch of Beam, it would be of
interest to many people.

Please let us know if we can do so. I can accordingly work on the same.

Regards,
Sandeep



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Including Apex runner in Beam tutorial at Strata - Singapore

[Sandeep Deshmukh]

Dear Beam Community,

There is a Beam tutorial in Strata-Singapore. I would like to explore
possibility of including the Apex runner as a part of that tutorial. As
Apex runner is recently merged into master branch of Beam, it would be of
interest to many people.

Please let us know if we can do so. I can accordingly work on the same.

Regards,
Sandeep

Re: Configuring Jenkins

[Dan Halperin]

Seems phenomenal!

Reading between the lines of your email, it sounds like changes to Jenkins
configuration will not actually be exercised on the PR that makes them. So,
we still need to work out a process of how we test changes that would
affect Jenkins config.

(That does not take away from the fact that DSL is a vast improvement!)
Dan

On Tue, Nov 15, 2016 at 12:52 PM, Aljoscha Krettek 
wrote:

> +1 I like this a lot!
>
> On Tue, 15 Nov 2016 at 10:37 Jean-Baptiste Onofré  wrote:
>
> > Fantastic Davor !
> >
> > I like this approach, I gonna take a deeper look.
> >
> > Thanks !
> >
> > Regards
> > JB
> >
> > On 11/15/2016 10:01 AM, Davor Bonaci wrote:
> > > Hi everybody,
> > > As I'm sure everybody knows, we use Apache's Jenkins instance for all
> our
> > > testing, including pre-commit, post-commit, nightly snapshot, etc.
> > (Travis
> > > CI is a backup system and recommended for individual forks only.)
> > >
> > > Managing Jenkins projects has been a big pain point so far. Among other
> > > reasons, only a few of us have access to configure it, way too few of
> us
> > > have visibility into what those jobs do, and nobody has any visibility
> > into
> > > changes being made or an opportunity to comment on them.
> > >
> > > Well, not any more! I was playing a little bit with Jenkins DSL plugin
> > and
> > > was able to move our configuration out of Jenkins and into the git
> > > repository. I've done it as a proof of concept for the website
> repository
> > > only [1], but Jason is planning on extending that work to the main
> > > repository. Look for a PR shortly!
> > >
> > > Going forward, anyone can see what our Jenkins jobs are doing, and
> anyone
> > > can add new jobs or improve existing ones by simply proposing a pull
> > > request to change the configuration. Finally, the project maintains a
> > > history in source repository, instead of direct changes without much
> > > accountability.
> > >
> > > How this works? There's a "seed" job that periodically applies
> > > configuration specified in the source repository into Jenkins.
> Currently,
> > > this happens once per day. If you modify the configuration in the
> source
> > > repository, it will be applied within 24 hours. If you, however, modify
> > the
> > > configuration in Jenkins directly, it will revert back to whatever is
> > > specified in the code repository also within 24 hours.
> > >
> > > How to understand Jenkins DSL? There are many resources available; I've
> > > found Jenkins Job DSL API [2] particularly helpful.
> > >
> > > I hope you are excited to have this feature available to us! If you
> have
> > > any thoughts on improving this further, please comment. Thanks!
> > >
> > > Davor
> > >
> > > [1] https://github.com/apache/incubator-beam-site/pull/80
> > > [2] https://jenkinsci.github.io/job-dsl-plugin/
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>

Re: Configuring Jenkins

[Daniel Kulp]

This is very cool….. I definitely need to look at this a bit as there a few 
there projects that would love to have something similar.  :)

Dan



> On Nov 15, 2016, at 4:01 AM, Davor Bonaci  wrote:
> 
> Hi everybody,
> As I'm sure everybody knows, we use Apache's Jenkins instance for all our
> testing, including pre-commit, post-commit, nightly snapshot, etc. (Travis
> CI is a backup system and recommended for individual forks only.)
> 
> Managing Jenkins projects has been a big pain point so far. Among other
> reasons, only a few of us have access to configure it, way too few of us
> have visibility into what those jobs do, and nobody has any visibility into
> changes being made or an opportunity to comment on them.
> 
> Well, not any more! I was playing a little bit with Jenkins DSL plugin and
> was able to move our configuration out of Jenkins and into the git
> repository. I've done it as a proof of concept for the website repository
> only [1], but Jason is planning on extending that work to the main
> repository. Look for a PR shortly!
> 
> Going forward, anyone can see what our Jenkins jobs are doing, and anyone
> can add new jobs or improve existing ones by simply proposing a pull
> request to change the configuration. Finally, the project maintains a
> history in source repository, instead of direct changes without much
> accountability.
> 
> How this works? There's a "seed" job that periodically applies
> configuration specified in the source repository into Jenkins. Currently,
> this happens once per day. If you modify the configuration in the source
> repository, it will be applied within 24 hours. If you, however, modify the
> configuration in Jenkins directly, it will revert back to whatever is
> specified in the code repository also within 24 hours.
> 
> How to understand Jenkins DSL? There are many resources available; I've
> found Jenkins Job DSL API [2] particularly helpful.
> 
> I hope you are excited to have this feature available to us! If you have
> any thoughts on improving this further, please comment. Thanks!
> 
> Davor
> 
> [1] https://github.com/apache/incubator-beam-site/pull/80
> [2] https://jenkinsci.github.io/job-dsl-plugin/

-- 
Daniel Kulp
dk...@apache.org - http://dankulp.com/blog
Talend Community Coder - http://coders.talend.com

Configuring Jenkins

[Davor Bonaci]

Hi everybody,
As I'm sure everybody knows, we use Apache's Jenkins instance for all our
testing, including pre-commit, post-commit, nightly snapshot, etc. (Travis
CI is a backup system and recommended for individual forks only.)

Managing Jenkins projects has been a big pain point so far. Among other
reasons, only a few of us have access to configure it, way too few of us
have visibility into what those jobs do, and nobody has any visibility into
changes being made or an opportunity to comment on them.

Well, not any more! I was playing a little bit with Jenkins DSL plugin and
was able to move our configuration out of Jenkins and into the git
repository. I've done it as a proof of concept for the website repository
only [1], but Jason is planning on extending that work to the main
repository. Look for a PR shortly!

Going forward, anyone can see what our Jenkins jobs are doing, and anyone
can add new jobs or improve existing ones by simply proposing a pull
request to change the configuration. Finally, the project maintains a
history in source repository, instead of direct changes without much
accountability.

How this works? There's a "seed" job that periodically applies
configuration specified in the source repository into Jenkins. Currently,
this happens once per day. If you modify the configuration in the source
repository, it will be applied within 24 hours. If you, however, modify the
configuration in Jenkins directly, it will revert back to whatever is
specified in the code repository also within 24 hours.

How to understand Jenkins DSL? There are many resources available; I've
found Jenkins Job DSL API [2] particularly helpful.

I hope you are excited to have this feature available to us! If you have
any thoughts on improving this further, please comment. Thanks!

Davor

[1] https://github.com/apache/incubator-beam-site/pull/80
[2] https://jenkinsci.github.io/job-dsl-plugin/