Re: Jenkinsfile and pipeline

[Stefan Seelmann]

There is now also a pipeline for ApacheDS [1][2].

It consists of following stages:
1. Build and run test on Linux with Java 8, 11, 12. (there are test
failures on Windows hence tests are skipped, to be investigated)
2. Deploy to snapshot repo
3. Build and archive the installers
4. Run installers tests

I disabled all old Jenkins jobs.

Kind Regards,
Stefan

[1] https://builds.apache.org/job/dir-server-pipeline/
[2]
https://builds.apache.org/blue/organizations/jenkins/dir-server-pipeline/activity/

Re: Jenkinsfile and pipeline

[Emmanuel Lécharny]

On 22/01/2019 10:23, Colm O hEigeartaigh wrote:

Just FYI with regards to Sonar, we can run instances at Apache as well:

https://builds.apache.org/analysis/



Yes, but it's a pretty old version of Sonar...

Re: Jenkinsfile and pipeline

[Colm O hEigeartaigh]

Just FYI with regards to Sonar, we can run instances at Apache as well:

https://builds.apache.org/analysis/

It's just a matter of setting up a Jenkins job to do it.

Colm.

On Mon, Jan 21, 2019 at 8:03 PM Emmanuel Lécharny 
wrote:

>
> On 21/01/2019 20:06, Stefan Seelmann wrote:
> > Hi,
> >
> > for the LDAP API I created a Jenkinsfile and a corresponding Jenkins
> > pipeline job (there is the classic view [1] and the more fancy "blue
> > ocean" UI [2]). This LDAP API pipeline is quite simple: The code is
> > built and tests are executed on various platforms (Windows with Java 8,
> > Linux with Java 8, 11, 12). When that is successful there is another
> > build that deploys the snapshot JARs to Nexus. I alrady disabled the old
> > LDAP API Jenkins job to not build twice.
>
> That is really great !
>
>
> > The Sonar Job is not included
> > yet, there I have to find out if/how it's possible to integrate.
>
>
> FTR, I'm running the latest Sonar version checks here :
>
> https://sonarcloud.io/dashboard?id=org.apache.directory.api%3Aapi-parent
>
>
> It's a manual job (ie I have to launch it), but at least it's up to
> date. There may be ways to include it in the pipeline, I may ask the
> Sonar people I know for that.
>
> >
> > I'd like to create similar pipelines for the other projects, then with
> > more steps (e.g. installer tests etc).
>
> FTR, ApacheDS won't build with Java 11 (because of Jaxb not being part
> of the JDK anymore, and I think it's because of ehcahe: another good
> reason to switch to Caffeine ?)
>
>
> >
> > The advantages are (IMHO):
> > * The Jenkins job is defined "as code" and is part of the Git repository
> > and versioned
> > * There is only one job with multiple stages/steps which makes the
> > Jenkins setup better manageable
> > * With the current configuration, in case of a failure, there is only
> > one failure email sent
> > * It's easy to add additional build steps
> >
> > Some build steps are based on a Docker image. This allows to define
> > custom tools and dependencies (e.g. new Java version not yet installed
> > on Jenkins, LDAP cmd line tools, other LDAP servers, Kerberos
> > configuration) for more sophisticated testing. All custom Docker images
> > are defined in [3].
> I'm pretty sure Shawn will be interested to have a pipeline for Fortress
> witha  Docker container running Openldap ;-)
> >
> > Let me know what you think or if you have objections or questions.
> > Otherwise I'd continue with more Jenkinsfiles :)
>
>
> More, more more !
>
>
> Thanks Stefan !
>
>

-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Jenkinsfile and pipeline

[Emmanuel Lécharny]

On 22/01/2019 07:40, Stefan Seelmann wrote:

On 1/21/19 11:23 PM, Emmanuel Lécharny wrote:

That makes me think :

what I would absolutely love to have is a pipeline of performance tests
that run against various versions of the server, and various versions of
Java...


What would it take to have that, assuming we have the perf tests already
existi,g ?

I'm not sure the ASF Jenkins infrastructure is well suited for that.
Each job (and also each stage within a pipeline job) is executed on
another machine with different resources (CPU, memeory, disk IO). And as
it's shared resources you can't be sure the perf test can use the full
resources, or if another job is running on the same machine. So the
results are hard to compare.


Agreed.


My idea was rather to have Jenkins running on my machine, to be able to 
start the test during night.




Anyway...

If the perf tests live in a separate git repo just create a
`Jenkinsfile`, otherwise if the are in the apacheds repo just create a
`Jenkinsfile.perftest` and a new Jenkins pipeline job where you
configure the "Script Path". Probably you also want to archive the test
results and change the build discarder to keep a longer history of builds.



I'll have a look at that.



The build stages can reuse the Maven docker images which have different
tags per Java versions. Within the stage steps you can first run a shell
script to download, install, configure and start a particular ApacheDS
version, then run the tests (e.g. with maven).



Thanks Stefan

Re: Jenkinsfile and pipeline

[Stefan Seelmann]

On 1/21/19 11:23 PM, Emmanuel Lécharny wrote:
> That makes me think :
> 
> what I would absolutely love to have is a pipeline of performance tests
> that run against various versions of the server, and various versions of
> Java...
> 
> 
> What would it take to have that, assuming we have the perf tests already
> existi,g ?

I'm not sure the ASF Jenkins infrastructure is well suited for that.
Each job (and also each stage within a pipeline job) is executed on
another machine with different resources (CPU, memeory, disk IO). And as
it's shared resources you can't be sure the perf test can use the full
resources, or if another job is running on the same machine. So the
results are hard to compare.

Anyway...

If the perf tests live in a separate git repo just create a
`Jenkinsfile`, otherwise if the are in the apacheds repo just create a
`Jenkinsfile.perftest` and a new Jenkins pipeline job where you
configure the "Script Path". Probably you also want to archive the test
results and change the build discarder to keep a longer history of builds.

The build stages can reuse the Maven docker images which have different
tags per Java versions. Within the stage steps you can first run a shell
script to download, install, configure and start a particular ApacheDS
version, then run the tests (e.g. with maven).

Re: Jenkinsfile and pipeline

[Emmanuel Lécharny]

That makes me think :

what I would absolutely love to have is a pipeline of performance tests 
that run against various versions of the server, and various versions of 
Java...



What would it take to have that, assuming we have the perf tests already 
existi,g ?



On 21/01/2019 21:53, Pierre Smits wrote:


Jenkins is great. My favourite.

We can even have pipelines that check our current code against latest 
release 3rd party libraries to ensure that our latest committed 
changes work not only against specific (identified) versions, but also 
major and/or minor upgrades.


Best regards,

Pierre Smits

*Apache Trafodion , Vice President*
*Apache Directory , PMC Member*
Apache Incubator , committer
*Apache OFBiz , contributor (without 
privileges) since 2008*

Apache Steve , committer


On Mon, Jan 21, 2019 at 9:17 PM Stefan Seelmann 
mailto:m...@stefan-seelmann.de>> wrote:


On 1/21/19 9:03 PM, Emmanuel Lécharny wrote:
>> I'd like to create similar pipelines for the other projects,
then with
>> more steps (e.g. installer tests etc).
>
> FTR, ApacheDS won't build with Java 11 (because of Jaxb not
being part
> of the JDK anymore, and I think it's because of ehcahe: another good
> reason to switch to Caffeine ?)

Yes, and Studio neither, which is a release blocker for me. But
Richard
will fix it :D

Re: Jenkinsfile and pipeline

[Pierre Smits]

Jenkins is great. My favourite.

We can even have pipelines that check our current code against latest
release 3rd party libraries to ensure that our latest committed changes
work not only against specific (identified) versions, but also major and/or
minor upgrades.

Best regards,

Pierre Smits

*Apache Trafodion , Vice President*
*Apache Directory , PMC Member*
Apache Incubator , committer
*Apache OFBiz , contributor (without privileges)
since 2008*
Apache Steve , committer


On Mon, Jan 21, 2019 at 9:17 PM Stefan Seelmann 
wrote:

> On 1/21/19 9:03 PM, Emmanuel Lécharny wrote:
> >> I'd like to create similar pipelines for the other projects, then with
> >> more steps (e.g. installer tests etc).
> >
> > FTR, ApacheDS won't build with Java 11 (because of Jaxb not being part
> > of the JDK anymore, and I think it's because of ehcahe: another good
> > reason to switch to Caffeine ?)
>
> Yes, and Studio neither, which is a release blocker for me. But Richard
> will fix it :D
>

Re: Jenkinsfile and pipeline

[Stefan Seelmann]

On 1/21/19 9:03 PM, Emmanuel Lécharny wrote:
>> I'd like to create similar pipelines for the other projects, then with
>> more steps (e.g. installer tests etc).
> 
> FTR, ApacheDS won't build with Java 11 (because of Jaxb not being part
> of the JDK anymore, and I think it's because of ehcahe: another good
> reason to switch to Caffeine ?)

Yes, and Studio neither, which is a release blocker for me. But Richard
will fix it :D

Re: Jenkinsfile and pipeline

[Emmanuel Lécharny]

On 21/01/2019 20:06, Stefan Seelmann wrote:

Hi,

for the LDAP API I created a Jenkinsfile and a corresponding Jenkins
pipeline job (there is the classic view [1] and the more fancy "blue
ocean" UI [2]). This LDAP API pipeline is quite simple: The code is
built and tests are executed on various platforms (Windows with Java 8,
Linux with Java 8, 11, 12). When that is successful there is another
build that deploys the snapshot JARs to Nexus. I alrady disabled the old
LDAP API Jenkins job to not build twice.


That is really great !



The Sonar Job is not included
yet, there I have to find out if/how it's possible to integrate.



FTR, I'm running the latest Sonar version checks here :

https://sonarcloud.io/dashboard?id=org.apache.directory.api%3Aapi-parent


It's a manual job (ie I have to launch it), but at least it's up to 
date. There may be ways to include it in the pipeline, I may ask the 
Sonar people I know for that.




I'd like to create similar pipelines for the other projects, then with
more steps (e.g. installer tests etc).


FTR, ApacheDS won't build with Java 11 (because of Jaxb not being part 
of the JDK anymore, and I think it's because of ehcahe: another good 
reason to switch to Caffeine ?)





The advantages are (IMHO):
* The Jenkins job is defined "as code" and is part of the Git repository
and versioned
* There is only one job with multiple stages/steps which makes the
Jenkins setup better manageable
* With the current configuration, in case of a failure, there is only
one failure email sent
* It's easy to add additional build steps

Some build steps are based on a Docker image. This allows to define
custom tools and dependencies (e.g. new Java version not yet installed
on Jenkins, LDAP cmd line tools, other LDAP servers, Kerberos
configuration) for more sophisticated testing. All custom Docker images
are defined in [3].
I'm pretty sure Shawn will be interested to have a pipeline for Fortress 
witha  Docker container running Openldap ;-)


Let me know what you think or if you have objections or questions.
Otherwise I'd continue with more Jenkinsfiles :)



More, more more !


Thanks Stefan !

Re: Jenkinsfile and pipeline

[Stefan Seelmann]

On 1/21/19 8:21 PM, Shawn McKinney wrote:
>> Let me know what you think or if you have objections or questions.
>> Otherwise I'd continue with more Jenkinsfiles :)
> 
> +1, sounds like a great idea to me.  Is this something we can divide and 
> conquer?  I’ve never done this, but it’s something I’ve been interested in 
> for some time.  Also, would be a great introduction to a project for a 
> newbie.  

Sure :)

Just take the existing Jenkinsfile as a template and adapt it. Then
create a Jenkins pipeline job similar the the LDAP API one. The official
Jenkins pipeline documentation might be useful [1].

You can easily develop the Jenkinsfile within a branch, then just
specify the branch to build in the Jenkins job.

To some extend it's also possible to run such a pipeline on your own
Jenkins server. This is especially useful to avoid long wait cycles when
no build slots are available on the ASF Jenkins. However at some point
that has limits, e.g. when using ASF Jenkins specific labels ("Windows")
or JDK installation paths or deployment credentials.

Kind Regards,
Stefan

[1] https://jenkins.io/doc/book/pipeline/

Re: Jenkinsfile and pipeline

[Shawn McKinney]

> 
> On Jan 21, 2019, at 1:06 PM, Stefan Seelmann  wrote:
> 
> Let me know what you think or if you have objections or questions.
> Otherwise I'd continue with more Jenkinsfiles :)

+1, sounds like a great idea to me.  Is this something we can divide and 
conquer?  I’ve never done this, but it’s something I’ve been interested in for 
some time.  Also, would be a great introduction to a project for a newbie.  

Thanks,
—Shawn


> On Jan 21, 2019, at 1:06 PM, Stefan Seelmann  wrote:
> 
> Hi,
> 
> for the LDAP API I created a Jenkinsfile and a corresponding Jenkins
> pipeline job (there is the classic view [1] and the more fancy "blue
> ocean" UI [2]). This LDAP API pipeline is quite simple: The code is
> built and tests are executed on various platforms (Windows with Java 8,
> Linux with Java 8, 11, 12). When that is successful there is another
> build that deploys the snapshot JARs to Nexus. I alrady disabled the old
> LDAP API Jenkins job to not build twice. The Sonar Job is not included
> yet, there I have to find out if/how it's possible to integrate.
> 
> I'd like to create similar pipelines for the other projects, then with
> more steps (e.g. installer tests etc).
> 
> The advantages are (IMHO):
> * The Jenkins job is defined "as code" and is part of the Git repository
> and versioned
> * There is only one job with multiple stages/steps which makes the
> Jenkins setup better manageable
> * With the current configuration, in case of a failure, there is only
> one failure email sent
> * It's easy to add additional build steps
> 
> Some build steps are based on a Docker image. This allows to define
> custom tools and dependencies (e.g. new Java version not yet installed
> on Jenkins, LDAP cmd line tools, other LDAP servers, Kerberos
> configuration) for more sophisticated testing. All custom Docker images
> are defined in [3].
> 
> Let me know what you think or if you have objections or questions.
> Otherwise I'd continue with more Jenkinsfiles :)
> 
> Kind Regards,
> Stefan
> 
> [1]
> https://builds.apache.org/view/D/view/Directory/job/dir-ldap-api-pipeline
> [2]
> https://builds.apache.org/blue/organizations/jenkins/dir-ldap-api-pipeline/activity
> [3] https://github.com/apache/directory-buildtools