Re: [OpenStack-Infra] JJB V2.0 planning

[Wayne Warren]

On Wed, Nov 9, 2016 at 6:41 PM, Thanh Ha 
wrote:

> Hi Everyone,
>
> I'd like to keep the momentum of JJB 2.0 work going as we're so close to
> the finish line. This Friday we're going to continue our sprint in
> #openstack-sprint for those who can make it. Wayne suggested we send a
> status to the mailing list for those who can't.
>
> To summarize some of the etherpad https://etherpad.
> openstack.org/p/jjb_api_v2.0 below.
>
> There's 2 patches that need one more core-review to get merged so
> hopefully someone can take a look at them soon:
>
> https://review.openstack.org/336091 Improve logger output for expanding
> templates
> https://review.openstack.org/206178 Add view management functionality
>

Merged, looking forward trying these out in the 2.0.0 release!


>
> And these patches need some reviews:
>
> https://review.openstack.org/395716 Add support for view-template
>

This is a feature addition that seems to me like it could just as easily
have landed in 1.x as in 2.0.0 or for that matter 2.1, 2.2, 2.3, etc. Are
we blocking a 2.0.0 release on this and if so is there a technical reason
for doing that rather than focusing on merging the changes necessary for
2.0.0--the goal of which remember was to refactor the API for improved
programmatic access to JJB internals, which benefits all of us even if the
view-template feature is not included in the first 2.x release.


>
> We also have 4 patches that still need work:
>
> https://review.openstack.org/333076 Move macro expansion into YamlParser
>

I will find time to update this per the review comments by Friday.


> https://review.openstack.org/309735 Output additional info when
> exceptions occur
> https://review.openstack.org/357960 Add convenience function for plugin
> namespace
> https://review.openstack.org/358019 Support explicit API and simple
> config creation
>
> There's also some TODO actions in the etherpad that we need to decide if
> we want to continue pushing for in the JJB 2.0 push.
>

I think the best forum for making a decision on these items would be here
on the mailing list. If everyone else agrees let's start that in this
thread.


>
> Thanh
>
>
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] JJB V2.0 planning

[Wayne Warren]

On Tue, Jul 5, 2016 at 6:32 AM, Darragh Bailey 
wrote:

>
>
> On 5 July 2016 at 05:44, Kien Ha  wrote:
>
>> HI Darragh,
>>
>> I would like to help where I can in the reviews, but Tuesdays and
>> Thursdays are my busiest days. I can only guarantee that I am free after
>> 19:00 UTC on Tuesdays and Thursdays. Monday and Fridays are days where I am
>> free from summer class obligations but I am not too sure if that will work
>> with everyone else.
>>
>
> I can do Monday, assuming Friday is a little more awkward for people in
> European timezones as they plan to wrap up for the day earlier.
>
> Woud 13:00-1700 UTC suit on those days?
>
>
>>
>> Regards,
>> Kien Ha
>>
>> On Mon, Jul 4, 2016 at 8:47 PM, Dong Ma  wrote:
>>
>>> Hey Darragh,
>>>
>>>
>>>
>>> I agree with your thought and would like to participate to the reviews,
>>> although the time slots is a little late for me but it works.
>>>
>>>
>>>
>>
> Would moving the time slot an hour earlier on either Mon/Fri suit you
> better?
>

For the proposed time slots so far, Friday would be better for me--I have a
team meeting I cannot miss between UTC 1530 and UTC 1630 (although it
usually runs a half hour short).

However, if Monday 1300 - 1700 UTC ends up being the best time for everyone
else, I can attempt to split my attention between the JJB 2.0 review
session and that meeting, or set aside dedicated time outside of the review
session to respond to comments and iterate on the 2.0 patches (really I
will want to do this anyway).


> --
> Darragh Bailey
> "Nothing is foolproof to a sufficiently talented fool"
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] jjb-2.0.0-api rebase onto master & next steps

[Wayne Warren]

On Sat, May 21, 2016 at 7:18 PM, Wayne Warren <wa...@puppet.com> wrote:
> Howdy folks!
>
> About a year ago I announced my intent to put serious effort into
> improving JJB's Python API:
> http://lists.openstack.org/pipermail/openstack-infra/2015-May/002761.html
>
> The currently-approved spec can be seen here:
> https://specs.openstack.org/openstack-infra/infra-specs/specs/jenkins-job-builder_2.0.0-api-changes.html
>
> Related changes can be seen using the "jjb-2.0.0-api" Gerrit topic:
> https://review.openstack.org/#/q/topic:jjb-2.0.0-api

Probably everyone on this list is more gerrit-savvy than myself but
here is a better search link anyway (excludes all the abandoned
feature/2.0.0 branch changes):
https://review.openstack.org/#/q/project:openstack-infra/jenkins-job-builder+topic:jjb-2.0.0-api+status:open

>
> Last month after lots of hard work by the core JJB team the final
> "1.x" version (1.5.0) of JJB was released. Shoutout in particular to
> Darragh Bailey, who took on the most difficult role of keeping track
> of which patches were highest priority for 1.5.0, coordinating the
> necessary reviews, and managing the release--thanks!
>
> Now that JJB 1.5.0 is released, I have rebased the 19 commits in the
> jjb-2.0.0-api topic onto master branch and am about to push those up
> to gerrit for review. But first I wanted to send this email with next
> steps and some notes about my approach in refactoring the API.
>
> Notes about the refactor approach:
> * While there are a few necessarily large commits, most commits are
> small and single-purpose. The small single-purpose commits serve two
> purposes--they made it easier to incrementally change and test the API
> (I was reading a book called The Mikado Method at the time, it's
> definitely worth checking out); they should also make it easier to
> understand and review.
> * Because much of the work was split across 19 commits, I am
> anticipating there may be feedback on the earlier commits that is
> addressed in later commits--rather than shuffing changes around
> between commits or squashing the whole line into one uber-commit
> (+2170,-1411), I am hoping it will suffice for me to point out where
> this is the case to satisfy reviewers.
>
> Next steps:
> * Remove the feature/2.0.x branch from the jenkins-job-builder repo.
> * Review the patches!
> * Merge them when they're ready.
> * Write some API unit tests and documentation.
> * Release 2.0.0!
>
> Happy reviewing!

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

[OpenStack-Infra] jjb-2.0.0-api rebase onto master & next steps

[Wayne Warren]

Howdy folks!

About a year ago I announced my intent to put serious effort into
improving JJB's Python API:
http://lists.openstack.org/pipermail/openstack-infra/2015-May/002761.html

The currently-approved spec can be seen here:
https://specs.openstack.org/openstack-infra/infra-specs/specs/jenkins-job-builder_2.0.0-api-changes.html

Related changes can be seen using the "jjb-2.0.0-api" Gerrit topic:
https://review.openstack.org/#/q/topic:jjb-2.0.0-api

Last month after lots of hard work by the core JJB team the final
"1.x" version (1.5.0) of JJB was released. Shoutout in particular to
Darragh Bailey, who took on the most difficult role of keeping track
of which patches were highest priority for 1.5.0, coordinating the
necessary reviews, and managing the release--thanks!

Now that JJB 1.5.0 is released, I have rebased the 19 commits in the
jjb-2.0.0-api topic onto master branch and am about to push those up
to gerrit for review. But first I wanted to send this email with next
steps and some notes about my approach in refactoring the API.

Notes about the refactor approach:
* While there are a few necessarily large commits, most commits are
small and single-purpose. The small single-purpose commits serve two
purposes--they made it easier to incrementally change and test the API
(I was reading a book called The Mikado Method at the time, it's
definitely worth checking out); they should also make it easier to
understand and review.
* Because much of the work was split across 19 commits, I am
anticipating there may be feedback on the earlier commits that is
addressed in later commits--rather than shuffing changes around
between commits or squashing the whole line into one uber-commit
(+2170,-1411), I am hoping it will suffice for me to point out where
this is the case to satisfy reviewers.

Next steps:
* Remove the feature/2.0.x branch from the jenkins-job-builder repo.
* Review the patches!
* Merge them when they're ready.
* Write some API unit tests and documentation.
* Release 2.0.0!

Happy reviewing!

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Austin Design Summit space needs

[Wayne Warren]

On Thu, Apr 14, 2016 at 2:42 PM, Darragh Bailey <daragh.bai...@gmail.com>
wrote:

> Hi Wayne,
>
>
> Unfortunately I won't be there to be able to catch up with you on the
> proposed changes, but I will try to be around on IRC more during the summit
> mornings, so if you happen to be covering anything involving specs or need
> some stuff reviewed/approved I'll try to make myself available.
>
> Course if we could manage it next week to set aside a 3-4 hours on one day
> to try and close out as many of the existing changes that would be
> applicable without your changes for 2.x it might make things easier for you
> if we could land them and cut a final stable 1.x release, with a view to
> focus on landing your proposed changes.
>

This sounds like a great idea! I'm also hoping to give the existing 1.x
changes some solid attention this week, maybe Wednesday or Thursday
afternoon.


>
>
> On 23 March 2016 at 18:12, Wayne Warren <wa...@puppetlabs.com> wrote:
>
>>
>>
>> On Tue, Mar 22, 2016 at 3:23 PM, Elizabeth K. Joseph <
>> l...@princessleia.com> wrote:
>>
>>> On Fri, Mar 11, 2016 at 9:27 AM, Elizabeth K. Joseph
>>> <l...@princessleia.com> wrote:
>>> > https://etherpad.openstack.org/p/infra-newton-summit-planning
>>>
>>> I've also gone ahead and added a section at the bottom for "Other
>>> sessions of interest" for non-infra sessions that an infra presence
>>> would be particularly valuable at.
>>>
>>
>> Would this be a good place to propose JJB design discussion (both my
>> recent 2.x work and more general YamlParser features moving forward)?
>>
>>
>>>
>>> We've been pretty good at divide and conquer on the fly at summits,
>>> but with the team growing I thought it would be valuable to call out
>>> some of the key sessions ahead of time to make sure we have coverage.
>>> I know I could always use some infra backup at the translations
>>> sessions, which I've added a reference to seed this section.
>>>
>>> --
>>> Elizabeth Krumbach Joseph || Lyz || pleia2
>>>
>>> ___
>>> OpenStack-Infra mailing list
>>> OpenStack-Infra@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>
>>
>>
>> ___
>> OpenStack-Infra mailing list
>> OpenStack-Infra@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>>
>
>
> --
> Darragh Bailey
> "Nothing is foolproof to a sufficiently talented fool"
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Austin Design Summit space needs

[Wayne Warren]

On Tue, Mar 22, 2016 at 3:23 PM, Elizabeth K. Joseph 
wrote:

> On Fri, Mar 11, 2016 at 9:27 AM, Elizabeth K. Joseph
>  wrote:
> > https://etherpad.openstack.org/p/infra-newton-summit-planning
>
> I've also gone ahead and added a section at the bottom for "Other
> sessions of interest" for non-infra sessions that an infra presence
> would be particularly valuable at.
>

Would this be a good place to propose JJB design discussion (both my recent
2.x work and more general YamlParser features moving forward)?


>
> We've been pretty good at divide and conquer on the fly at summits,
> but with the team growing I thought it would be valuable to call out
> some of the key sessions ahead of time to make sure we have coverage.
> I know I could always use some infra backup at the translations
> sessions, which I've added a reference to seed this section.
>
> --
> Elizabeth Krumbach Joseph || Lyz || pleia2
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Regression in JJB 1.4.0 maximum recursion depth exceeded

[Wayne Warren]

Hey Thanh,

Looking at that stack trace and the command that causes it, it appears
that the object whose encoding is being accessed is "sys.stdout" [1]

I would suggest trying to pass a specific "-o" argument to the "test"
subcommand to see if the stream.encoding __getattr__ recursion happens
there also.

Also, what specific minor version of Python are you using?

[1] 
https://review.openstack.org/gitweb?p=openstack-infra/jenkins-job-builder.git;a=blob;f=jenkins_jobs/cmd.py;h=efafdf05fa35f93a00633489dde160c06930641d;hb=b023d7e23f77e4de33e740dcc37af911e36fb189#l115

On Thu, Jan 7, 2016 at 5:40 PM, Thanh Ha  wrote:
> Hi JJB Devs,
>
> We discovered what seems to be a regression with JJB 1.4.0 which after some
> git bisecting found it was caused by this patch [1]. My Jenkins verify
> builds are failing to pass due to python runtime error:
>
> RuntimeError: maximum recursion depth exceeded
>
> This error follows what seems to be a recursive loop of python codecs
> __getattr__ attempts. I've pasted the full traceback below. This issue seems
> to only affect the command "jenkins-jobs test --recursive /path/to/jobs" and
> when as part of a Jenkins verify job. Oddly enough running "jenkins-jobs
> update --recursive /path/to/jobs" seems to pass just fine.
>
> Any ideas how to fix or workaround this issue?  (This issue is preventing us
> from upgrading to JJB 1.4.0)
>
> Thanks,
>
> Thanh
>
> [1] https://review.openstack.org/183939/
>
>
> Traceback (most recent call last):
>   File "/tmp/jjbtest/jjb/bin/jenkins-jobs", line 11, in 
> sys.exit(main())
>   File "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/cmd.py",
> line 172, in main
> execute(options, config)
>   File "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/cmd.py",
> line 337, in execute
> output=options.output_dir)
>   File
> "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/builder.py", line
> 326, in update_job
> output = utils.wrap_stream(output)
>   File "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/utils.py",
> line 25, in wrap_stream
> stream_enc = stream.encoding
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File 

Re: [OpenStack-Infra] Future JJB development

[Wayne Warren]

On Mon, Jun 29, 2015 at 11:00 AM, James E. Blair cor...@inaugust.com wrote:
 Hi,

 Jenkins Job builder is one of our more widely used projects.  It has
 served us extremely well and a lot of other projects have found it to be
 very useful.  Many of us are delighted and very proud of this.

 Recently I have proposed substantial changes to Zuul that I hope will,
 through the process of simplification, mean that we will eventually no
 longer need to use JJB in the OpenStack project.  However, I believe the
 project will continue to be useful for many others.  Meanwhile, others
 within the JJB community have started proposing major changes to JJB as
 well.  I wanted to talk about how development might proceed in order to
 provide minimal disruption for everyone.

 First, I think JJB should continue to at least maintain (and perhaps
 enhance) the current use case and syntax we are using in the OpenStack
 project infrastructure.  If major changes are to happen to JJB, I do not
 anticipate that we will want to make use of them in OpenStack, so we
 will be a good use case to ensure that we do not break compatibility for
 JJB's existing user base.

+1

We should take this one step further and be careful even in cases
where project-config isn't affected by changes. Some of the changes
being discussed in https://review.openstack.org/#/c/194497/ would
definitely have an adverse effect on the configuration I currently
manage.


 Having said that, if the Infrastructure Council, including the current
 JJB cores, feel that the proposed major changes to JJB are desirable, it
 will approve the proposed specs, and those changes can proceed.  If the
 changes need to break backwards compatibility, we can create a feature
 branch for that work (or a stable branch) so that we can continue to
 support the current 1.0 syntax (however, if we can evolve JJB in one
 branch, all the better).

What about API-specific changes that don't affect DSL or command line behavior?

Initially I was thinking that would happen on a feature branch but I'm
not sure how beneficial that would be at this point.


 Finally, assuming that we do accept the Zuulv3 spec and stop using JJB
 ourselves, I would expect us to remove JJB from the list of official
 OpenStack infrastructure projects, but owing to our responsibility to
 the community that has built up around it and our desire for its
 continued success, continue hosting development in OpenStack's project
 infrastructure as long as we are able and the future JJB development
 team desires.

+1, I'd like to help maintain this however I can.


 I hope that this sounds like a clear plan that benefits everyone.

 Thanks,

 Jim

 ___
 OpenStack-Infra mailing list
 OpenStack-Infra@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

[OpenStack-Infra] JJB 2.0.0 API Rewrite

[Wayne Warren]

I have identified in the openstack-infra/infra-specs addition linked
below a number of API shortcomings in Jenkins Job Builder that I
believe can be addressed with concerted effort which I am personally
willing to put in. I am hoping to solicit buy-in and guidance from
other JJB developers before doing too much more coding.

https://review.openstack.org/#/c/182542/

Last week as part of a proof of concept for generating Jenkins XML
with JJB using pure Python modules, I developed a series of patches
which actually begin to address some of the problems outlined in this
spec. The current head of this development effort can be seen here:

https://review.openstack.org/#/c/180315/

This series of patches does not currently include any work specific to
the Jenkins configuration in Python that I mentioned before primarily
because one of my goals with that work was to prove that I could
effectively utilize JJB as library in an external application.

I don't consider implementing this Python module Jenkins configuration
to be in-scope for JJB 2.0.0, but will consider pulling it in for a
later minor version release along with a set of base abstractions that
may also be useful in a future YamlParser rewrite. If anyone happens
to be interested in seeing this work in progress, I'd be happy to set
up a public repository that demonstrates with a very simple example.

Thanks for your time!

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] L summit planning

[Wayne Warren]

On Wed, Apr 22, 2015 at 7:31 AM, James E. Blair cor...@inaugust.com wrote:
 Hi,

 We have four traditional fishbowl summit sessions, then two 2-hour
 blocks in workrooms (which we can split into 1-hour blocks if we want),
 and a full sprint day.

 I've set up an etherpad where we can propose items for each of those
 sections:

   https://etherpad.openstack.org/p/infra-liberty-summit-planning

 There are three sections at the bottom, one for each of the meeting
 types.  Please add proposals there, and feel free to propose it to
 multiple sections if you think it might work in more than one context (I
 expect we will pick only one for a topic).  Please try to add proposals
 this week, or early next.

It looks like I am a little late to the party here and all slots have
been filled.

I was just collecting notes for my JJB2 proposal and it occurred to me
that the summit would be a great opportunity to discuss some of the
backwards-incompatible API changes I would like to make and to
demonstrate some of the progress I have made with JJB configuration
using Python modules rather than YAML.

So I am curious if there will be other opportunities to get a group
together to discuss this, if not is it too late to propose this as a
topic for one of the workroom/fishbowl sessions?

Apologies for the tardiness, if no opportunity is available to talk
about this I am sure discussion via specs/mailing list should be fine.


 -Jim

 ___
 OpenStack-Infra mailing list
 OpenStack-Infra@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra