Re: [OpenStack-Infra] Regression in JJB 1.4.0 maximum recursion depth exceeded

2016-01-11 Thread Thanh Ha 
On 9 January 2016 at 07:10, Darragh Bailey  wrote:

>
> Hi Thanh,
>
> On 8 Jan 2016 21:47, "Thanh Ha"  wrote:
> >
> > (I truncated some of the repetitive output below)
> >
> > I just tried with the -o argument and seems like it passes. I guess it
> only affects stdout output, good to know.
> >
> > We're using python 2.7.5 in production. I did some testing today with
> Python 3.4.3 and it seems the 3.4.x stream at least doesn't run into this
> issue so I'm guessing this only affects Python 2.7.x.
>
> I wonder if it's something that occurs with earlier 2.7 releases, I
> thought I checked that behavior out on 2.7.9
>
> I'll look to check with pyenv on Monday to go over a few python versions.
>
>
Hi Darragh,

The extremely strange thing about this one is I've only been able to
reproduce it in Jenkins. So imagine having JJB installed on the same server
as your Jenkins instance or slave. If you ssh to the machine and run JJB on
the commandline it works. Create a job running on the same system and
Jenkins will see the failure in console logs.

In case it helps with reproducing. The git repo I've been using to
reproduce this error is this one:

https://git.opendaylight.org/gerrit/#/admin/projects/releng/builder

After cloning this repo run "jenkins-jobs test -r jjb/".

Regards,

Thanh
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Regression in JJB 1.4.0 maximum recursion depth exceeded

2016-01-11 Thread Thanh Ha 
On 11 January 2016 at 11:03, Thanh Ha  wrote:

> On 9 January 2016 at 07:10, Darragh Bailey 
> wrote:
>
>> On 8 Jan 2016 21:47, "Thanh Ha"  wrote:
>>
>> >
>> > (I truncated some of the repetitive output below)
>> >
>> > I just tried with the -o argument and seems like it passes. I guess it
>> only affects stdout output, good to know.
>> >
>> > We're using python 2.7.5 in production. I did some testing today with
>> Python 3.4.3 and it seems the 3.4.x stream at least doesn't run into this
>> issue so I'm guessing this only affects Python 2.7.x.
>>
>> I wonder if it's something that occurs with earlier 2.7 releases, I
>> thought I checked that behavior out on 2.7.9
>>
>> I'll look to check with pyenv on Monday to go over a few python versions.
>>
>>
> Hi Darragh,
>
> The extremely strange thing about this one is I've only been able to
> reproduce it in Jenkins. So imagine having JJB installed on the same server
> as your Jenkins instance or slave. If you ssh to the machine and run JJB on
> the commandline it works. Create a job running on the same system and
> Jenkins will see the failure in console logs.
>
> In case it helps with reproducing. The git repo I've been using to
> reproduce this error is this one:
>
> https://git.opendaylight.org/gerrit/#/admin/projects/releng/builder
>
> After cloning this repo run "jenkins-jobs test -r jjb/".
>


I just tested this with Python 2.7.9 and confirm the issue still persists
for me even with 2.7.9. You can see my test here:

https://jenkins.opendaylight.org/sandbox/job/thanh-test/7/console

Regards,

Thanh
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Regression in JJB 1.4.0 maximum recursion depth exceeded

2016-01-11 Thread Darragh Bailey 
Btw, as a very dirty hack the following might work within the jenkins job:

jenkins-jobs test -r builder/jjb 2>&1 | cat

On 11 January 2016 at 17:49, Thanh Ha  wrote:
> On 11 January 2016 at 11:03, Thanh Ha  wrote:
>>
>> On 9 January 2016 at 07:10, Darragh Bailey 
>> wrote:
>>>
>>> On 8 Jan 2016 21:47, "Thanh Ha"  wrote:
>>>
>>> >
>>> > (I truncated some of the repetitive output below)
>>> >
>>> > I just tried with the -o argument and seems like it passes. I guess it
>>> > only affects stdout output, good to know.
>>> >
>>> > We're using python 2.7.5 in production. I did some testing today with
>>> > Python 3.4.3 and it seems the 3.4.x stream at least doesn't run into this
>>> > issue so I'm guessing this only affects Python 2.7.x.
>>>
>>> I wonder if it's something that occurs with earlier 2.7 releases, I
>>> thought I checked that behavior out on 2.7.9
>>>
>>> I'll look to check with pyenv on Monday to go over a few python versions.
>>
>>
>> Hi Darragh,
>>
>> The extremely strange thing about this one is I've only been able to
>> reproduce it in Jenkins. So imagine having JJB installed on the same server
>> as your Jenkins instance or slave. If you ssh to the machine and run JJB on
>> the commandline it works. Create a job running on the same system and
>> Jenkins will see the failure in console logs.
>>
>> In case it helps with reproducing. The git repo I've been using to
>> reproduce this error is this one:
>>
>> https://git.opendaylight.org/gerrit/#/admin/projects/releng/builder
>>
>> After cloning this repo run "jenkins-jobs test -r jjb/".
>
>
>
> I just tested this with Python 2.7.9 and confirm the issue still persists
> for me even with 2.7.9. You can see my test here:
>
> https://jenkins.opendaylight.org/sandbox/job/thanh-test/7/console
>
> Regards,
>
> Thanh
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>



-- 
Darragh Bailey
"Nothing is foolproof to a sufficiently talented fool"

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Regression in JJB 1.4.0 maximum recursion depth exceeded

2016-01-09 Thread Darragh Bailey 
Hi Thanh,

On 8 Jan 2016 21:47, "Thanh Ha"  wrote:
>
> (I truncated some of the repetitive output below)
>
> I just tried with the -o argument and seems like it passes. I guess it
only affects stdout output, good to know.
>
> We're using python 2.7.5 in production. I did some testing today with
Python 3.4.3 and it seems the 3.4.x stream at least doesn't run into this
issue so I'm guessing this only affects Python 2.7.x.

I wonder if it's something that occurs with earlier 2.7 releases, I thought
I checked that behavior out on 2.7.9

I'll look to check with pyenv on Monday to go over a few python versions.

--
Darragh Bailey
"Nothing is foolproof to a sufficiently talented fool" - unknown
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Regression in JJB 1.4.0 maximum recursion depth exceeded

2016-01-08 Thread Wayne Warren 
Hey Thanh,

Looking at that stack trace and the command that causes it, it appears
that the object whose encoding is being accessed is "sys.stdout" [1]

I would suggest trying to pass a specific "-o" argument to the "test"
subcommand to see if the stream.encoding __getattr__ recursion happens
there also.

Also, what specific minor version of Python are you using?

[1] 
https://review.openstack.org/gitweb?p=openstack-infra/jenkins-job-builder.git;a=blob;f=jenkins_jobs/cmd.py;h=efafdf05fa35f93a00633489dde160c06930641d;hb=b023d7e23f77e4de33e740dcc37af911e36fb189#l115

On Thu, Jan 7, 2016 at 5:40 PM, Thanh Ha  wrote:
> Hi JJB Devs,
>
> We discovered what seems to be a regression with JJB 1.4.0 which after some
> git bisecting found it was caused by this patch [1]. My Jenkins verify
> builds are failing to pass due to python runtime error:
>
> RuntimeError: maximum recursion depth exceeded
>
> This error follows what seems to be a recursive loop of python codecs
> __getattr__ attempts. I've pasted the full traceback below. This issue seems
> to only affect the command "jenkins-jobs test --recursive /path/to/jobs" and
> when as part of a Jenkins verify job. Oddly enough running "jenkins-jobs
> update --recursive /path/to/jobs" seems to pass just fine.
>
> Any ideas how to fix or workaround this issue?  (This issue is preventing us
> from upgrading to JJB 1.4.0)
>
> Thanks,
>
> Thanh
>
> [1] https://review.openstack.org/183939/
>
>
> Traceback (most recent call last):
>   File "/tmp/jjbtest/jjb/bin/jenkins-jobs", line 11, in 
> sys.exit(main())
>   File "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/cmd.py",
> line 172, in main
> execute(options, config)
>   File "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/cmd.py",
> line 337, in execute
> output=options.output_dir)
>   File
> "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/builder.py", line
> 326, in update_job
> output = utils.wrap_stream(output)
>   File "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/utils.py",
> line 25, in wrap_stream
> stream_enc = stream.encoding
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> __getattr__
> return getattr(self.stream, name)
>   File

Re: [OpenStack-Infra] Regression in JJB 1.4.0 maximum recursion depth exceeded

2016-01-08 Thread Thanh Ha 
(I truncated some of the repetitive output below)

I just tried with the -o argument and seems like it passes. I guess it only
affects stdout output, good to know.

We're using python 2.7.5 in production. I did some testing today with
Python 3.4.3 and it seems the 3.4.x stream at least doesn't run into this
issue so I'm guessing this only affects Python 2.7.x.

Regards,

Thanh


On 8 January 2016 at 12:44, Wayne Warren  wrote:

> Hey Thanh,
>
> Looking at that stack trace and the command that causes it, it appears
> that the object whose encoding is being accessed is "sys.stdout" [1]
>
> I would suggest trying to pass a specific "-o" argument to the "test"
> subcommand to see if the stream.encoding __getattr__ recursion happens
> there also.
>
> Also, what specific minor version of Python are you using?
>
> [1]
> https://review.openstack.org/gitweb?p=openstack-infra/jenkins-job-builder.git;a=blob;f=jenkins_jobs/cmd.py;h=efafdf05fa35f93a00633489dde160c06930641d;hb=b023d7e23f77e4de33e740dcc37af911e36fb189#l115
>
> On Thu, Jan 7, 2016 at 5:40 PM, Thanh Ha 
> wrote:
> > Hi JJB Devs,
> >
> > We discovered what seems to be a regression with JJB 1.4.0 which after
> some
> > git bisecting found it was caused by this patch [1]. My Jenkins verify
> > builds are failing to pass due to python runtime error:
> >
> > RuntimeError: maximum recursion depth exceeded
> >
> > This error follows what seems to be a recursive loop of python codecs
> > __getattr__ attempts. I've pasted the full traceback below. This issue
> seems
> > to only affect the command "jenkins-jobs test --recursive /path/to/jobs"
> and
> > when as part of a Jenkins verify job. Oddly enough running "jenkins-jobs
> > update --recursive /path/to/jobs" seems to pass just fine.
> >
> > Any ideas how to fix or workaround this issue?  (This issue is
> preventing us
> > from upgrading to JJB 1.4.0)
> >
> > Thanks,
> >
> > Thanh
> >
> > [1] https://review.openstack.org/183939/
> >
> >
> > Traceback (most recent call last):
> >   File "/tmp/jjbtest/jjb/bin/jenkins-jobs", line 11, in 
> > sys.exit(main())
> >   File
> "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/cmd.py",
> > line 172, in main
> > execute(options, config)
> >   File
> "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/cmd.py",
> > line 337, in execute
> > output=options.output_dir)
> >   File
> > "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/builder.py",
> line
> > 326, in update_job
> > output = utils.wrap_stream(output)
> >   File
> "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/utils.py",
> > line 25, in wrap_stream
> > stream_enc = stream.encoding
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> > return getattr(self.stream, name)
> >   File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in
> > __getattr__
> >

[OpenStack-Infra] Regression in JJB 1.4.0 maximum recursion depth exceeded

2016-01-07 Thread Thanh Ha 
Hi JJB Devs,

We discovered what seems to be a regression with JJB 1.4.0 which after some
git bisecting found it was caused by this patch [1]. My Jenkins verify
builds are failing to pass due to python runtime error:

RuntimeError: maximum recursion depth exceeded

This error follows what seems to be a recursive loop of python codecs
__getattr__ attempts. I've pasted the full traceback below. This issue
seems to only affect the command "jenkins-jobs test --recursive
/path/to/jobs" and when as part of a Jenkins verify job. Oddly enough
running "jenkins-jobs update --recursive /path/to/jobs" seems to pass just
fine.

Any ideas how to fix or workaround this issue?  (This issue is preventing
us from upgrading to JJB 1.4.0)

Thanks,

Thanh

[1] https://review.openstack.org/183939/


Traceback (most recent call last):
  File "/tmp/jjbtest/jjb/bin/jenkins-jobs", line 11, in 
sys.exit(main())
  File "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/cmd.py",
line 172, in main
execute(options, config)
  File "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/cmd.py",
line 337, in execute
output=options.output_dir)
  File "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/builder.py",
line 326, in update_job
output = utils.wrap_stream(output)
  File "/tmp/jjbtest/jjb/lib/python2.7/site-packages/jenkins_jobs/utils.py",
line 25, in wrap_stream
stream_enc = stream.encoding
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__
return getattr(self.stream, name)
  File "/tmp/jjbtest/jjb/lib64/python2.7/codecs.py", line 828, in __getattr__