Re: [Wikitech-l] Adventures in creating new repos / jenkins jobs
Le 28/02/2014 23:10, Antoine Musso a écrit : > python-jenkins then verify the job got created using the GET request > above. I found out tonight that our misc Varnish caches the 404 error > for up to a minute, and hence the second GET is being served the cached > 404 by Varnish. End result JJB consider the job hasn't been created and > bails out with the above stack trace. Hello, This is now fixed. I made the Apache proxy in front of Jenkins to send Cache-Control: no-cache,must-revalidate headers following a suggestion by Mark Bergsma. Related change is https://gerrit.wikimedia.org/r/116748 I confirmed jobs can now be created properly by JJB. Thanks Matthew to have raised the issue! -- Antoine "hashar" Musso ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Adventures in creating new repos / jenkins jobs
Hi, On Thu, Feb 27, 2014 at 04:28:52PM -0800, Matthew Walker wrote: > [ git repository creation and jenkins ] > > Would it make sense to have QChris / ^demon create the standard jobs when > they create the repository? Yes, have ^demon do it :-) I know nothing about our job creation, but sounds like I should start toying around with it. I like the idea of automatically setting up CI for new extensions. Have fun, Christian -- quelltextlich e.U. \\ Christian Aistleitner Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65aEmail: christ...@quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax:+43 732 / 26 95 63 Homepage: http://quelltextlich.at/ --- signature.asc Description: Digital signature ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Adventures in creating new repos / jenkins jobs
Le 28/02/2014 18:55, Mark Holmquist a écrit : > marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$ > jenkins-jobs --conf etc/jenkins_jobs.ini update config/ > 'mwext-MultimediaViewer-do-something' > INFO:root:Updating jobs in config/ (['mwext-MultimediaViewer-do-something']) > INFO:jenkins_jobs.builder:Creating jenkins job > mwext-MultimediaViewer-do-something > https://integration.wikimedia.org/ci/createItem?name=mwext-MultimediaViewer-do-something > jenkins.JenkinsException: create[mwext-MultimediaViewer-do-something] failed > marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$ > > > I just made a dummy job - commit here: > https://gerrit.wikimedia.org/r/116123 > > Obviously nothing special, but the issue is in the HTTP request code > anyway. Hi, Jenkins jobs builder use the python-jenkins module check for the existence of jobs using a simple GET request: GET /ci/job/mwext-MultimediaViewer-do-something/api/json?tree=name That throws a 404 and JJB then create the job: POST /ci/createItem?name=mwext-MultimediaViewer-do-something python-jenkins then verify the job got created using the GET request above. I found out tonight that our misc Varnish caches the 404 error for up to a minute, and hence the second GET is being served the cached 404 by Varnish. End result JJB consider the job hasn't been created and bails out with the above stack trace. The way to fix it: stop caching 404 on the misc varnish, at least when using the gallium backend. There is a cache4xx parameters to the varnish::instance puppet class. Will have to checkout out with some Varnish guru how to best fix it. Meanwhile I am entering sleep() mode.. -- Antoine "hashar" Musso ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Adventures in creating new repos / jenkins jobs
Yep; that's what I was saying above; the REST call to create the job 301
redirects back to integration.wikimedia.org/ when it should redirect to
integration.wikimedia.org/ci.
~Matt Walker
Wikimedia Foundation
Fundraising Technology Team
On Fri, Feb 28, 2014 at 9:55 AM, Mark Holmquist wrote:
> On Fri, Feb 28, 2014 at 06:43:04PM +0100, Antoine Musso wrote:
> > Got any trace to share?
>
> marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$
> jenkins-jobs --conf etc/jenkins_jobs.ini update config/
> 'mwext-MultimediaViewer-do-something'
> INFO:root:Updating jobs in config/
> (['mwext-MultimediaViewer-do-something'])
> INFO:jenkins_jobs.builder:Creating jenkins job
> mwext-MultimediaViewer-do-something
>
> https://integration.wikimedia.org/ci/createItem?name=mwext-MultimediaViewer-do-something
> Traceback (most recent call last):
> File "/usr/local/bin/jenkins-jobs", line 9, in
> load_entry_point('jenkins-job-builder==0.0.584.07fa712',
> 'console_scripts', 'jenkins-jobs')()
> File
> "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/cmd.py",
> line 127, in main
> jobs = builder.update_job(options.path, options.names)
> File
> "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/builder.py",
> line 581, in update_job
> self.jenkins.update_job(job.name, job.output())
> File
> "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/builder.py",
> line 476, in update_job
> self.jenkins.create_job(job_name, xml)
> File
> "/usr/local/lib/python2.7/dist-packages/python_jenkins-0.2.1-py2.7.egg/jenkins/__init__.py",
> line 400, in create_job
> raise JenkinsException('create[%s] failed' % (name))
> jenkins.JenkinsException: create[mwext-MultimediaViewer-do-something]
> failed
> marktraceur@midvalley-the-hornfreak
> :~/projects/wikimedia/integration/jenkins-job-builder$
>
> I just made a dummy job - commit here:
> https://gerrit.wikimedia.org/r/116123
>
> Obviously nothing special, but the issue is in the HTTP request code
> anyway.
>
> Cheers,
>
> --
> Mark Holmquist
> Software Engineer, Multimedia
> Wikimedia Foundation
> mtrac...@member.fsf.org
> https://wikimediafoundation.org/wiki/User:MHolmquist
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iQIbBAEBAgAGBQJTEM2sAAoJEEPl+wghkjzxctAP+MR4obFT1V2rhj0cCr2fs2sI
> /HqB4/3jT2berFZSrQROmS8BKKneNZa/15tvSE3CBOxna5dBAu5LPaR8oLSiD6G1
> Ncd3zC5gt8jxsf7CGDi4Op9hUvZ4jFpq7YqunFO84wns9kEO6BTwGZrpR8pLDJ87
> I6QyFPQmPE/oS9gib1q8c1VMSBgQOyonlWjk91uI1UkRpDO/v2QE3zteWmhTgfWj
> wp0XziQ4prbiwTHP2lXe+4GcOF0XFbTalqUbBrkbjPhqBeEEbBY4ER5lYsn+swS5
> sd6yGKM4WlCgYIjIBZDa4amoAUoj9hu5DvS3LtYdcWptS5NmlWMYkp1utlnNjdpa
> dzekPueAsDr7DcBvQ26O1XnwJ80zC6czZGvNIVAhC92i1rIv2ufAWZeqPYsU29Xb
> VvRtplq5DC3iiblWqyTuinHOUPcU/tXNsH093NkSfwYilSyF8+AbOm4C8ncejOO/
> FT0RbVjosHsBrxzVEMi7FBTBVqssj2dNuiloC//qu8GSaRD/V+B/Ma+SWOgfSjC5
> vacRZvcU8m/lZ13cdTndEKSD6q2a9C2Ld2+iUd3GNayXEnughhi7VIbMFF3ogFR5
> VQ1hKOpvAoGMEykm9tflVaDVsK+O4s/cj1zaDcxnUwaLOvYevojSNLgjZTNHPSZH
> LI11xv7omFaNz6caYvk=
> =z2n8
> -END PGP SIGNATURE-
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Adventures in creating new repos / jenkins jobs
On Fri, Feb 28, 2014 at 06:43:04PM +0100, Antoine Musso wrote:
> Got any trace to share?
marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$
jenkins-jobs --conf etc/jenkins_jobs.ini update config/
'mwext-MultimediaViewer-do-something'
INFO:root:Updating jobs in config/ (['mwext-MultimediaViewer-do-something'])
INFO:jenkins_jobs.builder:Creating jenkins job
mwext-MultimediaViewer-do-something
https://integration.wikimedia.org/ci/createItem?name=mwext-MultimediaViewer-do-something
Traceback (most recent call last):
File "/usr/local/bin/jenkins-jobs", line 9, in
load_entry_point('jenkins-job-builder==0.0.584.07fa712', 'console_scripts',
'jenkins-jobs')()
File
"/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/cmd.py",
line 127, in main
jobs = builder.update_job(options.path, options.names)
File
"/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/builder.py",
line 581, in update_job
self.jenkins.update_job(job.name, job.output())
File
"/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/builder.py",
line 476, in update_job
self.jenkins.create_job(job_name, xml)
File
"/usr/local/lib/python2.7/dist-packages/python_jenkins-0.2.1-py2.7.egg/jenkins/__init__.py",
line 400, in create_job
raise JenkinsException('create[%s] failed' % (name))
jenkins.JenkinsException: create[mwext-MultimediaViewer-do-something] failed
marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$
I just made a dummy job - commit here:
https://gerrit.wikimedia.org/r/116123
Obviously nothing special, but the issue is in the HTTP request code
anyway.
Cheers,
--
Mark Holmquist
Software Engineer, Multimedia
Wikimedia Foundation
mtrac...@member.fsf.org
https://wikimediafoundation.org/wiki/User:MHolmquist
signature.asc
Description: Digital signature
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Adventures in creating new repos / jenkins jobs
Le 28/02/2014 18:39, Mark Holmquist a écrit : > I have had this the entire time we were trying to create the jobs - it > did not help, I still saw the issue. Got any trace to share? On job creation, a POST is sent which is then redirected to a GET which has been cached by varnish previously and says the job does not work. That cause Jenkins Job Builder to choke with an error saying the created job does not exist :-( Workaround: disable Varnish caching entirely.. -- Antoine "hashar" Musso ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Adventures in creating new repos / jenkins jobs
On Fri, Feb 28, 2014 at 04:58:51PM +0100, Antoine Musso wrote: > The proper URL is https://integration.wikimedia.org/ci/ , the > integration.mediawiki.org redirects to the / (though it does not discard > the query string which is a bug). > > I have updated the wiki page, the jenkins_jobs.ini file should have: > > [jenkins] > url=https://integration.wikimedia.org/ci/ > user=... > password=... # actually an user API token I have had this the entire time we were trying to create the jobs - it did not help, I still saw the issue. -- Mark Holmquist Software Engineer, Multimedia Wikimedia Foundation mtrac...@member.fsf.org https://wikimediafoundation.org/wiki/User:MHolmquist signature.asc Description: Digital signature ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Adventures in creating new repos / jenkins jobs
Le 28/02/2014 01:28, Matthew Walker wrote: > > Would it make sense to have QChris / ^demon create the standard jobs when > they create the repository? > Hello, That is a good idea. Moreover we could ensure Bugzilla has a component. We might want to automatize a lot of the workflow as well. Two things that could help a bit is to run actions after a merge to the Zuul and Jenkins Jobs configuration repositories. Ie reload Zuul automatically and generate jobs on post merge. cheers, -- Antoine "hashar" Musso ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Adventures in creating new repos / jenkins jobs
Le 28/02/2014 01:28, Matthew Walker a écrit : > Hey all, > > I recently had a new repository created; and I wanted to create some jobs > for it. > > I dutifully created and had merged: > https://gerrit.wikimedia.org/r/#/c/115968/ > https://gerrit.wikimedia.org/r/#/c/115967/ > > Hashar told me I then needed to follow the instructions on [1] to push the > jobs to jenkins. Running the script myself was only pain; it kept erroring > out while trying to create the job. Marktraceur managed to create the jobs > after much "kicking down the door" aka running the script multiple times. > > It appears that the problem is that > https://integration.mediawiki.org/ci/createItem?name=mwext-FundraisingChart-lint301s > to > https://integration.mediawiki.org/?... > > So that's a problem? We're still not sure why Mark was able to create the > jobs with perseverance though. The proper URL is https://integration.wikimedia.org/ci/ , the integration.mediawiki.org redirects to the / (though it does not discard the query string which is a bug). I have updated the wiki page, the jenkins_jobs.ini file should have: [jenkins] url=https://integration.wikimedia.org/ci/ user=... password=... # actually an user API token While deploying some job to day, I have been hit by the issue of the jobs being created but not registered in Gearman. When posting to the Jenkins API, it issue a redirect to a status page which is cached by the misc Varnish. So we need to send headers to prevent page caching :/ -- Antoine "hashar" Musso ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Adventures in creating new repos / jenkins jobs
Le 28/02/2014 01:36, Mark Holmquist a écrit : > I don't see the code getting checked out on Gallium, and the jobs are all > marked "LOST" with no logs. I'm hopeful that this is an issue related to > the repository still being empty, but this may be too-wishful thinking. > > https://gerrit.wikimedia.org/r/116008 A job reported as LOST is always because Zuul could not find the build result in Jenkins. This can be caused by various situations: - Jenkins died and thus not reporting anything back - The job is not registered in the Gearman bus (the server is integrated in Zuul and Jenkins is a client of it, Jenkins is supposed to register jobs to Zuul Gearman server) In this case, the created jobs have not been properly registered. Which is because the job creation via Jenkins Job Builder did not work as expected. Although the job did get created, the Jenkins hook to register the job in Gearman did not trigger :-( -- Antoine "hashar" Musso ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Adventures in creating new repos / jenkins jobs
On Thu, Feb 27, 2014 at 4:36 PM, Mark Holmquist wrote: > On Thu, Feb 27, 2014 at 04:28:52PM -0800, Matthew Walker wrote: > > Hashar told me I then needed to follow the instructions on [1] to push > the > > jobs to jenkins. Running the script myself was only pain; it kept > erroring > > out while trying to create the job. Marktraceur managed to create the > jobs > > after much "kicking down the door" aka running the script multiple times. > > Hilariously, the jobs still don't run. > > I don't see the code getting checked out on Gallium, and the jobs are all > marked "LOST" with no logs. I'm hopeful that this is an issue related to > the repository still being empty, but this may be too-wishful thinking. > > https://gerrit.wikimedia.org/r/116008 No such luck; we pushed the initial rough code into the repo; and I rebased the above on top of it -- the jobs still fail. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Adventures in creating new repos / jenkins jobs
On Thu, Feb 27, 2014 at 04:28:52PM -0800, Matthew Walker wrote: > Hashar told me I then needed to follow the instructions on [1] to push the > jobs to jenkins. Running the script myself was only pain; it kept erroring > out while trying to create the job. Marktraceur managed to create the jobs > after much "kicking down the door" aka running the script multiple times. Hilariously, the jobs still don't run. I don't see the code getting checked out on Gallium, and the jobs are all marked "LOST" with no logs. I'm hopeful that this is an issue related to the repository still being empty, but this may be too-wishful thinking. https://gerrit.wikimedia.org/r/116008 -- Mark Holmquist Software Engineer, Multimedia Wikimedia Foundation mtrac...@member.fsf.org https://wikimediafoundation.org/wiki/User:MHolmquist signature.asc Description: Digital signature ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
