---------- Forwarded message ---------- From: Abhishek Shrivastava <abhis...@cloudbyte.com> Date: Tue, Dec 23, 2014 at 3:23 PM Subject: Re: [OpenStack-Infra] [Support] Problems installing zuul for jenkins-gerrit gateway (Jenkins does not run jobs) To: Florian Schmidt <florian.schmidt.wel...@t-online.de>
Hi Florian, Thanks for this article, but since we haven't got any issues like yours in our setup, therefore I am sending you one *automated procedure document* for setting up the CI environment that we are following. I hope it can help. On Tue, Dec 23, 2014 at 3:00 PM, Florian Schmidt < florian.schmidt.wel...@t-online.de> wrote: > Hello, > > thanks for your answer. For Jenkins i used this wiki article to install it > on Ubuntu server [1] and added Jobs via the Jenkins web frontend (if you > need the parameter for the test job, i can attach it, but not now, cause > i'm in a train :/) > > In order to install zuul i downloaded the gearman plugin from openstack's > git and build it with marven (clean install), uploaded the resulting plugin > binary to jenkins (again via the web frontned), test the connection in the > settings (works) and enabled gearman. > > [1] > https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu > > If you have further questions, i'm happy to answer :) > > Kind regards, > Florian > > Gesendet mit meinem HTC > > ----- Reply message ----- > Von: "Abhishek Shrivastava" <abhis...@cloudbyte.com> > An: "Florian Schmidt" <florian.schmidt.wel...@t-online.de>, < > openstack-infra@lists.openstack.org> > Cc: "Amit Das" <amit....@cloudbyte.com>, "Punith S" < > punit...@cloudbyte.com> > Betreff: [OpenStack-Infra] [Support] Problems installing zuul for > jenkins-gerrit gateway (Jenkins does not run jobs) > Datum: Di., Dez. 23, 2014 07:35 > > Hi Florian, > > Can you please tell me the procedure which you have followed for setting > up the CI environment, so that I can be clear with your *zuul* problem. > > ---------- Forwarded message ---------- >> From: Florian Schmidt <florian.schmidt.wel...@t-online.de> >> Date: Mon, Dec 22, 2014 at 7:04 PM >> Subject: [OpenStack-Infra] [Support] Problems installing zuul for >> jenkins-gerrit gateway (Jenkins does not run jobs) >> To: openstack-infra@lists.openstack.org >> >> >> Hello all together, >> >> i hope, someone of you can help me with my little problem. Actually I use >> Gerrit Trigger 8Jenkins plugin) to trigger Jenkins jobs for specific >> Gerrit >> events. Now I want to migrate from Gerrit Trigger to zuul (easier, layout >> based configuration and more trigger options and different pipelines). >> >> I installed zuul with this Blog post as a basis: >> >> http://ritchey98.blogspot.jp/2014/02/openstack-third-party-testing-how-to.ht >> ml >> <http://ritchey98.blogspot.jp/2014/02/openstack-third-party-testing-how-to.html> >> >> You can find my zuul configuration here[1] and my initial layout.yaml >> here[2] >> >> Now I have the problem, that it seems, that Jenkins doesn't run jobs added >> In gearman (they don't appear in Jenkins web frontend). Zuul's status page >> lists the change (added via a "recheck" comment) and the status of the job >> "Jenkins-test" is "queued" forever. That's why I think there is a problem >> somewhere between Jenkins and zuul, but I can't find it. I hope, that >> someone from here can assist me to find out the problem? I cleared the log >> and restarted Jenkins and zuul (stopped both and started zuul first and >> then >> Jenkins) and added a new comment "recheck" to my test change in gerrit to >> trigger a new build. I uploaded my log files here[3], maybe it helps to >> find >> out the problem. I have anonymized the domain to be "example.com" in the >> log >> files. >> >> I'm also online in #openstack-infra on freenode (nick: FlorianSW), so feel >> free to contact me there, I would be very grateful to get help to solve >> this >> problem :) >> >> [1] https://gist.github.com/Florianschmidtwelzow/ed0e3047f0ef0c5e5554 >> [2] https://gist.github.com/Florianschmidtwelzow/a101a87d653d5bbc6de6 >> [3] https://gist.github.com/Florianschmidtwelzow/12fff7c4530805dece0c >> >> Kind regards, >> Florian >> >> >> _______________________________________________ >> OpenStack-Infra mailing list >> OpenStack-Infra@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >> >> > > > -- > Thanks & Regards, > Abhishek > -- Thanks & Regards, Abhishek -- Thanks & Regards, Abhishek
================================================================================ README FOR SETUP CINDER CI SYSTEM FOR HDS DRIVERS ================================================================================ ================================================================================ ABOUT ================================================================================ The propose of this README is to set up a real external testing plataform for OpenStack using VMWare to handle nodes snapshots. This README contains the step-by-step to install Jenkins, Jenkins Job Builder (JJB), the Gerrit Jenkins plugin and auxiliary scripts and the HDS/Hitachi jobs. The most part of scripts used in this configuration was forked from: - jaypipes (https://github.com/jaypipes/os-ext-testing) - rasselin (https://github.com/rasselin/os-ext-testing) ================================================================================ PRE-REQUISITES ================================================================================ To set up the external OpenStack CI system you will need: 1. At least 2 Ubuntu 12.04 machines with at least 8GB each machine 2. A Gerrit account registered for your testing system 3. Install base dependency packages on your hosts/VMs 4. Set up your data repository 2. A Gerrit account registered for your testing system In order for your testing platform to post review comments to Gerrit code reviews on openstack.org, you will need to have a service account registered with the OpenStack Infra team. See this link (http://ci.openstack.org/third_party.html#requesting-a-service-account) for instructions on getting this account. In short, you will need to email the OpenStack Infra mailing list (http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra) an email that includes: - The email address to use for the system account (must be different from any other Gerrit account) - A short account username that will appear on code reviews - (optional) A longer account name or description - (optional but encouraged) Include your contact information (IRC handle, your email address, and maybe an alternate contact’s email address) to assist the upstream infrastructure team - The public key for an SSH key pair that the service account will use for Gerrit access.Please note that there should be no newlines in the SSH key Don’t have an SSH key pair for your Gerrit service account? You can create one like so: $ ssh-keygen -t rsa -b 1024 -N '' -f gerrit_key The above will produce the key pair: a pair of files called gerrit_key and gerrit_key.pub. Copy the text of the gerrit_key.pub into the email you send to the OpenStack Infra mailing list. Keep both the files handy for use in the next step. 3. Install base dependency packages on your hosts/VMs $ sudo apt-get update $ sudo apt-get install git wget openssl ssl-cert ca-certificates 4. Set up your data repository You will want to create a Git repository containing configuration data files -- such as the Gerrit username and private SSH key file for your testing account -- that are used in setting up the test platform. The easiest way to get your data repository set up is to make a copy of the example repository I set up here: http://github.com/rasselin/os-ext-testing-data Instead of forking the repository, like you might would normally do, I recommend instead just git clone’ing the repository to some local directory, and making it your own data repository: $ git clone https://github.com/rasselin/os-ext-testing-data.git \ ~/os-ext-testing-data $ cd os-ext-testing-data $ rm -rf .git $ git init . $ git add . $ git commit -a -m "My new data repository" There are a few things you will need to do in this data repository: I. Copy the private SSH key that you submitted when you registered with the upstream OpenStack Infrastructure team into somewhere in this repo. II. If you do not want to use the SSH key pair in the os-ext-testing-data example data repository and want to create your own SSH key pair, do this step. Create an SSH key pair that you will use for Jenkins. This SSH key pair will live in the /var/lib/jenkins/.ssh/ directory on the master Jenkins host, and it will be added to the /home/jenkins/.ssh/authorized_keys file of all slave hosts: $ ssh-keygen -t rsa -b 1024 -N '' -f jenkins_key Once you do the above, copy the jenkins_key and jenkins_key.pub files into your data repository. III. Open up vars.sh in an editor: a) Change the value of $GIT_EMAIL to the email you registered with the upstream OpenStack Infrastructure team b) Change the value of the $GIT_NAME shell variable to the Gerrit username you registered with the upstream OpenStack Infrastructure team as detailed in these instructions c) Change the value of the $UPSTREAM_GERRIT_USER shell variable to the Gerrit username you registered with the upstream OpenStack Infrastructure team as detailed in these instructions d) Change the value of the $UPSTREAM_GERRIT_SSH_KEY_PATH shell variable to the relative path of the private SSH key file you copied into the repository in step #1. For example, let's say you put your private SSH key file named mygerritkey into a directory called ssh within the repository, you would set the $UPSTREAM_GERRIT_SSH_KEY_PATH value to ssh/mygerritkey e) If for some reason, in step #2 above, you either used a different output filename than jenkins_key or put the key pair into some subdirectory of your data repository, then change the value of the $JENKINS_SSH_KEY_PATH variable in vars.sh to an appropriate value. f) Paste the jenkins public key with no whitespaces (only the key) to the $JENKINS_SSH_PUBLIC_KEY_NO_WHITESPACE g) Rename nodepool config file sample to script used by master: $ mv os-ext-testing-data/etc/nodepool/nodepool.yaml.erb.sample \ os-ext-testing-data/etc/nodepool/nodepool.yaml.erb IV. Next, open up the file etc/jenkins_jobs/config/projects.yaml in your data repository. Change the following line in it: vendor: MyVendor Change MyVendor to your organization’s name. V. Save changes in your data repository: $ git add . $ git commit -a -m "Added Gerrit SSH key and username" $ git push ================================================================================ Install Jenkins Master ================================================================================ On the host or virtual machine (or LXC container) you wish to run the Jenkins Master node on, run the following (the git account need to be set before run script - public key): $ git clone $YOUR_DATA_REPO $ wget https://raw.github.com/rasselin/os-ext-testing/master/puppet/install_master.sh $ bash install_master.sh The script will install Puppet, create an SSH key for the Jenkins master, create self-signed certificates for Apache, and then will ask you for the URL of the Git repository you are using as your data repository (see Prerequisites #4 above). Enter the URL of your data repository and hit Enter. Puppet will proceed to set up the Jenkins master. When Puppet completes, go ahead and open up the Jenkins web UI, which by default will be at http://$HOST_IP:8080. You will need to enable the Gearman workers that Zuul and Jenkins use to interact. To do this: 1. Restart jenkins: $ sudo service jenkins restart 2. Click the `Manage Jenkins` link on the left 3. Click the `Configure System` link 4. Scroll down until you see “Gearman Plugin Config”. Check the “Enable Gearman” checkbox. 5. Click the “Test Connection” button and verify Jenkins connects to Gearman. 6. Scroll down to the bottom of the page and click `Save` 7. Once you are done with that, it’s time to load up your Jenkins jobs and restart Zuul: $ sudo jenkins-jobs --flush-cache update /etc/jenkins_jobs/config/ $ sudo service zuul restart If you want to test the communication between Upstream and your master, you can do: $ sudo tail -f /var/log/zuul/debug.log | grep sandbox OK, now create a simple test patch in sandbox. Do this on your development workstation, not your Jenkins master (to do it, you will need to set your gerrit credentials): $ git clone g...@github.com:openstack-dev/sandbox /tmp/sandbox $ cd /tmp/sandbox $ git checkout -b testing-ext $ touch mytest $ git add mytest $ git commit -a -m "Testing comms" $ git review After do it, You will see in the debug.log a gerrit request to run the job pre-configured. ================================================================================ Install Jenkins Master Plugins ================================================================================ Now, you need to update the plugins installed and install anothers: 1. Click the `Manage Jenkins` link on the left 2. Click the `Manage Plugins` link 3. Update all plugins, except the LDAP Plugin 4. Go to available tab 5. Install the following plugins: Conditional-buildstep and Jenkins Parameterized Trigger plugin 6. Manual setup of Jenkins scp 1.9 plugin as follow: Version 1.8 is publicly available, but does not have all features (e.g. copy console log file, copy files after failure, etc.). Follow these steps to manually build and install the scp 1.9 plugin. Solve some dependencies is needed to build using "mvn": $ git clone https://github.com/jenkinsci/scp-plugin $ cd scp-plugin; mvn package When the plugin compilation was finished, go to Jenkins UI and upload the plugin: Jenkins Manage Plugins -> Advanced -> Upload Plugin (target/scp.hpi from the maven build) Source: http://lists.openstack.org/pipermail/openstack-infra/2013-December/000568.html 7. After install SCP Plugin, you need to configure it: a) Click the `Manage Jenkins` link on the left b) Click the `Configure System` link c) Scroll down until you see “SCP repository hosts”. d) Fill out all fields on the section (the Keyfile option is optional). ================================================================================ Install VMWare tools ================================================================================ On node that Jenkins Master was installed: # http://trickycloud.wordpress.com/2013/11/01/using-the-vsphere-sdkapi-in-ubuntu/ configuring vmware client: 1 - Install prereqs: $ sudo apt-get -y update && sudo apt-get install ia32-libs build-essential \ gcc uuid uuid-dev perl libssl-dev perl-doc liburi-perl libxml-libxml-perl \ libcrypt-ssleay-perl 2 - Update the deprecated modules using cpan. Hit <Enter> for all questions: $ sudo cpan install LWP::UserAgent $ sudo cpan install Compress::Zlib $ sudo cpan install Compress::Raw::Zlib $ sudo cpan install IO::Compress::Base $ sudo cpan install IO::Compress::Zlib::Constants 3 - Checking versions: $ perl -MLWP -e 'print "$LWP::VERSION\n"' $ perl -MNet::HTTP -e 'print "$Net::HTTP::VERSION\n"' 4 - Download and install the versions 6.03 of Net-HTTP and LWP: $ wget http://backpan.perl.org/authors/id/G/GA/GAAS/libwww-perl-6.03.tar.gz $ wget http://backpan.perl.org/authors/id/G/GA/GAAS/Net-HTTP-6.03.tar.gz PS: Unpack the files and follow the instructions of README 5 - Verify the new installed versions: $ perl -MLWP -e 'print "$LWP::VERSION\n"' $ perl -MNet::HTTP -e 'print "$Net::HTTP::VERSION\n"' 6 - Download and install vmware SDK client version 5.5 (https://my.vmware.com/group/vmware/details?downloadGroup=SDKPERL550&productId=353#) To do it, You need to register yourself in VMWare website. 7 - Test with the following command: $ vmware-cmd --server 172.24.44.14 --username 'USER' --password 'PASSWORD' \ --vihost <VCENTER IP> "[<DATASTORE>] <DIR PATH>/<FILENAME>.vmx" hassnapshot ================================================================================ Install Jenkins Slave ================================================================================ On each host or virtual machine (or LXC container) you wish to run the Jenkins Slave node on, run the following (the git account need to be set before run script - public key): $ git clone $YOUR_DATA_REPO $ wget https://raw.github.com/rasselin/os-ext-testing/master/puppet/install_slave.sh $ bash install_slave.sh The script will install Puppet, install a Jenkins slave, and install the Jenkins master's public SSH key in the authorized_keys of the Jenkins slave. Once the script completes successfully, you need to add the slave node to Jenkins master. To do so manually, follow these steps: 1. Go to the Jenkins web UI. By default, this will be http://$IP_OF_MASTER:8080 2. Click the Credentials link on the left 3. Click the Global credentials link 4. Click the Add credentials link on the left 5. Select SSH username with private key from the dropdown labeled "Kind" 6. Enter "jenkins" in the Username textbox 7. Select the "From a file on Jenkins master" radio button and enter /var/lib/jenkins/.ssh/id_rsa in the File textbox 8. Click the OK button 9. Click the "Jenkins" link in the upper left to go back to home page 10. Click the Manage Jenkins link on the left 11. Click the Manage Nodes link 12. Click the "New Node" link on the left 13. Enter devstack_slave1 in the Node name textbox 14. Select the Dumb Slave radio button 15. Click the OK button 16. Enter 2 in the Executors textbox 17. Enter /home/jenkins/workspaces in the Remote FS root textbox 18. Enter devstack_slave in the Labels textbox 19. Enter the IP Address of your slave host or VM in the Host textbox 20. Select jenkins from the Credentials dropdown 21. Click the Save button 22. Click the Log link on the left. The log should show the master connecting to the slave, and at the end of the log should be: "Slave successfully connected and online" ================================================================================ Configure Jobs on Master ================================================================================ On the Jenkins Master Node, you need to configure the desired jobs. To work with VMWare, we need to configure a especific type of job. In this toturial we will configure part of the jobs configuration via command line and another configs will be set via GUI. 1. Open the /etc/jenkins_jobs/config/projects.yaml and edit: - project: name: <gerrit-project-name> github-org: <git-repo-name gerrit-project> node: <master-node-name> vendor: <vendor> jobs: - <gerrit-name-job> - <auxiliary-job>: node: <slave-node-name> 2. Open the /etc/jenkins_jobs/config/examples.yaml and edit: - job-template: name: <gerrit-name-job> node: <master-node-name> - job-template: name: <auxialiry-job> node: '{node}' wrappers: - timeout: timeout: 120 # Timeout in *minutes* fail: true # A job run that exceeds the timeout will cause a failure - timestamps builders: - devstack-checkout - shell: | <scripts to configure driver and set env vars> - link-logs # In macros.yaml from os-ext-testing publishers: - devstack-logs # In macros.yaml from os-ext-testing - console-log # In macros.yaml from os-ext-testing PS: An auxialiry-job template can be found in ~/os-ext-testing-data/etc/jenkins_jobs/config/dsvm-cinder-driver.yaml.sample 3. Open the /etc/jenkins_jobs/config/macros.yaml and edit: - builder: name: link-logs builders: - shell: | #!/bin/sh echo "Detailed logs: http://<ip-log-server>/$LOG_PATH/" - publisher: name: console-log publishers: - scp: site: '<ip-log-server>' files: - target: '$LOG_PATH' copy-console: true copy-after-failure: true - publisher: name: devstack-logs publishers: - scp: site: '<ip-log-server>' files: - target: '$LOG_PATH' source: 'logs/**' keep-hierarchy: true copy-after-failure: true 4. Open the /etc/zuul/zuul.conf and edit: url_pattern=http://<ip-log-server>/{build.parameters[LOG_PATH]} status_url=http://<ip-log-server>/zuul/status 5. Update the changes: $ sudo jenkins-jobs update /etc/jenkins_jobs/config 6. On GUI: a) Go to the Jenkins web UI. By default, this will be http://$IP_OF_MASTER:8080 b) Click on <gerrit-name-job> c) Click the Configure link d) Configure "Dicard Olb Builds" as desired e) Uncheck "Execute concurrent buils if necessary" f) Set Time-out strategy to "Absolute" g) Set Timeout minutes to "180" h) Click the Add build step button i) Click the Conditional steps (multiple) option j) Set Run? to "Always" k) Click the Add step to condition button l) Click the Execute Shell option m) Add to the textbox: #!/bin/bash vmware-cmd --server <VCENTE IP> --username 'USERNAME' --password 'PASSWORD' --vihost <VCENTER HOST IP> "[<DATASTORE>] <DATASTORE DIR PATH>/<DATASTORE FILENAME>.vmx" revertsnapshot vmware-cmd --server <VCENTE IP> --username 'USERNAME' --password 'PASSWORD' --vihost <VCENTER HOST IP> "[<DATASTORE>] <DATASTORE DIR PATH>/<DATASTORE FILENAME>.vmx" start hard n) Click the Add step to condition button o) Click the Execute Shell option p) Add to the textbox: #!/bin/bash sleep 200 ssh jenkins@<JENKINS SLAVE IP> "exec java -jar /home/jenkins/workspaces/slave.jar &" sleep 160 q) Click the Add step to condition button r) Click the Trigger/call builds on other projects option s) Set <auxialiry-job-name> to 'Projects to build' t) Check the Block until the triggered projects... option u) Click the Add Parameters button v) Click the Predefined parameters option w) Add to the textbox: LOG_PATH=$LOG_PATH x) Save configuration ================================================================================ Configure Slave Node ================================================================================ 1. Configure NTP Open /etc/ntp.conf Edit the server section Add the Jenkins Master as first NTP server server <JENKINS MASTER IP> iburst restart ntp server sudo service ntp restart 2. Install driver deps, example: nfs-common, open-iscsi-utils, tool binaries,... 3. Add all config file needed, except the cinder.conf and tempest files 4. Change the MySQL password of root user to "secretmysql". That is the default password set on scripts. 5. Take a snapshot that will be used for CI to revert the state of the VM ============================================================================= Configure Log Server ============================================================================= 1. Setup a Machine with public IP, Ubuntu OS, apache and ssh installed 2. Create a user for Jenkins to allow it to copy the log files to the server
_______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra