RE: Call for jenkins slaves to improve multi operating system support
Loric, do you mean we need give the servers to you or we just build the testing inside our own server room to do all the testing? -jiangang -Original Message- From: ceph-devel-ow...@vger.kernel.org [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Loic Dachary Sent: Wednesday, April 08, 2015 6:56 PM To: Ceph Development Subject: Call for jenkins slaves to improve multi operating system support Hi Ceph, When a contribution is proposed to Ceph [1], a bot compiles and run tests with it to provide feedback to the developer [2]. When something goes wrong the failure can be repeated on the developer machine [3] for debug. This also helps the reviewer who knows the code compiles and does not break anything that would be detected by "make check". The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to older operating systems (headers, compiler version, etc.) may be detected later, when building packages [4] and after the pull request has been merged in master. This is rare but requires extra attention from the reviewer and needs to be dealt with urgently when it happens. If you can spare a machine to help expand the operating systems on which tests can run, it would be a great help. The minimum hardware configuration to run a slave is: * x86_64 architecture for CentOS 6, Fedora 21, OpenSUSE 13.2, Debian GNU/Linux Jessie, Ubuntu 14.02 32 GB RAM 200 GB SSD 8 core > 2.5Ghz * i386 architecture for CentOS 7, CentOS 6, Fedora 21, Debian GNU/Linux Jessie, Ubuntu 14.04, Ubuntu 14.02 4 GB RAM 200 GB disk 2 core * armv7, armv8 architecture for Ubuntu 14.04 4 GB RAM 200 GB disk 2 core Note that since the make check bot can run in a docker container, x86_64 machines can be used to run any of the operating systems for which a docker file has been prepared [5]. Cheers [1] pull requests https://github.com/ceph/ceph/pulls [2] make check bot feedback https://github.com/ceph/ceph/pull/4296#issuecomment-90812064 [3] run-make-check.sh https://github.com/ceph/ceph/blob/master/run-make-check.sh#L44 [4] gitbuilder http://ceph.com/gitbuilder.cgi [5] https://ceph.com/git/?p=ceph.git;a=blob;f=src/test/Makefile.am;hb=hammer#l91 -- Loïc Dachary, Artisan Logiciel Libre
Re: Call for jenkins slaves to improve multi operating system support
On 08/04/2015 15:21, Duan, Jiangang wrote: > Loric, > > do you mean we need give the servers to you or we just build the testing > inside our own server room to do all the testing? Thanks for asking, I realize that was not clear. The idea is not to donate hardware, because that would require manpower and extra costs to connect to the net. What would be useful is a machine connected to the net and dedicated to running a jenkins slave. It receives a build from the jenkins master (http://jenkins.ceph.dachary.org/) via ssh (possibly with a tunnel if behind a NAT), clone http://github.com/ceph/ceph, execute the run-make-check.sh script that is found at the root of the repository and reports failure / success back to the jenkins master. Does that make sense ? > > -jiangang > > -Original Message- > From: ceph-devel-ow...@vger.kernel.org > [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Loic Dachary > Sent: Wednesday, April 08, 2015 6:56 PM > To: Ceph Development > Subject: Call for jenkins slaves to improve multi operating system support > > Hi Ceph, > > When a contribution is proposed to Ceph [1], a bot compiles and run tests > with it to provide feedback to the developer [2]. When something goes wrong > the failure can be repeated on the developer machine [3] for debug. This also > helps the reviewer who knows the code compiles and does not break anything > that would be detected by "make check". > > The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to older > operating systems (headers, compiler version, etc.) may be detected later, > when building packages [4] and after the pull request has been merged in > master. This is rare but requires extra attention from the reviewer and needs > to be dealt with urgently when it happens. > > If you can spare a machine to help expand the operating systems on which > tests can run, it would be a great help. The minimum hardware configuration > to run a slave is: > > * x86_64 architecture for CentOS 6, Fedora 21, OpenSUSE 13.2, Debian > GNU/Linux Jessie, Ubuntu 14.02 > > 32 GB RAM > 200 GB SSD > 8 core > 2.5Ghz > > * i386 architecture for CentOS 7, CentOS 6, Fedora 21, Debian GNU/Linux > Jessie, Ubuntu 14.04, Ubuntu 14.02 > > 4 GB RAM > 200 GB disk > 2 core > > * armv7, armv8 architecture for Ubuntu 14.04 > > 4 GB RAM > 200 GB disk > 2 core > > Note that since the make check bot can run in a docker container, x86_64 > machines can be used to run any of the operating systems for which a docker > file has been prepared [5]. > > Cheers > > [1] pull requests https://github.com/ceph/ceph/pulls > [2] make check bot feedback > https://github.com/ceph/ceph/pull/4296#issuecomment-90812064 > [3] run-make-check.sh > https://github.com/ceph/ceph/blob/master/run-make-check.sh#L44 > [4] gitbuilder http://ceph.com/gitbuilder.cgi [5] > https://ceph.com/git/?p=ceph.git;a=blob;f=src/test/Makefile.am;hb=hammer#l91 > > -- > Loïc Dachary, Artisan Logiciel Libre > -- Loïc Dachary, Artisan Logiciel Libre signature.asc Description: OpenPGP digital signature
Re: Call for jenkins slaves to improve multi operating system support
On Wed, 8 Apr 2015, Loic Dachary wrote: > Hi Ceph, > > When a contribution is proposed to Ceph [1], a bot compiles and run > tests with it to provide feedback to the developer [2]. When something > goes wrong the failure can be repeated on the developer machine [3] for > debug. This also helps the reviewer who knows the code compiles and does > not break anything that would be detected by "make check". > > The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to > older operating systems (headers, compiler version, etc.) may be > detected later, when building packages [4] and after the pull request > has been merged in master. This is rare but requires extra attention > from the reviewer and needs to be dealt with urgently when it happens. Do additional slaves block the message from appearing on the pull request? I.e., what happens if a slave is very slow (e.g., armv7) or broken (network issue)? What are the connectivity requirements? Can slaves exist on other (private) networks? Thanks! sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Call for jenkins slaves to improve multi operating system support
Hi Sage, On 08/04/2015 18:59, Sage Weil wrote: > On Wed, 8 Apr 2015, Loic Dachary wrote: >> Hi Ceph, >> >> When a contribution is proposed to Ceph [1], a bot compiles and run >> tests with it to provide feedback to the developer [2]. When something >> goes wrong the failure can be repeated on the developer machine [3] for >> debug. This also helps the reviewer who knows the code compiles and does >> not break anything that would be detected by "make check". >> >> The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to >> older operating systems (headers, compiler version, etc.) may be >> detected later, when building packages [4] and after the pull request >> has been merged in master. This is rare but requires extra attention >> from the reviewer and needs to be dealt with urgently when it happens. > > Do additional slaves block the message from appearing on the pull > request? I.e., what happens if a slave is very slow (e.g., armv7) or > broken (network issue)? I will make it so a comment is posted as soon as the first slave succeeds / fails (it currently waits for all to finish which is inconvenient). The first slave will always be a CentOS 7 running on a fast machine so that the worst that can happen is that it's the only one to run. If slow slaves lag behind too much it would be nice to have a jenkins plugin that discards jobs randomly to prevent the queue from growing out of proportion on that specific slave. > What are the connectivity requirements? Nothing more than the ability to git pull from a ceph repository. > Can slaves exist on other > (private) networks? Yes. In that case a ssh -f -n -L tunnel to the jenkins master will be established to allow it to probe the slave when necessary. > > Thanks! > sage > Cheers -- Loïc Dachary, Artisan Logiciel Libre signature.asc Description: OpenPGP digital signature
Re: Call for jenkins slaves to improve multi operating system support
On 09/04/2015 12:30, Duan, Jiangang wrote: > Got you. > Let me see if we can help - it may take some time - we have the servers but > need find some people to set it up. Thanks a lot, I'm looking forward to that :-) > -Original Message- > From: Loic Dachary [mailto:l...@dachary.org] > Sent: Wednesday, April 08, 2015 9:33 PM > To: Duan, Jiangang; Ceph Development > Subject: Re: Call for jenkins slaves to improve multi operating system support > > > > On 08/04/2015 15:21, Duan, Jiangang wrote: >> Loric, >> >> do you mean we need give the servers to you or we just build the testing >> inside our own server room to do all the testing? > > Thanks for asking, I realize that was not clear. > > The idea is not to donate hardware, because that would require manpower and > extra costs to connect to the net. > > What would be useful is a machine connected to the net and dedicated to > running a jenkins slave. It receives a build from the jenkins master > (http://jenkins.ceph.dachary.org/) via ssh (possibly with a tunnel if behind > a NAT), clone http://github.com/ceph/ceph, execute the run-make-check.sh > script that is found at the root of the repository and reports failure / > success back to the jenkins master. > > Does that make sense ? > >> >> -jiangang >> >> -Original Message- >> From: ceph-devel-ow...@vger.kernel.org >> [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Loic Dachary >> Sent: Wednesday, April 08, 2015 6:56 PM >> To: Ceph Development >> Subject: Call for jenkins slaves to improve multi operating system support >> >> Hi Ceph, >> >> When a contribution is proposed to Ceph [1], a bot compiles and run tests >> with it to provide feedback to the developer [2]. When something goes wrong >> the failure can be repeated on the developer machine [3] for debug. This >> also helps the reviewer who knows the code compiles and does not break >> anything that would be detected by "make check". >> >> The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to >> older operating systems (headers, compiler version, etc.) may be detected >> later, when building packages [4] and after the pull request has been merged >> in master. This is rare but requires extra attention from the reviewer and >> needs to be dealt with urgently when it happens. >> >> If you can spare a machine to help expand the operating systems on which >> tests can run, it would be a great help. The minimum hardware configuration >> to run a slave is: >> >> * x86_64 architecture for CentOS 6, Fedora 21, OpenSUSE 13.2, Debian >> GNU/Linux Jessie, Ubuntu 14.02 >> >> 32 GB RAM >> 200 GB SSD >> 8 core > 2.5Ghz >> >> * i386 architecture for CentOS 7, CentOS 6, Fedora 21, Debian GNU/Linux >> Jessie, Ubuntu 14.04, Ubuntu 14.02 >> >> 4 GB RAM >> 200 GB disk >> 2 core >> >> * armv7, armv8 architecture for Ubuntu 14.04 >> >> 4 GB RAM >> 200 GB disk >> 2 core >> >> Note that since the make check bot can run in a docker container, x86_64 >> machines can be used to run any of the operating systems for which a docker >> file has been prepared [5]. >> >> Cheers >> >> [1] pull requests https://github.com/ceph/ceph/pulls >> [2] make check bot feedback >> https://github.com/ceph/ceph/pull/4296#issuecomment-90812064 >> [3] run-make-check.sh >> https://github.com/ceph/ceph/blob/master/run-make-check.sh#L44 >> [4] gitbuilder http://ceph.com/gitbuilder.cgi [5] >> https://ceph.com/git/?p=ceph.git;a=blob;f=src/test/Makefile.am;hb=hammer#l91 >> >> -- >> Loïc Dachary, Artisan Logiciel Libre >> > -- Loïc Dachary, Artisan Logiciel Libre signature.asc Description: OpenPGP digital signature
RE: Call for jenkins slaves to improve multi operating system support
Got you. Let me see if we can help - it may take some time - we have the servers but need find some people to set it up. -Original Message- From: Loic Dachary [mailto:l...@dachary.org] Sent: Wednesday, April 08, 2015 9:33 PM To: Duan, Jiangang; Ceph Development Subject: Re: Call for jenkins slaves to improve multi operating system support On 08/04/2015 15:21, Duan, Jiangang wrote: > Loric, > > do you mean we need give the servers to you or we just build the testing > inside our own server room to do all the testing? Thanks for asking, I realize that was not clear. The idea is not to donate hardware, because that would require manpower and extra costs to connect to the net. What would be useful is a machine connected to the net and dedicated to running a jenkins slave. It receives a build from the jenkins master (http://jenkins.ceph.dachary.org/) via ssh (possibly with a tunnel if behind a NAT), clone http://github.com/ceph/ceph, execute the run-make-check.sh script that is found at the root of the repository and reports failure / success back to the jenkins master. Does that make sense ? > > -jiangang > > -Original Message- > From: ceph-devel-ow...@vger.kernel.org > [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Loic Dachary > Sent: Wednesday, April 08, 2015 6:56 PM > To: Ceph Development > Subject: Call for jenkins slaves to improve multi operating system support > > Hi Ceph, > > When a contribution is proposed to Ceph [1], a bot compiles and run tests > with it to provide feedback to the developer [2]. When something goes wrong > the failure can be repeated on the developer machine [3] for debug. This also > helps the reviewer who knows the code compiles and does not break anything > that would be detected by "make check". > > The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to older > operating systems (headers, compiler version, etc.) may be detected later, > when building packages [4] and after the pull request has been merged in > master. This is rare but requires extra attention from the reviewer and needs > to be dealt with urgently when it happens. > > If you can spare a machine to help expand the operating systems on which > tests can run, it would be a great help. The minimum hardware configuration > to run a slave is: > > * x86_64 architecture for CentOS 6, Fedora 21, OpenSUSE 13.2, Debian > GNU/Linux Jessie, Ubuntu 14.02 > > 32 GB RAM > 200 GB SSD > 8 core > 2.5Ghz > > * i386 architecture for CentOS 7, CentOS 6, Fedora 21, Debian GNU/Linux > Jessie, Ubuntu 14.04, Ubuntu 14.02 > > 4 GB RAM > 200 GB disk > 2 core > > * armv7, armv8 architecture for Ubuntu 14.04 > > 4 GB RAM > 200 GB disk > 2 core > > Note that since the make check bot can run in a docker container, x86_64 > machines can be used to run any of the operating systems for which a docker > file has been prepared [5]. > > Cheers > > [1] pull requests https://github.com/ceph/ceph/pulls > [2] make check bot feedback > https://github.com/ceph/ceph/pull/4296#issuecomment-90812064 > [3] run-make-check.sh > https://github.com/ceph/ceph/blob/master/run-make-check.sh#L44 > [4] gitbuilder http://ceph.com/gitbuilder.cgi [5] > https://ceph.com/git/?p=ceph.git;a=blob;f=src/test/Makefile.am;hb=hammer#l91 > > -- > Loïc Dachary, Artisan Logiciel Libre > -- Loïc Dachary, Artisan Logiciel Libre
Re: Call for jenkins slaves to improve multi operating system support
Hi, Thanks to people who proposed to host jenkins slave: two new slaves should be added next week :-) The conversations we had have been summarized in a short document that will hopefully clarify how the slaves are put to work: http://ceph-workbench.readthedocs.org/en/latest/user/jenkins.html Cheers On 08/04/2015 12:56, Loic Dachary wrote: > Hi Ceph, > > When a contribution is proposed to Ceph [1], a bot compiles and run tests > with it to provide feedback to the developer [2]. When something goes wrong > the failure can be repeated on the developer machine [3] for debug. This also > helps the reviewer who knows the code compiles and does not break anything > that would be detected by "make check". > > The bot runs on CentOS 7 and Ubuntu 14.04 only, and problems related to older > operating systems (headers, compiler version, etc.) may be detected later, > when building packages [4] and after the pull request has been merged in > master. This is rare but requires extra attention from the reviewer and needs > to be dealt with urgently when it happens. > > If you can spare a machine to help expand the operating systems on which > tests can run, it would be a great help. The minimum hardware configuration > to run a slave is: > > * x86_64 architecture for CentOS 6, Fedora 21, OpenSUSE 13.2, Debian > GNU/Linux Jessie, Ubuntu 14.02 > > 32 GB RAM > 200 GB SSD > 8 core > 2.5Ghz > > * i386 architecture for CentOS 7, CentOS 6, Fedora 21, Debian GNU/Linux > Jessie, Ubuntu 14.04, Ubuntu 14.02 > > 4 GB RAM > 200 GB disk > 2 core > > * armv7, armv8 architecture for Ubuntu 14.04 > > 4 GB RAM > 200 GB disk > 2 core > > Note that since the make check bot can run in a docker container, x86_64 > machines can be used to run any of the operating systems for which a docker > file has been prepared [5]. > > Cheers > > [1] pull requests https://github.com/ceph/ceph/pulls > [2] make check bot feedback > https://github.com/ceph/ceph/pull/4296#issuecomment-90812064 > [3] run-make-check.sh > https://github.com/ceph/ceph/blob/master/run-make-check.sh#L44 > [4] gitbuilder http://ceph.com/gitbuilder.cgi > [5] > https://ceph.com/git/?p=ceph.git;a=blob;f=src/test/Makefile.am;hb=hammer#l91 > -- Loïc Dachary, Artisan Logiciel Libre signature.asc Description: OpenPGP digital signature
