Re: [libvirt] [jenkins-ci PATCH 2/2] jobs: use job_env in all job templates
On Thu, Nov 02, 2017 at 04:54:38PM +0100, Pavel Hrdina wrote: > On Thu, Nov 02, 2017 at 03:42:56PM +, Daniel P. Berrange wrote: > > On Thu, Nov 02, 2017 at 01:45:51PM +0100, Pavel Hrdina wrote: > > > This fixes libosinfo RPM build, we need to set "OSINFO_SYSTEM_DIR" for > > > RPM build as well since the spec file runs tests and they need valid > > > osinfo-db. Broken by <359b069f7e4700bee337588eab0a9d83cac94010>. > > > > I'm not suggesting you fix this right now, but our current RPM build > > process is very broken by design, because we are completely ignoring > > dependancies between packages. We've hacked around this by culling the > > BuildRequires lines from the spec file before running rpmbuild, and > > relying on a bit of luck to have the rpmbuild find the stuff we just > > built into $VIRT_PREFIX. As this test suite problem shows though, > > this is very fragile. > > I completely agree with it, it's fragile and it uses all the environment > variables to make sure that all the dependencies can be resolved. > > > One way to fix this is to create a yum repo populated with all RPMs > > we build during CI. Then setup a mock vroot to do the builds in, > > which includes this local yum repo. That way we can honour the RPM > > deps correctly getting a more accurate test of RPM build process. > > > > My main concern with this is that mock is kind of slow to bootstrap > > the initial vroot, so this would slow our CI somewhat. > > We need to look into it how much it will slow down our CI since we have > very limited resources. Maybe it's time to ask whether we can have more > resources. I think it is well overdue for us to acquire more hardware for Jenkins slaves. If CentOS have a 2nd machine they can spare for us then great, otherwise we should try to push RH to fund some... Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [jenkins-ci PATCH 2/2] jobs: use job_env in all job templates
On Thu, Nov 02, 2017 at 03:42:56PM +, Daniel P. Berrange wrote: > On Thu, Nov 02, 2017 at 01:45:51PM +0100, Pavel Hrdina wrote: > > This fixes libosinfo RPM build, we need to set "OSINFO_SYSTEM_DIR" for > > RPM build as well since the spec file runs tests and they need valid > > osinfo-db. Broken by <359b069f7e4700bee337588eab0a9d83cac94010>. > > I'm not suggesting you fix this right now, but our current RPM build > process is very broken by design, because we are completely ignoring > dependancies between packages. We've hacked around this by culling the > BuildRequires lines from the spec file before running rpmbuild, and > relying on a bit of luck to have the rpmbuild find the stuff we just > built into $VIRT_PREFIX. As this test suite problem shows though, > this is very fragile. I completely agree with it, it's fragile and it uses all the environment variables to make sure that all the dependencies can be resolved. > One way to fix this is to create a yum repo populated with all RPMs > we build during CI. Then setup a mock vroot to do the builds in, > which includes this local yum repo. That way we can honour the RPM > deps correctly getting a more accurate test of RPM build process. > > My main concern with this is that mock is kind of slow to bootstrap > the initial vroot, so this would slow our CI somewhat. We need to look into it how much it will slow down our CI since we have very limited resources. Maybe it's time to ask whether we can have more resources. Pavel signature.asc Description: PGP signature -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [jenkins-ci PATCH 2/2] jobs: use job_env in all job templates
On Thu, Nov 02, 2017 at 01:45:51PM +0100, Pavel Hrdina wrote: > This fixes libosinfo RPM build, we need to set "OSINFO_SYSTEM_DIR" for > RPM build as well since the spec file runs tests and they need valid > osinfo-db. Broken by <359b069f7e4700bee337588eab0a9d83cac94010>. I'm not suggesting you fix this right now, but our current RPM build process is very broken by design, because we are completely ignoring dependancies between packages. We've hacked around this by culling the BuildRequires lines from the spec file before running rpmbuild, and relying on a bit of luck to have the rpmbuild find the stuff we just built into $VIRT_PREFIX. As this test suite problem shows though, this is very fragile. One way to fix this is to create a yum repo populated with all RPMs we build during CI. Then setup a mock vroot to do the builds in, which includes this local yum repo. That way we can honour the RPM deps correctly getting a more accurate test of RPM build process. My main concern with this is that mock is kind of slow to bootstrap the initial vroot, so this would slow our CI somewhat. > > Signed-off-by: Pavel Hrdina > --- > jobs/autotools.yaml | 3 +++ > jobs/perl-makemaker.yaml | 3 +++ > 2 files changed, 6 insertions(+) > > diff --git a/jobs/autotools.yaml b/jobs/autotools.yaml > index 9ed5efe..e41d5ab 100644 > --- a/jobs/autotools.yaml > +++ b/jobs/autotools.yaml > @@ -41,6 +41,7 @@ > builders: >- shell: | >{make_env} > + {job_env} >mkdir build >cd build >../autogen.sh --prefix=$VIRT_PREFIX {autogen_args} > @@ -82,6 +83,7 @@ > builders: >- shell: | >{make_env} > + {job_env} >cd build >$MAKE -j{smp} syntax-check > publishers: > @@ -164,6 +166,7 @@ > builders: >- shell: | >{make_env} > + {job_env} >cd build >sed -i -e 's/BuildRequires: osinfo-db.*//' {name}.spec >sed -i -e 's/BuildRequires: libvirt.*devel.*//' {name}.spec > diff --git a/jobs/perl-makemaker.yaml b/jobs/perl-makemaker.yaml > index 0a3227a..3e53a29 100644 > --- a/jobs/perl-makemaker.yaml > +++ b/jobs/perl-makemaker.yaml > @@ -42,6 +42,7 @@ >- shell: | >perl Makefile.PL PREFIX="$VIRT_PREFIX" >{make_env} > + {job_env} >$MAKE >$MAKE -j{smp} install >$MAKE -j{smp} manifest > @@ -83,6 +84,7 @@ > builders: >- shell: | >{make_env} > + {job_env} >$MAKE -j{smp} test {test_args} > publishers: >- email: > @@ -121,6 +123,7 @@ > builders: >- shell: | >{make_env} > + {job_env} >sed -i -e 's/BuildRequires: *perl-Sys-Virt.*//' *.spec >sed -i -e 's/BuildRequires: *libvirt.*devel.*//' *.spec >rm -f *.tar.{archive_format} > -- > 2.13.6 > > -- > libvir-list mailing list > libvir-list@redhat.com > https://www.redhat.com/mailman/listinfo/libvir-list Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[libvirt] [jenkins-ci PATCH 2/2] jobs: use job_env in all job templates
This fixes libosinfo RPM build, we need to set "OSINFO_SYSTEM_DIR" for RPM build as well since the spec file runs tests and they need valid osinfo-db. Broken by <359b069f7e4700bee337588eab0a9d83cac94010>. Signed-off-by: Pavel Hrdina --- jobs/autotools.yaml | 3 +++ jobs/perl-makemaker.yaml | 3 +++ 2 files changed, 6 insertions(+) diff --git a/jobs/autotools.yaml b/jobs/autotools.yaml index 9ed5efe..e41d5ab 100644 --- a/jobs/autotools.yaml +++ b/jobs/autotools.yaml @@ -41,6 +41,7 @@ builders: - shell: | {make_env} + {job_env} mkdir build cd build ../autogen.sh --prefix=$VIRT_PREFIX {autogen_args} @@ -82,6 +83,7 @@ builders: - shell: | {make_env} + {job_env} cd build $MAKE -j{smp} syntax-check publishers: @@ -164,6 +166,7 @@ builders: - shell: | {make_env} + {job_env} cd build sed -i -e 's/BuildRequires: osinfo-db.*//' {name}.spec sed -i -e 's/BuildRequires: libvirt.*devel.*//' {name}.spec diff --git a/jobs/perl-makemaker.yaml b/jobs/perl-makemaker.yaml index 0a3227a..3e53a29 100644 --- a/jobs/perl-makemaker.yaml +++ b/jobs/perl-makemaker.yaml @@ -42,6 +42,7 @@ - shell: | perl Makefile.PL PREFIX="$VIRT_PREFIX" {make_env} + {job_env} $MAKE $MAKE -j{smp} install $MAKE -j{smp} manifest @@ -83,6 +84,7 @@ builders: - shell: | {make_env} + {job_env} $MAKE -j{smp} test {test_args} publishers: - email: @@ -121,6 +123,7 @@ builders: - shell: | {make_env} + {job_env} sed -i -e 's/BuildRequires: *perl-Sys-Virt.*//' *.spec sed -i -e 's/BuildRequires: *libvirt.*devel.*//' *.spec rm -f *.tar.{archive_format} -- 2.13.6 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list