Re: [libvirt] [jenkins-ci PATCH 2/2] jobs: use job_env in all job templates

2017-11-02 Thread Daniel P. Berrange 
On Thu, Nov 02, 2017 at 04:54:38PM +0100, Pavel Hrdina wrote:
> On Thu, Nov 02, 2017 at 03:42:56PM +, Daniel P. Berrange wrote:
> > On Thu, Nov 02, 2017 at 01:45:51PM +0100, Pavel Hrdina wrote:
> > > This fixes libosinfo RPM build, we need to set "OSINFO_SYSTEM_DIR" for
> > > RPM build as well since the spec file runs tests and they need valid
> > > osinfo-db.  Broken by <359b069f7e4700bee337588eab0a9d83cac94010>.
> > 
> > I'm not suggesting you fix this right now, but our current RPM build
> > process is very broken by design, because we are completely ignoring
> > dependancies between packages. We've hacked around this by culling the
> > BuildRequires lines from the spec file before running rpmbuild, and
> > relying on a bit of luck to have the rpmbuild find the stuff we just
> > built into  $VIRT_PREFIX. As this test suite problem shows though,
> > this is very fragile.
> 
> I completely agree with it, it's fragile and it uses all the environment
> variables to make sure that all the dependencies can be resolved.
> 
> > One way to fix this is to create a yum repo populated with all RPMs
> > we build during CI. Then setup a mock vroot to do the builds in,
> > which includes this local yum repo. That way we can honour the RPM
> > deps correctly getting a more accurate test of RPM build process.
> > 
> > My main concern with this is that mock is kind of slow to bootstrap
> > the initial vroot, so this would slow our CI somewhat.
> 
> We need to look into it how much it will slow down our CI since we have
> very limited resources.  Maybe it's time to ask whether we can have more
> resources.

I think it is well overdue for us to acquire more hardware for Jenkins
slaves. If CentOS have a 2nd machine they can spare for us then great,
otherwise we should try to push RH to fund some...

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [jenkins-ci PATCH 2/2] jobs: use job_env in all job templates

2017-11-02 Thread Pavel Hrdina 
On Thu, Nov 02, 2017 at 03:42:56PM +, Daniel P. Berrange wrote:
> On Thu, Nov 02, 2017 at 01:45:51PM +0100, Pavel Hrdina wrote:
> > This fixes libosinfo RPM build, we need to set "OSINFO_SYSTEM_DIR" for
> > RPM build as well since the spec file runs tests and they need valid
> > osinfo-db.  Broken by <359b069f7e4700bee337588eab0a9d83cac94010>.
> 
> I'm not suggesting you fix this right now, but our current RPM build
> process is very broken by design, because we are completely ignoring
> dependancies between packages. We've hacked around this by culling the
> BuildRequires lines from the spec file before running rpmbuild, and
> relying on a bit of luck to have the rpmbuild find the stuff we just
> built into  $VIRT_PREFIX. As this test suite problem shows though,
> this is very fragile.

I completely agree with it, it's fragile and it uses all the environment
variables to make sure that all the dependencies can be resolved.

> One way to fix this is to create a yum repo populated with all RPMs
> we build during CI. Then setup a mock vroot to do the builds in,
> which includes this local yum repo. That way we can honour the RPM
> deps correctly getting a more accurate test of RPM build process.
> 
> My main concern with this is that mock is kind of slow to bootstrap
> the initial vroot, so this would slow our CI somewhat.

We need to look into it how much it will slow down our CI since we have
very limited resources.  Maybe it's time to ask whether we can have more
resources.

Pavel


signature.asc
Description: PGP signature
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [jenkins-ci PATCH 2/2] jobs: use job_env in all job templates

2017-11-02 Thread Daniel P. Berrange 
On Thu, Nov 02, 2017 at 01:45:51PM +0100, Pavel Hrdina wrote:
> This fixes libosinfo RPM build, we need to set "OSINFO_SYSTEM_DIR" for
> RPM build as well since the spec file runs tests and they need valid
> osinfo-db.  Broken by <359b069f7e4700bee337588eab0a9d83cac94010>.

I'm not suggesting you fix this right now, but our current RPM build
process is very broken by design, because we are completely ignoring
dependancies between packages. We've hacked around this by culling the
BuildRequires lines from the spec file before running rpmbuild, and
relying on a bit of luck to have the rpmbuild find the stuff we just
built into  $VIRT_PREFIX. As this test suite problem shows though,
this is very fragile.

One way to fix this is to create a yum repo populated with all RPMs
we build during CI. Then setup a mock vroot to do the builds in,
which includes this local yum repo. That way we can honour the RPM
deps correctly getting a more accurate test of RPM build process.

My main concern with this is that mock is kind of slow to bootstrap
the initial vroot, so this would slow our CI somewhat.

> 
> Signed-off-by: Pavel Hrdina 
> ---
>  jobs/autotools.yaml  | 3 +++
>  jobs/perl-makemaker.yaml | 3 +++
>  2 files changed, 6 insertions(+)
> 
> diff --git a/jobs/autotools.yaml b/jobs/autotools.yaml
> index 9ed5efe..e41d5ab 100644
> --- a/jobs/autotools.yaml
> +++ b/jobs/autotools.yaml
> @@ -41,6 +41,7 @@
>  builders:
>- shell: |
>{make_env}
> +  {job_env}
>mkdir build
>cd build
>../autogen.sh --prefix=$VIRT_PREFIX {autogen_args}
> @@ -82,6 +83,7 @@
>  builders:
>- shell: |
>{make_env}
> +  {job_env}
>cd build
>$MAKE -j{smp} syntax-check
>  publishers:
> @@ -164,6 +166,7 @@
>  builders:
>- shell: |
>{make_env}
> +  {job_env}
>cd build
>sed -i -e 's/BuildRequires: osinfo-db.*//' {name}.spec
>sed -i -e 's/BuildRequires: libvirt.*devel.*//' {name}.spec
> diff --git a/jobs/perl-makemaker.yaml b/jobs/perl-makemaker.yaml
> index 0a3227a..3e53a29 100644
> --- a/jobs/perl-makemaker.yaml
> +++ b/jobs/perl-makemaker.yaml
> @@ -42,6 +42,7 @@
>- shell: |
>perl Makefile.PL PREFIX="$VIRT_PREFIX"
>{make_env}
> +  {job_env}
>$MAKE
>$MAKE -j{smp} install
>$MAKE -j{smp} manifest
> @@ -83,6 +84,7 @@
>  builders:
>- shell: |
>{make_env}
> +  {job_env}
>$MAKE -j{smp} test {test_args}
>  publishers:
>- email:
> @@ -121,6 +123,7 @@
>  builders:
>- shell: |
>{make_env}
> +  {job_env}
>sed -i -e 's/BuildRequires: *perl-Sys-Virt.*//' *.spec
>sed -i -e 's/BuildRequires: *libvirt.*devel.*//' *.spec
>rm -f *.tar.{archive_format}
> -- 
> 2.13.6
> 
> --
> libvir-list mailing list
> libvir-list@redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list