Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]

2017-10-15 Thread Yaniv Kaul 
On Sun, Oct 15, 2017 at 10:15 AM, Barak Korren  wrote:

>
>
> On 13 October 2017 at 14:31, Sandro Bonazzola  wrote:
>
>>
>> 2017-10-13 11:29 GMT+02:00 Dafna Ron :
>>
>>> Adding Eyal, Barak and Sandro and removing Victor.
>>>
>>> Personally, I do not mind taking on this task, but I think that I would
>>> need help in creating such a task.
>>>
>>
>> I think we are already mirroring everything and refreshing the mirror
>> every few hours.
>> Issue looks like we are not using them in some jobs.
>>
>
>
> I did not look deeply into this particular issue, but just to get everyone
> on the same page.
>
> * We have mirror system that is getting synced every 8 hours and has no
> known issues
>   ATM
> * All repo issues you're seeing in OST are due to one of the following two
> reasons:
>   1. We are white-listing packages into the OST environment and the list
> needs to
>  be maintained as package dependencies change
>   2. The OST VMs are not blocked from using the upstream CentOS
> repos/mirrors. And
>  the upstream repos are not updated in an atomic fashion
>
> We have ongoing work [1] to fix issue #2 above, it takes time because it
> requires meticulous work to get all the required things into the whitelist.
>

Since I send here and there patches to do this meticulous work, I know it's
not such a big deal.
Yes, it's annoying and I have not yet come up with an automated way to do
it (I'm sure there is!), but it takes few hours and we can do it once every
2 weeks or so.
It also has the nice benefit of reducing run time, sometimes dramatically.
Y.


>
> BTW when you see these issues in OST that are doe to upstream CentOS repos
> not being updated atomically, it usually correlates with a similar failure
> in the mirror sync job.
>
> [1]: https://ovirt-jira.atlassian.net/browse/OVIRT-1280
>
> --
> Barak Korren
> RHV DevOps team , RHCE, RHCi
> Red Hat EMEA
> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]

2017-10-15 Thread Barak Korren 
On 13 October 2017 at 14:31, Sandro Bonazzola  wrote:

>
> 2017-10-13 11:29 GMT+02:00 Dafna Ron :
>
>> Adding Eyal, Barak and Sandro and removing Victor.
>>
>> Personally, I do not mind taking on this task, but I think that I would
>> need help in creating such a task.
>>
>
> I think we are already mirroring everything and refreshing the mirror
> every few hours.
> Issue looks like we are not using them in some jobs.
>


I did not look deeply into this particular issue, but just to get everyone
on the same page.

* We have mirror system that is getting synced every 8 hours and has no
known issues
  ATM
* All repo issues you're seeing in OST are due to one of the following two
reasons:
  1. We are white-listing packages into the OST environment and the list
needs to
 be maintained as package dependencies change
  2. The OST VMs are not blocked from using the upstream CentOS
repos/mirrors. And
 the upstream repos are not updated in an atomic fashion

We have ongoing work [1] to fix issue #2 above, it takes time because it
requires meticulous work to get all the required things into the whitelist.

BTW when you see these issues in OST that are doe to upstream CentOS repos
not being updated atomically, it usually correlates with a similar failure
in the mirror sync job.

[1]: https://ovirt-jira.atlassian.net/browse/OVIRT-1280

-- 
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]

2017-10-13 Thread Sandro Bonazzola 
2017-10-13 11:29 GMT+02:00 Dafna Ron :

> Adding Eyal, Barak and Sandro and removing Victor.
>
> Personally, I do not mind taking on this task, but I think that I would
> need help in creating such a task.
>

I think we are already mirroring everything and refreshing the mirror every
few hours.
Issue looks like we are not using them in some jobs.



>
> On 10/12/2017 06:51 PM, Yaniv Kaul wrote:
>
>
>
> On Thu, Oct 12, 2017 at 2:36 PM, Dafna Ron  wrote:
>
>> Thank you.
>> I was not sure if this is repo related since I could not see a specific
>> package that it failed on.
>> Sandro believes this might have been a repo outage so I am opening a
>> ticket to try and investigate this issue and find the root cause.
>>
>> https://ovirt-jira.atlassian.net/browse/OVIRT-1693
>
>
> We have far too many repo outages.
> I believe it could be partially solved by properly and consistently
> keeping the reposync up-to-date.
> It's far from bullet-proof, and is annoying work, but we need to once
> every other week or so to just do it, to ensure we can perform offline
> installation (I don't believe it's a complete repo outage, but partial,
> which is why I think it'll help).
> Y.
>
>
>>
>>
>> Thanks,
>> Dafna
>>
>>
>> On 10/12/2017 11:37 AM, Viktor Mihajlovski wrote:
>> > On 12.10.2017 12:27, Yaniv Kaul wrote:
>> >> Repo issues (again?)
>> >> See log[1].
>> >> Y.
>> >>
>> >> [1]
>> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste
>> r/3104/artifact/exported-artifacts/basic-suit-master-el7/
>> 003_00_metrics_bootstrap.py.junit.xml
>> >>
>> >> On Thu, Oct 12, 2017 at 12:26 PM, Dafna Ron  wrote:
>> >>
>> > I agree, seems to be unrelated to my patch.
>> >
>>
>>
>
>


-- 

SANDRO BONAZZOLA

ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R

Red Hat EMEA 

TRIED. TESTED. TRUSTED. 

___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]

2017-10-13 Thread Dafna Ron 
Adding Eyal, Barak and Sandro and removing Victor.

Personally, I do not mind taking on this task, but I think that I would
need help in creating such a task.

On 10/12/2017 06:51 PM, Yaniv Kaul wrote:
>
>
> On Thu, Oct 12, 2017 at 2:36 PM, Dafna Ron  > wrote:
>
> Thank you.
> I was not sure if this is repo related since I could not see a
> specific
> package that it failed on.
> Sandro believes this might have been a repo outage so I am opening a
> ticket to try and investigate this issue and find the root cause.
>
> https://ovirt-jira.atlassian.net/browse/OVIRT-1693
> 
>
>
> We have far too many repo outages.
> I believe it could be partially solved by properly and consistently
> keeping the reposync up-to-date.
> It's far from bullet-proof, and is annoying work, but we need to once
> every other week or so to just do it, to ensure we can perform offline
> installation (I don't believe it's a complete repo outage, but
> partial, which is why I think it'll help).
> Y.
>  
>
>
>
> Thanks,
> Dafna
>
>
> On 10/12/2017 11:37 AM, Viktor Mihajlovski wrote:
> > On 12.10.2017 12:27, Yaniv Kaul wrote:
> >> Repo issues (again?)
> >> See log[1].
> >> Y.
> >>
> >> [1]
> >>
> 
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3104/artifact/exported-artifacts/basic-suit-master-el7/003_00_metrics_bootstrap.py.junit.xml
> 
> 
> >>
> >> On Thu, Oct 12, 2017 at 12:26 PM, Dafna Ron  > wrote:
> >>
> > I agree, seems to be unrelated to my patch.
> >
>
>

___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]

2017-10-12 Thread Yaniv Kaul 
On Thu, Oct 12, 2017 at 8:51 PM, Yaniv Kaul  wrote:

>
>
> On Thu, Oct 12, 2017 at 2:36 PM, Dafna Ron  wrote:
>
>> Thank you.
>> I was not sure if this is repo related since I could not see a specific
>> package that it failed on.
>> Sandro believes this might have been a repo outage so I am opening a
>> ticket to try and investigate this issue and find the root cause.
>>
>> https://ovirt-jira.atlassian.net/browse/OVIRT-1693
>
>
> We have far too many repo outages.
> I believe it could be partially solved by properly and consistently
> keeping the reposync up-to-date.
> It's far from bullet-proof, and is annoying work, but we need to once
> every other week or so to just do it, to ensure we can perform offline
> installation (I don't believe it's a complete repo outage, but partial,
> which is why I think it'll help).
>

Spoke too soon:
yum.Errors.NoMoreMirrorsRepoError: failure: repodata/repomd.xml from
centos-ovirt-4.2-el7: [Errno 256] No more mirrors to try.
http://cbs.centos.org/repos/virt7-ovirt-42-testing/x86_64/os/repodata/repomd.xml:
[Errno 14] HTTP Error 404 - Not Found


Perhaps we are not using mirror links properly.
Y.



> Y.
>
>
>>
>>
>> Thanks,
>> Dafna
>>
>>
>> On 10/12/2017 11:37 AM, Viktor Mihajlovski wrote:
>> > On 12.10.2017 12:27, Yaniv Kaul wrote:
>> >> Repo issues (again?)
>> >> See log[1].
>> >> Y.
>> >>
>> >> [1]
>> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste
>> r/3104/artifact/exported-artifacts/basic-suit-master-el7/
>> 003_00_metrics_bootstrap.py.junit.xml
>> >>
>> >> On Thu, Oct 12, 2017 at 12:26 PM, Dafna Ron  wrote:
>> >>
>> > I agree, seems to be unrelated to my patch.
>> >
>>
>>
>
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]

2017-10-12 Thread Yaniv Kaul 
On Thu, Oct 12, 2017 at 2:36 PM, Dafna Ron  wrote:

> Thank you.
> I was not sure if this is repo related since I could not see a specific
> package that it failed on.
> Sandro believes this might have been a repo outage so I am opening a
> ticket to try and investigate this issue and find the root cause.
>
> https://ovirt-jira.atlassian.net/browse/OVIRT-1693


We have far too many repo outages.
I believe it could be partially solved by properly and consistently keeping
the reposync up-to-date.
It's far from bullet-proof, and is annoying work, but we need to once every
other week or so to just do it, to ensure we can perform offline
installation (I don't believe it's a complete repo outage, but partial,
which is why I think it'll help).
Y.


>
>
> Thanks,
> Dafna
>
>
> On 10/12/2017 11:37 AM, Viktor Mihajlovski wrote:
> > On 12.10.2017 12:27, Yaniv Kaul wrote:
> >> Repo issues (again?)
> >> See log[1].
> >> Y.
> >>
> >> [1]
> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-
> tester/3104/artifact/exported-artifacts/basic-suit-master-
> el7/003_00_metrics_bootstrap.py.junit.xml
> >>
> >> On Thu, Oct 12, 2017 at 12:26 PM, Dafna Ron  wrote:
> >>
> > I agree, seems to be unrelated to my patch.
> >
>
>
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]

2017-10-12 Thread Dafna Ron 
Thank you.
I was not sure if this is repo related since I could not see a specific
package that it failed on.
Sandro believes this might have been a repo outage so I am opening a
ticket to try and investigate this issue and find the root cause.

https://ovirt-jira.atlassian.net/browse/OVIRT-1693

Thanks,
Dafna


On 10/12/2017 11:37 AM, Viktor Mihajlovski wrote:
> On 12.10.2017 12:27, Yaniv Kaul wrote:
>> Repo issues (again?)
>> See log[1].
>> Y.
>>
>> [1]
>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3104/artifact/exported-artifacts/basic-suit-master-el7/003_00_metrics_bootstrap.py.junit.xml
>>
>> On Thu, Oct 12, 2017 at 12:26 PM, Dafna Ron  wrote:
>>
> I agree, seems to be unrelated to my patch.
>

___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 12-10-2017 ] [003_00_metrics_bootstrap.configure_metrics ]

2017-10-12 Thread Yaniv Kaul 
Repo issues (again?)
See log[1].
Y.

[1]
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3104/artifact/exported-artifacts/basic-suit-master-el7/003_00_metrics_bootstrap.py.junit.xml

On Thu, Oct 12, 2017 at 12:26 PM, Dafna Ron  wrote:

> Hi,
>
> We had a failure to configure metrics in ovirt-system-tests which caused
> metrics_bootstrap to fail.
>
> The patch that was reported as the cause is below.
>
> *Link to suspected patches: https://gerrit.ovirt.org/#/c/82686/
> *
>
>
>
> * Link to Job:
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3104/
>  Link
> to all logs:
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3104/artifact/
> 
> (Relevant) error snippet from the log:  *
>
>  File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
> testMethod()
>   File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
> self.test(*self.arg)
>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in 
> wrapped_test
> test()
>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in 
> wrapper
> return func(get_test_prefix(), *args, **kwargs)
>   File 
> "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/003_00_metrics_bootstrap.py",
>  line 53, in configure_metrics
> ' Exit code is %s' % result.code
>   File "/usr/lib/python2.7/site-packages/nose/tools/trivial.py", line 29, in 
> eq_
> raise AssertionError(msg or "%r != %r" % (a, b))
> 'Configuring ovirt machines for metrics failed. Exit code is 2\n--
>
> **
>
>
> ___
> Devel mailing list
> de...@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra