[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-29 Thread Dafna Ron
Thank you Marcin :) I am also adding Didi to see if he knows of any changes. Anton, who from Engine/api side can help to take it from here? On Fri, Mar 29, 2019 at 9:37 AM Marcin Sobczyk wrote: > Hi, > > so the thing boils down to this: > > >

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-26 Thread Dafna Ron
We never remove any packages from tested :) I am actually suspecting a problem that is caused by something that is shared by all projects like the api. we currently have 3 different tests that are randomly failing because of engine/host communication and I am starting to think they are all

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-26 Thread Eyal Edri
On Tue, Mar 26, 2019 at 3:56 PM Sandro Bonazzola wrote: > > > Il giorno mar 26 mar 2019 alle ore 14:42 Marcin Sobczyk < > msobc...@redhat.com> ha scritto: > >> Ok, maybe I've found something. I was comparing two sets of log files - >> one for basic-suite-4.3 run that succeeded and one that did

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-26 Thread Sandro Bonazzola
Il giorno mar 26 mar 2019 alle ore 14:42 Marcin Sobczyk ha scritto: > Ok, maybe I've found something. I was comparing two sets of log files - > one for basic-suite-4.3 run that succeeded and one that did not. > > The deployment procedure for host-1 is not complete in the unsuccessful > case. The

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-26 Thread Marcin Sobczyk
Ok, maybe I've found something. I was comparing two sets of log files - one for basic-suite-4.3 run that succeeded and one that did not. The deployment procedure for host-1 is not complete in the unsuccessful case. The last log entry is: 2019-03-22 17:54:49,973-04 INFO 

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-25 Thread Dafna Ron
yes. I suspect this is the problem. In the host I can see in the yum log the times the vdsm packages are installed: [dron@dron post-002_bootstrap.py]$ less lago-basic-suite-4-3-host-1/_var_log/yum.log |grep vdsm Mar 25 07:48:49 Installed: vdsm-api-4.30.11-23.git276b602.el7.noarch Mar 25 07:48:52

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-25 Thread Dafna Ron
I am also adding didi should this not be logged in host-deploy? On Mon, Mar 25, 2019 at 1:33 PM Marcin Sobczyk wrote: > Indeed in my failed job, I can see, that 'vdsm-python' is installed > *after* engine's attempt to run 'vdsm-tool': > > lago-basic-suite-4-3-host-1/_var_log/yum.log > 403:Mar

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-25 Thread Marcin Sobczyk
Indeed in my failed job, I can see, that 'vdsm-python' is installed *after* engine's attempt to run 'vdsm-tool': lago-basic-suite-4-3-host-1/_var_log/yum.log 403:Mar 22 17:54:12 Installed: vdsm-python-4.30.11-23.git276b602.el7.noarch lago-basic-suite-4-3-engine/_var_log/ovirt-engine/engine.log

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-25 Thread Dafna Ron
and its only failing on one host as well. On Mon, Mar 25, 2019 at 1:17 PM Marcin Sobczyk wrote: > Hmmm, that would suggest, that 'vdsm-tool' command is not found on the > host. There was no work done around 'vdsm-tool's 'vdsm-id' recently. > On 3/25/19 2:08 PM, Dafna Ron wrote: > > Actually, I

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-25 Thread Marcin Sobczyk
Hmmm, that would suggest, that 'vdsm-tool' command is not found on the host. There was no work done around 'vdsm-tool's 'vdsm-id' recently. On 3/25/19 2:08 PM, Dafna Ron wrote: Actually, I can see in engine that it is trying to start vdsm installation on the host: 2019-03-25 07:48:33,123-04

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-25 Thread Dafna Ron
Actually, I can see in engine that it is trying to start vdsm installation on the host: 2019-03-25 07:48:33,123-04 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-1) [] Entered SsoRestApiAuthFilter 2019-03-25 07:48:33,123-04 DEBUG

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-25 Thread Dafna Ron
we have another failing patch with the same test: http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/360/ its obviously not related to the patch but there is something that is causing these failures randomly. >From what I can see in the current failed job, you are correct and host-1 and

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-25 Thread Marcin Sobczyk
For the failed job, the engine didn't even try to deploy on host-1: https://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/339/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-4.3/post-002_bootstrap.py/lago-basic-suite-4-3-engine/_var_log/ovirt-engine/host-deploy/ Martin, do you know

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-25 Thread Greg Sheremeta
Not related to ui-extensions (which is very frontend only, Javascript project) https://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/339/ > > On Fri, Mar 22, 2019 at 3:59 PM Dan Kenigsberg wrote: > >> >> >> On Fri, Mar 22, 2019 at 3:21 PM Marcin Sobczyk >> wrote: >> >>> Dafna, >>>

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-25 Thread Marcin Sobczyk
Hi all, After investigating, I really don't think my patch is the cause of the issue: - my patch touched 'vdsm-client' only and failing stages AFAICT have nothing to do with it - even now, I run basic-suite-4.3 successfully -

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-25 Thread Eyal Edri
On Mon, Mar 25, 2019 at 12:20 PM Greg Sheremeta wrote: > Not related to ui-extensions (which is very frontend only, Javascript > project) > I know its not, but this is just to surface that these failures needs to be investigated in depth by a relevant team in development to find root cause,

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-25 Thread Eyal Edri
Still fails, now on a different component. ( ovirt-web-ui-extentions ) https://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/339/ On Fri, Mar 22, 2019 at 3:59 PM Dan Kenigsberg wrote: > > > On Fri, Mar 22, 2019 at 3:21 PM Marcin Sobczyk > wrote: > >> Dafna, >> >> in 'verify_add_hosts'

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-22 Thread Dan Kenigsberg
On Fri, Mar 22, 2019 at 3:21 PM Marcin Sobczyk wrote: > Dafna, > > in 'verify_add_hosts' we specifically wait for single host to be up with a > timeout: > > 144 up_hosts = hosts_service.list(search='datacenter={} AND > status=up'.format(DC_NAME)) > 145 if len(up_hosts): > 146

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-22 Thread Marcin Sobczyk
Dafna, in 'verify_add_hosts' we specifically wait for single host to be up with a timeout:  144 up_hosts = hosts_service.list(search='datacenter={} AND status=up'.format(DC_NAME))  145 if len(up_hosts):  146 return True The log files say, that it took ~50 secs for one of the

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-22 Thread Dafna Ron
I wonder if there were any changes that merged around the same time that could have caused it? On Fri, Mar 22, 2019 at 12:17 PM Marcin Sobczyk wrote: > Hi, > > sure, I'm on it - it's weird though, I did ran 4.3 basic suite for this > patch manually and everything was ok. > On 3/22/19 1:05 PM,

[ovirt-devel] Re: [ OST Failure Report ] [ oVirt 4.3 (vdsm) ] [ 22-03-2019 ] [ 002_bootstrap.add_master_storage_domain ]

2019-03-22 Thread Marcin Sobczyk
Hi, sure, I'm on it - it's weird though, I did ran 4.3 basic suite for this patch manually and everything was ok. On 3/22/19 1:05 PM, Dafna Ron wrote: Hi, We are failing branch 4.3 for test: 002_bootstrap.add_master_storage_domain It seems that in one of the hosts, the vdsm is not