Re: [ OST Failure Report ] [ oVirt 4.1 ] [ 09/02/17 ] [ basic_sanity.hotplug_nic ]

2017-03-12 Thread Dan Kenigsberg 
For the record, we have realized that this failure is unrelated to the
log supplied in bug
https://bugzilla.redhat.com/show_bug.cgi?id=1417595 .

That log is due to a test requesting dhcp address, not receiving it,
and carrying on to next tests, while a Vdsm thread is still waiting
for dhcp.

The cause for this breakage is probably a broken jsonrpc connection,
hopefully fixed by
https://gerrit.ovirt.org/#/q/Idaa54767bb7e54bf13e89887ca34fa8e01ade420

On Thu, Mar 9, 2017 at 3:06 PM, Dan Kenigsberg  wrote:
> On Thu, Mar 9, 2017 at 10:00 AM, Daniel Belenky  wrote:
>> Test failed: basic_sanity.hotplug_nic
>>
>> Link to failed job: test-repo_ovirt_experimental_4.1/917/
>>
>> Link to all logs: logs from Jenkins
>>
>> May be related to: https://bugzilla.redhat.com/show_bug.cgi?id=1417595
>>
>> Error snippet from log: (from supervdsm.log):
>>
>> ifup/VLAN100_Network::DEBUG::2017-03-08
>> 22:24:22,832::commands::93::root::(execCmd) FAILED:  = 'Running scope
>> as unit
>> ff994cde-b66d-4d54-9e36-fc253682608f.scope.\n/etc/sysconfig/network-scripts/ifup-eth:
>> line 297: 16985 Terminated  /sbin/dhclient ${DHCLIENTARGS}
>> ${DEVICE}\nCannot find device "VLAN100_Network"\nDevice "VLAN100_Network"
>> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
>> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
>> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
>> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
>> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
>> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
>> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
>> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
>> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
>> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
>> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
>> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
>> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
>> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
>> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
>> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
>> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
>> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
>> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
>> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
>> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
>> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
>> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
>> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
>> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
>> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
>> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
>> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
>> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
>> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
>> does not exist.\n';  = 1
>> ifup/VLAN100_Network::ERROR::2017-03-08
>> 22:24:22,832::utils::371::root::(wrapper) Unhandled exception
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 368, in
>> wrapper
>> return f(*a, **kw)
>>   File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 180, in
>> run
>> return func(*args, **kwargs)
>>   File
>> "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line
>> 924, in _exec_ifup
>> _exec_ifup_by_name(iface.name, cgroup)
>>   File
>> "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line
>> 910, in _exec_ifup_by_name
>> raise ConfigNetworkError(ERR_FAILED_IFUP, out[-1] if out else '')
>> ConfigNetworkError: (29, 'Determining IPv6 information for
>> VLAN100_Network... failed.')
>
> Yes, this seems to be a reproduction of Vdsm bug 1417595 - hotplug_nic
> test fails in OST
>
> Eddy, can you tell how could it be that the VLAN100_NETWORK bridge
> does not exist when ifup is called?
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

Re: [ OST Failure Report ] [ oVirt 4.1 ] [ 09/02/17 ] [ basic_sanity.hotplug_nic ]

2017-03-09 Thread Dan Kenigsberg 
On Thu, Mar 9, 2017 at 10:00 AM, Daniel Belenky  wrote:
> Test failed: basic_sanity.hotplug_nic
>
> Link to failed job: test-repo_ovirt_experimental_4.1/917/
>
> Link to all logs: logs from Jenkins
>
> May be related to: https://bugzilla.redhat.com/show_bug.cgi?id=1417595
>
> Error snippet from log: (from supervdsm.log):
>
> ifup/VLAN100_Network::DEBUG::2017-03-08
> 22:24:22,832::commands::93::root::(execCmd) FAILED:  = 'Running scope
> as unit
> ff994cde-b66d-4d54-9e36-fc253682608f.scope.\n/etc/sysconfig/network-scripts/ifup-eth:
> line 297: 16985 Terminated  /sbin/dhclient ${DHCLIENTARGS}
> ${DEVICE}\nCannot find device "VLAN100_Network"\nDevice "VLAN100_Network"
> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
> does not exist.\nDevice "VLAN100_Network" does not exist.\nDevice
> "VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
> exist.\nDevice "VLAN100_Network" does not exist.\nDevice "VLAN100_Network"
> does not exist.\n';  = 1
> ifup/VLAN100_Network::ERROR::2017-03-08
> 22:24:22,832::utils::371::root::(wrapper) Unhandled exception
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 368, in
> wrapper
> return f(*a, **kw)
>   File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 180, in
> run
> return func(*args, **kwargs)
>   File
> "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line
> 924, in _exec_ifup
> _exec_ifup_by_name(iface.name, cgroup)
>   File
> "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line
> 910, in _exec_ifup_by_name
> raise ConfigNetworkError(ERR_FAILED_IFUP, out[-1] if out else '')
> ConfigNetworkError: (29, 'Determining IPv6 information for
> VLAN100_Network... failed.')

Yes, this seems to be a reproduction of Vdsm bug 1417595 - hotplug_nic
test fails in OST

Eddy, can you tell how could it be that the VLAN100_NETWORK bridge
does not exist when ifup is called?
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

[ OST Failure Report ] [ oVirt 4.1 ] [ 09/02/17 ] [ basic_sanity.hotplug_nic ]

2017-03-09 Thread Daniel Belenky 
*Test failed: *basic_sanity.hotplug_nic

*Link to failed job: *test-repo_ovirt_experimental_4.1/917/


*Link to all logs: *logs from Jenkins


*May be related to: *https://bugzilla.redhat.com/show_bug.cgi?id=1417595

*Error snippet from log: *(from supervdsm.log):

ifup/VLAN100_Network::DEBUG::2017-03-08
22:24:22,832::commands::93::root::(execCmd) FAILED:  = 'Running
scope as unit 
ff994cde-b66d-4d54-9e36-fc253682608f.scope.\n/etc/sysconfig/network-scripts/ifup-eth:
line 297: 16985 Terminated  /sbin/dhclient ${DHCLIENTARGS}
${DEVICE}\nCannot find device "VLAN100_Network"\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\nDevice
"VLAN100_Network" does not exist.\nDevice "VLAN100_Network" does not
exist.\nDevice "VLAN100_Network" does not exist.\n';  = 1
ifup/VLAN100_Network::ERROR::2017-03-08
22:24:22,832::utils::371::root::(wrapper) Unhandled exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 368, in wrapper
return f(*a, **kw)
  File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 180, in run
return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py",
line 924, in _exec_ifup
_exec_ifup_by_name(iface.name, cgroup)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py",
line 910, in _exec_ifup_by_name
raise ConfigNetworkError(ERR_FAILED_IFUP, out[-1] if out else '')
ConfigNetworkError: (29, 'Determining IPv6 information for
VLAN100_Network... failed.')

-- 

*Daniel Belenky*

*RHV DevOps*

*Red Hat Israel*
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra