Re: [ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 26.02.2017 ] [test-repo_ovirt_experimental_master]

2017-02-26 Thread Piotr Kliczewski 
Looking at the logs I see that the issue occurred when we were in
setupNetworks and in between we called getStats.

We called:

2017-02-26 05:43:44,655-05 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient]
(org.ovirt.thread.pool-7-thread-4) [] Message sent: SEND
destination:jms.topic.vdsm_requests
content-length:370
reply-to:jms.topic.vdsm_responses



in between we see:

2017-02-26 05:43:51,741-05 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message]
(DefaultQuartzScheduler5) [63e30973] SEND
destination:jms.topic.vdsm_requests
reply-to:jms.topic.vdsm_responses
content-length:98

{"jsonrpc":"2.0","method":"Host.getStats","params":{},"id":"27ffe862-ddda-46a1-a59f-5c5e3478a436"}\00

and setupNetworks response arrived at:

2017-02-26 05:43:51,762-05 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp
Reactor) [40fe71c] MESSAGE
content-length:106
destination:jms.topic.vdsm_responses
content-type:application/json
subscription:c1dfc84f-c309-4ea0-8f1c-92462aacbd8e

{"jsonrpc": "2.0", "id": "8eea12f1-f41a-4174-8032-03f34fa2e806", "result":
{"message": "Done", "code": 0}}\00

Response to Host.getStats arrived only partially. I thought that we hold
host level lock to stop host monitoring to run in parallel with
setupNetworks.

On Sun, Feb 26, 2017 at 2:23 PM, Nir Soffer  wrote:

> On Sun, Feb 26, 2017 at 3:18 PM, Yaniv Kaul  wrote:
>
>> On Sun, Feb 26, 2017 at 3:04 PM Shlomo Ben David 
>> wrote:
>>
>> Hi,
>>
>>
>> Test failed: [ test-repo_ovirt_experimental_master ]
>>
>> Link to Job: [1]
>>
>> Link to all logs: [2]
>>
>> Link to error log: [3]
>>
>>
>> [1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538
>>
>> [2] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma
>> ster/5538/artifact/exported-artifacts/basic-suit-master-el7/
>> test_logs/basic-suite-master/post-006_migrations.py/
>>
>> [3] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma
>> ster/5538/artifact/exported-artifacts/basic-suit-master-el7/
>> nosetests-006_migrations.py.xml
>>
>> Error snippet from the log:
>>
>>
>> The below is not the issue. The issue is (engine log):
>> 2017-02-26 05:43:52,178-05 ERROR 
>> [org.ovirt.engine.core.bll.network.host.HostValidator]
>> (default task-11) [d3b3a59d-6cc0-4896-b3f2-8483f9b77fe2] Unable to setup
>> network: operation can only be done when Host status is one of:
>> Maintenance, Up, NonOperational; current status is Connecting
>>
>> And it comes from the host disconnecting a bit from engine, with various
>> errors such as (again, engine log):
>>
>>
>> 2017-02-26 05:43:26,763-05 ERROR [org.ovirt.engine.core.dal.dbb
>> roker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5)
>> [63e30973] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID:
>> null, Call Stack: null, Custom Event ID: -1, Message: VDSM
>> lago-basic-suite-master-host1 *command FullListVDS failed: Unrecognized
>> message received*
>>
>
> The error Shlomo posted may not be the error failing the test, but is is
> very important error that must not be in vdsm logs.
>
> Nir
>
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

Re: [ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 26.02.2017 ] [test-repo_ovirt_experimental_master]

2017-02-26 Thread Nir Soffer 
On Sun, Feb 26, 2017 at 3:18 PM, Yaniv Kaul  wrote:

> On Sun, Feb 26, 2017 at 3:04 PM Shlomo Ben David 
> wrote:
>
> Hi,
>
>
> Test failed: [ test-repo_ovirt_experimental_master ]
>
> Link to Job: [1]
>
> Link to all logs: [2]
>
> Link to error log: [3]
>
>
> [1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538
>
> [2] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
> master/5538/artifact/exported-artifacts/basic-suit-master-
> el7/test_logs/basic-suite-master/post-006_migrations.py/
>
> [3] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
> master/5538/artifact/exported-artifacts/basic-suit-master-
> el7/nosetests-006_migrations.py.xml
>
> Error snippet from the log:
>
>
> The below is not the issue. The issue is (engine log):
> 2017-02-26 05:43:52,178-05 ERROR 
> [org.ovirt.engine.core.bll.network.host.HostValidator]
> (default task-11) [d3b3a59d-6cc0-4896-b3f2-8483f9b77fe2] Unable to setup
> network: operation can only be done when Host status is one of:
> Maintenance, Up, NonOperational; current status is Connecting
>
> And it comes from the host disconnecting a bit from engine, with various
> errors such as (again, engine log):
>
>
> 2017-02-26 05:43:26,763-05 ERROR [org.ovirt.engine.core.dal.
> dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5)
> [63e30973] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID:
> null, Call Stack: null, Custom Event ID: -1, Message: VDSM
> lago-basic-suite-master-host1 *command FullListVDS failed: Unrecognized
> message received*
>

The error Shlomo posted may not be the error failing the test, but is is
very important error that must not be in vdsm logs.

Nir
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

Re: [ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 26.02.2017 ] [test-repo_ovirt_experimental_master]

2017-02-26 Thread Yaniv Kaul 
On Sun, Feb 26, 2017 at 3:04 PM Shlomo Ben David 
wrote:

Hi,


Test failed: [ test-repo_ovirt_experimental_master ]

Link to Job: [1]

Link to all logs: [2]

Link to error log: [3]


[1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538

[2]
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-006_migrations.py/

[3]
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538/artifact/exported-artifacts/basic-suit-master-el7/nosetests-006_migrations.py.xml

Error snippet from the log:


The below is not the issue. The issue is (engine log):
2017-02-26 05:43:52,178-05 ERROR
[org.ovirt.engine.core.bll.network.host.HostValidator] (default task-11)
[d3b3a59d-6cc0-4896-b3f2-8483f9b77fe2] Unable to setup network: operation
can only be done when Host status is one of: Maintenance, Up,
NonOperational; current status is Connecting

And it comes from the host disconnecting a bit from engine, with various
errors such as (again, engine log):


2017-02-26 05:43:26,763-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler5) [63e30973] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null,
Custom Event ID: -1, Message: VDSM lago-basic-suite-master-host1 *command
FullListVDS failed: Unrecognized message received*


Y.





2017-02-26 05:35:53,340-0500 ERROR (jsonrpc/3)
[storage.TaskManager.Task]
(Task='22828901-d87f-4607-9690-a106c474ebe4') Unexpected error
(task:871)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line
878, in _run
return fn(*args, **kargs)
  File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 52, in wrapper
res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 3200, in teardownImage
dom.deactivateImage(imgUUID)
  File "/usr/share/vdsm/storage/blockSD.py", line 1278, in deactivateImage
lvm.deactivateLVs(self.sdUUID, volUUIDs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line
1304, in deactivateLVs
_setLVAvailability(vgName, toDeactivate, "n")
  File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line
845, in _setLVAvailability
raise error(str(e))
CannotDeactivateLogicalVolume: Cannot deactivate Logical Volume:
('General Storage Exception: ("5 [] [\'  Logical volume
1dd0ee2a-f26a-423c-90f9-79703343aa1e/8323e511-eb93-4b1c-a9fa-ad66409994e7
in use.\', \'  Logical volume
1dd0ee2a-f26a-423c-90f9-79703343aa1e/99662dfa-acf2-4392-8ab8-106412c2afa5
in 
use.\']\\n1dd0ee2a-f26a-423c-90f9-79703343aa1e/[\'99662dfa-acf2-4392-8ab8-106412c2afa5\',
\'8323e511-eb93-4b1c-a9fa-ad66409994e7\']",)',)
2017-02-26 05:35:53,347-0500 INFO  (jsonrpc/3)
[storage.TaskManager.Task]
(Task='22828901-d87f-4607-9690-a106c474ebe4') aborting: Task is
aborted: 'Cannot deactivate Logical Volume' - code 552 (task:1176)
2017-02-26 05:35:53,348-0500 ERROR (jsonrpc/3) [storage.Dispatcher]
{'status': {'message': 'Cannot deactivate Logical Volume: (\'General
Storage Exception: ("5 [] [\\\'  Logical volume
1dd0ee2a-f26a-423c-90f9-79703343aa1e/8323e511-eb93-4b1c-a9fa-ad66409994e7
in use.\\\', \\\'  Logical volume
1dd0ee2a-f26a-423c-90f9-79703343aa1e/99662dfa-acf2-4392-8ab8-106412c2afa5
in 
use.\\\']n1dd0ee2a-f26a-423c-90f9-79703343aa1e/[\\\'99662dfa-acf2-4392-8ab8-106412c2afa5\\\',
\\\'8323e511-eb93-4b1c-a9fa-ad66409994e7\\\']",)\',)', 'code': 552}}
(dispatcher:78)
2017-02-26 05:35:53,349-0500 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer]
RPC call Image.teardown failed (error 552) in 19.36 seconds
(__init__:552)




Best Regards,

Shlomi Ben-David | Software Engineer | Red Hat ISRAEL
RHCSA | RHCVA | RHCE
IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)

OPEN SOURCE - 1 4 011 && 011 4 1

___
Devel mailing list
de...@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

Re: [ OST Failure Report ] [ oVirt master ] [ 26.02.2017 ] [test-repo_ovirt_experimental_master]

2017-02-26 Thread Nir Soffer 
On Sun, Feb 26, 2017 at 3:02 PM, Shlomo Ben David 
wrote:

> Hi,
>
>
> Test failed: [ test-repo_ovirt_experimental_master ]
>
> Link to Job: [1]
>
> Link to all logs: [2]
>
> Link to error log: [3]
>
>
> [1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538
>
> [2] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
> master/5538/artifact/exported-artifacts/basic-suit-master-
> el7/test_logs/basic-suite-master/post-006_migrations.py/
>
> [3] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
> master/5538/artifact/exported-artifacts/basic-suit-master-
> el7/nosetests-006_migrations.py.xml
>
> Error snippet from the log:
>
> 
>
> 2017-02-26 05:35:53,340-0500 ERROR (jsonrpc/3) [storage.TaskManager.Task] 
> (Task='22828901-d87f-4607-9690-a106c474ebe4') Unexpected error (task:871)
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 878, in 
> _run
> return fn(*args, **kargs)
>   File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 52, in 
> wrapper
> res = f(*args, **kwargs)
>   File "/usr/share/vdsm/storage/hsm.py", line 3200, in teardownImage
> dom.deactivateImage(imgUUID)
>   File "/usr/share/vdsm/storage/blockSD.py", line 1278, in deactivateImage
> lvm.deactivateLVs(self.sdUUID, volUUIDs)
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 1304, in 
> deactivateLVs
> _setLVAvailability(vgName, toDeactivate, "n")
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 845, in 
> _setLVAvailability
> raise error(str(e))
> CannotDeactivateLogicalVolume: Cannot deactivate Logical Volume: ('General 
> Storage Exception: ("5 [] [\'  Logical volume 
> 1dd0ee2a-f26a-423c-90f9-79703343aa1e/8323e511-eb93-4b1c-a9fa-ad66409994e7 in 
> use.\', \'  Logical volume 
> 1dd0ee2a-f26a-423c-90f9-79703343aa1e/99662dfa-acf2-4392-8ab8-106412c2afa5 in 
> use.\']\\n1dd0ee2a-f26a-423c-90f9-79703343aa1e/[\'99662dfa-acf2-4392-8ab8-106412c2afa5\',
>  \'8323e511-eb93-4b1c-a9fa-ad66409994e7\']",)',)
>
> This means the logical volume is still used when we try to deactivate it.

Logical volume
1dd0ee2a-f26a-423c-90f9-79703343aa1e/8323e511-eb93-4b1c-a9fa-ad66409994e7
in use

This is serious error in the code trying to deactivating the volume.

Is this reproducible or it happens randomly?

Please file a bug for this.


> 2017-02-26 05:35:53,347-0500 INFO  (jsonrpc/3) [storage.TaskManager.Task] 
> (Task='22828901-d87f-4607-9690-a106c474ebe4') aborting: Task is aborted: 
> 'Cannot deactivate Logical Volume' - code 552 (task:1176)
> 2017-02-26 05:35:53,348-0500 ERROR (jsonrpc/3) [storage.Dispatcher] 
> {'status': {'message': 'Cannot deactivate Logical Volume: (\'General Storage 
> Exception: ("5 [] [\\\'  Logical volume 
> 1dd0ee2a-f26a-423c-90f9-79703343aa1e/8323e511-eb93-4b1c-a9fa-ad66409994e7 in 
> use.\\\', \\\'  Logical volume 
> 1dd0ee2a-f26a-423c-90f9-79703343aa1e/99662dfa-acf2-4392-8ab8-106412c2afa5 in 
> use.\\\']n1dd0ee2a-f26a-423c-90f9-79703343aa1e/[\\\'99662dfa-acf2-4392-8ab8-106412c2afa5\\\',
>  \\\'8323e511-eb93-4b1c-a9fa-ad66409994e7\\\']",)\',)', 'code': 552}} 
> (dispatcher:78)
> 2017-02-26 05:35:53,349-0500 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC 
> call Image.teardown failed (error 552) in 19.36 seconds (__init__:552)
>
> 
>
>
> Best Regards,
>
> Shlomi Ben-David | Software Engineer | Red Hat ISRAEL
> RHCSA | RHCVA | RHCE
> IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
>
> OPEN SOURCE - 1 4 011 && 011 4 1
>
>
> ___
> Infra mailing list
> Infra@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
>
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

[ OST Failure Report ] [ oVirt master ] [ 26.02.2017 ] [test-repo_ovirt_experimental_master]

2017-02-26 Thread Shlomo Ben David 
Hi,


Test failed: [ test-repo_ovirt_experimental_master ]

Link to Job: [1]

Link to all logs: [2]

Link to error log: [3]


[1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538

[2]
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-006_migrations.py/

[3]
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538/artifact/exported-artifacts/basic-suit-master-el7/nosetests-006_migrations.py.xml

Error snippet from the log:



2017-02-26 05:35:53,340-0500 ERROR (jsonrpc/3)
[storage.TaskManager.Task]
(Task='22828901-d87f-4607-9690-a106c474ebe4') Unexpected error
(task:871)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line
878, in _run
return fn(*args, **kargs)
  File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 52, in wrapper
res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 3200, in teardownImage
dom.deactivateImage(imgUUID)
  File "/usr/share/vdsm/storage/blockSD.py", line 1278, in deactivateImage
lvm.deactivateLVs(self.sdUUID, volUUIDs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line
1304, in deactivateLVs
_setLVAvailability(vgName, toDeactivate, "n")
  File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line
845, in _setLVAvailability
raise error(str(e))
CannotDeactivateLogicalVolume: Cannot deactivate Logical Volume:
('General Storage Exception: ("5 [] [\'  Logical volume
1dd0ee2a-f26a-423c-90f9-79703343aa1e/8323e511-eb93-4b1c-a9fa-ad66409994e7
in use.\', \'  Logical volume
1dd0ee2a-f26a-423c-90f9-79703343aa1e/99662dfa-acf2-4392-8ab8-106412c2afa5
in 
use.\']\\n1dd0ee2a-f26a-423c-90f9-79703343aa1e/[\'99662dfa-acf2-4392-8ab8-106412c2afa5\',
\'8323e511-eb93-4b1c-a9fa-ad66409994e7\']",)',)
2017-02-26 05:35:53,347-0500 INFO  (jsonrpc/3)
[storage.TaskManager.Task]
(Task='22828901-d87f-4607-9690-a106c474ebe4') aborting: Task is
aborted: 'Cannot deactivate Logical Volume' - code 552 (task:1176)
2017-02-26 05:35:53,348-0500 ERROR (jsonrpc/3) [storage.Dispatcher]
{'status': {'message': 'Cannot deactivate Logical Volume: (\'General
Storage Exception: ("5 [] [\\\'  Logical volume
1dd0ee2a-f26a-423c-90f9-79703343aa1e/8323e511-eb93-4b1c-a9fa-ad66409994e7
in use.\\\', \\\'  Logical volume
1dd0ee2a-f26a-423c-90f9-79703343aa1e/99662dfa-acf2-4392-8ab8-106412c2afa5
in 
use.\\\']n1dd0ee2a-f26a-423c-90f9-79703343aa1e/[\\\'99662dfa-acf2-4392-8ab8-106412c2afa5\\\',
\\\'8323e511-eb93-4b1c-a9fa-ad66409994e7\\\']",)\',)', 'code': 552}}
(dispatcher:78)
2017-02-26 05:35:53,349-0500 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer]
RPC call Image.teardown failed (error 552) in 19.36 seconds
(__init__:552)




Best Regards,

Shlomi Ben-David | Software Engineer | Red Hat ISRAEL
RHCSA | RHCVA | RHCE
IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)

OPEN SOURCE - 1 4 011 && 011 4 1
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra