On Tue, Jan 14, 2020 at 4:33 AM Jayme <jay...@gmail.com> wrote:
> My cluster appears to be experiencing an SPM problem. I recently placed
> each host in maintenance to move the ovirt management network to another
> interface. All was successful and all VMs are currently running. However,
> I'm not facing an SPM contending loop with data center going in and out of
> responsive status.
>
> I have a 3 server HCI setup and all volumes are active and healed, there
> are no unsynced entries or split brains.
>
> Does anyone know how I could diagnose the SPM issue?
>
Can you please try to select the host from web admin and make it SPM
manually ???
>
> engine.log:
>
> 2020-01-13 22:24:54,777-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
> (DefaultQuartzScheduler2) [213adf4f] START,
> GlusterTasksListVDSCommand(HostName = Orchard0,
> VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
> log id: 349f80a9
> 2020-01-13 22:24:55,231-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
> (DefaultQuartzScheduler2) [213adf4f] FINISH, GlusterTasksListVDSCommand,
> return: [], log id: 349f80a9
> 2020-01-13 22:24:58,245-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
> (DefaultQuartzScheduler3) [4f66c75b] START,
> GlusterServersListVDSCommand(HostName = Orchard0,
> VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
> log id: 7b04f110
> 2020-01-13 22:24:58,887-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Command
> 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
> return value '
> TaskStatusListReturn:{status='Status [code=654, message=Not SPM]'}
> '
> 2020-01-13 22:24:58,888-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] HostName = Orchard1
> 2020-01-13 22:24:58,888-04 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Command
> 'HSMGetAllTasksStatusesVDSCommand(HostName = Orchard1,
> VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'})'
> execution failed: IRSGenericException: IRSErrorException:
> IRSNonOperationalException: Not SPM
> 2020-01-13 22:24:59,034-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
> (DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterServersListVDSCommand,
> return: [10.12.0.220/24:CONNECTED, orchard1.grove.silverorange.com:CONNECTED,
> orchard2.grove.silverorange.com:DISCONNECTED], log id: 7b04f110
> 2020-01-13 22:24:59,049-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
> (DefaultQuartzScheduler3) [4f66c75b] START,
> GlusterServersListVDSCommand(HostName = Orchard2,
> VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
> log id: 43f1dd82
> 2020-01-13 22:24:59,099-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START,
> ConnectStoragePoolVDSCommand(HostName = Orchard1,
> ConnectStoragePoolVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d',
> vdsId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d',
> storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', masterVersion='1'}),
> log id: 2b397b31
> 2020-01-13 22:24:59,099-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Executing with
> domain map: {edc68a7c-7604-47e6-89bc-3738d727e8fc=active,
> 23c22a0f-0482-425e-8ada-730cf8ec0751=active,
> 390c0320-e843-4ff3-a4bb-a9973058447f=active,
> fb43d33a-82c8-44cb-8169-090cd0d8f56e=active,
> d70b171e-7488-4d52-8cad-bbc581dbf16e=active,
> 1f2e9989-9ab3-43d5-971d-568b8feca918=active}
> 2020-01-13 22:24:59,850-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
> (DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterServersListVDSCommand,
> return: [10.12.0.222/24:CONNECTED, 10.11.0.220:CONNECTED,
> orchard1.grove.silverorange.com:CONNECTED], log id: 43f1dd82
> 2020-01-13 22:24:59,852-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> (DefaultQuartzScheduler3) [4f66c75b] START,
> GlusterVolumesListVDSCommand(HostName = Orchard0,
> GlusterVolumesListVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
> log id: 263be6f8
> 2020-01-13 22:25:00,019-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH,
> ConnectStoragePoolVDSCommand, return: , log id: 2b397b31
> 2020-01-13 22:25:00,036-04 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) []
> hostFromVds::selectedVds - 'Orchard1', spmStatus 'Free', storage pool
> 'Default', storage pool version '4.3'
> 2020-01-13 22:25:00,056-04 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] starting spm on vds
> 'Orchard1', storage pool 'Default', prevId '-1', LVER '-1'
> 2020-01-13 22:25:00,057-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START,
> SpmStartVDSCommand(HostName = Orchard1,
> SpmStartVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d',
> storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', prevId='-1',
> prevLVER='-1', storagePoolFormatType='V4', recoveryMode='Manual',
> SCSIFencing='false'}), log id: 3dea111a
> 2020-01-13 22:25:00,065-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling
> started: taskId '671d5904-e062-4d45-9eb4-83a6f13657fe'
> 2020-01-13 22:25:00,500-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> (DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterVolumesListVDSCommand,
> return:
> {3f8f6a0f-aed4-48e3-9129-18a2a3f64eef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@743f4102,
> 71ff56d9-79b8-445d-b637-72ffc974f109=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@cad5f8f4,
> 752a9438-cd11-426c-b384-bc3c5f86ed07=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@f9f55499,
> c3e7447e-8514-4e4a-9ff5-a648fe6aa537=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7a2c16da,
> 79e8e93c-57c8-4541-a360-726cec3790cf=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@33e1ab37,
> 095fd8fc-5322-4741-8805-fc0bb64b554f=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b28195c0},
> log id: 263be6f8
> 2020-01-13 22:25:03,089-04 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Failed in
> 'HSMGetTaskStatusVDS' method
> 2020-01-13 22:25:03,090-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling
> ended: taskId '671d5904-e062-4d45-9eb4-83a6f13657fe' task status 'finished'
> 2020-01-13 22:25:03,090-04 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Start SPM Task
> failed - result: 'cleanSuccess', message: VDSGenericException:
> VDSErrorException: Failed to HSMGetTaskStatusVDS, error = TaskManager
> error, unable to add task: ('Task id already in use:
> 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',), code = 100
> 2020-01-13 22:25:03,104-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling
> ended, spm status: Free
> 2020-01-13 22:25:03,105-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START,
> HSMClearTaskVDSCommand(HostName = Orchard1,
> HSMTaskGuidBaseVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d',
> taskId='671d5904-e062-4d45-9eb4-83a6f13657fe'}), log id: 5335606f
> 2020-01-13 22:25:03,110-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH,
> HSMClearTaskVDSCommand, return: , log id: 5335606f
> 2020-01-13 22:25:03,110-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH,
> SpmStartVDSCommand, return: SpmStatusResult:{SPM Id='-1', SPM LVER='-1',
> SPM Status='Free'}, log id: 3dea111a
> 2020-01-13 22:25:03,117-04 INFO
> [org.ovirt.engine.core.bll.storage.pool.SetStoragePoolStatusCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Running
> command: SetStoragePoolStatusCommand internal: true. Entities affected :
> ID: a45e442e-9989-11e8-b0e4-00163e4bf18a Type: StoragePool
> 2020-01-13 22:25:03,179-04 ERROR
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac]
> IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
> IRSErrorException: SpmStart failed
> 2020-01-13 22:25:03,205-04 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Irs placed
> on server 'fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d' failed. Proceed Failover
> 2020-01-13 22:25:03,223-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] START,
> ConnectStoragePoolVDSCommand(HostName = Orchard2,
> ConnectStoragePoolVDSCommandParameters:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237',
> vdsId='fd0752d8-2d41-45b0-887a-0ffacbb8a237',
> storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', masterVersion='1'}),
> log id: 6a7302e9
> 2020-01-13 22:25:03,224-04 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Executing
> with domain map: {edc68a7c-7604-47e6-89bc-3738d727e8fc=active,
> 23c22a0f-0482-425e-8ada-730cf8ec0751=active,
> 390c0320-e843-4ff3-a4bb-a9973058447f=active,
> fb43d33a-82c8-44cb-8169-090cd0d8f56e=active,
> d70b171e-7488-4d52-8cad-bbc581dbf16e=active,
> 1f2e9989-9ab3-43d5-971d-568b8feca918=active}
>
>
> -----------------------------
>
> vdsm.log from one of my hosts:
>
> 2020-01-13 22:26:12,434-0400 ERROR (tasks/5) [storage.TaskManager.Task]
> Unexpected error (task:652)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
> in _loadMetaFile
> for line in getProcPool().readLines(filename):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
> line 334, in readLines
> return ioproc.readlines(path)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
> in readlines
> return self.readfile(path, direct).splitlines()
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
> in readfile
> "direct": direct}, self.timeout)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
> in _sendCommand
> raise OSError(errcode, errstr)
> OSError: [Errno 2] No such file or directory
> 2020-01-13 22:26:12,435-0400 ERROR (tasks/5) [storage.TaskManager]
> taskManager: Skipping directory: 88772af7-8cf5-433e-8be3-8d0adf0bbf04
> (taskManager:222)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 214, in loadDumpedTasks
> t = Task.loadTask(store, taskID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
> in loadTask
> t._load(store, ext)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746,
> in _load
> self._loadJobMetaFile(taskDir, jn)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696,
> in _loadJobMetaFile
> self._loadMetaFile(taskFile, self.jobs[n], Job.fields)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
> in _loadMetaFile
> raise se.TaskMetaDataLoadError(filename)
> TaskMetaDataLoadError: Can't load Task Metadata:
> ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/88772af7-8cf5-433e-8be3-8d0adf0bbf04/88772af7-8cf5-433e-8be3-8d0adf0bbf04.job.0',)
> 2020-01-13 22:26:12,462-0400 ERROR (tasks/5) [storage.TaskManager.Task]
> Unexpected error (task:652)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
> in _loadMetaFile
> for line in getProcPool().readLines(filename):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
> line 334, in readLines
> return ioproc.readlines(path)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
> in readlines
> return self.readfile(path, direct).splitlines()
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
> in readfile
> "direct": direct}, self.timeout)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
> in _sendCommand
> raise OSError(errcode, errstr)
> OSError: [Errno 2] No such file or directory
> 2020-01-13 22:26:12,462-0400 ERROR (tasks/5) [storage.TaskManager]
> taskManager: Skipping directory: a9b11e33-9b93-46a0-a36e-85063fd53ebe
> (taskManager:222)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 214, in loadDumpedTasks
> t = Task.loadTask(store, taskID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
> in loadTask
> t._load(store, ext)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751,
> in _load
> self._loadRecoveryMetaFile(taskDir, rn)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705,
> in _loadRecoveryMetaFile
> self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
> in _loadMetaFile
> raise se.TaskMetaDataLoadError(filename)
> TaskMetaDataLoadError: Can't load Task Metadata:
> ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/a9b11e33-9b93-46a0-a36e-85063fd53ebe/a9b11e33-9b93-46a0-a36e-85063fd53ebe.recover.0',)
> 2020-01-13 22:26:12,476-0400 ERROR (tasks/5) [storage.TaskManager.Task]
> Unexpected error (task:652)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
> in _loadMetaFile
> for line in getProcPool().readLines(filename):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
> line 334, in readLines
> return ioproc.readlines(path)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
> in readlines
> return self.readfile(path, direct).splitlines()
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
> in readfile
> "direct": direct}, self.timeout)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
> in _sendCommand
> raise OSError(errcode, errstr)
> OSError: [Errno 2] No such file or directory
> 2020-01-13 22:26:12,476-0400 ERROR (tasks/5) [storage.TaskManager]
> taskManager: Skipping directory: 650f2df4-6489-47e2-af5d-db86a22f01c0
> (taskManager:222)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 214, in loadDumpedTasks
> t = Task.loadTask(store, taskID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
> in loadTask
> t._load(store, ext)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751,
> in _load
> self._loadRecoveryMetaFile(taskDir, rn)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705,
> in _loadRecoveryMetaFile
> self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
> in _loadMetaFile
> raise se.TaskMetaDataLoadError(filename)
> TaskMetaDataLoadError: Can't load Task Metadata:
> ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/650f2df4-6489-47e2-af5d-db86a22f01c0/650f2df4-6489-47e2-af5d-db86a22f01c0.recover.0',)
> 2020-01-13 22:26:12,487-0400 ERROR (tasks/5) [storage.TaskManager.Task]
> Unexpected error (task:652)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
> in _loadMetaFile
> for line in getProcPool().readLines(filename):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
> line 334, in readLines
> return ioproc.readlines(path)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
> in readlines
> return self.readfile(path, direct).splitlines()
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
> in readfile
> "direct": direct}, self.timeout)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
> in _sendCommand
> raise OSError(errcode, errstr)
> OSError: [Errno 2] No such file or directory
> 2020-01-13 22:26:12,488-0400 ERROR (tasks/5) [storage.TaskManager]
> taskManager: Skipping directory: a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e
> (taskManager:222)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 214, in loadDumpedTasks
> t = Task.loadTask(store, taskID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
> in loadTask
> t._load(store, ext)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746,
> in _load
> self._loadJobMetaFile(taskDir, jn)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696,
> in _loadJobMetaFile
> self._loadMetaFile(taskFile, self.jobs[n], Job.fields)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
> in _loadMetaFile
> raise se.TaskMetaDataLoadError(filename)
> TaskMetaDataLoadError: Can't load Task Metadata:
> ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e/a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e.job.0',)
> 2020-01-13 22:26:12,493-0400 ERROR (tasks/5) [storage.TaskManager.Task]
> Unexpected error (task:652)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
> in _loadMetaFile
> for line in getProcPool().readLines(filename):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
> line 334, in readLines
> return ioproc.readlines(path)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
> in readlines
> return self.readfile(path, direct).splitlines()
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
> in readfile
> "direct": direct}, self.timeout)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
> in _sendCommand
> raise OSError(errcode, errstr)
> OSError: [Errno 2] No such file or directory
> 2020-01-13 22:26:12,493-0400 ERROR (tasks/5) [storage.TaskManager]
> taskManager: Skipping directory: eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743
> (taskManager:222)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 214, in loadDumpedTasks
> t = Task.loadTask(store, taskID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
> in loadTask
> t._load(store, ext)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 737,
> in _load
> self._loadTaskMetaFile(taskDir)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 688,
> in _loadTaskMetaFile
> self._loadMetaFile(taskFile, self, Task.fields)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
> in _loadMetaFile
> raise se.TaskMetaDataLoadError(filename)
> TaskMetaDataLoadError: Can't load Task Metadata:
> ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743/eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743.task',)
> 2020-01-13 22:26:12,505-0400 ERROR (tasks/5) [storage.TaskManager.Task]
> Unexpected error (task:652)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
> in _loadMetaFile
> for line in getProcPool().readLines(filename):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
> line 334, in readLines
> return ioproc.readlines(path)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
> in readlines
> return self.readfile(path, direct).splitlines()
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
> in readfile
> "direct": direct}, self.timeout)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
> in _sendCommand
> raise OSError(errcode, errstr)
> OSError: [Errno 2] No such file or directory
> 2020-01-13 22:26:12,506-0400 ERROR (tasks/5) [storage.TaskManager]
> taskManager: Skipping directory: 75318d21-45b2-4dbd-985c-a7851a10a463
> (taskManager:222)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 214, in loadDumpedTasks
> t = Task.loadTask(store, taskID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
> in loadTask
> t._load(store, ext)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746,
> in _load
> self._loadJobMetaFile(taskDir, jn)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696,
> in _loadJobMetaFile
> self._loadMetaFile(taskFile, self.jobs[n], Job.fields)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
> in _loadMetaFile
> raise se.TaskMetaDataLoadError(filename)
> TaskMetaDataLoadError: Can't load Task Metadata:
> ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/75318d21-45b2-4dbd-985c-a7851a10a463/75318d21-45b2-4dbd-985c-a7851a10a463.job.0',)
> 2020-01-13 22:26:12,517-0400 ERROR (tasks/5) [storage.TaskManager.Task]
> Unexpected error (task:652)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
> in _loadMetaFile
> for line in getProcPool().readLines(filename):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
> line 334, in readLines
> return ioproc.readlines(path)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
> in readlines
> return self.readfile(path, direct).splitlines()
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
> in readfile
> "direct": direct}, self.timeout)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
> in _sendCommand
> raise OSError(errcode, errstr)
> OSError: [Errno 2] No such file or directory
> 2020-01-13 22:26:12,517-0400 ERROR (tasks/5) [storage.TaskManager]
> taskManager: Skipping directory: 1779b352-022c-49a3-9388-2f688d33cdab
> (taskManager:222)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 214, in loadDumpedTasks
> t = Task.loadTask(store, taskID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
> in loadTask
> t._load(store, ext)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746,
> in _load
> self._loadJobMetaFile(taskDir, jn)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696,
> in _loadJobMetaFile
> self._loadMetaFile(taskFile, self.jobs[n], Job.fields)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
> in _loadMetaFile
> raise se.TaskMetaDataLoadError(filename)
> TaskMetaDataLoadError: Can't load Task Metadata:
> ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/1779b352-022c-49a3-9388-2f688d33cdab/1779b352-022c-49a3-9388-2f688d33cdab.job.0',)
> 2020-01-13 22:26:12,532-0400 ERROR (tasks/5) [storage.TaskManager.Task]
> Unexpected error (task:652)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
> in _loadMetaFile
> for line in getProcPool().readLines(filename):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
> line 334, in readLines
> return ioproc.readlines(path)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
> in readlines
> return self.readfile(path, direct).splitlines()
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
> in readfile
> "direct": direct}, self.timeout)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
> in _sendCommand
> raise OSError(errcode, errstr)
> OSError: [Errno 2] No such file or directory
> 2020-01-13 22:26:12,532-0400 ERROR (tasks/5) [storage.TaskManager]
> taskManager: Skipping directory: 0302036a-7d99-4685-befb-6fee1602feaf
> (taskManager:222)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 214, in loadDumpedTasks
> t = Task.loadTask(store, taskID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
> in loadTask
> t._load(store, ext)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746,
> in _load
> self._loadJobMetaFile(taskDir, jn)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696,
> in _loadJobMetaFile
> self._loadMetaFile(taskFile, self.jobs[n], Job.fields)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
> in _loadMetaFile
> raise se.TaskMetaDataLoadError(filename)
> TaskMetaDataLoadError: Can't load Task Metadata:
> ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/0302036a-7d99-4685-befb-6fee1602feaf/0302036a-7d99-4685-befb-6fee1602feaf.job.0',)
> 2020-01-13 22:26:12,545-0400 ERROR (tasks/5) [storage.TaskManager.Task]
> Unexpected error (task:652)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
> in _loadMetaFile
> for line in getProcPool().readLines(filename):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
> line 334, in readLines
> return ioproc.readlines(path)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
> in readlines
> return self.readfile(path, direct).splitlines()
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
> in readfile
> "direct": direct}, self.timeout)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
> in _sendCommand
> raise OSError(errcode, errstr)
> OSError: [Errno 2] No such file or directory
> 2020-01-13 22:26:12,546-0400 ERROR (tasks/5) [storage.TaskManager]
> taskManager: Skipping directory: 24a825b9-d48d-4134-8aa1-b4db7a9c6ab1
> (taskManager:222)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 214, in loadDumpedTasks
> t = Task.loadTask(store, taskID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
> in loadTask
> t._load(store, ext)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 737,
> in _load
> self._loadTaskMetaFile(taskDir)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 688,
> in _loadTaskMetaFile
> self._loadMetaFile(taskFile, self, Task.fields)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
> in _loadMetaFile
> raise se.TaskMetaDataLoadError(filename)
> TaskMetaDataLoadError: Can't load Task Metadata:
> ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/24a825b9-d48d-4134-8aa1-b4db7a9c6ab1.backup/24a825b9-d48d-4134-8aa1-b4db7a9c6ab1.task',)
> 2020-01-13 22:26:12,560-0400 ERROR (tasks/5) [storage.TaskManager.Task]
> Unexpected error (task:652)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
> in _loadMetaFile
> for line in getProcPool().readLines(filename):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
> line 334, in readLines
> return ioproc.readlines(path)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
> in readlines
> return self.readfile(path, direct).splitlines()
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
> in readfile
> "direct": direct}, self.timeout)
> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
> in _sendCommand
> raise OSError(errcode, errstr)
> OSError: [Errno 2] No such file or directory
> 2020-01-13 22:26:12,561-0400 ERROR (tasks/5) [storage.TaskManager]
> taskManager: Skipping directory: 90d88529-a051-4acd-bef2-d0aa034c15de
> (taskManager:222)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 214, in loadDumpedTasks
> t = Task.loadTask(store, taskID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
> in loadTask
> t._load(store, ext)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751,
> in _load
> self._loadRecoveryMetaFile(taskDir, rn)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705,
> in _loadRecoveryMetaFile
> self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
> in _loadMetaFile
> raise se.TaskMetaDataLoadError(filename)
> TaskMetaDataLoadError: Can't load Task Metadata:
> ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/90d88529-a051-4acd-bef2-d0aa034c15de/90d88529-a051-4acd-bef2-d0aa034c15de.recover.0',)
> 2020-01-13 22:26:12,607-0400 ERROR (tasks/5) [storage.StoragePool]
> Unexpected error (sp:383)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 378, in
> startSpm
> self.taskMng.recoverDumpedTasks()
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 227, in recoverDumpedTasks
> self.queueRecovery(task)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 48, in queueRecovery
> return self._queueTask(task, task.recover)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 54, in _queueTask
> 'Task id already in use: {0}'.format(task.id))
> AddTaskError: TaskManager error, unable to add task: ('Task id already in
> use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',)
> 2020-01-13 22:26:12,608-0400 ERROR (tasks/5) [storage.StoragePool] failed:
> TaskManager error, unable to add task: ('Task id already in use:
> 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',) (sp:384)
> 2020-01-13 22:26:12,635-0400 ERROR (tasks/5) [storage.TaskManager.Task]
> (Task='3c7de2a0-597c-4ebe-b4de-689dba26045b') Unexpected error (task:875)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882,
> in _run
> return fn(*args, **kargs)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336,
> in run
> return self.cmd(*self.argslist, **self.argsdict)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 378, in
> startSpm
> self.taskMng.recoverDumpedTasks()
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 227, in recoverDumpedTasks
> self.queueRecovery(task)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 48, in queueRecovery
> return self._queueTask(task, task.recover)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py",
> line 54, in _queueTask
> 'Task id already in use: {0}'.format(task.id))
> AddTaskError: TaskManager error, unable to add task: ('Task id already in
> use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',)
>
>
> _______________________________________________
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/5XPGERTWVSXUD665IVAKIO7RCY4PDUXO/
>
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/227MNBUK3BC5JNN2MEAHNQUZC7DJJ7A6/
