Change in vdsm[master]: sdm: Autoclear sdm jobs when finished
gerrit-hooks has posted comments on this change. Change subject: sdm: Autoclear sdm jobs when finished .. Patch Set 2: * Update tracker: IGNORE, no Bug-Url found -- To view, visit https://gerrit.ovirt.org/61257 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I666b65d3a73380d786ac1695e4de5c3c7923effd Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Freddy Rolland Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Liron Aravot Gerrit-Reviewer: Nir Soffer Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/vdsm-patches@lists.fedorahosted.org
Change in vdsm[master]: sdm: Autoclear sdm jobs when finished
Adam Litke has abandoned this change. Change subject: sdm: Autoclear sdm jobs when finished .. Abandoned Alternate solution merged. -- To view, visit https://gerrit.ovirt.org/61257 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I666b65d3a73380d786ac1695e4de5c3c7923effd Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Freddy Rolland Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Liron Aravot Gerrit-Reviewer: Nir Soffer Gerrit-Reviewer: gerrit-hooks ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/vdsm-patches@lists.fedorahosted.org
Change in vdsm[master]: sdm: Autoclear sdm jobs when finished
Nir Soffer has posted comments on this change.
Change subject: sdm: Autoclear sdm jobs when finished
..
Patch Set 2:
(2 comments)
https://gerrit.ovirt.org/#/c/61257/2/vdsm/storage/hsm.py
File vdsm/storage/hsm.py:
Line 3509
Line 3510
Line 3511
Line 3512
Line 3513
We should also get rid of this task manager for new jobs - when you add a job
to the jobs module it should schedule it on the jobs executor.
Line 3511: jobs.add(job)
Line 3512: delay = config.getint('irs', 'jobs_autoclear_delay')
Line 3513: if delay >= 0:
Line 3514: job.when_finished(self._scheduler.schedule, delay,
Line 3515: partial(jobs.delete, job.id))
I don't think hsm should do this, job clearing should be part of the jobs
module and job classes. hsm should not care about it, and we don't want to add
code in this hard to test junkyard.
I don't see any problem with passing the scheduler to the job, and leting the
job schedule its deletion. This will make the code easier to test, and I don't
see any disadvantage. Can you explain what is the advantage of adding this code
to hsm?
Line 3516: self.taskMng.scheduleJob("sdm", None, vars.task,
Line 3517: job.description, job.run)
Line 3518:
Line 3519: @public
--
To view, visit https://gerrit.ovirt.org/61257
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I666b65d3a73380d786ac1695e4de5c3c7923effd
Gerrit-PatchSet: 2
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Adam Litke
Gerrit-Reviewer: Francesco Romani
Gerrit-Reviewer: Freddy Rolland
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Liron Aravot
Gerrit-Reviewer: Nir Soffer
Gerrit-Reviewer: gerrit-hooks
Gerrit-HasComments: Yes
___
vdsm-patches mailing list
vdsm-patches@lists.fedorahosted.org
https://lists.fedorahosted.org/admin/lists/vdsm-patches@lists.fedorahosted.org
Change in vdsm[master]: sdm: Autoclear sdm jobs when finished
gerrit-hooks has posted comments on this change. Change subject: sdm: Autoclear sdm jobs when finished .. Patch Set 2: * Update tracker: IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.6', 'ovirt-4.0']) -- To view, visit https://gerrit.ovirt.org/61257 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I666b65d3a73380d786ac1695e4de5c3c7923effd Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Freddy Rolland Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Liron Aravot Gerrit-Reviewer: Nir Soffer Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/vdsm-patches@lists.fedorahosted.org
Change in vdsm[master]: sdm: Autoclear sdm jobs when finished
gerrit-hooks has posted comments on this change. Change subject: sdm: Autoclear sdm jobs when finished .. Patch Set 1: * Update tracker: IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.6', 'ovirt-4.0']) -- To view, visit https://gerrit.ovirt.org/61257 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I666b65d3a73380d786ac1695e4de5c3c7923effd Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/vdsm-patches@lists.fedorahosted.org
Change in vdsm[master]: sdm: Autoclear sdm jobs when finished
Adam Litke has uploaded a new change for review.
Change subject: sdm: Autoclear sdm jobs when finished
..
sdm: Autoclear sdm jobs when finished
The engine uses the Host.getJobs API to query the status of SDM jobs it has
run. In the common case engine polls for status and running, done, and failed
statuses as appropriate. Part of the contract between vdsm and engine states
that Host Jobs are nor persisted and can disappear at any time. Currently this
happens only at vdsm restart. When a job is missing, engine must resort to
entity polling (eg. Volume.getInfo) to determine the result of the missing job.
Our goal is to keep jobs around long enough so that engine can resolve the
status of flows efficiently but without crowding the Host.getJobs result set
with too many irrelevant jobs. This patch introduces a new configuration
setting that will delete a completed job (done, failed, aborted) after a
certain number of seconds have elapsed.
We prefer vdsm to clear jobs as opposed to engine clearing them (as is done
with the older tasks) because jobs can disappear at any time and engine must
always be prepared to handle that case.
Change-Id: I666b65d3a73380d786ac1695e4de5c3c7923effd
Signed-off-by: Adam Litke
---
M lib/vdsm/config.py.in
M tests/sdmtestlib.py
M tests/storage_hsm_test.py
M vdsm/storage/hsm.py
4 files changed, 55 insertions(+), 1 deletion(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/57/61257/1
diff --git a/lib/vdsm/config.py.in b/lib/vdsm/config.py.in
index 1ccb3f0..188b51e 100644
--- a/lib/vdsm/config.py.in
+++ b/lib/vdsm/config.py.in
@@ -392,6 +392,11 @@
'This feature requires a discard support from the storage server. '
'Physical discard operations are supported if the value of '
'/sys/block//queue/discard_max_bytes is not zero.'),
+
+ ('jobs_autoclear_delay', '60',
+'Time to wait (in seconds) before finished jobs are automatically '
+'cleared. Once cleared the job can no longer be checked with the '
+'Host.getJobs API. To disable autoclear set this to '-1'),
]),
# Section: [addresses]
diff --git a/tests/sdmtestlib.py b/tests/sdmtestlib.py
index 2844a6c..efc5899 100644
--- a/tests/sdmtestlib.py
+++ b/tests/sdmtestlib.py
@@ -43,3 +43,6 @@
assert(vars.job_id == self.id)
if self.exception:
raise self.exception
+
+def _abort(self):
+pass
diff --git a/tests/storage_hsm_test.py b/tests/storage_hsm_test.py
index e9e0448..30bc3b3 100644
--- a/tests/storage_hsm_test.py
+++ b/tests/storage_hsm_test.py
@@ -18,6 +18,7 @@
# Refer to the README and COPYING files for full details of the license
#
+import time
import uuid
from contextlib import contextmanager
@@ -25,19 +26,28 @@
from testlib import make_config
from testlib import VdsmTestCase
from testlib import permutations, expandPermutations
+from sdmtestlib import TestingJob, wait_for_job
from storagetestlib import fake_file_env
from storagetestlib import make_file_volume
+from vdsm import jobs
from vdsm import qemuimg
+from vdsm import schedule
from vdsm.storage import constants as sc
from vdsm.storage import exception as se
from storage import hsm
+class FakeTaskManager(object):
+
+def scheduleJob(self, type, store, task, jobName, func, *args):
+func(*args)
+
+
class FakeHSM(hsm.HSM):
def __init__(self):
-pass
+self.taskMng = FakeTaskManager()
@expandPermutations
@@ -117,3 +127,35 @@
make_file_volume(env.sd_manifest, self.SIZE, img_id, vol_id,
vol_format=vol_fmt)
yield env.sd_manifest.produceVolume(img_id, vol_id)
+
+
+@expandPermutations
+class SDMAutoclearTest(VdsmTestCase):
+
+def run_job(self, delay, error=None, abort=False):
+conf = make_config([('irs', 'jobs_autoclear_delay', str(delay))])
+with MonkeyPatchScope([(hsm, 'config', conf)]):
+scheduler = schedule.Scheduler()
+scheduler.start()
+h = FakeHSM()
+h.set_scheduler(scheduler)
+job = TestingJob(exception=error)
+if abort:
+job.abort()
+h.sdm_schedule(job)
+wait_for_job(job)
+time.sleep(0.1) # Give the scheduler a chance to run
+return job
+
+@permutations(((None,), (Exception(),)))
+def test_autoclear(self, error):
+job = self.run_job(delay=0, error=error)
+self.assertRaises(jobs.NoSuchJob, jobs.get, job.id)
+
+def test_autoclear_aborted(self):
+job = self.run_job(delay=0, abort=True)
+self.assertRaises(jobs.NoSuchJob, jobs.get, job.id)
+
+def test_autoclear_disabled(self):
+job = self.run_job(delay=-1)
+self.assertEqual(job.id, jobs.get(job.id).id)
diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py
index a4d4bf3..9c46352 100
