Re: [Devel] Vdsm functional tests
On Sun, Apr 06, 2014 at 11:18:19AM +0300, ybronhei wrote: On 04/03/2014 07:31 PM, Douglas Schilling Landgraf wrote: On 04/03/2014 11:08 AM, Dan Kenigsberg wrote: Functional tests are intended to verify that a running Vdsm instance does what it should, when treated as a black box, over its public API. They should be comprehensive and representative of a typical field usage of Vdsm. It is a sin to break such a test - but we must be able to know when such a sin is committed. We currently have the following functional tests modules: - sosPluginTests.py supervdsmFuncTests.py Sure, count with me. any news about it ? need my help around it? Douglas still owes me a time estimate on when this be done. supervdsmFuncTests.py doesn't really check much. we need to add much more logic there if we actually want to test the communication between vdsm and supervdsm (not sure if its really required.. its like checking calls to libvirt or sanlock or testing api calls) At the moment supervdsmFuncTests do test that supervdsm is reponsive and that supervdsm.getProxy() works. It's not like testing libvirt api calls since supervdsm is inside our tree. So it's like testing libvirt api calls - within the libvirt project. I would embrace more smarter logic into the test - but I'm not sure what you have in mind. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [Devel] cleaning statistics retrieval
On Tue, Apr 08, 2014 at 06:52:50AM -0400, Francesco Romani wrote: Hello VDSM developers, Please use devel@ovirt, and mention vdsm at the subject. This thread in particular involves Engine as well. I'd like to discuss and explain the plans for cleaning up Vm.getStats() in vdsm/virt/vm.py, and how it affects a bug we have: https://bugzilla.redhat.com/show_bug.cgi?id=1073478 Let's start from the bug. To make a long story short, there is a (small) race in VDSM, probably introduced by commit 8fedf8bde3c28edb07add23c3e9b72681cb48e49 The race can actually be triggered only if the VM is shutting down, so it is not that bad. Fixing the reported issue in the specific case can be done with a trivial if, and that it what I did in my initial http://gerrit.ovirt.org/#/c/25803/1/vdsm/vm.py,cm Could you explain how an AttributeError there moved the VM to Down? However, this is just a bandaid and a proper solution is needed. This is the reason why the following versions of http://gerrit.ovirt.org/#/c/25803 changed direction toward a more comprehensive approach. And this is the core of the issue. My initial idea is to either return success and a complete, well formed statistics set, or return an error. However current engine seems to not cope properly with this, and we cannot break backward compatibility. Would you be more precise? If getAllVmStats returns an errCode for one of the Vms, what happens? Looks like the only way to go is to always return success and to add a field to describe the content of the statistics (partial, complete...). THis is admittedly a far cry from the ideal solution, but it is dictated by the need to preserve the compatibility with current/old engines. I don't think that I understand your suggestion, but it does not sound right to send a partial dictionary and to appologize about its paritiality elsewhere. The dictionary may be partial for engine-4.0 yet complete for engine-3.5. It's not for Vdsm to grade its own output. Moreover, I would like to take the chance and cleanup/refactor the statistics collection. I already started adding the test infrastructure: http://gerrit.ovirt.org/#/c/26536/ To summarize, what I suggest to do is: * fix https://bugzilla.redhat.com/show_bug.cgi?id=1073478 using a simple ugly fix like the original http://gerrit.ovirt.org/#/c/25803/1/vdsm/vm.py,cm (which I'll resubmit as new change) * refactor and clean getStats() and friends * on the cleaner base, properly fix the statistics collection by let getStats() always succeed and return possibly partial stats, with a new field describing the content please note that I'm not really happy about this solution, but, given the constraint, I don't see better alternatives. Please explain the benefits of describing the partial content, as I do not see them. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [vdsm] cleaning statistics retrieval
On Wed, Apr 09, 2014 at 01:11:22PM +0200, Michal Skrivanek wrote: On Apr 8, 2014, at 18:57 , Dan Kenigsberg dan...@redhat.com wrote: On Tue, Apr 08, 2014 at 06:52:50AM -0400, Francesco Romani wrote: Hello VDSM developers, Please use devel@ovirt, and mention vdsm at the subject. This thread in particular involves Engine as well. I'd like to discuss and explain the plans for cleaning up Vm.getStats() in vdsm/virt/vm.py, and how it affects a bug we have: https://bugzilla.redhat.com/show_bug.cgi?id=1073478 Let's start from the bug. To make a long story short, there is a (small) race in VDSM, probably introduced by commit 8fedf8bde3c28edb07add23c3e9b72681cb48e49 The race can actually be triggered only if the VM is shutting down, so it is not that bad. Fixing the reported issue in the specific case can be done with a trivial if, and that it what I did in my initial http://gerrit.ovirt.org/#/c/25803/1/vdsm/vm.py,cm Could you explain how an AttributeError there moved the VM to Down? However, this is just a bandaid and a proper solution is needed. This is the reason why the following versions of http://gerrit.ovirt.org/#/c/25803 changed direction toward a more comprehensive approach. And this is the core of the issue. My initial idea is to either return success and a complete, well formed statistics set, or return an error. However current engine seems to not cope properly with this, and we cannot break backward compatibility. Would you be more precise? If getAllVmStats returns an errCode for one of the Vms, what happens? VM moves to Unknown after some time Looks like the only way to go is to always return success and to add a field to describe the content of the statistics (partial, complete...). THis is admittedly a far cry from the ideal solution, but it is dictated by the need to preserve the compatibility with current/old engines. I don't think that I understand your suggestion, but it does not sound right to send a partial dictionary and to appologize about its paritiality elsewhere. The dictionary may be partial for engine-4.0 yet complete for engine-3.5. It's not for Vdsm to grade its own output. Moreover, I would like to take the chance and cleanup/refactor the statistics collection. I already started adding the test infrastructure: http://gerrit.ovirt.org/#/c/26536/ To summarize, what I suggest to do is: * fix https://bugzilla.redhat.com/show_bug.cgi?id=1073478 using a simple ugly fix like the original http://gerrit.ovirt.org/#/c/25803/1/vdsm/vm.py,cm (which I'll resubmit as new change) * refactor and clean getStats() and friends * on the cleaner base, properly fix the statistics collection by let getStats() always succeed and return possibly partial stats, with a new field describing the content please note that I'm not really happy about this solution, but, given the constraint, I don't see better alternatives. Please explain the benefits of describing the partial content, as I do not see them. the main reason for me is that currently if we fail to fetch some parts of the stats we still return what we could get. E.g. try: stats.update(self.guestAgent.getGuestInfo()) except Exception: return stats currently in engine we can only tell it's partial by not seeing the missing bits… it makes sense since for lifecycle decisions we care for the VM status, but we don't necessarily care for other stuff Engine knows that the data is partial, since it sees only parts of it. What is the benefit of Vdsm declaring it as partial? ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [vdsm] cleaning statistics retrieval
On Wed, Apr 09, 2014 at 02:32:57PM +0200, Michal Skrivanek wrote: On Apr 9, 2014, at 14:21 , Dan Kenigsberg dan...@redhat.com wrote: On Wed, Apr 09, 2014 at 01:11:22PM +0200, Michal Skrivanek wrote: On Apr 8, 2014, at 18:57 , Dan Kenigsberg dan...@redhat.com wrote: On Tue, Apr 08, 2014 at 06:52:50AM -0400, Francesco Romani wrote: Hello VDSM developers, Please use devel@ovirt, and mention vdsm at the subject. This thread in particular involves Engine as well. I'd like to discuss and explain the plans for cleaning up Vm.getStats() in vdsm/virt/vm.py, and how it affects a bug we have: https://bugzilla.redhat.com/show_bug.cgi?id=1073478 Let's start from the bug. To make a long story short, there is a (small) race in VDSM, probably introduced by commit 8fedf8bde3c28edb07add23c3e9b72681cb48e49 The race can actually be triggered only if the VM is shutting down, so it is not that bad. Fixing the reported issue in the specific case can be done with a trivial if, and that it what I did in my initial http://gerrit.ovirt.org/#/c/25803/1/vdsm/vm.py,cm Could you explain how an AttributeError there moved the VM to Down? However, this is just a bandaid and a proper solution is needed. This is the reason why the following versions of http://gerrit.ovirt.org/#/c/25803 changed direction toward a more comprehensive approach. And this is the core of the issue. My initial idea is to either return success and a complete, well formed statistics set, or return an error. However current engine seems to not cope properly with this, and we cannot break backward compatibility. Would you be more precise? If getAllVmStats returns an errCode for one of the Vms, what happens? VM moves to Unknown after some time Looks like the only way to go is to always return success and to add a field to describe the content of the statistics (partial, complete...). THis is admittedly a far cry from the ideal solution, but it is dictated by the need to preserve the compatibility with current/old engines. I don't think that I understand your suggestion, but it does not sound right to send a partial dictionary and to appologize about its paritiality elsewhere. The dictionary may be partial for engine-4.0 yet complete for engine-3.5. It's not for Vdsm to grade its own output. Moreover, I would like to take the chance and cleanup/refactor the statistics collection. I already started adding the test infrastructure: http://gerrit.ovirt.org/#/c/26536/ To summarize, what I suggest to do is: * fix https://bugzilla.redhat.com/show_bug.cgi?id=1073478 using a simple ugly fix like the original http://gerrit.ovirt.org/#/c/25803/1/vdsm/vm.py,cm (which I'll resubmit as new change) * refactor and clean getStats() and friends * on the cleaner base, properly fix the statistics collection by let getStats() always succeed and return possibly partial stats, with a new field describing the content please note that I'm not really happy about this solution, but, given the constraint, I don't see better alternatives. Please explain the benefits of describing the partial content, as I do not see them. the main reason for me is that currently if we fail to fetch some parts of the stats we still return what we could get. E.g. try: stats.update(self.guestAgent.getGuestInfo()) except Exception: return stats currently in engine we can only tell it's partial by not seeing the missing bits… it makes sense since for lifecycle decisions we care for the VM status, but we don't necessarily care for other stuff Engine knows that the data is partial, since it sees only parts of it. What is the benefit of Vdsm declaring it as partial? there are a lot of variable fields so it's not that easy to say if it's partial or not. It does make sense to let someone know that there is an issue and you're getting partial data, yet you don't want to make any automatic actions... But Vdsm cannot make this decision. Soon, Vdsm is to report the host's boot time. Now assume that Vdsm fails to do so. Is the stats partial? It's partial for engine-3.5, but it's complete for engine-3.4. Vdsm should tell as much of the truth that it knows. We could extend the alerts mechanism to report non-lethal errors in getVmStats (we currently have it only in for storage domain status), where Engine is told what's missing and why. I'm not sure if this is really needed, though. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [vdsm] Missing numactl
On Tue, Apr 29, 2014 at 03:17:23PM +0200, Vinzenz Feenstra wrote: Seems pretty much like it On 04/28/2014 12:30 PM, Piotr Kliczewski wrote: Hello, I pulled the latest changes from master to test my code and during getCaps I got: Thread-14::ERROR::2014-04-28 12:11:44,394::__init__::484::jsonrpc.JsonRpcServer::(_serveRequest) [Errno 2] No such file or directory Traceback (most recent call last): File /usr/lib/python2.7/site-packages/yajsonrpc/__init__.py, line 480, in _serveRequest res = method(**params) File /usr/share/vdsm/Bridge.py, line 211, in _dynamicMethod result = fn(*methodArgs) File /usr/share/vdsm/API.py, line 1184, in getCapabilities c = caps.get() File /usr/share/vdsm/caps.py, line 459, in get caps['numaNodeDistance'] = _getNumaNodeDistance() File /usr/lib64/python2.7/site-packages/vdsm/utils.py, line 1003, in __call__ value = self.func(*args) File /usr/share/vdsm/caps.py, line 209, in _getNumaNodeDistance retcode, out, err = utils.execCmd(['numactl', '--hardware']) File /usr/lib64/python2.7/site-packages/vdsm/utils.py, line 732, in execCmd deathSignal=deathSignal, childUmask=childUmask) File /usr/lib64/python2.7/site-packages/cpopen/__init__.py, line 50, in __init__ stderr=PIPE) File /usr/lib64/python2.7/subprocess.py, line 711, in __init__ errread, errwrite) File /usr/lib64/python2.7/site-packages/cpopen/__init__.py, line 80, in _execute_child_v275 self._childUmask) OSError: [Errno 2] No such file or directory When I installed numactl rpm getCaps work again. Do we need to add this dependency to spec file? and for completeness, to debia/control too. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] short recap of today's vdsm call (29.4.2014)
Few things mentioned on today's call: - ovirt-3.3 is no longer supported: only security patches need to be merged there. - fenceSpmStorage turned into noop - intenetionally - infra code is expected to move around to its own subdir, as warned 2 weeks ago. - with pthreading used, any os.fork() causes a nasty warning Bug 1084500 - threading._after_fork spews cannot notify on un-acquired lock when os.fork is called from a thread - a huge json-rpc by Piotr needs to be broken to manageable pieces and reviewed. - Adam to create json-to-yaml schema conversion script (as a backward task) - sanlock fencing - Saggi's worries are to be addressed on the mailing list thread. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] short recap of last vdsm call (15.4.2014)
On Tue, Apr 22, 2014 at 02:54:29PM +0300, ybronhei wrote: hey, somehow we missed the summary of this call, and few big issues were raised there. so i would like to share it with all and hear more comments - task id in http header - allows engine to initiate calls with id instead of following vdsm response - federico already started this work, and this is mandatory for live merge feature afaiu. Adam, Federico, may I revisit this question from another angle? Why does Vdsm needs to know live-merge's task id? As far as I understand, (vmid, disk id) are enough to identify a live merge process. If we do not have a task id, we do not need to worry on how to pass it, and where to persist it. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] Fwd: rpm deps
On Wed, Apr 30, 2014 at 09:41:40AM -0400, Piotr Kliczewski wrote: Adding to the discussion. Your CC line was too huge. devel@ovirt.org should be used to such discussions. Currently vdsm-xmlrpc rpm is required by vdsm and vdsm-cli rpms. Initially I suggested to have vdsm-jsonrpc required by vdsm-api but it do not seems to be good idea. I think that we need to add that dept to vdsm rpm in the same way as it is for vdsm-xmlrpc. I do not understand why we have decided to split jsonrpc code between two rpms: vdms-api: - vdsmapi-schema.json vdsm-jsonrpc: - BindingJsonRpc - Bridge vdsm-yajsonrpc: - __init__ - betterAsyncore - stomp - stompReactor I understand that we may have schema in api package but do we really need two packages for jsonrpc? Currently we need a package for stomp reactor so I suggest for now to have one json(stomp) rpm and create second for amqp when there will be need for it. I believe that the motivation for the current packaging was that - yajsonrpc is not strictly vdsm-related. It is a candidate for a spin-off - jsonrpc is sever-side only - api might be needed by clients, too. But I hope we'd have reasonable inspection to avoid this need. As long as vdsm can work without jsonrpc, I'd rather that it does not require the package. When we move away of xmlrpc, we could drop the xmlrpc requirement, too. Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] short recap of last vdsm call (15.4.2014)
On Wed, Apr 30, 2014 at 01:26:18PM -0400, Adam Litke wrote: On 30/04/14 14:22 +0100, Dan Kenigsberg wrote: On Tue, Apr 22, 2014 at 02:54:29PM +0300, ybronhei wrote: hey, somehow we missed the summary of this call, and few big issues were raised there. so i would like to share it with all and hear more comments - task id in http header - allows engine to initiate calls with id instead of following vdsm response - federico already started this work, and this is mandatory for live merge feature afaiu. Adam, Federico, may I revisit this question from another angle? Why does Vdsm needs to know live-merge's task id? As far as I understand, (vmid, disk id) are enough to identify a live merge process. A vmId + diskId can uniquely identify a block job at a single moment in time since qemu guarantees that only a single block job can run at any given point in time. But this gives us no way to differentiate two sequential jobs that run on the same disk. Therefore, without having an engine-supplied jobID, we can never be sure if a one job finished and another started since the last time we polled stats. Why would Engine ever want to initiate a new live merge of a (vmId,diskId) before it has a conclusive result of the previous success/failure of the previous attempt? As far as I understand, this should never happen, and it's actually good for the API to force avoidence of such a case. Additionally, engine-supplied UUIDs is part of a developing framework for next-generation async tasks. Engine prefers to use a single identifier to represent any kind of task (rather than some problem domain specific combination of UUIDs). Adhering to this rule will help us to converge on a single implementation of ng async tasks moving forward. I do not think that having a (virtual) table of task_id - vmId,diskId in Vdsm is much simpler than having it on the Engine machine. I still find the nothion of a new framework for async tasks quite useful. But as I requested before, I think we should design it first, so it fits all conceivable users. In particular, if we should not tie it to the existence of a running VM. We'd better settle on persistence semantics that works for everybody (such as network tasks). Last time, the idea was struck down by Saggi and others from infra, who are afraid to repeat mistakes from the current task framework. If we do not have a task id, we do not need to worry on how to pass it, and where to persist it. There are at least 3 reasons to persist a block job ID: * To associate a specific block job operation with a specific engine-initiated flow. * So that you can clean up after a job that completed when vdsm could not receive the completion event. But if Vdsm dies before it managed to clean up, Engine would have to perform the cleanup via another host. So having this short-loop cleanup is redundant. * Since we must ask libvirt about block job events on a per VM, per disk basis, tracking the devices on which we expect block jobs enables us to eliminate wasteful calls to libvirt. This can be done by an in-memory cache. Hope this makes the rationale a bit clearer... Yes, but I am not yet convinced... ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [ioprocess] Karma please
On Mon, May 05, 2014 at 09:55:58AM +, Sven Kieske wrote: Am 05.05.2014 10:35, schrieb Saggi Mizrahi: I'd appreciate some karma. https://admin.fedoraproject.org/updates/ioprocess-0.3-1.fc20 Hi, I hope you don't mind if I hijack the thread: Is taken care of this package for el6 distributions? Will it get shipped by ovirt.org repo? I expect an EPEL6 (and EPEL7!) build to follow shortly. Hopefully we would not need ship the packages ourselves. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] vdsm disabling logical volumes
On Wed, May 07, 2014 at 09:56:03AM +0200, Jiri Moskovcak wrote: On 05/07/2014 09:28 AM, Nir Soffer wrote: - Original Message - From: Jiri Moskovcak jmosk...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: devel@ovirt.org, Federico Simoncelli fsimo...@redhat.com, Allon Mureinik amure...@redhat.com, Greg Padgett gpadg...@redhat.com, Doron Fediuck dfedi...@redhat.com Sent: Wednesday, May 7, 2014 10:21:28 AM Subject: Re: [ovirt-devel] vdsm disabling logical volumes On 05/05/2014 03:19 PM, Nir Soffer wrote: - Original Message - From: Jiri Moskovcak jmosk...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: devel@ovirt.org, Federico Simoncelli fsimo...@redhat.com, Allon Mureinik amure...@redhat.com, Greg Padgett gpadg...@redhat.com Sent: Monday, May 5, 2014 3:44:21 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes On 05/05/2014 02:37 PM, Nir Soffer wrote: - Original Message - From: Jiri Moskovcak jmosk...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: devel@ovirt.org, Federico Simoncelli fsimo...@redhat.com, Allon Mureinik amure...@redhat.com, Greg Padgett gpadg...@redhat.com Sent: Monday, May 5, 2014 3:16:37 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes On 05/05/2014 12:01 AM, Nir Soffer wrote: - Original Message - From: Jiri Moskovcak jmosk...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: devel@ovirt.org Sent: Sunday, May 4, 2014 9:23:49 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes On 05/04/2014 07:57 PM, Nir Soffer wrote: - Original Message - From: Jiri Moskovcak jmosk...@redhat.com To: devel@ovirt.org Sent: Sunday, May 4, 2014 8:08:33 PM Subject: [ovirt-devel] vdsm disabling logical volumes Greetings vdsm developers! While working on adding ISCSI support to the hosted engine tools, I ran into a problem with vdms. It seems that when stopped vdsm deactivates ALL logical volumes in it's volume group and when it starts it reactivates only specific logical volumes. This is a problem for hosted engine tools as they create logical volumes in the same volume group and when vdsm deactivates the LVs the hosted engine tools don't have a way to reactivate it, because the services drop the root permissions and are running as vdsm and apparently only root can activate LVs. Can you describe what volumes are you creating, and why? We create hosted-engine.lockspace (for sanlock) and hosted-engine.metadata (keeps data about hosted engine hosts) Do you create these lvs in every vdsm vg? - only in the first vg created by vdsm while deploying hosted-engine It seems that the hosted engine has single point of failure - the random vg that contains hosted engine data. Is this part of the domain structure used by hosted engine, or it has nothing to do with the storage domain? - sorry, I don't understand this question. How can I tell if it has something to do with the storage domain? It's for storing data about hosts set up to run the hosted-engine and data about state of engine and the state of VM running the engine. Can you tell us exactly what lvs you are creating, and on which vg? And how are you creating those lvs - I guess through vdsm? - no hosted-engine tools do that by calling: lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, args=[lvm, lvcreate, -L, str(size_bytes)+B, -n, lv_name, vg_uuid]) .. How do you ensure that another host is not modifying the same vg in the same time? If you are not ensuring this, you will corrupt this vg sooner or later. When a storage domain is detached from a host, for example when the host is in maintenance mode, lvs on the shared storage may be deleted, invalidating the devices mapper maps for these devices. If you write to an lv with wrong maps, you may be writing to an extent belonging to another lv, corrupting that lv data, or even worse corrupting the engine vg data. How do you ensure that the lvs are not deleted while you are using them? The output of lvs command on a host with hosted engine installed will help us to understand what you are doing, and then we can think more clearly what would be the best way to support this in vdsm. The output of lvs: http://fpaste.org/99196/93619139/ HE created these two LVs: ha_agent-hosted-engine.lockspace ha_agent-hosted-engine.metadata Why do you create these lvs on a vg owned by vdsm? If you want total control of these lvs, I suggest that you create your own vg and put what ever lvs you like there. I would rather not go this way (at least not for 3.5) as it's too much code changes in hosted-engine. On the other hand the logic in vdsm seems wrong because it's not complementary (disabling all LVs and then enabling just some of them) and should be fixed anyway. This problem is blocking one of our
Re: [ovirt-devel] [VDSM] [PATCH] Add the missing locked() interface
On Wed, May 07, 2014 at 12:48:39PM -0400, Nir Soffer wrote: threading.Lock has a little known locked() method, documented in https://docs.python.org/2.6/library/thread.html#thread.lock.locked This method is not very useful, but since pthreading.Lock must be drop-in replacment for threading.Lock, we must implement it. This patch adds the missing method, implementing it in the same way Python implemnts it. Since RLock does not have this method, RLock does not extend Lock now. Signed-off-by: Nir Soffer nsof...@redhat.com --- pthreading.py | 18 ++ tests.py | 7 +++ 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/pthreading.py b/pthreading.py index 2e9e2d6..f1ea056 100644 --- a/pthreading.py +++ b/pthreading.py @@ -51,9 +51,9 @@ import os import pthread -class Lock(pthread.Mutex): +class _Lock(pthread.Mutex): -Lock class mimics Python native threading.Lock() API on top of +_Lock class mimics Python native threading.Lock() API on top of the POSIX thread mutex synchronization primitive. def __enter__(self): @@ -78,9 +78,19 @@ class Lock(pthread.Mutex): self.unlock() -class RLock(Lock): +class Lock(_Lock): +def locked(self): +# Yes, this is horrible hack, and the same one used by Python +# threadmodule.c. But this is part of Python lock interface. +if self.acquire(blocking=False): +self.release() +return False +return True + + +class RLock(_Lock): def __init__(self): -pthread.Mutex.__init__(self, recursive=True) +_Lock.__init__(self, recursive=True) class Condition(object): diff --git a/tests.py b/tests.py index d651288..f4c9746 100644 --- a/tests.py +++ b/tests.py @@ -60,6 +60,13 @@ class LockTests(TestCaseBase): self.assertTrue(lock.acquire()) self.assertTrue(lock.acquire(False)) +def testLocked(self): +lock = pthreading.Lock() +self.assertFalse(lock.locked()) +with lock: +self.assertTrue(lock.locked()) +self.assertFalse(lock.locked()) + class Flag(object): def __init__(self): -- 1.8.3.1 ACK, thanks for the quick fix ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] Vdsm patches need review and merge
On Thu, May 08, 2014 at 08:48:52AM -0400, Francesco Romani wrote: - Original Message - From: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com To: Dan Kenigsberg (dan...@redhat.com) dan...@redhat.com, Francesco Romani from...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com (gchap...@redhat.com) gchap...@redhat.com, Doron Fediuck (dfedi...@redhat.com) dfedi...@redhat.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, devel@ovirt.org, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com Sent: Thursday, May 8, 2014 2:33:14 PM Subject: RE: Vdsm patches need review and merge Hi Dan/Francesco, Could you help to review and merge these patches: http://gerrit.ovirt.org/#/c/27515/ (This patch is needed by 26876 since some comments in that patch need to modify caps module) http://gerrit.ovirt.org/#/c/26876/ http://gerrit.ovirt.org/#/c/27403/ I didn't modify some comments in 26876 since I think they are related with codes refactor and literal syntax. I will submit a separate patch to modify these kinds of comments of numa feature altogether later. Could you please now focus on the functionality since we need to merge these patches ASAP? http://gerrit.ovirt.org/#/c/27403 I'm OK with it (gave +1) http://gerrit.ovirt.org/#/c/26876 I don't have objections and I don't see blockers for the functionality. I'm not 100% happy with the implementation, but the existing one is on par with most of code, and if we're in hurry, I guess refinements can wait. http://gerrit.ovirt.org/#/c/27515 good enough for the same reasons above (gave +1) Puritans would suggest to squash this into 26876, or fix the commit message - prior to 26876 nothing uses these private methods. But never mind, taken. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] supervdsm spams scans for scsi?
On Fri, May 09, 2014 at 01:04:18PM +, Sven Kieske wrote: Hi, just found this in supervdsm.log: MainProcess|storageRefresh::DEBUG::2014-03-19 17:34:57,752::iscsi::390::Storage.Misc.excCmd::(forceScsiScan) '/bin/dd of=/sys/class/scsi_host/host0/scan' (cwd None) MainProcess|storageRefresh::DEBUG::2014-03-19 17:34:57,764::iscsi::390::Storage.Misc.excCmd::(forceScsiScan) '/bin/dd of=/sys/class/scsi_host/host1/scan' (cwd None) MainProcess|storageRefresh::DEBUG::2014-03-19 17:34:57,766::iscsi::390::Storage.Misc.excCmd::(forceScsiScan) '/bin/dd of=/sys/class/scsi_host/host2/scan' (cwd None) MainProcess|storageRefresh::DEBUG::2014-03-19 17:34:57,768::iscsi::390::Storage.Misc.excCmd::(forceScsiScan) '/bin/dd of=/sys/class/scsi_host/host3/scan' (cwd None) MainProcess|storageRefresh::DEBUG::2014-03-19 17:34:58,771::iscsi::390::Storage.Misc.excCmd::(forceScsiScan) '/bin/dd of=/sys/class/scsi_host/host4/scan' (cwd None) MainProcess|storageRefresh::DEBUG::2014-03-19 17:34:58,774::iscsi::390::Storage.Misc.excCmd::(forceScsiScan) '/bin/dd of=/sys/class/scsi_host/host5/scan' (cwd None) MainProcess|storageRefresh::DEBUG::2014-03-19 17:34:58,776::iscsi::390::Storage.Misc.excCmd::(forceScsiScan) '/bin/dd of=/sys/class/scsi_host/host6/scan' (cwd None) Is there any reason why this is run on hosts which don't got any scsi volumes at all? These messages fill up to 10% of the whole supervdsm.log vdsm version: vdsm-4.13.3-3.el6.x86_64 Shouldn't this just be called if I invoke such a method via webadmin/api ? In ovirt-3.4 this is gone http://gerrit.ovirt.org/#/c/19256/15/vdsm/supervdsmServer ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] supervdsm defunct bug?
On Fri, May 09, 2014 at 02:39:29PM +, Sven Kieske wrote: I don't see anything in supervdsm.log Still, I believe that the bug is worth reopening - it's a process leak, and it should be avoided. I believe that it can be easily solved by adding zombiereaper to supervdsm. just one error at all related to a vm network. in vdsm log I got these timeouts repeatedly: Thread-25::DEBUG::2014-05-02 09:02:43,652::fileSD::222::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_home_DATA/9dc0fcb5-b0b0-47dd-b41f-d8709fd8cab2/dom_md/metadata bs=4096 count=1' (cwd None) Thread-25::DEBUG::2014-05-02 09:02:43,667::fileSD::222::Storage.Misc.excCmd::(getReadDelay) SUCCESS: err = '0+1 records in\n0+1 records out\n495 bytes (495 B) copied, 0.000160501 s, 3.1 MB/s\n'; rc = 0 VM Channels Listener::DEBUG::2014-05-02 09:02:43,940::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 65. VM Channels Listener::DEBUG::2014-05-02 09:02:45,384::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 106. This is unrelated - that's Vdsm complaining about guest agents not heart-beating. It's log noise that has very small value (particularly if you never run guest agents). It should not appear repeatedly more than once per guest. Solving this isn't hard, but would never get priority without an acclaimed user's BZ (hint hint). Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] vdsm disabling logical volumes
On Mon, May 12, 2014 at 01:26:33PM +0200, Jiri Moskovcak wrote: On 05/12/2014 11:22 AM, Dan Kenigsberg wrote: On Wed, May 07, 2014 at 02:16:34PM +0200, Jiri Moskovcak wrote: On 05/07/2014 11:37 AM, Dan Kenigsberg wrote: On Wed, May 07, 2014 at 09:56:03AM +0200, Jiri Moskovcak wrote: On 05/07/2014 09:28 AM, Nir Soffer wrote: - Original Message - From: Jiri Moskovcak jmosk...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: devel@ovirt.org, Federico Simoncelli fsimo...@redhat.com, Allon Mureinik amure...@redhat.com, Greg Padgett gpadg...@redhat.com, Doron Fediuck dfedi...@redhat.com Sent: Wednesday, May 7, 2014 10:21:28 AM Subject: Re: [ovirt-devel] vdsm disabling logical volumes On 05/05/2014 03:19 PM, Nir Soffer wrote: - Original Message - From: Jiri Moskovcak jmosk...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: devel@ovirt.org, Federico Simoncelli fsimo...@redhat.com, Allon Mureinik amure...@redhat.com, Greg Padgett gpadg...@redhat.com Sent: Monday, May 5, 2014 3:44:21 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes On 05/05/2014 02:37 PM, Nir Soffer wrote: - Original Message - From: Jiri Moskovcak jmosk...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: devel@ovirt.org, Federico Simoncelli fsimo...@redhat.com, Allon Mureinik amure...@redhat.com, Greg Padgett gpadg...@redhat.com Sent: Monday, May 5, 2014 3:16:37 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes On 05/05/2014 12:01 AM, Nir Soffer wrote: - Original Message - From: Jiri Moskovcak jmosk...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: devel@ovirt.org Sent: Sunday, May 4, 2014 9:23:49 PM Subject: Re: [ovirt-devel] vdsm disabling logical volumes On 05/04/2014 07:57 PM, Nir Soffer wrote: - Original Message - From: Jiri Moskovcak jmosk...@redhat.com To: devel@ovirt.org Sent: Sunday, May 4, 2014 8:08:33 PM Subject: [ovirt-devel] vdsm disabling logical volumes Greetings vdsm developers! While working on adding ISCSI support to the hosted engine tools, I ran into a problem with vdms. It seems that when stopped vdsm deactivates ALL logical volumes in it's volume group and when it starts it reactivates only specific logical volumes. This is a problem for hosted engine tools as they create logical volumes in the same volume group and when vdsm deactivates the LVs the hosted engine tools don't have a way to reactivate it, because the services drop the root permissions and are running as vdsm and apparently only root can activate LVs. Can you describe what volumes are you creating, and why? We create hosted-engine.lockspace (for sanlock) and hosted-engine.metadata (keeps data about hosted engine hosts) Do you create these lvs in every vdsm vg? - only in the first vg created by vdsm while deploying hosted-engine It seems that the hosted engine has single point of failure - the random vg that contains hosted engine data. Is this part of the domain structure used by hosted engine, or it has nothing to do with the storage domain? - sorry, I don't understand this question. How can I tell if it has something to do with the storage domain? It's for storing data about hosts set up to run the hosted-engine and data about state of engine and the state of VM running the engine. Can you tell us exactly what lvs you are creating, and on which vg? And how are you creating those lvs - I guess through vdsm? - no hosted-engine tools do that by calling: lvc = popen(stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, args=[lvm, lvcreate, -L, str(size_bytes)+B, -n, lv_name, vg_uuid]) .. How do you ensure that another host is not modifying the same vg in the same time? If you are not ensuring this, you will corrupt this vg sooner or later. When a storage domain is detached from a host, for example when the host is in maintenance mode, lvs on the shared storage may be deleted, invalidating the devices mapper maps for these devices. If you write to an lv with wrong maps, you may be writing to an extent belonging to another lv, corrupting that lv data, or even worse corrupting the engine vg data. How do you ensure that the lvs are not deleted while you are using them? The output of lvs command on a host with hosted engine installed will help us to understand what you are doing, and then we can think more clearly what would be the best way to support this in vdsm. The output of lvs: http://fpaste.org/99196/93619139/ HE created these two LVs: ha_agent-hosted-engine.lockspace ha_agent-hosted-engine.metadata Why do you create these lvs on a vg owned by vdsm? If you want total control of these lvs, I suggest that you create your own vg and put what ever lvs you like there. I would rather not go this way (at least not for 3.5
Re: [ovirt-devel] [vdsm] pep8 issue
On Tue, May 13, 2014 at 05:22:15PM -0300, Amador Pahim wrote: Building vdsm/master in F20, I've got: ./vdsm/virt/migration.py:223:19: E225 missing whitespace around operator In vdsm/virt/migration.py: 218 e.err = (libvirt.VIR_ERR_OPERATION_ABORTED, # error code$ 219 libvirt.VIR_FROM_QEMU, # error domain$ 220 'operation aborted',# error message$ 221 libvirt.VIR_ERR_WARNING,# error level$ 222 '', '', '', # str1, str2, str3,$ 223 -1, -1) # int1, int2$ 224 raise e$ pep8 is not accepting negative integer. Instead, it is handling the minus sign as an operator. Quick workaround is change -1 to int(-1). Is this a known? Now it is ;-) I'm using python-pep8-1.5.4-1.fc20.noarch It seems like a bug in pep8 tool, and I see that it does not reproduce with v1.5.6 thereof. I'm adding Matthias who may agree to backport the fix to Fedora 20 (or upgrade completely to 1.5.6) ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [vdsm] pep8 issue
On Wed, May 14, 2014 at 04:33:13PM +0200, Matthias Runge wrote: On Tue, May 13, 2014 at 06:15:31PM -0300, Amador Pahim wrote: I'm using python-pep8-1.5.4-1.fc20.noarch It seems like a bug in pep8 tool, and I see that it does not reproduce with v1.5.6 thereof. I'm adding Matthias who may agree to backport the fix to Fedora 20 (or upgrade completely to 1.5.6) Great, thank you. This update is pushed to updates-testing (for f20), though due the system maintenance currently running, there was no mail sent about that. Please leave feedback on bodhi. Thanks Thanks, but I do not get it with yum update --enablerepo=updates-testing python-pep8 nor see it in https://admin.fedoraproject.org/updates/search/python-pep8 ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [vdsm] VM volume chain information in getVMList API
On Tue, May 27, 2014 at 02:23:45PM +0100, Dan Kenigsberg wrote: On Thu, May 22, 2014 at 01:42:55PM -0400, Federico Simoncelli wrote: Francesco, you cannot answer with +1 to an question with an exclusive or ;-) Argh, sorry Francesco/Federico for the name mixup. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] discussion: URI format for libvirt metadata
On Wed, May 28, 2014 at 08:58:33AM -0400, Kobi Ianko wrote: - Original Message - From: Doron Fediuck dfedi...@redhat.com To: Kobi Ianko k...@redhat.com Cc: devel@ovirt.org Sent: Wednesday, May 28, 2014 3:06:44 PM Subject: Re: [ovirt-devel] discussion: URI format for libvirt metadata for the sake of future updates, I suggest we use current suggestion with a version in the URI. ie- http://ovirt.org/vm/tune/1.0 I prefer this one. me too, as it's clearer that the version relates to the tune element, and not to ovirt. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] [Hackathon] oVirt session non-notes
I've used the occasion of http://wiki.xen.org/wiki/Hackathon/May2014 to learn how Xen is managed, and to start scoping how it can be done within oVirt. I've laid out my impressions on http://www.ovirt.org/Xen; some of it may false. For example, up until few minutes ago, it stated that Xen is missing spice support. Feel free to fix it, it's a wiki. The bottom line is that, with a bit of help, and with limited feature set, oVirt can manage Xen-based hypervisors. I've posted a couple of Vdsm tweaks to support that http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:xen,n,z The nicest patch there is a brute script that munches an oVirt-generated libvirt domxml and strips it from anything that it finds offensive to libvirt's xl driver. http://gerrit.ovirt.org/#/c/28353/1/vdsm_hooks/xen/before_vm_start.py The most painful of the things dropped is qcow2 images - I did not find a way to specify them for Xen domains (admittedly, I did not search too deep). Another hurdle was my failure to boot the guest from its cdrom. To have something running in the guest, I copied http://download.fedoraproject.org/pub/fedora/linux/updates/20/Images/x86_64/Fedora-x86_64-20-20140407-sda.raw.xz onto the VM disk volume. Plenty of other caveats are listed in http://www.ovirt.org/Xen#What.27s_not_Done. But hey, it's working!(ish) I'd like to thank the all nice and helpful people at the Xen hackathon (too many to mention by name!), who invited me there and helped me debug numerous failures and misconfigurations. Regards, Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] local vdsm build fails
On Fri, Jun 06, 2014 at 09:19:11AM +0200, Piotr Kliczewski wrote: All, I pulled the latest vdsm from master and noticed that build is failing. Here is the patch that causes the failuer: http://gerrit.ovirt.org/#/c/28226 and looking at jenkins comments I can see that jenkins was failing with the same reason: http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_gerrit/1064/console Thanks for your report. Nir has already fixed this in http://gerrit.ovirt.org/28426. It was introduced in http://gerrit.ovirt.org/#/c/28226/ but missed also because we have turned PYFLAKES off in unit test jobs. We must turn it on in at least one of the tests (or initiate a new jenkins job for `make check-local`). As a quick fix, David has re-enabled PYFLAKES in http://jenkins.ovirt.org/view/By%20Project/view/vdsm/job/vdsm_master_unit_tests/configure Regards, Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] [REVIEW] Profiling patches
On Tue, Jun 17, 2014 at 03:20:58AM -0400, Nir Soffer wrote: Hi all, I have few easy to review patches adding useful features to vdsm profiler. Please review: - http://gerrit.ovirt.org/27786 profile: Test profile decorator format - http://gerrit.ovirt.org/27787 profile: Support yappi builtins option - http://gerrit.ovirt.org/27816 profile: Support yappi clock types - http://gerrit.ovirt.org/27894 profile: Allow disabling threads when using the decorator Thanks for your code and persistency to get them in; all are merged now. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] supervdsm defunct bug?
On Wed, Jun 18, 2014 at 09:20:06AM +, Sven Kieske wrote: Am 18.06.2014 11:05, schrieb Dan Kenigsberg: Does the suggested patch http://gerrit.ovirt.org/#/c/27627/ solves the problem and does not introduce new ones? If so, please mark it as verified. It awaits verification and more reviews, while we are busy in even-more-urgent 3.5 chores. Regards, Dan. Thanks for pushing with me for the review in order to avoid log spam. You do not happen to know anything about the other bug(supervdsm defunct), though? Oh sorry, this email thread is overloaded a bit. I'm adding Yaniv to the nag list. I hope he or Dima can prepare a quick fix for the pid leak. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] F20 python 2.7.5-12 breaks cpopen
It's a bit confusing, but this is not a task for infra@ovirt, but for the the similarly-named but utterly different team of ovirt developers, who prefer patches to be sent as pull requests to https://github.com/ovirt-infra/cpopen On Mon, Jun 23, 2014 at 05:19:01AM -0400, Francesco Romani wrote: The attached patch fixes locally. - Original Message - From: Francesco Romani from...@redhat.com To: infra in...@ovirt.org Sent: Monday, June 23, 2014 10:48:23 AM Subject: F20 python 2.7.5-12 breaks cpopen Hi Infra, I just discovered the hard way that the last python package in F20 # rpm -qi python Name: python Version : 2.7.5 Release : 12.fc20 Architecture: x86_64 Install Date: Mon 23 Jun 2014 08:28:32 AM CEST Group : Development/Languages Size: 80819 License : Python Signature : RSA/SHA256, Thu 19 Jun 2014 04:49:28 PM CEST, Key ID 2eb161fa246110c1 Source RPM : python-2.7.5-12.fc20.src.rpm Build Date : Thu 19 Jun 2014 02:54:35 PM CEST Build Host : buildvm-24.phx2.fedoraproject.org Relocations : (not relocatable) Packager: Fedora Project Vendor : Fedora Project URL : http://www.python.org/ Summary : An interpreted, interactive, object-oriented programming language [...] Includes this apparently innocuous change: * Thu Jun 19 2014 Bohuslav Kabrda bkab...@redhat.com - 2.7.5-12 - Fix test failures with SQLite 3.8.4 - Fix double close of subprocess pipes when child process fails Resolves: rhbz#1103450 THIS ONE! Which is http://hg.python.org/cpython/rev/43749cb6bdbd If we try to run VDSM tests using cpopen 1.3-2[1] we'll get: http://fpaste.org/112115/ the fix is simple: just use execute_child_v276 and everything will work again. How can we detect and react properly to this schenario, however? Thanks, +++ [1] # rpm -qi python-cpopen Name: python-cpopen Version : 1.3 Release : 2.fc20 Architecture: x86_64 Install Date: Sat 15 Feb 2014 09:50:01 PM CET -- Francesco Romani RedHat Engineering Virtualization R D Phone: 8261328 IRC: fromani ___ Infra mailing list in...@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra -- Francesco Romani RedHat Engineering Virtualization R D Phone: 8261328 IRC: fromani From a5895505d051dfe387c573474e71b3faf696fcf0 Mon Sep 17 00:00:00 2001 From: Francesco Romani from...@redhat.com Date: Mon, 23 Jun 2014 11:17:47 +0200 Subject: [PATCH] compatibility with F20's python 2.7.5-12 I'd prefer the commit message to state the exact exception seen, so that other user could find the fixing patch more easily. --- cpopen/__init__.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/cpopen/__init__.py b/cpopen/__init__.py index 46ed446..cea001e 100644 --- a/cpopen/__init__.py +++ b/cpopen/__init__.py @@ -26,6 +26,7 @@ This is a simpler method of execing that doesn't go back to python after forking. This allows for faster safer exec. +import inspect import os import sys from subprocess import Popen, PIPE @@ -101,5 +102,7 @@ class CPopen(Popen): if sys.version_info[0:3] = (2, 7, 6): _execute_child = _execute_child_v276 +elif 'to_close' in inspect.getargspec(Popen._execute_child).args: +_execute_child = _execute_child_v276 else: _execute_child = _execute_child_v275 -- 1.9.3 ___ Infra mailing list in...@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] ioprocess on debian
On Tue, Jun 24, 2014 at 08:33:00AM -0400, Yeela Kaplan wrote: Hi, There's a new dependency for vdsm: python-ioprocess. We also need to build ioprocess for debian. Do you know who can be helping with that or what's the process? ZhengSheng, maybe you can help us here? ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] VDSM sync meeting minutes June 24th, 2014
We have discussed the following matters. , I had very bad sound quality, so please follow up on this message with things that I have missed. 3.5 features - https://admin.fedoraproject.org/updates/FEDORA-EPEL-2014-1683/mom-0.4.1-1.el6 is deemed unstable (but is required by vdsm) - Last 3.5 feature has been merged in today, and the 3.5 clusterLevel is now reported. - Branching is slated to 2014-06-26 but since we already have a way-too-long merge queue on the master branch, I may branch a bit earlier (to re-open the flood gates). 3.5 bugs - We have a quite old bug in pthreading, causing callers of wait() to be woken up needlessly. A pending patch by Nir should be reviewed by Saggi and yours truly. - lastClientIface is the only current hinderence known to me to moving to jsonrpc. Help us review Piotr's patches. 3.6 changes === - New getHostLeaseStatus verb should be approved by infra - Francesco posted a new threadpool implementation. It's implemented with statistics threads in mind, but should be used by storage/tasks, too. As such infra and storage folks should review it. - Yoav suggests in http://gerrit.ovirt.org/28869 to change our ages-old log format. I like double-colon as delimiter, but I may be old fasioned. It's a tiny oneliner, but everybody is going to be affected by it, so please review it. In particular, if anybody is using an automated tool to parse the logs - come forward, as we may break things up for you. - Domain monitor changes by Nir should be reviewed and approved by Federico. Thanks for joining! Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [QE][ACTION NEEDED] oVirt 3.5.0 Beta status
On Wed, Jun 25, 2014 at 10:25:09AM +0200, Sandro Bonazzola wrote: Hi, We're going to compose oVirt 3.5.0 Beta on Thu *2014-06-26 08:00 UTC*. Maintainers: - Please be sure that master snapshot allow to create VMs before today *2014-06-25 15:00 UTC* The bug tracker [1] shows the following proposed blockers to be reviewed: Bug IDWhiteboard Status Summary 1073944 integration ASSIGNEDAdd log gathering for a new ovirt module (External scheduler) 1060198 integration NEW [RFE] add support for Fedora 20 About Add log gathering for a new ovirt module (External scheduler), patch have been merged on upstream sos master branch. For Fedora = 19 sos-3.1 package includes the fix. For EL 6 patches need to be backported to ovirt-log-collector. Feature freeze is now effective, no more features should be pushed. Features included in 3.5.0 Beta are marked in green on Features Status Table [2] Some more features may be included since they were near to be completed on last sync meeting. The table will be updated on next sync meeting scheduled for today 2014-06-25. There are still 357 bugs [3] targeted to 3.5.0. Excluding node and documentation bugs we still have 294 bugs [4] targeted to 3.5.0. Maintainers / Assignee: - Please remember to rebuild your packages before today *2014-06-25 15:00* if needed, otherwise nightly snapshot will be taken. - Please be sure that master snapshot allow to create VMs before today *2014-06-25 15:00 UTC* I have bumped vdsm's version to 4.16.0 for this occasion, and have created a new ovirt-3.5 branch. Douglas, you may build from that branch whenever you want. - If you find a blocker bug please remember to add it to the tracker [1] - Please start filling release notes, the page has been created here [5] - Please review and add test cases to oVirt 3.5 Test Day [6] Community: - You're welcome to join us testing this beta release and getting involved in oVirt Quality Assurance[7]! [1] http://bugzilla.redhat.com/1073943 [2] http://bit.ly/17qBn6F [3] http://red.ht/1pVEk7H [4] http://red.ht/1rLCJwF [5] http://www.ovirt.org/OVirt_3.5_Release_Notes [6] http://www.ovirt.org/OVirt_3.5_TestDay [7] http://www.ovirt.org/OVirt_Quality_Assurance Thanks, -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] VDSM sync meeting minutes June 24th, 2014
On Wed, Jun 25, 2014 at 07:46:52AM -0400, Antoni Segura Puimedon wrote: - Original Message - From: Francesco Romani from...@redhat.com To: Dan Kenigsberg dan...@redhat.com, Saggi Mizrahi smizr...@redhat.com Cc: devel@ovirt.org Sent: Wednesday, June 25, 2014 1:15:03 PM Subject: Re: [ovirt-devel] VDSM sync meeting minutes June 24th, 2014 - Original Message - From: Francesco Romani from...@redhat.com To: Dan Kenigsberg dan...@redhat.com, Saggi Mizrahi smizr...@redhat.com Cc: devel@ovirt.org Sent: Wednesday, June 25, 2014 10:06:37 AM Subject: Re: [ovirt-devel] VDSM sync meeting minutes June 24th, 2014 - Original Message - From: Dan Kenigsberg dan...@redhat.com To: devel@ovirt.org Sent: Tuesday, June 24, 2014 4:13:05 PM Subject: [ovirt-devel] VDSM sync meeting minutes June 24th, 2014 [...] - Francesco posted a new threadpool implementation. It's implemented with statistics threads in mind, but should be used by storage/tasks, too. As such infra and storage folks should review it. The code is on my github. The history may be messed, but the concepts are in there and it is reviewable: https://github.com/mojaves/vdsm/tree/master/lib/threadpool it is used only here: https://github.com/mojaves/vdsm/blob/master/vdsm/virt/sampling.py I'm now adding at least the initial test and polishing the code. I'll post a proper patch on gerrit hopefully within the end of the week, in order to ease the review. Initial draft posted in the hope to ease the review: http://gerrit.ovirt.org/#/c/29189/ http://gerrit.ovirt.org/#/c/29190/ http://gerrit.ovirt.org/#/c/29191/ These patches will be kept in-sync with changes I make on github: testing and docs, mostly, and adjustement of the code to make testing easier. The core concepts are there. This is the core part - the new pool implementation; the port of the virt sampling code and the other pieces will follow soon in separate patches. A very valid point was raised by Nir in #vdsm: if we choose to go ahead with this new thread pool, should this code put in a separate project like cpopen or ioprocess? I vote for separate oVirt gerrit project. Do we expect anyone outside Vdsm to make use of it soon? If so, go ahead! If not, this would excert release-engineering costs, with no clear benefits. Someone would have to tag a build, and push it to Fedora/debian. cpopen and ioprocess are written in C and as such are less fitting to be inside Vdsm. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [ACTION REQUIRED] ovirt-3.4-snapshot repository closure failures
On Thu, Jun 26, 2014 at 05:59:47AM -0400, Francesco Romani wrote: - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: devel@ovirt.org, infra in...@ovirt.org Sent: Thursday, June 26, 2014 11:56:19 AM Subject: [ACTION REQUIRED] ovirt-3.4-snapshot repository closure failures Hi, ovirt-3.4-snapshot is currently failing repository closure test: http://jenkins.ovirt.org/job/repos_3.4_check-closure_merged/DISTRIBUTION=centos6/80/console due to missing vdsm packages. VDSM is not published since May 29th due to job failing: http://jenkins.ovirt.org/job/vdsm_3.4_create-rpms_merged/ Note that http://jenkins.ovirt.org/view/Publishers/job/publish_ovirt_rpms_nightly_3.4 is not marked as failing also if a required package is missing. Can infra take care of the publisher job and vdsm team of the failing build? Thanks, Looks like we need to update the cpopen package to 1.3-3 on the F20 host. Background: http://lists.ovirt.org/pipermail/devel/2014-June/007813.html In case it was not clear - the cpopen bug is the reason of recent failures. Please join me in giving karma to https://admin.fedoraproject.org/updates/FEDORA-2014-7758/python-cpopen-1.3-3.fc20 so it can be pushed to Fedora 20. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] multipath configuration
On Wed, Jun 25, 2014 at 09:58:52AM -0400, Yeela Kaplan wrote: Hi, Currently multipath.conf is being rotated each time we reconfigure it. We'd like to change the behaviour for multipath.conf so that current configuration will be commented out and we'd stop rotating (in the same manner as libvirt conf works today). Does anybody have any comment for/ against? I'd like to present the problem in a slightly different light. It is highly uncommon for a service to mangle the configuration files of another service. Administrator do not expect they complain (e.g. Bug 1076531 - vdsm overwrites multipath.conf at every startup) We used to have this problem with libvirtd.conf, but it has been aleviated by moving the configuration to vdsm-tool, which should do its thing when asked by the local admin, or once during automatic host installation. We should do this with multipath.conf, too. And we should deprecate the RHEV trademark that is stuck there while at it. The only question is how. Should we place the old configuration in a rotated file (a-la .rpmsave) or should we place them in the same file, completely commented out. We've taken the comment-out approach for libvirtd.conf, logrotate etc., but I must admit that moving that saving a copy of the original file is simpler and cleaner. Note that we do not need to rotate more than one file. On upgrade, we should overwrite our version of the multipath.conf with our new favorite version. The original version should be kept just in case we want to uninstall Vdsm and revert to the original. Another question is how we should write the configuration file: should we do it as simple text, or should we take a smarter augeas-based approach. In my opinion, as long as we want to dump our own conf file, there is not motivation for augeas. If, on the other hand, we would like to replace only a section of the file, it should come into play. Is this what you have in mind for multipath.conf, Federico? Could you elaborate? Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] multipath configuration
On Fri, Jun 27, 2014 at 09:10:51AM -0400, Federico Simoncelli wrote: - Original Message - From: Antoni Segura Puimedon asegu...@redhat.com To: Nir Soffer nsof...@redhat.com Cc: Federico Simoncelli fsimo...@redhat.com, devel@ovirt.org Sent: Friday, June 27, 2014 2:57:59 PM Subject: Re: [ovirt-devel] multipath configuration - Original Message - From: Nir Soffer nsof...@redhat.com To: Federico Simoncelli fsimo...@redhat.com Cc: devel@ovirt.org Sent: Friday, June 27, 2014 2:12:50 PM Subject: Re: [ovirt-devel] multipath configuration - Original Message - From: Federico Simoncelli fsimo...@redhat.com To: Dan Kenigsberg dan...@redhat.com Cc: devel@ovirt.org Sent: Friday, June 27, 2014 3:03:56 PM Subject: Re: [ovirt-devel] multipath configuration - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Yeela Kaplan ykap...@redhat.com Cc: devel@ovirt.org, Federico Simoncelli fsimo...@redhat.com, Allon Mureinik amure...@redhat.com Sent: Thursday, June 26, 2014 1:56:32 PM Subject: Re: multipath configuration On Wed, Jun 25, 2014 at 09:58:52AM -0400, Yeela Kaplan wrote: Hi, Currently multipath.conf is being rotated each time we reconfigure it. We'd like to change the behaviour for multipath.conf so that current configuration will be commented out and we'd stop rotating (in the same manner as libvirt conf works today). Does anybody have any comment for/ against? I'd like to present the problem in a slightly different light. It is highly uncommon for a service to mangle the configuration files of another service. Administrator do not expect they complain (e.g. Bug 1076531 - vdsm overwrites multipath.conf at every startup) We used to have this problem with libvirtd.conf, but it has been aleviated by moving the configuration to vdsm-tool, which should do its thing when asked by the local admin, or once during automatic host installation. We should do this with multipath.conf, too. And we should deprecate the RHEV trademark that is stuck there while at it. +2 The only question is how. Should we place the old configuration in a rotated file (a-la .rpmsave) or should we place them in the same file, completely commented out. That is an all-or-nothing approach. VDSM probably relies only on few config entries. One thing is for sure, VDSM should be able to detect if the relevant multipath configs are not properly set and refuse to start. For the discovery of these values (read) we could use augeas. If the sysadmin already uses the correct parameters we may not even touch the file at all. Another question is how we should write the configuration file: should we do it as simple text, or should we take a smarter augeas-based approach. In my opinion, as long as we want to dump our own conf file, there is not motivation for augeas. If, on the other hand, we would like to replace only a section of the file, it should come into play. Is this what you have in mind for multipath.conf, Federico? Could you elaborate? My approach would be: 1. vdsm (or its service file) should discover if multipath.conf is configured properly and eventually refuse to start vdsm, the less the service file does the more likely we can have the same systemd unit file for all the distros without init common adapters. We need this check somewhere whether it is in the service file (maybe using ExecStartPre for cleanliness) or in the vdsm daemon the problem of supporting multiple distro remains. task_check_is_configured calls `vdsm-tool is-configured` on service start in all distros. Extending is-configured to check multipath.conf is the natural thing to do. 2. vdsm-tool being able to configure multipath.conf (only the required entries), this would be very much similar to what we do with libvirt. If we want to use augeas it is just an implementation detail. 3. vdsm-tool being able to install/replace entire multipath.conf in case it wasn't already configured (or in case you want to force the default strict configuration, e.g. during initial automated deployment?) It is always polite to backup the previous config in .rpmsave. Point 1 implicitly allows the sysadmin to change multipath.conf, which could be harder to support (he could use conflicting entries) but anyway it gives the flexibility to customize multipath.conf if needed (which is something that we already have). Point 1 and 2 seem just read and write of the same parameters, which could be cleanly done with augeas (if the support is good). We
Re: [ovirt-devel] [VDSM] Re: XML benchmarks
On Fri, Jun 27, 2014 at 08:43:01AM -0400, Francesco Romani wrote: 1000 VMs python thread error :) File /usr/lib64/python2.7/threading.py, line 746, in start _start_new_thread(self.__bootstrap, ()) thread.error: can't start new thread I think this is another proof (if we need more of them) that * we _really need_ to move away from the 1 thread per VM model - http://gerrit.ovirt.org/#/c/29189/ and friends! Let's fire up the discussion! True, but it's interesting to understand what is the limit that you have banged into. Not enough PIDs? Memory? ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] scale hooks
On Mon, Jun 30, 2014 at 01:04:09PM +0200, Michal Skrivanek wrote: Hi, just thinking about the recent stats hook[1] by Dima and scale, It is a very nice hook and quite useful for the scale testing - ( though when focus is on engine I'd suggest to use FakeVDSM instead to eliminate all the bottlenecks on VDSM side.) One issue I see - since this is a very high-profile call, every 15s time number of VMs - is with the hook invocation. Currently the hook mechanism always go through the hooks dit and does stat() call to see script is there or not… it certainly doesn't matter on small scale but when the system is on the edge every IO on local FS cost something…. Would be nice if the hook mechanism is enhanced in general, figure out which hooks are relevant only at the startup and don't waste time passing domainXML every single time when the hook is not there But a caching mechanism for hook scripts costs something, too (code size, bugs, simplicty of testing new script). Before complicating the mechanism, I'd rather see numbers: how much would be saved? I would guess that filesystem caching already shaves costs of stat()s. Writing domxml to a memory-map file before starting a vm may be insignificant, too. Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] test day upgrade failure
On Wed, Jul 02, 2014 at 05:28:24AM -0400, Eli Mesika wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: devel@ovirt.org, emes...@redhat.com Sent: Wednesday, July 2, 2014 11:27:19 AM Subject: test day upgrade failure For the test day I've tried to upgrade my ovirt-engine-3.4.2-1.el6.noarch to 3.5 beta. I've encountered psql:/usr/share/ovirt-engine/dbscripts/upgrade/03_05_0050_event_notification_methods.sql:2: ERROR: constraint fk_event_subscriber_event_notification_methods of relation event_subscriber does not exist FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/upgrade/03_05_0050_event_notification_methods.sql 2014-07-02 09:16:13 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File /usr/lib/python2.6/site-packages/otopi/context.py, line 142, in _executeMethod method['method']() File /usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/db/schema.py, line 291, in _misc oenginecons.EngineDBEnv.PGPASS_FILE File /usr/lib/python2.6/site-packages/otopi/plugin.py, line 871, in execute command=args[0], RuntimeError: Command '/usr/share/ovirt-engine/dbscripts/schema.sh' failed to execute 2014-07-02 09:16:13 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Misc configuration': Command '/usr/share/ovirt-engine/dbscripts/schema.sh' failed to execute Is this failure known? What is the remedy? Hi Dan I had created a 3.4.2 DB and upgraded it to 3.5 with no issues fk_event_subscriber_event_notification_methods FK is created when creating the initial base tales and nothing drops it except the failing upgrade script. I assume that a manual operation had dropped this FK on your DB, please try to run the following before teh upgrade on your 3.4.2 DB and after that try to upgrade again ALTER TABLE ONLY event_subscriber ADD CONSTRAINT fk_event_subscriber_event_notification_methods FOREIGN KEY (method_id) REFERENCES event_notification_methods(method_id) ON DELETE CASCADE; How can I tell that I'm on the right DB? The refered table event_notification_methods seems to be missing. # su - postgres -bash-4.1$ psql engine psql (8.4.20) Type help for help. engine=# ALTER TABLE ONLY event_subscriber engine-# ADD CONSTRAINT fk_event_subscriber_event_notification_methods FOREIGN KEY (method_id) REFERENCES event_notification_methods(method_id) ON DELETE CASCADE; ERROR: relation event_notification_methods does not exist ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] VDSM threads as in vdsm 4.15.0
On Fri, Jul 04, 2014 at 06:09:44AM -0400, Francesco Romani wrote: Hi, I'm trying to enumerate all the daemon threads VDSM uses, as in 4.15.0 (to be exact, as in master 3228f5dad8f488192ea62f4e3e39172832435e80) So far I got: === SocketServer: 1 service, short lived thread per request? VDSM alone, with no VM running num whatwhere 1 libVirtEventLooplib/vdsm/libvirtconnection.py P TaskManager vdsm/storage/hsm.py 1 StorageServer vdsm/storage/storageServer.py 1 StorageRefresh vdsm/storage/hsm.py 1 ChannelListener vdsm/virt/vmchannels.py 1 HostStatsThread vdsm/virt/sampling.py 1 MomThread vdsm/momIF.py 1 clientIFvdsm/clientIF.py 1 HostMonitor mom/HostMonitor.py 1 StompReactorvdsm/rpc/BindingJsonRpc.py 1 JsonRpcServer vdsm/rpc/BidningJsonRpc.py 1 GuestManagermom/GuestManager.py 1 PolicyEnginemom/PolicyEngine.py 1 XmlRpcServer? vdsm/rpc/BindingXMLRPC.py 1 Detectorvdsm/clientIF.py 1 RPCServer mom/RPCServer.py 1 SocketServerstdlib, triggered by vdsm/rpc/BindingXMLRPC.py 1 IOProcess? vdsm/storage/outOfProcess.py S DomainMonitor vdsm/storage/domainMonitor.py P/2 HSM_MailMonitor vdsm/storage/storage_mailbox.py P/2 SPM_MailMonitor vdsm/storage/storage_mailbox.py P = pool size (config/irs/thread_pool_size) Total: 17 + number(StorageDomains) + P * 2 Total with defaults (P=10) = 37 + number(StorageDomains) Minimum storage domains: 2 (ISO + data) Baseline: 39 threads Per VM 1 samplingvdsm/virt/sampling.py 1 GuestMonitormom/GuestMonitor.py === More precise update will follow. In the meantime, anyone please feel free to point out omissions/errors. You must have dropped the migration-monitoring threads on purposed! (but they still pop up). There are other ocasions when (hopefully) short-lived threads are created, such as during getStorageDomainsList() XmlRpcServer springs a new thread per http connection. What's P/2 ? What are the question marks in the above table? And in that line - what's the IOProcess thead? Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [vdsm] logging levels and noise in the log
On Thu, Jul 10, 2014 at 10:15:23AM -0400, Nir Soffer wrote:
- Original Message -
From: Martin Sivak msi...@redhat.com
To: devel@ovirt.org
Sent: Thursday, July 10, 2014 1:09:14 PM
Subject: [ovirt-devel] [vdsm] logging levels and noise in the log
Hi,
we discussed the right amount of logging with Nir and Francesco while
reviewing my patches. Francesco was against one DEBUG log message that could
potentially flood the logs. But the message in question was important for me
and SLA, because it was logging the VDSM changes in response to MoM command.
Since DEBUG really is meant to be used for debug messages, I have one
proposal:
1) Change the default log level of vdsm.log to INFO
Not good enough - our log are full or tracebacks logged in error
and warning levels, which are not helpful to the administrator.
So this will create log that is not useful for anything.
2) Log (only) DEBUG messages to a separate file vdsm-debug.log
The debug log should include *all* log levels, basically being the same as
the current vdsm log. When you need the logs, you want all the information in
the same file.
3) Make the vdsm-debug.log rotate faster (every hour, keep only last couple
of hours?) so it does not grow too much
I just debugged a problem in production, where the mailbox stopped working
86 logs ago - so having only couple of hours is too little.
However when using proper logging, This issue should be clear even
in info log level, which show error, warnings, and info message such
as mailbox stopping.
This way the customer would be able to monitor INFO logs (much smaller)
without all the noise and we would be able to collect the DEBUG part in case
something happens.
I think you got it right with the dual log approach. We should have two logs:
1. Admin log (events.log?)
This log should show important events and errors, but it should not contain
any tracebacks, thread name, file names etc. Basically, something like the
events log in the engine.
If we take storage for example, this log can contain storage domain state
change, or configuration changes triggered by the engine, or events like
disk that becoming too full.
I have recently introduced a connectivity.log, that tracks changes in
host network devices. Adding parallel logs for other important
admin-understandable changes makes sense.
I'd be happy to hear which bits of information do oVirt admins wish to
see in this events.log.
2. Developer log (vdsm.log)
There is lot of work to to here:
- Disable library logs - we have lot of infrastructure code that log
useless
messages. This code should be verified by tests, not by looking at the
logs.
- Removing junk log messages:
- new ioprocess logs: https://bugzilla.redhat.com/1114908
- new jsonrpc logs: same issue - no bug yet
- Adding useful info like VDSM changes in response to MoM command.
I did not look at the patch, but if vdsm changes state, it must be logged
- Improving log format
http://gerrit.ovirt.org/28869
The only problem is to agree which format is better. I personally like
condensed format, that is possibly machine-readable. Spaces, in
function and repeating module name and its full path, are suboptimal
imo.
- Removing duplicate errors/tracebacks for same error, caused by code that
does
try:
something()
except:
log.error('something failed...')
raise
+1, I hate those.
Should be:
something()
And let the upper level handle the failure, logging a trackback.
Or:
try:
something()
Except LowLevelError:
raise FlowError(description...)
FlowError can log a traceback of both errors by using sys.exec_info.
See http://hg.moinmo.in/moin/1.9/file/274a7f675261/MoinMoin/error.py
- Disabling this log by default?
Only if an admin can easily enable or disable the log, ideally from the
ui.
It all too often that a user comes up with a bug or a question. Seeing
his logs usually gives a good hint regarding the solution, or at least
who is to blame. It is important to me to keep this tool handy.
So for example, when flow fails, the event log will contain the a single
error message
related to the flow, while the debug log will contains the same error message
with
a traceback and useful debugging messages during the flow.
Lets make this a feature for 3.6?
Vdsm has a setLogLevel; it must have rotton out of disuse. But I'd love
to see an Engine option for cluster debug level, that is overridable
per-host. Each admin should decide what does they care more about:
debugablility by developers, or the bliss of quiet logs.
Dan.
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [action required] vdsm build job failing on Fedora 20
On Wed, Jul 23, 2014 at 06:04:06AM -0400, Nir Soffer wrote: - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: devel@ovirt.org, infra in...@ovirt.org Sent: Wednesday, July 23, 2014 10:19:43 AM Subject: [action required] vdsm build job failing on Fedora 20 Hi, VDSM build job is currently failing on Fedora 20 for all branches: 3.4: pep8 issues http://jenkins.ovirt.org/view/Stable%20branches%20per%20project/view/vdsm/job/vdsm_3.4_create-rpms_merged/label=fedora20/256/console 07:51:15 /usr/bin/pep8 --version 07:51:15 1.5.6 This test is wrong in two ways: 1. There is no reason to check current source with pep8, only new changes. We should not change working code because new pep8 version changed the rules 2. This should not be part of the build. pep8 is a tool for checking new source before we add it. It does not validate the application in any way. To workaround this, either downgrade pep8 or disable this job. ... or disable pep8 in this specific job, by setting PEP8=true on the `make rpm` line. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] VDSM mom dependancy on RHEL7
On Wed, Jul 23, 2014 at 12:00:36PM -0400, Adam Litke wrote: On 23/07/14 13:18 +0100, Dan Kenigsberg wrote: On Wed, Jul 23, 2014 at 07:05:07AM -0400, Martin Polednik wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Martin Polednik mpoled...@redhat.com, dc...@redhat.com Cc: devel@ovirt.org Sent: Wednesday, July 23, 2014 12:19:42 PM Subject: Re: [ovirt-devel] VDSM mom dependancy on RHEL7 On Tue, Jul 22, 2014 at 06:36:10PM -0400, Martin Polednik wrote: Hi, I've gone through installing VDSM on RHEL7 host (Red Hat Enterprise Linux Server release 7.0 (Maipo)) and encountered issue with mom: Error: Package: mom-0.4.1-2.el6.noarch (ovirt-3.5-epel) Requires: python(abi) = 2.6 Installed: python-2.7.5-16.el7.x86_64 (@rhel7) python(abi) = 2.7 python(abi) = 2.7 Repositories used were master, master-snapshots and 3.5 + dependancies. Although it is possible to get it working by getting mom source and rebuilding it on RHEL7, I'd like to know if there is different RHEL7 repo or this is mistake in repos. http://resources.ovirt.org/pub/ovirt-3.5-pre/rpm/el7/ is quite empty, I'm afraid. The problem actually occurs when getting mom from http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/ mom there http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/noarch/mom-0.4.1-0.0.master.20140718.gitfcedb2d.el7.noarch.rpm is of older version than the one in your ovirt-3.5-epel (mom-0.4.1-2.el6.noarch). Try disabling ovirt-3.5-epel repo. Since el6-el7 upgrade should be possible, el7 packages must have a higher n-v-r than their el6 diblings. Who is the person that is actually building the mom rpms for the release? It's actually not me :) I am actually a bit confused about how to handle this. Since mom is actually an upstream project that is used by people outside of oVirt releases, I have been maintaining upstream packages in Fedora/epel. We haven't really encountered a situation where oVirt needs to ship something different than what is upstream (but this may come to pass in the future). A couple of questions... 1) Should we even be building mom in the ovirt repos at all? If Fedora/EPEL is good enough - we'd rather use it, instead of duplicating the effort. 2) If we need to ensure a smooth upgrade between el6 and el7, does this mean that I need to bump the el7 version every time I upgrade the el6 packages? This seems strange to me. It's strange and can be anoying, but that's life on Fedora, too. If the package in F21 declares itself as older than the one in F20, it won't be updated on a platform upgrade, and it can even hamper the upgrade altogether. When building a new version of a package in Fedora/EPEL you should start with the devel (rawhide) branch, then backward in time until the oldest supported version. I must admit that I'm too lazy to do that when I need to build only f19, but in that case I try to bump only the minor number of the release, so that at least in the next release of f20 things would straighten up. I think the better solution is to not require a specific version of python (2.6, 2.7, etc) but just 3 until we support python3. The python version dependency is implicit, since rpm ship with .pyc, which has a different signature in 2.7. 3) We should probably make vdsm depend on a specific version of MOM. That way we can upgrade the fedora package but have an ovirt host stay on a current stabilized version. But the downside is that every build - no matter how minute - of mom, would require a contemporary build of vdsm. Seems painful. Alternatively, before a new mom is released, it should be tested against current vdsm. If things break, mom should declare a Conflicts: vdsm = current-vdsm-version ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] noarch vdsm.rpm
I'd like to make Vdsm a pure noarc package, by shedding the remaining safelease binary to a helper package and posted http://gerrit.ovirt.org/30678 on these lines. However, this patch is not enough - when built on an x86_64 host, rpm still recognizes vdsm.rpm and vdsm-python.rpm as x86_64 instead of noarch. Dropping all mentions of x86_64 from the spec did not help, either. Is there an RPM wizard around who can tell me why is this auto-detection happening, and how can it be avoided? Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] noarch vdsm.rpm
On Thu, Jul 24, 2014 at 07:37:45AM -0400, Martin Sivak wrote: I am pretty sure you have to tell rpm that it is supposed to be noarch package: BuildArch: noarch I do not see that anywhere in that patch. Thanks, it did the trick. Now the only problem is the x86_64-only dmidecode requirement. Yaniv, can it be changed to a soft requirement? ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] VDSM sync meeting minutes August 5th, 2014
Yesterdays we discussed various pending issues, and among them: - live merge for 3.5 has backported patches pending merge http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:ovirt-3.5+topic:livemerge-3.5,n,z - jsonrpc inroduced logging Engine command arguments, equivalent to what we have in xmlrpc - We have a ppc-specific bug - on vdsm restart, previously-running VMs are not recovered. This is very dangerous, as it may lead to Engine restarting the same VM elsewhere, causing data curruption. Michal found the bug and its cause - in ppc, we do not inject the vm uuid to the guest bios, and clientIF.isVDSMVm() fails to recognize such Vms. Is there a BZ open on that? - There was an agreement that we configure libvirtd.log to be too verbose. Yaniv/Mooli, please consider something like my untested http://gerrit.ovirt.org/31135 Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] VDSM sync meeting minutes August 5th, 2014
On Wed, Aug 06, 2014 at 11:28:14AM +0100, Dan Kenigsberg wrote: Yesterdays we discussed various pending issues, and among them: - live merge for 3.5 has backported patches pending merge http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:ovirt-3.5+topic:livemerge-3.5,n,z - jsonrpc inroduced logging Engine command arguments, equivalent to what we have in xmlrpc - We have a ppc-specific bug - on vdsm restart, previously-running VMs are not recovered. This is very dangerous, as it may lead to Engine restarting the same VM elsewhere, causing data curruption. Michal found the bug and its cause - in ppc, we do not inject the vm uuid to the guest bios, and clientIF.isVDSMVm() fails to recognize such Vms. Is there a BZ open on that? There is: 1126887 recovery of VMs after VDSM restart doesn't work on PPC - There was an agreement that we configure libvirtd.log to be too verbose. Yaniv/Mooli, please consider something like my untested http://gerrit.ovirt.org/31135 Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] Management network as a role - design proposal
On Thu, Aug 14, 2014 at 11:52:41AM -0400, Yevgeny Zaspitsky wrote: Hi All, The proposed feature will allow defining an arbitrary network in the DC as the management network for the cluster, which in its turn will allow assigning different VLANs for the management networks in the same DC. Feature page can be found here - http://www.ovirt.org/Features/Management_Network_As_A_Role . Please take a look into the page especially into Open issues section. I'd like to have your opinions on that. May I ask why you change the default management network from ovirtmgmt to Management? (And why the upercase M?) Regarding your open question: Creating new cluster would have to receive the new parameter (management network) This new parameter should be kept optional, with a default value of ovirtmgmt. This way, a user that is unaware of the new feature, would see no change in functionality. The feature page should discuss the possibility of chaning the management role. Is it supported after there are hosts in the cluster? If we allow that, there's a bit of complexity. The management network's gateway is used as the default route of each host. If you change the role, all hosts should be notified (with a new setupNetwork command). I think that the cleanest solution would be to allow editing, but report the hosts as out-of-sync. This approach requires a Vdsm-side change - it would need to report which of its network is the default route. Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] Problems installing ovirt 3.5 on Centos 7
On Wed, Aug 27, 2014 at 08:49:32AM +0200, Sandro Bonazzola wrote:
Il 26/08/2014 18:59, Andy Ryan ha scritto:
I'm trying to install oVirt 3.5 on a CentOS 7 system that has a bonded
network interface. ovirt-hosted-engine-setup fails to complete. I get the
follow messages in the install log:
Adding users list.
Running configure...
Reconfiguration of libvirt is done.
Done configuring modules to VDSM.
2014-08-26 11:40:02 DEBUG
otopi.plugins.ovirt_hosted_engine_setup.system.vdsmenv plugin.execute:866
execute-output: ('/bin/vdsm-tool', 'config
2014-08-26 11:40:02 DEBUG otopi.plugins.otopi.services.systemd
systemd.state:134 starting service vdsmd
2014-08-26 11:40:02 DEBUG otopi.plugins.otopi.services.systemd
plugin.executeRaw:785 execute: ('/bin/systemctl', 'start',
'vdsmd.service'), ex
2014-08-26 11:41:32 DEBUG otopi.plugins.otopi.services.systemd
plugin.executeRaw:803 execute-result: ('/bin/systemctl', 'start',
'vdsmd.servic
2014-08-26 11:41:32 DEBUG otopi.plugins.otopi.services.systemd
plugin.execute:861 execute-output: ('/bin/systemctl', 'start',
'vdsmd.service')
2014-08-26 11:41:32 DEBUG otopi.plugins.otopi.services.systemd
plugin.execute:866 execute-output: ('/bin/systemctl', 'start',
'vdsmd.service')
Job for vdsmd.service failed. See 'systemctl status vdsmd.service' and
'journalctl -xn' for details.
2014-08-26 11:41:32 DEBUG otopi.context context._executeMethod:152 method
exception
Traceback (most recent call last):
File /usr/lib/python2.7/site-packages/otopi/context.py, line 142, in
_executeMethod
method['method']()
File
/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/system/vdsmenv.py,
line 155, in _late_setup
state=True
File /usr/share/otopi/plugins/otopi/services/systemd.py, line 138, in
state
'start' if state else 'stop'
File /usr/share/otopi/plugins/otopi/services/systemd.py, line 77, in
_executeServiceCommand
raiseOnError=raiseOnError
File /usr/lib/python2.7/site-packages/otopi/plugin.py, line 871, in
execute
command=args[0],
RuntimeError: Command '/bin/systemctl' failed to execute
2014-08-26 11:41:32 ERROR otopi.context context._executeMethod:161 Failed
to execute stage 'Environment setup': Command '/bin/systemctl' faile
2014-08-26 11:41:32 DEBUG otopi.context context.dumpEnvironment:468
ENVIRONMENT DUMP - BEGIN
2014-08-26 11:41:32 DEBUG otopi.context context.dumpEnvironment:478 ENV
BASE/error=bool:'True'
2014-08-26 11:41:32 DEBUG otopi.context context.dumpEnvironment:478 ENV
BASE/exceptionInfo=list:'[(type 'exceptions.RuntimeError', RuntimeEr
2014-08-26 11:41:32 DEBUG otopi.context context.dumpEnvironment:482
ENVIRONMENT DUMP - END
2014-08-26 11:41:32 INFO otopi.context context.runSequence:395 Stage: Clean
up
2014-08-26 11:41:32 DEBUG otopi.context context.runSequence:399 STAGE
cleanup
And in /var/log/messages
Aug 26 11:55:03 localhost systemd: Starting Virtual Desktop Server
Manager...
Aug 26 11:55:03 localhost vdsmd_init_common.sh: vdsm: Running mkdirs
Aug 26 11:55:03 localhost vdsmd_init_common.sh: vdsm: Running
configure_coredump
Aug 26 11:55:03 localhost vdsmd_init_common.sh: vdsm: Running
configure_vdsm_logs
Aug 26 11:55:03 localhost vdsmd_init_common.sh: vdsm: Running run_init_hooks
Aug 26 11:55:03 localhost vdsmd_init_common.sh: vdsm: Running gencerts
Aug 26 11:55:03 localhost vdsmd_init_common.sh: vdsm: Running
check_is_configured
Aug 26 11:55:03 localhost vdsmd_init_common.sh: libvirt is already
configured for vdsm
Aug 26 11:55:03 localhost vdsmd_init_common.sh: vdsm: Running
validate_configuration
Aug 26 11:55:03 localhost vdsmd_init_common.sh: SUCCESS: ssl configured to
true. No conflicts
Aug 26 11:55:04 localhost vdsmd_init_common.sh: vdsm: Running
prepare_transient_repository
Aug 26 11:55:04 localhost vdsmd_init_common.sh: vdsm: Running
syslog_available
Aug 26 11:55:04 localhost vdsmd_init_common.sh: vdsm: Running nwfilter
Aug 26 11:55:04 localhost vdsmd_init_common.sh: vdsm: Running dummybr
Aug 26 11:55:04 localhost vdsmd_init_common.sh: vdsm: Running
load_needed_modules
Aug 26 11:55:04 localhost vdsmd_init_common.sh: vdsm: Running tune_system
Aug 26 11:55:04 localhost vdsmd_init_common.sh: vdsm: Running test_space
Aug 26 11:55:04 localhost vdsmd_init_common.sh: vdsm: Running test_lo
Aug 26 11:55:04 localhost vdsmd_init_common.sh: vdsm: Running
unified_network_persistence_upgrade
Aug 26 11:55:04 localhost vdsmd_init_common.sh: vdsm: Running restore_nets
Aug 26 11:55:05 localhost kernel: bonding: g7je1p3LjMcQbb3 is being
created...
Aug 26 11:55:05 localhost kernel: bonding: g7je1p3LjMcQbb3: setting mode to
balance-rr (0).
Aug 26 11:55:05 localhost kernel: bonding: g7je1p3LjMcQbb3: setting mode to
active-backup (1).
Aug 26 11:55:05 localhost kernel:
Re: [ovirt-devel] feature review - ReportGuestDisksLogicalDeviceName
On Sun, Aug 31, 2014 at 07:20:04AM -0400, Liron Aravot wrote: Feel free to review the the following feature. http://www.ovirt.org/Features/ReportGuestDisksLogicalDeviceName Thanks for posting this feature page. Two things worry me about this feature. The first is timing. It is not reasonable to suggest an API change, and expect it to get to ovirt-3.5.0. We are two late anyway. The other one is the suggested API. You suggest placing volatile and optional infomation in getVMList. It won't be the first time that we have it (guestIPs, guestFQDN, clientIP, and displayIP are there) but it's foreign to the notion of conf reported by getVMList() - the set of parameters needed to recreate the VM. Would you describe the motivation for using this verb? What are the pros and cons? Which other ideas where considered? Regards, Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [QE][ACTION REQUIRED] oVirt 3.5.0 RC2 status
On Mon, Sep 08, 2014 at 10:04:30AM -0400, Nir Soffer wrote: - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: devel@ovirt.org, us...@ovirt.org, Juan Hernandez juan.hernan...@redhat.com, Yair Zaslavsky yzasl...@redhat.com, Piotr Kliczewski pklic...@redhat.com, el...@redhat.com, Tal Nisan tni...@redhat.com, Liron Aravot lara...@redhat.com, Alexander Wels aw...@redhat.com, Tomas Jelinek tjeli...@redhat.com, Michal Skrivanek michal.skriva...@redhat.com, rk...@redhat.com, mgr...@redhat.com, Martin Sivak msi...@redhat.com, Jiri Moskovcak jmosk...@redhat.com Sent: Monday, September 8, 2014 9:41:07 AM Subject: [ovirt-devel] [QE][ACTION REQUIRED] oVirt 3.5.0 RC2 status Hi, We were supposed to start composing oVirt 3.5.0 RC2 today from 3.5 branch, however we still have blockers open so we should postpone the build. In order to re-schedule the build I need ETA on the open blockers, please provide it. Maintainers: - Please be sure that 3.5 snapshot satisfy release criteria[9] - Please be sure that no pending patches are going to block the release - If any patch must block the RC release please raise the issue as soon as possible. The bug tracker [1] shows the following proposed blockers to be reviewed: Bug ID Whiteboard Status Summary 1138986 infra NEW UploadStreamVDSCommand fails using JsonRpc 1136843 network NEW NetworkManager ifups VDSM's dummy bond 1138731 network NEW firewall-cmd gets stuck when run as superVDSM subsubprocess 1138340 sla NEW vdsm relies on unsupported 'min_guarantee' VM configuration parameter 1129261 sla POSTprepareImage api call fails with [Errno 2] No such file or directory And the following dependencies still open: Bug 1041569 - [NFR] libvirt: Returning the watermark for all the images opened for writing Bug 1102881 - virDomainBlockCommit fails with live snapshots on oVirt block storage The following bugs are keyworded as Regression and not marked as blockers[10] Bug ID Whiteboard Status Summary 1138983 NEW [CLI] Update template is not actually updating template. 1106435 infra NEW [AAA] builtin kerbldap provider does not show error if search fails 1138981 network POSTno validation before calling to AddSubnetToProvider 1136895 network NEW Import VmTemplate fails 1116572 storage NEW [engine-backend] Storage domain activation isn't executed as part of storage domain creation 1138144 storage NEW Failed to autorecover storage domain after unblocking connection with host This is an issue with lvm, not ovirt/vdsm regression - we waiting for help from lvm developers on this. Does it reproduce only on el7? ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [QE][ACTION REQUIRED] oVirt 3.5.0 RC2 status
On Mon, Sep 08, 2014 at 05:19:01PM -0400, Rashid Khan wrote: Hi As for https://bugzilla.redhat.com/show_bug.cgi?id=1136843 NetworkManager ifups VDSM's reference bond Bug in POST state now. Work around and scratch build provided. We will roll it in the next NM build shortly. Thanks. But to make use of it (call nmcli con load), Ondrej needs to clone this bug to Vdsm. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] feature review - ReportGuestDisksLogicalDeviceName
On Wed, Sep 03, 2014 at 08:50:12AM +0200, Michal Skrivanek wrote: On Sep 2, 2014, at 21:03 , Nir Soffer nsof...@redhat.com wrote: - Original Message - From: Michal Skrivanek mskri...@redhat.com To: Liron Aravot lara...@redhat.com, Dan Kenigsberg dan...@redhat.com, Federico Simoncelli fsimo...@redhat.com, Vinzenz Feenstra vfeen...@redhat.com Cc: us...@ovirt.org, devel@ovirt.org Sent: Tuesday, September 2, 2014 3:29:57 PM Subject: Re: [ovirt-devel] feature review - ReportGuestDisksLogicalDeviceName On Sep 2, 2014, at 13:11 , Liron Aravot lara...@redhat.com wrote: - Original Message - From: Federico Simoncelli fsimo...@redhat.com To: devel@ovirt.org Cc: Liron Aravot lara...@redhat.com, us...@ovirt.org, smizr...@redhat.com, Michal Skrivanek mskri...@redhat.com, Vinzenz Feenstra vfeen...@redhat.com, Allon Mureinik amure...@redhat.com, Dan Kenigsberg dan...@redhat.com Sent: Tuesday, September 2, 2014 12:50:28 PM Subject: Re: feature review - ReportGuestDisksLogicalDeviceName - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Liron Aravot lara...@redhat.com Cc: us...@ovirt.org, devel@ovirt.org, smizr...@redhat.com, fsimo...@redhat.com, Michal Skrivanek mskri...@redhat.com, Vinzenz Feenstra vfeen...@redhat.com, Allon Mureinik amure...@redhat.com Sent: Monday, September 1, 2014 11:23:45 PM Subject: Re: feature review - ReportGuestDisksLogicalDeviceName On Sun, Aug 31, 2014 at 07:20:04AM -0400, Liron Aravot wrote: Feel free to review the the following feature. http://www.ovirt.org/Features/ReportGuestDisksLogicalDeviceName Thanks for posting this feature page. Two things worry me about this feature. The first is timing. It is not reasonable to suggest an API change, and expect it to get to ovirt-3.5.0. We are two late anyway. The other one is the suggested API. You suggest placing volatile and optional infomation in getVMList. It won't be the first time that we have it (guestIPs, guestFQDN, clientIP, and displayIP are there) but it's foreign to the notion of conf reported by getVMList() - the set of parameters needed to recreate the VM. The fact is that today we return guest information in list(Full=true), We decide on it's notion and it seems like we already made our minds when guest info was added there :) . I don't see any harm in returning the disk mapping there and if we'll want to extract the guest info out, we can extract all of it in later version (4?) without need for BC. Having the information spread between different verbs is no better imo. At first sight this seems something belonging to getVmStats (which is reporting already other guest agent information). Fede, I've mentioned in the wiki, getVmStats is called by the engine every few seconds and therefore that info wasn't added there but to list() which is called only when the hash is changed. If everyone is in for that simple solution i'm fine with that, but Michal/Vincenz preferred it that way. yes, that was the main reason me and Vinzenz suggested to use list(). 15s is a reasonable compromise, IMHO. And since it's also reported by guest agent in a similar manner (and actually via the same vdsm-ga API call) as other guest information I think it should sit alongside guestIPs, FQDN, etc… Maybe not the best place, but I would leave that for a bigger discussion if/when we want to refactor reporting of the guest agent information Given that we are late, adding disk mapping where other guest info is in a backward compatible way looks reasonable. Did you consider adding a new verb for getting guest information? This verb can be called once after starting/recovering a vm, and then only when guest info hash changes (like the xml hash). yes, was considered but turned down. Main reason is an additional overhead and pollution of the API for a tiny little item which is in the same group of guest agent reported items. It doesn't make sense to introduce an inconsistency just this one item. Refactoring of the frequency of what we report is indeed long overdue, but we shouldn't start here…(the first and most offending is still the application list) Ok, lacking another alternative, let's dump the maps to list(). ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [ovirt-users] [QE][ACTION REQUIRED] oVirt 3.5.0 status
On Wed, Sep 17, 2014 at 09:12:28AM -0400, Oved Ourfali wrote: - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: Oved Ourfali ov...@redhat.com Cc: us...@ovirt.org, devel@ovirt.org Sent: Wednesday, September 17, 2014 4:10:33 PM Subject: Re: [ovirt-users] [ovirt-devel] [QE][ACTION REQUIRED] oVirt 3.5.0 status Il 17/09/2014 15:09, Oved Ourfali ha scritto: Bug ID Whiteboard Status Summary 1135909infra POST[Event log] Wrong warning about available swap memory of host [1023MB], when actually host has [1024MB] memory size We're still discussing whether to address that on engine/VDSM side. Doesn't seem urgent, though, so I wouldn't consider that as a blocker. We didn't see any change with regards to that, so I'm surprised it is a Regression at all... If it's not a regression, please remove the keyword :-) Well... the reporter claims it is... we haven't tested it yet due to other more urgent issues. In particular, real-life installation should use much more thatn 1GiB of swap space, so this problem is moot. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] vdsm 3.5 rpm build is failing
On Mon, Sep 22, 2014 at 11:49:02AM +0200, Piotr Kliczewski wrote: All, I noticed that vdsm rpm build is failing: http://jenkins-ci.eng.lab.tlv.redhat.com/job/vdsm_3.5_rpm_factory/779/artifact/logs/make_rpm.log Superficially, it seems like a bad backport of http://gerrit.ovirt.org/#/c/33007/3/vdsm.spec.in but I do not know for sure. This log is part of a downstream build, and is inaccessible to the public. As such, it should not be discussed in this mailing list. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] vdsm 3.5 rpm build is failing
On Tue, Sep 23, 2014 at 12:48:25AM +0100, Dan Kenigsberg wrote: On Mon, Sep 22, 2014 at 11:49:02AM +0200, Piotr Kliczewski wrote: All, I noticed that vdsm rpm build is failing: http://jenkins-ci.eng.lab.tlv.redhat.com/job/vdsm_3.5_rpm_factory/779/artifact/logs/make_rpm.log Superficially, it seems like a bad backport of http://gerrit.ovirt.org/#/c/33007/3/vdsm.spec.in but I do not know for sure. This log is part of a downstream build, and is inaccessible to the public. As such, it should not be discussed in this mailing list. Oh, the issue affects upstream as well. Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/jenkins/workspace/vdsm_3.5_rpm_factory/rpmbuild/BUILDROOT/vdsm-4.16.4-11.git0337e7f.el6.x86_64 error: Installed (but unpackaged) file(s) found: /usr/share/vdsm/rpc/vdsmapi-gluster-schema.json http://gerrit.ovirt.org/#/c/33007 installs /usr/share/vdsm/rpc/vdsmapi-gluster-schema.json, but if vdsm is built with_gluster=0, this file is not collected by anything. http://gerrit.ovirt.org/#/c/33240/ should fix that. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] Building vdsm within Fedora
Since Vdsm was open-sourced, it was built and deployed via Fedora. Recently [http://gerrit.ovirt.org/31214] vdsm introduced a spec-file dependency onf qemu-kvm-rhev, and considered to backport it to the ovirt-3.4 brach. Requiring qemu-kvm-rhev, which is not part of Fedora's EPEL6 branch, violates Fedora's standards. So basically we have two options: 1. Revert the qemu-kvm-rhev dependency. 2. Drop vdsm from EPEL6 (or completely from Fedora); ship Vdsm only within the oVirt repositories. A third option would be to have one rpm, with qemu-kvm-rhev, shipped in ovirt, and another without it - shipped in Fedora. I find this overly complex and confusing. I favor option 2. The Fedora deployment platform served us well for a long time, but now that ovirt is maturing, we no longer need it for building vdsm. This has the added benefit of removing the need to pass through Fedora's ghastly gateway when adding a Vdsm dependency. Sandro, what should be done in order to build Vdsm by ovirt, occording to the most up-to-date tag in a stable branch? Does anybody object this? If no one does, we would stop updating Vdsm in Fedora, and obsolete it in the future. Regards, Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [ovirt-users] Building vdsm within Fedora
On Wed, Sep 24, 2014 at 10:57:21AM +0200, Sandro Bonazzola wrote: Il 24/09/2014 09:44, Sven Kieske ha scritto: On 24/09/14 09:13, Federico Simoncelli wrote: You probably missed the first part we were using qemu-kvm/qemu-img in the spec file. In that case you won't fail in any requirement. Basically the question is: was there any problem on centos6 before committing http://gerrit.ovirt.org/31214 ? Federico: as we checked a few minutes ago, it seems there's no problem in requiring qemu-kvm/qemu-img in the spec file. Only issue is that if non rhev version is installed a manual yum update is required for moving to the rhevm version. Right. Without the patch, RPM does not enforce qemu-kvm-rhev. So our code has to check for qemu-kvm-rhev functionality, instead of knowing that it is there. Furthermore, we had several reports of users finding themselves without qemu-kvm-rhev on their node, and not understanding why they do not have live merge. Of course there was a problem, please follow the link in this very commit to the according bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1127763 In short: you can not use live snapshots without this updated spec file. And it's a PITA to install this package by hand, you must track it's versions yourself etc pp. you basically lose all the stuff a proper spec file gives you. Well, since the -rhev package is now available in 3.4, 3.5 and master repos it shouldn't be a PITA anymore. PS: I also don't get the we want to get vdsm in every distribution a) it was never in any distro, it was in epel, which is a third party repository anyway, so you can just provide it via ovirt repo imho. Historically, Vdsm has been part of Fedora before it has been part of ovirt! https://bugzilla.redhat.com/show_bug.cgi?id=745510 The EPEL build was added much later b) no one packages vdsm for debian, ubuntu, gentoo, arch, suse, $nameyourdistro or I completely missed it, so why treat fedora in a special way? Don't misunderstand me, it would be cool if you have packages for every distro, or even bsd based stuff, but I think this is still a long way. Indeed. But it would be even longer if we take my suggested step backwards. c) will anyone use vdsm without ovirt? is this even possible? so imho you need ovirt repos anyway? I don't belive Vdsm is soon to be used by anything outside oVirt. But if software purists win, oVirt would publish only tarballs. Fedora/Debian/whatever would build, package, and deploy them all, and the ovirt repo would become redundant. I did not expect to hear much support for keeping Vdsm in Fedora. Given what I've heard, how about taking the in-between road? - Keep Vdsm in Fedora, abiding to Fedora rules. - Hope that Engine and qemu-kvm-rhev join, too. - Until they do, build vdsm.rpm with non-Fedora quirks (such as the qemu-kvm-rhev requirement) http://gerrit.ovirt.org/33367 spec: do not require qemu-kvm-rhev on Fedora http://gerrit.ovirt.org/33368 spec: allow all archs in Fedora Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [ovirt-users] Building vdsm within Fedora
On Fri, Sep 26, 2014 at 05:42:41AM -0400, Federico Simoncelli wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Sandro Bonazzola sbona...@redhat.com Cc: crobi...@redhat.com, users us...@ovirt.org, devel@ovirt.org Sent: Thursday, September 25, 2014 3:06:01 PM Subject: Re: [ovirt-devel] [ovirt-users] Building vdsm within Fedora On Wed, Sep 24, 2014 at 10:57:21AM +0200, Sandro Bonazzola wrote: Il 24/09/2014 09:44, Sven Kieske ha scritto: On 24/09/14 09:13, Federico Simoncelli wrote: You probably missed the first part we were using qemu-kvm/qemu-img in the spec file. In that case you won't fail in any requirement. Basically the question is: was there any problem on centos6 before committing http://gerrit.ovirt.org/31214 ? Federico: as we checked a few minutes ago, it seems there's no problem in requiring qemu-kvm/qemu-img in the spec file. Only issue is that if non rhev version is installed a manual yum update is required for moving to the rhevm version. Right. Without the patch, RPM does not enforce qemu-kvm-rhev. So our code has to check for qemu-kvm-rhev functionality, instead of knowing that it is there. Furthermore, we had several reports of users finding themselves without qemu-kvm-rhev on their node, and not understanding why they do not have live merge. Live merge? The biggest problem with live merge is libvirt not qemu. Sorry, I meant to say live snapshot and refer to http://gerrit.ovirt.org/26149 reporting to Engine if it's available. Anyway the qemu-kvm/qemu-kvm-rhev problem is relevant only for centos and centos has a specific way to address these special needs: http://www.centos.org/variants/ A CentOS variant is a special edition of CentOS Linux that starts with the core distribution, then replaces or supplements a specific subset of packages. This may include replacing everything down to the kernel, networking, and other subsystems. I think the plan was to have our own centos variant (shipping qemu-kvm-rhev). I remember Doron participated to the centos meetings but I don't remember the outcome. That would be lovely. EPEL's vdsm can then ship there, in case Fedora cannot depend on a centos variant. Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] Master cannot be installed on rhel7 - broken libvirt-dependency
On Sun, Oct 19, 2014 at 09:15:49AM -0400, Nir Soffer wrote: Hi all, Commit 395c77c90c0b (vdsm: Add support for VIR_MIGRATE_AUTO_CONVERGE parameter) bumped libvirt-daemon dependency on rhel7 to version 1.2.3. It seems that there is no such version in rhel7 or rhel7.z repositories, which means that master cannot be installed now on rhel7 machines. Anyone knows on which repository this package exists? Thanks Nir for your reprot. el7.1 should see a libvirt rebase, but at the moment, I am not aware of such a repo. Martin, would you drop the hard requirement from the spec, until it is available on the el7 platform? Nothing else is required, as we already have a soft dependency in place. We should make a better effort to avoid requiring a package before it is universally available. Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] SR-IOV feature
On Sun, Oct 26, 2014 at 06:39:00AM -0400, Alona Kaplan wrote:
On 10/05/2014 07:02 AM, Alona Kaplan wrote:
Hi all,
Currently SR-IOV in oVirt is only supported using vdsm-hook [1].
This feature will add SR-IOV support to oVirt management system
(including
migration).
You are more than welcome to review the feature page-
http://www.ovirt.org/Feature/SR-IOV
Thanks,
Alona.
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
Glad to see this.
some questions:
Note: this feature is about exposing a virtualized (or VirtIO) vNic to
the
guest, and not about exposing the PCI device to it. This restriction is
necessary for migration to be supported.
did not understand this sentence - are you hinting to macvtap?
Most likely macvtap, yes.
Additionally I think Martin Poledník is looking into direct sr-iov
attachment
to VMs as part of the pci passthrough work he is doing.
add/edit profile
so i gather the implementation is at profile level, which is at logical
network level?
how does this work exactly? can this logical network be vlan tagged or
must be native? if vlan tagged who does the tagging for the passthrough
device? (I see later on vf_vlan is one of the parameters to vdsm, just
wondering how the mapping can be at host level if this is a passthrough
device)?
is this because the use of virtio (macvtap)?
The logical network can be vlan tagged.
As you mentioned the vf_vlan is one of the parameters to the vdsm (on create
verb).
Setting the vlan on the vf is done as follows-
ip link set {DEVICE} vf {NUM} [ vlan VLANID ]
It is written in the notes section.
It is not related to the use of virtio. The vlan can be set on the vf whether
it
is connected to the vm via macvtap or directly.
Are you sure about this? I think that when a host device is attached to
a VM, it disappears from the host, and the the guest can send arbitrary
unmodified packets through the wire. But I may well be wrong.
wouldn't it be better to support both macvtap and passthrough and just
flag the VM as non migratable in that case?
Martin Polednik is working on pci-passthrough-
http://www.ovirt.org/Features/hostdev_passthrough
Maybe we should wait for his feature to be ready and then combine it with the
sr-iov feature.
As I see in his feature page he plans to attach a specific device directly
to the vm.
We can combine his feature with the sr-iov feature-
1. The network profile will have type property-
bridge (the regular configuration we have today, vnic-tap-bridge-physical
nic).
virtio(in the current feature design it is called passthrough,
vnic-macvtap-vf)
pci-passthrough(vnic-vf)
2. Attaching a network profile with pci-passthrough type to a vnic will mark
the vm as non-migratable.
This marking can be tuned by the admin. If the admin requests migration
despite the pci-passthrough type, Vdsm can auto-unplug the PCI device
before migration, and plug it back on the destination.
That would allow some kind of migration to guests that are willing to
see a PCI device disappear and re-appear.
3. When running a vm with pci-passthrough vnic a free VF will be attached to
the vm with the vlan and mtu
configuration of the profile/network (same as for virio profile, as described
in the feature page).
The benefit of it is that the user won't have to choose the vf directly and
will
be able to set vlan and mtu on the vf.
also (and doesn't have to be in first phase) what happens if i ran out
of hosts with sr-iov (or they failed) - can i fail back to non
pcipassthrough profile for backup (policy question at vm level if more
important to have sr-iov or more important it will run even without it
since it provides a critical service, with a [scheduling] preference to
run on sr-iov?
(oh, i see this is in the futures section already.
:)
A benefit of this Nice to have passthrough is that one could set it on
vNic profiles that are already used by VMs. Once they are migrated to a
new host, the passthrough-ness request would take effect.
management, display and migration properties are not relevant for the
VFs
configuration
just wondering - any technical reason we can't put the management on a
VF (not saying its a priority to do so)?
Today we mark the logical network with a role (management/display/migration)
when attaching it to the cluster.
A logical network can be attached to one physical nic (PF).
We can't use the current attachment of a role for sr-iov, since the network
can
be configured as vf allowed on more than one nic (maybe even on all the
nics).
If the network is vf allowed on the nic,
a vnic with this network can be attached to a free vf on the nic.
So we can't use the logical network
Re: [ovirt-devel] [QE][ACTION NEEDED] oVirt 3.5.1 RC status - branching
On Wed, Nov 12, 2014 at 09:37:03AM +0100, Sandro Bonazzola wrote: Hi, We're going to start composing oVirt 3.5.1 RC on *2014-11-25 08:00 UTC* from 3.5 branch. In order to stabilize the release a new branch ovirt-engine-3.5.1 will be created from the same git hash used for composing the RC. Maintainers: - Please be sure that 3.5 snapshot allow to create VMs before *2014-11-24 15:00 UTC* - Please be sure that no pending patches are going to block the release before *2014-11-24 15:00 UTC* - If any patch must block the RC release please raise the issue as soon as possible. A bug tracker [1] has been opened and shows 2 open blocker: Bug IDWhiteboard Status Summary 1142710 integration POSTVolume creation failed while deploying Hosted Engine on iSCSI 1160846 sla NEW Can't add disk to VM without specifying disk profile when the storage domain has more than one disk profile The following bugs have been keyworded as Regression and not marked as blockers: Bug IDWhiteboard Status Summary 1106435 infra POST[AAA] builtin kerbldap provider does not show error if search fails 1138144 storage NEW [BLOCKED]Failed to autorecover storage domain after unblocking connection with host 1118349 storage NEW [vdsm] Creating DataCenter 3.5 using master domain V1 fails with InquireNotSupportedError I've suggested Bug 1162640 - supervdsm segfault in libgfapi while querying volume status detail as a blocker. If this is not solved in a timely fasion on libgfapi, we'd need to disable glusterVolumeStatsInfoGet in ovirt-3.5.1. Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [ACTION REQUIRED] vdsm_master_install-rpm-sanity-el6_created bugged and vdsm spec file broken
On Mon, Nov 10, 2014 at 08:24:40AM -0500, Ondřej Svoboda wrote: Hi, I am in favor of tagging e.g. the commit introducing the hook in question [1] with v4.16.99. (Similarly, in the message to ovirt-users [2] I suggested that anyone interested in the hook tag their master branch with v4.17-fake.) I think that the tag v4.17 is very similar to the eventual tag v4.17.0 and might be confusing or sounding like a hack; but they are different tags nonetheless so I am not opposed to this solution either. Introducing the tag would make installing the hook painless to both groups (those on v4.16.7; and those on master, i.e., post-v4.16.0). [1] hooks: Add a hook to configure IPv6 networking through custom properties e42c9f3e9a2a4f759a66f11b3572d21c2b28d7c6 [2] IPv6 configuration through custom network properties and vdsm-hook-ipv6 I've ended up tagging master with v4.17.0. - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: devel@ovirt.org, infra in...@ovirt.org Sent: Thursday, November 6, 2014 8:25:09 AM Subject: [ovirt-devel] [ACTION REQUIRED] vdsm_master_install-rpm-sanity-el6_created bugged and vdsm spec file broken Looking at: http://jenkins.ovirt.org/job/vdsm_master_install-rpm-sanity-el6_created/570/console The job is trying to install the src.rpm and fails: 16:45:09 Cannot add package /tmp/vdsm-4.16.0-501.gitf981b80.el6.src.rpm to transaction. Not a compatible architecture: src 16:45:11 Cannot add package /tmp/vdsm-4.16.0-501.gitf981b80.fc20.src.rpm to transaction. Not a compatible architecture: src After that, some rpms fails the install: Package: vdsm-hook-ipv6-4.16.0-501.gitf981b80.el6.noarch (/vdsm-hook-ipv6-4.16.0-501.gitf981b80.el6.noarch) 16:45:34Requires: vdsm = 4.16.7 Master must bump version to 4.16.99 or 4.17 or something like that in order to be installable. Last thing: Cents 6.6 now deliver glusterfs 3.6.0.28 and vdsm is explicitly requiring 3.5.2. Anything preventing to move to = 3.5.2 instead of = 3.5.2? -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
On Fri, Nov 14, 2014 at 04:24:17PM -0500, Nir Soffer wrote:
Hi everyone,
the recent JSONRPC discussion and my last work around vdsClient reminded
me
that
since a long time there it was vague plans for a vdsClient overhaul
or at least I haven't ever found a detailed plan :)
Since we have to add jsonrpc support anyway, it could be a good chance to
bootstrap
a new vdsClient, and incrementally enhance it as needs arises; the old
vdsClient will
not go away very soon, I guess.
At least, we can to put together a wishlist for a new vdsClient and add
it
to
the wiki.
My one, so far:
* client library package: we should have a program-friendly python
package
to
access
VDSM, in order
* JSON for input: I's like to open logs, grab a line including a command
and
feed
vdsClient with the very same line
Currently we don't see commands json input in the logs, but this can be a
good
idea (in debug mode).
+1
JSON for output: current vdsClient format replies in a mix of text and
json. We
need to print the json returned from vdsm as is.
but a pretty-printed output can be useful, too.
I started discussion about it and here is what I was able to gather:
- we should start new vdsClient like tool which holds current vdsClient
command line interface.
- vdsClient should stay as it is and we can extend new client with
additional
functionality as we need (adding debug mode etc)
Current vdsClient horrible - we need to replace it with something better.
and never look back. it's beyond redemtion.
- The interface is not consistent, trying to convert command line options to
xmlrpc call
- Error handling is not helpful, showing an error message and then printing
help for all verbse, so the error message is hidden above your screen
- Help message is not useful, we need only the names of the parameters
- We need extended help for specific verb - like git help commit
- Command output is not consistent and unusable for parsing in scripts
We need to print the returned value from vdsm as is, or prety printed.
- there was a suggestion to reuse ovirt-engine-cli ideas and it should
use schema
to determine the API
Using the schema can be nice to generate help for the commands.
I started a simple jsonrpc client that can be used for testing *now*:
http://gerrit.ovirt.org/35181
Example usage:
Calling method without arguements:
# jsonrpc Host.getVMList
{
jsonrpc: 2.0,
id: 0e043d83-294a-4d31-b1b6-6dc2f2747494,
result: [
b3f6fa00-b315-4ad4-8108-f73da817b5c5
]
}
Calling method with arguements:
# jsonrpc VM.getStats '{vmID: b3f6fa00-b315-4ad4-8108-f73da817b5c5}'
{
jsonrpc: 2.0,
id: cefd25a3-6250-4123-8a56-d7047899e19e,
result: [
{
status: Down,
exitMessage: Admin shut down from the engine,
vmId: b3f6fa00-b315-4ad4-8108-f73da817b5c5,
exitReason: 6,
timeOffset: 0,
exitCode: 0
}
]
}
Thanks, this looks very nice and ready for contrib, imho.
I'd love to see something similar done to make vdscli connect over
jsonrpc. When I need to do something less than trivial from the command
line, I start python and
from vdsm import vdscli
s = vdscli.connect() # does a horribly useful heuristic, to connect
# to local host over xmlrpc
print s.getVMList()
I'd like to see vdscli.connect() extended to jsonrpc. This would
simplify
Bug 1112594 - Use jsonrpc during migration of vms
too.
Dan.
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
On Tue, Nov 11, 2014 at 05:31:37AM -0500, Nir Soffer wrote: - Original Message - From: Francesco Romani from...@redhat.com To: devel@ovirt.org Cc: Nir Soffer nsof...@redhat.com Sent: Tuesday, November 11, 2014 12:24:43 PM Subject: Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6 - Original Message - From: Nir Soffer nsof...@redhat.com To: Francesco Romani from...@redhat.com Cc: devel@ovirt.org Sent: Monday, November 10, 2014 1:21:53 PM Subject: Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance onRHEL/Centos 6 We have vdsm/lib/compat.py for this stuff. Then everyone needing json will do from vdsm.compat import json Do you think is better to have an hard dependency on 'simplejson' or a soft one like I showed above hidden on compat.py? It depends on the availability of simplejson. If it available on all platforms, we can require it. We've learned the hard way that this was not the case.. http://gerrit.ovirt.org/#/c/35354 ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
On Thu, Nov 20, 2014 at 11:12:44AM +0100, Sven Kieske wrote: On 19/11/14 18:36, Dan Kenigsberg wrote: We've learned the hard way that this was not the case.. http://gerrit.ovirt.org/#/c/35354 Why don't you just require this one package from epel than? epel is included in EL7, you just need to activate it. Furthermore, some engine packages are also just available via epel, so I see no reason to not require epel for vdsm too? RHEV cannot depend on EPEL. If a package is not in RHEL, RHEV must consume it, which is something that's not worth doing in this case (where there's not functional benefit). Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
On Thu, Nov 20, 2014 at 12:36:20PM +0100, Sven Kieske wrote: On 20/11/14 12:18, Dan Kenigsberg wrote: RHEV cannot depend on EPEL. If a package is not in RHEL, RHEV must consume it, which is something that's not worth doing in this case (where there's not functional benefit). As I said: it already does that: an excerpt from the latest ovirt-release35-001-1.noarch.rpm: in /./usr/share/ovirt-release35/ovirt-el6-deps.repo you find: [ovirt-3.5-epel] name=Extra Packages for Enterprise Linux 6 - $basearch #baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6arch=$basearch failovermethod=priority enabled=1 includepkgs=epel-release,python-uinput,puppet,python-lockfile,python-cpopen,python-ordereddict,python-pthreading,python-inotify,python-argparse,novnc,python-ply,python-kitchen,python-daemon,python-websockify,livecd-tools,spice-html5,mom,python-IPy,python-ioprocess,ioprocess,rubygem-rgen gpgcheck=1 gpgkey=https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6 So why can engine depend on epel and vdsm can not? engine and vdsm both can, and do, depend on epel. However, RHEV cannot. Taking an epel package has a price of needing to ship it in http://ftp.redhat.com/pub/redhat/rhev-m/3.x/SRPMS/ (see python-inotify there, as a vdsm dependency). In case of simplejson for el7, it simply does not worth it. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
On Sun, Nov 23, 2014 at 03:08:59PM -0500, Saggi Mizrahi wrote:
It's well known.
I'd drop the well - it was not known to me, and to others.
The problem is that there is no simplejson in EL6\7
(unless the situation changed since I last checked).
Francesco verified that RHEL6 does have simplejson. RHEL7 does not have
it (and does not need it as anxiously).
I already did extensive profiling on the jsonrpc framework.
The patches [1] move us on par with XML-RPC for sequential requests and
*way* faster for concurrent commands.
Thanks!
There are thoughts about adding msgpack [2] as an optional encoding for
responses as it's much smaller and less cpu intensive to encode\decode
but we have bigger fish to fry ATM (patches welcome).
[1]
http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:stomp_performance,n,z
[2] http://msgpack.org/
- Original Message -
From: Francesco Romani from...@redhat.com
To: devel@ovirt.org
Sent: Monday, November 10, 2014 10:02:44 AM
Subject: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on
RHEL/Centos 6
Hi everyone,
I was doing JSON-RPC investigation recently, running VDSM on RHEL6.{5,6},
and while some initial profile work, I discovered a (little) performance
degradation
about pure JSONRPC coding/encoding, *only* on the above platforms. Here's
why:
- the JSON codec module in the python stdlib is based on simplejson
- (thus) simplejson API is identical to stdlib's json module API
- python 2.6 JSON codec is (based on) simplejson 1.9
- python 2.7 JSON codec is (based on) simplejson 2.0.9, with significant
speedups. See
https://docs.python.org/2/whatsnew/2.7.html#new-and-improved-modules and
https://bugs.python.org/issue4136
- RHEL6.x/Centos 6.x includes python 2.6, so here (and only here) JSON codec
is unnecessarily slower.
- RHEL6.x (surely) and CentOS 6.x (need to check, don't see why not)
includes
anyway simplejson 2.0.9 as external package
So, it seems to me that we can get python 2.7 speed even on stable platform
with
a five line patch:
=== cut here ===
diff --git a/lib/yajsonrpc/__init__.py b/lib/yajsonrpc/__init__.py
index b3fd590..6682fc3 100644
--- a/lib/yajsonrpc/__init__.py
+++ b/lib/yajsonrpc/__init__.py
@@ -12,7 +12,10 @@
# You should have received a copy of the GNU General Public
# License along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
-import json
+try:
+import simplejson as json
+except ImportError
+import json
import logging
from functools import partial
from Queue import Queue
=== cut here ===
Here's an excerpt of the effects of this patch, using an initial still rough
bencmark
- once again 100 VMs boot, is the beginning of every testing routine from
mine,
so nothing new here.
Let's consider two initial profiles. Please note that
1) the amount of calls is not the same (error from mine in profiling, need
more precise bench)
but still
2) using 2.0.9 module JSON just vanishes from the profile alltogether
Vanilla VDSM, stdlib json, top 20 expensive calls
ncalls tottime percall cumtime percall filename:lineno(function)
19/100 127.2166.6969.9740.100
/usr/lib/python2.6/site-packages/mom/GuestMonitor.py:51(GuestMonitor.run)
1555/1635 41.1800.026 58.5660.036
/usr/lib64/python2.6/threading.py:481(Thread.run)
11708/1553160 11.9670.001 37.9210.000
/usr/lib64/python2.6/copy.py:144(deepcopy)
12686136.1540.0009.2090.000
/usr/lib64/python2.6/copy.py:261(_keep_alive)
4890/1300936.1490.001 37.5960.000
/usr/lib64/python2.6/copy.py:251(_deepcopy_dict)
497858/20700785.2240.000 11.1020.000
/usr/lib64/python2.6/json/encoder.py:284(JSONEncoder._iterencode)
498440/13334744.1870.000 10.3680.000
/usr/lib64/python2.6/json/encoder.py:213(JSONEncoder._iterencode_dict)
12/1004.1150.3434.1300.041
/usr/share/vdsm/virt/sampling.py:424(VmStatsThread.run)
438263.6920.0006.5160.000
/usr/lib64/python2.6/xml/dom/expatbuilder.py:743(ExpatBuilderNS.start_element_handler)
73453.6570.000 11.0470.002
/usr/share/vdsm/virt/vm.py:2264(Vm._getRunningVmStats)
9666/3205083.5680.0003.5720.000
/usr/lib64/python2.6/xml/dom/minidom.py:305(_get_elements_by_tagName_helper)
274994/2750003.3390.0005.5930.000
/usr/lib64/python2.6/StringIO.py:208(StringIO.write)
902171.8230.0001.8700.000
/usr/lib64/python2.6/xml/dom/minidom.py:349(Attr.__init__)
44025/440361.6660.0002.4700.000
/usr/share/vdsm/storage/lvm.py:217(makeLV)
144212/1442261.2870.0001.2920.000
Re: [ovirt-devel] [QE][ACTION REQUIRED] oVirt 3.5.1 RC status - postponed
On Tue, Nov 25, 2014 at 07:07:14AM +0100, Sandro Bonazzola wrote: Hi, We'were supposed to start composing oVirt 3.5.1 RC today *2014-11-25 08:00 UTC* from 3.5 branch. We have still blockers for oVirt 3.5.1 RC release so we need to postpone it untill they'll be fixed. The new tentative date for 3.5.1 RC composition is set to *2014-12-02 08:00 UTC The bug tracker [1] shows 4 open blocker: Bug IDWhiteboard Status Summary 1164218 POSTglfs_set_volfile_server() method causes segmentation fault when bad arguments are passed. Removed from tracker. The following BZ is enough for 3.5.1: Bala, Darshan: http://gerrit.ovirt.org/#/c/35256/ needs approval and a quick backport + merge. It would not solve the not-yet-understood failure of glfs_new, but it would avoid the segfault. 1162640 gluster POSTsupervdsm segfault in libgfapi while querying volume status detail ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] engine-setup fails on first attempt due to Command '/bin/systemctl' failed to execute then succeeds.
(Poor mailing list, needs to duplicate so much logs to so many
subscribers. Pastebin is so more efficient)
On Wed, Nov 26, 2014 at 06:57:27AM -0500, Mooli Tayer wrote:
2014-11-26 12:17:45 DEBUG otopi.plugins.otopi.services.systemd
plugin.execute:936 execute-output: ('/bin/systemctl', 'stop',
'nfs-server.service') stderr:
2014-11-26 12:17:45 DEBUG otopi.plugins.otopi.services.systemd
systemd.state:134 starting service nfs-server
2014-11-26 12:17:45 DEBUG otopi.plugins.otopi.services.systemd
plugin.executeRaw:823 execute: ('/bin/systemctl', 'start',
'nfs-server.service'), executable='None', cwd='None', env=None
2014-11-26 12:17:45 DEBUG otopi.plugins.otopi.services.systemd
plugin.executeRaw:873 execute-result: ('/bin/systemctl', 'start',
'nfs-server.service'), rc=1
2014-11-26 12:17:45 DEBUG otopi.plugins.otopi.services.systemd
plugin.execute:931 execute-output: ('/bin/systemctl', 'start',
'nfs-server.service') stdout:
2014-11-26 12:17:45 DEBUG otopi.plugins.otopi.services.systemd
plugin.execute:936 execute-output: ('/bin/systemctl', 'start',
'nfs-server.service') stderr:
Job for nfs-server.service failed. See 'systemctl status nfs-server.service'
and 'journalctl -xn' for details.
could you look for details of the nfs-server.service startup failure?
Dan.
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] Fwd: Got an error after buuilding VDSM on latest upstream commit
On Sun, Nov 30, 2014 at 01:00:28PM +0200, ybronhei wrote: forwarding to devels anyone familiar with the issue already ? any ready patch? Forwarded Message Subject: Got an error after buuilding VDSM on latest upstream commit Date: Sun, 30 Nov 2014 05:54:45 -0500 (EST) From: Eli Mesika emes...@redhat.com To: Dan Kenigsberg dan...@redhat.com, Saggi Mizrahi smizr...@redhat.com, ybronhei ybron...@redhat.com, Dima Kuznetsov dkuzn...@redhat.com CC: Oved Ourfali oourf...@redhat.com, Barak Azulay bazu...@redhat.com vdsm: Running restore_nets libvirt: Network Driver error : Network not found: no network with matching name 'vdsm-ovirtmgmt' ^X^CTraceback (most recent call last): File /usr/share/vdsm/vdsm-restore-net-config, line 139, in module restore() File /usr/share/vdsm/vdsm-restore-net-config, line 125, in restore unified_restoration() File /usr/share/vdsm/vdsm-restore-net-config, line 70, in unified_restoration setupNetworks(nets, bonds, connectivityCheck=False, _inRollback=True) File /usr/share/vdsm/network/api.py, line 715, in setupNetworks implicitBonding=True, _netinfo=_netinfo, **d) File /usr/share/vdsm/network/api.py, line 221, in wrapped ret = func(**attrs) File /usr/share/vdsm/network/api.py, line 310, in addNetwork netEnt.configure(**options) File /usr/share/vdsm/network/models.py, line 196, in configure self.configurator.configureBridge(self, **opts) File /usr/share/vdsm/network/configurators/ifcfg.py, line 92, in configureBridge ifup(bridge.name, bridge.ipConfig.async) File /usr/share/vdsm/network/configurators/ifcfg.py, line 820, in ifup rc, out, err = _ifup(iface) File /usr/share/vdsm/network/configurators/ifcfg.py, line 804, in _ifup rc, out, err = utils.execCmd([constants.EXT_IFUP, netIf], raw=False) File /usr/lib/python2.6/site-packages/vdsm/utils.py, line 622, in execCmd (out, err) = p.communicate(data) File /usr/lib64/python2.6/subprocess.py, line 732, in communicate stdout, stderr = self._communicate(input, endtime) File /usr/lib64/python2.6/subprocess.py, line 1316, in _communicate stdout, stderr = self._communicate_with_poll(input, endtime) File /usr/lib64/python2.6/subprocess.py, line 1388, in _communicate_with_poll ready = poller.poll(self._remaining_time(endtime)) The traceback seems snipped. I woder which exception is raised deep done. As it happens with our new unified_restoration() of network definitions, please share the content of your /var/lib/vdsm/netconf/* Could it be that a network there referes to a nic that no longer exists? ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [QE][ACTION REQUIRED] oVirt 3.5.1 RC status - postponed
On Tue, Dec 02, 2014 at 06:44:50AM +0100, Sandro Bonazzola wrote: Hi, We were supposed to start composing oVirt 3.5.1 RC today *2014-12-02 08:00 UTC* from 3.5 branch. We have still blockers for oVirt 3.5.1 RC release so we need to postpone it untill they'll be fixed. The new tentative date for 3.5.1 RC composition is set to *2014-12-09 08:00 UTC The bug tracker [1] shows 2 open blocker: Bug IDWhiteboard Status Summary 1168689 infra ASSIGNEDUpgrading from 3.4.4 to 3.5.1 snapshot the vdsm rpm throw an exception This one actuall has two patches merged. I moved it back to ASSIGNED by mistake, as I did not notice the section added to http://gerrit.ovirt.org/#/c/35763/2/vdsm.spec.in Anyway, this bloke is behind us. 1160846 sla NEW Can't add disk to VM without specifying disk profile when the storage domain has more than one disk profile In order to stabilize the release a new branch ovirt-engine-3.5.1 will be created from the same git hash used for composing the RC. Maintainers: - Please be sure that 3.5 snapshot allow to create VMs before *2014-12-08 15:00 UTC* - Please be sure that no pending patches are going to block the release before *2014-12-08 15:00 UTC* - If any patch must block the RC release please raise the issue as soon as possible. There are still 69 bugs [2] targeted to 3.5.1. Excluding node and documentation bugs we still have 46 bugs [3] targeted to 3.5.1. Maintainers / Assignee: - Please add the bugs to the tracker if you think that 3.5.1 should not be released without them fixed. - Please update the target to 3.5.2 or later for bugs that won't be in 3.5.1: it will ease gathering the blocking bugs for next releases. - Please fill release notes, the page has been created here [4] Community: - If you're testing oVirt 3.5 nightly snapshot, please add yourself to the test page [5] [1] http://bugzilla.redhat.com/1155170 [2] http://goo.gl/7G0PDV [3] http://goo.gl/6gUbVr [4] http://www.ovirt.org/OVirt_3.5.1_Release_Notes [5] http://www.ovirt.org/Testing/oVirt_3.5.1_Testing -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] VDSM sync meeting minutes December 9th, 2014
- Everybody should review Piotr's
http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:schema_verification,n,z
and approve the API of their respective component.
Francesco: virt
Nir: storage
Adam: live{merge,snapshot}, mother patch
Ido: network
- Nir suggest to generate API from schema (or vice versa) instead of verifying
that the duplicated API definition is consistent.
Adam agrees that API.py can go away; but that Piotr's work is helpful sanity
check for the short term.
- F21 needs testing. Petr Horacek reports that vdsmd works exteremely
slowly when started via systemd on Fedora 21. Please report to this
list if you have good/bad experience with it.
- el6 nearing its end of support. clusterLevel 3.6 features would be
limitted to el7.
- Adam suggests to tag legacy code with
# REQUIRED_FOR: el6
Or
# REQUIRED_FOR: engine-3.0
So it is easier to find and drop when we remove support of legacy
platform or legacy client.
- Nir and Federico to review Amador's iscsiadm.iface_list()
http://gerrit.ovirt.org/35807 and follow-up patches.
Dan
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] 3.5.1 RC schedule [was Re: [ovirt-users] oVirt Weekly Sync Meeting: Dec. 10, 2014]
On Thu, Dec 11, 2014 at 08:36:46AM +0100, Sandro Bonazzola wrote: Il 10/12/2014 16:54, Brian Proffitt ha scritto: (On time this week!) = #ovirt: oVirt Weekly Sync = [cut] * 3.5.z updates Two blockers have postponed RC again. New RC date will be discussed and decided in mailing list. (bkp, 15:30:01) [cut] Let's start the discussion and decision task. We have an ETA on the last pending blocker for 3.5.1 on next week. I suggest to postpone RC and GA after winter holidays. What about: RC - 2015-01-07 GA - 2015-01-14 Can we have a vdsm respin before that? We get repeated reports of bugs that are already solved in the stable branch. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] oVirt 3.6 Feature: Cumulative Network Usage Statistics
On Mon, Dec 22, 2014 at 01:40:06PM +0200, Lior Vernia wrote: Hello users and developers, Just put up a feature page for the aforementioned feature; in summary, to report total RX/TX statistics for hosts and VMs in oVirt. This has been requested several times on the users mailing list, and is especially useful for accounting in VDI deployments. You're more than welcome to review the feature page: http://www.ovirt.org/Features/Cumulative_RX_TX_Statistics Sorry for the late review; I have a couple of questions/comments. - What do you mean by VDI use cases in the Benefit to oVirt sanpshot section? Do you refer to hosting services who would like to charge their customers based on actual bandwidth usage? - I've added another motivation: currently-reported rxRate/txRate can be utterly meaningless. I don't see reference to nasty negative flows: what happens if a host disappears? Or a VM? I suppose there's always a chance that some traffic would go unaccounted for. But do you expect to extract this information somehow? Either way, it should be mentioned as a caveat on the feature page. Note that this only deals with network usage - it'll be great if we have similar features for CPU and disk usage! There's a formal feature request about this: Bug 1172153 - [RFE] Collect CPU, IO and network accounting information Dan ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] python-blivet requirement not available on el6
On Mon, Jan 19, 2015 at 11:55:48AM +0100, Sandro Bonazzola wrote: Hi, looks like vdsm added a dependency not available on EPEL 6. http://jenkins.ovirt.org/job/repos_master_check-closure_merged/DISTRIBUTION=centos6/388/console package: vdsm-4.17.0-283.gite127687.el6.x86_64 from check-custom-el6 unresolved deps: python-blivet package: vdsm-4.17.0-292.git0b55852.el6.x86_64 from check-custom-el6 unresolved deps: python-blivet Can you share some details about this requirement? Is anybody working with epel maintainer to get this in el6? I'm adding Bala and Sahina for more authoritative response. (which should have happened before having merged commit 7e49db74d) On the mean while, I've added http://gerrit.ovirt.org/37010 to sweep the problem under the carpet. Please rebase on top of master to obtain it. Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] vdsm: unit test breakage on el6
Some of oVirt-3.6 new features such as http://www.ovirt.org/Features/hostdev_passthrough would not be available on el6. However, we would like to be able to build and run legacy features on el6. Therefore, please avoid using fancy python 2.7 syntax. Unit tests on el6 should keep passing. Regards, Dan. http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/14115/console ERROR: testParseSRIOV_PFDeviceParams (hostdevTests.HostdevTests) -- Traceback (most recent call last): File /home/jenkins/workspace/vdsm_master_unit_tests_gerrit_el/vdsm/tests/hostdevTests.py, line 470, in testParseSRIOV_PFDeviceParams deviceXML = hostdev._parse_device_params(_SRIOV_PF_XML) File /home/jenkins/workspace/vdsm_master_unit_tests_gerrit_el/vdsm/vdsm/hostdev.py, line 77, in _parse_device_params params['physfn'] = _pci_address_to_name(**address.attrib) File /home/jenkins/workspace/vdsm_master_unit_tests_gerrit_el/vdsm/vdsm/hostdev.py, line 42, in _pci_address_to_name function[2:]) ValueError: zero length field name in format ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] Adding support for 3.6 in engine database
On Wed, Jan 14, 2015 at 08:19:00AM -0500, Oved Ourfali wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: Oved Ourfali ov...@redhat.com, fsimo...@redhat.com Cc: engine-de...@ovirt.org devel@ovirt.org Sent: Wednesday, January 14, 2015 3:15:02 PM Subject: Re: [ovirt-devel] Adding support for 3.6 in engine database On Thu, Jan 08, 2015 at 08:46:09AM -0500, Oved Ourfali wrote: - Original Message - From: Eli Mesika emes...@redhat.com To: Lior Vernia lver...@redhat.com, Oved Ourfali oourf...@redhat.com Cc: engine-de...@ovirt.org devel@ovirt.org, Dan Kenigsberg dan...@redhat.com, Yaniv Bronheim ybron...@redhat.com Sent: Thursday, January 8, 2015 3:41:31 PM Subject: Re: [ovirt-devel] Adding support for 3.6 in engine database - Original Message - From: Lior Vernia lver...@redhat.com To: Eli Mesika emes...@redhat.com Cc: engine-de...@ovirt.org devel@ovirt.org, Dan Kenigsberg dan...@redhat.com, Yaniv Bronheim ybron...@redhat.com Sent: Thursday, January 8, 2015 3:08:24 PM Subject: Re: [ovirt-devel] Adding support for 3.6 in engine database Tried to work with it, and noticed that: 1. The engine doesn't list 4.17 as a supported vdsm version. 2. 4.17 vdsm doesn't report 3.6 as a supported engine version. This basically means that no host could be operational in a 3.6 cluster, as to my understanding 4.17 is exactly the version supporting 3.6 functionality. May I send a fix for (1), or is there any argument against? And who could take care of (2)? I had understood deom Oved that this is 4.16 see patch http://gerrit.ovirt.org/#/c/36511/ Oved ??? I don't know when we should add 4.17. I remember there is some policy for that. Dan? Yes, there is. Vdsm would like to declare its support of clusterLevel 3.6 only when it actually does. This is not yet the case, as we are not yet in 3.6 feature freeze (heck, we're not yet in feature definition). To test cluster level 3.6 on the master branch, someone has to lie. It may be Vdsm (by claiming that it supports 3.6 while it does not) or Engine (by allowing vdsm 4.17 into cluster 3.6, even though it does not). I prefer the latter, as the Engine-side hack is eaiser to undo on a distributed system. If today's Vdsm claims that it already support 3.6, future Engines would add it to their cluster, only to find that random APIs fails. If the hack is Engine-side, it would be gone when 3.6 reaches feature freeze. We don't have a mechanism to allow specific version of VDSM to a specific cluster level. For this we only rely on the reported supported cluster levels. I know. I'm asking Engine to add it. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [ovirt-users] [Feature review] Select network to be used for glusterfs
On Thu, Jan 15, 2015 at 12:34:18PM +0530, Sahina Bose wrote: I've updated the feature page with the REST API and other comments. On further thought, there will be no change to Add brick API, as the engine will select the network to be used based on the networks setup for the host. If Storage network role is associated with any of the networks, this will be used. Otherwise, the host's address will be used to add the brick. snip The paragraph above rules out the use case I lay below. Could you relate to it? Isn't it a reasonable use case? If I am not mistaken, it could make sense to have a setup with one brick using network A and another - using network B. Does your design support this? I think that this would be particularly important on upgraded clusters, where the management network is already used, but newly created bricks should start using another network. May I repeat my follow request? It would help me understand the content of the feature. Would you add a feature page section regarding modification to the Vdsm/Engine API? One last comment - may I ask that new APIs accept both ipv4 and ipv6 addresses? There is an ongoing effort to support ipv6 on Vdsm. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [ovirt-users] [Feature review] Select network to be used for glusterfs
On Mon, Jan 12, 2015 at 02:59:50PM +0200, Lior Vernia wrote: On 12/01/15 14:44, Oved Ourfali wrote: Hi Sahina, Some comments: 1. As far as I understand, you might not have an IP available immediately after setupNetworks runs (getCapabilities should run, but it isn't run automatically, afair). 2. Perhaps you should pass not the IP but the name of the network? IPs might change. Actually, IP address can indeed change - which would be very bad for gluster functioning! I think moving networks or changing their IP addresses via Setup Networks should be blocked if they're used by gluster bricks. In the suggested feature, there is no real storage role. The storage role title means only default value for glusterfs IP. For example, once a brick was created, nothing protects the admin from accidently removing the storage network, or changing its IP address. Another proof that this is not a real role, is that it affects only GUI: I am guessing that REST API would not make use of it at all. (maybe I'm wrong; for sure, REST must be defined in the feature page) Maybe that's the behavior we want. But alternatively, Engine can enforce a stronger linkage between the brick to the network that it uses. When adding a brick, the dialog would list available networks instead of the specific IP. As long as the brick is being used, the admin would be blocked/warned against deleting the network. I'm missing a discussion regarding the upgrade path. If we would opt to requiring a single storage role network in a cluster, in an upgraded cluster the management network should take this role. 3. Adding to 2, perhaps using DNS names is a more valid approach? 4. You're using the terminology role, but it might be confusing, as we have roles with regards to permissions. Consider changing storage usage and not storage role in the feature page. Well, we've already been using this terminology for a while now concerning display/migration roles for networks... That's probably the terminology to use. Thanks, Oved - Original Message - From: Sahina Bose sab...@redhat.com To: devel@ovirt.org, users us...@ovirt.org Sent: Monday, January 12, 2015 2:00:16 PM Subject: [ovirt-users] [Feature review] Select network to be used for glusterfs Hi all, Please review the feature page for this proposed solution and provide your inputs - http://www.ovirt.org/Features/Select_Network_For_Gluster thanks sahina ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] vdsm to require newer pyflakes on el6
As mentioned on http://gerrit.ovirt.org/#/c/36434/ , vdsm's `make pyflakes` has been broken so that it gives a false sense of security, as it tests very few modules. During this period, we've acquired code that triggers pyflakes-0.4.0-2.el6.noarch.rpm of epel6. ./vdsm/sos/vdsm.py.in:25: redefinition of unused 'RedHatPlugin' from line 22 ./vdsm/sos/vdsm.py.in:25: redefinition of unused 'Plugin' from line 22 ./vdsm/virt/guestagent.py:144: redefinition of function 'guestDiskMapping' from line 140 vdsm/get-conf-item:11: redefinition of unused 'config' from line 8 vdsm/set-conf-item:13: redefinition of unused 'config' from line 10 make: *** [pyflakes] Error 1 I suggest requiring Vdsm developers to use a newer version of pyflakes, even on el6, such as the one provided by http://danken.fedorapeople.org/pyflakes-0.8.1-3.el6.noarch.rpm as the alternative is to add silly code in order to quiesce an ancient pyflakes. This would server as a stop-gap until we live up to our plan of including pep8 and pyflakes as git submodules, so that all vdsm developers use the same verification tool on all platforms. Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] vdsm to require newer pyflakes on el6
On Wed, Jan 07, 2015 at 08:05:33AM +0100, Sandro Bonazzola wrote: Il 06/01/2015 22:42, Dan Kenigsberg ha scritto: As mentioned on http://gerrit.ovirt.org/#/c/36434/ , vdsm's `make pyflakes` has been broken so that it gives a false sense of security, as it tests very few modules. During this period, we've acquired code that triggers pyflakes-0.4.0-2.el6.noarch.rpm of epel6. Please open an EPEL bug asking for an update of pyflakes on epel6. el6 is an aging platform, that is not expected to interest us for to long. But I've asked to rebase pyflakes there, why not https://bugzilla.redhat.com/show_bug.cgi?id=1080877#c1 Until this is accepted, please use my build (or your own). Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [ovirt-users] [Feature review] Select network to be used for glusterfs
On Tue, Jan 13, 2015 at 02:51:34PM +0200, Lior Vernia wrote: On 13/01/15 10:21, Sahina Bose wrote: On 01/12/2015 08:52 PM, Dan Kenigsberg wrote: On Mon, Jan 12, 2015 at 02:59:50PM +0200, Lior Vernia wrote: On 12/01/15 14:44, Oved Ourfali wrote: Hi Sahina, Some comments: 1. As far as I understand, you might not have an IP available immediately after setupNetworks runs (getCapabilities should run, but it isn't run automatically, afair). 2. Perhaps you should pass not the IP but the name of the network? IPs might change. Actually, IP address can indeed change - which would be very bad for gluster functioning! I think moving networks or changing their IP addresses via Setup Networks should be blocked if they're used by gluster bricks. In the suggested feature, there is no real storage role. The storage role title means only default value for glusterfs IP. For example, once a brick was created, nothing protects the admin from accidently removing the storage network, or changing its IP address. Another proof that this is not a real role, is that it affects only GUI: I am guessing that REST API would not make use of it at all. (maybe I'm wrong; for sure, REST must be defined in the feature page) REST API that lists the available networks (with IP addresses) would be used to select the network and pass to the create gluster volume API My question regarded the argument of the add brick API (in Engine level). Is it an IPv4 address (like it seems) or could it be a network name? I'll update the feature page with the REST API changes as well. If REST allows to choose the network used for gluster traffic, then I think so should the GUI - I would not drop the list box from the design in that case. Maybe that's the behavior we want. But alternatively, Engine can enforce a stronger linkage between the brick to the network that it uses. When adding a brick, the dialog would list available networks instead of the specific IP. As long as the brick is being used, the admin would be blocked/warned against deleting the network. Is there a way to block against changing IP address used by a network? Yes, this should be implemented at least in the canDoAction() method of SetupNetworksCommand (most of it is done in the SetupNetworksHelper class). And perhaps this should be blocked in the GUI as well. Note that by the time 3.6 is released, the REST (and probably GUI) are supposed to work with a different backend command that is currently being implemented - so maybe you'll need to modify that instead, or on top of the changes in SetupNetworksHelper. I'm missing a discussion regarding the upgrade path. If we would opt to requiring a single storage role network in a cluster, in an upgraded cluster the management network should take this role. There would not be any change to existing volumes on upgrade, as bricks have already been added. Users can use the Edit brick option to update the network to be used, if required as mentioned in Change network used by brick I suspect Dan referred to the upgrade path of the engine itself - if you add a new Gluster Network boolean column to the DB, it will initially be null for all current networks. You'd likely need to write an upgrade script to assign the role by default to the existing management networks in each cluster. yep. 3. Adding to 2, perhaps using DNS names is a more valid approach? 4. You're using the terminology role, but it might be confusing, as we have roles with regards to permissions. Consider changing storage usage and not storage role in the feature page. Well, we've already been using this terminology for a while now concerning display/migration roles for networks... That's probably the terminology to use. If I am not mistaken, it could make sense to have a setup with one brick using network A and another - using network B. Does your design support this? I think that this would be particularly important on upgraded clusters, where the management network is already used, but newly created bricks should start using another network. Would you add a feature page section regarding modification to the Vdsm/Engine API? One last comment - may I ask that new APIs accept both ipv4 and ipv6 addresses? There is an ongoing effort to support ipv6 on Vdsm. Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] Adding support for 3.6 in engine database
On Thu, Jan 08, 2015 at 08:46:09AM -0500, Oved Ourfali wrote: - Original Message - From: Eli Mesika emes...@redhat.com To: Lior Vernia lver...@redhat.com, Oved Ourfali oourf...@redhat.com Cc: engine-de...@ovirt.org devel@ovirt.org, Dan Kenigsberg dan...@redhat.com, Yaniv Bronheim ybron...@redhat.com Sent: Thursday, January 8, 2015 3:41:31 PM Subject: Re: [ovirt-devel] Adding support for 3.6 in engine database - Original Message - From: Lior Vernia lver...@redhat.com To: Eli Mesika emes...@redhat.com Cc: engine-de...@ovirt.org devel@ovirt.org, Dan Kenigsberg dan...@redhat.com, Yaniv Bronheim ybron...@redhat.com Sent: Thursday, January 8, 2015 3:08:24 PM Subject: Re: [ovirt-devel] Adding support for 3.6 in engine database Tried to work with it, and noticed that: 1. The engine doesn't list 4.17 as a supported vdsm version. 2. 4.17 vdsm doesn't report 3.6 as a supported engine version. This basically means that no host could be operational in a 3.6 cluster, as to my understanding 4.17 is exactly the version supporting 3.6 functionality. May I send a fix for (1), or is there any argument against? And who could take care of (2)? I had understood deom Oved that this is 4.16 see patch http://gerrit.ovirt.org/#/c/36511/ Oved ??? I don't know when we should add 4.17. I remember there is some policy for that. Dan? Yes, there is. Vdsm would like to declare its support of clusterLevel 3.6 only when it actually does. This is not yet the case, as we are not yet in 3.6 feature freeze (heck, we're not yet in feature definition). To test cluster level 3.6 on the master branch, someone has to lie. It may be Vdsm (by claiming that it supports 3.6 while it does not) or Engine (by allowing vdsm 4.17 into cluster 3.6, even though it does not). I prefer the latter, as the Engine-side hack is eaiser to undo on a distributed system. If today's Vdsm claims that it already support 3.6, future Engines would add it to their cluster, only to find that random APIs fails. If the hack is Engine-side, it would be gone when 3.6 reaches feature freeze. Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] el6 future in ovirt-3.6
Current master branch of vdsm (destined for our 3.6 release) uses el7 as its main platform. New features are expected to be available only on el7 (and modern Fedoras) but not on el6. On el6, vdsm still builds, runs, and exports clusterLevel = 3.5, with no feature loss relative to 3.5. This has been done per gluster request. However, maintaining this furhter as high costs: we keep testing el6, we need to make sure nothing breaks there, and there's a lot of legacy code that we could start deleting. Sahina, would you explain (again... sorry for not recalling the details) why ovirt-3.6's vdsm should keep running on el6? I'd like to see if there's a nother means to solve the underlying gluster issue. Regards, Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] el6 future in ovirt-3.6
On Mon, Mar 16, 2015 at 06:02:36PM +0200, Sahina Bose wrote: On 03/16/2015 03:28 PM, Dan Kenigsberg wrote: Current master branch of vdsm (destined for our 3.6 release) uses el7 as its main platform. New features are expected to be available only on el7 (and modern Fedoras) but not on el6. On el6, vdsm still builds, runs, and exports clusterLevel = 3.5, with no feature loss relative to 3.5. This has been done per gluster request. However, maintaining this furhter as high costs: we keep testing el6, we need to make sure nothing breaks there, and there's a lot of legacy code that we could start deleting. Sahina, would you explain (again... sorry for not recalling the details) why ovirt-3.6's vdsm should keep running on el6? I'd like to see if there's a nother means to solve the underlying gluster issue. This was only because downstream gluster + vdsm would continue to be supported on RHEL 6. Could you provide more details about your needs? Catering for d/s RHS is very important, but since it's expensive to maintain, I'd like to understand more. Please note again that when installed on el6, ovirt-3.6's vdsm would NOT expose clusterLevel=3.6, so theoretically you can keep ovirt-3.5's vdsm. Do you plan an Engine-side hack in order to use new 3.6 gluster functionality despite clusterLevel=3.5? Regards, Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] el6 future in ovirt-3.6
On Wed, Mar 18, 2015 at 09:39:45PM +0200, Sahina Bose wrote: On 03/17/2015 11:12 PM, Dan Kenigsberg wrote: On Mon, Mar 16, 2015 at 06:02:36PM +0200, Sahina Bose wrote: On 03/16/2015 03:28 PM, Dan Kenigsberg wrote: Current master branch of vdsm (destined for our 3.6 release) uses el7 as its main platform. New features are expected to be available only on el7 (and modern Fedoras) but not on el6. On el6, vdsm still builds, runs, and exports clusterLevel = 3.5, with no feature loss relative to 3.5. This has been done per gluster request. However, maintaining this furhter as high costs: we keep testing el6, we need to make sure nothing breaks there, and there's a lot of legacy code that we could start deleting. Sahina, would you explain (again... sorry for not recalling the details) why ovirt-3.6's vdsm should keep running on el6? I'd like to see if there's a nother means to solve the underlying gluster issue. This was only because downstream gluster + vdsm would continue to be supported on RHEL 6. Could you provide more details about your needs? Catering for d/s RHS is very important, but since it's expensive to maintain, I'd like to understand more. Please note again that when installed on el6, ovirt-3.6's vdsm would NOT expose clusterLevel=3.6, so theoretically you can keep ovirt-3.5's vdsm. Do you plan an Engine-side hack in order to use new 3.6 gluster functionality despite clusterLevel=3.5? I wasn't aware that ovirt-3.6 would not expose clusterLevel 3.6 when installed on el6. Downstream RHS will need to be supported on both el6 and el7. And we have added some apis to vdsm master that we require for brick management, geo-rep management and gluster volume snapshot management. We have 2 options - fork off vdsm if it needs to move on to el7 support only in master. We (developers) WANT to shed off el7. But RHS is an important user; and there are other users that are simply happy to use el6 and don't want to desert it yet. We can keep el6 support even on 3.6 (with clusterLevel=3.5). The only problem (beside maintanence cost) is how gluster can tell these hosts apart. Can Engine manage a clusterLevel=3.5.5? Vdsm-3.6 can report it even on el6 hosts. - backport to vdsm 3.5 (depending on what we decide on cluster level support for downstream) I find it dangerous to introduce new features to a stable branch. It may introduce unwarrented instability to non-gluster users. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] Adding dependency on glusterfs-3.7dev for vdsm-gluster patch.
On Fri, Mar 20, 2015 at 08:03:32AM -0400, Darshan Narayana Murthy wrote: Hi All, In the vdsm-gluster patch [1] we will be adding a dependency on glusterfs-3.7dev(alfa) as the dependent feature was added to glusterfs recently and its not present in any release of glusterfs. Since glusterfs-3.7dev is alfa, vdsm-gluster package installation will not directly pull this package. Users have to manually add the repo for getting this package. Repo for glusterfs-3.7dev can be added by installing suitable (depending on your os and its version) rpm located at [2]. For example install the rpm dgo-nightly-master-1.0-0.1.fc20.noarch.rpm located at [2] if your os is fedora-20. [1] https://gerrit.ovirt.org/#/c/38444/ [2] http://download.gluster.org/pub/gluster/glusterfs/nightly/glusterfs/ In my opinion this is acceptable for developers. +1 from me. However, this has implication to continuous integration. Hence I'd like to see Eyal and David sign off this, too. Dan ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] vdsm broke repoclosure (again) and node build
On Tue, Mar 31, 2015 at 08:50:05AM +0200, Sandro Bonazzola wrote: Hi, vdsm broke repository closure again: 06:37:42 package: vdsm-4.17.0-601.gitff96629.el7.x86_64 from check-custom-el7 06:37:42 unresolved deps: 06:37:42 libvirt-daemon = 0:1.2.8-15.el7 06:37:49 package: vdsm-4.17.0-601.gitff96629.fc20.x86_64 from check-custom-fc20 06:37:49 unresolved deps: 06:37:49 libvirt = 0:1.2.9 For Fedora 20 I'm solving by adding virt-preview repo to the repoclosure job. Couldn't the EL7 update wait a couple of days until CentOS 7.1 will be out? Sorry. We should have waited with https://gerrit.ovirt.org/#/c/38538/4/vdsm.spec.in However, the required versions of libvirt is actually out - though not on the official centos location yet http://mirror.centos.org/centos-7/7.1.1503/updates/x86_64/Packages/libvirt-1.2.8-16.el7_1.2.x86_64.rpm could this help saving the day? Above breakage in repository closure caused ovirt node iso build failures: http://jenkins.ovirt.org/job/ovirt-node_master_create-iso-el7_merged/57/ http://jenkins.ovirt.org/job/ovirt-node_master_create-iso-fc20_merged/105/ Tolik, please update http://jenkins.ovirt.org/job/ovirt-node_master_create-iso-fc20_merged adding virt-preview repo. Tolik, while at it, can you also check why the el6 job is requiring the hosted-engine plugin? It was supposed to not require it since in 3.6 hosted-engine won't be available on EL6. http://jenkins.ovirt.org/job/ovirt-node_master_create-iso-el6_merged/37/ ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] el6 future in ovirt-3.6
On Tue, Mar 17, 2015 at 09:47:17AM +0100, Sven Kieske wrote: +1 also take a look at the results of the user survey: most users use el6 so even the discussion about dropping support will make people uncertain how long they will be able to use el6. me included. Given the recent discussions, this seems the way we are going to take: ovirt-3.5 would keep support of el6 as long as it (ovirt-3.5) is maintained. ovirt-3.6 (current master) would no longer support el6 for the hypervisor. Vdsm would not be tested on el6, it won't build there, and most probably not run properly/at all on el6. If someone thinks that this is a wrong step to take - say so now, or be silent forever. Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] Proposing Sahina Bose as oVirt core maintainer
On Sun, Mar 01, 2015 at 06:56:55AM -0500, Oved Ourfali wrote: Hi all! I would like to propose Sahina Bose as an engine-core maintainer, with regards to Gluster related code. Sahina joined the oVirt project two years ago, and has since contributed over 150 patches, and has been instrumental in reviews of all Gluster patches. Will appreciate your response! +1, assuming she merges the patch that makes http://jenkins.ovirt.org/job/vdsm_master_verify-error-codes_merged/ fail for way too long! ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [ovirt-users] [ANN] oVirt 3.5.2 First Release Candidate is now available for testing
On Mon, Mar 02, 2015 at 11:35:00AM +0100, Sandro Bonazzola wrote: Il 02/03/2015 11:30, Dan Kenigsberg ha scritto: On Mon, Mar 02, 2015 at 08:46:51AM +0100, Sandro Bonazzola wrote: Il 27/02/2015 23:43, Dan Kenigsberg ha scritto: On Fri, Feb 27, 2015 at 03:48:22PM -0500, Nir Soffer wrote: - Original Message - From: Nir Soffer nsof...@redhat.com To: Sandro Bonazzola sbona...@redhat.com Cc: annou...@ovirt.org, us...@ovirt.org, devel@ovirt.org Sent: Friday, February 27, 2015 10:35:52 PM Subject: Re: [ovirt-users] [ovirt-devel] [ANN] oVirt 3.5.2 First ReleaseCandidate is nowavailable for testing - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: annou...@ovirt.org, devel@ovirt.org, us...@ovirt.org Sent: Friday, February 27, 2015 12:00:10 PM Subject: [ovirt-devel] [ANN] oVirt 3.5.2 First Release Candidate is now available for testing -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The oVirt team is pleased to announce that the 3.5.2 First Release Candidate is now available for testing as of Feb 27th 2015. The release candidate is available now for Fedora 20... Do we support Fedora 21? I don't recall a formal statement about it, but I surely WANT to support it, as nowadays it is the run-of-the-mill Fedora version. Please be sure that all relevant rpms for VDSM Fedora 21 support are published in ovirt-3.5-snapshot. Latest job run is:http://jenkins.ovirt.org/view/Publishers/job/publish_ovirt_rpms_nightly_3.5/298/console Important point in deed: no Vdsm build is there or in http://resources.ovirt.org/pub/ovirt-3.5-snapshot/rpm/fc21/noarch/ What should be done to enabled f21 builds of Vdsm? change the vdsm build job for 3.5 in order to make it building for fedora 21 too (yaml code in jenkins repo) This https://gerrit.ovirt.org/#/c/38273/ would be a starat? change the publisher job for 3.5 in order to publish the artifacts from the fedora 21 vdsm build job (manual change in jenkins job, not yet migrated to yaml) Same for all the vdsm deps. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [ovirt-users] [ANN] oVirt 3.5.2 First Release Candidate is now available for testing
On Mon, Mar 02, 2015 at 08:46:51AM +0100, Sandro Bonazzola wrote: Il 27/02/2015 23:43, Dan Kenigsberg ha scritto: On Fri, Feb 27, 2015 at 03:48:22PM -0500, Nir Soffer wrote: - Original Message - From: Nir Soffer nsof...@redhat.com To: Sandro Bonazzola sbona...@redhat.com Cc: annou...@ovirt.org, us...@ovirt.org, devel@ovirt.org Sent: Friday, February 27, 2015 10:35:52 PM Subject: Re: [ovirt-users] [ovirt-devel] [ANN] oVirt 3.5.2 First Release Candidate is nowavailable for testing - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: annou...@ovirt.org, devel@ovirt.org, us...@ovirt.org Sent: Friday, February 27, 2015 12:00:10 PM Subject: [ovirt-devel] [ANN] oVirt 3.5.2 First Release Candidate is now available for testing -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The oVirt team is pleased to announce that the 3.5.2 First Release Candidate is now available for testing as of Feb 27th 2015. The release candidate is available now for Fedora 20... Do we support Fedora 21? I don't recall a formal statement about it, but I surely WANT to support it, as nowadays it is the run-of-the-mill Fedora version. Please be sure that all relevant rpms for VDSM Fedora 21 support are published in ovirt-3.5-snapshot. Latest job run is:http://jenkins.ovirt.org/view/Publishers/job/publish_ovirt_rpms_nightly_3.5/298/console Important point in deed: no Vdsm build is there or in http://resources.ovirt.org/pub/ovirt-3.5-snapshot/rpm/fc21/noarch/ What should be done to enabled f21 builds of Vdsm? ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [ovirt-users] [ANN] oVirt 3.5.2 First Release Candidate is now available for testing
On Fri, Feb 27, 2015 at 03:48:22PM -0500, Nir Soffer wrote: - Original Message - From: Nir Soffer nsof...@redhat.com To: Sandro Bonazzola sbona...@redhat.com Cc: annou...@ovirt.org, us...@ovirt.org, devel@ovirt.org Sent: Friday, February 27, 2015 10:35:52 PM Subject: Re: [ovirt-users] [ovirt-devel] [ANN] oVirt 3.5.2 First Release Candidate is nowavailable for testing - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: annou...@ovirt.org, devel@ovirt.org, us...@ovirt.org Sent: Friday, February 27, 2015 12:00:10 PM Subject: [ovirt-devel] [ANN] oVirt 3.5.2 First Release Candidate is now available for testing -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The oVirt team is pleased to announce that the 3.5.2 First Release Candidate is now available for testing as of Feb 27th 2015. The release candidate is available now for Fedora 20... Do we support Fedora 21? I don't recall a formal statement about it, but I surely WANT to support it, as nowadays it is the run-of-the-mill Fedora version. May be also relevant for Fedora 20 with virt-preview repository. Note also that we cannot use libvirt 1.2.8 on Fedora 20, since it contains critical bug that breaks live storage migration. This bug was fixed in libvirt 1.2.9. See https://bugzilla.redhat.com/1176673 Summary: - Fedora 20 is ok since it uses libvirt 1.1.3. - Fefora 20 with virt-preview must provide libvirt = 1.2.9 and vdsm must include the patch bellow - Fedora 21 provides libvirt 1.2.9 and vdsm must include the patch bellow. These requirement are not enforced in vdsm spec yet. If we do, then this fix _must_ be included in 3.5.2: https://gerrit.ovirt.org/38201 vm: Fix LSM when using libvirt = 1.2.8 Please mark the relevant bug as a blocker of the 3.5.2 tracker. That would make sure we don't release without it. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] dynamic tuning of logging
On Thu, Mar 05, 2015 at 09:50:26AM +0100, Sven Kieske wrote: On 04/03/15 18:24, Nir Soffer wrote: vdsClient -s 0 setLogLevel 20 Does this survive a restart of the vdsm service? No, this is not persisted. If you want that, you should edit /etc/vdsm/vdsm/logger.conf ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] about improved libvirt-python bindings
On Tue, Feb 24, 2015 at 05:40:07AM -0500, Francesco Romani wrote:
- Original Message -
From: Dan Kenigsberg dan...@redhat.com
To: Francesco Romani from...@redhat.com
Cc: devel@ovirt.org
Sent: Tuesday, February 24, 2015 11:19:20 AM
Subject: Re: [ovirt-devel] [VDSM] about improved libvirt-python bindings
On Wed, Feb 18, 2015 at 07:53:17AM -0500, Francesco Romani wrote:
Hi,
during my sampling overhaul effort I faced various times a mismatch
between
what the libvirt-python API offers for bulk stats and what VDSM needs.
The return value of bulk stats is a something like
[(dom1, {dict_of_bulk_stats1}), {dom2, {dict_of_bulk_stats2}), ...]
While VDSM almost all the times really wants to use
{ dom_uuid_1: dict_of_bulk_stats_1, dom_uuid_2: dict_of_bulk_stats_2 }
translation is trivial to be coded, but bulk stats retrieval is quite
always in the hot path. Moreover, this is just wasteful.
How big is the waste?
Will provide hard numbers (and possibly pretty graphs)
But in general, there will always be a not-perfect match from
libvirt-python
and VDSM needs. This is expected: libvirt-python needs to be more generic.
So, I coded a Proof of Concept extension module which *complements*
and not *replaces* the libvirt-python API.
Does it require Vdsm to hold two open connections to libvirtd? [...]
Nope, it is intended as transparent replacement, much like our pthreading
does for threading standard module.
ah, ok. please read again your I coded a Proof...API sentence to see
why I was confused ;-)
[...]
It was a nice and fun hack, but now, the question is: is that a concept
worth developing further? Do we want to use and depend on this module?
I believe that with such a buffer we can gain some agility with respect
to libvirt plans and needs, and the maintainership cost _seems_ fairly
low.
Thoughts welcome.
To me, adding this module seems quite expensive: new ovirt project,
inclusion in fedora/el/debian, versioning and release-engineering
hassle.
Right, but much of this cost should be bootstrap, not recurring, right?
Not trying to push things, just to assess the cons.
Well, maintaining another package per distribution has some recurrent
costs. It has to be rebuilt on each ovirt release (assuming no urgent
security bugs), and there might be unexpected needs to recompile (python
change, libvirt upgrade).
Regards,
Dan.
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [URGENT][ACTION REQUIRED] please fix VDSM requirements
On Wed, Mar 25, 2015 at 03:38:06AM -0400, Timothy Asir Jeyasingh wrote: Please give me one more day time. The fix patch is waiting for review. Regards, Tim - Original Message - From: Sandro Bonazzola sbona...@redhat.com To: devel@ovirt.org, Dan Kenigsberg dan...@redhat.com, Timothy Asir Jeyasingh tjeya...@redhat.com, Balamurugan Arumugam barum...@redhat.com Sent: Wednesday, March 25, 2015 12:42:37 PM Subject: [URGENT][ACTION REQUIRED] please fix VDSM requirements Hi, we still have repository closure error and CI jobs broken by the blivet requirement introduced with https://gerrit.ovirt.org/35498 on Mar 18. The merged patch https://gerrit.ovirt.org/38974 requiring it only at runtime is not enough to fix CI and repository issues. Please either revert the blivet requirement at all or try to get https://gerrit.ovirt.org/38942 in as soon as possible. It's one week since last time VDSM has been successfully tested by CI, please address the issue. Thanks, Tim, if this is not done by today, I'd hide the python-blivet = 0.61.14 dependency (which is slightly proplematic for gluster, but will keep the rest of ovirt tested). Sandro, we may have a similar issue with glusterfs: I've merged https://gerrit.ovirt.org/39075 that has a glusterfs = 3.6.999 requirement without an all-clear from you. Is glusterfs 3.7dev available on CI? If not, please direct the gluster folks what they must do to make it available there. Regards, Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] Relocating the vdsm upgrade logic into a new or existing service at boot time
On Thu, Apr 23, 2015 at 08:58:28AM -0400, Fabian Deutsch wrote: Hey, the logic which is run on vdsm upgrades is located in the specfile, and thus run when the rpm get's an update. This is fine as long as rpm is involved (like when rpm or yum or dnf is used), but it becomes - and already is - an issue for flows where the upgrade happens without rpm, like on oVirt Node. On oVirt Node the image for booting the host is getting replaced, we currently mimic the upgrade process of the specfile. But this error prone - we had a lot of bugs around this area (remember the problems with the hooks?), and we need to watch out for changes in the upgrade logic. To simplify and especially unify the logic between Node and normal hosts, I'd like to discuss if the upgrade logic can be moved to a separate service or included in one of the existsing services. The simplest approach to me would be top create a new service which is started before libvirtd and vdsmd/supervdsmd to run some logic if an upgrade of vdsm is detected. This habit - to do the upgrade logic during boot - is actually not new and systemd provides some helpers (ConditionNeedsUpdate=) around this to ease the implementation. Thoughts? The problem is real: we have too much logic on %post and %postun scripts, and end up with upgrade bugs litterally on every release. However, for non-node installation we still need to allow configuration after upgrade, but before next boot. This means that everything in %post should be converted into vdsm-tool configurators. https://gerrit.ovirt.org/#/c/39823/3/vdsm.spec.in is one small step in that direction. When Vdsm starts up, it performs several upgrade modules, which are used to upgrade vdsm-specific configurations. These too should operate properly also when run much after boot. Hence, I'm not sure about the benefit of adding an early systemd service, but I may be missing something. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] Maintainer rights on vdsm - ovirt-3.5-gluster
On Mon, Apr 20, 2015 at 03:20:18PM +0530, Sahina Bose wrote: Hi! On the vdsm branch ovirt-3.5-gluster, could you provide merge rights to Bala (barum...@redhat.com) ? +1 from me. ovirt-3.5-gluster needs a rebase on top of the current ovirt-3.5 ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] Relocating the vdsm upgrade logic into a new or existing service at boot time
On Fri, Apr 24, 2015 at 04:53:43AM -0400, Fabian Deutsch wrote: - Original Message - On Thu, Apr 23, 2015 at 08:58:28AM -0400, Fabian Deutsch wrote: Hey, the logic which is run on vdsm upgrades is located in the specfile, and thus run when the rpm get's an update. This is fine as long as rpm is involved (like when rpm or yum or dnf is used), but it becomes - and already is - an issue for flows where the upgrade happens without rpm, like on oVirt Node. On oVirt Node the image for booting the host is getting replaced, we currently mimic the upgrade process of the specfile. But this error prone - we had a lot of bugs around this area (remember the problems with the hooks?), and we need to watch out for changes in the upgrade logic. To simplify and especially unify the logic between Node and normal hosts, I'd like to discuss if the upgrade logic can be moved to a separate service or included in one of the existsing services. The simplest approach to me would be top create a new service which is started before libvirtd and vdsmd/supervdsmd to run some logic if an upgrade of vdsm is detected. This habit - to do the upgrade logic during boot - is actually not new and systemd provides some helpers (ConditionNeedsUpdate=) around this to ease the implementation. Thoughts? The problem is real: we have too much logic on %post and %postun scripts, and end up with upgrade bugs litterally on every release. However, for non-node installation we still need to allow configuration after upgrade, but before next boot. Can you tell what kind of configuration needs to be done after upgrade, before the next reboot? All of them. When adding a new non-node host, we no longer require a reboot before the host is deemed operational. yum install vdsm ; vdsm-tool configure; service vdsmd start should do everything. I'm saying that we should take logic out of rpm and into `vdsm-tool configure` This means that everything in %post should be converted into vdsm-tool configurators. https://gerrit.ovirt.org/#/c/39823/3/vdsm.spec.in is one small step in that direction. I agree, that change is exactly what I'm looking for. When Vdsm starts up, it performs several upgrade modules, which are used to upgrade vdsm-specific configurations. These too should operate properly also when run much after boot. Hence, I'm not sure about the benefit of adding an early systemd service, but I may be missing something. I was only thinking about the separate service, to make sure that any service who's configuration file get's possibly updated (i.e. libvirtd) has not yet been started. But if vdsmd and the configurators take care to restart the service who's configuration file got changed, then we do not need a separate one. It's somewhat ugly, but that's what vdsm-tool --nodule libvirt configure already does. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [oVirt Jenkins] vdsm_master_unit-tests_merged - Build # 4878 - Still Failing!
On Sun, Apr 19, 2015 at 01:26:47PM +0100, Jenkins ci oVirt Server wrote: Project: http://jenkins.ovirt.org/job/vdsm_master_unit-tests_merged/ Build: http://jenkins.ovirt.org/job/vdsm_master_unit-tests_merged/4878/ Build Number: 4878 Build Status: Still Failing Triggered By: Triggered by Gerrit: https://gerrit.ovirt.org/39806 Hi Martin, I have merged your https://gerrit.ovirt.org/39411 prematurely, and provided an insufficient fix to it. Could you fix it promptly? Regards, Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] vdsm: python3 support?
Vdsm depends on Python 2, but the developers of Python 2 wants us (and everybody else) to move to the slightly-different language of Python 3. There is a lot to be fixed for this to happen: - depency on libraries with no python3 port: - mom - cpopen - m2crypto - use of implicit relative imports, mostly in the storage package - use of renamed/moved functions, methods and modules (e.g. dict.iteritems, io.StringIO, itertools.izip) - use of contextlib.nested - unsafe unicode treatment - and probably much more... Fixing it all would take a long time, and may introduce regressions. We must perform it gradually. The very first step is to use syntax that is compatible with both languages, and make sure we do not regress on that front. The remaining bit is our usages of raise E, V, T statement, which has been dropped in PEP-3109. https://gerrit.ovirt.org/#/c/40214/ suggests to replace it with six.reraise(), which adds a new run-time dependency to Vdsm. https://gerrit.ovirt.org/#/c/40215/ adds an attempt to python3-compile all vdsm code, so that we do not introduce new python2-only code. Please approve/comment! ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [vdsm] VmDevices rework
On Tue, Apr 28, 2015 at 04:28:12AM -0400, Martin Polednik wrote: Hello everyone, I have started working on line of patches that deal with current state of VM devices in VDSM. There is a wiki page at [1] describing the issues, phases and final goals for this work. Additionally, there is formal naming system for code related to devices. I would love to hear your opinions and comments about this effort! (and please note that this is very long term work) [1] http://www.ovirt.org/Feature/VmDevices_rework Hi Martin, It's a great and just initiative. Renaming is good, but I'd like to hear more about your final goal - besides the hope that it the code is shorter. In my opion you should state your high-level goal explicitly: device_object should be self-sufficient regarding: initiation, state change, stat extraction and stat reporting. Comments: 1. We want to drop the legacy conf support. In https://gerrit.ovirt.org/40104 we have removed support of engine3.3 so production environment does not need it any more. There might be an issue with command-line test scripts, though that might be abusing the legacy args. At the very least it should be marked as deprecated. 2. dev_map is an unfortunate historical mistake. The fact that it is a map of device type to device_object lists helps no one. It should be hidden behind one or two lookup functions. Each object class should be able to iterate its instances, and add filtering on top of that. 3. A definition like Type of the device is dev_type does not help for someone who knows what is dev_type. Please give examples. The formal naming system could use some wiki styling. 4. I did not understand Libvirt XML parsing and processing is dumb - missing orchestration object to delegate XML chunks to devices themself 5. It seems that you have further phases planned, but not written. Could you provide at least the title, or charter of each phase? 1.1 is about renaming. 1.2 is about legacy. Do you have phases 2, 3..? Death (or at least diet) to vm.py! Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] ipwrapperTests.TestDrvinfo failing on f21 slave, pass on f20 slave
On Wed, May 06, 2015 at 05:28:14PM -0400, Nir Soffer wrote: Hi infra, These tests fail on Fedora 21 slave, and pass on Fedora 20 slave. http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/1888/console - FAIL http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/1908/console - PASS Infra: please check the Fedora 21 slave, maybe this is configuration issue? Ido: please check if the tests are valid Generally these tests are not unitests, and should not run for validating unrelated patches. They should move to the functional tests. Could you explain your assertions? This tests do not require a functioning vdsm. They don't test vdsm's API, so they should not end up in the functional directory. The ipwrapperTests tests the ipwrapper unit of vdsm. But in order to do so, they interact with the host system, and attempt to create a bridge. When this setup fails for some reason (I do not know why `brctl addbr vdsmtest-O8JmI` failed) the test ends up in an ERROR state. If we fail to fix this condition ASAP, we catch this failure and skip the test as broken. But I don't think we should thrown it out of the test suite altogether. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
