Re: [Engine-devel] Fwd: [vdsm] ovirt-guest-agent on debian
On 03/11/2014 09:23 AM, Itamar Heim wrote: On 03/11/2014 10:09 AM, Sven Kieske wrote: Forwarding to engine devel, because it seems there is no interest in this feature on vdsm-devel. There is definitely an interest, I just haven't seen this email yesterday but this morning. Sorry for that. Could this get integrated somehow? I managed to reuse the ubuntu build of ovirt-guest-agent for debian. vinznenz? Original-Nachricht Betreff: [vdsm] ovirt-guest-agent on debian Datum: Mon, 10 Mar 2014 09:36:11 + Von: Sven Kieske An: VDSM Project Development Hi, I just did some additional research, here are my results: on a clean debian 7 x64 VM inside oVirt: I did as root: apt-get install python-software-properties add-apt-repository -y ppa:zhshzhou/vdsm-ubuntu vim /etc/apt/sources.list.d/zhshzhou-vdsm-ubuntu-wheezy.list change the following lines from: deb http://ppa.launchpad.net/zhshzhou/vdsm-ubuntu/ubuntu wheezy main deb-src http://ppa.launchpad.net/zhshzhou/vdsm-ubuntu/ubuntu wheezy main to: deb http://ppa.launchpad.net/zhshzhou/vdsm-ubuntu/ubuntu precise main deb-src http://ppa.launchpad.net/zhshzhou/vdsm-ubuntu/ubuntu precise main then: apt-get update apt-get install ovirt-guest-agent You get an error: Setting up ovirt-guest-agent (1.0.8.201309301944.gitb7f8f2-1ppa1) ... chown: cannot access `/var/log/ovirt-guest-agent/ovirt-guest-agent.log': No such file or directory but it works: ls -lashi /var/log/ovirt-guest-agent/ovirt-guest-agent.log 533212 4.0K -rw-r--r-- 1 ovirtagent ovirtagent 208 Mar 10 10:20 /var/log/ovirt-guest-agent/ovirt-guest-agent.log service ovirt-guest-agent status [ ok ] ovirt-guest-agent is running. IP information shows up in ovirt. it would be cool if the folder "precise" on the ppa could be cloned to be a folder "wheezy". So you would not need to change the apt-sources entries. I don't know why this chown error occurs, but it seems to work! -- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Install rhevm-guest-agent on SLES guest
On 03/02/2014 10:46 PM, Itamar Heim wrote: On 02/21/2014 03:43 PM, Alexander Wels wrote: Doesn't look like the package exists yet, there is an open bug for it: https://bugzilla.redhat.com/show_bug.cgi?id=1043471 On Friday, February 21, 2014 11:00:40 AM Vikas Kokare wrote: We want to install the rhevm-guest-agent package on a virtual machine with Suse Linux as the guest OS. The RHEVM 3.2 documentation provides install instructions for RHEL and Windows guests but not for Suse Linux. This agent is important to monitor virtual resources on guest machines, hence the need. How can this be done? ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel is the suse one relevant for this? http://www.ovirt.org/Feature/GuestAgentOpenSUSE Only if it is openSUSE, SLES won't currently work though. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel -- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] [ovirt-testday 2] Fake PPC Support - Feedback
616f-ce98-44e2-90e5-6956a457d8a0', 'specParams': {}, 'readonly':
'false', 'domainID': '5b2e0ba6-4457-4929-b925-f1c0d8d37b75', 'optional':
'false', 'deviceId': '61b8616f-ce98-44e2-90e5-6956a457d8a0', 'poolID':
'0002-0002-0002-0002-0002', 'device': 'disk', 'shared':
'false', 'propagateErrors': 'off', 'type': 'disk'}, {'device': 'scsi',
'index': '0', 'type': 'controller', 'address': {'type': 'spapr-vio'}},
{'nicModel': 'pv', 'macAddr': '00:1a:4a:fd:38:d2', 'linkActive': 'true',
'network': 'ovirtmgmt', 'filter': 'vdsm-no-mac-spoofing', 'specParams':
{}, 'deviceId': '814af769-c14c-42df-84b6-075930067cfb', 'device':
'bridge', 'type': 'interface'}, {'device': 'memballoon', 'specParams':
{'model': 'virtio'}, 'type': 'balloon', 'deviceId':
'2f41cde8-4660-4e2a-a364-79535be662cf'}, {'index': '1', 'specParams':
{}, 'deviceId': '2e48b9cb-d3f6-470f-b47e-f68850d34303', 'device':
'scsi', 'model': 'virtio-scsi', 'type': 'controller'}], 'smp': '1',
'vmType': 'kvm', 'memSize': 1024, 'displayIp': '0',
'spiceSecureChannels':
'smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard',
'smpCoresPerSocket': '1', 'vmName': 'test', 'display': 'vnc', 'nice': '0'}}
Thread-1110::DEBUG::2014-02-11 15:31:28,321::vm::3119::vm.Vm::(_run)
vmId=`e8aa218a-7d9c-46a8-885d-2ea33c3432c2`::encoding="utf-8"?>xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0";>
test
e8aa218a-7d9c-46a8-885d-2ea33c3432c2
262144
262144
1
1048576
path="/var/lib/libvirt/qemu/channels/e8aa218a-7d9c-46a8-885d-2ea33c3432c2.com.redhat.rhevm.vdsm"/>
path="/var/lib/libvirt/qemu/channels/e8aa218a-7d9c-46a8-885d-2ea33c3432c2.org.qemu.guest_agent.0"/>
passwd="*" port="-1" type="vnc"/>
/usr/bin/qemu-system-ppc64
unit="0"/>
file="/rhev/data-center/mnt/str-02.rhev.lab.eng.brq.redhat.com:_mnt_export_nfs_150_nfs03/5b2e0ba6-4457-4929-b925-f1c0d8d37b75/images/61b8616f-ce98-44e2-90e5-6956a457d8a0/06abed34-5464-46d4-9527-3c4e38132587"/>
61b8616f-ce98-44e2-90e5-6956a457d8a0
name="qemu" type="raw"/>
hvm
--
Regards,
Vinzenz Feenstra | Senior Software Engineer
RedHat Engineering Virtualization R & D
Phone: +420 532 294 625
IRC: vfeenstr or evilissimo
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] [ovirt-testday] Fake PPC Support - Feedback
Hi,
On the test day I was testing the Fake PPC support. I managed to
register the Fake PPC host into the engine. However when I tried to run
the VM it failed with libvirt responding:
*
libvirtError: internal error: process exited while connecting to
monitor: "kvm" accelerator does not exist.**
**No accelerator found!**
**KVM not supported for this target**
***
Which does not make much sense though. Since it should not try to use
'kvm'. And I am not really sure why it does.
Please let me know if there's anything else I would need to provide you
so you can figure out what is going on. Right now the server is still in
the same state. But I eventually will have to use it for something else.
Here's the part of the log which prints the generated DomainXML:
Thread-77::DEBUG::2014-01-23 15:43:46,834::vm::3130::vm.Vm::(_run)
vmId=`a5409fe7-8054-46ce-91c2-2d565e7edc5b`::encoding="utf-8"?>
http://libvirt.org/schemas/domain/qemu/1.0";>
asdfasd
a5409fe7-8054-46ce-91c2-2d565e7edc5b
524288
524288
1
524288
type="virtio"/>
path="/var/lib/libvirt/qemu/channels/a5409fe7-8054-46ce-91c2-2d565e7edc5b.com.redhat.rhevm.vdsm"/>
type="virtio"/>
path="/var/lib/libvirt/qemu/channels/a5409fe7-8054-46ce-91c2-2d565e7edc5b.org.qemu.guest_agent.0"/>
passwd="*" passwdValidTo="1970-01-01T00:00:01" port="-1" type="vnc"/>
/usr/bin/qemu-system-ppc64
type="drive" unit="0"/>
file="/rhev/data-center/mnt/str-02.rhev.lab.eng.brq.redhat.com:_mnt_export_nfs_150_nfs04/efd2a9ce-6417-44e9-ac59-bb577f22bbc3/images/4f6be340-f7b1-4b78-9aa3-7390ffd6bec8/e4f6d99b-c94c-4ae9-8bd6-5212a1d4e9df"/>
4f6be340-f7b1-4b78-9aa3-7390ffd6bec8
io="threads" name="qemu" type="raw"/>
hvm
Thread-77::DEBUG::2014-01-23
15:43:47,566::libvirtconnection::124::root::(wrapper) Unknown
libvirterror: ecode: 1 edom: 10 level: 2 message: internal error:
process exited while connecting to monitor: "kvm" accelerator does not
exist.
No accelerator found!
KVM not supported for this target
Thread-77::DEBUG::2014-01-23
15:43:47,567::vm::2247::vm.Vm::(_startUnderlyingVm)
vmId=`a5409fe7-8054-46ce-91c2-2d565e7edc5b`::_ongoingCreations released
Thread-77::ERROR::2014-01-23
15:43:47,567::vm::2273::vm.Vm::(_startUnderlyingVm)
vmId=`a5409fe7-8054-46ce-91c2-2d565e7edc5b`::The vm start process failed
Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 2233, in _startUnderlyingVm
self._run()
File "/usr/share/vdsm/vm.py", line 3164, in _run
self._connection.createXML(domxml, flags),
File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.py",
line 92, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2920, in
createXML
if ret is None:raise libvirtError('virDomainCreateXML() failed',
conn=self)
libvirtError: internal error: process exited while connecting to
monitor: "kvm" accelerator does not exist.
No accelerator found!
KVM not supported for this target
Extraction of the capabilities related to this:
hvm
32
/usr/bin/qemu-system-ppc
g3beige
prep
mpc8544ds
mac99
ppce500
virtex-ml507
bamboo
taihu
ref405ep
none
hvm
64
/usr/bin/qemu-system-ppc64
mac99
prep
mpc8544ds
g3beige
ppce500
virtex-ml507
pseries
bamboo
taihu
ref405ep
none
--
Regards,
Vinzenz Feenstra | Senior Software Engineer
RedHat Engineering Virtualization R & D
Phone: +420 532 294 625
IRC: vfeenstr or evilissimo
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Gerrit NEW Change Screen
On 01/16/2014 10:52 AM, Yedidyah Bar David wrote: - Original Message - From: "Vinzenz Feenstra" To: "Itamar Heim" , vdsm-de...@lists.fedorahosted.org, "engine-devel" Sent: Thursday, January 16, 2014 11:40:01 AM Subject: Re: [Engine-devel] Gerrit NEW Change Screen On 01/16/2014 10:38 AM, Vinzenz Feenstra wrote: On 01/15/2014 11:08 PM, Itamar Heim wrote: with gerrit 2.8, there is a new change screen. its not enabled by default (yet), please use and see what you think. to enable, go to settings (click the top-right arrow next to your name, and choose settings). select preferences and set "Change View:" to "New Screen". Well I had this enabled for some time today but I had to switch back, since I don't see how I am able to apply code review +/- or anything like this. Correction, I just found it. It's hidden under the "Reply..." button And if you hover on it you see that you can also press 'a'. In 2.6 'r' did this, not sure why they had to change that. And if you press 'r' inside a diff you get to the old review/reply page... On the other hand, it is way faster on the diff view even for vdsm's vm.py which has a whopping 5k lines. I like that :-) This new change screen is something one really needs to get used to. Indeed. One thing I like in it is that in "History" it shows also all the inline comments. And one thing I miss is a solution to [1] - am I the only one bugged by it? No, I am also not a big fan about it, but this is not a regression to the previous version, thanks for the link :-) [1] http://code.google.com/p/gerrit/issues/detail?id=217 -- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] stripping ' from log messages
On 01/15/2014 10:33 AM, Jiri Moskovcak wrote: Hi, while working on a patch for engine I ran into a code which removes ' from log messages. It's in Log.java: 165 method transform. Seems like it's there since the conversion to java happened. This code cripples the log messages i.e can't -> cant, it's -> its ..etc. Is there some reason why it's there? Or could it be removed? Even worse, if you are trying to log some variables and put them in single quotes to see that the variable was empty (e.g. for Strings) you won't be seeing anything. Thank you, Jirka ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel -- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Gerrit NEW Change Screen
On 01/16/2014 10:38 AM, Vinzenz Feenstra wrote: On 01/15/2014 11:08 PM, Itamar Heim wrote: with gerrit 2.8, there is a new change screen. its not enabled by default (yet), please use and see what you think. to enable, go to settings (click the top-right arrow next to your name, and choose settings). select preferences and set "Change View:" to "New Screen". Well I had this enabled for some time today but I had to switch back, since I don't see how I am able to apply code review +/- or anything like this. Correction, I just found it. It's hidden under the "Reply..." button On the other hand, it is way faster on the diff view even for vdsm's vm.py which has a whopping 5k lines. I like that :-) This new change screen is something one really needs to get used to. Thanks, Itamar ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel -- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Gerrit NEW Change Screen
On 01/15/2014 11:08 PM, Itamar Heim wrote: with gerrit 2.8, there is a new change screen. its not enabled by default (yet), please use and see what you think. to enable, go to settings (click the top-right arrow next to your name, and choose settings). select preferences and set "Change View:" to "New Screen". Well I had this enabled for some time today but I had to switch back, since I don't see how I am able to apply code review +/- or anything like this. On the other hand, it is way faster on the diff view even for vdsm's vm.py which has a whopping 5k lines. I like that :-) This new change screen is something one really needs to get used to. Thanks, Itamar ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel -- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] [vdsm] Proposal VDSM <=> Engine Data Statistics Retrieval Optimization
anged. if hashes.info changed => request getConfInfo with all vmIDs on that host where the hash changed if hashes.status changed => request getStatus with all vmIDs on that host where the hash changed if hashes.guestDetails changed => request getGuestDetails with all vmIDs on that host where the hash changed Request getAllDeviceStats periodically e.g. every 5 minutes, which should be sufficient for the DWH, in case it is not it could be even configurable. On 03/17/2013 04:30 PM, Dan Kenigsberg wrote: On Sun, Mar 17, 2013 at 10:28:15AM -0400, Ayal Baron wrote: - Original Message - On 17/03/13 15:13, Ayal Baron wrote: - Original Message - On 03/13/2013 11:55 PM, Ayal Baron wrote: ... The only reason we have this problem is because there is this thing against making multiple calls. Just split it up. getVmRuntimeStats() - transient things like mem and cpu% getVmInformation() - (semi)static things like disk\networking layout etc. Each updated at different intervals. +1 on splitting the data up into 2 separate API calls. You could potentially add a checksum (md5, or any other way) of the "static" data to getVmRuntimeStats and not bother even with polling the VmInformation if this hasn't changed. Then you could poll as often as you'd like the stats and immediately see if you also need to retrieve VmInfo or not (you rarely would). +1 To Ayal's suggestion except that instead of the engine hashing the data VDSM sends the key which is opaque to the engine. This can be a local timestap or a generation number. Of course vdsm does the hash, otherwise you'd need to pass all the data to engine which would beat the purpose. I thought you meant engine will be sending the hash of previous requests per VM to vdsm, then vdsm will reply back with vm's removed, vm's added, and the details for vm's that changed (i.e., engine would be doing something like if-modified-since-checksum per vm). benefit is reducing a round trip. but first would need to split to calls of stats (always changing) and slowly/never changing data. If vdms accepts the hash then in your method engine would have to periodically call getVmInfo(hash). What I was suggesting is that getVmStats would return vmInfo hash so that we could avoid calling getVmInfo altogether. The stats *always* change so there is no need for checking if that info has changed. What we could do is avoid the split into 2 verbs by calling getVmStats(hash) and then have getVmStats return everything if the hash has changed or only the stats if it hasn't. This would be the least number of roundtrips and avoid the split. If you don't pass a hash it would return everything so this way it's also fully backward compatible. For the 'static' data, why is there a need for a hash? If VDSM sends in each update a timestamp, can't RHEVM just use if-modified-since with the last timestamp it got from VDSM? Is it cheaper for VDSM to calculate the hash, than update the timestamp per change in any of the fields? It doesn't really need to update the timestamp per change, only for the first change since last update sent actually (so 'dirty' flag in a way, to signify data that RHEVM hasn't seen yet). Y. As Saggi mentioned: "VDSM sends the key which is opaque to the engine. This can be a local timestap or a generation number." The content doesn't matter, what matters is that it has changed. timestamp assumes that vdsm will track changes and send only delta. Although possible this would be an overkill (for every value in the dict you'd have to hold a timestamp of last change and send only those which have changed since the timestamp which was passed by the user). If we're in the spirit of quoting Saggi, this suggestion is not compatible with "...mak[ing] the return value differ according to input ... is a big no no when talking about type safe APIs.". Dan. _______ vdsm-devel mailing list vdsm-de...@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel -- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] [vdsm] Proposal VDSM <=> Engine Data Statistics Retrieval Optimization
On 04/22/2013 10:39 PM, Ayal Baron wrote:
- Original Message -
I have left this for a while without continuing because I had to focus
on other things. However this is still in progress :-)
Are you writing patches? (if so, what solution are you pursuing)
I haven't started writing patches for this particular thing yet, however
I want to, but before I want to know that the approach will be
acceptable and so I won't have to throw away work because it's not
accepted by the project that way.
My ultimate goal is to reduce the overall traffic and the amount of data
the engine has to parse on a per Host basis.
This can be achieved without having push notifications working already.
Imagine that oVirt engine currently has to parse for 1000 VMs running,
~16MiB of XML data and that every 15 seconds. And that's just the result
from getAllVmStats.
Additionally it's polling every 2-3 seconds 'list' to retrieve the VmIds
and statuses.
Now I have split the values which the engine would return in getVmStats
into groups based on how likely the are to change AND their size.
getAllVmRuntimeStats() for example, returns the hashes to the different
parts where it makes sense, however it won't give you the devices for
example. Because the engine still has to poll that every 5 minutes or so
to retrieve stats for the the data warehouse.
I am personally for having push notifications properly implemented and
working however the base of this idea is reducing parser and validation
work and will help the engine backend to scale better. Especially it
should theoretically reduce the load on the database as well, at least
theoretically (I have no numbers for this one ;-))
On 03/13/2013 10:55 PM, Ayal Baron wrote:
- Original Message -
- Original Message -
From: "Ayal Baron"
To: "Saggi Mizrahi"
Cc: engine-devel@ovirt.org, vdsm-de...@lists.fedorahosted.org,
"Vinzenz Feenstra"
Sent: Wednesday, March 13, 2013 5:39:24 PM
Subject: Re: [vdsm] [Engine-devel] Proposal VDSM <=> Engine Data
Statistics RetrievalOptimization
- Original Message -
I am completely against this.
It make the return value differ according to input which
is a big no no when talking about type safe APIs.
The only reason we have this problem is because there is this
thing against making multiple calls.
Which is totally contra productive because multiple calls, if properly
split up, will actually lead to less data sent for frequent needed data
calls. And the others shall be triggered when necessary.
Just split it up.
getVmRuntimeStats() - transient things like mem and cpu%
getVmInformation() - (semi)static things like disk\networking
layout
etc.
Each updated at different intervals.
+1 on splitting the data up into 2 separate API calls.
You could potentially add a checksum (md5, or any other way) of the
"static" data to getVmRuntimeStats and not bother even with polling
the VmInformation if this hasn't changed. Then you could poll as
often as you'd like the stats and immediately see if you also need
to retrieve VmInfo or not (you rarely would).
+1 To Ayal's suggestion
except that instead of the engine hashing the data VDSM sends the
key which is opaque to the engine.
This can be a local timestap or a generation number.
Of course vdsm does the hash, otherwise you'd need to pass all the data to
engine which would beat the purpose.
We need the hash if we can't have dynamic content. Generation numbers
aren't really helpful as every call aggregates the statistics data
newly, at the moment at least.
But, we might want to consider that when we add events polling
becomes (much) less frequent so maybe it'll be an overkill.
You'd still need to compare versions of the data in vdsm and send only if
it changed. If you don't persist what was received last then potentially
you could have a monday morning effect where upon on system startup you'd
be sending everything. So I still think you'd want to have the hash.
We do a hash already on the XML and include it in the getStats response.
Hashes should show enough difference.
Now to the non-dynamic responses and 'type-safe' API: If we would go for
non dynamic responses we would need for sure 5 new API calls to achieve
some gain on the amount of data sent.
*getAllVmRuntimeStats() "returns a map of vmId/data pairs for all vms"*
# All the time changing data which is needed by the oVirt Engine, or so
often changing that it does not make sense
# to place it anywhere else
{
VmId: {
cpuSys --> Could be potentially summarized
cpuUser -/
memUsage
elapsedTime,
status
statsAge
hashes = {
conf,# Hased information of the XML
(This one is called "hash" in
Re: [Engine-devel] [vdsm] Proposal VDSM <=> Engine Data Statistics Retrieval Optimization
I have left this for a while without continuing because I had to focus
on other things. However this is still in progress :-)
On 03/13/2013 10:55 PM, Ayal Baron wrote:
- Original Message -
- Original Message -
From: "Ayal Baron"
To: "Saggi Mizrahi"
Cc: engine-devel@ovirt.org, vdsm-de...@lists.fedorahosted.org,
"Vinzenz Feenstra"
Sent: Wednesday, March 13, 2013 5:39:24 PM
Subject: Re: [vdsm] [Engine-devel] Proposal VDSM <=> Engine Data
Statistics RetrievalOptimization
- Original Message -
I am completely against this.
It make the return value differ according to input which
is a big no no when talking about type safe APIs.
The only reason we have this problem is because there is this
thing against making multiple calls.
Which is totally contra productive because multiple calls, if properly
split up, will actually lead to less data sent for frequent needed data
calls. And the others shall be triggered when necessary.
Just split it up.
getVmRuntimeStats() - transient things like mem and cpu%
getVmInformation() - (semi)static things like disk\networking
layout
etc.
Each updated at different intervals.
+1 on splitting the data up into 2 separate API calls.
You could potentially add a checksum (md5, or any other way) of the
"static" data to getVmRuntimeStats and not bother even with polling
the VmInformation if this hasn't changed. Then you could poll as
often as you'd like the stats and immediately see if you also need
to retrieve VmInfo or not (you rarely would).
+1 To Ayal's suggestion
except that instead of the engine hashing the data VDSM sends the
key which is opaque to the engine.
This can be a local timestap or a generation number.
Of course vdsm does the hash, otherwise you'd need to pass all the data to
engine which would beat the purpose.
We need the hash if we can't have dynamic content. Generation numbers
aren't really helpful as every call aggregates the statistics data
newly, at the moment at least.
But, we might want to consider that when we add events polling
becomes (much) less frequent so maybe it'll be an overkill.
You'd still need to compare versions of the data in vdsm and send only if it
changed. If you don't persist what was received last then potentially you
could have a monday morning effect where upon on system startup you'd be
sending everything. So I still think you'd want to have the hash.
We do a hash already on the XML and include it in the getStats response.
Hashes should show enough difference.
Now to the non-dynamic responses and 'type-safe' API: If we would go for
non dynamic responses we would need for sure 5 new API calls to achieve
some gain on the amount of data sent.
*getAllVmRuntimeStats() "returns a map of vmId/data pairs for all vms"*
# All the time changing data which is needed by the oVirt Engine, or so
often changing that it does not make sense
# to place it anywhere else
{
VmId: {
cpuSys --> Could be potentially summarized
cpuUser -/
memUsage
elapsedTime,
status
statsAge
hashes = {
conf,# Hased information of the XML
(This one is called "hash" in getStats())
info, # Hashed information of semi
static items
statusHash: # Hashed information of items with are
likely to change however not that often
guestDetails:# Hashed value of the guest details
(applicationList, network information)
}
}
**getVmStatuses([vmId1, vmId2, ...])*"Returns a vmId/data pair for
each vm requested"**
*# This data does not change that often and can be retrieved on demand
once the hash changes
return {
vmId: {
timeOffset,
monitorResponse
clientIp,
lastLogin,
username,
session,
guestIPs,
}
}
*getAllVmDeviceStatistics():**"Returns a vmId/data pair for all vms"*
# This data has to be requested all the time however in lower
intervals (e.g. every 5 minutes)
# And is usually needed for all the VMs anyway
return {
vmId: {
network,
disksUsage, # Might be improved by summarizing?
disks,
balloonInfo,
memoryStats
}
}
*getVmInfo([vmId1, vmId2, ...]) "Returns a vmId/data pair for each vm
requested"
* # Basically this should be almost constant, except if there have
been changes like migrations, pausing, errors etc
return {
vmId: {
acpiEnable,
vmType,
guestName,
guestOS,
kvmEnable,
pauseCode,
displayIp,
displayPort,
displaySecurePort,
Re: [Engine-devel] [vdsm] Proposal VDSM <=> Engine Data Statistics Retrieval Optimization
On 03/08/2013 03:30 AM, Mark Wu wrote:
On 03/08/2013 06:11 AM, Dan Kenigsberg wrote:
On Thu, Mar 07, 2013 at 12:25:54PM +0100, Vinzenz Feenstra wrote:
Please find the prettier version on the wiki:
http://www.ovirt.org/Proposal_VDSM_-_Engine_Data_Statistics_Retrieval
Proposal VDSM - Engine Data Statistics Retrieval
VDSM <=> Engine data retrieval optimization
Motivation:
Currently the RHEVM engine is polling the a lot of data from VDSM
every 15 seconds. This should be optimized and the amount of data
requested should be more specific.
It feels like a good idea, but do you have numbers? How much traffic
would be saved? Remember the added computation incurred on each host -
there's always a price to pay.
Well the data of a single really basic simple VM has about 4 KiB data in
the output of vdsClient, the XMLRPC formatted body part has almost 16KiB.
The thing is that this data is queried every 15 seconds (previously 10)
with little value for having ALL data sent all the time, the engine is
not even using all of the data all the time.
This optimization must be seen on a bigger scale, if you have a
datacenter with let's say 1000 VMs then the data needed to be
transmitted and parsed by the engine every 15 seconds is about 16MiB.
This optimization wouldn't pay off that much in a 2 server 20 VM
datacenter however on a larger scale it has quite a big impact.
For each VM the data currently contains much more information than
actually needed which blows up the size of the XML content quite
big. We could optimize this by splitting the reply on the getVmStats
based on the request of the engine into sections. For this reason
Omer Frenkel and me have split up the data into parts based on their
usage.
This data can and usually does change during the lifetime of the VM.
Rarely Changed:
This data is change not very frequent and it should be enough to
update this only once in a while. Most commonly this data changes
after changes made in the UI or after a migration of the VM to
another Host.
*Status* = Running
Status does not change much, but when it does, it is important to report
that quickly.
This is done by the list command which is executed every 2 seconds
(maybe 3?)
For this kind of data, it is suitable to use an event report, which
should be available in the jsonrpc API.
*acpiEnable* = true
*vmType* = kvm
*guestName* = W864GUESTAGENTT
*displayType* = qxl
*guestOs* = Win 8
*kvmEnable* = true #/*this should be constant and never changed*/
*pauseCode* = NOERR
*monitorResponse* = 0
*session* = Locked # unused
*netIfaces* = [{'name': 'Realtek RTL8139C+ Fast Ethernet NIC',
'inet6': ['fe80::490c:92bb:bbcc:9f87'], 'inet': ['10.34.60.148'],
'hw': '00:1a:4a:22:3c:db'}]
*appsList* = ['RHEV-Tools 3.2.4', 'RHEV-Agent64 3.2.3',
'RHEV-Serial64 3.2.3', 'RHEV-Network64 3.2.2', 'RHEV-Network64
3.2.3', 'RHEV-Block64 3.2.3', 'RHEV-Balloon64 3.2.3',
'RHEV-Balloon64 3.2.2', 'RHEV-Agent64 3.2.2', 'RHEV-USB 3.2.3',
'RHEV-Block64 3.2.2', 'RHEV-Serial64 3.2.2']
*pid* = 11314
*guestIPs* = 10.34.60.148 # duplicated info
*displayIp* = 0
*displayPort* = 5902
*displaySecurePort* = 5903
*username* = user@W864GUESTAGENTT
*clientIp* =
*lastLogin* = 1361976900.67
Often Changed:
This data is changed quite often however it is not necessary to
update this data every 15 seconds. As this is cumulative data and
reflects the current status, and it does not need to be snapshotted
every 15 seconds to retrieve statistics. The data can be retrieved
in much more generous time slices. (e.g. Every 5 minutes)
*network* = {'vnet1': {'macAddr': '00:1a:4a:22:3c:db',
'rxDropped': '0', 'txDropped': '0', 'rxErrors': '0', 'txRate':
'0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown',
'speed': '100', 'name': 'vnet1'}}
*disksUsage* = [{'path': 'c:\\', 'total': '64055406592', 'fs':
'NTFS', 'used': '19223846912'}, {'path': 'd:\\', 'total':
'3490912256', 'fs': 'UDF', 'used': '3490912256'}]
*timeOffset* = 14422
*elapsedTime* = 68591
*hash* = 2335461227228498964
*statsAge* = 0.09 # unused
Often Changed but unused
This data does not seem to be used in the engine at all. It is *not*
even used in the data warehouse.
*memoryStats* = {'swap_out': '0',
[Engine-devel] Proposal VDSM <=> Engine Data Statistics Retrieval Optimization
Engine. This data could be requested every 15 seconds to keep things as they are now. *cpuSys* = 2.32 *cpuUser* = 1.34 *memUsage* = 30 Proposed Solution for VDSM & Engine: We will introduce new optional parameters to getVmStats, getAllVmStats and list to allow a finer grained specification of data which should be included. *Parameter:* *statsType*=/**/ (getVmStats, getAllVmStats only) *Allowed values:* * full (default to keep backwards compatibility) * app-list (Just send the application list) * rare (include everything from rarely changed to very frequent) * often (include everything from often changed to very frequent) * frequent (only send the very frequently changed items) *Parameter:* *clientId*=** The client id is specified by the client and should be unique however constantly used. *Parameter:* *diff*=** In combination with the clientId VDSM will send only differences to the previous request from the named clientId. (if diff=true) Additional Change: Besides the introduction of the new parameters for list, getVmStats and getAllVmStats it might make sense to include a hash for the appList into the rarely changed section of the response which would allow to identify changes and avoid having to sent the complete appList every so often and only if the hash known to the client is outdated. *Note:* The appList (Application List) reported by the guest agent could be fully implemented on request only, as long as the guest agent installed supports this. As there seems to be a request to have the complete list of installed applications on all guests this data could be quite extensive and a huge list. On the other hand this data is only rarely visible and therefore it should not be requested all the time and only on demand. Improvement of the Guest Agent: As part of the proposed solution it is necessary to improve the guest agent as well. For the full application list there should be implemented a caching system which will be fully reactive and should not poll the application list for example all the time. The guest can create a prepared data file containing all data in the JSON format (as used for the communication with VDSM via VIO) and just have to read that file from disk and directly sends it to VDSM. However it is quite possible that this list is to big and it might have to be chunked into pieces. (Multiple messages, which would have to be supported by VDSM then as well) The solution for this is to make VDSM request this data and it will retrieve the data necessary on request only. -- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Report vNic Implementation Details
On 11/22/2012 08:48 AM, Livnat Peer wrote: On 21/11/12 22:53, Andrew Cathrow wrote: - Original Message - From: "Moti Asayag" To: engine-devel@ovirt.org Sent: Wednesday, November 21, 2012 12:36:58 PM Subject: [Engine-devel] Report vNic Implementation Details Hi all, This is a proposal for reporting the vNic implementation details as reported by the guest agent per each vNic. Please review the wiki and add your comments. While we're making the change is there anything else we'd want to report - MTU, MAC (since a user might try to override), etc ? About the MAC address - the engine uses the MAC address to correlate between the guest-agent report and the VNICs defined in the engine. If the GA reports a MAC that the engine does not recognize the engine can not associate it with a VNIC. What the engine can do is to issue a warning to the audit log in case such a mismatch is recognized, Is that good enough? About MTU - It is not being reported by the guest agent ATM (while IPv4 IPv6 and MAC are), If we'd like to have it I can open RFE for the GA to report additional fields. Well if there are changes needed for the GA, please let me know soon, if it should make it into 3.2. Thanks. Livnat http://wiki.ovirt.org/wiki/Feature/ReportingVnicImplementation Thanks, Moti ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel -- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
