Re: [libvirt] Yet another RFC for CAT

2017-09-04 Thread Daniel P. Berrange 
On Mon, Sep 04, 2017 at 04:14:00PM +0200, Martin Kletzander wrote:
> * The current design (finally something libvirt-related, right?)
> 
> The discussion ended with a conclusion of the following (with my best
> knowledge, there were so many discussions about so many things that I
> would spend too much time looking up all of them):
> 
> - Users should not need to specify bit masks, such complexity should be
>   abstracted.  We'll use sizes (e.g. 4MB)
> 
> - Multiple vCPUs might need to share the same allocation.
> 
> - Exclusivity of allocations is to be assumed, that is only unoccupied
>   cache should be used for new allocations.
> 
> The last point seems trivial but it's actually very specific condition
> that, if removed, can cause several problems.  If it's hard to grasp the
> last point together with the second one, you're on the right track.  If
> not, then I'll try to make a point for why the last point should be
> removed in 3... 2... 1...
> 
> * Design flaws
> 
> 1) Users cannot specify any allocation that would share only part with
>some other allocation of the domain or the default group.
> 
> 2) It was not specified what to do with the default resource group.
>There might be several ways to approach this, with varying pros and
>cons:
> 
> a) Treat it as any other group.  That is any bit set for this group
>will be excluded from usable bits when creating new allocation
>for a domain.
> 
> - Very predictable behaviour
> 
> - You will not be able to allocate any amount of cache without
>   previous setting for the default group as that will have all
>   the bits set which will make all the cache unusable
> 
> b) Automatically remove the appropriate amount of bits that are
>needed for new domains.
> 
> - No need to do any change to the system settings in order to
>   use this new feature
> 
> - We would have to change system settings, which is generally
>   frowned upon when done "automatically" as a side effect of
>   starting a domain, especially for such scarce resource as
>   cache
> 
> - The change to system settings would not be entirely
>   predictable
> 
> c) Act like it doesn't exist and don't remove its allocations from
>consideration
> 
> - Doesn't really make sense as system processes might be
>   trashing the cache as any VM, moreover when all VM processes
>   without allocations will be based in the default group as
>   well
> 
> 3) There is no way for users to know what the particular settings are
>for any running domain.
> 
> The first point was deemed a corner case.  Fair enough on its own, but
> considering point 2 and its solutions, it is rather difficult for me to
> justify it.  Also, let's say you have domain with 4 vCPUs out of which
> you know 1 might be trashing the cache, but you don't want to restrict
> it completely, but others will utilize it very nicely.  Sensible
> allocations for such domain's vCPUs might be:
> 
>  vCPU  0:   000f
>  vCPUs 1-3: 
> 
> as you want vCPUs 1-3 to utilize even the part of cache that might get
> trashed by vCPU 0.  Or they might share some data (especially
> guest-memory-related).
> 
> The case above is not possible to set up with only per-vcpu(s) scalar
> setting.  And there are more as you might imagine now.  For example how
> do we behave with iothreads and emulator threads?

Ok, I see what you're getting at.  I've actually forgotten what
our current design looks like though :-)

What level of granularity were we allowing within a guest ?
All vCPUs use separate cache regions from each other, or all
vCPUs use a share cached region, but separate from other guests,
or a mix ?

> * My suggestion:
> 
> - Provide an API for querying and changing the allocation of the
>   default resource group.  This would be similar to setting and
>   querying hugepage allocations (see virsh's freepages/allocpages
>   commands).

Reasonable

> - Let users specify the starting position in addition to the size, i.e.
>   not only specifying "size", but also "from".  If "from" is not
>   specified, the whole allocation must be exclusive.  If "from" is
>   specified it will be set without checking for collisions.  The latter
>   needs them to query the system or know what settings are applied
>   (this should be the case all the time), but is better then adding
>   non-specific and/or meaningless exclusivity settings (how do you
>   specify part-exclusivity of the cache as in the example above)

I'm concerned about the idea of not checking 'from' for collisions,
if there's allowed a mix of guests with & within 'from'.

eg consider

 * Initially 24 MB of cache is free, starting at 8MB
 * run guest A   from=8M, size=8M
 * run guest B   size=8M
 => libvirt sets from=16M, so doesn't clash with A
 * stop guest A
 * run guest C   size=8M
 => libvirt sets from=8M, so

[libvirt] Yet another RFC for CAT

2017-09-04 Thread Martin Kletzander 

Hello everyone.

Last couple of weeks [1] I was working on CAT for libvirt.  Only
clean-ups and minor things were pushed into upstream, but as I'm getting
closer and closer to actual functionality I'm seeing a problem with our
current (already discussed and approved) design.  And I would like to
know your thoughts about this, even if you are not familiar with CAT,
feel free to keep the questions coming.

* Little bit of background about CAT

[I wanted to say "Long story short...", but after reading the mail in
its entirety before sending it I see it would end up like in "I Should
Have Never Gone Ziplining", so I'll rather let you brace for quite
elongated or, dare I say, endless stream of words]

Since the interface for CAT in the Linux kernel is quite hairy (together
with cache information reporting, don't even get me started on that) and
might feel pretty inconsistent if you are used to any Linux kernel
interface, I would like to summarize how is it used [2].  Feel free to
skip this part if you are familiar with it.

You can tune how much cache which processes can utilize.  Let's talk
only about L3 for now, also let's assume only unified caches (no
code/data prioritization).  For simplicity.

The cache is split into parts and when describing the allocation we use
hexadecimal representation of bit masks where each bit is the smallest
addressable (or rather allocable) part of the cache.  Let's say you have
16MB L3 cache which the CPU is able to allocate by chunks of 1MB, so the
allocation is represented by 32 bits => 8 hexadecimal characters.  Yes,
there can be minimum of continuous bits that need to be specified, you
can have multiple L3 caches, etc., but that's yet another thing that's
not important to what I need to discuss.  The whole cache is then
referred to as "" in this particular case.  Again, for simplicity
sake, let's assume the above hardware is constant in future examples.

Now, when you want to work with the allocations, it behaves similarly
(not the same way, though) as cgroups.  The default group, which
contains all processes, is in /sys/fs/resctrl, and you can create
additional groups (directories under /sys/fs/resctrl).  These are flat,
not hierarchical, meaning they cannot have subdirectories.  Each
resource group represents a group of processes (PIDs are written in
"tasks" file) that share the same resource settings.  One of the
settings is the allocation of caches.  By default there are no
additional resource groups (subdirectories of /sys/fs/resctrl) and the
default one occupies all the cache.

(IIRC, all bit masks must have only consecutive bits, but I cannot find
this in the documentation; let's assume this as well, but feel free to
correct me)

* Example time (we're almost there)

Let's say you have the default group with this setting:

 L3:0=00ff

That is setting of allocation for L3 cache, both code and data, cache id
0 and the occupancy rate is 50% (lower 8MB of the only L3 cache in our
example to be precise).

If you now create additional resource group, let's say
"libvirt-qemu-3-alpine-vcpu3" (truly random name, right?) and set the
following allocation:

 L3:0=0ff0

That specifies it will be allowed to use also 8MB of the cache, but this
time from the middle.  Half of that will be shared between this group
and the default one, the rest is exclusive to this group only.

* The current design (finally something libvirt-related, right?)

The discussion ended with a conclusion of the following (with my best
knowledge, there were so many discussions about so many things that I
would spend too much time looking up all of them):

- Users should not need to specify bit masks, such complexity should be
  abstracted.  We'll use sizes (e.g. 4MB)

- Multiple vCPUs might need to share the same allocation.

- Exclusivity of allocations is to be assumed, that is only unoccupied
  cache should be used for new allocations.

The last point seems trivial but it's actually very specific condition
that, if removed, can cause several problems.  If it's hard to grasp the
last point together with the second one, you're on the right track.  If
not, then I'll try to make a point for why the last point should be
removed in 3... 2... 1...

* Design flaws

1) Users cannot specify any allocation that would share only part with
   some other allocation of the domain or the default group.

2) It was not specified what to do with the default resource group.
   There might be several ways to approach this, with varying pros and
   cons:

a) Treat it as any other group.  That is any bit set for this group
   will be excluded from usable bits when creating new allocation
   for a domain.

- Very predictable behaviour

- You will not be able to allocate any amount of cache without
  previous setting for the default group as that will have all
  the bits set which will make all the cache unusable

b) Automatically remove the appropriate amount of bits that are

Re: [libvirt] [PATCH] Makefile.nonreentrant: Rebuild against Fedora 26

2017-09-04 Thread Daniel P. Berrange 
On Mon, Sep 04, 2017 at 03:06:16PM +0200, Andrea Bolognani wrote:
> According to the comments in the file and the git history, the
> list of forbidden symbols was originally built against Fedora 9
> in 2009 (!) and pretty much never refreshed afterwards.
> 
> Signed-off-by: Andrea Bolognani 
> ---
> I didn't put too much thought into this, but the results looks
> sane enough as we are only *adding* to the list.
> 
>  Makefile.nonreentrant | 8 ++--
>  1 file changed, 6 insertions(+), 2 deletions(-)

Reviewed-by: Daniel P. Berrange 

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH python] Remove unused variables for event callbacks

2017-09-04 Thread Daniel P. Berrange 
Signed-off-by: Daniel P. Berrange 
---
 libvirt-override.c | 22 --
 1 file changed, 22 deletions(-)

diff --git a/libvirt-override.c b/libvirt-override.c
index c04ce2e..a3a0508 100644
--- a/libvirt-override.c
+++ b/libvirt-override.c
@@ -5182,19 +5182,11 @@ libvirt_virConnectDomainEventDeregister(PyObject *self 
ATTRIBUTE_UNUSED,
  * Event Impl
  ***/
 static PyObject *addHandleObj;
-static char *addHandleName;
 static PyObject *updateHandleObj;
-static char *updateHandleName;
 static PyObject *removeHandleObj;
-static char *removeHandleName;
 static PyObject *addTimeoutObj;
-static char *addTimeoutName;
 static PyObject *updateTimeoutObj;
-static char *updateTimeoutName;
 static PyObject *removeTimeoutObj;
-static char *removeTimeoutName;
-
-#define NAME(fn) ( fn ## Name ? fn ## Name : # fn )
 
 static int
 libvirt_virEventAddHandleFunc(int fd,
@@ -5442,12 +5434,6 @@ libvirt_virEventRegisterImpl(PyObject *self 
ATTRIBUTE_UNUSED,
 Py_XDECREF(addTimeoutObj);
 Py_XDECREF(updateTimeoutObj);
 Py_XDECREF(removeTimeoutObj);
-VIR_FREE(addHandleName);
-VIR_FREE(updateHandleName);
-VIR_FREE(removeHandleName);
-VIR_FREE(addTimeoutName);
-VIR_FREE(updateTimeoutName);
-VIR_FREE(removeTimeoutName);
 
 /* Parse and check arguments */
 if (!PyArg_ParseTuple(args, (char *) "OO:virEventRegisterImpl",
@@ -5462,14 +5448,6 @@ libvirt_virEventRegisterImpl(PyObject *self 
ATTRIBUTE_UNUSED,
 !PyCallable_Check(removeTimeoutObj))
 return NULL;
 
-/* Get argument string representations (for error reporting) */
-addHandleName = py_str(addHandleObj);
-updateHandleName = py_str(updateHandleObj);
-removeHandleName = py_str(removeHandleObj);
-addTimeoutName = py_str(addTimeoutObj);
-updateTimeoutName = py_str(updateTimeoutObj);
-removeTimeoutName = py_str(removeTimeoutObj);
-
 /* Inc refs since we're holding on to these objects until
  * the next call (if any) to this function.
  */
-- 
2.13.5

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH python] Report an error if registering an event loop twice

2017-09-04 Thread Daniel P. Berrange 
The C library will now ignore an attempt to register an event
loop twice. It is unable to report an error in this case though
due to the C API returning 'void'. To improve this we must
manually report an error at the python level.

Signed-off-by: Daniel P. Berrange 
---
 libvirt-override.c | 13 ++---
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/libvirt-override.c b/libvirt-override.c
index a3a0508..9eba4ed 100644
--- a/libvirt-override.c
+++ b/libvirt-override.c
@@ -5427,13 +5427,12 @@ static PyObject *
 libvirt_virEventRegisterImpl(PyObject *self ATTRIBUTE_UNUSED,
  PyObject *args)
 {
-/* Unref the previously-registered impl (if any) */
-Py_XDECREF(addHandleObj);
-Py_XDECREF(updateHandleObj);
-Py_XDECREF(removeHandleObj);
-Py_XDECREF(addTimeoutObj);
-Py_XDECREF(updateTimeoutObj);
-Py_XDECREF(removeTimeoutObj);
+if (addHandleObj || updateHandleObj || removeHandleObj ||
+addTimeoutObj || updateTimeoutObj || removeTimeoutObj) {
+PyErr_SetString(PyExc_RuntimeError,
+"Event loop is already registered");
+return NULL;
+}
 
 /* Parse and check arguments */
 if (!PyArg_ParseTuple(args, (char *) "OO:virEventRegisterImpl",
-- 
2.13.5

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH] Makefile.nonreentrant: Rebuild against Fedora 26

2017-09-04 Thread Andrea Bolognani 
According to the comments in the file and the git history, the
list of forbidden symbols was originally built against Fedora 9
in 2009 (!) and pretty much never refreshed afterwards.

Signed-off-by: Andrea Bolognani 
---
I didn't put too much thought into this, but the results looks
sane enough as we are only *adding* to the list.

 Makefile.nonreentrant | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/Makefile.nonreentrant b/Makefile.nonreentrant
index 5cc64c0d5..a83298322 100644
--- a/Makefile.nonreentrant
+++ b/Makefile.nonreentrant
@@ -15,9 +15,9 @@
 ## .
 
 #
-# Generated by running the following on Fedora 9:
+# Generated by running the following on Fedora 26:
 #
-#  nm -D --defined-only /lib/libc.so.6  \
+#  nm -D --defined-only /lib64/libc.so.6  \
 #  | grep '_r$' \
 #  | awk '{print $3}' \
 #  | grep -v __ \
@@ -43,6 +43,7 @@ NON_REENTRANT += ether_ntoa
 NON_REENTRANT += fcvt
 NON_REENTRANT += fgetgrent
 NON_REENTRANT += fgetpwent
+NON_REENTRANT += fgetsgent
 NON_REENTRANT += fgetspent
 NON_REENTRANT += getaliasbyname
 NON_REENTRANT += getaliasent
@@ -72,6 +73,8 @@ NON_REENTRANT += getrpcent
 NON_REENTRANT += getservbyname
 NON_REENTRANT += getservbyport
 NON_REENTRANT += getservent
+NON_REENTRANT += getsgent
+NON_REENTRANT += getsgnam
 NON_REENTRANT += getspent
 NON_REENTRANT += getspnam
 NON_REENTRANT += getutent
@@ -95,6 +98,7 @@ NON_REENTRANT += random
 NON_REENTRANT += rand
 NON_REENTRANT += seed48
 NON_REENTRANT += setstate
+NON_REENTRANT += sgetsgent
 NON_REENTRANT += sgetspent
 NON_REENTRANT += srand48
 NON_REENTRANT += srandom
-- 
2.13.5

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] docs: Fix typo deamon -> daemon

2017-09-04 Thread Martin Kletzander 

On Mon, Sep 04, 2017 at 02:04:10PM +0200, Andrea Bolognani wrote:

Suggested-by: Martin Kletzander 
Signed-off-by: Andrea Bolognani 
---
I would push this as trivial, but I feel like someone might not be
okay with touching the release notes for libvirt versions that are
already out. I think in this case it's perfectly justified, but if
you disagree this is your chance to point it out :)



Precedent-set-by: f546d09362a01473adf2a2c781faa6fdb27ec2e3
Reviewed-by: Martin Kletzander 


signature.asc
Description: Digital signature
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v2 0/2] dac: relabel spice rendernode

2017-09-04 Thread Andrea Bolognani 
On Sun, 2017-08-27 at 12:20 -0400, Cole Robinson wrote:
> This fixes the last issue preventing qemu:///system spice GL from working
> out of the box: chown'ing the rendernode path so we have permissions
> to open it.
> 
> We skip this if mount namespaces are disabled, so the chown'ing won't
> interfere with other rendernode users on the host.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1460804
> 
> v2:
> Add the MOUNT_NAMESPACE handling
> Drop DAC restore of rendernode
> 
> Cole Robinson (2):
>   security: add MANAGER_MOUNT_NAMESPACE flag
>   security: dac: relabel spice rendernode
> 
>  src/qemu/qemu_driver.c  |  2 ++
>  src/security/security_dac.c | 68 
> +
>  src/security/security_dac.h |  3 ++
>  src/security/security_manager.c |  4 ++-
>  src/security/security_manager.h |  1 +
>  5 files changed, 77 insertions(+), 1 deletion(-)

Looks reasonable and works as expected on my Fedora 26
installation, so for the entire series:

  Reviewed-by: Andrea Bolognani 

You should document this in the release notes, though :)

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH] Add libxslt as build requires for mingw RPMs

2017-09-04 Thread Daniel P. Berrange 
The libxslt package is needed since:

  commit 94d2d6429d686c5af95115d09c01f3c6bd5ea7c6
  Author: Daniel P. Berrange 
  Date:   Wed Jul 26 17:40:44 2017 +0100

docs: make xmllint & xsltproc compulsory

The native RPM had it already, but mingw build was missing it.

Signed-off-by: Daniel P. Berrange 
---

Pushed as a build breaker & trivial fix

 mingw-libvirt.spec.in | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mingw-libvirt.spec.in b/mingw-libvirt.spec.in
index 6b0ce0bf5..dc18d055b 100644
--- a/mingw-libvirt.spec.in
+++ b/mingw-libvirt.spec.in
@@ -60,6 +60,7 @@ BuildRequires:  mingw64-dlfcn
 BuildRequires:  pkgconfig
 # Need native version for msgfmt
 BuildRequires:  gettext
+BuildRequires:  libxslt
 BuildRequires:  python
 %if 0%{?fedora} >= 27
 BuildRequires:  perl-interpreter
-- 
2.13.5

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH] docs: Fix typo deamon -> daemon

2017-09-04 Thread Andrea Bolognani 
Suggested-by: Martin Kletzander 
Signed-off-by: Andrea Bolognani 
---
I would push this as trivial, but I feel like someone might not be
okay with touching the release notes for libvirt versions that are
already out. I think in this case it's perfectly justified, but if
you disagree this is your chance to point it out :)

 docs/internals/rpc.html.in | 2 +-
 docs/news.xml  | 2 +-
 src/libxl/libxl.conf   | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/docs/internals/rpc.html.in b/docs/internals/rpc.html.in
index 45c07cb61..fca9cb784 100644
--- a/docs/internals/rpc.html.in
+++ b/docs/internals/rpc.html.in
@@ -534,7 +534,7 @@ C --  |32| 8 | 1 | 3 | 1 | 1 | 0 | .o.oOo |  -- S  
(reply)
 
   virNetDaemonPtr (virnetdaemon.h)
   The virNetDaemon APIs are used to manage a daemon process. A
-deamon is a process that might expose one or more servers.  It
+daemon is a process that might expose one or more servers.  It
 handles most process-related details, network-related should
 be part of the underlying server.
   
diff --git a/docs/news.xml b/docs/news.xml
index 452711717..483f9d6d1 100644
--- a/docs/news.xml
+++ b/docs/news.xml
@@ -195,7 +195,7 @@
   Fix --verbose option for all daemons
 
 
-  Since v3.0.0, the option had been ignored by all libvirt deamons
+  Since v3.0.0, the option had been ignored by all libvirt daemons
   (libvirtd, virtlogd and
   virtlockd); it's now working as intended once again.
 
diff --git a/src/libxl/libxl.conf b/src/libxl/libxl.conf
index 5c9bdaac2..264af7cf9 100644
--- a/src/libxl/libxl.conf
+++ b/src/libxl/libxl.conf
@@ -28,7 +28,7 @@
 # is sent to the daemon after keepalive_interval seconds of inactivity
 # to check if the daemon is still responding; keepalive_count is a
 # maximum number of keepalive messages that are allowed to be sent to
-# the deamon without getting any response before the connection is
+# the daemon without getting any response before the connection is
 # considered broken.  In other words, the connection is automatically
 # closed after approximately keepalive_interval * (keepalive_count + 1)
 # seconds since the last message was received from the daemon. If
-- 
2.13.5

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] Availability of libvirt 3.7.0

2017-09-04 Thread Daniel Veillard 
  I pushed the final version earlier today, it is tagged in git and
I made available sigend tarball and rpms at the usual place:

   ftp://libvirt.org/libvirt/

I also rolled out the 3.7.0 version of libvirt-python bindings, those are
available from:

   ftp://libvirt.org/libvirt/python/

This is a balanced release with new features available from virsh command
line and API, as well a general improvements and a fair amount of bug fixes:


* New features

- qemu: Add managedsave-edit commands
Using managedsave-dumpxml, managedsave-define and managedsave-edit
commands, now we can dump and edit the XML configuration of domain
which has managedsave image.

- qemu: Add migrate-getmaxdowntime command
Currently, the maximum tolerable downtime for a domain being migrated
is write-only from libvirt, via migrate-setmaxdowntime. This implements
a complementary migrate-getmaxdowntime command

- bhyve: Support autoport for VNC ports
It's no longer necessary to explicitly specify VNC port for the bhyve
guests. With the autoport feature it will be allocated automatically.
Please refer to the bhyve driver documentation for examples.

- qemu: Added support for setting heads of virtio GPU

- qemu: Added support to configure reconnect timeout for chardev devices
When you have a TCP or UNIX chardev device and it's connected somewhere
you can configure reconnect timeout if the connection is closed.

* Improvements

- qemu: Report a clear error when dropping a VM during startup
"Failed to load config for domain 'DOMNAME'" is now reported if a VM
config can't be parsed for some reason, and thus provides a clear
indication for users (and devs).

- apparmor: Update for QEMU 2.10 compatibility
Starting with QEMU 2.10, disk images and NVRAM files get automatically
locked to prevent them from being corrupted; however, file locking
needs to be explicitly allowed through virt-aa-helper or AppArmor will
reject the requests and the guest will not be able to run.

- virsh: List Unix sockets in 'domdisplay' output
VNC and SPICE graphics can use Unix sockets instead of TCP/IP sockets
as connection endpoints, but such a configuration was not handled
correctly by virsh domdisplay, causing the respective endpoints to be
missing from the output.

- qemu: Don't check whether offline migration is safe
Since offline migration only copies the guest definition to the
destination host, data corruption is not a concern and the operation
can always be performed safely.

- virt-host-validate: Fix IOMMU detection on ppc64

* Bug fixes

- qemu: Better support for international domain names (with wide
  characters)
There were some issues with multi-byte domains getting lost on daemon
restart due to truncation, so the code now handles multi-byte names a
bit better.

- qemu: Support long domain names with namespaces
Domains with extremely long names would fail to start due to temporary
namespace paths being created with the whole name. The path is now
generated with shortened name instead.

- qemu: Tolerate missing emulator binary during libvirtd restart
For some time libvirt required qemu capabilities being present when
parsing VM configs during startup. As a side effect VM configs would
fail to parse and thus vanish, if the emulator binary would be
uninstalled or broken. Libvirt now tolerates when capabilities are
missing during startup.

- qemu: Prevent pSeries guests from disappearing in some situations
pSeries guest would disappear if any of the host devices they were
configured to use was not available during libvirtd startup, which
could easily happen for SR-IOV Virtual Functions. This scenario is now
handled correctly.

- qemu: Honor  setting
The setting was accepted by the parser, but not actually implemented.

- Fix --verbose option for all daemons
Since v3.0.0, the option had been ignored by all libvirt deamons
(libvirtd, virtlogd and virtlockd); it's now working as intended once
again.

  Thanks everybody for your help with this release, be it with code, patches,
review, documentation, but reports, etc ...

   Enjoy this release !

Daniel

-- 
Daniel Veillard  | Red Hat Developers Tools http://developer.redhat.com/
veill...@redhat.com  | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | virtualization library  http://libvirt.org/

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 1/4] numa: describe siblings distances within cells

2017-09-04 Thread Martin Kletzander 

On Mon, Sep 04, 2017 at 11:51:21AM +0200, Wim ten Have wrote:

On Mon, 4 Sep 2017 08:49:33 +0200
Martin Kletzander  wrote:


On Fri, Sep 01, 2017 at 04:31:50PM +0200, Wim ten Have wrote:
>On Thu, 31 Aug 2017 16:36:58 +0200
>Martin Kletzander  wrote:
>
>> >diff --git a/src/conf/cpu_conf.c b/src/conf/cpu_conf.c
>> >index c21d11d..8d804a1 100644
>> >--- a/src/conf/cpu_conf.c
>> >+++ b/src/conf/cpu_conf.c
>> >@@ -642,7 +642,7 @@ virCPUDefFormatBufFull(virBufferPtr buf,
>> > if (virCPUDefFormatBuf(, def, updateCPU) < 0)
>> > goto cleanup;
>> >
>> >-if (virDomainNumaDefCPUFormat(, numa) < 0)
>> >+if (virDomainNumaDefCPUFormatXML(, numa) < 0)
>> > goto cleanup;
>
>> Changing function names should be separate patch.  Why is this
>> changed anyway?
>
>I renamed virDomainNumaDefCPUFormat() to virDomainNumaDefCPUFormatXML()
>to make it consistent with already existing function names like
>virDomainNumaDefCPUParseXML()
>

Then put it in a separate patch.


Sure. Do you advise me to put this patch in same or in a separated set?



Whatever suits you, I usually put clean-ups in the series as first
patches so that it is cleanly prepared for the actual changes.  But it's
only a matter of not doing multiple things in one patch in case someone
would be targetting one change in the future (finding a regression,
back-porting it, reverting it).  It also reads a bit more nicely.


- Wim.


signature.asc
Description: Digital signature
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 1/4] numa: describe siblings distances within cells

2017-09-04 Thread Wim ten Have 
On Mon, 4 Sep 2017 08:49:33 +0200
Martin Kletzander  wrote:

> On Fri, Sep 01, 2017 at 04:31:50PM +0200, Wim ten Have wrote:
> >On Thu, 31 Aug 2017 16:36:58 +0200
> >Martin Kletzander  wrote:
> >  
> >> >diff --git a/src/conf/cpu_conf.c b/src/conf/cpu_conf.c
> >> >index c21d11d..8d804a1 100644
> >> >--- a/src/conf/cpu_conf.c
> >> >+++ b/src/conf/cpu_conf.c
> >> >@@ -642,7 +642,7 @@ virCPUDefFormatBufFull(virBufferPtr buf,
> >> > if (virCPUDefFormatBuf(, def, updateCPU) < 0)
> >> > goto cleanup;
> >> >
> >> >-if (virDomainNumaDefCPUFormat(, numa) < 0)
> >> >+if (virDomainNumaDefCPUFormatXML(, numa) < 0)
> >> > goto cleanup;  
> >  
> >> Changing function names should be separate patch.  Why is this
> >> changed anyway?  
> >
> >I renamed virDomainNumaDefCPUFormat() to virDomainNumaDefCPUFormatXML()
> >to make it consistent with already existing function names like
> >virDomainNumaDefCPUParseXML()
> >  
> 
> Then put it in a separate patch.

Sure. Do you advise me to put this patch in same or in a separated set?

- Wim.

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v2] vmx: Expose VMware Managed Object Reference (moref) in XML.

2017-09-04 Thread Michal Privoznik 
On 09/01/2017 01:06 PM, Richard W.M. Jones wrote:
> If you use the VDDK library to access virtual machines remotely, you
> really need to know the Managed Object Reference ("moref") of the VM.
> This must be passed each time you connect to the API.
> 
> For example nbdkit's VDDK plugin requires a moref to be passed to
> mount up a VM's disk remotely:
> 
>  nbdkit vddk user=root password=+/tmp/rootpw \
>  server=esxi.example.com thumbprint=xx:xx:xx:... \
>  vm=moref=2 \
>  file="[datastore1] Fedora/Fedora.vmdk"
> 
> Getting the moref is a huge pain.  To get some idea of what it is, why
> it is needed, and how much trouble it is to get it, see:
> https://blogs.vmware.com/vsphere/2012/02/uniquely-identifying-virtual-machines-in-vsphere-and-vcloud-part-1-overview.html
> https://blogs.vmware.com/vsphere/2012/02/uniquely-identifying-virtual-machines-in-vsphere-and-vcloud-part-2-technical.html
> 
> However the moref is available conveniently in the internals of the
> libvirt VMX driver.  This patch exposes it as a custom XML element
> using the same "vmware:" namespace which was previously used for the
> datacenterpath (see libvirt commit 636a99058758a044).
> 
> It appears in the XML like this:
> 
>  xmlns:vmware='http://libvirt.org/schemas/domain/vmware/1.0'>
>   Fedora
> ...
>   ha-datacenter
>   2
> 
> 
> Note that the moref can appear as either a simple ID (for esx://
> connections) or as a "vm-" (for vpx:// connections).  It should be
> treated by users as an opaque string.
> ---
>  src/esx/esx_driver.c |  7 +++
>  src/esx/esx_vi.c | 13 +
>  src/esx/esx_vi.h |  4 
>  src/vmx/vmx.c| 43 +--
>  src/vmx/vmx.h|  2 ++
>  5 files changed, 59 insertions(+), 10 deletions(-)

ACK

Michal

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] event: ignore attempts to replace the event loop impl

2017-09-04 Thread Andrea Bolognani 
On Fri, 2017-09-01 at 13:49 +0100, Daniel P. Berrange wrote:
> Although not previously explicitly documented, the expectation for
> the libvirt event loop is that an implementation is registered early
> in application startup, before calling any libvirt APIs and then
> run forever after. Replacing a previously registered event loop is
> not safe & subject to races even if virConnectClose has been called
> on open handles, due to delayed deregistration of callbacks during
> conenction close.
> 
> Signed-off-by: Daniel P. Berrange 
> ---
>  src/util/virevent.c | 13 +
>  1 file changed, 13 insertions(+)

Reviewed-by: Andrea Bolognani 

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] docs: Remove from entries due to RNG error

2017-09-04 Thread Andrea Bolognani 
On Sat, 2017-09-02 at 08:09 -0400, John Ferlan wrote:
> commit '96e55048' caused make check failure for virschematest:
> 
> 1929) Checking ../docs/news.xml against ../news.rng ... 
> libvirt: XML Util error : XML document failed to validate against schema: 
> Unable to validate doc against 
> /home/jferlan/git/libvirt.work/docs/schemas/../news.rng
> Datatype element summary has child elements
> Element summary failed to validate content
> Datatype element summary has child elements
> Element summary failed to validate content
> ^[[31m^[[1mFAILED^[[0m
> 
> That's because  elements don't appear to be allowed in the schema.
> Rather than attempt to fix the schema, figured it was simpler to just
> remove them and let the schema fix happen later.
> 
> Signed-off-by: John Ferlan 
> ---
>  Pushed as build breaker.

Thanks ;)

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] qemu: Remove unused params from qemuDomainDeviceDefValidate

2017-09-04 Thread Martin Kletzander 

On Fri, Sep 01, 2017 at 01:23:09PM -0400, John Ferlan wrote:

Neither @cfg nor (now) @driver is used in the API, so remove them
and mark @opaque as UNUSED.

NB: Commit id 'fa3c558596' dropped the unused @qemuCaps which was the
last consumer of @driver other than @cfg, but even @cfg was never used
even in the original implementation from commit id 'd987f63a'.

Signed-off-by: John Ferlan 
---
Another rogue piece of unused data seen whilst working through the
VxHS patches.

src/qemu/qemu_domain.c | 5 +
1 file changed, 1 insertion(+), 4 deletions(-)



ACK


signature.asc
Description: Digital signature
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 1/4] numa: describe siblings distances within cells

2017-09-04 Thread Martin Kletzander 

On Fri, Sep 01, 2017 at 04:31:50PM +0200, Wim ten Have wrote:

On Thu, 31 Aug 2017 16:36:58 +0200
Martin Kletzander  wrote:


On Thu, Aug 31, 2017 at 04:02:38PM +0200, Wim Ten Have wrote:
>From: Wim ten Have 
>



I haven't seen the previous versions, so sorry if I point out something
that got changed already.


There was a v1 and v2 cycle by Jim and Daniel 2 month back.
Due to personal issues whole got delayed at my end where i am
catching up.


>Add libvirtd NUMA cell domain administration functionality to
>describe underlying cell id sibling distances in full fashion
>when configuring HVM guests.
>
>[below is an example of a 4 node setup]
>
>  
>
>  
>
>  
>  
>  
>  
>
>  
>  
>
>  
>  
>  
>  
>
>  
>  
>
>  
>  
>  
>  
>
>  
>
>  
>  
>  
>  
>
>  
>
>  
>

Are all those required?  I don't see the point in requiring duplicate
values.


Sorry I am not sure I understand what you mean by "duplicate values"
here. Can you elaborate?



Sure.  For simplicity let's assume 2 cells:


 
   
   
 


 

   




>Changes under this commit concern all those that require adding
>the valid data structures, virDomainNuma* functional routines and the
>XML doc schema enhancements to enforce appropriate administration.

I don't understand the point of this paragraph.


Let me rephrase this paragraph in future version.


>These changes are accompanied with topic related documentation under
>docs/formatdomain.html within the "CPU model and topology" extending the
>"Guest NUMA topology" paragraph.

This can be seen from the patch.


Agree, so I'll address this in future version too.


>For terminology we refer to sockets as "nodes" where access to each
>others' distinct resources such as memory make them "siblings" with a
>designated "distance" between them.  A specific design is described under
>the ACPI (Advanced Configuration and Power Interface Specification)
>within the chapter explaining the system's SLIT (System Locality Distance
>Information Table).

The above paragraph is also being added in the docs.


True so I'll scratch this part of the commit message.


>These patches extend core libvirt's XML description of a virtual machine's
>hardware to include NUMA distance information for sibling nodes, which
>is then passed to Xen guests via libxl. Recently qemu landed support for
>constructing the SLIT since commit 0f203430dd ("numa: Allow setting NUMA
>distance for different NUMA nodes"), hence these core libvirt extensions
>can also help other drivers in supporting this feature.

This is not true, libxl changes are in separate patch.


Like the previous comments and replies, I'll rearrange the commit
messages and rephrase them to be more specific to the patch.


There's a lot of copy-paste from the cover-letter...


There is indeed room for improvement.


>These changes alter the docs/schemas/cputypes.rng enforcing
>domain administration to follow the syntax below per numa cell id.
>
>These changes also alter docs/schemas/basictypes.rng to add
>"numaDistanceValue" which is an "unsignedInt" with a minimum value
>of 10 as 0-9 are reserved values and can not be used as System
>Locality Distance Information Table data.
>
>Signed-off-by: Wim ten Have 
>---
>Changes on v1:
>- Add changes to docs/formatdomain.html.in describing schema update.
>Changes on v2:
>- Automatically apply distance symmetry maintaining cell <-> sibling.
>- Check for maximum '255' on numaDistanceValue.
>- Automatically complete empty distance ranges.
>- Check that sibling_id's are in range with cell identifiers.
>- Allow non-contiguous ranges, starting from any node id.
>- Respect parameters as ATTRIBUTE_NONNULL fix functions and callers.
>- Add and apply topoly for LOCAL_DISTANCE=10 and REMOTE_DISTANCE=20.
>---
> docs/formatdomain.html.in   |  70 +-
> docs/schemas/basictypes.rng |   9 ++
> docs/schemas/cputypes.rng   |  18 +++
> src/conf/cpu_conf.c |   2 +-
> src/conf/numa_conf.c| 323 +++-
> src/conf/numa_conf.h|  25 +++-
> src/libvirt_private.syms|   6 +
> 7 files changed, 444 insertions(+), 9 deletions(-)
>
>diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in
>index 8ca7637..19a2ac7 100644
>--- a/docs/formatdomain.html.in
>+++ b/docs/formatdomain.html.in
>@@ -1529,7 +1529,75 @@
> 
>
> 
>-  This guest NUMA specification is currently available only for QEMU/KVM.
>+  This guest NUMA specification is currently available only for
>+  QEMU/KVM and Xen.  Whereas Xen driver also allows for a distinct
>+  description of NUMA arranged sibling cell
>+  distances Since 3.6.0.
>+
>+
>+
>+  Under NUMA h/w architecture, distinct