from:"Qiao, Liyong"

Re: [libvirt] [PATCH] Extend l3 cache to nodeinfo

2017-01-11 Thread Qiao, Liyong

Hi Daniel,

I agree that to expose othe level cache but the only can be tuned (allocation). 
If we expose such kinds of information, the host should have ability to control 
such kinds of resources.

In another thread, Martin told that there are cases which multiple sockets may 
has different values, I kinds of agree(but I don’t see that case), I agree to 
expose cache per socket, just wonder if 

cell == socket in libvirt? In my environment, I can see all socket_id in a cell 
are the same, wonder if I can extend cache information to cell node?

Best Regards

Eli Qiao(乔立勇）OpenStack Core team OTC Intel.
-- 

On 11/01/2017, 9:31 AM, "Qiao, Liyong" <liyong.q...@intel.com> wrote:

>Also, why only l3 cache. Why not expose full info about
 >   the CPU cache hierarchy. It feels wrong to expose only
 >  L3 cache and ignore other levels of cache.

Okay, I’ll think how to expose there into capabilities. This is related to 
enable cache tune support in [1]
The status in kernel is that only L3 cache can be tuned(by cat_l3 support 
in kernel) for now.

Could you help to give some input for the RFC of cache tune?

[1]https://www.redhat.com/archives/libvir-list/2017-January/msg00354.html

Thanks Eli.

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] Paradox cpu topology in capabilities outputs

2017-01-10 Thread Qiao, Liyong

Okay, I got some answer myself.



This indicate the topology in one of a numa node. But what if I want to get the 
node’s physical socket number?


Best Regards

Eli Qiao(乔立勇）OpenStack Core team OTC Intel.
--


From: "Qiao, Liyong" <liyong.q...@intel.com>
Date: Wednesday, 11 January 2017 at 2:52 PM
To: "libvir-list@redhat.com" <libvir-list@redhat.com>
Cc: "Daniel P. Berrange" <berra...@redhat.com>
Subject: Paradox cpu topology in capabilities outputs

Hi,

I observe that virsh capabilities give wrong cpu topology on a multiple sockets 
host

taget@jfz1r04h13:~/libvirt$ lscpu
Architecture:  x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):72
On-line CPU(s) list:   0-71
Thread(s) per core:2
Core(s) per socket:18
Socket(s): 2 <
NUMA node(s):  2
Vendor ID: GenuineIntel
CPU family:6
Model: 63
Model name:Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
Stepping:  2
CPU MHz:   1201.660
CPU max MHz:   3600.
CPU min MHz:   1200.
BogoMIPS:  4590.78
Virtualization:VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache:  256K
L3 cache:  46080K
NUMA node0 CPU(s): 0-17,36-53
NUMA node1 CPU(s): 18-35,54-71

But output of virsh capabilities only gives.



looking into code and got this:
https://libvirt.org/git/?p=libvirt.git;a=blob;f=src/util/virhostcpu.c;h=f29f3122acee018b9fd7dca06fd7ae1fc118b210;hb=HEAD#l703

should we change it into

704<https://libvirt.org/git/?p=libvirt.git;a=blob;f=src/util/virhostcpu.c;h=f29f3122acee018b9fd7dca06fd7ae1fc118b210;hb=HEAD#l704>
 *sockets  += nodesockets;


This also affect nodeinfo.sockets.

Attached file is the full output of capabilities of the host

Best Regards

Eli Qiao(乔立勇）OpenStack Core team OTC Intel.
--

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] Paradox cpu topology in capabilities outputs

2017-01-10 Thread Qiao, Liyong

Hi,

I observe that virsh capabilities give wrong cpu topology on a multiple sockets 
host

taget@jfz1r04h13:~/libvirt$ lscpu
Architecture:  x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):72
On-line CPU(s) list:   0-71
Thread(s) per core:2
Core(s) per socket:18
Socket(s): 2 <
NUMA node(s):  2
Vendor ID: GenuineIntel
CPU family:6
Model: 63
Model name:Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
Stepping:  2
CPU MHz:   1201.660
CPU max MHz:   3600.
CPU min MHz:   1200.
BogoMIPS:  4590.78
Virtualization:VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache:  256K
L3 cache:  46080K
NUMA node0 CPU(s): 0-17,36-53
NUMA node1 CPU(s): 18-35,54-71

But output of virsh capabilities only gives.



looking into code and got this:
https://libvirt.org/git/?p=libvirt.git;a=blob;f=src/util/virhostcpu.c;h=f29f3122acee018b9fd7dca06fd7ae1fc118b210;hb=HEAD#l703

should we change it into

704
 *sockets  += nodesockets;


This also affect nodeinfo.sockets.

Attached file is the full output of capabilities of the host

Best Regards

Eli Qiao(乔立勇）OpenStack Core team OTC Intel.
--



cap_2_sockets
Description: cap_2_sockets
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] Extend l3 cache to nodeinfo

2017-01-10 Thread Qiao, Liyong

Best Regards

Eli Qiao(乔立勇）OpenStack Core team OTC Intel.
-- 

On 10/01/2017, 8:32 PM, "Martin Kletzander"  wrote:

On Tue, Jan 10, 2017 at 04:11:03PM +0800, Eli Qiao wrote:
>This patch extends l3 cache infomation to nodeinfo output.
>

This doesn't make sense, in case there are multiple sockets with
different L3 cache sizes, it can't be represented in nodeinfo.  If there
can be multiple values, you need to extend virsh capabilities instead.

I don’t think it will work if we plug 2 different type of socket, if they can 
work together, they should have same size of L3 cache size, anyway, Daniel 
don’t agree to extend nodeinfo, I am trying to consider it as another way.

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] Extend l3 cache to nodeinfo

2017-01-10 Thread Qiao, Liyong


>Also, why only l3 cache. Why not expose full info about
 >   the CPU cache hierarchy. It feels wrong to expose only
 >  L3 cache and ignore other levels of cache.
   
Okay, I’ll think how to expose there into capabilities. This is related to 
enable cache tune support in [1]
The status in kernel is that only L3 cache can be tuned(by cat_l3 support in 
kernel) for now.

Could you help to give some input for the RFC of cache tune?

[1]https://www.redhat.com/archives/libvir-list/2017-January/msg00354.html

Thanks Eli.


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] Extend l3 cache to nodeinfo

2017-01-10 Thread Qiao, Liyong

Hi Daniel,

Thanks for your comments.

I will not change public ABI nor break RPC.

I wonder if you would agree with extend virHostCPUGetInfoPopulateLinux to get 
l3 cache size ?
Such as:
virHostCPUGetInfoPopulateLinux(FILE *cpuinfo,
   virArch arch,
   unsigned int *cpus,
   unsigned int *mhz,
   unsigned int *nodes,
   unsigned int *sockets,
   unsigned int *cores,
   unsigned int *threads,
   unsigned int *l3_cache)  ←-

virHostCPUGetInfoPopulateLinux is doing parsing /proc/cpuinfo , so I think it’s 
the best way to get l3 cache size.

cat /proc/cpuinfo

…
cpu MHz : 2821.435
cache size  : 56320 KB
…


Best Regards

Eli Qiao(乔立勇）OpenStack Core team OTC Intel.
-- 


On 10/01/2017, 5:44 PM, "Daniel P. Berrange"  wrote:

On Tue, Jan 10, 2017 at 04:11:03PM +0800, Eli Qiao wrote:
> This patch extends l3 cache infomation to nodeinfo output.
> 
> Signed-off-by: Eli Qiao 
> ---
>  include/libvirt/libvirt-host.h |  1 +
>  src/nodeinfo.c |  3 ++-
>  src/remote/remote_protocol.x   |  1 +
>  src/test/test_driver.c |  1 +
>  src/util/virhostcpu.c  | 29 +
>  src/util/virhostcpu.h  |  3 ++-
>  src/util/virhostcpupriv.h  |  3 ++-
>  tests/virhostcputest.c |  3 ++-
>  tools/virsh-host.c |  1 +
>  9 files changed, 37 insertions(+), 8 deletions(-)
> 
> diff --git a/include/libvirt/libvirt-host.h 
b/include/libvirt/libvirt-host.h
> index 07b5d15..ba926df 100644
> --- a/include/libvirt/libvirt-host.h
> +++ b/include/libvirt/libvirt-host.h
> @@ -167,6 +167,7 @@ struct _virNodeInfo {
>   processors in case of unusual NUMA 
topology*/
>  unsigned int threads; /* number of threads per core, 1 in case of
>   unusual numa topology */
> +unsigned int l3_cache; /* l3 cache in kilobytes */
>  };

NACK, it is *forbidden* to change public structs as this breaks
ABI compatibility.

Okay, get it.

> diff --git a/src/remote/remote_protocol.x b/src/remote/remote_protocol.x
> index b846ef2..6a16b4e 100644
> --- a/src/remote/remote_protocol.x
> +++ b/src/remote/remote_protocol.x
> @@ -489,6 +489,7 @@ struct remote_node_get_info_ret { /* insert@1 */
>  int sockets;
>  int cores;
>  int threads;
> +int l3_cache;
>  };

Likewise this breaks RPC compatibility.

  This info wil need to be reported in the capabilities XML instead.

Sure.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ 
:|
|: http://libvirt.org  -o- http://virt-manager.org 
:|
|: http://entangle-photo.org   -o-http://search.cpan.org/~danberr/ 
:|



--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [V3] RFC for support cache tune in libvirt

2017-01-09 Thread Qiao, Liyong

Add support for cache allocation.

Thanks Martin for the previous version comments, this is the v3 version for RFC
, I’v have some PoC code [2]. The follow changes are partly finished by the PoC.

#Propose Changes

## virsh command line

1. Extend output of nodeinfo, to expose L3 cache size for Level 3 (last level
cache size).

This will expose how many cache on a host which can be used.

root@s2600wt:~/linux# virsh nodeinfo | grep L3
L3 cache size: 56320 KiB

2. Extend capabilities outputs.

virsh capabilities | grep resctrl

...

This will tell that the host have enabled resctrl(which you can find it in
/sys/fs/resctrl),
And it supports to allocate 'L3' type cache, total 'L3' cache size is 56320
KiB, and the minimum unit size of 'L3' cache is 2816 KiB.
P.S. L3 cache size unit is the minum l3 cache unit can be allocated. It's
hardware related and can not be changed.

3. Add new virsh command 'nodecachestats':
This API is to expose vary cache resouce left on each hardware (cpu socket).

It will be formated as:

.: left size KiB

for example I have a 2 socket cpus host, and I'v enabled cat_l3 feature only

root@s2600wt:~/linux# virsh nodecachestats
L3.0 : 56320 KiB
L3.1 : 56320 KiB

P.S. resource_type can be L3, L3DATA, L3CODE, L2 for now.

4. Add new interface to manage how many cache can be allociated for a domain

root@s2600wt:~/linux# virsh cachetune kvm02 --l3.count 2

root@s2600wt:~/linux# virsh cachetune kvm02
l3.count : 2

This will allocate 2 units(2816 * 2) l3 cache for domain kvm02

## Domain XML changes

Cache Tuneing

...

## Restriction for using cache tune on multiple sockets' host.

The l3 cache is per socket resource, kernel need to konw about what's affinity
looks like, so for a VM which running on a mulitple socket's host, it should
have NUMA setting or vcpuset pin setting. Or cache tune will fail.

[1] kernel support
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/tree/arch/x86/kernel/cpu/intel_rdt.c?h=x86/cache

[2] libvirt PoC(not finished yet)
https://github.com/taget/libvirt/commits/cat_new

Best Regards

Eli Qiao(乔立勇）OpenStack Core team OTC Intel.
--

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [V2]RFC for support cache tune in libvirt

2017-01-09 Thread Qiao, Liyong

Hi Martin,

Thanks for your comments, it’s really helpful, I’v changed some of the design 
and with some PoC code.
And will later sent the v3 version of cachetune for libvirt, will add you in CC 
list.

I have some PoC code based on 3.0.0, not finished yet. But I’v learned that 
3.0.0 will be released soon, seems there will be some rebase work to do.

Best Regards

Eli Qiao(乔立勇）OpenStack Core team OTC Intel.
-- 

On 09/01/2017, 9:09 PM, "Martin Kletzander" <mklet...@redhat.com> wrote:

On Fri, Dec 23, 2016 at 05:38:13AM +0000, Qiao, Liyong wrote:
>
>Hi folks
>
>I’v send the V1 version RFC early this week, but no reply yet.
>

I replied to that and since I was going through the mail
chronologically, and this is not a reply to that one, I am getting to
this one only now.

>Thanks Qiaowei for the comment, I’v change the RFC much more libvirt 
specify, please help to comments on.
>
>##Propose Changes
>
>#Libvirtd configure changes
>
>Add a new configure option cache_allocation_ratio to libvirtd, which let 
libvirt to allocate how many cache to domains.
>

Where would be this option?  In libvirtd.conf?  Why?

>Default is 0.5
>
>Eg.
>On a host which has 55M cache, libvirt can allocate 55M * 
cache_allocation_ratio cache to domains
>

Why does this have to be statically limited?

>## Virsh command line changes:
>
> NAME
>   cachetune - control or query domain cache allocation
>
> SYNOPSIS
>   cachetune  [--enabled true/false] [--type ][--size 
] [--config] [--live] [--current]
>
> DESCRIPTION
>   Allocate cache usage for domain.
>
> OPTIONS
>   [--domain]   domain name, id or uuid
>   --enabled   enable cache allocation

So if I want to clear it out, I do "cachetune --enable false"?  That's a
mouthful.

>   --type   cache allocations type, support l3/l2 etc
>   --size  the cache size in KB
>   --config affect next boot
>   --live   affect running domain
>   --currentaffect current domain
>
>This will allow libvirt to allocate specify type l3 cache for a domain
>
>Domain xml changes:
>
>
>

Re: [libvirt] [PATCH] qemuDomainGetPreservedMounts: Fetch list of /dev/* mounts dynamically

2017-01-05 Thread Qiao, Liyong

Hi Michal

☹ Nothing changes with your patch(without workaround and with /dev/mqueue 
mounted)

root@s2600wt:~/linux# cat /proc/mounts | grep mqueue
none /dev/mqueue mqueue rw,relatime 0 0
root@s2600wt:~/linux# vim /etc/libvirt/qemu.conf
root@s2600wt:~/linux# grep namespace /etc/libvirt/qemu.conf
# To enhance security, QEMU driver is capable of creating private namespaces
# for each domain started. Well, so far only "mount" namespace is supported. If
# devices entries throughout the domain lifetime. This namespace is turned on
#namespaces = [ "mount" ]
#namespaces = []
root@s2600wt:~/linux# virsh start kvm02
error: Failed to start domain kvm02
error: An error occurred, but the cause is unknown

Attach kvm02.log, but seems nothing debug information helpful.

2017-01-06 02:47:37.544+: 74279: debug : virCommandHandshakeChild:435 : 
Notifying parent for handshake start on 26
2017-01-06 02:47:37.544+: 74279: debug : virCommandHandshakeChild:443 : 
Waiting on parent for handshake complete on 27
libvirt:  error : libvirtd quit during handshake: Input/output error

Best Regards

Eli Qiao(乔立勇）OpenStack Core team OTC Intel.
-- 

On 05/01/2017, 10:46 PM, "Martin Kletzander"  wrote:

On Thu, Jan 05, 2017 at 02:41:02PM +0100, Michal Privoznik wrote:
>With my namespace patches, we are spawning qemu in its own
>namespace so that we can manage /dev entries ourselves. However,
>some filesystems mounted under /dev needs to be preserved in
>order to be shared with the parent namespace (e.g. /dev/pts).
>Currently, the list of mount points to preserve is hardcoded
>which ain't right - on some systems there might be less or more
>items under real /dev that on our list. The solution is to parse
>/proc/mounts and fetch the list from there.
>

Works for me, ACK.  I still wonder whether we should mark the mounts as
slaves.  I was also wondering if there are multiple sub-subtrees and
from the code it looks like it will 'just work'.  Nice.

Martin

kvm02.log.namespace
Description: kvm02.log.namespace
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] libvirt failed to spawn with namespace

2017-01-05 Thread Qiao, Liyong

Hi Michal
Thanks for your reply, the workaround works well for me.
Recreate kvm02.log and attach it for you.

BTW, can you help me to understand how can you make this [1] work if it’s 
/dev/mqueue.
As far as I know /dev/mqueue is not a block device, how can we mount 
/dev/mqueue to somewhere ?

[1] 
https://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_domain.c;h=67e8836f3bfb722aae2921843659c3f7907b3dd1;hb=HEAD#l7329

Best Regards

Eli Qiao(乔立勇）OpenStack Core team OTC Intel.
-- 

On 05/01/2017, 4:34 PM, "Michal Privoznik" <mpriv...@redhat.com> wrote:

On 01/05/2017 08:48 AM, Qiao, Liyong wrote:
> Hi Michal
> 
> Build with the latest libvirt source code, I found failed to spawn a qemu 
process, not sure if something wrong with my environment, can you please help 
to take a look at it?
> 
> After build the latest (3.0.0) libvirt, found failed to start an existed 
domain.
> 
> root@s2600wt:/home/taget/qemu# virsh start kvm02
> error: Failed to start domain kvm02
> error: internal error: Process exited prior to exec: libvirt: QEMU Driver 
error : Unable to move /dev/mqueue mount: Invalid argument

D'oh. There's a list of FSs that libvirt tries to preserve and bind
mount from the parent namespace. But it is not fault tolerant.

> 
> then I found you had some code get merged, and required to use /dev/mqueue

Yes. Exactly.

> 
> so mount it then start the domain again:
> 
> mount -t mqueue none /dev/mqueue
> 
> root@s2600wt:/home/taget/qemu# virsh start kvm02
> error: Failed to start domain kvm02
> error: An error occurred, but the cause is unknown

Can you get /var/log/libvirt/qemu/kvm02.log?

As a workaround you can set namespaces=[] in qemu.conf, but that is
really meant just like a workaround until all of these edge cases are fixed.

Meanwhile, I started working on a fix that will drop the persistent list
and parse /proc/mounts instead.

Michal

kvm02.log
Description: kvm02.log
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] libvirt failed to spawn with namespace

2017-01-04 Thread Qiao, Liyong

Hi Michal

Build with the latest libvirt source code, I found failed to spawn a qemu 
process, not sure if something wrong with my environment, can you please help 
to take a look at it?

After build the latest (3.0.0) libvirt, found failed to start an existed domain.

root@s2600wt:/home/taget/qemu# virsh start kvm02
error: Failed to start domain kvm02
error: internal error: Process exited prior to exec: libvirt: QEMU Driver error 
: Unable to move /dev/mqueue mount: Invalid argument

then I found you had some code get merged, and required to use /dev/mqueue

so mount it then start the domain again:

mount -t mqueue none /dev/mqueue

root@s2600wt:/home/taget/qemu# virsh start kvm02
error: Failed to start domain kvm02
error: An error occurred, but the cause is unknown


some debug log are:

…
2017-01-05 07:48:44.195+: 37227: debug : virFileMakePathHelper:2899 : 
path=/var/run/libvirt/qemu/kvm02.dev/ mode=0777
2017-01-05 07:48:44.195+: 37227: debug : virFileMakeParentPath:2971 : 
path=/var/run/libvirt/qemu/kvm02.dev//hpet
2017-01-05 07:48:44.195+: 37227: debug : virFileMakePathHelper:2899 : 
path=/var/run/libvirt/qemu/kvm02.dev/ mode=0777
2017-01-05 07:48:44.195+: 37227: debug : virFileMakeParentPath:2971 : 
path=/var/run/libvirt/qemu/kvm02.dev//vfio/vfio
2017-01-05 07:48:44.195+: 37227: debug : virFileMakePathHelper:2899 : 
path=/var/run/libvirt/qemu/kvm02.dev//vfio mode=0777
2017-01-05 07:48:44.195+: 37227: debug : virFileMakePathHelper:2899 : 
path=/var/run/libvirt/qemu/kvm02.dev/ mode=0777
2017-01-05 07:48:44.195+: 37227: info : virObjectUnref:259 : OBJECT_UNREF: 
obj=0x7f7dc812fad0
2017-01-05 07:48:44.195+: 37227: debug : qemuDomainSetupAllDisks:7076 : 
Setting up disks
2017-01-05 07:48:44.195+: 37227: debug : qemuDomainSetupAllDisks:7085 : 
Setup all disks
2017-01-05 07:48:44.195+: 37227: debug : qemuDomainSetupAllHostdevs:7124 : 
Setting up hostdevs
2017-01-05 07:48:44.195+: 37227: debug : qemuDomainSetupAllHostdevs:7131 : 
Setup all hostdevs
2017-01-05 07:48:44.195+: 37227: debug : qemuDomainSetupAllChardevs:7155 : 
Setting up chardevs
2017-01-05 07:48:44.195+: 37227: debug : qemuDomainSetupAllChardevs:7163 : 
Setup all chardevs
2017-01-05 07:48:44.195+: 37227: debug : qemuDomainSetupAllInputs:7231 : 
Setting up disks
2017-01-05 07:48:44.195+: 37227: debug : qemuDomainSetupAllInputs:7238 : 
Setup all disks
2017-01-05 07:48:44.195+: 37227: debug : qemuDomainSetupAllRNGs:7270 : 
Setting up RNGs
2017-01-05 07:48:44.195+: 37227: debug : qemuDomainSetupAllRNGs:7278 : 
Setup all RNGs
2017-01-05 07:48:44.195+: 37227: debug : virFileMakePathHelper:2899 : 
path=/dev/pts mode=0777
2017-01-05 07:48:44.195+: 37227: debug : virFileMakePathHelper:2899 : 
path=/dev mode=0777
2017-01-05 07:48:44.195+: 37227: debug : virFileMakePathHelper:2899 : 
path=/dev/shm mode=0777
2017-01-05 07:48:44.195+: 37227: debug : virFileMakePathHelper:2899 : 
path=/dev mode=0777
2017-01-05 07:48:44.195+: 37227: debug : virFileMakePathHelper:2899 : 
path=/dev/mqueue mode=0777
2017-01-05 07:48:44.195+: 37227: debug : virFileMakePathHelper:2899 : 
path=/dev mode=0777
2017-01-05 07:48:44.195+: 37227: info : virObjectUnref:259 : OBJECT_UNREF: 
obj=0x7f7dc812fad0
2017-01-05 07:48:44.195+: 37227: info : virObjectUnref:259 : OBJECT_UNREF: 
obj=0x7f7dc812fad0
2017-01-05 07:48:44.195+: 37227: debug : qemuProcessHook:2693 : Hook 
complete ret=0
2017-01-05 07:48:44.195+: 37227: debug : virExec:699 : Done hook 0
2017-01-05 07:48:44.195+: 37227: debug : virExec:736 : Setting child 
uid:gid to 0:0 with caps 0
2017-01-05 07:48:44.195+: 37227: debug : virCommandHandshakeChild:435 : 
Notifying parent for handshake start on 25
2017-01-05 07:48:44.195+: 37227: debug : virCommandHandshakeChild:443 : 
Waiting on parent for handshake complete on 26
libvirt:  error : libvirtd quit during handshake: Input/output error
2017-01-05 07:48:44.216+: shutting down, reason=failed


/\
I work on the latest libvirt git tree.


commit 866641d4c5706413393913fdb3bb1cd077683d21
Author: Michal Privoznik 
Date:   Sat Dec 24 17:55:48 2016 +0100

NEWS: Update after qemu namespace fix

Signed-off-by: Michal Privoznik 



Best Regards

Eli Qiao(乔立勇）OpenStack Core team OTC Intel.
--

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [V2]RFC for support cache tune in libvirt

2016-12-22 Thread Qiao, Liyong


Hi folks

I’v send the V1 version RFC early this week, but no reply yet.

Thanks Qiaowei for the comment, I’v change the RFC much more libvirt specify, 
please help to comments on.

##Propose Changes

#Libvirtd configure changes

Add a new configure option cache_allocation_ratio to libvirtd, which let 
libvirt to allocate how many cache to domains.

Default is 0.5

Eg.
On a host which has 55M cache, libvirt can allocate 55M * 
cache_allocation_ratio cache to domains

## Virsh command line changes:

 NAME
   cachetune - control or query domain cache allocation

 SYNOPSIS
   cachetune  [--enabled true/false] [--type ][--size ] 
[--config] [--live] [--current]

 DESCRIPTION
   Allocate cache usage for domain.

 OPTIONS
   [--domain]   domain name, id or uuid
   --enabled   enable cache allocation
   --type   cache allocations type, support l3/l2 etc
   --size  the cache size in KB
   --config affect next boot
   --live   affect running domain
   --currentaffect current domain

This will allow libvirt to allocate specify type l3 cache for a domain

Domain xml changes:

[libvirt] RFC for support Intel RDT/CAT in libvirt

2016-12-21 Thread Qiao, Liyong

Hi folks

I would like to start a discussion about how to support a new cpu feature in
libvirt. CAT support is not fully merged into linux kernel yet, the target
release is 4.10, and all patches has been merged into linux tip branch. So
there won’t be interface/design changes.

## Background

Intel RDT is a toolkit to do resource Qos for cpu such as llc(l3) cache,
memory bandwidth usage, these fine granularity resource control features are
very useful in a cloud environment which will run logs of noisy instances.
Currently, Libvirt has supported CAT/MBMT/MBML already, they are only for
resource usage monitor, propose to supporting CAT to control VM’s l3 cache
quota.

## CAT interface in kernel

In kernel, a new resource interface has been introduced under /sys/fs/resctrl,
it’s used for resource control, for more information, refer
Intel_rdt_ui [
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/tree/Documentation/x86/intel_rdt_ui.txt?h=x86/cache
]

Kernel requires to provide schemata for l3 cache before add a task to a new
partition, these interface is too much detail for a virtual machine user, so
propose to let Libvirt manage schemata on the host.

## What will libvirt do?

### Questions:

To enable CAT support in libvirt, we need to think about follow questions

1. Only set CAT when an VM has CPU pin, which is to say, l3 cache is per cpu
socket resources. On a host which has 2 cpu sockets, each cpu socket has it own
cache, and can not be shared..
2. What the cache allocation policy should be used, this will be looks like:
a. VM has it’s own dedicated l3 cache and also can share other
l3 cache.
b. VM can only use the caches which allocated to it.
c. Has some pre-defined policies and priority for a VM
Like COB [1]

1. Should reserve some l3 cache for host’s system usage (related to 2)
2. What’s the unit for l3 cache allocation? (related to 2)

### Propose Changes

XML domain user interface changes:

Option 1: explicit specify cache allocation for a VM

1 work with numa node

Some cloud orchestration software use numa + vcpu pin together, so we can
enable cat supporting with numa infra.

Expose how many l3 cache a VM want to reserved and we require that the l3 cache
should be bind on some specify cpu socket, just like what we did for numa node.

This is an domain xml example which is generated by OpenStack Nova for allocate
llc(l3 cache) when booting a new VM

…

...

Refer to [http://libvirt.org/formatdomain.html#elementsCPUTuning]

So finally we can calculate on which CPU socket(cell) we need to allocate how
may l3cache for a VM.

2. work with vcpu pin

Forget numa part, CAT setting should have relationship with cpu core setting,
we can apply CAT policy if VM has set cpu pin setting (only VM won’t be
schedule to another CPU sockets)

Cache allocation on which CPU socket can be calculate as just as 1.

We may need to enable both 1 and 2.

There are several policy for cache allocation:

Let’ take some examples:

For intel e5 v4 2699(Signal socket), there are 55M l3 cache on the chip , the
default of L3 schemata is L3:0=ff , it represents to use 20 bit to control
l3 cache, each bit will represent 2.75M, which will be the minimal unit on this
host.
The allocation policy could be 3 policies :

1. One priority VM:
A priority import VM can be allocated a dedicated amount l3 cache (let’s say
2.75 * 4 = 11M) and it can also reach the left 44 M cache which will be shared
with other process and VM on the same host.
So that we need to create a new ‘Partition’ n-20371

root@s2600wt:/sys/fs/resctrl# ls
cpus info n-20371 schemata tasks

Inside of n-20371 directory:

root@s2600wt:/sys/fs/resctrl# ls n-20371/
cpus schemata tasks

The schemata content will be L3:0=f
The tasks content will be the pids of that VM

Along we need to change the default schemata of system:

root@s2600wt:/sys/fs/resctrl# cat schemata
L3:0= # which can not use the highest 4 bits, only tasks in n-20371 can
reach that.

In this design , we can only get 1 priority VM.

Let’s change it a bit to have 2 VMs

The schemata of the 2 VMs could be:

1. L3:0=0 # could not use the 4 low bits 11M l3 cache
2. L3:0=0 # could not use the 4 high bits 11M l3 cache

Default schemata changes to

L3:0=0fff0 # default system process could only use the middle 33M l3 cache

2. Isolated l3 cache dedicated allocation for each VM(if required)

A VM can only use the cache allocated to it.

For example
VM 1 requires to allocate 11 M
It’s schemata will be L3:0=f #
VM 2 requires to allocate 11M
It’s schemata will be L3:0=f000

And default schemata will be L3:0=fff

In this case, we can create multiple VMs which each of them can have dedicated
l3 cache.
The disadvantage is that we the allocated cache could be not be shared
efficiently.

Re: [libvirt] [PATCH] Add Skylake Cpu Model

2016-05-17 Thread Qiao, Liyong

Hi Jirka
Thanks for your comments,, the qemu patch is on the way, check 
https://lists.gnu.org/archive/html/qemu-devel/2016-04/msg03951.html

Even we cann't use qemu to boot a Skylake cpu virtual machine, but if we 
specify in host mode, we still can  
And besides, the follow cpu feature has already in qemu tree.

+
+
+  
+
+
+  
+
+
+  
+
+
+  
+

Best Regards,
Qiao, Liyong (Eli) OTC SSG Intel

   此致
敬礼！
英特尔（中国）有限公司软件与服务部开源技术中心 乔立勇



  -Original Message-
  From: Jiri Denemark [mailto:jdene...@redhat.com]
  Sent: Monday, May 16, 2016 9:03 PM
  To: Qiao, Liyong <liyong.q...@intel.com>
  Cc: libvir-list@redhat.com
  Subject: Re: [libvirt] [PATCH] Add Skylake Cpu Model
  
  On Mon, May 16, 2016 at 17:15:44 +0800, Eli Qiao wrote:
  > Add Skylake Cpu model to cpu maps to let libvirt discover host
  Skylake
  > cpu model correctly.
  >
  > Currently, libvirt still think a Skylake cpu as Broadwell-noTSX
  > Signed-off-by: Eli Qiao <liyong.q...@intel.com>
  > ---
  >  src/cpu/cpu_map.xml | 76
  >
  
  +
  >  1 file changed, 76 insertions(+)
  
  Adding a new model to libvirt when there is no support for it in
  QEMU makes no sense. The only benefit will be seeing Skylake in
  capabilities XML, but users won't be able to use it anyway.
  
  Jirka

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] cpu: add ABM to Haswell* and Broadwell* CPU models

2016-05-08 Thread Qiao, Liyong

Hi jiri,

Thanks for you kindly reviewing, I wonder what cpu/cpu_map.xml does in libvirt 
if QEMU already provide all features to guest?

Best Regards,
Qiao, Liyong (Eli) OTC SSG Intel

   此致
敬礼！
英特尔（中国）有限公司软件与服务部开源技术中心 乔立勇

  -Original Message-
  From: Jiri Denemark [mailto:jdene...@redhat.com]
  Sent: Friday, May 06, 2016 6:56 PM
  To: Qiao, Liyong <liyong.q...@intel.com>
  Cc: libvir-list@redhat.com
  Subject: Re: [libvirt] [PATCH] cpu: add ABM to Haswell* and
  Broadwell* CPU models

  On Fri, May 06, 2016 at 17:53:04 +0800, Eli Qiao wrote:
  > Corresponding QEMU commits:
  > becb66673ec30cb604926d247ab9449a60ad8b11
  >
  > Signed-off-by: Eli Qiao <liyong.q...@intel.com>
  > ---
  >  src/cpu/cpu_map.xml | 4 
  >  1 file changed, 4 insertions(+)

  NACK. We don't add features to models since it could break
  existing setups. In any case, presence of any flag in CPU models
  does not really influence whether that flag is going to be 
provided
  to a guest. That part is driven by QEMU.

  Jirka

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [RFC] libvirt support multi-thread compression for live migration

2015-06-04 Thread Qiao,Liyong




On 2015年06月04日 20:09, Jiri Denemark wrote:

On Tue, Jun 02, 2015 at 11:02:27 -0600, Eric Blake wrote:

On 06/02/2015 07:56 AM, Qiao, Liyong wrote:

Hi Eric
Thanks for replying the mail, replied in lines.


+virdomainMigrateGetParameters(virDomainPtr domain,
+  int *level,
+  int *threads,
+  int *dthreads,
+  int flags)
+

I'd rather we used virTypedParameters, to make it easier to use the same API to 
pass ALL future possible tuning parameters, rather than just hard-coding to 
only the parameters of this one feature.

Okay ,that sound good, but if virTypedParameters can be passed to qemu_driver 
such as qemu_monitor_json.c qemu_monitor_text.c ?

[Your quoting style makes it very hard to distinguish original text from
added text.  Please consider changing your outgoing mail process to
ensure that you add another level of quoting to all previous text so
that your added text is the only unquoted text.  Also, configure your
mailer to wrap long lines.]

Use existing API for a guide - for example, virDomainSetBlockIoTune
takes virTypedParamters, as well as defines a specific set of parameters
that it will understand.

And do we actually need to changed these migration parameters on the fly
when migration is already running. I can imagine we would need to do so
for some parameters but multithreaded compression sounds like something
that needs to be configured when migration starts and there's nothing to
be tuned on the fly. If this is the case, I think we should just add
these new parameters to virDomainMigrate3 instead of requiring another
API to be called to actually configure multithreaded compression.

The separate API makes it a bit harder to repeat migration (which
previously failed) with a different parameters (e.g., without
multithreaded compression).

hi jirka
thanks for your advice, that make sense for me.

if I understanding you correctly,
then we need only do this

virsh migrate --live --compressed --compress-level 1--compress-threads 8 
--decompress-threads 2


and all compressed parameters (compress-level ,compress-threads, 
decompress-threads)  only
be specified when we do call virDomainMigrate3 instead of expose to user 
by 'virsh migrate-setparameters' ?


I think that's good for me, and make user easy to start LM.

thanks,
Eli.


Jirka


--
BR, Eli(Li Yong)Qiao

attachment: liyong_qiao.vcf--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [RFC] libvirt support multi-thread compression for live migration

2015-06-04 Thread Qiao,Liyong




On 2015年06月04日 20:01, Jiri Denemark wrote:

On Wed, Jun 03, 2015 at 15:13:17 +, Feng, Shaohe wrote:

Eric,
Thank you for reply.

Is it necessary to expose these two APIs to user application?
+ virdomainMigrateSetCapabilities
+ virdomainMigrateGetCapabilities

For qemu , the migration capabilities are xbzrle, rdma-pin-all,  
auto-converge,
  zero-blocks and compress.

No, these capabilities are either turned on/off based on flags passed to
virDomainMigrate3 API.
I think we need to call qemuMonitorGetMigrationCapability when 
qemuMigrationSetCompression
if the source/dest node doesn't support  'compress' capability, we need 
to stop migration if
flags contain VIR_MIGRATE_COMPRESSED (currently it stands for xbzrle, 
and Eric's previous mail

suggest we share this flag with 'compress')

-Eli.

Jirka


--
BR, Eli(Li Yong)Qiao

attachment: liyong_qiao.vcf--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [RFC] libvirt support multi-thread compression for live migration

2015-06-02 Thread Qiao,Liyong




On 2015年06月03日 01:02, Eric Blake wrote:

On 06/02/2015 07:56 AM, Qiao, Liyong wrote:

Hi Eric
Thanks for replying the mail, replied in lines.


+virdomainMigrateGetParameters(virDomainPtr domain,
+  int *level,
+  int *threads,
+  int *dthreads,
+  int flags)
+

I'd rather we used virTypedParameters, to make it easier to use the same API to 
pass ALL future possible tuning parameters, rather than just hard-coding to 
only the parameters of this one feature.

Okay ,that sound good, but if virTypedParameters can be passed to qemu_driver 
such as qemu_monitor_json.c qemu_monitor_text.c ?

[Your quoting style makes it very hard to distinguish original text from
added text.  Please consider changing your outgoing mail process to
ensure that you add another level of quoting to all previous text so
that your added text is the only unquoted text.  Also, configure your
mailer to wrap long lines.]
hi Eric, sorry about the inconvenient mail client, I switch outlook to 
thunder

bird, hopes it will be better.

Use existing API for a guide - for example, virDomainSetBlockIoTune
takes virTypedParamters, as well as defines a specific set of parameters
that it will understand.  The set of parameters can be enlarged without
needing a new API (good for backporting features without requiring a .so
version bump), but for any given libvirt version, the set of features
understood is finite and limited to what you can translate to the QMP
call.  So qemu_driver.c would take the virTypedParameters, reject the
ones it does not understand, and convert the ones it does understand
into the 'int level, int threads, int dthreads' parameters used in
qemu_monitor_json.c where you drive the actual QMP command with fixed
parameters and types.
ah, that helps, thanks for kindly supporting, we will think a bit more 
to how

implement this API (taken virTypedParamters as parameter)




If we think that it is worth always turning on both compression styles 
simultaneously, then reuse the bit.  Otherwise, we need a different bit, and 
users can choose which (or both) of the two compression styles as desired.

+1 for reuse compressed, and we can set compress-level, compress-threads, 
compress-dthreads by virdomainMigrateSetParameters(maybe some new virsh 
command--- migrate-setparameter)
But how can we passing these parameter when we using `virsh migrate `, is there 
any parameter we can use in 'virsh migrate' command ?
Can you point me out ?

The underlying API already has a form that takes virTypedParameters (see
virDomainMigrate3()), so your first task is to figure out how to extend
the API to expose new typed parameters for your new migration tunings.
Once that is done, then you can worry about how to tweak the 'virsh
migrate' client to pass in those new parameters.


--
BR, Eli(Li Yong)Qiao

attachment: liyong_qiao.vcf--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [RFC] libvirt support multi-thread compression for live migration

2015-06-02 Thread Qiao, Liyong

Hi Eric
Thanks for replying the mail, replied in lines.

BR, Eli(Li Yong)Qiao

-Original Message-
From: Eric Blake [mailto:ebl...@redhat.com] 
Sent: Tuesday, June 02, 2015 8:21 PM
To: Feng, Shaohe; libvir-list@redhat.com
Cc: Qiao, Liyong; Bhandaru, Malini K; Ding, Jian-feng; Chylinski, Arek; 
Koniszewski, Pawel; Li, Liang Z; berra...@redhat.com; veill...@redhat.com
Subject: Re: [RFC] libvirt support multi-thread compression for live migration

[correcting list address: libvirt-list, not libver-list]

On 06/02/2015 05:58 AM, Feng, Shaohe wrote:
 Hi folks:
 I'd like to request some comments on enabling multi-thread compression in 
 libvirt.

 Currently, qemu upstream has supported multi-thread compression for live 
 migration.
 $ git log 263170e~1..362ba4e -oneline
 This can preserve bandwidth and speed up the live migration process, 
 so this is an import feature we need to enable so for other high level such 
 as openstack will be benefit.

 We plan to add feature of multi-thread compression and actually the patch is 
 working in process.

 Now some things need for comments.

 1.  Expose new set/get multi-thread compression parameters API for live 
 migration.
 Now qemu only supports zlib compression. Maybe LZ4 and gzip will be 
 supported later.
 but there are 3 parameters needed by qemu currently.
 compress-level:  compression level, default is 1. For zlib compression, it 
 is from 0-9, 0 means use default level, 9 means high compressed.
 compress-threads:  compression thread count for multi-thread migration, 
 default is 8 in qemu.
 decompress-threads: decompression thread count for multi-thread 
 migration, default is 2 in qemu

 So in libvirt we will expose the public symbols as follow, we only 3 
 parameters, missing compression method(zlib, LZ4 and gzip) parameters, for 
 qemu do not support it at present.
 index d851225..103b3b9 100644
 --- a/include/libvirt/libvirt-domain.h
 +++ b/include/libvirt/libvirt-domain.h
 @@ -798,6 +798,17 @@ int virDomainMigrateGetMaxSpeed(virDomainPtr domain,
  unsigned long *bandwidth,
  unsigned int flags);

 +int virdomainMigrateSetParameters(virDomainPtr domain,
 +  int level,
 +  int threads,
 +  int dthreads,
 +  int flags) int 
 +virdomainMigrateGetParameters(virDomainPtr domain,
 +  int *level,
 +  int *threads,
 +  int *dthreads,
 +  int flags)
 +

I'd rather we used virTypedParameters, to make it easier to use the same API to 
pass ALL future possible tuning parameters, rather than just hard-coding to 
only the parameters of this one feature.

Okay ,that sound good, but if virTypedParameters can be passed to qemu_driver 
such as qemu_monitor_json.c qemu_monitor_text.c ?

 For virdomainMigrateSetParameters, if specifying a negative value to 
 level, threads, and dthreads parameters, such as -1, means do not set the 
 parameters.
 The underlying code will not pass the corresponding parameters to qemu 
 monitor.
 Such as threads and dthreads is -1, then pass the following json streaming to 
 qemu.
 { execute: migrate-set-parameters , arguments:  { 
 compress-level: 1 } }

 The virsh will expose the two commands:
 # virsh migrate-setparameters  domain [--level level] [--threads 
 threads] [--dthreads dthread] # virsh migrate-getparameters  domain

 2.  How to enable multi-thread compression in application interface?
There is not a special interface for setting migration capabilities.
 But we can set the capabilities when execute an virsh command as 
 following:
 $ migrate --live] [--offline] [--p2p] [--direct] [--tunnelled] 
 [--persistent] [--undefinesource] [--suspend] [--copy-storage-all] 
 [--copy-storage-inc] [--change-protection] [--unsafe] [--verbose] 
 [--compressed] [--abort-on-error] domain desturi [migrateuri] 
 [graphicsuri] [listen-address] [dname] [--timeout number] 
 [--xml string]

 There is already a compressed option,  here compressed means xbzrle not 
 multi- compressed.
 can I rename the compressed to xbzrle?

If we think that it is worth always turning on both compression styles 
simultaneously, then reuse the bit.  Otherwise, we need a different bit, and 
users can choose which (or both) of the two compression styles as desired.

+1 for reuse compressed, and we can set compress-level, compress-threads, 
compress-dthreads by virdomainMigrateSetParameters(maybe some new virsh 
command--- migrate-setparameter)
But how can we passing these parameter when we using `virsh migrate `, is there 
any parameter we can use in 'virsh migrate' command ?
Can you point me out ?

 so we add another option for multi-thread compression, which is better option 
 name? -multi- compressed ?
 Any way

Re: [libvirt] [PATCH] Extend l3 cache to nodeinfo

Re: [libvirt] Paradox cpu topology in capabilities outputs

[libvirt] Paradox cpu topology in capabilities outputs

Re: [libvirt] [PATCH] Extend l3 cache to nodeinfo

Re: [libvirt] [PATCH] Extend l3 cache to nodeinfo

Re: [libvirt] [PATCH] Extend l3 cache to nodeinfo

[libvirt] [V3] RFC for support cache tune in libvirt

Re: [libvirt] [V2]RFC for support cache tune in libvirt

Re: [libvirt] [PATCH] qemuDomainGetPreservedMounts: Fetch list of /dev/* mounts dynamically

Re: [libvirt] libvirt failed to spawn with namespace

[libvirt] libvirt failed to spawn with namespace

[libvirt] [V2]RFC for support cache tune in libvirt

[libvirt] RFC for support Intel RDT/CAT in libvirt

Re: [libvirt] [PATCH] Add Skylake Cpu Model

Re: [libvirt] [PATCH] cpu: add ABM to Haswell* and Broadwell* CPU models

Re: [libvirt] [RFC] libvirt support multi-thread compression for live migration

Re: [libvirt] [RFC] libvirt support multi-thread compression for live migration

Re: [libvirt] [RFC] libvirt support multi-thread compression for live migration

Re: [libvirt] [RFC] libvirt support multi-thread compression for live migration

19 matches

Site Navigation

Mail list logo

Footer information