[Engine-devel] FW: Discussion about NUMA feature and CPU pinning

2014-04-02 Thread Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC) 
Forward to public group

Best Regards,
Jason Liao

From: Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC)
Sent: 2014年4月1日 16:56
To: 'Gilad Chaplik'
Cc: Doron Fediuck; Vinod, Chegu; Liang, Shang-Chun (David Liang, 
HPservers-Core-OE-PSC); Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ); 
'de...@ovirt.org'
Subject: Discussion about NUMA feature and CPU pinning

Hi Gilad,

When I define VdcActionType to manage Virtual NUMA node and pin Virtual NUMA 
node to host NUMA node.

There are some operations in list:

1.  New Virtual NUMA node ( set vcpus count, set total memory )

2.  Pin Virtual NUMA node to host NUMA node ( save host NUMA node ID into 
Virtual NUMA node )

3.  Save a VM’s all Virtual NUMA node

a) Calculate VM pin to host property from host NUMA node’s host ID.

b) Calculate VM NUMA tuning nodeset from all Virtual NUMA node’s pin to 
host NUMA node ID.

c) Calculate VM CPU pinning from all Virtual NUMA node’s vcpus and 
related pin to host NUMA node cpus

e.g.

host NUMA node 0 cpus 0, 1, 2, 3

host NUMA node 1 cpus 4, 5, 6, 7

host NUMA node 2 cpus 8, 9, 10, 11

host NUMA node 3 cpus 12, 13, 14, 15

virtual NUMA node 0 vcpus count 2

virtual NUMA node 1 vcpus count 2



pin virtual NUMA node 0 to host NUMA node 1

pin virtual NUMA node 1 to host NUMA node 3



b. Calculate result: 1,3

c. Calculate result: 0#4,5,6,7_1#4,5,6,7_2#12,13,14,15_3#12,13,14,15

Discussion:

1.  Do we need these operations, especially method c. ?

2.  If the answer is Yes, this operation will replace the Cpu pinning 
configuration from VM Cpu pining input, Do we need to notify the owner ?

Best Regards,
Jason Liao

___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel

[Engine-devel] NUMA feature for oVirt Scheduling interface

2014-04-02 Thread Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC) 
Hi Gilad,

I have update the wiki page
http://www.ovirt.org/Features/Detailed_NUMA_and_Virtual_NUMA#Interface_and_data_structure_in_ovirt_scheduler

The ovirt scheduler section.

We are appreciate that you or some community members could give us some 
feedback and comments, and sorry for the nag.

Best Regards,
Jason Liao

___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel

Re: [Engine-devel] Numa feature entities patch

2014-04-01 Thread Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC) 
Hi Gilad,

Did you see updated wiki page design with VdcQueryType, VdcActionType ?

Feel free to let us know if you have any comment.

Best Regards,
Jason Liao

-Original Message-
From: Gilad Chaplik [mailto:gchap...@redhat.com] 
Sent: 2014年4月1日 15:51
To: Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC)
Cc: Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); Doron Fediuck; Vinod, 
Chegu; Martin Sivak; Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ); 
engine-devel@ovirt.org; Roy Golan
Subject: Re: Numa feature entities patch

Great to hear :-)

VdcQueryType and VdcActionType with parameters are not part of the initial BE 
patch; should part of the server's queries and commands patch.



Thanks, 
Gilad.


- Original Message -
 From: Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) 
 shangchun.li...@hp.com
 To: Gilad Chaplik gchap...@redhat.com, Chuan Liao (Jason Liao, 
 HPservers-Core-OE-PSC) chuan.l...@hp.com
 Cc: Doron Fediuck dfedi...@redhat.com, Chegu Vinod 
 chegu_vi...@hp.com, Martin Sivak msi...@redhat.com,
 Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com, 
 engine-devel@ovirt.org, Roy Golan
 rgo...@redhat.com
 Sent: Tuesday, April 1, 2014 3:38:20 AM
 Subject: RE: Numa feature entities patch
 
 Hi Gilad,
 The below are our understanding. Please correct me if anything wrong. :)
 For VdcQueryType and VdcActionType with parameters, we think it's need to
 update in doc asap because the code of this part has not started yet.
 For other parts that started the coding, We will reach an agreement in
 gerrit/review and then update in the docs accordingly.
 
 Best Regards,
 David Liang
 
 -Original Message-
 From: Gilad Chaplik [mailto:gchap...@redhat.com]
 Sent: Tuesday, April 01, 2014 1:05 AM
 To: Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC)
 Cc: Doron Fediuck; Vinod, Chegu; Martin Sivak; Shi, Xiao-Lei (Bruce, HP
 Servers-PSC-CQ); Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC);
 engine-devel@ovirt.org; Roy Golan
 Subject: Re: Numa feature entities patch
 
 Hi Jason,
 
 Please reply on the comments in the patch.
 Once the code is written I think it's better not to context switch back to
 the design.
 we will reach an agreement in gerrit/review and then update the design
 respectively.
 
 
 Thanks,
 Gilad.
 
 - Original Message -
  From: Chuan Liao (Jason Liao, HPservers-Core-OE-PSC)
  chuan.l...@hp.com
  To: Gilad Chaplik gchap...@redhat.com
  Cc: Doron Fediuck dfedi...@redhat.com, Chegu Vinod
  chegu_vi...@hp.com, Martin Sivak msi...@redhat.com, Xiao-Lei
  Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com, Shang-Chun
  Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com,
  engine-devel@ovirt.org
  Sent: Monday, March 31, 2014 7:48:28 PM
  Subject: RE: Numa feature entities patch
  
  Hi Gilad,
  
  I have update the wiki page design of BE patch 1. add VdcQueryType and
  VdcActionType with parameters.
  2. merge some of your comments and Eli's feedback.
  
  Please take a look at the section
  http://www.ovirt.org/Features/Detailed_NUMA_and_Virtual_NUMA#Interface
  _and_data_structure_in_engine_core
  
  We appreciate any of the comments from community, and sorry for the nag.
  
  Best Regards,
  Jason Liao
  
  -Original Message-
  From: Gilad Chaplik [mailto:gchap...@redhat.com]
  Sent: 2014年3月30日 22:43
  To: Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); Shi, Xiao-Lei
  (Bruce, HP Servers-PSC-CQ); Liang, Shang-Chun (David Liang,
  HPservers-Core-OE-PSC)
  Cc: Doron Fediuck; Vinod, Chegu; Martin Sivak
  Subject: Numa feature entities patch
  
  Hi all,
  
  I had some comments :-) IMO this patch [1] has maximum priority.
  let's try to merge it ASAP, I will be very responsive to every new upload.
  
  Thanks,
  Gilad.
  
  [1] http://gerrit.ovirt.org/#/c/23702/
 
 
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel

[Engine-devel] NUMA feature for MOM interface

2014-04-01 Thread Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC) 
Hi Martin  Adam

I have update the wiki page design
http://www.ovirt.org/Features/Detailed_NUMA_and_Virtual_NUMA#Interface_between_VDSM_and_Host
I-1.4 Data structure that provided to MOM component

From this design MOM could use VDSM HypervisorInterface to get NUMA from host.


We appreciate any of the comments from you and community, and sorry for the nag.

Best Regards,
Jason Liao

___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel

Re: [Engine-devel] Numa feature entities patch

2014-03-31 Thread Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC) 
Hi Gilad,

I have update the wiki page design of BE patch
1. add VdcQueryType and VdcActionType with parameters.
2. merge some of your comments and Eli's feedback.

Please take a look at the section 
http://www.ovirt.org/Features/Detailed_NUMA_and_Virtual_NUMA#Interface_and_data_structure_in_engine_core

We appreciate any of the comments from community, and sorry for the nag.

Best Regards,
Jason Liao

-Original Message-
From: Gilad Chaplik [mailto:gchap...@redhat.com] 
Sent: 2014年3月30日 22:43
To: Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); Shi, Xiao-Lei (Bruce, HP 
Servers-PSC-CQ); Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC)
Cc: Doron Fediuck; Vinod, Chegu; Martin Sivak
Subject: Numa feature entities patch

Hi all, 

I had some comments :-) IMO this patch [1] has maximum priority.
let's try to merge it ASAP, I will be very responsive to every new upload.

Thanks, 
Gilad.

[1] http://gerrit.ovirt.org/#/c/23702/
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel

Re: [Engine-devel] debug ovirt-engine

2014-03-26 Thread Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC) 
Hi,

Do you go through the wiki page OVirt Engine Development Environment
http://www.ovirt.org/OVirt_Engine_Development_Environment

you can debug with
Debug port is available via port 8787, to be used by Eclipse or any other 
debugger.

Best Regards,
Jason Liao

From: engine-devel-boun...@ovirt.org [mailto:engine-devel-boun...@ovirt.org] On 
Behalf Of 彭春洪
Sent: 2014年3月26日 16:13
To: engine-devel@ovirt.org
Subject: [Engine-devel] debug ovirt-engine

Hello,everybody.when I develop with ovirt-engine,I have meet some obstacles 
because of without the debug environment. So I really want to know how to debug 
the frontend and backend of ovirt-engine by break points or by step . Thank you 
for help!
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel

[Engine-devel] Help to review the design of NUMA feature for oVirt

2014-03-25 Thread Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC) 
Hi All,

Please help us to review the design of NUMA feature for oVirt,

Here are two design on wiki page
General Feature Design
http://www.ovirt.org/Features/NUMA_and_Virtual_NUMA

Detailed Design
http://www.ovirt.org/Features/Detailed_NUMA_and_Virtual_NUMA

Please feel free to have any comments, and we will reply as quickly as we could.


And we are in a tight schedule - we want it in to ovirt 3.5 release

Best Regards,
Jason Liao

___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel

[Engine-devel] Consider iowait add into usage CPU percentage

2014-03-11 Thread Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC) 
Hi All,

On engine core, The usage CPU percentage is calculated by %sys + %usr

class VdsBrokerObjectsBuilder
function updateVDSStatisticsData
vds.setCpuSys(AssignDoubleValue(xmlRpcStruct, VdsProperties.cpu_sys));
vds.setCpuUser(AssignDoubleValue(xmlRpcStruct, VdsProperties.cpu_user));
if (vds.getCpuSys() != null  vds.getCpuUser() != null) {
vds.setUsageCpuPercent((int) (vds.getCpuSys() + vds.getCpuUser()));
}

On vdsm, The %sys, %usr and %idle is calculated like below workflow,

class API
function getStats
decStats = self._cif._hostStats.get()
class clientIF
function __init__
   self._hostStats = sampling.HostStatsThread(log=log)
self._hostStats.start()
class HostStatsThread
function get
   hs0, hs1 = self._samples[0], self._samples[-1]
   ...
   jiffies = (hs1.totcpu.user - hs0.totcpu.user) % (2 ** 32)
stats['cpuUser'] = jiffies / interval / self._ncpus
jiffies = (hs1.totcpu.sys - hs0.totcpu.sys) % (2 ** 32)
stats['cpuSys'] = jiffies / interval / self._ncpus
stats['cpuIdle'] = max(0.0, 100.0 - stats['cpuUser'] - stats['cpuSys'])
class HostSample
function __init__
   self.totcpu = TotalCpuSample()
class TotalCpuSample
function __init__
 self.user, userNice, self.sys, self.idle = \
map(int, file('/proc/stat').readline().split()[1:5])
self.user += userNice

Question1: Why stats['cpuIdle'] do not use the sampling data totcpu.idle for 
calculating?

Question2: There is another data named iowait in /proc/stat, do we need to 
consider add this into usage CPU percentage for calculating?

Best Regards,
Jason Liao

___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel

Re: [Engine-devel] Consider iowait add into usage CPU percentage

2014-03-11 Thread Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC) 
Hi All,

Any feedback for this topic?
Add Doron and Martin in the mail list.

Best Regards,
Jason Liao

From: vdsm-devel-boun...@lists.fedorahosted.org 
[mailto:vdsm-devel-boun...@lists.fedorahosted.org] On Behalf Of Liao, Chuan 
(Jason Liao, HPservers-Core-OE-PSC)
Sent: 2014年3月11日 14:02
To: vdsm-devel; engine-devel@ovirt.org
Cc: Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Vinod, Chegu; Gilad 
Chaplik
Subject: [vdsm] Consider iowait add into usage CPU percentage

Hi All,

On engine core, The usage CPU percentage is calculated by %sys + %usr

class VdsBrokerObjectsBuilder
function updateVDSStatisticsData
vds.setCpuSys(AssignDoubleValue(xmlRpcStruct, VdsProperties.cpu_sys));
vds.setCpuUser(AssignDoubleValue(xmlRpcStruct, VdsProperties.cpu_user));
if (vds.getCpuSys() != null  vds.getCpuUser() != null) {
vds.setUsageCpuPercent((int) (vds.getCpuSys() + vds.getCpuUser()));
}

On vdsm, The %sys, %usr and %idle is calculated like below workflow,

class API
function getStats
decStats = self._cif._hostStats.get()
class clientIF
function __init__
   self._hostStats = sampling.HostStatsThread(log=log)
self._hostStats.start()
class HostStatsThread
function get
   hs0, hs1 = self._samples[0], self._samples[-1]
   …
   jiffies = (hs1.totcpu.user - hs0.totcpu.user) % (2 ** 32)
stats['cpuUser'] = jiffies / interval / self._ncpus
jiffies = (hs1.totcpu.sys - hs0.totcpu.sys) % (2 ** 32)
stats['cpuSys'] = jiffies / interval / self._ncpus
stats['cpuIdle'] = max(0.0, 100.0 - stats['cpuUser'] - stats['cpuSys'])
class HostSample
function __init__
   self.totcpu = TotalCpuSample()
class TotalCpuSample
function __init__
 self.user, userNice, self.sys, self.idle = \
map(int, file('/proc/stat').readline().split()[1:5])
self.user += userNice

Question1: Why stats[‘cpuIdle’] do not use the sampling data totcpu.idle for 
calculating?

Question2: There is another data named iowait in /proc/stat, do we need to 
consider add this into usage CPU percentage for calculating?

Best Regards,
Jason Liao

___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel

[Engine-devel] Supposed to add per CPU usage related infomation into engine core and vdsm

2014-03-10 Thread Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC) 
Hi All,

In order to support NUMA and guest NUMA feature in ovirt.
We need per NUMA node CPU usage or per CPU usage related information on engine 
core.

This information will be used for VM who have NUMA aware information and find 
the best Host to run it on with best performance.

Approach 1:

1.  Sample data in vdsm for each CPU stats. (sys, usr, iowait, idle)

2.  Calculate the each CPU usage information. (%sys, %usr, %iowait, %idle)

3.  Transport the usage data to engine core.

4.  Engine core merge per NUMA node CPU usage. (%sys, %usr, %iowait, %idle)

Approach 2:

1.  Sample data in vdsm for each CPU stats. (sys, usr, iowait, idle)

2.  Calculate the each CPU usage information. (%sys, %usr, %iowait, %idle)

3.  VDSM merge per NUMA node CPU usage. (%sys, %usr, %iowait, %idle)

4.  Transport the usage data to engine core.

Which one do you prefer, and why, or other solution.

Best Regards,
Jason Liao

___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel

[Engine-devel] which data structure is better

2014-02-20 Thread Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC) 
Hi All,

I am Jason Liao from HP who are in charge of NUMA and Virtual NUMA feature.

Now I have some concept about the host NUMA topology data structure on engine 
core

We have VDS, VDSDynamic, VDSStatic, VdsStatistics object on engine core.

And we have NUMA topology information:
ListNumaNode numaNodeList
 NumaNode
  String ID # 
update from GetCapabilitiesVDSCommand
   ListString cpuList   # update from 
GetCapabilitiesVDSCommand
Int totalMem# update from 
GetCapabilitiesVDSCommand
   Int freeMem  # 
update from GetStatsVDSCommand


A.  Add this data structure into VDSDynamic
We should change the GetStatsVDSCommand update the VDSDynamic data.

B.  Add this data structure into VDS, and build the data structure from 
VDSDynamic, VdsStatistics



I prefer B. does anybody have some comments?

Best Regards,
Jason Liao

___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel

[Engine-devel] oVirt NUMA and Virual NUMA feature wiki page and detailed design page update

2014-02-14 Thread Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC) 
Hi All,

I am Jason Liao from HP who are in charge of NUMA and Virtual NUMA feature 
design.

Now I have updated the page of NUMA and Virual NUMA feature wiki page and 
detailed design page

http://www.ovirt.org/Features/NUMA_and_Virtual_NUMA

http://www.ovirt.org/Features/Detailed_NUMA_and_Virtual_NUMA

It is not the finally version, if anyone have some comment, that would be 
useful for us.

Best Regards,
Jason Liao

___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel