[Xen-devel] [TESTDAY] Test report - xl sched-rtds
* Hardware: CPU: Intel Core2 Quad Q9400 Total Memory: 2791088 kB * Software: Ubuntu 14.04 Linux kernel: 3.13.0-68 * Guest operating systems: Ubuntu 14.04 (PV) * Functionality tested: xl sched-rtds (for set/get per-VCPU parameters) * Comments: All examples about "xl sched-rtds" in xl mannual page <http://xenbits.xen.org/docs/unstable/man/xl.1.html#DOMAIN-SUBCOMMANDS> have been tested, and all run successfully. If users type in wrong parameters (e.g., budget > period), the error/warnning messages are returned correctly as well. Chong Li -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v1] libxc: fix uninitialized variable when changing rtds scheduling parameters
Commit 046c2b503a89d21b41e4d555a9f75d02af00dbc6 introduces a build
failure: in some cases (e.g., num_vcpus <=0),
xc_sched_rtds_vcpu_get/set returns an uninitialized variable.
Fix it.
Signed-off-by: Chong Li
---
CC:
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxc/xc_rt.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/libxc/xc_rt.c b/tools/libxc/xc_rt.c
index 221d17f..77aaa36 100644
--- a/tools/libxc/xc_rt.c
+++ b/tools/libxc/xc_rt.c
@@ -68,7 +68,7 @@ int xc_sched_rtds_vcpu_set(xc_interface *xch,
struct xen_domctl_schedparam_vcpu *vcpus,
uint32_t num_vcpus)
{
-int rc;
+int rc = 0;
unsigned processed = 0;
DECLARE_DOMCTL;
DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
@@ -102,7 +102,7 @@ int xc_sched_rtds_vcpu_get(xc_interface *xch,
struct xen_domctl_schedparam_vcpu *vcpus,
uint32_t num_vcpus)
{
-int rc;
+int rc = 0;
unsigned processed = 0;
DECLARE_DOMCTL;
DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
--
1.9.1
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v9 for Xen 4.7 3/4] libxl: enable per-VCPU parameter for RTDS
On Wed, Apr 6, 2016 at 2:30 PM, Wei Liu wrote:
> On Wed, Apr 06, 2016 at 02:20:55PM -0500, Chong Li wrote:
>> On Wed, Apr 6, 2016 at 1:54 PM, Andrew Cooper
>> wrote:
>> > On 06/04/16 17:41, Chong Li wrote:
>> >> On Wed, Apr 6, 2016 at 11:36 AM, Dario Faggioli
>> >> wrote:
>> >>> On Wed, 2016-04-06 at 16:38 +0100, Ian Jackson wrote:
>> >>>> Ian Jackson writes ("Re: [PATCH v9 for Xen 4.7 3/4] libxl: enable
>> >>>> per-VCPU parameter for RTDS"):
>> >>>>> Dario points out on irc that perhaps the problem is that I didn't
>> >>>>> apply 2/4. I wasn't CC'd on 2/4, so I foolishly assumed it was a
>> >>>>> hypervisor patch (and the HV parts are already in tree).
>> >>>>>
>> >>>>> I will check my view of the xen-devel list.
>> >>>> Indeed. With 2/4 it builds. 4/4 was also not CC'd to me. I used a
>> >>>> copy from the list.
>> >>>>
>> >>>> I have pushed all four.
>> >>>>
>> >>> Thanks Ian!
>> >>>
>> >>> So, Chong, clearly, the build failure was not your fault (as there is
>> >>> no actual build failure), but please, always double check (even when
>> >>> sending new versions of a series) that the appropriate maintainers are
>> >>> Cc-ed... This would help limiting problems like this one we've seen
>> >>> here.
>> >>>
>> >> Yes, I'll.
>> >>
>> >> Thanks for your help on this.
>> >> Chong
>> >
>> > Yet another build failure on CentOS.
>> >
>> > xc_rt.c: In function 'xc_sched_rtds_vcpu_set':
>> > xc_rt.c:71:9: error: 'rc' may be used uninitialized in this function
>> > [-Werror=maybe-uninitialized]
>> > int rc;
>> > ^
>> > xc_rt.c: In function 'xc_sched_rtds_vcpu_get':
>> > xc_rt.c:105:9: error: 'rc' may be used uninitialized in this function
>> > [-Werror=maybe-uninitialized]
>> > int rc;
>> > ^
>> > cc1: all warnings being treated as errors
>> >
>> > In both cases, if your while loop doesn't execute (i.e. the user passes
>> > num_vcpus = 0), rc is genuinely uninialised when used at the end of the
>> > function.
>> >
>> > ~Andrew
>>
>> I see. I can do a sanity check on num_vcpus before the while loop.
>>
>
> Not sure what kind of sanity check you were thinking about. But you can
> just set rc = 0 at the beginning of each function. That semantics should
> be sensible enough. What do you think?
Yes, I can do this change. But didn't you or Dario say that rc should not be
initialized at the beginning of a function?
>
>> Do I have to re-send the whole patch series? Or maybe just something
>> like a bug fix patch?
>>
>
> Please send a patch on top of staging branch. This series has already
> been committed.
>
Sure.
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v9 for Xen 4.7 3/4] libxl: enable per-VCPU parameter for RTDS
On Wed, Apr 6, 2016 at 1:54 PM, Andrew Cooper wrote:
> On 06/04/16 17:41, Chong Li wrote:
>> On Wed, Apr 6, 2016 at 11:36 AM, Dario Faggioli
>> wrote:
>>> On Wed, 2016-04-06 at 16:38 +0100, Ian Jackson wrote:
>>>> Ian Jackson writes ("Re: [PATCH v9 for Xen 4.7 3/4] libxl: enable
>>>> per-VCPU parameter for RTDS"):
>>>>> Dario points out on irc that perhaps the problem is that I didn't
>>>>> apply 2/4. I wasn't CC'd on 2/4, so I foolishly assumed it was a
>>>>> hypervisor patch (and the HV parts are already in tree).
>>>>>
>>>>> I will check my view of the xen-devel list.
>>>> Indeed. With 2/4 it builds. 4/4 was also not CC'd to me. I used a
>>>> copy from the list.
>>>>
>>>> I have pushed all four.
>>>>
>>> Thanks Ian!
>>>
>>> So, Chong, clearly, the build failure was not your fault (as there is
>>> no actual build failure), but please, always double check (even when
>>> sending new versions of a series) that the appropriate maintainers are
>>> Cc-ed... This would help limiting problems like this one we've seen
>>> here.
>>>
>> Yes, I'll.
>>
>> Thanks for your help on this.
>> Chong
>
> Yet another build failure on CentOS.
>
> xc_rt.c: In function 'xc_sched_rtds_vcpu_set':
> xc_rt.c:71:9: error: 'rc' may be used uninitialized in this function
> [-Werror=maybe-uninitialized]
> int rc;
> ^
> xc_rt.c: In function 'xc_sched_rtds_vcpu_get':
> xc_rt.c:105:9: error: 'rc' may be used uninitialized in this function
> [-Werror=maybe-uninitialized]
> int rc;
> ^
> cc1: all warnings being treated as errors
>
> In both cases, if your while loop doesn't execute (i.e. the user passes
> num_vcpus = 0), rc is genuinely uninialised when used at the end of the
> function.
>
> ~Andrew
I see. I can do a sanity check on num_vcpus before the while loop.
Do I have to re-send the whole patch series? Or maybe just something
like a bug fix patch?
Chong
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v9 for Xen 4.7 3/4] libxl: enable per-VCPU parameter for RTDS
On Wed, Apr 6, 2016 at 11:36 AM, Dario Faggioli
wrote:
> On Wed, 2016-04-06 at 16:38 +0100, Ian Jackson wrote:
>> Ian Jackson writes ("Re: [PATCH v9 for Xen 4.7 3/4] libxl: enable
>> per-VCPU parameter for RTDS"):
>> >
>> > Dario points out on irc that perhaps the problem is that I didn't
>> > apply 2/4. I wasn't CC'd on 2/4, so I foolishly assumed it was a
>> > hypervisor patch (and the HV parts are already in tree).
>> >
>> > I will check my view of the xen-devel list.
>> Indeed. With 2/4 it builds. 4/4 was also not CC'd to me. I used a
>> copy from the list.
>>
>> I have pushed all four.
>>
> Thanks Ian!
>
> So, Chong, clearly, the build failure was not your fault (as there is
> no actual build failure), but please, always double check (even when
> sending new versions of a series) that the appropriate maintainers are
> Cc-ed... This would help limiting problems like this one we've seen
> here.
>
Yes, I'll.
Thanks for your help on this.
Chong
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v4] xen: sched: fix deadlock when changing scheduling parameters
Commit f7b87b0745b4 ("enable per-VCPU parameter for RTDS") introduced
a bug: it made it possible, in Credit and Credit2, when doing domain
or vcpu parameters' manipulation, to leave the hypervisor with a
spinlock held and interrupts disabled.
Fix it.
Signed-off-by: Chong Li
Acked-by: Dario Faggioli
---
CC:
CC:
CC:
CC:
CC:
CC:
---
xen/common/sched_credit.c | 6 --
xen/common/sched_credit2.c | 6 --
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
index e5d15d8..4c4927f 100644
--- a/xen/common/sched_credit.c
+++ b/xen/common/sched_credit.c
@@ -1075,6 +1075,7 @@ csched_dom_cntl(
struct csched_dom * const sdom = CSCHED_DOM(d);
struct csched_private *prv = CSCHED_PRIV(ops);
unsigned long flags;
+int rc = 0;
/* Protect both get and put branches with the pluggable scheduler
* lock. Runq lock not needed anywhere in here. */
@@ -1101,12 +1102,13 @@ csched_dom_cntl(
sdom->cap = op->u.credit.cap;
break;
default:
-return -EINVAL;
+rc = -EINVAL;
+break;
}
spin_unlock_irqrestore(&prv->lock, flags);
-return 0;
+return rc;
}
static inline void
diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index d48ed5a..b8c8e40 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -1416,6 +1416,7 @@ csched2_dom_cntl(
struct csched2_dom * const sdom = CSCHED2_DOM(d);
struct csched2_private *prv = CSCHED2_PRIV(ops);
unsigned long flags;
+int rc = 0;
/* Must hold csched2_priv lock to read and update sdom,
* runq lock to update csvcs. */
@@ -1457,12 +1458,13 @@ csched2_dom_cntl(
}
break;
default:
-return -EINVAL;
+rc = -EINVAL;
+break;
}
spin_unlock_irqrestore(&prv->lock, flags);
-return 0;
+return rc;
}
static void *
--
1.9.1
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[Xen-devel] [PATCH] xen: enable per-VCPU parameter for RTDS
Commit f7b87b0745b4 ("enable per-VCPU parameter for RTDS") introduced
a bug: it made it possible, in Credit and Credit2, when doing domain
or vcpu parameters' manipulation, to leave the hypervisor with a
spinlock held and interrupts disabled.
Fix it.
Signed-off-by: Chong Li
Acked-by: Dario Faggioli
---
CC:
CC:
CC:
CC:
CC:
CC:
---
xen/common/sched_credit.c | 6 --
xen/common/sched_credit2.c | 6 --
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
index e5d15d8..4c4927f 100644
--- a/xen/common/sched_credit.c
+++ b/xen/common/sched_credit.c
@@ -1075,6 +1075,7 @@ csched_dom_cntl(
struct csched_dom * const sdom = CSCHED_DOM(d);
struct csched_private *prv = CSCHED_PRIV(ops);
unsigned long flags;
+int rc = 0;
/* Protect both get and put branches with the pluggable scheduler
* lock. Runq lock not needed anywhere in here. */
@@ -1101,12 +1102,13 @@ csched_dom_cntl(
sdom->cap = op->u.credit.cap;
break;
default:
-return -EINVAL;
+rc = -EINVAL;
+break;
}
spin_unlock_irqrestore(&prv->lock, flags);
-return 0;
+return rc;
}
static inline void
diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index d48ed5a..b8c8e40 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -1416,6 +1416,7 @@ csched2_dom_cntl(
struct csched2_dom * const sdom = CSCHED2_DOM(d);
struct csched2_private *prv = CSCHED2_PRIV(ops);
unsigned long flags;
+int rc = 0;
/* Must hold csched2_priv lock to read and update sdom,
* runq lock to update csvcs. */
@@ -1457,12 +1458,13 @@ csched2_dom_cntl(
}
break;
default:
-return -EINVAL;
+rc = -EINVAL;
+break;
}
spin_unlock_irqrestore(&prv->lock, flags);
-return 0;
+return rc;
}
static void *
--
1.9.1
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[Xen-devel] [PATCH] xen: enable per-VCPU parameter for RTDS
From: Chong-Li
Commit f7b87b0745b4 ("enable per-VCPU parameter for RTDS") introduced
a bug: it made it possible, in Credit and Credit2, when doing domain
or vcpu parameters' manipulation, to leave the hypervisor with a
spinlock held.
Fix it.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
Acked-by: Dario Faggioli
---
CC:
CC:
CC:
CC:
CC:
CC:
---
xen/common/sched_credit.c | 1 +
xen/common/sched_credit2.c | 1 +
2 files changed, 2 insertions(+)
diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
index e5d15d8..fa6b7f0 100644
--- a/xen/common/sched_credit.c
+++ b/xen/common/sched_credit.c
@@ -1101,6 +1101,7 @@ csched_dom_cntl(
sdom->cap = op->u.credit.cap;
break;
default:
+spin_unlock_irqrestore(&prv->lock, flags);
return -EINVAL;
}
diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index d48ed5a..cf444c9 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -1457,6 +1457,7 @@ csched2_dom_cntl(
}
break;
default:
+spin_unlock_irqrestore(&prv->lock, flags);
return -EINVAL;
}
--
1.9.1
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[Xen-devel] [PATCH] xen: enable per-VCPU parameter for RTDS
From: Chong-Li Fix a bug in sched_credit.c and sched_credit2.c: in the default case of csched_dom_cntl and csched2_dom_cntl, function returns without unlocking prv->lock. Signed-off-by: Chong Li Signed-off-by: Meng Xu Signed-off-by: Sisu Xi --- CC: CC: CC: CC: CC: CC: --- xen/common/sched_credit.c | 1 + xen/common/sched_credit2.c | 1 + 2 files changed, 2 insertions(+) diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c index e5d15d8..fa6b7f0 100644 --- a/xen/common/sched_credit.c +++ b/xen/common/sched_credit.c @@ -1101,6 +1101,7 @@ csched_dom_cntl( sdom->cap = op->u.credit.cap; break; default: +spin_unlock_irqrestore(&prv->lock, flags); return -EINVAL; } diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c index d48ed5a..cf444c9 100644 --- a/xen/common/sched_credit2.c +++ b/xen/common/sched_credit2.c @@ -1457,6 +1457,7 @@ csched2_dom_cntl( } break; default: +spin_unlock_irqrestore(&prv->lock, flags); return -EINVAL; } -- 1.9.1 ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [hypervisor deadlock] Re: [PATCH v9 for Xen 4.7 1/4] xen: enable per-VCPU parameter for RTDS
On Mon, Apr 4, 2016 at 11:47 AM, Wei Liu wrote:
> On Mon, Apr 04, 2016 at 06:32:48PM +0200, Dario Faggioli wrote:
>> On Mon, 2016-04-04 at 17:05 +0100, George Dunlap wrote:
>> > On 04/04/16 16:58, Chong Li wrote:
>> > > On Mon, Apr 4, 2016 at 10:14 AM, Andrew Cooper
>> > > wrote:
>> > > > On 01/04/16 05:59, Chong Li wrote:
>> > > > >
>> > > > > --- a/xen/common/sched_credit2.c
>> > > > > +++ b/xen/common/sched_credit2.c
>> > > > > @@ -1421,14 +1421,12 @@ csched2_dom_cntl(
>> > > > > * runq lock to update csvcs. */
>> > > > > spin_lock_irqsave(&prv->lock, flags);
>> > > > >
>> > > > > -if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
>> > > > > +switch ( op->cmd )
>> > > > > {
>> > > > > +case XEN_DOMCTL_SCHEDOP_getinfo:
>> > > > > op->u.credit2.weight = sdom->weight;
>> > > > > -}
>> > > > > -else
>> > > > > -{
>> > > > > -ASSERT(op->cmd == XEN_DOMCTL_SCHEDOP_putinfo);
>> > > > > -
>> > > > > +break;
>> > > > > +case XEN_DOMCTL_SCHEDOP_putinfo:
>> > > > > if ( op->u.credit2.weight != 0 )
>> > > > > {
>> > > > > struct vcpu *v;
>> > > > > @@ -1457,6 +1455,9 @@ csched2_dom_cntl(
>> > > > > vcpu_schedule_unlock(lock, svc->vcpu);
>> > > > > }
>> > > > > }
>> > > > > +break;
>> > > > > +default:
>> > > > > +return -EINVAL;
>> > > > As does this.
>> > > >
>> > > > Please submit a bugfix ASAP. This will become a security
>> > > > vulnerability
>> > > > if Xen 4.7 is shipped without it being fixed.
>> > > >
>> > > > >
>> > > > > }
>> > > > >
>> > > > > spin_unlock_irqrestore(&prv->lock, flags);
>> > > Thanks for pointing this out.
>> > >
>> > > Dario, do you want to include this bugfix in your cleanup patch, or
>> > > let me submit this?
>> > If you're around and can test it, it's probably better if you can
>> > send a
>> > patch right a way.
>> >
>> Exactly. In fact:
>> - we don't fold bugfixes in clanups,
>> - I think I mentioned wanting to cleanup some code duplication in
>>libxl, this is in xen,
>> - cleanups are delayed to 4.8, while this must be fixed before
>>release (or the patch/the whole feature be reverted).
>>
>> So, if you can't work out a fix today or, at most, tomorrow, let me
>> know and I'll do it myself.
>>
>
> Yes, please fix this by tomorrow. I think it is quite straightforward to
> fix anyway.
>
> Wei.
>
Will do.
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [hypervisor deadlock] Re: [PATCH v9 for Xen 4.7 1/4] xen: enable per-VCPU parameter for RTDS
On Mon, Apr 4, 2016 at 10:14 AM, Andrew Cooper
wrote:
> On 01/04/16 05:59, Chong Li wrote:
>> diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
>> index 305889a..e5d15d8 100644
>> --- a/xen/common/sched_credit.c
>> +++ b/xen/common/sched_credit.c
>> @@ -1080,15 +1080,13 @@ csched_dom_cntl(
>> * lock. Runq lock not needed anywhere in here. */
>> spin_lock_irqsave(&prv->lock, flags);
>>
>> -if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
>> +switch ( op->cmd )
>> {
>> +case XEN_DOMCTL_SCHEDOP_getinfo:
>> op->u.credit.weight = sdom->weight;
>> op->u.credit.cap = sdom->cap;
>> -}
>> -else
>> -{
>> -ASSERT(op->cmd == XEN_DOMCTL_SCHEDOP_putinfo);
>> -
>> +break;
>> +case XEN_DOMCTL_SCHEDOP_putinfo:
>> if ( op->u.credit.weight != 0 )
>> {
>> if ( !list_empty(&sdom->active_sdom_elem) )
>> @@ -1101,7 +1099,9 @@ csched_dom_cntl(
>>
>> if ( op->u.credit.cap != (uint16_t)~0U )
>> sdom->cap = op->u.credit.cap;
>> -
>> +break;
>> +default:
>> +return -EINVAL;
>
> This path returns without unlocking prv->lock.
>
>> }
>>
>> spin_unlock_irqrestore(&prv->lock, flags);
>> diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
>> index 7ddad38..d48ed5a 100644
>> --- a/xen/common/sched_credit2.c
>> +++ b/xen/common/sched_credit2.c
>> @@ -1421,14 +1421,12 @@ csched2_dom_cntl(
>> * runq lock to update csvcs. */
>> spin_lock_irqsave(&prv->lock, flags);
>>
>> -if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
>> +switch ( op->cmd )
>> {
>> +case XEN_DOMCTL_SCHEDOP_getinfo:
>> op->u.credit2.weight = sdom->weight;
>> -}
>> -else
>> -{
>> -ASSERT(op->cmd == XEN_DOMCTL_SCHEDOP_putinfo);
>> -
>> +break;
>> +case XEN_DOMCTL_SCHEDOP_putinfo:
>> if ( op->u.credit2.weight != 0 )
>> {
>> struct vcpu *v;
>> @@ -1457,6 +1455,9 @@ csched2_dom_cntl(
>> vcpu_schedule_unlock(lock, svc->vcpu);
>> }
>> }
>> +break;
>> +default:
>> +return -EINVAL;
>
> As does this.
>
> Please submit a bugfix ASAP. This will become a security vulnerability
> if Xen 4.7 is shipped without it being fixed.
>
>> }
>>
>> spin_unlock_irqrestore(&prv->lock, flags);
>
Thanks for pointing this out.
Dario, do you want to include this bugfix in your cleanup patch, or
let me submit this?
Chong
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v9 for Xen 4.7 4/4] xl: enable per-VCPU parameter for RTDS
Change main_sched_rtds and related output functions to support
per-VCPU settings.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
Acked-by: Wei Liu
---
Changes on PATCH v8:
1) Improve the example in xl.pod.1
2) Add sched_rtds_vcpu_output_all() and sched_vcpu_get_all()
to output all vcpus of a domain
3) Fix some coding style issues
Changes on PATCH v7:
1) Add example to xl.pod.1
Changes on PATCH v6:
1) More explain in xl.pod.1 and cmdtable.c
2) Resolve some coding sytle issues
Changes on PATCH v5:
1) Add sched_vcpu_set_all() for the cases that all vcpus of a
domain need to be changed together.
Changes on PATCH v4:
1) Coding style changes
Changes on PATCH v3:
1) Support commands, e.g., "xl sched-rtds -d vm1" to output the
default scheduling parameters
Changes on PATCH v2:
1) Remove per-domain output functions for RTDS scheduler.
2) Users now use '-v all' to specify all VCPUs.
3) Support outputting a subset of the parameters of the VCPUs
of a specific domain.
4) When setting all VCPUs with the same parameters (by only one
command), no per-domain function is invoked.
CC:
CC:
CC:
CC:
CC:
CC:
---
docs/man/xl.pod.1 | 67 +
tools/libxl/xl_cmdimpl.c | 365 +-
tools/libxl/xl_cmdtable.c | 4 +-
3 files changed, 400 insertions(+), 36 deletions(-)
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 4d4b333..9e89e0a 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -1056,6 +1056,10 @@ B
Specify domain for which scheduler parameters are to be modified or retrieved.
Mandatory for modifying scheduler parameters.
+=item B<-v VCPUID/all>, B<--vcpuid=VCPUID/all>
+
+Specify vcpu for which scheduler parameters are to be modified or retrieved.
+
=item B<-p PERIOD>, B<--period=PERIOD>
Period of time, in microseconds, over which to replenish the budget.
@@ -1071,6 +1075,69 @@ Restrict output to domains in the specified cpupool.
=back
+B
+
+=over 4
+
+1) Use B<-v all> to see the budget and period of all the VCPUs of
+all the domains:
+
+xl sched-rtds -v all
+Cpupool Pool-0: sched=RTDS
+NameID VCPUPeriodBudget
+Domain-0 00 1 4000
+vm1 10 300 150
+vm1 11 400 200
+vm1 12 1 4000
+vm1 13 1000 500
+vm2 20 1 4000
+vm2 21 1 4000
+
+Without any arguments, it will output the default scheduing
+parameters for each domain:
+
+xl sched-rtds
+Cpupool Pool-0: sched=RTDS
+NameIDPeriodBudget
+Domain-0 0 1 4000
+vm1 1 1 4000
+vm2 2 1 4000
+
+
+2) Use, for instance B<-d vm1, -v all> to see the budget and
+period of all VCPUs of a specific domain (B):
+
+xl sched-rtds -d vm1 -v all
+NameID VCPUPeriodBudget
+vm1 10 300 150
+vm1 11 400 200
+vm1 12 1 4000
+vm1 13 1000 500
+
+To see the parameters of a subset of the VCPUs of a domain, use:
+
+xl sched-rtds -d vm1 -v 0 -v 3
+NameID VCPUPeriodBudget
+vm1 10 300 150
+vm1 13 1000 500
+
+If no B<-v> is speficified, the default scheduling parameter for the
+domain are shown:
+
+xl sched-rtds -d vm1
+NameIDPeriodBudget
+vm1 1 1 4000
+
+
+3) Users can set the budget and period of multiple VCPUs of a
+specific domain with only one command,
+e.g., "xl sched-rtds -d vm1 -v 0 -p 100 -b 50 -v 3 -p 300 -b 150".
+
+To change the parameters of all the VCPUs of a domain, use B<-v all>,
+e.g., "xl sched-rtds -d vm1 -v all -p 500 -b 250".
+
+=back
+
=back
=head1 CPUPOOLS COMMANDS
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 7750995..dc77689 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -6159,6 +6159,72 @@ static int sched_domain_set(int domid, const
libxl_domain_sched_params *scinfo)
return 0;
}
+static int sched_vcpu_get(libxl_scheduler sched, int domid,
+ libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_get(ctx, domid, scinfo);
+if (rc) {
+fprintf(stderr, "libxl_vcpu_sched_params_get failed.\n"
[Xen-devel] [PATCH v9 for Xen 4.7 2/4] libxc: enable per-VCPU parameter for RTDS
Add xc_sched_rtds_vcpu_get/set functions to interact with
Xen to get/set a domain's per-VCPU parameters.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
Acked-by: Wei Liu
Reviewed-by: Dario Faggioli
---
Changes on PATCH v7:
1) rc in xc_sched_rtds_vcpu_set() should not be initialized to 0. It
is fixed.
Changes on PATCH v6:
1) Resolve some coding sytle issues
Changes on PATCH v5:
1) In xc_sched_rtds_vcpu_get/set, re-issueing the hypercall
if it is preempted.
Changes on PATCH v4:
1) Minor modifications on the function parameters.
Changes on PATCH v2:
1) Minor modifications due to the change of struct xen_domctl_scheduler_op.
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxc/include/xenctrl.h | 8 +
tools/libxc/xc_rt.c | 68 +++
2 files changed, 76 insertions(+)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index a9e4dc1..f430ef9 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -878,6 +878,14 @@ int xc_sched_rtds_domain_set(xc_interface *xch,
int xc_sched_rtds_domain_get(xc_interface *xch,
uint32_t domid,
struct xen_domctl_sched_rtds *sdom);
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus);
+int xc_sched_rtds_vcpu_get(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus);
int
xc_sched_arinc653_schedule_set(
diff --git a/tools/libxc/xc_rt.c b/tools/libxc/xc_rt.c
index d59e5ce..221d17f 100644
--- a/tools/libxc/xc_rt.c
+++ b/tools/libxc/xc_rt.c
@@ -62,3 +62,71 @@ int xc_sched_rtds_domain_get(xc_interface *xch,
return rc;
}
+
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus)
+{
+int rc;
+unsigned processed = 0;
+DECLARE_DOMCTL;
+DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
+ XC_HYPERCALL_BUFFER_BOUNCE_IN);
+
+if ( xc_hypercall_bounce_pre(xch, vcpus) )
+return -1;
+
+domctl.cmd = XEN_DOMCTL_scheduler_op;
+domctl.domain = (domid_t) domid;
+domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
+domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_putvcpuinfo;
+
+while ( processed < num_vcpus )
+{
+domctl.u.scheduler_op.u.v.nr_vcpus = num_vcpus - processed;
+set_xen_guest_handle_offset(domctl.u.scheduler_op.u.v.vcpus, vcpus,
+processed);
+if ( (rc = do_domctl(xch, &domctl)) != 0 )
+break;
+processed += domctl.u.scheduler_op.u.v.nr_vcpus;
+}
+
+xc_hypercall_bounce_post(xch, vcpus);
+
+return rc;
+}
+
+int xc_sched_rtds_vcpu_get(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus)
+{
+int rc;
+unsigned processed = 0;
+DECLARE_DOMCTL;
+DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
+ XC_HYPERCALL_BUFFER_BOUNCE_BOTH);
+
+if ( xc_hypercall_bounce_pre(xch, vcpus) )
+return -1;
+
+domctl.cmd = XEN_DOMCTL_scheduler_op;
+domctl.domain = (domid_t) domid;
+domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
+domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_getvcpuinfo;
+
+while ( processed < num_vcpus )
+{
+domctl.u.scheduler_op.u.v.nr_vcpus = num_vcpus - processed;
+set_xen_guest_handle_offset(domctl.u.scheduler_op.u.v.vcpus, vcpus,
+processed);
+if ( (rc = do_domctl(xch, &domctl)) != 0 )
+break;
+processed += domctl.u.scheduler_op.u.v.nr_vcpus;
+}
+
+xc_hypercall_bounce_post(xch, vcpus);
+
+return rc;
+}
--
1.9.1
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v9 for Xen 4.7 3/4] libxl: enable per-VCPU parameter for RTDS
Add libxl_vcpu_sched_params_get/set and sched_rtds_vcpu_get/set
functions to support per-VCPU settings.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
Acked-by: Wei Liu
Reviewed-by: Dario Faggioli
---
Changes on PATCH v8:
1) Add libxl_vcpu_sched_params_get_all() and
sched_rtds_vcpu_get_all() to output all vcpus of a domain.
Changes on PATCH v7:
1) Fix a mistake in sched_rtds_domain_set()
Changes on PATCH v6:
1) Resolve some coding style issues
2) Change sched_rtds_validate_params()
3) Small changes for sched_rtds_vcpus_params_set(all) functions
Changes on PATCH v5:
1) Add a seperate function, sched_rtds_vcpus_params_set_all(), to set
the parameters of all vcpus of a domain.
2) Add libxl_vcpu_sched_params_set_all() to invoke the above function.
3) Coding style changes. (I didn't find the indentation rules for function
calls with long parameters (still 4 spaces?), so I just imitated the
indentation style of some existing functions)
Changes on PATCH v4:
1) Coding style changes
Changes on PATCH v3:
1) Add sanity check on vcpuid
2) Add comments on per-domain and per-vcpu functions for libxl
users
Changes on PATCH v2:
1) New data structure (libxl_vcpu_sched_params and libxl_sched_params)
to help per-VCPU settings.
2) sched_rtds_vcpu_get now can return a random subset of the parameters
of the VCPUs of a specific domain.
CC:
CC:
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxl/libxl.c | 386 +---
tools/libxl/libxl.h | 41 +
tools/libxl/libxl_types.idl | 14 ++
3 files changed, 420 insertions(+), 21 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index ac4d1f6..7d6ea91 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -5779,6 +5779,236 @@ static int sched_credit2_domain_set(libxl__gc *gc,
uint32_t domid,
return 0;
}
+static int sched_rtds_validate_params(libxl__gc *gc, int period, int budget)
+{
+int rc;
+
+if (period < 1) {
+LOG(ERROR, "Invalid VCPU period of %d (it should be >= 1)", period);
+rc = ERROR_INVAL;
+goto out;
+}
+
+if (budget < 1) {
+LOG(ERROR, "Invalid VCPU budget of %d (it should be >= 1)", budget);
+rc = ERROR_INVAL;
+goto out;
+}
+
+if (budget > period) {
+LOG(ERROR, "VCPU budget must be smaller than or equal to period, "
+ "but %d > %d", budget, period);
+rc = ERROR_INVAL;
+goto out;
+}
+rc = 0;
+out:
+return rc;
+}
+
+/* Get the RTDS scheduling parameters of vcpu(s) */
+static int sched_rtds_vcpu_get(libxl__gc *gc, uint32_t domid,
+ libxl_vcpu_sched_params *scinfo)
+{
+uint32_t num_vcpus;
+int i, r, rc;
+xc_dominfo_t info;
+struct xen_domctl_schedparam_vcpu *vcpus;
+
+r = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (r < 0) {
+LOGE(ERROR, "getting domain info");
+rc = ERROR_FAIL;
+goto out;
+}
+
+if (scinfo->num_vcpus <= 0) {
+rc = ERROR_INVAL;
+goto out;
+} else {
+num_vcpus = scinfo->num_vcpus;
+GCNEW_ARRAY(vcpus, num_vcpus);
+for (i = 0; i < num_vcpus; i++) {
+if (scinfo->vcpus[i].vcpuid < 0 ||
+scinfo->vcpus[i].vcpuid > info.max_vcpu_id) {
+LOG(ERROR, "VCPU index is out of range, "
+ "valid values are within range from 0 to %d",
+ info.max_vcpu_id);
+rc = ERROR_INVAL;
+goto out;
+}
+vcpus[i].vcpuid = scinfo->vcpus[i].vcpuid;
+}
+}
+
+r = xc_sched_rtds_vcpu_get(CTX->xch, domid, vcpus, num_vcpus);
+if (r != 0) {
+LOGE(ERROR, "getting vcpu sched rtds");
+rc = ERROR_FAIL;
+goto out;
+}
+scinfo->sched = LIBXL_SCHEDULER_RTDS;
+for (i = 0; i < num_vcpus; i++) {
+scinfo->vcpus[i].period = vcpus[i].u.rtds.period;
+scinfo->vcpus[i].budget = vcpus[i].u.rtds.budget;
+scinfo->vcpus[i].vcpuid = vcpus[i].vcpuid;
+}
+rc = 0;
+out:
+return rc;
+}
+
+/* Get the RTDS scheduling parameters of all vcpus of a domain */
+static int sched_rtds_vcpu_get_all(libxl__gc *gc, uint32_t domid,
+ libxl_vcpu_sched_params *scinfo)
+{
+uint32_t num_vcpus;
+int i, r, rc;
+xc_dominfo_t info;
+struct xen_domctl_schedparam_vcpu *vcpus;
+
+r = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (r < 0) {
+LOGE(ERROR, "getting domain info");
+rc = ERROR_FAIL;
+goto out;
+}
+
+if (scinfo->num_vcpus > 0) {
+rc = ERROR_INVAL;
+goto out;
+} else {
+num_vcpus = info.max_vcpu_id + 1;
+
[Xen-devel] [PATCH v9 for Xen 4.7 1/4] xen: enable per-VCPU parameter for RTDS
Add XEN_DOMCTL_SCHEDOP_getvcpuinfo and _putvcpuinfo hypercalls
to independently get and set the scheduling parameters of each
vCPU of a domain.
Also fix a bug in XEN_DOMCTL_SCHEDOP_getinfo, where PERIOD and
BUDGET are not divided by MICROSECS(1) before being retruned
to the caller.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v8:
1) Move BUG fix description before SoB.
2) Remove the unuseful padding in struct xen_domctl_schedparam_vcpu.
Changes on PATCH v7:
1) A bug in case XEN_DOMCTL_SCHEDOP_getinfo (in Xen 4.6) is fixed:
The default PERIOD or BUDGET should be divided by MICROSECS(1),
before returned to upper caller.
2) In the *cntl() functions in sched_credit.c and sched_credit2.c, remove
two unhandled cases in "switch".
Changes on PATCH v6:
1) Add explain for nr_vcpus in struct xen_domctl_scheduler_op, because it is
used
in both IN and OUT ways.
2) Remove the check and warning for vcpu settings with low budget or budget.
Making this
feature "per-domain" or "per-operation" is one of the future work.
3) In the *cntl() functions in sched_credit.c and sched_credit2.c, change the
"if-else"
structure to "switch" structure.
4) In rt_dom_cntl(), use copy_to_guest* instead of __copy_to_guest*, because
the latter one
requires lock protection.
Changes on PATCH v5:
1) When processing XEN_DOMCTL_SCHEDOP_get/putvcpuinfo, we do
preemption check in a similar way to XEN_SYSCTL_pcitopoinfo
Changes on PATCH v4:
1) Add uint32_t vcpu_index to struct xen_domctl_scheduler_op.
When processing XEN_DOMCTL_SCHEDOP_get/putvcpuinfo, we call
hypercall_preemption_check in case the current hypercall lasts
too long. If we decide to preempt the current hypercall, we record
the index of the most-recent finished vcpu into the vcpu_index of
struct xen_domctl_scheduler_op. So when we resume the hypercall after
preemption, we start processing from the posion specified by vcpu_index,
and don't need to repeat the work that has already been done in the
hypercall before the preemption.
(This design is based on the do_grant_table_op() in grant_table.c)
2) Coding style changes
Changes on PATCH v3:
1) Remove struct xen_domctl_schedparam_t.
2) Change struct xen_domctl_scheduler_op.
3) Check if period/budget is within a validated range
Changes on PATCH v2:
1) Change struct xen_domctl_scheduler_op, for transferring per-vcpu parameters
between libxc and hypervisor.
2) Handler of XEN_DOMCTL_SCHEDOP_getinfo now just returns the default budget
and
period values of RTDS scheduler.
3) Handler of XEN_DOMCTL_SCHEDOP_getvcpuinfo now can return a random subset of
the parameters of the VCPUs of a specific domain
CC:
CC:
CC:
CC:
CC:
CC:
---
xen/common/sched_credit.c | 14 +++
xen/common/sched_credit2.c | 13 ---
xen/common/sched_rt.c | 93 ++---
xen/common/schedule.c | 15 ++--
xen/include/public/domctl.h | 63 --
5 files changed, 156 insertions(+), 42 deletions(-)
diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
index 305889a..e5d15d8 100644
--- a/xen/common/sched_credit.c
+++ b/xen/common/sched_credit.c
@@ -1080,15 +1080,13 @@ csched_dom_cntl(
* lock. Runq lock not needed anywhere in here. */
spin_lock_irqsave(&prv->lock, flags);
-if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
+switch ( op->cmd )
{
+case XEN_DOMCTL_SCHEDOP_getinfo:
op->u.credit.weight = sdom->weight;
op->u.credit.cap = sdom->cap;
-}
-else
-{
-ASSERT(op->cmd == XEN_DOMCTL_SCHEDOP_putinfo);
-
+break;
+case XEN_DOMCTL_SCHEDOP_putinfo:
if ( op->u.credit.weight != 0 )
{
if ( !list_empty(&sdom->active_sdom_elem) )
@@ -1101,7 +1099,9 @@ csched_dom_cntl(
if ( op->u.credit.cap != (uint16_t)~0U )
sdom->cap = op->u.credit.cap;
-
+break;
+default:
+return -EINVAL;
}
spin_unlock_irqrestore(&prv->lock, flags);
diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index 7ddad38..d48ed5a 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -1421,14 +1421,12 @@ csched2_dom_cntl(
* runq lock to update csvcs. */
spin_lock_irqsave(&prv->lock, flags);
-if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
+switch ( op->cmd )
{
+case XEN_DOMCTL_SCHEDOP_getinfo:
op->u.credit2.weight = sdom->weight;
-}
-else
-{
-ASSERT(op->cmd == XEN_DOMCTL_SCHEDOP_putinfo);
-
+break;
+case XEN_DOMCTL_SCHEDOP_putinfo:
if ( op->u.credit2.weight != 0 )
{
struct vcpu *v;
@@ -1457,6 +1455,9 @@ csched2_dom_cntl(
vcpu_schedule_unlock(lock, svc->vcpu);
}
}
+break;
+d
[Xen-devel] [PATCH v9 for Xen 4.7 0/4] Enable per-VCPU parameter for RTDS
[Goal] The current xl sched-rtds tool can only set the VCPUs of a domain to the same parameter although the scheduler supports VCPUs with different parameters. This patchset is to enable xl sched-rtds tool to configure the VCPUs of a domain with different parameters. This per-VCPU settings can be used in many scenarios. For example, based on Dario's statement in our pervious discussion (http://lists.xen.org/archives/html/xen-devel/2014-09/msg00423.html), if there are two real-time applications, which have different timing requirements, running in a multi-VCPU guest domain, it is beneficial to pin these two applications to two seperate VCPUs with different scheduling parameters. What this patchset includes is a wanted and planned feature for RTDS scheudler(http://wiki.xenproject.org/wiki/RTDS-Based-Scheduler) in Xen 4.7. The interface design of the xl sched-rtds tool is based on Meng's previous discussion with Dario, George and Wei (http://lists.xen.org/archives/html/xen-devel/2015-02/msg02606.html). Basically, there are two main changes: 1) in xl, we create an array that records all VCPUs whose parameters are about to modify or output. 2) in libxl, we receive the array and call different xc functions to handle it. 3) in xen and libxc, we use XEN_DOMCTL_SCHEDOP_getvcpuinfo/putvcpuinfo(introduced by this patchset) as the hypercall for per-VCPU operations(get/set method). [Usage] With this patchset in use, xl sched-rtds tool can: 1) show the budget and period of each VCPU of each domain, by using "xl sched-rtds -v all" command. An example would be like: # xl sched-rtds -v all Cpupool Pool-0: sched=RTDS NameID VCPUPeriodBudget Domain-0 00 1 4000 vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 vm2 20 1 4000 vm2 21 1 4000 Using "xl sched-rtds" will output the default scheduling parameters for each domain. An example would be like: # xl sched-rtds Cpupool Pool-0: sched=RTDS NameIDPeriodBudget Domain-0 0 1 4000 vm1 1 1 4000 vm2 2 1 4000 2) show the budget and period of each VCPU of a specific domain, by using, e.g., "xl sched-rtds -d vm1 -v all" command. The output would be like: # xl sched-rtds -d vm1 -v all NameID VCPUPeriodBudget vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 To show a subset of the parameters of the VCPUs of a specific domain, please use, e.g.,"xl sched-rtds -d vm1 -v 0 -v 3" command. The output would be: # xl sched-rtds -d vm1 -v 0 -v 3 NameID VCPUPeriodBudget vm1 10 300 150 vm1 13 1000 500 Using command, e.g., "xl sched-rtds -d vm1" will output the default scheduling parameters of vm1. An example would be like: # xl sched-rtds -d vm1 NameIDPeriodBudget vm1 1 1 4000 3) Users can set the budget and period of multiple VCPUs of a specific domain with only one command, e.g., "xl sched-rtds -d vm1 -v 0 -p 100 -b 50 -v 3 -p 300 -b 150". Users can set all VCPUs with the same parameters, by one command. e.g., "xl sched-rtds -d vm1 -v all -p 500 -b 250". The github repo for this patch is at: https://github.com/Chong-Li/xen.git And the branch name is: vcpu-rtds (https://github.com/Chong-Li/xen/tree/vcpu-rtds) --- CC: CC: CC: CC: CC: CC: CC: CC: CC: Chong Li (4): xen: enable per-VCPU parameter for RTDS libxc: enable per-VCPU parameter for RTDS libxl: enable per-VCPU parameter for RTDS xl: enable per-VCPU parameter for RTDS docs/man/xl.pod.1 | 67 tools/libxc/include/xenctrl.h | 8 + tools/libxc/xc_rt.c | 68 tools/libxl/libxl.c | 386 +++--- tools/libxl/libxl.h | 41 + tools/libxl/libxl_types.idl | 14 ++ tools/libxl/xl_cmdimpl.c | 365 +++ tools/libxl/xl_cmdtable.c | 4 +- xen/common/sched_credit.c | 14 +- xen/common/sched_credit2.c| 13 +- xen/common/sched_rt.c
Re: [Xen-devel] [PATCH v8 for Xen 4.7 1/4] xen: enable per-VCPU parameter for RTDS
On Mon, Mar 21, 2016 at 11:25 AM, Jan Beulich wrote:
>>>> On 21.03.16 at 17:03, wrote:
>> On Mon, Mar 21, 2016 at 10:49 AM, Jan Beulich wrote:
>>>>>> On 21.03.16 at 16:18, wrote:
>>>> On Mon, Mar 21, 2016 at 8:35 AM, Jan Beulich wrote:
>>>>>>>> On 18.03.16 at 22:26, wrote:
>>>>>> --- a/xen/include/public/domctl.h
>>>>>> +++ b/xen/include/public/domctl.h
>>>>>> @@ -338,24 +338,64 @@ DEFINE_XEN_GUEST_HANDLE(xen_domctl_max_vcpus_t);
>>>>>> #define XEN_SCHEDULER_ARINC653 7
>>>>>> #define XEN_SCHEDULER_RTDS 8
>>>>>>
>>>>>> -/* Set or get info? */
>>>>>> +typedef struct xen_domctl_sched_credit {
>>>>>> +uint16_t weight;
>>>>>> +uint16_t cap;
>>>>>> +} xen_domctl_sched_credit_t;
>>>>>> +
>>>>>> +typedef struct xen_domctl_sched_credit2 {
>>>>>> +uint16_t weight;
>>>>>> +} xen_domctl_sched_credit2_t;
>>>>>> +
>>>>>> +typedef struct xen_domctl_sched_rtds {
>>>>>> +uint32_t period;
>>>>>> +uint32_t budget;
>>>>>> +} xen_domctl_sched_rtds_t;
>>>>>> +
>>>>>> +typedef struct xen_domctl_schedparam_vcpu {
>>>>>> +union {
>>>>>> +xen_domctl_sched_credit_t credit;
>>>>>> +xen_domctl_sched_credit2_t credit2;
>>>>>> +xen_domctl_sched_rtds_t rtds;
>>>>>> +} u;
>>>>>> +uint32_t vcpuid;
>>>>>> +uint16_t padding[2];
>>>>>
>>>>> So why uint16_t[2] instead of just uint32_t? And what's the
>>>>> padding needed for in the first place?
>>>>
>>>> You're right. It's better to use uint32_t, which pads (the struct) to
>>>> the 64-bit boundary.
>>>
>>> Which doesn't answer the "why in the first place" part - I don't
>>> see why structure size needs to be a multiple of 64 bits.
>>>
>> In this patch post,
>>
>> http://lists.xen.org/archives/html/xen-devel/2015-07/msg02334.html
>>
>> you had a comment about the structure size, which I think you mean
>> the struct size should be a multiple of 64 bits.
>
> Looks like I had got mislead by there being struct
> xen_domctl_sched_sedf, but it not being part of the union. I'm
> sorry for that.
>
Ok. I've already fixed all problems pointed out by George and Jan.
Dario and Meng, do you have any other comments on this patch?
Chong
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v8 for Xen 4.7 1/4] xen: enable per-VCPU parameter for RTDS
On Mon, Mar 21, 2016 at 10:49 AM, Jan Beulich wrote:
>>>> On 21.03.16 at 16:18, wrote:
>> On Mon, Mar 21, 2016 at 8:35 AM, Jan Beulich wrote:
>>>>>> On 18.03.16 at 22:26, wrote:
>>>> --- a/xen/include/public/domctl.h
>>>> +++ b/xen/include/public/domctl.h
>>>> @@ -338,24 +338,64 @@ DEFINE_XEN_GUEST_HANDLE(xen_domctl_max_vcpus_t);
>>>> #define XEN_SCHEDULER_ARINC653 7
>>>> #define XEN_SCHEDULER_RTDS 8
>>>>
>>>> -/* Set or get info? */
>>>> +typedef struct xen_domctl_sched_credit {
>>>> +uint16_t weight;
>>>> +uint16_t cap;
>>>> +} xen_domctl_sched_credit_t;
>>>> +
>>>> +typedef struct xen_domctl_sched_credit2 {
>>>> +uint16_t weight;
>>>> +} xen_domctl_sched_credit2_t;
>>>> +
>>>> +typedef struct xen_domctl_sched_rtds {
>>>> +uint32_t period;
>>>> +uint32_t budget;
>>>> +} xen_domctl_sched_rtds_t;
>>>> +
>>>> +typedef struct xen_domctl_schedparam_vcpu {
>>>> +union {
>>>> +xen_domctl_sched_credit_t credit;
>>>> +xen_domctl_sched_credit2_t credit2;
>>>> +xen_domctl_sched_rtds_t rtds;
>>>> +} u;
>>>> +uint32_t vcpuid;
>>>> +uint16_t padding[2];
>>>
>>> So why uint16_t[2] instead of just uint32_t? And what's the
>>> padding needed for in the first place?
>>
>> You're right. It's better to use uint32_t, which pads (the struct) to
>> the 64-bit boundary.
>
> Which doesn't answer the "why in the first place" part - I don't
> see why structure size needs to be a multiple of 64 bits.
>
In this patch post,
http://lists.xen.org/archives/html/xen-devel/2015-07/msg02334.html
you had a comment about the structure size, which I think you mean
the struct size should be a multiple of 64 bits.
Chong
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v8 for Xen 4.7 1/4] xen: enable per-VCPU parameter for RTDS
On Mon, Mar 21, 2016 at 8:35 AM, Jan Beulich wrote:
>>>> On 18.03.16 at 22:26, wrote:
>> Add XEN_DOMCTL_SCHEDOP_getvcpuinfo and _putvcpuinfo hypercalls
>> to independently get and set the scheduling parameters of each
>> vCPU of a domain
>>
>> Signed-off-by: Chong Li
>> Signed-off-by: Meng Xu
>> Signed-off-by: Sisu Xi
>>
>> ---
>> Changes on PATCH v7:
>> 1) A bug in case XEN_DOMCTL_SCHEDOP_getinfo (in Xen 4.6) is fixed:
>> The default PERIOD or BUDGET should be divided by MICROSECS(1),
>> before returned to upper caller.
>
> Seems like there's still some misunderstanding here: Anything
> past the first --- won't make it into the repo, yet the description
> of what bug you fix should end up there.
I see. Do I put the "bug fix" before "Signed-off-by ..." or after it?
>
>> --- a/xen/include/public/domctl.h
>> +++ b/xen/include/public/domctl.h
>> @@ -338,24 +338,64 @@ DEFINE_XEN_GUEST_HANDLE(xen_domctl_max_vcpus_t);
>> #define XEN_SCHEDULER_ARINC653 7
>> #define XEN_SCHEDULER_RTDS 8
>>
>> -/* Set or get info? */
>> +typedef struct xen_domctl_sched_credit {
>> +uint16_t weight;
>> +uint16_t cap;
>> +} xen_domctl_sched_credit_t;
>> +
>> +typedef struct xen_domctl_sched_credit2 {
>> +uint16_t weight;
>> +} xen_domctl_sched_credit2_t;
>> +
>> +typedef struct xen_domctl_sched_rtds {
>> +uint32_t period;
>> +uint32_t budget;
>> +} xen_domctl_sched_rtds_t;
>> +
>> +typedef struct xen_domctl_schedparam_vcpu {
>> +union {
>> +xen_domctl_sched_credit_t credit;
>> +xen_domctl_sched_credit2_t credit2;
>> +xen_domctl_sched_rtds_t rtds;
>> +} u;
>> +uint32_t vcpuid;
>> +uint16_t padding[2];
>
> So why uint16_t[2] instead of just uint32_t? And what's the
> padding needed for in the first place?
You're right. It's better to use uint32_t, which pads (the struct) to
the 64-bit boundary.
> Also, while for a domctl it's
> not as strictly needed as for other hypercalls, checking that all
> padding fields are zero would still seem to be rather desirable.
>
Do you mean:
+case XEN_DOMCTL_SCHEDOP_getvcpuinfo:
+case XEN_DOMCTL_SCHEDOP_putvcpuinfo:
+if ( op->u.v.padding !=0 )
+{
+rc = -EINVAL;
+break;
+}
+while ( index < op->u.v.nr_vcpus )
+{
+if ( copy_from_guest_offset(&local_sched,
+op->u.v.vcpus, index, 1) )
+{
+rc = -EFAULT;
+break;
+ }
+if ( local_sched.padding != 0 )
+{
+rc = -EINVAL;
+break;
+}
+if ( local_sched.vcpuid >= d->max_vcpus ||
+ d->vcpu[local_sched.vcpuid] == NULL )
+{
+rc = -EINVAL;
+break;
+}
?
Thanks,
Chong
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] List of projects for 4.7
On Fri, Mar 18, 2016 at 7:07 AM, Wei Liu wrote: > Hi all > > Today is that last posting day for new features. And we are two weeks > away from the anticipated freeze date. > > I've gone through the outstanding patch series on the list and ask for > input from various core community members. I've enumerated a list > here, which covers several areas of this release and can be used as a > guideline. > > Important and close patch, new features: > 1. xSplice > 2. CPUID levelling > 3. ARM ACPI > 4. COLO HA > 5. RTDS per-vcpu parameter setting The current status: 1) patch for xen: The "mentioning a bug fixed" issue 2) patch for libxc: Acked-by: Wei Liu; Reviewed-by: Dario Faggioli 3) patch for libxl: Acked-by: Wei Liu 4) patch for xl: More examples need to be added to man page Chong -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v7 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
Add XEN_DOMCTL_SCHEDOP_getvcpuinfo and _putvcpuinfo hypercalls
to independently get and set the scheduling parameters of each
vCPU of a domain
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v6:
1) Add explain for nr_vcpus in struct xen_domctl_scheduler_op, because it is
used
in both IN and OUT ways.
2) Remove the check and warning for vcpu settings with low budget or budget.
Making this
feature "per-domain" or "per-operation" is one of the future work.
3) In the *cntl() functions in sched_credit.c and sched_credit2.c, change the
"if-else"
structure to "switch" structure.
4) In rt_dom_cntl(), use copy_to_guest* instead of __copy_to_guest*, because
the latter one
requires lock protection.
Changes on PATCH v5:
1) When processing XEN_DOMCTL_SCHEDOP_get/putvcpuinfo, we do
preemption check in a similar way to XEN_SYSCTL_pcitopoinfo
Changes on PATCH v4:
1) Add uint32_t vcpu_index to struct xen_domctl_scheduler_op.
When processing XEN_DOMCTL_SCHEDOP_get/putvcpuinfo, we call
hypercall_preemption_check in case the current hypercall lasts
too long. If we decide to preempt the current hypercall, we record
the index of the most-recent finished vcpu into the vcpu_index of
struct xen_domctl_scheduler_op. So when we resume the hypercall after
preemption, we start processing from the posion specified by vcpu_index,
and don't need to repeat the work that has already been done in the
hypercall before the preemption.
(This design is based on the do_grant_table_op() in grant_table.c)
2) Coding style changes
Changes on PATCH v3:
1) Remove struct xen_domctl_schedparam_t.
2) Change struct xen_domctl_scheduler_op.
3) Check if period/budget is within a validated range
Changes on PATCH v2:
1) Change struct xen_domctl_scheduler_op, for transferring per-vcpu parameters
between libxc and hypervisor.
2) Handler of XEN_DOMCTL_SCHEDOP_getinfo now just returns the default budget and
period values of RTDS scheduler.
3) Handler of XEN_DOMCTL_SCHEDOP_getvcpuinfo now can return a random subset of
the parameters of the VCPUs of a specific domain
CC:
CC:
CC:
CC:
CC:
CC:
---
xen/common/sched_credit.c | 17 ---
xen/common/sched_credit2.c | 16 ---
xen/common/sched_rt.c | 114 ++--
xen/common/schedule.c | 15 --
xen/include/public/domctl.h | 63 +++-
5 files changed, 183 insertions(+), 42 deletions(-)
diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
index 0dce790..82b0d14 100644
--- a/xen/common/sched_credit.c
+++ b/xen/common/sched_credit.c
@@ -1054,15 +1054,16 @@ csched_dom_cntl(
* lock. Runq lock not needed anywhere in here. */
spin_lock_irqsave(&prv->lock, flags);
-if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
+switch ( op->cmd )
{
+case XEN_DOMCTL_SCHEDOP_putvcpuinfo:
+case XEN_DOMCTL_SCHEDOP_getvcpuinfo:
+return -EINVAL;
+case XEN_DOMCTL_SCHEDOP_getinfo:
op->u.credit.weight = sdom->weight;
op->u.credit.cap = sdom->cap;
-}
-else
-{
-ASSERT(op->cmd == XEN_DOMCTL_SCHEDOP_putinfo);
-
+break;
+case XEN_DOMCTL_SCHEDOP_putinfo:
if ( op->u.credit.weight != 0 )
{
if ( !list_empty(&sdom->active_sdom_elem) )
@@ -1075,7 +1076,9 @@ csched_dom_cntl(
if ( op->u.credit.cap != (uint16_t)~0U )
sdom->cap = op->u.credit.cap;
-
+break;
+default:
+return -EINVAL;
}
spin_unlock_irqrestore(&prv->lock, flags);
diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index 3c49ffa..46d54bc 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -1421,14 +1421,15 @@ csched2_dom_cntl(
* runq lock to update csvcs. */
spin_lock_irqsave(&prv->lock, flags);
-if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
+switch ( op->cmd )
{
+case XEN_DOMCTL_SCHEDOP_putvcpuinfo:
+case XEN_DOMCTL_SCHEDOP_getvcpuinfo:
+return -EINVAL;
+case XEN_DOMCTL_SCHEDOP_getinfo:
op->u.credit2.weight = sdom->weight;
-}
-else
-{
-ASSERT(op->cmd == XEN_DOMCTL_SCHEDOP_putinfo);
-
+break;
+case XEN_DOMCTL_SCHEDOP_putinfo:
if ( op->u.credit2.weight != 0 )
{
struct vcpu *v;
@@ -1457,6 +1458,9 @@ csched2_dom_cntl(
vcpu_schedule_unlock(lock, svc->vcpu);
}
}
+break;
+default:
+return -EINVAL;
}
spin_unlock_irqrestore(&prv->lock, flags);
diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c
index 3f1d047..359c2db 100644
--- a/xen/common/sched_rt.c
+++ b/xen/common/sched_rt.c
@@ -86,6 +86,22 @@
#define RTDS_DEFAULT_PERIOD (MICROSECS(1))
#define RTDS_DEFAULT_BUDGET (MICROSECS(4000))
+/*
+ * M
[Xen-devel] [PATCH v8 for Xen 4.7 0/4] Enable per-VCPU parameter for RTDS
[Goal] The current xl sched-rtds tool can only set the VCPUs of a domain to the same parameter although the scheduler supports VCPUs with different parameters. This patchset is to enable xl sched-rtds tool to configure the VCPUs of a domain with different parameters. This per-VCPU settings can be used in many scenarios. For example, based on Dario's statement in our pervious discussion (http://lists.xen.org/archives/html/xen-devel/2014-09/msg00423.html), if there are two real-time applications, which have different timing requirements, running in a multi-VCPU guest domain, it is beneficial to pin these two applications to two seperate VCPUs with different scheduling parameters. What this patchset includes is a wanted and planned feature for RTDS scheudler(http://wiki.xenproject.org/wiki/RTDS-Based-Scheduler) in Xen 4.7. The interface design of the xl sched-rtds tool is based on Meng's previous discussion with Dario, George and Wei (http://lists.xen.org/archives/html/xen-devel/2015-02/msg02606.html). Basically, there are two main changes: 1) in xl, we create an array that records all VCPUs whose parameters are about to modify or output. 2) in libxl, we receive the array and call different xc functions to handle it. 3) in xen and libxc, we use XEN_DOMCTL_SCHEDOP_getvcpuinfo/putvcpuinfo(introduced by this patchset) as the hypercall for per-VCPU operations(get/set method). [Usage] With this patchset in use, xl sched-rtds tool can: 1) show the budget and period of each VCPU of each domain, by using "xl sched-rtds -v all" command. An example would be like: # xl sched-rtds -v all Cpupool Pool-0: sched=RTDS NameID VCPUPeriodBudget Domain-0 00 1 4000 vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 vm2 20 1 4000 vm2 21 1 4000 Using "xl sched-rtds" will output the default scheduling parameters for each domain. An example would be like: # xl sched-rtds Cpupool Pool-0: sched=RTDS NameIDPeriodBudget Domain-0 0 1 4000 vm1 1 1 4000 vm2 2 1 4000 2) show the budget and period of each VCPU of a specific domain, by using, e.g., "xl sched-rtds -d vm1 -v all" command. The output would be like: # xl sched-rtds -d vm1 -v all NameID VCPUPeriodBudget vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 To show a subset of the parameters of the VCPUs of a specific domain, please use, e.g.,"xl sched-rtds -d vm1 -v 0 -v 3" command. The output would be: # xl sched-rtds -d vm1 -v 0 -v 3 NameID VCPUPeriodBudget vm1 10 300 150 vm1 13 1000 500 Using command, e.g., "xl sched-rtds -d vm1" will output the default scheduling parameters of vm1. An example would be like: # xl sched-rtds -d vm1 NameIDPeriodBudget vm1 1 1 4000 3) Users can set the budget and period of multiple VCPUs of a specific domain with only one command, e.g., "xl sched-rtds -d vm1 -v 0 -p 100 -b 50 -v 3 -p 300 -b 150". Users can set all VCPUs with the same parameters, by one command. e.g., "xl sched-rtds -d vm1 -v all -p 500 -b 250". --- CC: CC: CC: CC: CC: CC: CC: CC: CC: Chong Li (4): xen: enable per-VCPU parameter for RTDS libxc: enable per-VCPU parameter for RTDS libxl: enable per-VCPU parameter for RTDS xl: enable per-VCPU parameter for RTDS docs/man/xl.pod.1 | 77 +++ tools/libxc/include/xenctrl.h | 8 ++ tools/libxc/xc_rt.c | 68 + tools/libxl/libxl.c | 314 +++--- tools/libxl/libxl.h | 37 + tools/libxl/libxl_types.idl | 14 ++ tools/libxl/xl_cmdimpl.c | 301 +++- tools/libxl/xl_cmdtable.c | 4 +- xen/common/sched_credit.c | 14 +- xen/common/sched_credit2.c| 13 +- xen/common/sched_rt.c | 93 +++-- xen/common/schedule.c | 15 +- xen/include/public/domctl.h | 64 +++-- 13 files changed, 925 insertions(+), 97 del
[Xen-devel] [PATCH v7 for Xen 4.7 0/4] Enable per-VCPU parameter settings for RTDS scheduler
[Goal] The current xl sched-rtds tool can only set the VCPUs of a domain to the same parameter although the scheduler supports VCPUs with different parameters. This patchset is to enable xl sched-rtds tool to configure the VCPUs of a domain with different parameters. This per-VCPU settings can be used in many scenarios. For example, based on Dario's statement in our pervious discussion (http://lists.xen.org/archives/html/xen-devel/2014-09/msg00423.html), if there are two real-time applications, which have different timing requirements, running in a multi-VCPU guest domain, it is beneficial to pin these two applications to two seperate VCPUs with different scheduling parameters. What this patchset includes is a wanted and planned feature for RTDS scheudler(http://wiki.xenproject.org/wiki/RTDS-Based-Scheduler) in Xen 4.7. The interface design of the xl sched-rtds tool is based on Meng's previous discussion with Dario, George and Wei (http://lists.xen.org/archives/html/xen-devel/2015-02/msg02606.html). Basically, there are two main changes: 1) in xl, we create an array that records all VCPUs whose parameters are about to modify or output. 2) in libxl, we receive the array and call different xc functions to handle it. 3) in xen and libxc, we use XEN_DOMCTL_SCHEDOP_getvcpuinfo/putvcpuinfo(introduced by this patchset) as the hypercall for per-VCPU operations(get/set method). [Usage] With this patchset in use, xl sched-rtds tool can: 1) show the budget and period of each VCPU of each domain, by using "xl sched-rtds -v all" command. An example would be like: # xl sched-rtds -v all Cpupool Pool-0: sched=RTDS NameID VCPUPeriodBudget Domain-0 00 1 4000 vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 vm2 20 1 4000 vm2 21 1 4000 Using "xl sched-rtds" will output the default scheduling parameters for each domain. An example would be like: # xl sched-rtds Cpupool Pool-0: sched=RTDS NameIDPeriodBudget Domain-0 0 1 4000 vm1 1 1 4000 vm2 2 1 4000 2) show the budget and period of each VCPU of a specific domain, by using, e.g., "xl sched-rtds -d vm1 -v all" command. The output would be like: # xl sched-rtds -d vm1 -v all NameID VCPUPeriodBudget vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 To show a subset of the parameters of the VCPUs of a specific domain, please use, e.g.,"xl sched-rtds -d vm1 -v 0 -v 3" command. The output would be: # xl sched-rtds -d vm1 -v 0 -v 3 NameID VCPUPeriodBudget vm1 10 300 150 vm1 13 1000 500 Using command, e.g., "xl sched-rtds -d vm1" will output the default scheduling parameters of vm1. An example would be like: # xl sched-rtds -d vm1 NameIDPeriodBudget vm1 1 1 4000 3) Users can set the budget and period of multiple VCPUs of a specific domain with only one command, e.g., "xl sched-rtds -d vm1 -v 0 -p 100 -b 50 -v 3 -p 300 -b 150". Users can set all VCPUs with the same parameters, by one command. e.g., "xl sched-rtds -d vm1 -v all -p 500 -b 250". --- CC: CC: CC: CC: CC: CC: CC: CC: CC: Chong Li (4): xen: enable per-VCPU parameter settings for RTDS scheduler libxc: enable per-VCPU parameter settings for RTDS scheduler libxl: enable per-VCPU parameter settings for RTDS scheduler xl: enable per-VCPU parameter settings for RTDS scheduler docs/man/xl.pod.1 | 38 + tools/libxc/include/xenctrl.h | 8 ++ tools/libxc/xc_rt.c | 68 + tools/libxl/libxl.c | 321 +++--- tools/libxl/libxl.h | 37 + tools/libxl/libxl_types.idl | 14 ++ tools/libxl/xl_cmdimpl.c | 301 ++- tools/libxl/xl_cmdtable.c | 16 ++- xen/common/sched_credit.c | 17 ++- xen/common/sched_credit2.c| 16 ++- xen/common/sched_rt.c | 114 +-- xen/common/schedule.c | 15 +- xen/include/publi
Re: [Xen-devel] [PATCH v6 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Wed, Mar 16, 2016 at 9:37 AM, Meng Xu wrote: > On Wed, Mar 16, 2016 at 4:23 AM, Dario Faggioli > wrote: >> On Tue, 2016-03-15 at 23:43 -0400, Meng Xu wrote: >>> On Tue, Mar 15, 2016 at 11:32 PM, Chong Li >>> wrote: >>> > > > How about: >>> > > > >>> > > > We create a global variable in sched_rt.c: >>> > > > /* This variable holds its value through hyerpcall re- >>> > > > issueing. >>> > > > * When finding vcpu settings with too low budget or period >>> > > > (e.g, >>> > > > 100 us), we print a warning >>> > > > * and set this variable "true". No more warnings are >>> > > > printed >>> > > > until this variable >>> > > > * becomes false. >>> > > > */ >>> > > > static bool warned; >>> > > > Initialize it as "false" in rt_init(). >>> > > > In your example, >>> > > > we "warned = true" when we find the first vcpu has budget less >>> > > > than >>> > > > 100 us. Outside >>> > > > of the while loop, we do: >>> > > > if ( index == op->u.v.nr_vcpus ) /* no more hypercall re- >>> > > > issueing */ >>> > > > warned = false; >>> > > > >>> > > >>> > > >>> > If we define >>> > >>> >static bool warned; >>> > >>> > at the beginning of rt_dom_cntl(), do we need to initialize it? If >>> > without >>> > initialization, I think its default value is "false", which is just >>> > what we need. >>> > >>> We need initializing any variable we are going to use, of course. We >>> should not reply on the compiler to give an initialized value. :-) >>> >> We need to initialize any variable that would be used uninitialized, if >> we don't initialize it. :-) >> >> However, something along the line of a static variable was also what I >> was thinking to, but I don't think it works sufficiently well for >> justifying it being introduced. And making things work well is proving >> to be too hard to keep bothering. >> >> Reasons why I'm saying I don't think it works well are that: (a) there >> may be more than one CPU executing this hypercall, and they'd race on >> the value of the static flag; (b) what if the hypercall finishes >> processing the first lot of 64 vCPUs with the flag set to false, are we >> sure it can't be anything than "still false", when the new hypercal, >> for the next lot of vCPUs of the same domain, is re-issued? >> >> I continue to think that it could be useful to have this logged, but >> I'm leaning toward just killing it for now (and maybe finding another >> way to check and warn about the same thing or one of the effects it >> produces, later). >> >> Meng, what do you think? > > I'm thinking about if it may not be worthwhile *for now only* to > provide such information with so much effort and the danger of > introducing more serious issues. > > Right, race condition occurs on the global variable and I believe we > don't want to encounter this race condition. > So let's just not use the global variable. > > We should definitely put a large warning in the wiki for the RTDS > scheduler about the parameter settings. Incorrect setting should never > crash system but may lead to poor real-time performance users want. > > Once this patch is done, I will modify the wiki then. (Chong, could > you remind me if I happen to forget?) > Sure, I'll. Chong -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v7 for Xen 4.7 2/4] libxc: enable per-VCPU parameter settings for RTDS scheduler
Add xc_sched_rtds_vcpu_get/set functions to interact with
Xen to get/set a domain's per-VCPU parameters.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v6:
1) Resolve some coding sytle issues
Changes on PATCH v5:
1) In xc_sched_rtds_vcpu_get/set, re-issueing the hypercall
if it is preempted.
Changes on PATCH v4:
1) Minor modifications on the function parameters.
Changes on PATCH v2:
1) Minor modifications due to the change of struct xen_domctl_scheduler_op.
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxc/include/xenctrl.h | 8 +
tools/libxc/xc_rt.c | 68 +++
2 files changed, 76 insertions(+)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 01a6dda..4567585 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -893,6 +893,14 @@ int xc_sched_rtds_domain_set(xc_interface *xch,
int xc_sched_rtds_domain_get(xc_interface *xch,
uint32_t domid,
struct xen_domctl_sched_rtds *sdom);
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus);
+int xc_sched_rtds_vcpu_get(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus);
int
xc_sched_arinc653_schedule_set(
diff --git a/tools/libxc/xc_rt.c b/tools/libxc/xc_rt.c
index d59e5ce..d1d1aa5 100644
--- a/tools/libxc/xc_rt.c
+++ b/tools/libxc/xc_rt.c
@@ -62,3 +62,71 @@ int xc_sched_rtds_domain_get(xc_interface *xch,
return rc;
}
+
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus)
+{
+int rc = 0;
+unsigned processed = 0;
+DECLARE_DOMCTL;
+DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
+ XC_HYPERCALL_BUFFER_BOUNCE_IN);
+
+if ( xc_hypercall_bounce_pre(xch, vcpus) )
+return -1;
+
+domctl.cmd = XEN_DOMCTL_scheduler_op;
+domctl.domain = (domid_t) domid;
+domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
+domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_putvcpuinfo;
+
+while ( processed < num_vcpus )
+{
+domctl.u.scheduler_op.u.v.nr_vcpus = num_vcpus - processed;
+set_xen_guest_handle_offset(domctl.u.scheduler_op.u.v.vcpus, vcpus,
+processed);
+if ( (rc = do_domctl(xch, &domctl)) != 0 )
+break;
+processed += domctl.u.scheduler_op.u.v.nr_vcpus;
+}
+
+xc_hypercall_bounce_post(xch, vcpus);
+
+return rc;
+}
+
+int xc_sched_rtds_vcpu_get(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus)
+{
+int rc;
+unsigned processed = 0;
+DECLARE_DOMCTL;
+DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
+ XC_HYPERCALL_BUFFER_BOUNCE_BOTH);
+
+if ( xc_hypercall_bounce_pre(xch, vcpus) )
+return -1;
+
+domctl.cmd = XEN_DOMCTL_scheduler_op;
+domctl.domain = (domid_t) domid;
+domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
+domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_getvcpuinfo;
+
+while ( processed < num_vcpus )
+{
+domctl.u.scheduler_op.u.v.nr_vcpus = num_vcpus - processed;
+set_xen_guest_handle_offset(domctl.u.scheduler_op.u.v.vcpus, vcpus,
+processed);
+if ( (rc = do_domctl(xch, &domctl)) != 0 )
+break;
+processed += domctl.u.scheduler_op.u.v.nr_vcpus;
+}
+
+xc_hypercall_bounce_post(xch, vcpus);
+
+return rc;
+}
--
1.9.1
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v6 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Wed, Mar 16, 2016 at 3:23 AM, Dario Faggioli wrote: > On Tue, 2016-03-15 at 23:43 -0400, Meng Xu wrote: >> On Tue, Mar 15, 2016 at 11:32 PM, Chong Li >> wrote: >> > > > How about: >> > > > >> > > > We create a global variable in sched_rt.c: >> > > > /* This variable holds its value through hyerpcall re- >> > > > issueing. >> > > > * When finding vcpu settings with too low budget or period >> > > > (e.g, >> > > > 100 us), we print a warning >> > > > * and set this variable "true". No more warnings are >> > > > printed >> > > > until this variable >> > > > * becomes false. >> > > > */ >> > > > static bool warned; >> > > > Initialize it as "false" in rt_init(). >> > > > In your example, >> > > > we "warned = true" when we find the first vcpu has budget less >> > > > than >> > > > 100 us. Outside >> > > > of the while loop, we do: >> > > > if ( index == op->u.v.nr_vcpus ) /* no more hypercall re- >> > > > issueing */ >> > > > warned = false; >> > > > >> > > >> > > >> > If we define >> > >> >static bool warned; >> > >> > at the beginning of rt_dom_cntl(), do we need to initialize it? If >> > without >> > initialization, I think its default value is "false", which is just >> > what we need. >> > >> We need initializing any variable we are going to use, of course. We >> should not reply on the compiler to give an initialized value. :-) >> > We need to initialize any variable that would be used uninitialized, if > we don't initialize it. :-) > > However, something along the line of a static variable was also what I > was thinking to, but I don't think it works sufficiently well for > justifying it being introduced. And making things work well is proving > to be too hard to keep bothering. > > Reasons why I'm saying I don't think it works well are that: (a) there > may be more than one CPU executing this hypercall, and they'd race on > the value of the static flag; (b) what if the hypercall finishes > processing the first lot of 64 vCPUs with the flag set to false, are we > sure it can't be anything than "still false", when the new hypercal, > for the next lot of vCPUs of the same domain, is re-issued? > > I continue to think that it could be useful to have this logged, but > I'm leaning toward just killing it for now (and maybe finding another > way to check and warn about the same thing or one of the effects it > produces, later). > By "killing it", do you mean we don't do this check nor print the warning? Or just print the warning globally once? Chong -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v8 for Xen 4.7 1/4] xen: enable per-VCPU parameter for RTDS
Add XEN_DOMCTL_SCHEDOP_getvcpuinfo and _putvcpuinfo hypercalls
to independently get and set the scheduling parameters of each
vCPU of a domain
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v7:
1) A bug in case XEN_DOMCTL_SCHEDOP_getinfo (in Xen 4.6) is fixed:
The default PERIOD or BUDGET should be divided by MICROSECS(1),
before returned to upper caller.
2) In the *cntl() functions in sched_credit.c and sched_credit2.c, remove
two unhandled cases in "switch".
Changes on PATCH v6:
1) Add explain for nr_vcpus in struct xen_domctl_scheduler_op, because it is
used
in both IN and OUT ways.
2) Remove the check and warning for vcpu settings with low budget or budget.
Making this
feature "per-domain" or "per-operation" is one of the future work.
3) In the *cntl() functions in sched_credit.c and sched_credit2.c, change the
"if-else"
structure to "switch" structure.
4) In rt_dom_cntl(), use copy_to_guest* instead of __copy_to_guest*, because
the latter one
requires lock protection.
Changes on PATCH v5:
1) When processing XEN_DOMCTL_SCHEDOP_get/putvcpuinfo, we do
preemption check in a similar way to XEN_SYSCTL_pcitopoinfo
Changes on PATCH v4:
1) Add uint32_t vcpu_index to struct xen_domctl_scheduler_op.
When processing XEN_DOMCTL_SCHEDOP_get/putvcpuinfo, we call
hypercall_preemption_check in case the current hypercall lasts
too long. If we decide to preempt the current hypercall, we record
the index of the most-recent finished vcpu into the vcpu_index of
struct xen_domctl_scheduler_op. So when we resume the hypercall after
preemption, we start processing from the posion specified by vcpu_index,
and don't need to repeat the work that has already been done in the
hypercall before the preemption.
(This design is based on the do_grant_table_op() in grant_table.c)
2) Coding style changes
Changes on PATCH v3:
1) Remove struct xen_domctl_schedparam_t.
2) Change struct xen_domctl_scheduler_op.
3) Check if period/budget is within a validated range
Changes on PATCH v2:
1) Change struct xen_domctl_scheduler_op, for transferring per-vcpu parameters
between libxc and hypervisor.
2) Handler of XEN_DOMCTL_SCHEDOP_getinfo now just returns the default budget and
period values of RTDS scheduler.
3) Handler of XEN_DOMCTL_SCHEDOP_getvcpuinfo now can return a random subset of
the parameters of the VCPUs of a specific domain
CC:
CC:
CC:
CC:
CC:
CC:
---
xen/common/sched_credit.c | 14 +++
xen/common/sched_credit2.c | 13 ---
xen/common/sched_rt.c | 93 ++---
xen/common/schedule.c | 15 ++--
xen/include/public/domctl.h | 64 +--
5 files changed, 157 insertions(+), 42 deletions(-)
diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
index 0dce790..b504702 100644
--- a/xen/common/sched_credit.c
+++ b/xen/common/sched_credit.c
@@ -1054,15 +1054,13 @@ csched_dom_cntl(
* lock. Runq lock not needed anywhere in here. */
spin_lock_irqsave(&prv->lock, flags);
-if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
+switch ( op->cmd )
{
+case XEN_DOMCTL_SCHEDOP_getinfo:
op->u.credit.weight = sdom->weight;
op->u.credit.cap = sdom->cap;
-}
-else
-{
-ASSERT(op->cmd == XEN_DOMCTL_SCHEDOP_putinfo);
-
+break;
+case XEN_DOMCTL_SCHEDOP_putinfo:
if ( op->u.credit.weight != 0 )
{
if ( !list_empty(&sdom->active_sdom_elem) )
@@ -1075,7 +1073,9 @@ csched_dom_cntl(
if ( op->u.credit.cap != (uint16_t)~0U )
sdom->cap = op->u.credit.cap;
-
+break;
+default:
+return -EINVAL;
}
spin_unlock_irqrestore(&prv->lock, flags);
diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index 3c49ffa..c2dcf0f 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -1421,14 +1421,12 @@ csched2_dom_cntl(
* runq lock to update csvcs. */
spin_lock_irqsave(&prv->lock, flags);
-if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
+switch ( op->cmd )
{
+case XEN_DOMCTL_SCHEDOP_getinfo:
op->u.credit2.weight = sdom->weight;
-}
-else
-{
-ASSERT(op->cmd == XEN_DOMCTL_SCHEDOP_putinfo);
-
+break;
+case XEN_DOMCTL_SCHEDOP_putinfo:
if ( op->u.credit2.weight != 0 )
{
struct vcpu *v;
@@ -1457,6 +1455,9 @@ csched2_dom_cntl(
vcpu_schedule_unlock(lock, svc->vcpu);
}
}
+break;
+default:
+return -EINVAL;
}
spin_unlock_irqrestore(&prv->lock, flags);
diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c
index 3f1d047..b640e59 100644
--- a/xen/common/sched_rt.c
+++ b/xen/common/sched_rt.c
@@ -86,6 +86,22 @@
#define RTDS_DEFAULT_
[Xen-devel] [PATCH v8 for Xen 4.7 4/4] xl: enable per-VCPU parameter for RTDS
Change main_sched_rtds and related output functions to support per-VCPU settings. Signed-off-by: Chong Li Signed-off-by: Meng Xu Signed-off-by: Sisu Xi --- Changes on PATCH v7: 1) Add example to xl.pod.1 Changes on PATCH v6: 1) More explain in xl.pod.1 and cmdtable.c 2) Resolve some coding sytle issues Changes on PATCH v5: 1) Add sched_vcpu_set_all() for the cases that all vcpus of a domain need to be changed together. Changes on PATCH v4: 1) Coding style changes Changes on PATCH v3: 1) Support commands, e.g., "xl sched-rtds -d vm1" to output the default scheduling parameters Changes on PATCH v2: 1) Remove per-domain output functions for RTDS scheduler. 2) Users now use '-v all' to specify all VCPUs. 3) Support outputting a subset of the parameters of the VCPUs of a specific domain. 4) When setting all VCPUs with the same parameters (by only one command), no per-domain function is invoked. CC: CC: CC: CC: CC: CC: --- docs/man/xl.pod.1 | 77 tools/libxl/xl_cmdimpl.c | 301 +- tools/libxl/xl_cmdtable.c | 4 +- 3 files changed, 348 insertions(+), 34 deletions(-) diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1 index 4279c7c..0350baa 100644 --- a/docs/man/xl.pod.1 +++ b/docs/man/xl.pod.1 @@ -1051,6 +1051,10 @@ B Specify domain for which scheduler parameters are to be modified or retrieved. Mandatory for modifying scheduler parameters. +=item B<-v VCPUID/all>, B<--vcpuid=VCPUID/all> + +Specify vcpu for which scheduler parameters are to be modified or retrieved. + =item B<-p PERIOD>, B<--period=PERIOD> Period of time, in microseconds, over which to replenish the budget. @@ -1066,6 +1070,79 @@ Restrict output to domains in the specified cpupool. =back +B + +=over 4 + +1) show the budget and period of each VCPU of each domain, +by using "xl sched-rtds -v all" command. An example would be like: + +xl sched-rtds -v all + +Cpupool Pool-0: sched=RTDS + +NameID VCPUPeriodBudget +Domain-0 00 1 4000 +vm1 10 300 150 +vm1 11 400 200 +vm1 12 1 4000 +vm1 13 1000 500 +vm2 20 1 4000 +vm2 21 1 4000 + +Using "xl sched-rtds" will output the default scheduling parameters +for each domain. An example would be like: + +xl sched-rtds + +Cpupool Pool-0: sched=RTDS + +NameIDPeriodBudget +Domain-0 0 1 4000 +vm1 1 1 4000 +vm2 2 1 4000 + + +2) show the budget and period of each VCPU of a specific domain, +by using, e.g., "xl sched-rtds -d vm1 -v all" command. The output +would be like: + +xl sched-rtds -d vm1 -v all + +NameID VCPUPeriodBudget +vm1 10 300 150 +vm1 11 400 200 +vm1 12 1 4000 +vm1 13 1000 500 + +To show a subset of the parameters of the VCPUs of a specific domain, +please use, e.g.,"xl sched-rtds -d vm1 -v 0 -v 3" command. +The output would be: + +xl sched-rtds -d vm1 -v 0 -v 3 + +NameID VCPUPeriodBudget +vm1 10 300 150 +vm1 13 1000 500 + +Using command, e.g., "xl sched-rtds -d vm1" will output the default +scheduling parameters of vm1. An example would be like: + +xl sched-rtds -d vm1 + +NameIDPeriodBudget +vm1 1 1 4000 + + +3) Users can set the budget and period of multiple VCPUs of a +specific domain with only one command, +e.g., "xl sched-rtds -d vm1 -v 0 -p 100 -b 50 -v 3 -p 300 -b 150". + +Users can set all VCPUs with the same parameters, by one command. +e.g., "xl sched-rtds -d vm1 -v all -p 500 -b 250". + +=back + =back =head1 CPUPOOLS COMMANDS diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c index 2b6371d..97e8b33 100644 --- a/tools/libxl/xl_cmdimpl.c +++ b/tools/libxl/xl_cmdimpl.c @@ -5823,6 +5823,52 @@ static int sched_domain_set(int domid, const libxl_domain_sched_params *scinfo) return 0; } +static int sched_vcpu_get(libxl_scheduler sched, int domid, + lib
[Xen-devel] [PATCH v7 for Xen 4.7 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
Add libxl_vcpu_sched_params_get/set and sched_rtds_vcpu_get/set
functions to support per-VCPU settings.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v6:
1) Resolve some coding style issues
2) Change sched_rtds_validate_params()
3) Small changes for sched_rtds_vcpus_params_set(all) functions
Changes on PATCH v5:
1) Add a seperate function, sched_rtds_vcpus_params_set_all(), to set
the parameters of all vcpus of a domain.
2) Add libxl_vcpu_sched_params_set_all() to invoke the above function.
3) Coding style changes. (I didn't find the indentation rules for function
calls with long parameters (still 4 spaces?), so I just imitated the
indentation style of some existing functions)
Changes on PATCH v4:
1) Coding style changes
Changes on PATCH v3:
1) Add sanity check on vcpuid
2) Add comments on per-domain and per-vcpu functions for libxl
users
Changes on PATCH v2:
1) New data structure (libxl_vcpu_sched_params and libxl_sched_params)
to help per-VCPU settings.
2) sched_rtds_vcpu_get now can return a random subset of the parameters
of the VCPUs of a specific domain.
CC:
CC:
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxl/libxl.c | 321
tools/libxl/libxl.h | 37 +
tools/libxl/libxl_types.idl | 14 ++
3 files changed, 349 insertions(+), 23 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index bd3aac8..0f9fb7e 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -5770,6 +5770,201 @@ static int sched_credit2_domain_set(libxl__gc *gc,
uint32_t domid,
return 0;
}
+static int sched_rtds_validate_params(libxl__gc *gc, int period, int budget)
+{
+int rc;
+
+if (period < 1) {
+LOG(ERROR, "VCPU period is out of range, "
+ "valid values are larger than or equal to 1");
+rc = ERROR_INVAL; /* error scheduling parameter */
+goto out;
+}
+
+if (budget < 1) {
+LOG(ERROR, "VCPU budget is not set or out of range, "
+ "valid values are larger than or equal to 1");
+rc = ERROR_INVAL;
+goto out;
+}
+
+if (budget > period) {
+LOG(ERROR, "VCPU budget must be smaller than "
+ "or equal to VCPU period");
+rc = ERROR_INVAL;
+goto out;
+}
+rc = 0;
+out:
+return rc;
+}
+
+/* Get the RTDS scheduling parameters of vcpu(s) */
+static int sched_rtds_vcpu_get(libxl__gc *gc, uint32_t domid,
+ libxl_vcpu_sched_params *scinfo)
+{
+uint32_t num_vcpus;
+int i, r, rc;
+xc_dominfo_t info;
+struct xen_domctl_schedparam_vcpu *vcpus;
+
+r = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (r < 0) {
+LOGE(ERROR, "getting domain info");
+rc = ERROR_FAIL;
+goto out;
+}
+
+num_vcpus = scinfo->num_vcpus ? scinfo->num_vcpus :
+info.max_vcpu_id + 1;
+
+GCNEW_ARRAY(vcpus, num_vcpus);
+
+if (scinfo->num_vcpus > 0) {
+for (i = 0; i < num_vcpus; i++) {
+if (scinfo->vcpus[i].vcpuid < 0 ||
+scinfo->vcpus[i].vcpuid > info.max_vcpu_id) {
+LOG(ERROR, "VCPU index is out of range, "
+ "valid values are within range from 0 to %d",
+ info.max_vcpu_id);
+rc = ERROR_INVAL;
+goto out;
+}
+vcpus[i].vcpuid = scinfo->vcpus[i].vcpuid;
+}
+} else
+for (i = 0; i < num_vcpus; i++)
+vcpus[i].vcpuid = i;
+
+r = xc_sched_rtds_vcpu_get(CTX->xch, domid, vcpus, num_vcpus);
+if (r != 0) {
+LOGE(ERROR, "getting vcpu sched rtds");
+rc = ERROR_FAIL;
+goto out;
+}
+scinfo->sched = LIBXL_SCHEDULER_RTDS;
+if (scinfo->num_vcpus == 0) {
+scinfo->num_vcpus = num_vcpus;
+scinfo->vcpus = libxl__calloc(NOGC, num_vcpus,
+ sizeof(libxl_sched_params));
+}
+for (i = 0; i < num_vcpus; i++) {
+scinfo->vcpus[i].period = vcpus[i].u.rtds.period;
+scinfo->vcpus[i].budget = vcpus[i].u.rtds.budget;
+scinfo->vcpus[i].vcpuid = vcpus[i].vcpuid;
+}
+rc = 0;
+out:
+return rc;
+}
+
+/* Set the RTDS scheduling parameters of vcpu(s) */
+static int sched_rtds_vcpus_params_set(libxl__gc *gc, uint32_t domid,
+ const libxl_vcpu_sched_params *scinfo)
+{
+int r, rc;
+int i;
+uint16_t max_vcpuid;
+xc_dominfo_t info;
+struct xen_domctl_schedparam_vcpu *vcpus;
+
+r = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (r < 0) {
+LOGE(ERROR, "getting domain info");
+rc = ERROR_FAI
Re: [Xen-devel] [PATCH v7 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Thu, Mar 17, 2016 at 5:03 AM, Dario Faggioli
wrote:
> On Wed, 2016-03-16 at 11:47 -0500, Chong Li wrote:
>
>> --- a/xen/common/sched_rt.c
>> +++ b/xen/common/sched_rt.c
>> @@ -1129,24 +1145,22 @@ rt_dom_cntl(
>> struct vcpu *v;
>> unsigned long flags;
>> int rc = 0;
>> +xen_domctl_schedparam_vcpu_t local_sched;
>> +s_time_t period, budget;
>> +uint32_t index = 0;
>>
>> switch ( op->cmd )
>> {
>> case XEN_DOMCTL_SCHEDOP_getinfo:
>> -if ( d->max_vcpus > 0 )
>> -{
>> -spin_lock_irqsave(&prv->lock, flags);
>> -svc = rt_vcpu(d->vcpu[0]);
>> -op->u.rtds.period = svc->period / MICROSECS(1);
>> -op->u.rtds.budget = svc->budget / MICROSECS(1);
>> -spin_unlock_irqrestore(&prv->lock, flags);
>> -}
>> -else
>> -{
>> -/* If we don't have vcpus yet, let's just return the
>> defaults. */
>> -op->u.rtds.period = RTDS_DEFAULT_PERIOD;
>> -op->u.rtds.budget = RTDS_DEFAULT_BUDGET;
>> -}
>> +/* Return the default parameters.
>> + * A previous bug fixed here:
>> + * The default PERIOD or BUDGET should be divided by
>> MICROSECS(1),
>> + * before returned to upper caller.
>> + */
> Comment style (missing opening '/*').
>
> Also, putting this kind of things in comments is not at all ideal. If
> doing this in this patch, you should mention it in the changelog. Or
> you do it in a separate patch (that you put before this one in the
> series).
>
> I'd say that, in this specific case, is not a big deal which one of the
> two approaches you take (mentioning or separate patch), but the having
> a separate one is indeed almost always preferable (e.g., the fix can be
> backported).
If I choose mentioning, do I move the comment to the changelog? Or do I keep
it here and say it again in the changelog?
Chong
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v7 for Xen 4.7 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
On Wed, Mar 16, 2016 at 11:05 PM, Dario Faggioli
wrote:
> On Wed, 2016-03-16 at 11:47 -0500, Chong Li wrote:
>> Add libxl_vcpu_sched_params_get/set and sched_rtds_vcpu_get/set
>> functions to support per-VCPU settings.
>>
>> +/* Set the RTDS scheduling parameters of all vcpus of a domain */
>> +static int sched_rtds_vcpus_params_set_all(libxl__gc *gc, uint32_t
>> domid,
>> + const libxl_vcpu_sched_params
>> *scinfo)
>>
> Indentation?
If I follow the indentation rule, the second line would be longer than
80 characters.
The function name is just too long.
>
>> @@ -5802,30 +5997,10 @@ static int sched_rtds_domain_set(libxl__gc
>> *gc, uint32_t domid,
>> LOGE(ERROR, "getting domain sched rtds");
>> return ERROR_FAIL;
>> }
>> -
>> -if (scinfo->period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT) {
>> -if (scinfo->period < 1) {
>> -LOG(ERROR, "VCPU period is not set or out of range, "
>> - "valid values are larger than 1");
>> -return ERROR_INVAL;
>> -}
>> -sdom.period = scinfo->period;
>> -}
>> -
>> -if (scinfo->budget != LIBXL_DOMAIN_SCHED_PARAM_BUDGET_DEFAULT) {
>> -if (scinfo->budget < 1) {
>> -LOG(ERROR, "VCPU budget is not set or out of range, "
>> - "valid values are larger than 1");
>> +if (scinfo->period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT &&
>> +scinfo->budget != LIBXL_DOMAIN_SCHED_PARAM_BUDGET_DEFAULT)
>> +if (sched_rtds_validate_params(gc, scinfo->period, scinfo-
>> >budget))
>> return ERROR_INVAL;
>>
> I'm not sure I understand. What's happening in this function?
>
> As it stands after this patch, it looks to me that:
> - we read the default scheduling parameter from Xen,
>via xc_sched_rtds_domain_get()
> - we (possibly, if both are non-default) validate a new period and
>budget couple of values
> - we don't use such values for anything, and set back what we got
>from Xen, via xc_sched_rtds_domain_set()
>
> Either I'm missing something very basic, or this is not what Wei said
> when reviewing v6:
>
> "Then at callsites you set those values with two direct assignment:
>
>if (validate(period_value, budget_value) != 0) {
>error;
>}
>period = period_value;
>budget = budget_value;"
>
> Is it?
The current RTDS (xen 4.6) is:
1) Read the (per-domain) scheduling params from Xen, and store them to sdom
2) If period equals LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT (which is
-1), use sdom.period;
else sdom.period = new period (if new period is valid);
If budget equals LIBXL_DOMAIN_SCHED_PARAM_BUDGET_DEFAULT (which is
-1), use sdom.budget;
else sdom.budget = new budget;
3) xc_sched_rtds_domain_set (sdom);
In our patch, my plan is:
1) Read the default params from Xen, and store them to sdom
2) If period equals LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT (which is
-1), use sdom.period;
else sdom.period = new period (if new period is valid);
If budget equals LIBXL_DOMAIN_SCHED_PARAM_BUDGET_DEFAULT (which is
-1), use sdom.budget;
else sdom.budget = new budget;
3) xc_sched_rtds_domain_set (sdom);
Even though I made some mistakes in this post (forgot the two "else"s
in my plan), is this plan ok?
Chong
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v7 for Xen 4.7 4/4] xl: enable per-VCPU parameter settings for RTDS scheduler
Change main_sched_rtds and related output functions to support
per-VCPU settings.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v6:
1) More explain in xl.pod.1 and cmdtable.c
2) Resolve some coding sytle issues
Changes on PATCH v5:
1) Add sched_vcpu_set_all() for the cases that all vcpus of a
domain need to be changed together.
Changes on PATCH v4:
1) Coding style changes
Changes on PATCH v3:
1) Support commands, e.g., "xl sched-rtds -d vm1" to output the
default scheduling parameters
Changes on PATCH v2:
1) Remove per-domain output functions for RTDS scheduler.
2) Users now use '-v all' to specify all VCPUs.
3) Support outputting a subset of the parameters of the VCPUs
of a specific domain.
4) When setting all VCPUs with the same parameters (by only one
command), no per-domain function is invoked.
CC:
CC:
CC:
CC:
CC:
CC:
---
docs/man/xl.pod.1 | 38 ++
tools/libxl/xl_cmdimpl.c | 301 +-
tools/libxl/xl_cmdtable.c | 16 ++-
3 files changed, 320 insertions(+), 35 deletions(-)
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 4279c7c..1a8aacd 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -1051,6 +1051,10 @@ B
Specify domain for which scheduler parameters are to be modified or retrieved.
Mandatory for modifying scheduler parameters.
+=item B<-v VCPUID/all>, B<--vcpuid=VCPUID/all>
+
+Specify vcpu for which scheduler parameters are to be modified or retrieved.
+
=item B<-p PERIOD>, B<--period=PERIOD>
Period of time, in microseconds, over which to replenish the budget.
@@ -1066,6 +1070,40 @@ Restrict output to domains in the specified cpupool.
=back
+B
+
+=over 4
+
+=item B<-d [domid]>
+
+List default per-domain params for a domain
+
+=item B<-d [domid] [params (period and budget)]>
+
+Set per-domain params for a domain
+
+=item B<-d [domid] -v [vcpuid] -v [vcpuid] ...>
+
+List per-VCPU params for a domain
+
+=item B<-d [domid] -v all>
+
+List all per-VCPU params for a domain
+
+=item B<-v all>
+
+List all per-VCPU params for all domains
+
+=item B<-d [domid] -v [vcpuid] [params] -v [vcpuid] [params] ...>
+
+Set per-VCPU params for a domain
+
+=item B<-d [domid] -v all [params]>
+
+Set all per-VCPU params for a domain
+
+=back
+
=back
=head1 CPUPOOLS COMMANDS
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 2b6371d..131541f 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -5823,6 +5823,52 @@ static int sched_domain_set(int domid, const
libxl_domain_sched_params *scinfo)
return 0;
}
+static int sched_vcpu_get(libxl_scheduler sched, int domid,
+ libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_get(ctx, domid, scinfo);
+if (rc) {
+fprintf(stderr, "libxl_vcpu_sched_params_get failed.\n");
+exit(-1);
+}
+if (scinfo->sched != sched) {
+fprintf(stderr, "libxl_vcpu_sched_params_get returned %s not %s.\n",
+libxl_scheduler_to_string(scinfo->sched),
+libxl_scheduler_to_string(sched));
+return 1;
+}
+
+return 0;
+}
+
+static int sched_vcpu_set(int domid, const libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_set(ctx, domid, scinfo);
+if (rc) {
+fprintf(stderr, "libxl_vcpu_sched_params_set failed.\n");
+exit(-1);
+}
+
+return rc;
+}
+
+static int sched_vcpu_set_all(int domid, const libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_set_all(ctx, domid, scinfo);
+if (rc) {
+fprintf(stderr, "libxl_vcpu_sched_params_set failed.\n");
+exit(-1);
+}
+
+return rc;
+}
+
static int sched_credit_params_set(int poolid, libxl_sched_credit_params
*scinfo)
{
if (libxl_sched_credit_params_set(ctx, poolid, scinfo)) {
@@ -5942,6 +5988,37 @@ static int sched_rtds_domain_output(
return 0;
}
+static int sched_rtds_vcpu_output(int domid, libxl_vcpu_sched_params *scinfo)
+{
+char *domname;
+int rc = 0;
+int i;
+
+if (domid < 0) {
+printf("%-33s %4s %4s %9s %9s\n", "Name", "ID",
+ "VCPU", "Period", "Budget");
+return 0;
+}
+
+rc = sched_vcpu_get(LIBXL_SCHEDULER_RTDS, domid, scinfo);
+if (rc)
+goto out;
+
+domname = libxl_domid_to_name(ctx, domid);
+for ( i = 0; i < scinfo->num_vcpus; i++ ) {
+printf("%-33s %4d %4d %9"PRIu32" %9"PRIu32"\n",
+ domname,
+ domid,
+ scinfo->vcpus[i].vcpuid,
+ scinfo->vcpus[i].period,
+ scinfo->vcpus[i].budget);
+}
+free(domn
[Xen-devel] [PATCH v8 for Xen 4.7 2/4] libxc: enable per-VCPU parameter for RTDS
Add xc_sched_rtds_vcpu_get/set functions to interact with
Xen to get/set a domain's per-VCPU parameters.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v6:
1) Resolve some coding sytle issues
Changes on PATCH v5:
1) In xc_sched_rtds_vcpu_get/set, re-issueing the hypercall
if it is preempted.
Changes on PATCH v4:
1) Minor modifications on the function parameters.
Changes on PATCH v2:
1) Minor modifications due to the change of struct xen_domctl_scheduler_op.
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxc/include/xenctrl.h | 8 +
tools/libxc/xc_rt.c | 68 +++
2 files changed, 76 insertions(+)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 01a6dda..4567585 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -893,6 +893,14 @@ int xc_sched_rtds_domain_set(xc_interface *xch,
int xc_sched_rtds_domain_get(xc_interface *xch,
uint32_t domid,
struct xen_domctl_sched_rtds *sdom);
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus);
+int xc_sched_rtds_vcpu_get(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus);
int
xc_sched_arinc653_schedule_set(
diff --git a/tools/libxc/xc_rt.c b/tools/libxc/xc_rt.c
index d59e5ce..221d17f 100644
--- a/tools/libxc/xc_rt.c
+++ b/tools/libxc/xc_rt.c
@@ -62,3 +62,71 @@ int xc_sched_rtds_domain_get(xc_interface *xch,
return rc;
}
+
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus)
+{
+int rc;
+unsigned processed = 0;
+DECLARE_DOMCTL;
+DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
+ XC_HYPERCALL_BUFFER_BOUNCE_IN);
+
+if ( xc_hypercall_bounce_pre(xch, vcpus) )
+return -1;
+
+domctl.cmd = XEN_DOMCTL_scheduler_op;
+domctl.domain = (domid_t) domid;
+domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
+domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_putvcpuinfo;
+
+while ( processed < num_vcpus )
+{
+domctl.u.scheduler_op.u.v.nr_vcpus = num_vcpus - processed;
+set_xen_guest_handle_offset(domctl.u.scheduler_op.u.v.vcpus, vcpus,
+processed);
+if ( (rc = do_domctl(xch, &domctl)) != 0 )
+break;
+processed += domctl.u.scheduler_op.u.v.nr_vcpus;
+}
+
+xc_hypercall_bounce_post(xch, vcpus);
+
+return rc;
+}
+
+int xc_sched_rtds_vcpu_get(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus)
+{
+int rc;
+unsigned processed = 0;
+DECLARE_DOMCTL;
+DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
+ XC_HYPERCALL_BUFFER_BOUNCE_BOTH);
+
+if ( xc_hypercall_bounce_pre(xch, vcpus) )
+return -1;
+
+domctl.cmd = XEN_DOMCTL_scheduler_op;
+domctl.domain = (domid_t) domid;
+domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
+domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_getvcpuinfo;
+
+while ( processed < num_vcpus )
+{
+domctl.u.scheduler_op.u.v.nr_vcpus = num_vcpus - processed;
+set_xen_guest_handle_offset(domctl.u.scheduler_op.u.v.vcpus, vcpus,
+processed);
+if ( (rc = do_domctl(xch, &domctl)) != 0 )
+break;
+processed += domctl.u.scheduler_op.u.v.nr_vcpus;
+}
+
+xc_hypercall_bounce_post(xch, vcpus);
+
+return rc;
+}
--
1.9.1
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v7 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Fri, Mar 18, 2016 at 5:47 AM, Dario Faggioli wrote: > On Fri, 2016-03-18 at 01:39 -0600, Jan Beulich wrote: >> > >> > > > On 17.03.16 at 21:42, wrote: >> > On Thu, Mar 17, 2016 at 5:03 AM, Dario Faggioli >> > wrote: >> > > >> > > I'd say that, in this specific case, is not a big deal which one >> > > of the >> > > two approaches you take (mentioning or separate patch), but the >> > > having >> > > a separate one is indeed almost always preferable (e.g., the fix >> > > can be >> > > backported). >> > If I choose mentioning, do I move the comment to the changelog? Or >> > do I keep >> > it here and say it again in the changelog? >> Just consider what would happen if everyone mentioned in >> comments the bugs they fixed. I think the answer to you question >> is obvious then... >> > Exactly! :-) > > And, please (Chong), let me restate this: "having a separate one is > indeed almost always preferable (e.g., the fix can be backported)" > Considering the complexity of making a new patch and today's the last day for posting, I'll mention this bug fix in the changlog. Chong -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v8 for Xen 4.7 3/4] libxl: enable per-VCPU parameter for RTDS
Add libxl_vcpu_sched_params_get/set and sched_rtds_vcpu_get/set
functions to support per-VCPU settings.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v7:
1) Fix a mistake in sched_rtds_domain_set()
Changes on PATCH v6:
1) Resolve some coding style issues
2) Change sched_rtds_validate_params()
3) Small changes for sched_rtds_vcpus_params_set(all) functions
Changes on PATCH v5:
1) Add a seperate function, sched_rtds_vcpus_params_set_all(), to set
the parameters of all vcpus of a domain.
2) Add libxl_vcpu_sched_params_set_all() to invoke the above function.
3) Coding style changes. (I didn't find the indentation rules for function
calls with long parameters (still 4 spaces?), so I just imitated the
indentation style of some existing functions)
Changes on PATCH v4:
1) Coding style changes
Changes on PATCH v3:
1) Add sanity check on vcpuid
2) Add comments on per-domain and per-vcpu functions for libxl
users
Changes on PATCH v2:
1) New data structure (libxl_vcpu_sched_params and libxl_sched_params)
to help per-VCPU settings.
2) sched_rtds_vcpu_get now can return a random subset of the parameters
of the VCPUs of a specific domain.
CC:
CC:
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxl/libxl.c | 314 +---
tools/libxl/libxl.h | 37 ++
tools/libxl/libxl_types.idl | 14 ++
3 files changed, 344 insertions(+), 21 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index bd3aac8..b8d7f07 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -5770,6 +5770,196 @@ static int sched_credit2_domain_set(libxl__gc *gc,
uint32_t domid,
return 0;
}
+static int sched_rtds_validate_params(libxl__gc *gc, int period, int budget)
+{
+int rc;
+
+if (period < 1) {
+LOG(ERROR, "Invalid VCPU period of %d (it should be >= 1)", period);
+rc = ERROR_INVAL;
+goto out;
+}
+
+if (budget < 1) {
+LOG(ERROR, "Invalid VCPU budget of %d (it should be >= 1)", budget);
+rc = ERROR_INVAL;
+goto out;
+}
+
+if (budget > period) {
+LOG(ERROR, "VCPU budget must be smaller than or equal to period, "
+ "but %d > %d", budget, period);
+rc = ERROR_INVAL;
+goto out;
+}
+rc = 0;
+out:
+return rc;
+}
+
+/* Get the RTDS scheduling parameters of vcpu(s) */
+static int sched_rtds_vcpu_get(libxl__gc *gc, uint32_t domid,
+ libxl_vcpu_sched_params *scinfo)
+{
+uint32_t num_vcpus;
+int i, r, rc;
+xc_dominfo_t info;
+struct xen_domctl_schedparam_vcpu *vcpus;
+
+r = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (r < 0) {
+LOGE(ERROR, "getting domain info");
+rc = ERROR_FAIL;
+goto out;
+}
+
+num_vcpus = scinfo->num_vcpus ? scinfo->num_vcpus :
+info.max_vcpu_id + 1;
+
+GCNEW_ARRAY(vcpus, num_vcpus);
+
+if (scinfo->num_vcpus > 0) {
+for (i = 0; i < num_vcpus; i++) {
+if (scinfo->vcpus[i].vcpuid < 0 ||
+scinfo->vcpus[i].vcpuid > info.max_vcpu_id) {
+LOG(ERROR, "VCPU index is out of range, "
+ "valid values are within range from 0 to %d",
+ info.max_vcpu_id);
+rc = ERROR_INVAL;
+goto out;
+}
+vcpus[i].vcpuid = scinfo->vcpus[i].vcpuid;
+}
+} else
+for (i = 0; i < num_vcpus; i++)
+vcpus[i].vcpuid = i;
+
+r = xc_sched_rtds_vcpu_get(CTX->xch, domid, vcpus, num_vcpus);
+if (r != 0) {
+LOGE(ERROR, "getting vcpu sched rtds");
+rc = ERROR_FAIL;
+goto out;
+}
+scinfo->sched = LIBXL_SCHEDULER_RTDS;
+if (scinfo->num_vcpus == 0) {
+scinfo->num_vcpus = num_vcpus;
+scinfo->vcpus = libxl__calloc(NOGC, num_vcpus,
+ sizeof(libxl_sched_params));
+}
+for (i = 0; i < num_vcpus; i++) {
+scinfo->vcpus[i].period = vcpus[i].u.rtds.period;
+scinfo->vcpus[i].budget = vcpus[i].u.rtds.budget;
+scinfo->vcpus[i].vcpuid = vcpus[i].vcpuid;
+}
+rc = 0;
+out:
+return rc;
+}
+
+/* Set the RTDS scheduling parameters of vcpu(s) */
+static int sched_rtds_vcpu_set(libxl__gc *gc, uint32_t domid,
+ const libxl_vcpu_sched_params *scinfo)
+{
+int r, rc;
+int i;
+uint16_t max_vcpuid;
+xc_dominfo_t info;
+struct xen_domctl_schedparam_vcpu *vcpus;
+
+r = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (r < 0) {
+LOGE(ERROR, "getting domain info");
+rc = ERROR_FAIL;
+goto out;
+}
+max_vcpuid = in
Re: [Xen-devel] [PATCH v6 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Mar 15, 2016 at 10:14 PM, Meng Xu wrote: > On Tue, Mar 15, 2016 at 1:22 PM, Chong Li wrote: >> On Tue, Mar 15, 2016 at 11:41 AM, Dario Faggioli >> wrote: >>> On Tue, 2016-03-15 at 11:22 -0500, Chong Li wrote: >>>> On Mon, Mar 14, 2016 at 5:05 AM, Dario Faggioli >>>> wrote: >>>> > >>> >>> We said 'once' and then 'once per domain', but something I'd be fine >>> with (coupled with keeping G_WARNING) would be 'once per operation'. >>> Basically, if a domain has 128 vcpus, and an hypercall tries to set all >>> of them to period=100, budget=50, we just print the warning once. Then, >>> if after a while the sysadmin tries the same again, we again just log >>> once, etc. >>> >>> Doing this seems much easier, as the 'warned' flag could just be a >>> local variable of the hypercall implementation. I'm quite sure that >>> would work if there is not any continuation/re-issueing mechanism in >>> the hypercall in question. BUT in our case there is, so things may be >>> more complicated... :-/ >>> >>> Had you thought about a solution like this already? If no, can you see >>> whether there is a nice and easy way to make something like what I just >>> described above to work in our case? >>> >> How about: >> >> We create a global variable in sched_rt.c: >> /* This variable holds its value through hyerpcall re-issueing. >> * When finding vcpu settings with too low budget or period (e.g, >> 100 us), we print a warning >> * and set this variable "true". No more warnings are printed >> until this variable >> * becomes false. >> */ >> static bool warned; >> Initialize it as "false" in rt_init(). >> In your example, >> we "warned = true" when we find the first vcpu has budget less than >> 100 us. Outside >> of the while loop, we do: >> if ( index == op->u.v.nr_vcpus ) /* no more hypercall re-issueing */ >> warned = false; >> > Hi Chong, > > I don't think creating a global variable just for the warning thing is > a better idea. Even if we do want such a variable, it should only > occur in rt_dom_cntl() function, since it is only used in > rt_dom_cntl(). > Global variable should be used "globally", isn't it. ;-) > You're right. If we define static bool warned; at the beginning of rt_dom_cntl(), do we need to initialize it? If without initialization, I think its default value is "false", which is just what we need. Chong -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v6 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Mar 15, 2016 at 11:41 AM, Dario Faggioli wrote: > On Tue, 2016-03-15 at 11:22 -0500, Chong Li wrote: >> On Mon, Mar 14, 2016 at 5:05 AM, Dario Faggioli >> wrote: >> > > > We said 'once' and then 'once per domain', but something I'd be fine > with (coupled with keeping G_WARNING) would be 'once per operation'. > Basically, if a domain has 128 vcpus, and an hypercall tries to set all > of them to period=100, budget=50, we just print the warning once. Then, > if after a while the sysadmin tries the same again, we again just log > once, etc. > > Doing this seems much easier, as the 'warned' flag could just be a > local variable of the hypercall implementation. I'm quite sure that > would work if there is not any continuation/re-issueing mechanism in > the hypercall in question. BUT in our case there is, so things may be > more complicated... :-/ > > Had you thought about a solution like this already? If no, can you see > whether there is a nice and easy way to make something like what I just > described above to work in our case? > How about: We create a global variable in sched_rt.c: /* This variable holds its value through hyerpcall re-issueing. * When finding vcpu settings with too low budget or period (e.g, 100 us), we print a warning * and set this variable "true". No more warnings are printed until this variable * becomes false. */ static bool warned; Initialize it as "false" in rt_init(). In your example, we "warned = true" when we find the first vcpu has budget less than 100 us. Outside of the while loop, we do: if ( index == op->u.v.nr_vcpus ) /* no more hypercall re-issueing */ warned = false; Chong -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v6 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Mon, Mar 14, 2016 at 5:05 AM, Dario Faggioli wrote: > On Mon, 2016-03-14 at 03:15 -0600, Jan Beulich wrote: >> > > > On 14.03.16 at 10:10, wrote: >> > >> > I think it has some value to have this in the logs. In fact, >> > someone >> > that ended up with small values --either by bug/chance, or in >> > general >> > without a specific need for them-- and is seeing >> > performance/scheduling >> > issues will still be able to find this. >> > >> > And printing it somewhere else than in Xen is impractical (see my >> > reply >> > to Wei). >> > >> > However, we may well print it just once, as soon as the first vcpu >> > with >> > potentially problematic parameters is hit, and then silence it. >> > >> > Linux has things like WARN_ON_ONCE to do this: >> > http://lxr.free-electrons.com/source/include/asm-generic/bug.h#L109 >> > >> > >> > I don't see anything like it in Xen, so you'd need to either >> > implement >> > it, or arrange for something with the same effect locally. >> One of the reasons we don't (yet) have this is because commonly >> (like is the case here) printing such once globally isn't really what >> you want - you'd much rather see it once per domain. >> > Ah, I did not know that was part of the reason, but it makes sense. > > However, for this specific case, I was also considering that, and I > agree, if possible/not to difficult to implement, once per domain would > indeed be better. > It is easy to implement a global "warn_on_once". To implement a per-domain "warn_on_once", I may need a hash table (name it as "dom_warned"), and dom_warned[domid] returns true or false. I don't know if hash table is supported in Xen. Or do we have any list structure which is very good at searching? Another way is to add a "bool" field in struct domain, but I don't think that would be a good design. Any ideas? Thanks, Chong -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v6 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Wed, Mar 9, 2016 at 10:38 AM, Jan Beulich wrote: >>>> On 09.03.16 at 17:10, wrote: >> On Tue, 2016-03-08 at 19:09 +, Wei Liu wrote: >>> On Sun, Mar 06, 2016 at 11:55:55AM -0600, Chong Li wrote: >>> >>> > +spin_lock_irqsave(&prv->lock, flags); >>> > +svc = rt_vcpu(d->vcpu[local_sched.vcpuid]); >>> > +svc->period = period; >>> > +svc->budget = budget; >>> > +spin_unlock_irqrestore(&prv->lock, flags); >>> > + >>> And this locking pattern seems sub-optimal. You might be able to move >>> the lock and unlock outside the while loop? >>> >> Yes, unless I'm missing something, that looks possible to me, and would >> save a lot of acquire/release ping pong on the lock. > > Well, there are guest memory accesses (which may fault) in the > loop body. While this may work right now, I don't think doing so > is a good idea. So I still keep my design here? Dario, Jan and Wei, I almost finished a new version, but since this part is critical for the whole patch, let me summarize the feedbacks here. Please correct me if my understanding is wrong. 1) We don't need "guest_handle_is_null()" check, because null handle could be used in some special cases. And normally, handle is checked by copy_from(to)_guest* functions. 2) In domctl.h, add explain for nr_vcpus, because it is used in both IN and OUT ways. 3) Use printk(XENLOG_G_WARNING ...) here, because of its rate limit feature. 4) Do I still keep the spin_lock inside the loop body? Chong -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v6 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Mar 8, 2016 at 1:09 PM, Wei Liu wrote:
> On Sun, Mar 06, 2016 at 11:55:55AM -0600, Chong Li wrote:
> [...]
>> @@ -1163,6 +1173,96 @@ rt_dom_cntl(
>> }
>> spin_unlock_irqrestore(&prv->lock, flags);
>> break;
>> +case XEN_DOMCTL_SCHEDOP_getvcpuinfo:
>> +
>> +period = MICROSECS(local_sched.s.rtds.period);
>> +budget = MICROSECS(local_sched.s.rtds.budget);
>> +if ( period > RTDS_MAX_PERIOD || budget < RTDS_MIN_BUDGET ||
>> + budget > period || period < RTDS_MIN_PERIOD )
>> +{
>> +rc = -EINVAL;
>> +break;
>> +}
>> +
>> +/*
>> + * We accept period/budget less than 100 us, but will warn
>> users about
>> + * the large scheduling overhead due to it
>> + */
>> +if ( period < MICROSECS(100) || budget < MICROSECS(100) )
>> +printk("Warning: period or budget set to less than 100us.\n"
>> + "This may result in high scheduling overhead.\n");
>> +
>
> I'm not the maintainer, but I think having printk here is bad idea
> because the toolstack can then DoS the hypervisor.
>
>
> Wei.
So what function should I use here? I see many LOG() calls in libxl,
but I'm not sure whether that can be used here.
Chong
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v6 for Xen 4.7 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
On Wed, Mar 9, 2016 at 11:28 AM, Dario Faggioli wrote: > On Tue, 2016-03-08 at 19:12 +, Wei Liu wrote: > >> Overall I think the patch is moving towards the right direction. >> > I also think so. Thanks a lot Wei for the review, BTW. > >> Just >> that there are too many places where indentation need fixing. Please >> fix them in the next iteration. I really don't like holding back >> patches >> just because of indentation issues. >> > Exactly. Trying to elaborate a bit more, this series does not include > any complex algorithm or similar, so one may thing that it is not that > hard to review. But it is indeed complex and hard to review, because > the patches are rather big and because the API and the command line > syntax we want to support is complex. > > If there are too many style issue, any review will likely end up > focusing mostly, if not only, on them, for various reasons. E.g., when > one finds a stile issue, avoiding commenting on it (e.g., because one > wants to focus on "more important" aspects) means risking forgetting > about it and, in the end, letting it hit the repository (if others also > miss it or does the same). Also, we're all used to look at code that > (well, mostly :-D) conforms to coding style, so it's harder to focus on > code that does not. And more. > > Add to this that the most difficult part of the tools side of this > series (like API and data structures) is actually ok. > > So, Chong, for us to be able to quickly and effectively help you > forward, we need to ask you to do your best and get rid of (ideally) > all coding style problems. > > Once that's done, we're not far from calling this all a done deal. :-) > > Thanks and Regards, > Dario Wei and Dario, thanks for your help reviewing this post. I'm now working on the coding style issues and will send out the next version when all problems are figured out. Chong > -- > <> (Raistlin Majere) > ----- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) > -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v6 for Xen 4.7 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Mar 8, 2016 at 1:12 PM, Wei Liu wrote:
> On Sun, Mar 06, 2016 at 11:55:57AM -0600, Chong Li wrote:
> [...]
>> tools/libxl/libxl.c | 326
>>
>> tools/libxl/libxl.h | 37 +
>> tools/libxl/libxl_types.idl | 14 ++
>> 3 files changed, 354 insertions(+), 23 deletions(-)
>>
>> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
>> index bd3aac8..4532e86 100644
>> --- a/tools/libxl/libxl.c
>> +++ b/tools/libxl/libxl.c
>> @@ -5770,6 +5770,207 @@ static int sched_credit2_domain_set(libxl__gc *gc,
>> uint32_t domid,
>> return 0;
>> }
>>
>> +static int sched_rtds_validate_params(libxl__gc *gc, int period,
>> +int budget, uint32_t *sdom_period,
>> +uint32_t *sdom_budget)
>
>
> This is a strange pattern. It does more than what it's name suggests.
>
> It seems this function just returns the vanilla values in period and
> budget. It can be simplified by removing sdom_period and sdom_budget all
> together. Do you agree?
Yes.
>> +
>> +/* Get the RTDS scheduling parameters of vcpu(s) */
>> +static int sched_rtds_vcpu_get(libxl__gc *gc, uint32_t domid,
>> + libxl_vcpu_sched_params *scinfo)
>> +{
>> +uint32_t num_vcpus;
>> +int i, r, rc;
>> +xc_dominfo_t info;
>> +struct xen_domctl_schedparam_vcpu *vcpus;
>> +
>> +r = xc_domain_getinfo(CTX->xch, domid, 1, &info);
>> +if (r < 0) {
>> +LOGE(ERROR, "getting domain info");
>> +rc = ERROR_FAIL;
>> +goto out;
>> +}
>> +
>> +num_vcpus = scinfo->num_vcpus ? scinfo->num_vcpus :
>> +info.max_vcpu_id + 1;
>> +
>> +GCNEW_ARRAY(vcpus, num_vcpus);
>> +
>> +if (scinfo->num_vcpus > 0) {
>> +for (i = 0; i < num_vcpus; i++) {
>> +if (scinfo->vcpus[i].vcpuid < 0 ||
>> +scinfo->vcpus[i].vcpuid > info.max_vcpu_id) {
>> +LOG(ERROR, "VCPU index is out of range, "
>> + "valid values are within range from 0 to %d",
>> + info.max_vcpu_id);
>> +rc = ERROR_INVAL;
>> +goto out;
>> +}
>> +vcpus[i].vcpuid = scinfo->vcpus[i].vcpuid;
>> +}
>> +} else
>> +for (i = 0; i < num_vcpus; i++)
>> +vcpus[i].vcpuid = i;
>> +
>> +r = xc_sched_rtds_vcpu_get(CTX->xch, domid, vcpus, num_vcpus);
>> +if (r != 0) {
>> +LOGE(ERROR, "getting vcpu sched rtds");
>> +rc = ERROR_FAIL;
>> +goto out;
>> +}
>> +scinfo->sched = LIBXL_SCHEDULER_RTDS;
>> +if (scinfo->num_vcpus == 0) {
>> +scinfo->num_vcpus = num_vcpus;
>> +scinfo->vcpus = libxl__calloc(NOGC, num_vcpus,
>> +sizeof(libxl_sched_params));
>> +}
>> +for(i = 0; i < num_vcpus; i++) {
>> +scinfo->vcpus[i].period = vcpus[i].s.rtds.period;
>> +scinfo->vcpus[i].budget = vcpus[i].s.rtds.budget;
>> +scinfo->vcpus[i].vcpuid = vcpus[i].vcpuid;
>> +}
>> +return r;
>
> Just set rc = 0 here?
Ok.
>
>> +out:
>> +return rc;
>> +}
>> +
>> +/* Set the RTDS scheduling parameters of vcpu(s) */
>> +static int sched_rtds_vcpus_params_set(libxl__gc *gc, uint32_t domid,
>> + const libxl_vcpu_sched_params *scinfo)
>> +{
>> +int r, rc;
>> +int i;
>> +uint16_t max_vcpuid;
>> +xc_dominfo_t info;
>> +struct xen_domctl_schedparam_vcpu *vcpus;
>> +uint32_t num_vcpus;
>> +
>> +r = xc_domain_getinfo(CTX->xch, domid, 1, &info);
>> +if (r < 0) {
>> +LOGE(ERROR, "getting domain info");
>> +rc = ERROR_FAIL;
>> +goto out;
>> +}
>> +max_vcpuid = info.max_vcpu_id;
>> +
>> +if (scinfo->num_vcpus > 0) {
>> +num_vcpus = scinfo->num_vcpus;
>> +GCNEW_ARRAY(vcpus, num_vcpus);
>> +for (i = 0; i < num_vcpus; i++) {
>> +if (scinfo->vcpus[i].vcpuid < 0 ||
>> +scinfo->vcpus[i].vcpuid > max_vcpuid) {
>> +LOG(ERROR, "VCPU index is out of ra
Re: [Xen-devel] [PATCH v6 for Xen 4.7 4/4] xl: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Mar 8, 2016 at 1:12 PM, Wei Liu wrote:
> On Sun, Mar 06, 2016 at 11:55:58AM -0600, Chong Li wrote:
> [...]
>> @@ -6222,77 +6359,166 @@ int main_sched_rtds(int argc, char **argv)
>> {
>> const char *dom = NULL;
>> const char *cpupool = NULL;
>> -int period = 0; /* period is in microsecond */
>> -int budget = 0; /* budget is in microsecond */
>> +int *vcpus = (int *)xmalloc(sizeof(int)); /* IDs of VCPUs that change */
>> +int *periods = (int *)xmalloc(sizeof(int)); /* period is in microsecond
>> */
>> +int *budgets = (int *)xmalloc(sizeof(int)); /* budget is in microsecond
>> */
>> +int v_size = 1; /* size of vcpus array */
>> +int p_size = 1; /* size of periods array */
>> +int b_size = 1; /* size of budgets array */
>> +int v_index = 0; /* index in vcpus array */
>> +int p_index =0; /* index in periods array */
>> +int b_index =0; /* index for in budgets array */
>> bool opt_p = false;
>> bool opt_b = false;
>> -int opt, rc;
>> +bool opt_v = false;
>> +bool opt_all = false; /* output per-dom parameters */
>> +int opt, i;
>> +int rc = 0;
>> static struct option opts[] = {
>> {"domain", 1, 0, 'd'},
>> {"period", 1, 0, 'p'},
>> {"budget", 1, 0, 'b'},
>> +{"vcpuid",1, 0, 'v'},
>> {"cpupool", 1, 0, 'c'},
>> COMMON_LONG_OPTS
>> };
>>
>> -SWITCH_FOREACH_OPT(opt, "d:p:b:c:", opts, "sched-rtds", 0) {
>> +SWITCH_FOREACH_OPT(opt, "d:p:b:v:c", opts, "sched-rtds", 0) {
>> case 'd':
>> dom = optarg;
>> break;
>> case 'p':
>> -period = strtol(optarg, NULL, 10);
>> +if (p_index >= p_size) { /* periods array is full */
>> +p_size *= 2;
>> +periods = xrealloc(periods, p_size);
>> +}
>> +periods[p_index++] = strtol(optarg, NULL, 10);
>> opt_p = 1;
>> break;
>> case 'b':
>> -budget = strtol(optarg, NULL, 10);
>> +if (b_index >= b_size) { /* budgets array is full */
>> +b_size *= 2;
>> +budgets = xrealloc(budgets, b_size);
>> +}
>> +budgets[b_index++] = strtol(optarg, NULL, 10);
>> opt_b = 1;
>> break;
>> +case 'v':
>> +if (!strcmp(optarg, "all")) { /* get or set all vcpus of a domain */
>> +opt_all = 1;
>> +break;
>> +}
>> +if (v_index >= v_size) { /* vcpus array is full */
>> +v_size *= 2;
>> +vcpus = xrealloc(vcpus, v_size);
>> +}
>> +vcpus[v_index++] = strtol(optarg, NULL, 10);
>> +opt_v = 1;
>> +break;
>
> I'm still not quite sure why this is written like this. What's the
> expected syntax of this command? The hunk to patch xl manpage is very
> terse...
We have three arrays here, vcpus[], periods[] and budgets[]. If the xl
command is like (more examples at cover letter):
xl sched-rtds -d vm1 -v 3 -p 500 -b 200 -v 1 -p 600 -b 300 (set the
scheduling parameters of two vcpus of vm1)
then, the three arrays would be like:
vcpus: [3, 1]
periods: [500, 600]
budgets: [200, 300]
What makes this code complicated is the size of these three arrays
grows along with the
reading of OPTS. At the beginning, all three arrays have the size for
only one int. When
one array becomes full, we double its size.
Chong
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v6 for Xen 4.7 2/4] libxc: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Mar 8, 2016 at 1:09 PM, Wei Liu wrote: > On Sun, Mar 06, 2016 at 11:55:56AM -0600, Chong Li wrote: >> Add xc_sched_rtds_vcpu_get/set functions to interact with >> Xen to get/set a domain's per-VCPU parameters. >> >> Signed-off-by: Chong Li >> Signed-off-by: Meng Xu >> Signed-off-by: Sisu Xi >> >> --- >> Changes on PATCH v5: >> 1) In xc_sched_rtds_vcpu_get/set, re-issueing the hypercall >> if it is preempted. >> >> Changes on PATCH v4: >> 1) Minor modifications on the function parameters. >> >> Changes on PATCH v2: >> 1) Minor modifications due to the change of struct xen_domctl_scheduler_op. >> >> CC: >> CC: >> CC: >> CC: >> CC: >> CC: >> --- >> tools/libxc/include/xenctrl.h | 16 +++--- >> tools/libxc/xc_rt.c | 68 >> +++ >> 2 files changed, 80 insertions(+), 4 deletions(-) >> >> diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h >> index 01a6dda..9462271 100644 >> --- a/tools/libxc/include/xenctrl.h >> +++ b/tools/libxc/include/xenctrl.h >> @@ -888,11 +888,19 @@ int xc_sched_credit2_domain_get(xc_interface *xch, >> struct xen_domctl_sched_credit2 *sdom); >> >> int xc_sched_rtds_domain_set(xc_interface *xch, >> -uint32_t domid, >> -struct xen_domctl_sched_rtds *sdom); >> + uint32_t domid, >> + struct xen_domctl_sched_rtds *sdom); >> int xc_sched_rtds_domain_get(xc_interface *xch, >> -uint32_t domid, >> -struct xen_domctl_sched_rtds *sdom); >> + uint32_t domid, >> + struct xen_domctl_sched_rtds *sdom); > > Please don't mix fixing indentation with new functionality. Besides, the > new indentation looks wrong. > >> +int xc_sched_rtds_vcpu_set(xc_interface *xch, >> + uint32_t domid, >> + struct xen_domctl_schedparam_vcpu *vcpus, >> + uint32_t num_vcpus); >> +int xc_sched_rtds_vcpu_get(xc_interface *xch, >> + uint32_t domid, >> + struct xen_domctl_schedparam_vcpu *vcpus, >> + uint32_t num_vcpus); >> > > Indentation also looks wrong. Maybe you used tab? Please use spaces > instead. > I'm not using tab. This is just what I'm confused with. The indentation for these four *rtds* functions is the same as the two *credit2* functions above. I can not find the indentation rules for function calls with many / long parameters. Chong -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v6 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Mon, Mar 7, 2016 at 11:53 AM, Dario Faggioli
wrote:
> On Mon, 2016-03-07 at 09:40 -0700, Jan Beulich wrote:
>> > > > On 07.03.16 at 17:28, wrote:
>> > On Mon, Mar 7, 2016 at 6:59 AM, Jan Beulich
>> > wrote:
>> > >
>> > > > @@ -1163,6 +1173,96 @@ rt_dom_cntl(
>> > > >
>> > > > +case XEN_DOMCTL_SCHEDOP_getvcpuinfo:
>> > > > +if ( guest_handle_is_null(op->u.v.vcpus) )
>> > > > +{
>> > > > +rc = -EINVAL;
>
>> > > > +{
>> > > > +rc = -EINVAL;
>> > > > +break;
>> > > > +}
>> > > > +
>> > > > +spin_lock_irqsave(&prv->lock, flags);
>> > > > +svc = rt_vcpu(d->vcpu[local_sched.vcpuid]);
>> > > > +local_sched.s.rtds.budget = svc->budget /
>> > > > MICROSECS(1);
>> > > > +local_sched.s.rtds.period = svc->period /
>> > > > MICROSECS(1);
>> > > > +spin_unlock_irqrestore(&prv->lock, flags);
>> > > > +
>> > > > +if ( __copy_to_guest_offset(op->u.v.vcpus, index,
>> > > > +&local_sched, 1) )
>> > > > +{
>> > > > +rc = -EFAULT;
>> > > > +break;
>> > > > +}
>> > > > +if ( (++index > 0x3f) && hypercall_preempt_check()
>> > > > )
>> > > > +break;
>
>> > > > +}
>> > > > +
>> > > > +if ( !rc && (op->u.v.nr_vcpus != index) )
>> > > > +op->u.v.nr_vcpus = index;
>> > > I don't think the right side of the && is really necessary /
>> > > useful.
>> > The right side is to check whether the vcpus array is fully
>> > processed.
>> > When it is true and no error occurs (rc == 0), we
>> > update op->u.v.nr_vcpus, which is returned to libxc, and helps xc
>> > function figuring out how many un-processed vcpus should
>> > be taken care of in the next hypercall.
>> Just consider what the contents of op->u.v.nr_vcpus is after
>> this piece of code was executed, once with the full conditional,
>> and another time with the right side of the && omitted.
>>
> BTW, Chong, I'm not sure this has to do with what Jan is saying, but
> looking again at XEN_SYSCTL_pcitopoinfo, it looks to me you're missing
> copying nr_vcpus back up to the guest (which is actually what makes
> libxc knows whether all vcpus have been processed or now).
I think by "op->u.v.nr_vcpus = index", we already make the new nr_vcpus
seen by the guest (I've verified it).
In the case XEN_DOMCTL_scheduler_op of do_domctl(),
we make "copyback = 1" after calling sched_adjust(), which means all
fields in op (including
the new nr_vcpus) will be copied to u_domctl (at the end of
do_domctl()). This operation
ensures the new nr_vcpus is copied back up to the guest.
Please correct me if my understanding is wrong.
Chong
>
> Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v6 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Mon, Mar 7, 2016 at 6:59 AM, Jan Beulich wrote:
>>>> On 06.03.16 at 18:55, wrote:
>
>> @@ -1130,23 +1146,17 @@ rt_dom_cntl(
>> unsigned long flags;
>> int rc = 0;
>>
>> +xen_domctl_schedparam_vcpu_t local_sched;
>> +s_time_t period, budget;
>> +uint32_t index = 0;
>> +
>
> There's a stray blank line left ahead of this addition.
>
>> switch ( op->cmd )
>> {
>> -case XEN_DOMCTL_SCHEDOP_getinfo:
>> -if ( d->max_vcpus > 0 )
>> -{
>> -spin_lock_irqsave(&prv->lock, flags);
>> -svc = rt_vcpu(d->vcpu[0]);
>> -op->u.rtds.period = svc->period / MICROSECS(1);
>> -op->u.rtds.budget = svc->budget / MICROSECS(1);
>> -spin_unlock_irqrestore(&prv->lock, flags);
>> -}
>> -else
>> -{
>> -/* If we don't have vcpus yet, let's just return the defaults.
>> */
>> -op->u.rtds.period = RTDS_DEFAULT_PERIOD;
>> -op->u.rtds.budget = RTDS_DEFAULT_BUDGET;
>> -}
>> +case XEN_DOMCTL_SCHEDOP_getinfo: /* return the default parameters */
>> +spin_lock_irqsave(&prv->lock, flags);
>> +op->u.rtds.period = RTDS_DEFAULT_PERIOD / MICROSECS(1);
>> +op->u.rtds.budget = RTDS_DEFAULT_BUDGET / MICROSECS(1);
>> +spin_unlock_irqrestore(&prv->lock, flags);
>> break;
>
> This alters the values returned when d->max_vcpus == 0 - while
> this looks to be intentional, I think calling out such a bug fix in the
> description is a must.
Based on previous discussion, XEN_DOMCTL_SCHEDOP_getinfo only returns
the default parameters,
no matter whether vcpu is created yet or not. But I can absolutely
explain this in the description.
>
>> @@ -1163,6 +1173,96 @@ rt_dom_cntl(
>> }
>> spin_unlock_irqrestore(&prv->lock, flags);
>> break;
>> +case XEN_DOMCTL_SCHEDOP_getvcpuinfo:
>> +if ( guest_handle_is_null(op->u.v.vcpus) )
>> +{
>> +rc = -EINVAL;
>
> Perhaps rather -EFAULT? But then again - what is this check good for
> (considering that it doesn't cover other obviously bad handle values)?
Dario suggested this in the last post, because vcpus is a handle and
needs to be validated.
>> +{
>> +rc = -EINVAL;
>> +break;
>> +}
>> +
>> +spin_lock_irqsave(&prv->lock, flags);
>> +svc = rt_vcpu(d->vcpu[local_sched.vcpuid]);
>> +local_sched.s.rtds.budget = svc->budget / MICROSECS(1);
>> +local_sched.s.rtds.period = svc->period / MICROSECS(1);
>> +spin_unlock_irqrestore(&prv->lock, flags);
>> +
>> +if ( __copy_to_guest_offset(op->u.v.vcpus, index,
>> +&local_sched, 1) )
>> +{
>> +rc = -EFAULT;
>> +break;
>> +}
>> +if ( (++index > 0x3f) && hypercall_preempt_check() )
>> +break;
>
> So how is the caller going to be able to reliably read all vCPU-s'
> information for a guest with more than 64 vCPU-s?
In libxc, we re-issue hypercall if the current one is preempted.
>
>> +}
>> +
>> +if ( !rc && (op->u.v.nr_vcpus != index) )
>> +op->u.v.nr_vcpus = index;
>
> I don't think the right side of the && is really necessary / useful.
The right side is to check whether the vcpus array is fully processed.
When it is true and no error occurs (rc == 0), we
update op->u.v.nr_vcpus, which is returned to libxc, and helps xc
function figuring out how many un-processed vcpus should
be taken care of in the next hypercall.
>
>> +typedef struct xen_domctl_schedparam_vcpu {
>> +union {
>> +xen_domctl_sched_credit_t credit;
>> +xen_domctl_sched_credit2_t credit2;
>> +xen_domctl_sched_rtds_t rtds;
>> +} s;
>
> Please call such unions "u", as done everywhere else.
>
>> +uint16_t vcpuid;
>
> Any particular reason to limit this to 16 bits, when elsewhere
> we commonly use 32 bits for vCPU IDs?
I'll change it.
Thanks for your comments.
Chong
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v6 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
Add XEN_DOMCTL_SCHEDOP_getvcpuinfo and _putvcpuinfo hypercalls
to independently get and set the scheduling parameters of each
vCPU of a domain
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v5:
1) When processing XEN_DOMCTL_SCHEDOP_get/putvcpuinfo, we do
preemption check in a similar way to XEN_SYSCTL_pcitopoinfo
Changes on PATCH v4:
1) Add uint32_t vcpu_index to struct xen_domctl_scheduler_op.
When processing XEN_DOMCTL_SCHEDOP_get/putvcpuinfo, we call
hypercall_preemption_check in case the current hypercall lasts
too long. If we decide to preempt the current hypercall, we record
the index of the most-recent finished vcpu into the vcpu_index of
struct xen_domctl_scheduler_op. So when we resume the hypercall after
preemption, we start processing from the posion specified by vcpu_index,
and don't need to repeat the work that has already been done in the
hypercall before the preemption.
(This design is based on the do_grant_table_op() in grant_table.c)
2) Coding style changes
Changes on PATCH v3:
1) Remove struct xen_domctl_schedparam_t.
2) Change struct xen_domctl_scheduler_op.
3) Check if period/budget is within a validated range
Changes on PATCH v2:
1) Change struct xen_domctl_scheduler_op, for transferring per-vcpu parameters
between libxc and hypervisor.
2) Handler of XEN_DOMCTL_SCHEDOP_getinfo now just returns the default budget and
period values of RTDS scheduler.
3) Handler of XEN_DOMCTL_SCHEDOP_getvcpuinfo now can return a random subset of
the parameters of the VCPUs of a specific domain
CC:
CC:
CC:
CC:
CC:
CC:
---
xen/common/sched_credit.c | 4 ++
xen/common/sched_credit2.c | 4 ++
xen/common/sched_rt.c | 130 +++-
xen/common/schedule.c | 15 -
xen/include/public/domctl.h | 59
5 files changed, 182 insertions(+), 30 deletions(-)
diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
index 0dce790..455c684 100644
--- a/xen/common/sched_credit.c
+++ b/xen/common/sched_credit.c
@@ -1054,6 +1054,10 @@ csched_dom_cntl(
* lock. Runq lock not needed anywhere in here. */
spin_lock_irqsave(&prv->lock, flags);
+if ( op->cmd == XEN_DOMCTL_SCHEDOP_putvcpuinfo ||
+ op->cmd == XEN_DOMCTL_SCHEDOP_getvcpuinfo )
+return -EINVAL;
+
if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
{
op->u.credit.weight = sdom->weight;
diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index 3c49ffa..c3049a0 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -1421,6 +1421,10 @@ csched2_dom_cntl(
* runq lock to update csvcs. */
spin_lock_irqsave(&prv->lock, flags);
+if ( op->cmd == XEN_DOMCTL_SCHEDOP_putvcpuinfo ||
+ op->cmd == XEN_DOMCTL_SCHEDOP_getvcpuinfo )
+return -EINVAL;
+
if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
{
op->u.credit2.weight = sdom->weight;
diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c
index 3f1d047..4fcbf40 100644
--- a/xen/common/sched_rt.c
+++ b/xen/common/sched_rt.c
@@ -86,6 +86,22 @@
#define RTDS_DEFAULT_PERIOD (MICROSECS(1))
#define RTDS_DEFAULT_BUDGET (MICROSECS(4000))
+/*
+ * Max period: max delta of time type, because period is added to the time
+ * a vcpu activates, so this must not overflow.
+ * Min period: 10 us, considering the scheduling overhead (when period is
+ * too low, scheduling is invoked too frequently, causing high overhead).
+ */
+#define RTDS_MAX_PERIOD (STIME_DELTA_MAX)
+#define RTDS_MIN_PERIOD (MICROSECS(10))
+
+/*
+ * Min budget: 10 us, considering the scheduling overhead (when budget is
+ * consumed too fast, scheduling is invoked too frequently, causing
+ * high overhead).
+ */
+#define RTDS_MIN_BUDGET (MICROSECS(10))
+
#define UPDATE_LIMIT_SHIFT 10
#define MAX_SCHEDULE(MILLISECS(1))
/*
@@ -1130,23 +1146,17 @@ rt_dom_cntl(
unsigned long flags;
int rc = 0;
+xen_domctl_schedparam_vcpu_t local_sched;
+s_time_t period, budget;
+uint32_t index = 0;
+
switch ( op->cmd )
{
-case XEN_DOMCTL_SCHEDOP_getinfo:
-if ( d->max_vcpus > 0 )
-{
-spin_lock_irqsave(&prv->lock, flags);
-svc = rt_vcpu(d->vcpu[0]);
-op->u.rtds.period = svc->period / MICROSECS(1);
-op->u.rtds.budget = svc->budget / MICROSECS(1);
-spin_unlock_irqrestore(&prv->lock, flags);
-}
-else
-{
-/* If we don't have vcpus yet, let's just return the defaults. */
-op->u.rtds.period = RTDS_DEFAULT_PERIOD;
-op->u.rtds.budget = RTDS_DEFAULT_BUDGET;
-}
+case XEN_DOMCTL_SCHEDOP_getinfo: /* return the default parameters */
+spin_lock_irqsave(&prv->lock, flags);
+op->u
[Xen-devel] [PATCH v6 for Xen 4.7 4/4] xl: enable per-VCPU parameter settings for RTDS scheduler
Change main_sched_rtds and related output functions to support
per-VCPU settings.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v5:
1) Add sched_vcpu_set_all() for the cases that all vcpus of a
domain need to be changed together.
Changes on PATCH v4:
1) Coding style changes
Changes on PATCH v3:
1) Support commands, e.g., "xl sched-rtds -d vm1" to output the
default scheduling parameters
Changes on PATCH v2:
1) Remove per-domain output functions for RTDS scheduler.
2) Users now use '-v all' to specify all VCPUs.
3) Support outputting a subset of the parameters of the VCPUs
of a specific domain.
4) When setting all VCPUs with the same parameters (by only one
command), no per-domain function is invoked.
CC:
CC:
CC:
CC:
CC:
CC:
---
docs/man/xl.pod.1 | 4 +
tools/libxl/xl_cmdimpl.c | 292 --
tools/libxl/xl_cmdtable.c | 10 +-
3 files changed, 269 insertions(+), 37 deletions(-)
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 4279c7c..f9ff917 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -1051,6 +1051,10 @@ B
Specify domain for which scheduler parameters are to be modified or retrieved.
Mandatory for modifying scheduler parameters.
+=item B<-v VCPUID/all>, B<--vcpuid=VCPUID/all>
+
+Specify vcpu for which scheduler parameters are to be modified or retrieved.
+
=item B<-p PERIOD>, B<--period=PERIOD>
Period of time, in microseconds, over which to replenish the budget.
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 2b6371d..7d5620f 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -5823,6 +5823,52 @@ static int sched_domain_set(int domid, const
libxl_domain_sched_params *scinfo)
return 0;
}
+static int sched_vcpu_get(libxl_scheduler sched, int domid,
+ libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_get(ctx, domid, scinfo);
+if (rc) {
+fprintf(stderr, "libxl_vcpu_sched_params_get failed.\n");
+exit(-1);
+}
+if (scinfo->sched != sched) {
+fprintf(stderr, "libxl_vcpu_sched_params_get returned %s not %s.\n",
+libxl_scheduler_to_string(scinfo->sched),
+libxl_scheduler_to_string(sched));
+return 1;
+}
+
+return 0;
+}
+
+static int sched_vcpu_set(int domid, const libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_set(ctx, domid, scinfo);
+if (rc) {
+fprintf(stderr, "libxl_vcpu_sched_params_set failed.\n");
+exit(-1);
+}
+
+return rc;
+}
+
+static int sched_vcpu_set_all(int domid, const libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_set_all(ctx, domid, scinfo);
+if (rc) {
+fprintf(stderr, "libxl_vcpu_sched_params_set failed.\n");
+exit(-1);
+}
+
+return rc;
+}
+
static int sched_credit_params_set(int poolid, libxl_sched_credit_params
*scinfo)
{
if (libxl_sched_credit_params_set(ctx, poolid, scinfo)) {
@@ -5942,6 +5988,38 @@ static int sched_rtds_domain_output(
return 0;
}
+static int sched_rtds_vcpu_output(
+ int domid, libxl_vcpu_sched_params *scinfo)
+{
+char *domname;
+int rc = 0;
+int i;
+
+if (domid < 0) {
+printf("%-33s %4s %4s %9s %9s\n", "Name", "ID",
+"VCPU", "Period", "Budget");
+return 0;
+}
+
+rc = sched_vcpu_get(LIBXL_SCHEDULER_RTDS, domid, scinfo);
+if (rc)
+goto out;
+
+domname = libxl_domid_to_name(ctx, domid);
+for( i = 0; i < scinfo->num_vcpus; i++ ) {
+printf("%-33s %4d %4d %9"PRIu32" %9"PRIu32"\n",
+domname,
+domid,
+scinfo->vcpus[i].vcpuid,
+scinfo->vcpus[i].period,
+scinfo->vcpus[i].budget);
+}
+free(domname);
+
+out:
+return rc;
+}
+
static int sched_rtds_pool_output(uint32_t poolid)
{
char *poolname;
@@ -6015,6 +6093,65 @@ static int sched_domain_output(libxl_scheduler sched,
int (*output)(int),
return 0;
}
+static int sched_vcpu_output(libxl_scheduler sched,
+ int (*output)(int, libxl_vcpu_sched_params
*),
+ int (*pooloutput)(uint32_t), const char
*cpupool)
+{
+libxl_dominfo *info;
+libxl_cpupoolinfo *poolinfo = NULL;
+uint32_t poolid;
+int nb_domain, n_pools = 0, i, p;
+int rc = 0;
+
+if (cpupool) {
+if (libxl_cpupool_qualifier_to_cpupoolid(ctx, cpupool, &poolid, NULL)
+|| !libxl_cpupoolid_is_valid(ctx, poolid)) {
+fprintf(stderr, "unknown cpupool \'
[Xen-devel] [PATCH v6 for Xen 4.7 2/4] libxc: enable per-VCPU parameter settings for RTDS scheduler
Add xc_sched_rtds_vcpu_get/set functions to interact with
Xen to get/set a domain's per-VCPU parameters.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v5:
1) In xc_sched_rtds_vcpu_get/set, re-issueing the hypercall
if it is preempted.
Changes on PATCH v4:
1) Minor modifications on the function parameters.
Changes on PATCH v2:
1) Minor modifications due to the change of struct xen_domctl_scheduler_op.
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxc/include/xenctrl.h | 16 +++---
tools/libxc/xc_rt.c | 68 +++
2 files changed, 80 insertions(+), 4 deletions(-)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 01a6dda..9462271 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -888,11 +888,19 @@ int xc_sched_credit2_domain_get(xc_interface *xch,
struct xen_domctl_sched_credit2 *sdom);
int xc_sched_rtds_domain_set(xc_interface *xch,
-uint32_t domid,
-struct xen_domctl_sched_rtds *sdom);
+ uint32_t domid,
+ struct xen_domctl_sched_rtds *sdom);
int xc_sched_rtds_domain_get(xc_interface *xch,
-uint32_t domid,
-struct xen_domctl_sched_rtds *sdom);
+ uint32_t domid,
+ struct xen_domctl_sched_rtds *sdom);
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus);
+int xc_sched_rtds_vcpu_get(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus);
int
xc_sched_arinc653_schedule_set(
diff --git a/tools/libxc/xc_rt.c b/tools/libxc/xc_rt.c
index d59e5ce..4be9624 100644
--- a/tools/libxc/xc_rt.c
+++ b/tools/libxc/xc_rt.c
@@ -62,3 +62,71 @@ int xc_sched_rtds_domain_get(xc_interface *xch,
return rc;
}
+
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus)
+{
+int rc = 0;
+unsigned processed = 0;
+DECLARE_DOMCTL;
+DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
+XC_HYPERCALL_BUFFER_BOUNCE_IN);
+
+if ( xc_hypercall_bounce_pre(xch, vcpus) )
+return -1;
+
+domctl.cmd = XEN_DOMCTL_scheduler_op;
+domctl.domain = (domid_t) domid;
+domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
+domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_putvcpuinfo;
+
+while ( processed < num_vcpus )
+{
+domctl.u.scheduler_op.u.v.nr_vcpus = num_vcpus - processed;
+set_xen_guest_handle_offset(domctl.u.scheduler_op.u.v.vcpus, vcpus,
+processed);
+if ( (rc = do_domctl(xch, &domctl)) != 0 )
+break;
+processed += domctl.u.scheduler_op.u.v.nr_vcpus;
+}
+
+xc_hypercall_bounce_post(xch, vcpus);
+
+return rc;
+}
+
+int xc_sched_rtds_vcpu_get(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus)
+{
+int rc;
+unsigned processed = 0;
+DECLARE_DOMCTL;
+DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
+XC_HYPERCALL_BUFFER_BOUNCE_BOTH);
+
+if ( xc_hypercall_bounce_pre(xch, vcpus) )
+return -1;
+
+domctl.cmd = XEN_DOMCTL_scheduler_op;
+domctl.domain = (domid_t) domid;
+domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
+domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_getvcpuinfo;
+
+while ( processed < num_vcpus )
+{
+domctl.u.scheduler_op.u.v.nr_vcpus = num_vcpus - processed;
+set_xen_guest_handle_offset(domctl.u.scheduler_op.u.v.vcpus, vcpus,
+processed);
+if ( (rc = do_domctl(xch, &domctl)) != 0 )
+break;
+processed += domctl.u.scheduler_op.u.v.nr_vcpus;
+}
+
+xc_hypercall_bounce_post(xch, vcpus);
+
+return rc;
+}
--
1.9.1
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v6 for Xen 4.7 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
Add libxl_vcpu_sched_params_get/set and sched_rtds_vcpu_get/set
functions to support per-VCPU settings.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v5:
1) Add a seperate function, sched_rtds_vcpus_params_set_all(), to set
the parameters of all vcpus of a domain.
2) Add libxl_vcpu_sched_params_set_all() to invoke the above function.
3) Coding style changes. (I didn't find the indentation rules for function
calls with long parameters (still 4 spaces?), so I just imitated the
indentation style of some existing functions)
Changes on PATCH v4:
1) Coding style changes
Changes on PATCH v3:
1) Add sanity check on vcpuid
2) Add comments on per-domain and per-vcpu functions for libxl
users
Changes on PATCH v2:
1) New data structure (libxl_vcpu_sched_params and libxl_sched_params)
to help per-VCPU settings.
2) sched_rtds_vcpu_get now can return a random subset of the parameters
of the VCPUs of a specific domain.
CC:
CC:
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxl/libxl.c | 326
tools/libxl/libxl.h | 37 +
tools/libxl/libxl_types.idl | 14 ++
3 files changed, 354 insertions(+), 23 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index bd3aac8..4532e86 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -5770,6 +5770,207 @@ static int sched_credit2_domain_set(libxl__gc *gc,
uint32_t domid,
return 0;
}
+static int sched_rtds_validate_params(libxl__gc *gc, int period,
+int budget, uint32_t *sdom_period,
+uint32_t *sdom_budget)
+{
+int rc = 0;
+if (period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT) {
+if (period < 1) {
+LOG(ERROR, "VCPU period is out of range, "
+ "valid values are larger than or equal to 1");
+rc = ERROR_INVAL; /* error scheduling parameter */
+goto out;
+}
+*sdom_period = period;
+}
+
+if (budget != LIBXL_DOMAIN_SCHED_PARAM_BUDGET_DEFAULT) {
+if (budget < 1) {
+LOG(ERROR, "VCPU budget is not set or out of range, "
+ "valid values are larger than or equal to 1");
+rc = ERROR_INVAL;
+goto out;
+}
+*sdom_budget = budget;
+}
+
+if (*sdom_budget > *sdom_period) {
+LOG(ERROR, "VCPU budget must be smaller than "
+ "or equal to VCPU period");
+rc = ERROR_INVAL;
+}
+out:
+return rc;
+}
+
+/* Get the RTDS scheduling parameters of vcpu(s) */
+static int sched_rtds_vcpu_get(libxl__gc *gc, uint32_t domid,
+ libxl_vcpu_sched_params *scinfo)
+{
+uint32_t num_vcpus;
+int i, r, rc;
+xc_dominfo_t info;
+struct xen_domctl_schedparam_vcpu *vcpus;
+
+r = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (r < 0) {
+LOGE(ERROR, "getting domain info");
+rc = ERROR_FAIL;
+goto out;
+}
+
+num_vcpus = scinfo->num_vcpus ? scinfo->num_vcpus :
+info.max_vcpu_id + 1;
+
+GCNEW_ARRAY(vcpus, num_vcpus);
+
+if (scinfo->num_vcpus > 0) {
+for (i = 0; i < num_vcpus; i++) {
+if (scinfo->vcpus[i].vcpuid < 0 ||
+scinfo->vcpus[i].vcpuid > info.max_vcpu_id) {
+LOG(ERROR, "VCPU index is out of range, "
+ "valid values are within range from 0 to %d",
+ info.max_vcpu_id);
+rc = ERROR_INVAL;
+goto out;
+}
+vcpus[i].vcpuid = scinfo->vcpus[i].vcpuid;
+}
+} else
+for (i = 0; i < num_vcpus; i++)
+vcpus[i].vcpuid = i;
+
+r = xc_sched_rtds_vcpu_get(CTX->xch, domid, vcpus, num_vcpus);
+if (r != 0) {
+LOGE(ERROR, "getting vcpu sched rtds");
+rc = ERROR_FAIL;
+goto out;
+}
+scinfo->sched = LIBXL_SCHEDULER_RTDS;
+if (scinfo->num_vcpus == 0) {
+scinfo->num_vcpus = num_vcpus;
+scinfo->vcpus = libxl__calloc(NOGC, num_vcpus,
+sizeof(libxl_sched_params));
+}
+for(i = 0; i < num_vcpus; i++) {
+scinfo->vcpus[i].period = vcpus[i].s.rtds.period;
+scinfo->vcpus[i].budget = vcpus[i].s.rtds.budget;
+scinfo->vcpus[i].vcpuid = vcpus[i].vcpuid;
+}
+return r;
+out:
+return rc;
+}
+
+/* Set the RTDS scheduling parameters of vcpu(s) */
+static int sched_rtds_vcpus_params_set(libxl__gc *gc, uint32_t domid,
+ const libxl_vcpu_sched_params *scinfo)
+{
+int r, rc;
+int i;
+uint16_t max_vcpuid;
+xc_dominfo_t info;
+struct xen_domctl_schedpara
[Xen-devel] [PATCH v6 for Xen 4.7 0/4] Enable per-VCPU parameter settings for RTDS scheduler
[Goal] The current xl sched-rtds tool can only set the VCPUs of a domain to the same parameter although the scheduler supports VCPUs with different parameters. This patchset is to enable xl sched-rtds tool to configure the VCPUs of a domain with different parameters. This per-VCPU settings can be used in many scenarios. For example, based on Dario's statement in our pervious discussion (http://lists.xen.org/archives/html/xen-devel/2014-09/msg00423.html), if there are two real-time applications, which have different timing requirements, running in a multi-VCPU guest domain, it is beneficial to pin these two applications to two seperate VCPUs with different scheduling parameters. What this patchset includes is a wanted and planned feature for RTDS scheudler(http://wiki.xenproject.org/wiki/RTDS-Based-Scheduler) in Xen 4.7. The interface design of the xl sched-rtds tool is based on Meng's previous discussion with Dario, George and Wei (http://lists.xen.org/archives/html/xen-devel/2015-02/msg02606.html). Basically, there are two main changes: 1) in xl, we create an array that records all VCPUs whose parameters are about to modify or output. 2) in libxl, we receive the array and call different xc functions to handle it. 3) in xen and libxc, we use XEN_DOMCTL_SCHEDOP_getvcpuinfo/putvcpuinfo(introduced by this patchset) as the hypercall for per-VCPU operations(get/set method). [Usage] With this patchset in use, xl sched-rtds tool can: 1) show the budget and period of each VCPU of each domain, by using "xl sched-rtds -v all" command. An example would be like: # xl sched-rtds -v all Cpupool Pool-0: sched=RTDS NameID VCPUPeriodBudget Domain-0 00 1 4000 vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 vm2 20 1 4000 vm2 21 1 4000 Using "xl sched-rtds" will output the default scheduling parameters for each domain. An example would be like: # xl sched-rtds Cpupool Pool-0: sched=RTDS NameIDPeriodBudget Domain-0 0 1 4000 vm1 1 1 4000 vm2 2 1 4000 2) show the budget and period of each VCPU of a specific domain, by using, e.g., "xl sched-rtds -d vm1 -v all" command. The output would be like: # xl sched-rtds -d vm1 -v all NameID VCPUPeriodBudget vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 To show a subset of the parameters of the VCPUs of a specific domain, please use, e.g.,"xl sched-rtds -d vm1 -v 0 -v 3" command. The output would be: # xl sched-rtds -d vm1 -v 0 -v 3 NameID VCPUPeriodBudget vm1 10 300 150 vm1 13 1000 500 Using command, e.g., "xl sched-rtds -d vm1" will output the default scheduling parameters of vm1. An example would be like: # xl sched-rtds -d vm1 NameIDPeriodBudget vm1 1 1 4000 3) Users can set the budget and period of multiple VCPUs of a specific domain with only one command, e.g., "xl sched-rtds -d vm1 -v 0 -p 100 -b 50 -v 3 -p 300 -b 150". Users can set all VCPUs with the same parameters, by one command. e.g., "xl sched-rtds -d vm1 -v all -p 500 -b 250". --- Previous conclusion: On PATCH v4, our concern was about the usage of hypercall_preemption_check and the print of warning message (both in xen). These issues are addressed in this patch. CC: CC: CC: CC: CC: CC: CC: CC: CC: Chong Li (4): xen: enable per-VCPU parameter settings for RTDS scheduler libxc: enable per-VCPU parameter settings for RTDS scheduler libxl: enable per-VCPU parameter settings for RTDS scheduler xl: enable per-VCPU parameter settings for RTDS scheduler docs/man/xl.pod.1 | 4 + tools/libxc/include/xenctrl.h | 16 ++- tools/libxc/xc_rt.c | 68 + tools/libxl/libxl.c | 326 +++--- tools/libxl/libxl.h | 37 + tools/libxl/libxl_types.idl | 14 ++ tools/libxl/xl_cmdimpl.c | 292 - tools/libxl/xl_cmdtable.c | 10 +- xen/comm
Re: [Xen-devel] [PATCH v5 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Feb 9, 2016 at 12:17 PM, Dario Faggioli
wrote:
> On Thu, 2016-02-04 at 16:50 -0600, Chong Li wrote:
>> Add XEN_DOMCTL_SCHEDOP_getvcpuinfo and _putvcpuinfo hypercalls
>> to independently get and set the scheduling parameters of each
>> vCPU of a domain
>>
>> Signed-off-by: Chong Li
>> Signed-off-by: Meng Xu
>> Signed-off-by: Sisu Xi
>>
>> ---
>> Changes on PATCH v4:
>> 1) Add uint32_t vcpu_index to struct xen_domctl_scheduler_op.
>> When processing XEN_DOMCTL_SCHEDOP_get/putvcpuinfo, we call
>> hypercall_preemption_check in case the current hypercall lasts
>> too long. If we decide to preempt the current hypercall, we record
>> the index of the most-recent finished vcpu into the vcpu_index of
>> struct xen_domctl_scheduler_op. So when we resume the hypercall after
>> preemption, we start processing from the posion specified by
>> vcpu_index,
>> and don't need to repeat the work that has already been done in the
>> hypercall before the preemption.
>> (This design is based on the do_grant_table_op() in grant_table.c)
>>
>
>> diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c
>> index 3f1d047..34ae48d 100644
>> --- a/xen/common/sched_rt.c
>> +++ b/xen/common/sched_rt.c
>
>> +for ( index = op->u.v.vcpu_index; index < op->u.v.nr_vcpus;
>> index++ )
>> +{
>> +spin_lock_irqsave(&prv->lock, flags);
>> +if ( copy_from_guest_offset(&local_sched,
>> + op->u.v.vcpus, index, 1) )
>> +{
>> +rc = -EFAULT;
>> +spin_unlock_irqrestore(&prv->lock, flags);
>> +break;
>> +}
>> +if ( local_sched.vcpuid >= d->max_vcpus ||
>> + d->vcpu[local_sched.vcpuid] == NULL )
>> +{
>> +rc = -EINVAL;
>> +spin_unlock_irqrestore(&prv->lock, flags);
>> +break;
>> +}
>> +svc = rt_vcpu(d->vcpu[local_sched.vcpuid]);
>> +period = MICROSECS(local_sched.s.rtds.period);
>> +budget = MICROSECS(local_sched.s.rtds.budget);
>> +if ( period > RTDS_MAX_PERIOD || budget <
>> RTDS_MIN_BUDGET ||
>> + budget > period )
>>
> Isn't checking against RTDS_MIN_PERIOD missing?
Because RTDS_MIN_PERIOD==RTDS_MIN_BUDGET, by checking budget <
RTDS_MIN_BUDGET and budget > period, the checking against
RTDS_MIN_PERIOD is already covered.
>> diff --git a/xen/common/schedule.c b/xen/common/schedule.c
>> index c195129..f4a4032 100644
>> --- a/xen/common/schedule.c
>> +++ b/xen/common/schedule.c
>> @@ -1148,10 +1148,19 @@ long sched_adjust(struct domain *d, struct
>> xen_domctl_scheduler_op *op)
>> if ( ret )
>> return ret;
>>
>> -if ( (op->sched_id != DOM2OP(d)->sched_id) ||
>> - ((op->cmd != XEN_DOMCTL_SCHEDOP_putinfo) &&
>> - (op->cmd != XEN_DOMCTL_SCHEDOP_getinfo)) )
>> +if ( op->sched_id != DOM2OP(d)->sched_id )
>> return -EINVAL;
>> +else
>> + switch ( op->cmd )
>> +{
>> +case XEN_DOMCTL_SCHEDOP_putinfo:
>> +case XEN_DOMCTL_SCHEDOP_getinfo:
>> +case XEN_DOMCTL_SCHEDOP_putvcpuinfo:
>> +case XEN_DOMCTL_SCHEDOP_getvcpuinfo:
>> +break;
>> +default:
>> +return -EINVAL;
>> +}
>>
> Maybe I'm misremembering (in which case, sorry), was using a switch
> like this instead of an if suggested during one of the previous rounds?
>
Yes. In patch v3, Jan suggested that.
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v5 for Xen 4.7 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Feb 9, 2016 at 6:00 AM, Dario Faggioli
wrote:
> On Thu, 2016-02-04 at 16:50 -0600, Chong Li wrote:
>>
>> +{
>> +int rc;
>> +int i;
>> +uint16_t max_vcpuid;
>> +xc_dominfo_t info;
>> +struct xen_domctl_schedparam_vcpu *vcpus;
>> +uint32_t num_vcpus;
>> +
>> +rc = xc_domain_getinfo(CTX->xch, domid, 1, &info);
>> +if (rc < 0) {
>> +LOGE(ERROR, "getting domain info");
>> +return ERROR_FAIL;
>> +}
>> +max_vcpuid = info.max_vcpu_id;
>> +
>> +if (scinfo->num_vcpus > 0) {
>> +num_vcpus = scinfo->num_vcpus;
>> +GCNEW_ARRAY(vcpus, num_vcpus);
>> +for (i = 0; i < num_vcpus; i++) {
>> +if (scinfo->vcpus[i].vcpuid < 0 ||
>> +scinfo->vcpus[i].vcpuid > max_vcpuid) {
>> +LOG(ERROR, "VCPU index is out of range, "
>> + "valid values are within range from 0 to
>> %d",
>> + max_vcpuid);
>> +return ERROR_INVAL;
>> +}
>> +vcpus[i].vcpuid = scinfo->vcpus[i].vcpuid;
>> +
>> +rc = sched_rtds_validate_params(gc,
>> +scinfo->vcpus[i].period, scinfo-
>> >vcpus[i].budget,
>> +&vcpus[i].s.rtds.period,
>> &vcpus[i].s.rtds.budget);
>> +if (rc)
>> +return ERROR_INVAL;
>> +}
>> +} else {
>>
> So, it looks to me that this function can be split in two. One would be
> the actual sched_rtds_vcpus_params_set(), and it will do what is being
> done above here.
>
> The other one would be something like
> sched_rtds_vcpus_params_set_all(), and it will do what is being done
> below here.
>
> About scinfo->num_vcpus, I think it would be fine for
> sched_rtds_vcpus_params_set() to enforce it being > 0, and erroring out
> if not.
>
> On the other hand, in sched_rtds_vcpus_params_set_all(), since the
> semantic is "use this set of params for all vcpus", I think it would be
> fine to enforce scinfo->num_vcpus == 1 (and maybe even
> scinfo.vcpus[0].vcpuid == LIBXL_SCHED_PARAM_VCPU_INDEX_DEFAULT).
>
>
> Now, for external callers (like xl, but also like any other toolstack
> wanting to build on top of libxl).
>
> If you think a 'set all vcpus' function would be useufl (as it is
> probably the case), you can define a libxl API function called
> libxl_vcpus_params_set_all(), doing exactly the same thing that
> libxl_vcpus_params_set() is doing, but calling the
> sched_rtds_vcpus_params_set_all() internal function.
>
> Chong, do you think this could work?
I think it would work. Thanks for this suggestion.
Chong
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v5 for Xen 4.7 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
On Mon, Feb 8, 2016 at 5:07 AM, Wei Liu wrote: > > > [...] > > >> +num_vcpus = max_vcpuid + 1; > > >> +GCNEW_ARRAY(vcpus, num_vcpus); > > >> +if (sched_rtds_validate_params(gc, scinfo->vcpus[0].period, > > >> + scinfo->vcpus[0].budget, > > > > > > This doesn't make sense. You take this path because scinfo->num_vcpus is > > > 0 but now you're dereferencing scinfo->vcpus[0]. Do I miss anything? > > For commands like " xl sched-rtds -d vm1 -v all -p 1000 -b 1000" > > (which sets all vcpus with > > the same scheduling parameters), we pass the budget and period via > > scinfo->vcpus[0]. > > > > I'll add more explanation here. > > No, adding more explanation won't help. > > Let me explain a bit. Libxl is the library that can be used by multiple > applications. Xl is just one of the applications. The other application > that I know of is libvirt. > > So, the incarnation of a particular xl command is of no concern how we > define the semantics of a libxl API. That is, you can come up with an > unambiguous API but still support the same xl command. > > Currently the semantics of this (new?) libxl API seems to be broken, > because you're (ab)using num_vcpus to represent a special case. In > effect you can't really whether the array is empty. When num_vcpus is 0, > you shouldn't dereference vcpus array, at all, because the semantics of > num_vcpus == 0 is that the array is empty. > > Wei. I see. I'll think about re-designing the data structure of libxl_vcpu_sched_params. Chong > > > > > > >> + &vcpus[0].s.rtds.period, > > >> + &vcpus[0].s.rtds.budget)) > > > > > > > -- > > Chong Li > > Department of Computer Science and Engineering > > Washington University in St.louis -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v5 for Xen 4.7 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
I'll fix these coding style issues.
On Fri, Feb 5, 2016 at 8:44 AM, Wei Liu wrote:
> On Thu, Feb 04, 2016 at 04:50:43PM -0600, Chong Li wrote:
>> Add libxl_vcpu_sched_params_get/set and sched_rtds_vcpu_get/set
>> functions to support per-VCPU settings.
>>
>
> I will need Dario or George to review the logic of the code.
>
> If some of the comments below don't make sense, just ask. I'm sure I
> make stupid comments at times.
>
>> Signed-off-by: Chong Li
>> Signed-off-by: Meng Xu
>> Signed-off-by: Sisu Xi
>>
>> ---
>> Changes on PATCH v4:
>> 1) Coding style changes
>>
>> Changes on PATCH v3:
>> 1) Add sanity check on vcpuid
>>
>> 2) Add comments on per-domain and per-vcpu functions for libxl
>> users
>>
>> Changes on PATCH v2:
>> 1) New data structure (libxl_vcpu_sched_params and libxl_sched_params)
>> to help per-VCPU settings.
>>
>> 2) sched_rtds_vcpu_get now can return a random subset of the parameters
>> of the VCPUs of a specific domain.
>>
>> CC:
>> CC:
>> CC:
>> CC:
>> CC:
>> CC:
>> CC:
>> CC:
>> ---
>> tools/libxl/libxl.c | 249
>>
>> tools/libxl/libxl.h | 19
>> tools/libxl/libxl_types.idl | 16 +++
>> 3 files changed, 262 insertions(+), 22 deletions(-)
>>
>> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
>> index bd3aac8..ac4a103 100644
>> --- a/tools/libxl/libxl.c
>> +++ b/tools/libxl/libxl.c
>> +
>> +static int sched_rtds_vcpu_set(libxl__gc *gc, uint32_t domid,
>> + const libxl_vcpu_sched_params *scinfo)
>> +{
>
> Again, please document the semantics of this function.
>
>> +int rc;
>
> int r;
>
> And please use it to store return value from xc_ functions.
>
>> +int i;
>> +uint16_t max_vcpuid;
>> +xc_dominfo_t info;
>> +struct xen_domctl_schedparam_vcpu *vcpus;
>> +uint32_t num_vcpus;
>> +
>> +rc = xc_domain_getinfo(CTX->xch, domid, 1, &info);
>> +if (rc < 0) {
>> +LOGE(ERROR, "getting domain info");
>> +return ERROR_FAIL;
>
> Please use goto style error handling.
>
>> +}
>> +max_vcpuid = info.max_vcpu_id;
>> +
>> +if (scinfo->num_vcpus > 0) {
>> +num_vcpus = scinfo->num_vcpus;
>> +GCNEW_ARRAY(vcpus, num_vcpus);
>> +for (i = 0; i < num_vcpus; i++) {
>> +if (scinfo->vcpus[i].vcpuid < 0 ||
>> +scinfo->vcpus[i].vcpuid > max_vcpuid) {
>> +LOG(ERROR, "VCPU index is out of range, "
>> + "valid values are within range from 0 to %d",
>> + max_vcpuid);
>> +return ERROR_INVAL;
>> +}
>> +vcpus[i].vcpuid = scinfo->vcpus[i].vcpuid;
>> +
>> +rc = sched_rtds_validate_params(gc,
>> +scinfo->vcpus[i].period, scinfo->vcpus[i].budget,
>> +&vcpus[i].s.rtds.period, &vcpus[i].s.rtds.budget);
>> +if (rc)
>> +return ERROR_INVAL;
>> +}
>> +} else {
>> +num_vcpus = max_vcpuid + 1;
>> +GCNEW_ARRAY(vcpus, num_vcpus);
>> +if (sched_rtds_validate_params(gc, scinfo->vcpus[0].period,
>> + scinfo->vcpus[0].budget,
>
> This doesn't make sense. You take this path because scinfo->num_vcpus is
> 0 but now you're dereferencing scinfo->vcpus[0]. Do I miss anything?
For commands like " xl sched-rtds -d vm1 -v all -p 1000 -b 1000"
(which sets all vcpus with
the same scheduling parameters), we pass the budget and period via
scinfo->vcpus[0].
I'll add more explanation here.
>
>> + &vcpus[0].s.rtds.period,
>> + &vcpus[0].s.rtds.budget))
>
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v5 for Xen 4.7 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
Add XEN_DOMCTL_SCHEDOP_getvcpuinfo and _putvcpuinfo hypercalls
to independently get and set the scheduling parameters of each
vCPU of a domain
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v4:
1) Add uint32_t vcpu_index to struct xen_domctl_scheduler_op.
When processing XEN_DOMCTL_SCHEDOP_get/putvcpuinfo, we call
hypercall_preemption_check in case the current hypercall lasts
too long. If we decide to preempt the current hypercall, we record
the index of the most-recent finished vcpu into the vcpu_index of
struct xen_domctl_scheduler_op. So when we resume the hypercall after
preemption, we start processing from the posion specified by vcpu_index,
and don't need to repeat the work that has already been done in the
hypercall before the preemption.
(This design is based on the do_grant_table_op() in grant_table.c)
2) Coding style changes
Changes on PATCH v3:
1) Remove struct xen_domctl_schedparam_t.
2) Change struct xen_domctl_scheduler_op.
3) Check if period/budget is within a validated range
Changes on PATCH v2:
1) Change struct xen_domctl_scheduler_op, for transferring per-vcpu parameters
between libxc and hypervisor.
2) Handler of XEN_DOMCTL_SCHEDOP_getinfo now just returns the default budget and
period values of RTDS scheduler.
3) Handler of XEN_DOMCTL_SCHEDOP_getvcpuinfo now can return a random subset of
the parameters of the VCPUs of a specific domain
CC:
CC:
CC:
CC:
CC:
CC:
---
xen/common/domctl.c | 5 ++
xen/common/sched_credit.c | 4 ++
xen/common/sched_credit2.c | 4 ++
xen/common/sched_rt.c | 127 ++--
xen/common/schedule.c | 15 --
xen/include/public/domctl.h | 57 +++-
6 files changed, 180 insertions(+), 32 deletions(-)
diff --git a/xen/common/domctl.c b/xen/common/domctl.c
index 46b967e..b294221 100644
--- a/xen/common/domctl.c
+++ b/xen/common/domctl.c
@@ -847,9 +847,14 @@ long do_domctl(XEN_GUEST_HANDLE_PARAM(xen_domctl_t)
u_domctl)
}
case XEN_DOMCTL_scheduler_op:
+{
ret = sched_adjust(d, &op->u.scheduler_op);
+if ( ret == -ERESTART )
+ret = hypercall_create_continuation(
+__HYPERVISOR_domctl, "h", u_domctl);
copyback = 1;
break;
+}
case XEN_DOMCTL_getdomaininfo:
{
diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
index 0dce790..455c684 100644
--- a/xen/common/sched_credit.c
+++ b/xen/common/sched_credit.c
@@ -1054,6 +1054,10 @@ csched_dom_cntl(
* lock. Runq lock not needed anywhere in here. */
spin_lock_irqsave(&prv->lock, flags);
+if ( op->cmd == XEN_DOMCTL_SCHEDOP_putvcpuinfo ||
+ op->cmd == XEN_DOMCTL_SCHEDOP_getvcpuinfo )
+return -EINVAL;
+
if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
{
op->u.credit.weight = sdom->weight;
diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index 3c49ffa..c3049a0 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -1421,6 +1421,10 @@ csched2_dom_cntl(
* runq lock to update csvcs. */
spin_lock_irqsave(&prv->lock, flags);
+if ( op->cmd == XEN_DOMCTL_SCHEDOP_putvcpuinfo ||
+ op->cmd == XEN_DOMCTL_SCHEDOP_getvcpuinfo )
+return -EINVAL;
+
if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
{
op->u.credit2.weight = sdom->weight;
diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c
index 3f1d047..34ae48d 100644
--- a/xen/common/sched_rt.c
+++ b/xen/common/sched_rt.c
@@ -86,8 +86,21 @@
#define RTDS_DEFAULT_PERIOD (MICROSECS(1))
#define RTDS_DEFAULT_BUDGET (MICROSECS(4000))
+/*
+ * Max period: max delta of time type
+ * Min period: 100 us, considering the scheduling overhead
+ */
+#define RTDS_MAX_PERIOD (STIME_DELTA_MAX)
+#define RTDS_MIN_PERIOD (MICROSECS(10))
+
+/*
+ * Min budget: 100 us
+ */
+#define RTDS_MIN_BUDGET (MICROSECS(10))
+
#define UPDATE_LIMIT_SHIFT 10
#define MAX_SCHEDULE(MILLISECS(1))
+
/*
* Flags
*/
@@ -1129,26 +1142,18 @@ rt_dom_cntl(
struct vcpu *v;
unsigned long flags;
int rc = 0;
-
+xen_domctl_schedparam_vcpu_t local_sched;
+s_time_t period, budget;
+uint32_t index;
switch ( op->cmd )
{
-case XEN_DOMCTL_SCHEDOP_getinfo:
-if ( d->max_vcpus > 0 )
-{
-spin_lock_irqsave(&prv->lock, flags);
-svc = rt_vcpu(d->vcpu[0]);
-op->u.rtds.period = svc->period / MICROSECS(1);
-op->u.rtds.budget = svc->budget / MICROSECS(1);
-spin_unlock_irqrestore(&prv->lock, flags);
-}
-else
-{
-/* If we don't have vcpus yet, let's just return the defaults. */
-op->u.rtds.period = RTDS_DEFAULT_PERIOD;
-op-&g
[Xen-devel] [PATCH v5 for Xen 4.7 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
Add libxl_vcpu_sched_params_get/set and sched_rtds_vcpu_get/set
functions to support per-VCPU settings.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v4:
1) Coding style changes
Changes on PATCH v3:
1) Add sanity check on vcpuid
2) Add comments on per-domain and per-vcpu functions for libxl
users
Changes on PATCH v2:
1) New data structure (libxl_vcpu_sched_params and libxl_sched_params)
to help per-VCPU settings.
2) sched_rtds_vcpu_get now can return a random subset of the parameters
of the VCPUs of a specific domain.
CC:
CC:
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxl/libxl.c | 249
tools/libxl/libxl.h | 19
tools/libxl/libxl_types.idl | 16 +++
3 files changed, 262 insertions(+), 22 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index bd3aac8..ac4a103 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -5770,6 +5770,151 @@ static int sched_credit2_domain_set(libxl__gc *gc,
uint32_t domid,
return 0;
}
+static int sched_rtds_validate_params(libxl__gc *gc, int period,
+ int budget, uint32_t *sdom_period,
+ uint32_t *sdom_budget)
+{
+if (period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT) {
+if (period < 1) {
+LOG(ERROR, "VCPU period is out of range, "
+ "valid values are larger than or equal to 1");
+return 1; /* error scheduling parameter */
+}
+*sdom_period = period;
+}
+
+if (budget != LIBXL_DOMAIN_SCHED_PARAM_BUDGET_DEFAULT) {
+if (budget < 1) {
+LOG(ERROR, "VCPU budget is not set or out of range, "
+ "valid values are larger than or equal to 1");
+return 1;
+}
+*sdom_budget = budget;
+}
+
+if (budget > period) {
+LOG(ERROR, "VCPU budget must be smaller than "
+ "or equal to VCPU period");
+return 1;
+}
+
+return 0; /* period and budget are valid */
+}
+
+static int sched_rtds_vcpu_get(libxl__gc *gc, uint32_t domid,
+ libxl_vcpu_sched_params *scinfo)
+{
+uint32_t num_vcpus;
+int rc, i;
+xc_dominfo_t info;
+struct xen_domctl_schedparam_vcpu *vcpus;
+
+rc = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (rc < 0) {
+LOGE(ERROR, "getting domain info");
+return ERROR_FAIL;
+}
+
+num_vcpus = scinfo->num_vcpus ? scinfo->num_vcpus :
+info.max_vcpu_id + 1;
+
+GCNEW_ARRAY(vcpus, num_vcpus);
+
+if (scinfo->num_vcpus > 0)
+for (i=0; i < num_vcpus; i++) {
+if (scinfo->vcpus[i].vcpuid < 0 ||
+scinfo->vcpus[i].vcpuid > info.max_vcpu_id) {
+LOG(ERROR, "VCPU index is out of range, "
+ "valid values are within range from 0 to %d",
+ info.max_vcpu_id);
+return ERROR_INVAL;
+}
+vcpus[i].vcpuid = scinfo->vcpus[i].vcpuid;
+} else
+for (i=0; i < num_vcpus; i++)
+vcpus[i].vcpuid = i;
+
+rc = xc_sched_rtds_vcpu_get(CTX->xch, domid, vcpus, num_vcpus);
+if (rc != 0) {
+LOGE(ERROR, "getting vcpu sched rtds");
+return ERROR_FAIL;
+}
+scinfo->sched = LIBXL_SCHEDULER_RTDS;
+if (scinfo->num_vcpus == 0) {
+scinfo->num_vcpus = num_vcpus;
+scinfo->vcpus = libxl__calloc(NOGC, num_vcpus,
+sizeof(libxl_sched_params));
+}
+for(i = 0; i < num_vcpus; i++) {
+scinfo->vcpus[i].period = vcpus[i].s.rtds.period;
+scinfo->vcpus[i].budget = vcpus[i].s.rtds.budget;
+scinfo->vcpus[i].vcpuid = vcpus[i].vcpuid;
+}
+
+return 0;
+}
+
+static int sched_rtds_vcpu_set(libxl__gc *gc, uint32_t domid,
+ const libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+int i;
+uint16_t max_vcpuid;
+xc_dominfo_t info;
+struct xen_domctl_schedparam_vcpu *vcpus;
+uint32_t num_vcpus;
+
+rc = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (rc < 0) {
+LOGE(ERROR, "getting domain info");
+return ERROR_FAIL;
+}
+max_vcpuid = info.max_vcpu_id;
+
+if (scinfo->num_vcpus > 0) {
+num_vcpus = scinfo->num_vcpus;
+GCNEW_ARRAY(vcpus, num_vcpus);
+for (i = 0; i < num_vcpus; i++) {
+if (scinfo->vcpus[i].vcpuid < 0 ||
+scinfo->vcpus[i].vcpuid > max_vcpuid) {
+LOG(ERROR, "VCPU index is out of range, "
+ "valid values are within range from
[Xen-devel] [PATCH v5 for Xen 4.7 2/4] libxc: enable per-VCPU parameter settings for RTDS scheduler
Add xc_sched_rtds_vcpu_get/set functions to interact with
Xen to get/set a domain's per-VCPU parameters.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v4:
1) Minor modifications on the function parameters.
Changes on PATCH v2:
1) Minor modifications due to the change of struct xen_domctl_scheduler_op.
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxc/include/xenctrl.h | 8 +++
tools/libxc/xc_rt.c | 56 +++
2 files changed, 64 insertions(+)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 01a6dda..db13434 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -893,6 +893,14 @@ int xc_sched_rtds_domain_set(xc_interface *xch,
int xc_sched_rtds_domain_get(xc_interface *xch,
uint32_t domid,
struct xen_domctl_sched_rtds *sdom);
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+uint32_t domid,
+struct xen_domctl_schedparam_vcpu *vcpus,
+uint32_t num_vcpus);
+int xc_sched_rtds_vcpu_get(xc_interface *xch,
+uint32_t domid,
+struct xen_domctl_schedparam_vcpu *vcpus,
+uint32_t num_vcpus);
int
xc_sched_arinc653_schedule_set(
diff --git a/tools/libxc/xc_rt.c b/tools/libxc/xc_rt.c
index d59e5ce..cb7bc1a 100644
--- a/tools/libxc/xc_rt.c
+++ b/tools/libxc/xc_rt.c
@@ -62,3 +62,59 @@ int xc_sched_rtds_domain_get(xc_interface *xch,
return rc;
}
+
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus)
+{
+int rc;
+DECLARE_DOMCTL;
+DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
+XC_HYPERCALL_BUFFER_BOUNCE_IN);
+
+if ( xc_hypercall_bounce_pre(xch, vcpus) )
+return -1;
+
+domctl.cmd = XEN_DOMCTL_scheduler_op;
+domctl.domain = (domid_t) domid;
+domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
+domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_putvcpuinfo;
+domctl.u.scheduler_op.u.v.nr_vcpus = num_vcpus;
+set_xen_guest_handle(domctl.u.scheduler_op.u.v.vcpus, vcpus);
+domctl.u.scheduler_op.u.v.vcpu_index = 0;
+
+rc = do_domctl(xch, &domctl);
+
+xc_hypercall_bounce_post(xch, vcpus);
+
+return rc;
+}
+
+int xc_sched_rtds_vcpu_get(xc_interface *xch,
+ uint32_t domid,
+ struct xen_domctl_schedparam_vcpu *vcpus,
+ uint32_t num_vcpus)
+{
+int rc;
+DECLARE_DOMCTL;
+DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
+XC_HYPERCALL_BUFFER_BOUNCE_BOTH);
+
+if ( xc_hypercall_bounce_pre(xch, vcpus) )
+return -1;
+
+domctl.cmd = XEN_DOMCTL_scheduler_op;
+domctl.domain = (domid_t) domid;
+domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
+domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_getvcpuinfo;
+domctl.u.scheduler_op.u.v.nr_vcpus = num_vcpus;
+set_xen_guest_handle(domctl.u.scheduler_op.u.v.vcpus, vcpus);
+domctl.u.scheduler_op.u.v.vcpu_index = 0;
+
+rc = do_domctl(xch, &domctl);
+
+xc_hypercall_bounce_post(xch, vcpus);
+
+return rc;
+}
--
1.9.1
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v5 for Xen 4.7 4/4] xl: enable per-VCPU parameter settings for RTDS scheduler
Change main_sched_rtds and related output functions to support
per-VCPU settings.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v4:
1) Coding style changes
Changes on PATCH v3:
1) Support commands, e.g., "xl sched-rtds -d vm1" to output the
default scheduling parameters
Changes on PATCH v2:
1) Remove per-domain output functions for RTDS scheduler.
2) Users now use '-v all' to specify all VCPUs.
3) Support outputting a subset of the parameters of the VCPUs
of a specific domain.
4) When setting all VCPUs with the same parameters (by only one
command), no per-domain function is invoked.
CC:
CC:
CC:
CC:
CC:
CC:
---
docs/man/xl.pod.1 | 4 +
tools/libxl/xl_cmdimpl.c | 305 --
tools/libxl/xl_cmdtable.c | 10 +-
3 files changed, 281 insertions(+), 38 deletions(-)
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 4279c7c..f9ff917 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -1051,6 +1051,10 @@ B
Specify domain for which scheduler parameters are to be modified or retrieved.
Mandatory for modifying scheduler parameters.
+=item B<-v VCPUID/all>, B<--vcpuid=VCPUID/all>
+
+Specify vcpu for which scheduler parameters are to be modified or retrieved.
+
=item B<-p PERIOD>, B<--period=PERIOD>
Period of time, in microseconds, over which to replenish the budget.
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 2b6371d..b843fa5 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -5823,6 +5823,39 @@ static int sched_domain_set(int domid, const
libxl_domain_sched_params *scinfo)
return 0;
}
+static int sched_vcpu_get(libxl_scheduler sched, int domid,
+libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_get(ctx, domid, scinfo);
+if (rc) {
+fprintf(stderr, "libxl_vcpu_sched_params_get failed.\n");
+exit(-1);
+}
+if (scinfo->sched != sched) {
+fprintf(stderr, "libxl_vcpu_sched_params_get returned %s not %s.\n",
+libxl_scheduler_to_string(scinfo->sched),
+libxl_scheduler_to_string(sched));
+return 1;
+}
+
+return 0;
+}
+
+static int sched_vcpu_set(int domid, const libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_set(ctx, domid, scinfo);
+if (rc) {
+fprintf(stderr, "libxl_vcpu_sched_params_set failed.\n");
+exit(-1);
+}
+
+return rc;
+}
+
static int sched_credit_params_set(int poolid, libxl_sched_credit_params
*scinfo)
{
if (libxl_sched_credit_params_set(ctx, poolid, scinfo)) {
@@ -5942,6 +5975,38 @@ static int sched_rtds_domain_output(
return 0;
}
+static int sched_rtds_vcpu_output(
+int domid, libxl_vcpu_sched_params *scinfo)
+{
+char *domname;
+int rc = 0;
+int i;
+
+if (domid < 0) {
+printf("%-33s %4s %4s %9s %9s\n", "Name", "ID",
+"VCPU", "Period", "Budget");
+return 0;
+}
+
+rc = sched_vcpu_get(LIBXL_SCHEDULER_RTDS, domid, scinfo);
+if (rc)
+goto out;
+
+domname = libxl_domid_to_name(ctx, domid);
+for( i = 0; i < scinfo->num_vcpus; i++ ) {
+printf("%-33s %4d %4d %9"PRIu32" %9"PRIu32"\n",
+domname,
+domid,
+scinfo->vcpus[i].vcpuid,
+scinfo->vcpus[i].period,
+scinfo->vcpus[i].budget);
+}
+free(domname);
+
+out:
+return rc;
+}
+
static int sched_rtds_pool_output(uint32_t poolid)
{
char *poolname;
@@ -6015,6 +6080,65 @@ static int sched_domain_output(libxl_scheduler sched,
int (*output)(int),
return 0;
}
+static int sched_vcpu_output(libxl_scheduler sched,
+ int (*output)(int, libxl_vcpu_sched_params *),
+ int (*pooloutput)(uint32_t), const char
*cpupool)
+{
+libxl_dominfo *info;
+libxl_cpupoolinfo *poolinfo = NULL;
+uint32_t poolid;
+int nb_domain, n_pools = 0, i, p;
+int rc = 0;
+
+if (cpupool) {
+if (libxl_cpupool_qualifier_to_cpupoolid(ctx, cpupool, &poolid, NULL)
+|| !libxl_cpupoolid_is_valid(ctx, poolid)) {
+fprintf(stderr, "unknown cpupool \'%s\'\n", cpupool);
+return -ERROR_FAIL;
+}
+}
+
+info = libxl_list_domain(ctx, &nb_domain);
+if (!info) {
+fprintf(stderr, "libxl_list_domain failed.\n");
+return 1;
+}
+poolinfo = libxl_list_cpupool(ctx, &n_pools);
+if (!poolinfo) {
+fprintf(stderr, "error getting cpupool info\n");
+libxl_dominfo_list_free(info, nb_domain);
+re
[Xen-devel] [PATCH v5 for Xen 4.7 0/4] Enable per-VCPU parameter settings for RTDS scheduler
[Goal] The current xl sched-rtds tool can only set the VCPUs of a domain to the same parameter although the scheduler supports VCPUs with different parameters. This patchset is to enable xl sched-rtds tool to configure the VCPUs of a domain with different parameters. This per-VCPU settings can be used in many scenarios. For example, based on Dario's statement in our pervious discussion (http://lists.xen.org/archives/html/xen-devel/2014-09/msg00423.html), if there are two real-time applications, which have different timing requirements, running in a multi-VCPU guest domain, it is beneficial to pin these two applications to two seperate VCPUs with different scheduling parameters. What this patchset includes is a wanted and planned feature for RTDS scheudler(http://wiki.xenproject.org/wiki/RTDS-Based-Scheduler) in Xen 4.6. The interface design of the xl sched-rtds tool is based on Meng's previous discussion with Dario, George and Wei (http://lists.xen.org/archives/html/xen-devel/2015-02/msg02606.html). Basically, there are two main changes: 1) in xl, we create an array that records all VCPUs whose parameters are about to modify or output. 2) in libxl, we receive the array and call different xc functions to handle it. 3) in xen and libxc, we use XEN_DOMCTL_SCHEDOP_getvcpuinfo/putvcpuinfo(introduced by this patchset) as the hypercall for per-VCPU operations(get/set method). [Usage] With this patchset in use, xl sched-rtds tool can: 1) show the budget and period of each VCPU of each domain, by using "xl sched-rtds -v all" command. An example would be like: # xl sched-rtds -v all Cpupool Pool-0: sched=RTDS NameID VCPUPeriodBudget Domain-0 00 1 4000 vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 vm2 20 1 4000 vm2 21 1 4000 Using "xl sched-rtds" will output the default scheduling parameters for each domain. An example would be like: # xl sched-rtds Cpupool Pool-0: sched=RTDS NameIDPeriodBudget Domain-0 0 1 4000 vm1 1 1 4000 vm2 2 1 4000 2) show the budget and period of each VCPU of a specific domain, by using, e.g., "xl sched-rtds -d vm1 -v all" command. The output would be like: # xl sched-rtds -d vm1 -v all NameID VCPUPeriodBudget vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 To show a subset of the parameters of the VCPUs of a specific domain, please use, e.g.,"xl sched-rtds -d vm1 -v 0 -v 3" command. The output would be: # xl sched-rtds -d vm1 -v 0 -v 3 NameID VCPUPeriodBudget vm1 10 300 150 vm1 13 1000 500 Using command, e.g., "xl sched-rtds -d vm1" will output the default scheduling parameters of vm1. An example would be like: # xl sched-rtds -d vm1 NameIDPeriodBudget vm1 1 1 4000 3) Users can set the budget and period of multiple VCPUs of a specific domain with only one command, e.g., "xl sched-rtds -d vm1 -v 0 -p 100 -b 50 -v 3 -p 300 -b 150". Users can set all VCPUs with the same parameters, by one command. e.g., "xl sched-rtds -d vm1 -v all -p 500 -b 250". --- Previous conclusion: On PATCH v4, our concern was about the usage of hypercall_preemption_check and the print of warning message (both in xen). These issues are addressed in this patch. CC: CC: CC: CC: CC: CC: CC: CC: CC: Chong Li (4): xen: enable per-VCPU parameter settings for RTDS scheduler libxc: enable per-VCPU parameter settings for RTDS scheduler libxl: enable per-VCPU parameter settings for RTDS scheduler xl: enable per-VCPU parameter settings for RTDS scheduler docs/man/xl.pod.1 | 4 + tools/libxc/include/xenctrl.h | 8 ++ tools/libxc/xc_rt.c | 56 tools/libxl/libxl.c | 249 +++--- tools/libxl/libxl.h | 19 +++ tools/libxl/libxl_types.idl | 16 +++ tools/libxl/xl_cmdimpl.c | 305 +- tools/libxl/xl_cmdtable.c | 10 +- xen/common/domctl.c
Re: [Xen-devel] [PATCH v4 for Xen 4.6 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Mon, Jul 27, 2015 at 10:51 AM, Dario Faggioli
wrote:
> On Fri, 2015-07-10 at 23:52 -0500, Chong Li wrote:
>> Add XEN_DOMCTL_SCHEDOP_getvcpuinfo and _putvcpuinfo hypercalls
>> to independently get and set the scheduling parameters of each
>> vCPU of a domain
>>
> I'd add a note about the fact that, for now, this is only supported and
> being implemented for RTDs.
>
>> Signed-off-by: Chong Li
>> Signed-off-by: Meng Xu
>> Signed-off-by: Sisu Xi
>>
>> ---
>> Changes on PATCH v3:
>>
>> 1) Remove struct xen_domctl_schedparam_t.
>>
>> 2) Change struct xen_domctl_scheduler_op.
>>
>> 3) Check if period/budget is within a validated range
>>
> Don't drop the 'Changes on PATCH v2' part, please. That helps having an
> idea of the full history, reminding about previous comments being done,
> etc.
>
>> diff --git a/xen/common/domctl.c b/xen/common/domctl.c
>> index 2a2d203..349f68b 100644
>> --- a/xen/common/domctl.c
>> +++ b/xen/common/domctl.c
>
>> @@ -1162,8 +1176,82 @@ rt_dom_cntl(
>
>> +case XEN_DOMCTL_SCHEDOP_putvcpuinfo:
>> +spin_lock_irqsave(&prv->lock, flags);
>> +for( index = 0; index < op->u.v.nr_vcpus; index++ )
>> +{
>> +if ( copy_from_guest_offset(&local_sched,
>> + op->u.v.vcpus, index, 1) )
>> +{
>> +rc = -EFAULT;
>> +break;
>> +}
>> +if ( local_sched.vcpuid >= d->max_vcpus ||
>> + d->vcpu[local_sched.vcpuid] == NULL )
>> +{
>> +rc = -EINVAL;
>> +break;
>> +}
>> +svc = rt_vcpu(d->vcpu[local_sched.vcpuid]);
>> +period = MICROSECS(local_sched.s.rtds.period);
>> +budget = MICROSECS(local_sched.s.rtds.budget);
>> +if ( period < MICROSECS(10) || period > RTDS_MAX_PERIOD ||
>> + budget < MICROSECS(10) || budget > period )
>> +{
>> +rc = -EINVAL;
>> +break;
>> +}
>> +if ( period < RTDS_MIN_PERIOD || budget < RTDS_MIN_BUDGET )
>> +warn = 1;
>> +
>> +svc->period = period;
>> +svc->budget = budget;
>> +if( hypercall_preempt_check() )
>> +{
>> +rc = -ERESTART;
>> +break;
>> +}
>> +}
>> +spin_unlock_irqrestore(&prv->lock, flags);
>> +break;
>> }
>> -
>> +if ( rc == 0 && warn == 1 ) /* print warning in libxl */
>> +rc = 1;
>> return rc;
>> }
>>
> Mmm.. And where is the documentation about the fact that return value of
> 1 means that the call succeeded, but some upper layer should print a
> warning?
>
> Remember that, when working in Xen, libxl is not the only toolstack one
> should bear in mind.
>
> In any case, even if documented, I don't like it that much... Why can't
> we print a warning here, and just return success?
Do I just use printk here? Is there any other more appropriate print
function in xen?
>
>> diff --git a/xen/common/schedule.c b/xen/common/schedule.c
>> index ecf1545..886e1b5 100644
>> --- a/xen/common/schedule.c
>> +++ b/xen/common/schedule.c
>> @@ -1052,10 +1052,22 @@ long sched_adjust(struct domain *d, struct
>> xen_domctl_scheduler_op *op)
>> if ( ret )
>> return ret;
>>
>> -if ( (op->sched_id != DOM2OP(d)->sched_id) ||
>> - ((op->cmd != XEN_DOMCTL_SCHEDOP_putinfo) &&
>> - (op->cmd != XEN_DOMCTL_SCHEDOP_getinfo)) )
>> +if ( op->sched_id != DOM2OP(d)->sched_id )
>> return -EINVAL;
>> +else
>> +switch ( op->cmd )
>> +{
>> +case XEN_DOMCTL_SCHEDOP_putinfo:
>> +break;
>> +case XEN_DOMCTL_SCHEDOP_getinfo:
>> +break;
>> +case XEN_DOMCTL_SCHEDOP_putvcpuinfo:
>> +break;
>> +case XEN_DOMCTL_SCHEDOP_getvcpuinfo:
>> +break;
>> +default:
>> +return -EINVAL;
>> +}
>>
>> /* NB: the pluggable scheduler code needs to take care
>> * of locking by itself. */
>>
> So, what happens if someone calls XEN_DOMCTL_SCHED
Re: [Xen-devel] [PATCH v4 for Xen 4.6 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
On Mon, Jul 13, 2015 at 3:37 AM, Jan Beulich wrote:
>>>> On 11.07.15 at 06:52, wrote:
>> @@ -1162,8 +1176,82 @@ rt_dom_cntl(
>> }
>> spin_unlock_irqrestore(&prv->lock, flags);
>> break;
>> +case XEN_DOMCTL_SCHEDOP_getvcpuinfo:
>> +spin_lock_irqsave(&prv->lock, flags);
>> +for ( index = 0; index < op->u.v.nr_vcpus; index++ )
>> +{
>> +if ( copy_from_guest_offset(&local_sched,
>> + op->u.v.vcpus, index, 1) )
>> +{
>> +rc = -EFAULT;
>> +break;
>> +}
>> +if ( local_sched.vcpuid >= d->max_vcpus ||
>> + d->vcpu[local_sched.vcpuid] == NULL )
>> +{
>> +rc = -EINVAL;
>> +break;
>> +}
>> +svc = rt_vcpu(d->vcpu[local_sched.vcpuid]);
>> +
>> +local_sched.s.rtds.budget = svc->budget / MICROSECS(1);
>> +local_sched.s.rtds.period = svc->period / MICROSECS(1);
>> +
>> +if ( __copy_to_guest_offset(op->u.v.vcpus, index,
>> +&local_sched, 1) )
>> +{
>> +rc = -EFAULT;
>> +break;
>> +}
>> +if( hypercall_preempt_check() )
>> +{
>> +rc = -ERESTART;
>> +break;
>> +}
>
> I still don't see how this is supposed to work.
I return -ERESTART here, and the upper layer function (do_domctl) will
handle this error code by calling hypercall_create_continuation.
>> +} xen_domctl_schedparam_vcpu_t;
>> +DEFINE_XEN_GUEST_HANDLE(xen_domctl_schedparam_vcpu_t);
>> +
>> /* Set or get info? */
>> #define XEN_DOMCTL_SCHEDOP_putinfo 0
>> #define XEN_DOMCTL_SCHEDOP_getinfo 1
>> +#define XEN_DOMCTL_SCHEDOP_putvcpuinfo 2
>> +#define XEN_DOMCTL_SCHEDOP_getvcpuinfo 3
>> struct xen_domctl_scheduler_op {
>> uint32_t sched_id; /* XEN_SCHEDULER_* */
>> uint32_t cmd; /* XEN_DOMCTL_SCHEDOP_* */
>> union {
>> -struct xen_domctl_sched_sedf {
>> -uint64_aligned_t period;
>> -uint64_aligned_t slice;
>> -uint64_aligned_t latency;
>> -uint32_t extratime;
>> -uint32_t weight;
>> -} sedf;
>> -struct xen_domctl_sched_credit {
>> -uint16_t weight;
>> -uint16_t cap;
>> -} credit;
>> -struct xen_domctl_sched_credit2 {
>> -uint16_t weight;
>> -} credit2;
>> -struct xen_domctl_sched_rtds {
>> -uint32_t period;
>> -uint32_t budget;
>> -} rtds;
>> +xen_domctl_sched_sedf_t sedf;
>> +xen_domctl_sched_credit_t credit;
>> +xen_domctl_sched_credit2_t credit2;
>> +xen_domctl_sched_rtds_t rtds;
>> +struct {
>> +XEN_GUEST_HANDLE_64(xen_domctl_schedparam_vcpu_t) vcpus;
>> + uint16_t nr_vcpus;
>> +} v;
>
> And there's still no explicit padding here at all (nor am I convinced
> that uint16_t is really a good choice for nr_vcpus - uint32_t would
> seem more natural without causing any problems or structure
> growth).
I think the size of union u is equal to the size of
xen_domctl_sched_sedf_t, which is 64*4 bits (if "vcpus" in struct v is
just a pointer).
The "nr_vcpus" is indeed better to be uint32_t. I'll change it in the
next version.
Chong
>
> Jan
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v4 for Xen 4.6 4/4] xl: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Jul 28, 2015 at 4:25 AM, Dario Faggioli
wrote:
> On Fri, 2015-07-10 at 23:52 -0500, Chong Li wrote:
>> Change main_sched_rtds and related output functions to support
>> per-VCPU settings.
>>
> This patch also looks nice. A few comments provided inline.
>
>
>> diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
>> index c858068..da7c8a6 100644
>> --- a/tools/libxl/xl_cmdimpl.c
>> +++ b/tools/libxl/xl_cmdimpl.c
>> @@ -5630,6 +5630,37 @@ static int sched_domain_set(int domid, const
>> libxl_domain_sched_params *scinfo)
>> return rc;
>> }
>>
>> +static int sched_vcpu_get(libxl_scheduler sched, int domid,
>> +libxl_vcpu_sched_params *scinfo)
>> +{
>> +int rc;
>> +
>> +rc = libxl_vcpu_sched_params_get(ctx, domid, scinfo);
>> +if (rc) {
>> +fprintf(stderr, "libxl_vcpu_sched_params_get failed.\n");
>> +return rc;
>> +}
>> +if (scinfo->sched != sched) {
>> +fprintf(stderr, "libxl_vcpu_sched_params_get returned %s not %s.\n",
>> +libxl_scheduler_to_string(scinfo->sched),
>> +libxl_scheduler_to_string(sched));
>> +return ERROR_INVAL;
>>
> We are in xl, so there's no need to use libxl error code. It's probably
> not strictly forbidden either, and xl itself is really inconsistent
> about this, I know.
>
> Personally, I never use them, and I think that using them may be
> confusing, and create even more inconsistency.
>
What error code should I use here? Do I also need to change the
"ERROR_INVAL" in function sched_domain_get?
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v4 for Xen 4.6 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Jul 28, 2015 at 4:15 AM, Dario Faggioli
wrote:
> On Fri, 2015-07-10 at 23:52 -0500, Chong Li wrote:
>
>> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
>> index e9a2d26..9f7f859 100644
>> --- a/tools/libxl/libxl.c
>> +++ b/tools/libxl/libxl.c
>>
>> +static int sched_rtds_validate_params(libxl__gc *gc, int period,
>> + int budget, uint32_t *sdom_period,
>> + uint32_t *sdom_budget)
>> +{
>> +if (period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT) {
>> +if (period < 1) {
>> +LOG(ERROR, "VCPU period is not set or out of range, "
>> + "valid values are larger than or equal to 1");
>>
> That's probably a nit, but, if period is not set, as the error message
> says, it means it stays LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT? In
> which case you do not enter this branch, and you do not print this
> message, do you?
>
> What I mean is that, it looks to me that it's not accurate for the
> message to say "period not set", or am I overlooking something?
If the period is empty (which means its value == 0), we consider it as
"period not set". Actually, I think maybe we can just delete this "not
set" warning, because in xl, we've already done a sanity check to make
sure period is not empty.
>
>> +return 1;
>>
> So, 1 is error, 0 is ok. That's fine, as this is an internal function,
> but please, add a quick doc comment above it.
Sure.
>
>
>> +return 1;
>> +}
>
>> +}
>> +} else {
>> +num_vcpus = max_vcpuid + 1;
>> +GCNEW_ARRAY(vcpus, num_vcpus);
>> +rc = sched_rtds_validate_params(gc,
>> +scinfo->vcpus[0].period, scinfo->vcpus[0].budget,
>> +&vcpus[0].s.rtds.period, &vcpus[0].s.rtds.budget);
>> +if (rc)
>> +return ERROR_INVAL;
>> +for (i = 0; i < num_vcpus; i++) {
>> +vcpus[i].vcpuid = i;
>> +vcpus[i].s.rtds.period = scinfo->vcpus[0].period;
>> +vcpus[i].s.rtds.budget = scinfo->vcpus[0].budget;
>> +}
>> +}
>> +
>> +rc = xc_sched_rtds_vcpu_set(CTX->xch, domid,
>> +vcpus, num_vcpus);
>> +if (rc == 1) {
>> +printf("WARN: too small period or budget may "
>> + "result in large scheduling overhead\n");
>> +rc = 0;
>>
> As said, do the logging in Xen directly, and ditch this.
Do I use "printk" function in Xen to output warning message?
>
>> +} else if (rc != 0) {
>> +LOGE(ERROR, "setting vcpu sched rtds");
>> +return ERROR_FAIL;
>> +}
>> +
>> +return rc;
>> +}
>> diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
>> index a1c5d15..040a248 100644
>> --- a/tools/libxl/libxl.h
>> +++ b/tools/libxl/libxl.h
>> @@ -200,6 +200,16 @@
>> #define LIBXL_HAVE_DEVICETREE_PASSTHROUGH 1
>>
>> /*
>> + * libxl_vcpu_sched_params is used to store per-vcpu params
>> +*/
>> +#define LIBXL_HAVE_VCPU_SCHED_PARAMS 1
>> +
>> +/*
>> + * libxl_sched_params is used to store the array of per-vcpu params
>> +*/
>> +#define LIBXL_HAVE_SCHED_PARAMS 1
>> +
>>
> I may be misremembering, but did not we say that one of these
> LIBXL_HAVE_* was enough?
>
> If we want to have two, I'd much rather have a generic one
> (LIBXL_HAVE_VCPU_SCHED_PARAMS) and one announcing that the feature is
> supported by RTDS (e.g., LIBXL_HAVE_SCHED_RTDS_VCPU_PARAMS).
>
> This way, if/when we'll add support for per-VCPU parameters in Credit,
> we'd add LIBXL_HAVE_SCHED_CREDIT_VCPU_PARAMS.
>
> But that's just an idea, it's best to see what tools maintainers think
> of this.
>
> BTW, be a little more verbose in commenting these macros. Just have a
> look and take inspiration from the already existing ones.
Ok. I'll rewrite it.
Thanks,
Chong
>
> Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v4 for Xen 4.6 2/4] libxc: enable per-VCPU parameter settings for RTDS scheduler
On Mon, Jul 27, 2015 at 11:11 AM, Dario Faggioli
wrote:
> On Fri, 2015-07-10 at 23:52 -0500, Chong Li wrote:
>
>> diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
>> index d1d2ab3..58f1a7a 100644
>> --- a/tools/libxc/include/xenctrl.h
>> +++ b/tools/libxc/include/xenctrl.h
>> @@ -915,6 +915,15 @@ int xc_sched_rtds_domain_get(xc_interface *xch,
>> uint32_t domid,
>> struct xen_domctl_sched_rtds *sdom);
>>
> So, here in the header, you're not changing the prototype of
> xc_sched_rtds_domain_{get,set} to use the typedef.
>
> However...
>
>> +int xc_sched_rtds_vcpu_set(xc_interface *xch,
>> +uint32_t domid,
>> +xen_domctl_schedparam_vcpu_t *vcpus,
>> +uint16_t num_vcpus);
>>
> ... you hare using the typedef for the newly introduced functions
> xc_sched_rtds_vcpu_{get,set}(), and...
>
>> --- a/tools/libxc/xc_rt.c
>> +++ b/tools/libxc/xc_rt.c
>> @@ -27,7 +27,7 @@
>>
>> int xc_sched_rtds_domain_set(xc_interface *xch,
>> uint32_t domid,
>> - struct xen_domctl_sched_rtds *sdom)
>> + xen_domctl_sched_rtds_t *sdom)
>>
> ...you are actually changing the _implementation_ of
> xc_sched_rtds_domain_{get,set}().
>
> That's all rather inconsistent.
You're right. I'll change it.
>
> Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v4 for Xen 4.6 0/4] Enable per-VCPU parameter settings for RTDS scheduler
On Mon, Jul 27, 2015 at 10:14 AM, Dario Faggioli wrote: > On Fri, 2015-07-10 at 23:52 -0500, Chong Li wrote: > > > Or by "just" adding a note before the actual output: > > # xl sched-rtds -d vm1 > Showing per-vm(s) default scheduling parameters, > use `-v' for seeing the actual parameters of each vcpu > NameIDPeriodBudget > vm1 1 1 4000 > > The latter is more verbose, but I think is what I see as more useful > (and, probably, the easier to implement). I agree. I'll add it in the next version. > > Regards, > Dario > -- > <> (Raistlin Majere) > - > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH] Enable per-VCPU parameter settings for RTDS scheduler
On Fri, Jul 24, 2015 at 10:54 AM, Jan Beulich wrote: >>>> On 24.07.15 at 17:39, wrote: >> I sent out a new version of our patchset (enable per-VCPU parameter >> settings for RTDS scheduler) two weeks ago. Can I get some feedbacks >> and comments about it? I'm waiting for these so that I can work out >> the next version soon. > > First of all, such a ping would better be sent as a reply to the original > submission. > > And then, if you mean > http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg02240.html Yes. That's my original submission. > then I replied (without any further reaction) in > http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg02334.html > and considering the feature freeze you probably should be a little > more patient (e.g. ping when the development tree re-opened). Sure, thanks for your reply. > > Jan > -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] [PATCH] Enable per-VCPU parameter settings for RTDS scheduler
Hi, I sent out a new version of our patchset (enable per-VCPU parameter settings for RTDS scheduler) two weeks ago. Can I get some feedbacks and comments about it? I'm waiting for these so that I can work out the next version soon. Best! Chong -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v4 for Xen 4.6 2/4] libxc: enable per-VCPU parameter settings for RTDS scheduler
Add xc_sched_rtds_vcpu_get/set functions to interact with
Xen to get/set a domain's per-VCPU parameters.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxc/include/xenctrl.h | 9 +++
tools/libxc/xc_rt.c | 58 +--
2 files changed, 65 insertions(+), 2 deletions(-)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index d1d2ab3..58f1a7a 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -915,6 +915,15 @@ int xc_sched_rtds_domain_get(xc_interface *xch,
uint32_t domid,
struct xen_domctl_sched_rtds *sdom);
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+uint32_t domid,
+xen_domctl_schedparam_vcpu_t *vcpus,
+uint16_t num_vcpus);
+int xc_sched_rtds_vcpu_get(xc_interface *xch,
+uint32_t domid,
+xen_domctl_schedparam_vcpu_t *vcpus,
+uint16_t num_vcpus);
+
int
xc_sched_arinc653_schedule_set(
xc_interface *xch,
diff --git a/tools/libxc/xc_rt.c b/tools/libxc/xc_rt.c
index b2d1cc5..257a962 100644
--- a/tools/libxc/xc_rt.c
+++ b/tools/libxc/xc_rt.c
@@ -27,7 +27,7 @@
int xc_sched_rtds_domain_set(xc_interface *xch,
uint32_t domid,
- struct xen_domctl_sched_rtds *sdom)
+ xen_domctl_sched_rtds_t *sdom)
{
int rc;
DECLARE_DOMCTL;
@@ -46,7 +46,7 @@ int xc_sched_rtds_domain_set(xc_interface *xch,
int xc_sched_rtds_domain_get(xc_interface *xch,
uint32_t domid,
- struct xen_domctl_sched_rtds *sdom)
+ xen_domctl_sched_rtds_t *sdom)
{
int rc;
DECLARE_DOMCTL;
@@ -63,3 +63,57 @@ int xc_sched_rtds_domain_get(xc_interface *xch,
return rc;
}
+
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+ uint32_t domid,
+ xen_domctl_schedparam_vcpu_t *vcpus,
+ uint16_t num_vcpus)
+{
+int rc;
+DECLARE_DOMCTL;
+DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
+XC_HYPERCALL_BUFFER_BOUNCE_IN);
+
+if ( xc_hypercall_bounce_pre(xch, vcpus) )
+return -1;
+
+domctl.cmd = XEN_DOMCTL_scheduler_op;
+domctl.domain = (domid_t) domid;
+domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
+domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_putvcpuinfo;
+domctl.u.scheduler_op.u.v.nr_vcpus = num_vcpus;
+set_xen_guest_handle(domctl.u.scheduler_op.u.v.vcpus, vcpus);
+
+rc = do_domctl(xch, &domctl);
+
+xc_hypercall_bounce_post(xch, vcpus);
+
+return rc;
+}
+
+int xc_sched_rtds_vcpu_get(xc_interface *xch,
+ uint32_t domid,
+ xen_domctl_schedparam_vcpu_t *vcpus,
+ uint16_t num_vcpus)
+{
+int rc;
+DECLARE_DOMCTL;
+DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
+XC_HYPERCALL_BUFFER_BOUNCE_BOTH);
+
+if ( xc_hypercall_bounce_pre(xch, vcpus) )
+return -1;
+
+domctl.cmd = XEN_DOMCTL_scheduler_op;
+domctl.domain = (domid_t) domid;
+domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
+domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_getvcpuinfo;
+domctl.u.scheduler_op.u.v.nr_vcpus = num_vcpus;
+set_xen_guest_handle(domctl.u.scheduler_op.u.v.vcpus, vcpus);
+
+rc = do_domctl(xch, &domctl);
+
+xc_hypercall_bounce_post(xch, vcpus);
+
+return rc;
+}
--
1.9.1
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v4 for Xen 4.6 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
Add libxl_vcpu_sched_params_get/set and sched_rtds_vcpu_get/set
functions to support per-VCPU settings.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v3:
1) Add sanity check on vcpuid
2) Add comments on per-domain and per-vcpu functions for libxl
users
CC:
CC:
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxl/libxl.c | 265
tools/libxl/libxl.h | 17 +++
tools/libxl/libxl_types.idl | 16 +++
3 files changed, 276 insertions(+), 22 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index e9a2d26..9f7f859 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -5854,6 +5854,156 @@ static int sched_sedf_domain_set(libxl__gc *gc,
uint32_t domid,
return 0;
}
+static int sched_rtds_validate_params(libxl__gc *gc, int period,
+ int budget, uint32_t *sdom_period,
+ uint32_t *sdom_budget)
+{
+if (period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT) {
+if (period < 1) {
+LOG(ERROR, "VCPU period is not set or out of range, "
+ "valid values are larger than or equal to 1");
+return 1;
+}
+*sdom_period = period;
+}
+
+if (budget != LIBXL_DOMAIN_SCHED_PARAM_BUDGET_DEFAULT) {
+if (budget < 1) {
+LOG(ERROR, "VCPU budget is not set or out of range, "
+ "valid values are larger than or equal to 1");
+return 1;
+}
+*sdom_budget = budget;
+}
+
+if (budget > period) {
+LOG(ERROR, "VCPU budget is larger than VCPU period");
+return 1;
+}
+
+return 0;
+}
+
+static int sched_rtds_vcpu_get(libxl__gc *gc, uint32_t domid,
+ libxl_vcpu_sched_params *scinfo)
+{
+uint16_t num_vcpus;
+int rc, i;
+xc_dominfo_t info;
+xen_domctl_schedparam_vcpu_t *vcpus;
+
+rc = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (rc < 0) {
+LOGE(ERROR, "getting domain info");
+return ERROR_FAIL;
+}
+
+if (scinfo->num_vcpus == 0)
+num_vcpus = info.max_vcpu_id + 1;
+else
+num_vcpus = scinfo->num_vcpus;
+
+GCNEW_ARRAY(vcpus, num_vcpus);
+
+if (scinfo->num_vcpus > 0)
+for (i=0; i < num_vcpus; i++) {
+if (scinfo->vcpus[i].vcpuid < 0 ||
+scinfo->vcpus[i].vcpuid > info.max_vcpu_id) {
+LOG(ERROR, "VCPU index is out of range, "
+ "valid values are within range from 0 to %d",
+ info.max_vcpu_id);
+return ERROR_INVAL;
+}
+vcpus[i].vcpuid = scinfo->vcpus[i].vcpuid;
+} else
+for (i=0; i < num_vcpus; i++)
+vcpus[i].vcpuid = i;
+
+rc = xc_sched_rtds_vcpu_get(CTX->xch, domid, vcpus, num_vcpus);
+if (rc != 0) {
+LOGE(ERROR, "getting vcpu sched rtds");
+return ERROR_FAIL;
+}
+
+scinfo->sched = LIBXL_SCHEDULER_RTDS;
+if (scinfo->num_vcpus == 0) {
+scinfo->num_vcpus = num_vcpus;
+scinfo->vcpus = libxl__calloc(NOGC, num_vcpus,
sizeof(libxl_sched_params));
+}
+for(i = 0; i < num_vcpus; i++) {
+scinfo->vcpus[i].period = vcpus[i].s.rtds.period;
+scinfo->vcpus[i].budget = vcpus[i].s.rtds.budget;
+scinfo->vcpus[i].vcpuid = vcpus[i].vcpuid;
+}
+
+return 0;
+}
+
+static int sched_rtds_vcpu_set(libxl__gc *gc, uint32_t domid,
+ const libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+int i;
+uint16_t max_vcpuid;
+xc_dominfo_t info;
+xen_domctl_schedparam_vcpu_t *vcpus;
+int num_vcpus;
+
+rc = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (rc < 0) {
+LOGE(ERROR, "getting domain info");
+return ERROR_FAIL;
+}
+max_vcpuid = info.max_vcpu_id;
+
+if (scinfo->num_vcpus > 0) {
+num_vcpus = scinfo->num_vcpus;
+GCNEW_ARRAY(vcpus, num_vcpus);
+for (i = 0; i < num_vcpus; i++) {
+if (scinfo->vcpus[i].vcpuid < 0 ||
+scinfo->vcpus[i].vcpuid > max_vcpuid) {
+LOG(ERROR, "VCPU index is out of range, "
+ "valid values are within range from 0 to %d",
+ max_vcpuid);
+return ERROR_INVAL;
+}
+vcpus[i].vcpuid = scinfo->vcpus[i].vcpuid;
+
+rc = sched_rtds_validate_params(gc,
+scinfo->vcpus[i].period, scinfo->vcpus[i].budget,
+&vcpus[i].s.rtds.period, &vcpus[i].s.rtds.budget);
+if (rc)
+
[Xen-devel] [PATCH v4 for Xen 4.6 4/4] xl: enable per-VCPU parameter settings for RTDS scheduler
Change main_sched_rtds and related output functions to support
per-VCPU settings.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v3:
1) Support commands, e.g., "xl sched-rtds -d vm1" to output the
default scheduling parameters
CC:
CC:
CC:
CC:
CC:
CC:
---
docs/man/xl.pod.1 | 4 +
tools/libxl/xl_cmdimpl.c | 292 +-
tools/libxl/xl_cmdtable.c | 10 +-
3 files changed, 274 insertions(+), 32 deletions(-)
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 4eb929d..d35e169 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -1093,6 +1093,10 @@ B
Specify domain for which scheduler parameters are to be modified or retrieved.
Mandatory for modifying scheduler parameters.
+=item B<-v VCPUID/all>, B<--vcpuid=VCPUID/all>
+
+Specify vcpu for which scheduler parameters are to be modified or retrieved.
+
=item B<-p PERIOD>, B<--period=PERIOD>
Period of time, in microseconds, over which to replenish the budget.
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index c858068..da7c8a6 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -5630,6 +5630,37 @@ static int sched_domain_set(int domid, const
libxl_domain_sched_params *scinfo)
return rc;
}
+static int sched_vcpu_get(libxl_scheduler sched, int domid,
+libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_get(ctx, domid, scinfo);
+if (rc) {
+fprintf(stderr, "libxl_vcpu_sched_params_get failed.\n");
+return rc;
+}
+if (scinfo->sched != sched) {
+fprintf(stderr, "libxl_vcpu_sched_params_get returned %s not %s.\n",
+libxl_scheduler_to_string(scinfo->sched),
+libxl_scheduler_to_string(sched));
+return ERROR_INVAL;
+}
+
+return 0;
+}
+
+static int sched_vcpu_set(int domid, const libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_set(ctx, domid, scinfo);
+if (rc)
+fprintf(stderr, "libxl_vcpu_sched_params_set failed.\n");
+
+return rc;
+}
+
static int sched_credit_params_set(int poolid, libxl_sched_credit_params
*scinfo)
{
int rc;
@@ -5786,6 +5817,38 @@ out:
return rc;
}
+static int sched_rtds_vcpu_output(
+int domid, libxl_vcpu_sched_params *scinfo)
+{
+char *domname;
+int rc = 0;
+int i;
+
+ if (domid < 0) {
+printf("%-33s %4s %4s %9s %9s\n", "Name", "ID",
+"VCPU", "Period", "Budget");
+return 0;
+}
+
+rc = sched_vcpu_get(LIBXL_SCHEDULER_RTDS, domid, scinfo);
+if (rc)
+goto out;
+
+domname = libxl_domid_to_name(ctx, domid);
+ for( i = 0; i < scinfo->num_vcpus; i++ ) {
+printf("%-33s %4d %4d %9"PRIu32" %9"PRIu32"\n",
+domname,
+domid,
+scinfo->vcpus[i].vcpuid,
+scinfo->vcpus[i].period,
+scinfo->vcpus[i].budget);
+}
+free(domname);
+
+out:
+return rc;
+}
+
static int sched_rtds_pool_output(uint32_t poolid)
{
char *poolname;
@@ -5859,6 +5922,65 @@ static int sched_domain_output(libxl_scheduler sched,
int (*output)(int),
return 0;
}
+static int sched_vcpu_output(libxl_scheduler sched,
+ int (*output)(int, libxl_vcpu_sched_params *),
+ int (*pooloutput)(uint32_t), const char
*cpupool)
+{
+libxl_dominfo *info;
+libxl_cpupoolinfo *poolinfo = NULL;
+uint32_t poolid;
+int nb_domain, n_pools = 0, i, p;
+int rc = 0;
+
+if (cpupool) {
+if (libxl_cpupool_qualifier_to_cpupoolid(ctx, cpupool, &poolid, NULL)
||
+!libxl_cpupoolid_is_valid(ctx, poolid)) {
+fprintf(stderr, "unknown cpupool \'%s\'\n", cpupool);
+return -ERROR_FAIL;
+}
+}
+
+info = libxl_list_domain(ctx, &nb_domain);
+if (!info) {
+fprintf(stderr, "libxl_list_domain failed.\n");
+return 1;
+}
+poolinfo = libxl_list_cpupool(ctx, &n_pools);
+if (!poolinfo) {
+fprintf(stderr, "error getting cpupool info\n");
+libxl_dominfo_list_free(info, nb_domain);
+return -ERROR_NOMEM;
+}
+
+for (p = 0; !rc && (p < n_pools); p++) {
+if ((poolinfo[p].sched != sched) ||
+(cpupool && (poolid != poolinfo[p].poolid)))
+continue;
+
+pooloutput(poolinfo[p].poolid);
+
+libxl_vcpu_sched_params scinfo_out;
+libxl_vcpu_sched_params_init(&scinfo_out);
+output(-1, &scinfo_out);
+libxl_vcpu_sched_params_dispose(&scinfo_out);
+for (i = 0; i < nb_doma
[Xen-devel] [PATCH v4 for Xen 4.6 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
Add XEN_DOMCTL_SCHEDOP_getvcpuinfo and _putvcpuinfo hypercalls
to independently get and set the scheduling parameters of each
vCPU of a domain
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
Changes on PATCH v3:
1) Remove struct xen_domctl_schedparam_t.
2) Change struct xen_domctl_scheduler_op.
3) Check if period/budget is within a validated range
CC:
CC:
CC:
CC:
CC:
CC:
---
xen/common/domctl.c | 3 ++
xen/common/sched_rt.c | 100 +---
xen/common/schedule.c | 18 ++--
xen/include/public/domctl.h | 61 +++
4 files changed, 155 insertions(+), 27 deletions(-)
diff --git a/xen/common/domctl.c b/xen/common/domctl.c
index 2a2d203..349f68b 100644
--- a/xen/common/domctl.c
+++ b/xen/common/domctl.c
@@ -839,6 +839,9 @@ long do_domctl(XEN_GUEST_HANDLE_PARAM(xen_domctl_t)
u_domctl)
case XEN_DOMCTL_scheduler_op:
ret = sched_adjust(d, &op->u.scheduler_op);
+if ( ret == -ERESTART )
+ret = hypercall_create_continuation(
+__HYPERVISOR_domctl, "h", u_domctl);
copyback = 1;
break;
diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c
index 4372486..fed9e09 100644
--- a/xen/common/sched_rt.c
+++ b/xen/common/sched_rt.c
@@ -78,7 +78,6 @@
*vcpu_insert, vcpu_remove, context_saved, __runq_insert
*/
-
/*
* Default parameters:
* Period and budget in default is 10 and 4 ms, respectively
@@ -86,6 +85,18 @@
#define RTDS_DEFAULT_PERIOD (MICROSECS(1))
#define RTDS_DEFAULT_BUDGET (MICROSECS(4000))
+/*
+ * Max period: max delta of time type
+ * Min period: 100 us, considering the scheduling overhead
+ */
+#define RTDS_MAX_PERIOD (STIME_DELTA_MAX)
+#define RTDS_MIN_PERIOD (MICROSECS(100))
+
+/*
+ * Min budget: 100 us
+ */
+#define RTDS_MIN_BUDGET (MICROSECS(100))
+
#define UPDATE_LIMIT_SHIFT 10
#define MAX_SCHEDULE(MILLISECS(1))
/*
@@ -1137,14 +1148,17 @@ rt_dom_cntl(
struct list_head *iter;
unsigned long flags;
int rc = 0;
+int warn = 0;
+xen_domctl_schedparam_vcpu_t local_sched;
+s_time_t period, budget;
+unsigned int index;
switch ( op->cmd )
{
-case XEN_DOMCTL_SCHEDOP_getinfo:
+case XEN_DOMCTL_SCHEDOP_getinfo: /* return the default parameters */
spin_lock_irqsave(&prv->lock, flags);
-svc = list_entry(sdom->vcpu.next, struct rt_vcpu, sdom_elem);
-op->u.rtds.period = svc->period / MICROSECS(1); /* transfer to us */
-op->u.rtds.budget = svc->budget / MICROSECS(1);
+op->u.rtds.period = RTDS_DEFAULT_PERIOD / MICROSECS(1); /* transfer to
us */
+op->u.rtds.budget = RTDS_DEFAULT_BUDGET / MICROSECS(1);
spin_unlock_irqrestore(&prv->lock, flags);
break;
case XEN_DOMCTL_SCHEDOP_putinfo:
@@ -1162,8 +1176,82 @@ rt_dom_cntl(
}
spin_unlock_irqrestore(&prv->lock, flags);
break;
+case XEN_DOMCTL_SCHEDOP_getvcpuinfo:
+spin_lock_irqsave(&prv->lock, flags);
+for ( index = 0; index < op->u.v.nr_vcpus; index++ )
+{
+if ( copy_from_guest_offset(&local_sched,
+ op->u.v.vcpus, index, 1) )
+{
+rc = -EFAULT;
+break;
+}
+if ( local_sched.vcpuid >= d->max_vcpus ||
+ d->vcpu[local_sched.vcpuid] == NULL )
+{
+rc = -EINVAL;
+break;
+}
+svc = rt_vcpu(d->vcpu[local_sched.vcpuid]);
+
+local_sched.s.rtds.budget = svc->budget / MICROSECS(1);
+local_sched.s.rtds.period = svc->period / MICROSECS(1);
+
+if ( __copy_to_guest_offset(op->u.v.vcpus, index,
+&local_sched, 1) )
+{
+rc = -EFAULT;
+break;
+}
+if( hypercall_preempt_check() )
+{
+rc = -ERESTART;
+break;
+}
+}
+spin_unlock_irqrestore(&prv->lock, flags);
+break;
+case XEN_DOMCTL_SCHEDOP_putvcpuinfo:
+spin_lock_irqsave(&prv->lock, flags);
+for( index = 0; index < op->u.v.nr_vcpus; index++ )
+{
+if ( copy_from_guest_offset(&local_sched,
+ op->u.v.vcpus, index, 1) )
+{
+rc = -EFAULT;
+break;
+}
+if ( local_sched.vcpuid >= d->max_vcpus ||
+ d->vcpu[local_sched.vcpuid] == NULL )
+{
+rc = -EINVAL;
+break;
+}
+svc = rt_vcpu(d->vcpu[local_sched.vcpuid]);
+period = MICROSECS(local_sche
[Xen-devel] [PATCH v4 for Xen 4.6 0/4] Enable per-VCPU parameter settings for RTDS scheduler
[Goal] The current xl sched-rtds tool can only set the VCPUs of a domain to the same parameter although the scheduler supports VCPUs with different parameters. This patchset is to enable xl sched-rtds tool to configure the VCPUs of a domain with different parameters. This per-VCPU settings can be used in many scenarios. For example, based on Dario's statement in our pervious discussion (http://lists.xen.org/archives/html/xen-devel/2014-09/msg00423.html), if there are two real-time applications, which have different timing requirements, running in a multi-VCPU guest domain, it is beneficial to pin these two applications to two seperate VCPUs with different scheduling parameters. What this patchset includes is a wanted and planned feature for RTDS scheudler(http://wiki.xenproject.org/wiki/RTDS-Based-Scheduler) in Xen 4.6. The interface design of the xl sched-rtds tool is based on Meng's previous discussion with Dario, George and Wei (http://lists.xen.org/archives/html/xen-devel/2015-02/msg02606.html). Basically, there are two main changes: 1) in xl, we create an array that records all VCPUs whose parameters are about to modify or output. 2) in libxl, we receive the array and call different xc functions to handle it. 3) in xen and libxc, we use XEN_DOMCTL_SCHEDOP_getvcpuinfo/putvcpuinfo(introduced by this patchset) as the hypercall for per-VCPU operations(get/set method). [Usage] With this patchset in use, xl sched-rtds tool can: 1) show the budget and period of each VCPU of each domain, by using "xl sched-rtds -v all" command. An example would be like: # xl sched-rtds -v all Cpupool Pool-0: sched=RTDS NameID VCPUPeriodBudget Domain-0 00 1 4000 vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 vm2 20 1 4000 vm2 21 1 4000 Using "xl sched-rtds" will output the default scheduling parameters for each domain. An example would be like: # xl sched-rtds Cpupool Pool-0: sched=RTDS NameIDPeriodBudget Domain-0 0 1 4000 vm1 1 1 4000 vm2 2 1 4000 2) show the budget and period of each VCPU of a specific domain, by using, e.g., "xl sched-rtds -d vm1 -v all" command. The output would be like: # xl sched-rtds -d vm1 -v all NameID VCPUPeriodBudget vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 To show a subset of the parameters of the VCPUs of a specific domain, please use, e.g.,"xl sched-rtds -d vm1 -v 0 -v 3" command. The output would be: # xl sched-rtds -d vm1 -v 0 -v 3 NameID VCPUPeriodBudget vm1 10 300 150 vm1 13 1000 500 Using command, e.g., "xl sched-rtds -d vm1" will output the default scheduling parameters of vm1. An example would be like: # xl sched-rtds -d vm1 NameIDPeriodBudget vm1 1 1 4000 3) Users can set the budget and period of multiple VCPUs of a specific domain with only one command, e.g., "xl sched-rtds -d vm1 -v 0 -p 100 -b 50 -v 3 -p 300 -b 150". Users can set all VCPUs with the same parameters, by one command. e.g., "xl sched-rtds -d vm1 -v all -p 500 -b 250". --- CC: CC: CC: CC: CC: CC: CC: CC: CC: Chong Li (4): xen: enable per-VCPU parameter settings for RTDS scheduler libxc: enable per-VCPU parameter settings for RTDS scheduler libxl: enable per-VCPU parameter settings for RTDS scheduler xl: enable per-VCPU parameter settings for RTDS scheduler docs/man/xl.pod.1 | 4 + tools/libxc/include/xenctrl.h | 9 ++ tools/libxc/xc_rt.c | 58 - tools/libxl/libxl.c | 265 ++ tools/libxl/libxl.h | 17 +++ tools/libxl/libxl_types.idl | 16 +++ tools/libxl/xl_cmdimpl.c | 292 ++ tools/libxl/xl_cmdtable.c | 10 +- xen/common/domctl.c | 3 + xen/common/sched_rt.c | 100 ++- xen/common/schedule.c | 18 ++- xen/include/public/domctl.h
Re: [Xen-devel] [PATCH v3 for Xen 4.6 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Jul 7, 2015 at 11:23 AM, Dario Faggioli wrote: > On Sun, 2015-06-28 at 21:44 -0500, Chong Li wrote: >> Add libxl_vcpu_sched_params_get/set and sched_rtds_vcpu_get/set functions to >> support >> per-VCPU settings. >> >> >> +/* Per-VCPU parameters*/ >> +#define LIBXL_DOMAIN_SCHED_PARAM_VCPU_INDEX_DEFAULT -1 >> + >> int libxl_domain_sched_params_get(libxl_ctx *ctx, uint32_t domid, >>libxl_domain_sched_params *params); >> int libxl_domain_sched_params_set(libxl_ctx *ctx, uint32_t domid, >>const libxl_domain_sched_params *params); >> +int libxl_vcpu_sched_params_get(libxl_ctx *ctx, uint32_t domid, >> + libxl_vcpu_sched_params *params); >> +int libxl_vcpu_sched_params_set(libxl_ctx *ctx, uint32_t domid, >> + const libxl_vcpu_sched_params *params); >> > Didn't we say that we wanted the fact that now, at least for RTDS, > libxl_domain_sched_params_*() deals with default parameters to be > documented in a comment somewhere? Have I missed it? I'll implement it in the next version. Do both get/set deal with default parameters? I can easily implement it in hypervisor, because default parameters (RTDS_DEFAULT_PERIOD/BUDGET) are just defined there. But for libxl, how do I deal with it without knowing the default values? > > I appreciate that this is happening in lower levels, and that libxl is > just reporting it, but, still, it is something that I would like to find > in the headers of the library, in order to be able to tell the > differences between libxl_vcpu_sched_params_* and > libxl_domain_sched_params_* ... Ian? > > Regards, > Dario > -- > <> (Raistlin Majere) > --------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v3 for Xen 4.6 0/4] Enable per-VCPU parameter settings for RTDS scheduler
On Tue, Jul 7, 2015 at 10:16 AM, Dario Faggioli wrote: > On Sun, 2015-06-28 at 21:44 -0500, Chong Li wrote: > >> [Usage] >> With this patchset in use, xl sched-rtds tool can: >> >> 1) show the budget and period of each VCPU of each domain, by using "xl >> sched-rtds -v all" command. An example would be like: >> > Ok. Thanks for this summary, and for including the actual output. > >> # xl sched-rtds -v all >> Cpupool Pool-0: sched=RTDS >> NameID VCPUPeriodBudget >> Domain-0 00 1 4000 >> vm1 10 300 150 >> vm1 11 400 200 >> vm1 12 1 4000 >> vm1 13 1000 500 >> vm2 20 1 4000 >> vm2 21 1 4000 >> > Right. What happens with just `xl sched-rtds'? >> >> 2) show the budget and period of each VCPU of a specific domain, by using, >> e.g., "xl sched-rtds -d vm1 -v all" command. The output would be like: >> >> # xl sched-rtds -d vm1 -v all >> NameID VCPUPeriodBudget >> vm1 10 300 150 >> vm1 11 400 200 >> vm1 12 1 4000 >> vm1 13 1000 500 >> > Same as above: what happens with just `xl sched-rtds -d vm1'? 'xl sched-rtds ( -d vm1)' will return an error message. As you suggested, I'll make it return per-domain default parameters in the next version. > >> To show a subset of the parameters of the VCPUs of a specific domain, please >> use, e.g.,"xl sched-rtds -d vm1 -v 0 -v 3" command. The output would be: >> >> # xl sched-rtds -d vm1 -v 0 -v 3 >> NameID VCPUPeriodBudget >> vm1 10 300 150 >> vm1 13 1000 500 >> >> >> 3) Users can set the budget and period of multiple VCPUs of a specific domain >> with only one command, e.g., "xl sched-rtds -d vm1 -v 0 -p 100 -b 50 -v 3 -p >> 300 -b 150". >> >> Users can set all VCPUs with the same parameters, by one command. >> e.g., "xl sched-rtds -d vm1 -v all -p 500 -b 250". >> > Forgive me for asking, as I really think it's rather obvious you've done > this, but I guess you've stress tested this by feeding the various > switches with arbitrary bad arguments, and checked that things do not > explode? > > If you've done it, then fine. It's not necessary that you include (all) > the results of that too in here... I just wanted do double check because > this is certainly not the easiest piece of interface we have in xl. Yes, I've done stress test. > > Anyway, I'll give it a go too. > > Regards, > Dario > >> >> --- >> CC: >> CC: >> CC: >> CC: >> CC: >> CC: >> CC: >> CC: >> CC: >> >> >> Chong Li (4): >> xen: enable per-VCPU parameter settings for RTDS scheduler >> libxc: enable per-VCPU parameter settings for RTDS scheduler >> libxl: enable per-VCPU parameter settings for RTDS scheduler >> xl: enable per-VCPU parameter settings for RTDS scheduler >> >> docs/man/xl.pod.1 | 4 + >> tools/libxc/include/xenctrl.h | 9 ++ >> tools/libxc/xc_csched.c | 4 +- >> tools/libxc/xc_csched2.c | 4 +- >> tools/libxc/xc_rt.c | 64 +- >> tools/libxc/xc_sedf.c | 4 +- >> tools/libxl/libxl.c | 209 +++ >> tools/libxl/libxl.h | 17 +++ >> tools/libxl/libxl_types.idl | 16 +++ >> tools/libxl/xl_cmdimpl.c | 284 >> +++--- >> tools/libxl/xl_cmdtable.c | 10 +- >> xen/common/Makefile | 1 - >> xen/common/domctl.c | 3 + >> xen/common/sched_credit.c | 14 +-- >> xen/common/sched_credit2.c| 6 +- >> xen/common/sched_rt.c | 80 +++- >> xen/common/schedule.c | 5 +- >> xen/include/public/domctl.h | 64 +++--- >> 18 files changed, 680 insertions(+), 118 deletions(-) >> > > -- > <> (Raistlin Majere) > - > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v3 for Xen 4.6 4/4] xl: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Jul 7, 2015 at 10:34 AM, Dario Faggioli wrote: > On Sun, 2015-06-28 at 21:44 -0500, Chong Li wrote: >> Change main_sched_rtds and related output functions to support per-VCPU >> settings. >> >> Changes on PATCH v2: >> >> 1) Remove per-domain output functions for RTDS scheduler. >> >> 2) Users now use '-v all' to specify all VCPUs. >> >> 3) Support outputting a subset of the parameters of the VCPUs of a specific >> domain. >> >> 4) When setting all VCPUs with the same parameters (by only one command), no >> per-domain function is invoked. >> >> Signed-off-by: Chong Li >> Signed-off-by: Meng Xu >> Signed-off-by: Sisu Xi >> > I think this patch is fine. I've asked already, when replying to the > cover message, what happens if one uses just: > > # xl sched-rtds > > or > > # xl sched-rtds -d vm1 > > and it looks to me that, instead of getting rid of > sched_rtds_domain_output(), you can leave it there and make it respond > to the above calls, with the effect of returning the default values, as > agreed and implemented. I agree. I'll do that in the next version. Chong > > Dario > -- > <> (Raistlin Majere) > ----- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v3 for Xen 4.6 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
-struct xen_domctl_sched_credit {
>> -uint16_t weight;
>> -uint16_t cap;
>> -} credit;
>> -struct xen_domctl_sched_credit2 {
>> -uint16_t weight;
>> -} credit2;
>> -struct xen_domctl_sched_rtds {
>> -uint32_t period;
>> -uint32_t budget;
>> -} rtds;
>> +xen_domctl_schedparam_t d;
>
> With this type gone I'm not even sure we need to wrap this in
> another union; not doing so would eliminate some of the other
> changes in this patch.
I see your point. Because of xen_domctl_schedparam_vcpu_t, we still
need to define struct xen_domctl_sched_sedf/credit/credit2/rtds
outside of struct xen_domctl_scheduler_op. Then the struct would be
like:
struct xen_domctl_scheduler_op {
uint32_t sched_id; /* XEN_SCHEDULER_* */
uint32_t cmd; /* XEN_DOMCTL_SCHEDOP_* */
union {
struct xen_domctl_sched_sedf sedf;
struct xen_domctl_sched_credit credit;
struct xen_domctl_sched_credit2 credit2;
struct xen_domctl_sched_rtds rtds;
struct {
XEN_GUEST_HANDLE_64(xen_domctl_schedparam_vcpu_t) vcpus;
uint16_t nr_vcpus;
} v;
} u;
};
typedef struct xen_domctl_scheduler_op xen_domctl_scheduler_op_t;
This design is good for compatibility. Dario, what do you think?
Chong
>
> Jan
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v3 for Xen 4.6 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
On Wed, Jul 1, 2015 at 7:50 AM, Dario Faggioli wrote: > On Wed, 2015-07-01 at 09:48 +0100, Ian Campbell wrote: >> On Tue, 2015-06-30 at 17:54 -0700, Meng Xu wrote: >> > 2015-06-30 9:19 GMT-07:00 Ian Campbell : >> > > Note that this field is not the same as the others in this struct, it is >> > > in effect part of the "key" while the others are the "values". >> > > >> > >> > vcpuid in the libxl is not the key but the value. >> > >> > When it is passed into hypervisor, vcpuid acts as the key to identify >> > which VCPU. vcpuid should be be negative when it is passed to >> > hypervisor. >> > >> > Do you have any concerns about assigning the initial value of vcpuid as -1? >> >> libxl needs to do _something_ if it is passed a vcpuid=-1. What is that >> something? >> > Exactly. > > Meng, Chong, maybe what you are missing from Ian's point is the fact > that libxl is a library on top of which to build more advanced > toolstack, it is not just something that the xl command line tool links > to. > > Therefore, it has to work even when it is not xl that makes the API > calls. Actually, I better say that --not only it has to work-- it has to > be easy and comfortable to use from high level components different than > xl. > > The outcome of this reasoning is that, you can't just assume that a > default value does not matter much, because every code path in _xl_ > overrides it. Everything that is in libxl must make sense in libxl > per-se, even without bringing xl into the picture. > > So, in this case, you need to think whether it is convenient > _in_general_, not only wrt xl, to have -1 as default value for vcpuid > and, if yes, what happens if such default is not overridden and libxl > gets to deal with it! > > Hope this helps... If this was not the issue, sorry for the noise. :-) I did some sanity check about the vcpuid in hypervisor. Maybe that's not enough. I'll add other check on vcpuid in libxl (sched_rtds_vcpu_get/set) to ensure vcpuid has a valid value. Chong > > Regards, > Dario > > -- > <> (Raistlin Majere) > - > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v3 for Xen 4.6 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Jun 30, 2015 at 11:19 AM, Ian Campbell wrote: > On Tue, 2015-06-30 at 11:10 -0500, Chong Li wrote: >> On Tue, Jun 30, 2015 at 10:57 AM, Ian Campbell >> wrote: >> > On Tue, 2015-06-30 at 10:42 -0500, Chong Li wrote: >> >> >> @@ -1554,10 +1564,17 @@ int libxl_sched_credit_params_set(libxl_ctx >> >> >> *ctx, uint32_t poolid, >> >> >> #define LIBXL_DOMAIN_SCHED_PARAM_EXTRATIME_DEFAULT -1 >> >> >> #define LIBXL_DOMAIN_SCHED_PARAM_BUDGET_DEFAULT-1 >> >> >> >> >> >> +/* Per-VCPU parameters*/ >> >> >> +#define LIBXL_DOMAIN_SCHED_PARAM_VCPU_INDEX_DEFAULT -1 >> >> > >> >> > What is the effect of passing vcpuid == -1 to this interface? >> >> >> >> I think the right way should be: >> >> >> >> #define LIBXL_SCHED_PARAM_VCPU_INDEX_DEFAULT -1 >> >> >> >> which is the default value for the vcpuid in libxl_sched_params. Even >> >> though libxl_sched_params is used for per-vcpu settings, all the other >> >> fields (except vcpuid) would still use >> >> LIBXL_DOMAIN_SCHED_PARAM_*_DEFAULT values, just for convenience. >> > >> > My question is what does the concept of vcpu==-1 mean in this interface? >> >> We need a default value for vcpuid when initiating libxl_sched_params, >> just like other fields (e.g., budget, weight ...) in the struct. But >> the default value could be others, maybe "0". I'm not sure. > > What will you do if you come across a -1 in this field? What are the > semantics of the interface WRT this field. > > Note that this field is not the same as the others in this struct, it is > in effect part of the "key" while the others are the "values". > For example, if vm1 has 4 VCPUs, and the user types in: "xl rtds -d vm1 -v 1 -v 3" to get (output) the per-vcpu information of VCPU 1 and VCPU 3. In libxl.c, we create an array (say vcpus), which has two elements (each one is a libxl_sched_params). Then we do: vcpus[0].vcpuid = 1; vcpus[1].vcpuid = 3; and all the other fields in each libxl_sched_params keep empty, but will be filled in by the hypercall handler. So vcpuid == -1 only exists during initiation, and it will be definitely given a valid value later. Chong > Ian. > -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v3 for Xen 4.6 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Jun 30, 2015 at 10:57 AM, Ian Campbell wrote: > On Tue, 2015-06-30 at 10:42 -0500, Chong Li wrote: >> >> @@ -1554,10 +1564,17 @@ int libxl_sched_credit_params_set(libxl_ctx *ctx, >> >> uint32_t poolid, >> >> #define LIBXL_DOMAIN_SCHED_PARAM_EXTRATIME_DEFAULT -1 >> >> #define LIBXL_DOMAIN_SCHED_PARAM_BUDGET_DEFAULT-1 >> >> >> >> +/* Per-VCPU parameters*/ >> >> +#define LIBXL_DOMAIN_SCHED_PARAM_VCPU_INDEX_DEFAULT -1 >> > >> > What is the effect of passing vcpuid == -1 to this interface? >> >> I think the right way should be: >> >> #define LIBXL_SCHED_PARAM_VCPU_INDEX_DEFAULT -1 >> >> which is the default value for the vcpuid in libxl_sched_params. Even >> though libxl_sched_params is used for per-vcpu settings, all the other >> fields (except vcpuid) would still use >> LIBXL_DOMAIN_SCHED_PARAM_*_DEFAULT values, just for convenience. > > My question is what does the concept of vcpu==-1 mean in this interface? We need a default value for vcpuid when initiating libxl_sched_params, just like other fields (e.g., budget, weight ...) in the struct. But the default value could be others, maybe "0". I'm not sure. > > Ian. > -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v3 for Xen 4.6 2/4] libxc: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Jun 30, 2015 at 10:32 AM, Ian Campbell wrote: > On Tue, 2015-06-30 at 10:18 -0500, Chong Li wrote: >> On Tue, Jun 30, 2015 at 7:22 AM, Ian Campbell >> wrote: >> > On Sun, 2015-06-28 at 21:44 -0500, Chong Li wrote: >> > >> >> diff --git a/tools/libxc/xc_csched.c b/tools/libxc/xc_csched.c >> >> index 390c645..5a2bdf4 100644 >> >> --- a/tools/libxc/xc_csched.c >> >> +++ b/tools/libxc/xc_csched.c >> >> @@ -36,7 +36,7 @@ xc_sched_credit_domain_set( >> >> domctl.domain = (domid_t) domid; >> >> domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_CREDIT; >> >> domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_putinfo; >> >> -domctl.u.scheduler_op.u.credit = *sdom; >> >> +domctl.u.scheduler_op.u.d.credit = *sdom; >> > >> > I don't see any change to any types in this series, which makes me >> > suspect it is in another patch and that bisectability is therefore not >> > maintained through the series. Each patch in the series needs to build >> > when they are applied one by one. >> > >> > This probably means that at to at least some degree hypercall interface >> > changes need to be done in the same time as the libxc adjustments. You >> > can still keep the _new_ functionality in a separate patch of course (if >> > that makes sense in this case). >> >> Because both per-domain and per-vcpu parameters are now unioned in >> struct xen_domctl_scheduler_op, we need to specify per-domain or >> per-vcpu when using domctl.u.scheduler_op.u (.d means per-domain, .v >> means per-vcpu). This change also happens to the XEN_DOMCTL_SCHEDOP_* >> handlers (of all schedulers), where the domctl.u.scheduler_op.u.* is >> passed to. > > I understand the interface change, but the problem is that the interface > change and the change ot the users of that interface seem to be in > different patches, which breaks bisectability. > > Does this series compile and work at every step? > > i.e. the following combinations of patches from this 4 patch series all > need to build and work successfully: > > Patch #1 > Patch #1+#2 > Patch #1+#2+#3 > Patch #1+#2+#3+#4 > > Is that the case? I see your point. Do you mean the change of struct xen_domctl_scheduler_op, the change of XEN_DOMCTL_SCHEDOP_* handlers (in xen) and the change of libxc (just shown above) should be in the same patch? Should the title of that patch be something like "xen+libxc: enable per-vcpu parameter settings for RTDS scheduler" ? Chong > > Ian. > -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v3 for Xen 4.6 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Jun 30, 2015 at 7:26 AM, Ian Campbell wrote:
> On Sun, 2015-06-28 at 21:44 -0500, Chong Li wrote:
>
>> +GC_INIT(ctx);
>> +libxl_scheduler sched = scinfo->sched;
>> +int ret;
>> +
>> +if (sched == LIBXL_SCHEDULER_UNKNOWN)
>> +sched = libxl__domain_scheduler(gc, domid);
>> +
>> +switch (sched) {
>> +case LIBXL_SCHEDULER_RTDS:
>> +ret = sched_rtds_vcpu_set(gc, domid, scinfo);
>> +break;
>> +default:
>> +LOG(ERROR, "Unknown scheduler");
>
> There are several other known schedulers, I don't think the fact that
> they don't have any need for per-vcpu variables should result in this
> error message, they should get something more specific to that
> situation. Only truly unknown values for the scheduler should result in
> this message.
Right, I'll change it.
>
> [...]
>> /*
>> + * libxl_vcpu_sched_params is used to get/set per-vcpu params
>> +*/
>> +#define LIBXL_HAVE_VCPU_SCHED_PARAMS 1
>> +
>> +/*
>> + * libxl_sched_params is used to store per-vcpu params
>> +*/
>> +#define LIBXL_SCHED_PARAMS 1
>
> Do we need both of these? In particular I'm not sure what the second one
> is indicating has changed.
Sorry for the vague description there, and there is an error there.
The right way should be:
/* libxl_sched_params is introduced to store per-vcpu params. */
#define LIBXL_HAVE_SCHED_PARAMS
/* libxl_vcpu_sched_params is introduced to store the array of
libxl_sched_params (per-vcpu params). */
#define LIBXL_HAVE_VCPU_SCHED_PARAMS
Actually, now I'm not sure if both them should be there, because I've
not found "LIBXL_HAVE_DOMAIN_SCHED_PARAMS" there.
>
>> +
>> +/*
>> * libxl ABI compatibility
>> *
>> * The only guarantee which libxl makes regarding ABI compatibility
>> @@ -1554,10 +1564,17 @@ int libxl_sched_credit_params_set(libxl_ctx *ctx,
>> uint32_t poolid,
>> #define LIBXL_DOMAIN_SCHED_PARAM_EXTRATIME_DEFAULT -1
>> #define LIBXL_DOMAIN_SCHED_PARAM_BUDGET_DEFAULT-1
>>
>> +/* Per-VCPU parameters*/
>> +#define LIBXL_DOMAIN_SCHED_PARAM_VCPU_INDEX_DEFAULT -1
>
> What is the effect of passing vcpuid == -1 to this interface?
I think the right way should be:
#define LIBXL_SCHED_PARAM_VCPU_INDEX_DEFAULT -1
which is the default value for the vcpuid in libxl_sched_params. Even
though libxl_sched_params is used for per-vcpu settings, all the other
fields (except vcpuid) would still use
LIBXL_DOMAIN_SCHED_PARAM_*_DEFAULT values, just for convenience.
Chong
>
> Ian.
>
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v3 for Xen 4.6 2/4] libxc: enable per-VCPU parameter settings for RTDS scheduler
On Tue, Jun 30, 2015 at 7:22 AM, Ian Campbell wrote: > On Sun, 2015-06-28 at 21:44 -0500, Chong Li wrote: >> Add xc_sched_rtds_vcpu_get/set functions to interact with Xen to get/set a >> domain's >> per-VCPU parameters. >> >> Changes on PATCH v2: >> >> 1) Minor modifications due to the change of struct xen_domctl_scheduler_op. > > This intra-version changelog should go after the -- so it doesn't appear > in the final commit message. Ok, I see. > >> diff --git a/tools/libxc/xc_csched.c b/tools/libxc/xc_csched.c >> index 390c645..5a2bdf4 100644 >> --- a/tools/libxc/xc_csched.c >> +++ b/tools/libxc/xc_csched.c >> @@ -36,7 +36,7 @@ xc_sched_credit_domain_set( >> domctl.domain = (domid_t) domid; >> domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_CREDIT; >> domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_putinfo; >> -domctl.u.scheduler_op.u.credit = *sdom; >> +domctl.u.scheduler_op.u.d.credit = *sdom; > > I don't see any change to any types in this series, which makes me > suspect it is in another patch and that bisectability is therefore not > maintained through the series. Each patch in the series needs to build > when they are applied one by one. > > This probably means that at to at least some degree hypercall interface > changes need to be done in the same time as the libxc adjustments. You > can still keep the _new_ functionality in a separate patch of course (if > that makes sense in this case). Because both per-domain and per-vcpu parameters are now unioned in struct xen_domctl_scheduler_op, we need to specify per-domain or per-vcpu when using domctl.u.scheduler_op.u (.d means per-domain, .v means per-vcpu). This change also happens to the XEN_DOMCTL_SCHEDOP_* handlers (of all schedulers), where the domctl.u.scheduler_op.u.* is passed to. Chong > > Ian. > -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v3 for Xen 4.6 2/4] libxc: enable per-VCPU parameter settings for RTDS scheduler
Add xc_sched_rtds_vcpu_get/set functions to interact with Xen to get/set a
domain's
per-VCPU parameters.
Changes on PATCH v2:
1) Minor modifications due to the change of struct xen_domctl_scheduler_op.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxc/include/xenctrl.h | 9 ++
tools/libxc/xc_csched.c | 4 +--
tools/libxc/xc_csched2.c | 4 +--
tools/libxc/xc_rt.c | 64 +++
tools/libxc/xc_sedf.c | 4 +--
5 files changed, 74 insertions(+), 11 deletions(-)
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index d1d2ab3..58f1a7a 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -915,6 +915,15 @@ int xc_sched_rtds_domain_get(xc_interface *xch,
uint32_t domid,
struct xen_domctl_sched_rtds *sdom);
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+uint32_t domid,
+xen_domctl_schedparam_vcpu_t *vcpus,
+uint16_t num_vcpus);
+int xc_sched_rtds_vcpu_get(xc_interface *xch,
+uint32_t domid,
+xen_domctl_schedparam_vcpu_t *vcpus,
+uint16_t num_vcpus);
+
int
xc_sched_arinc653_schedule_set(
xc_interface *xch,
diff --git a/tools/libxc/xc_csched.c b/tools/libxc/xc_csched.c
index 390c645..5a2bdf4 100644
--- a/tools/libxc/xc_csched.c
+++ b/tools/libxc/xc_csched.c
@@ -36,7 +36,7 @@ xc_sched_credit_domain_set(
domctl.domain = (domid_t) domid;
domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_CREDIT;
domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_putinfo;
-domctl.u.scheduler_op.u.credit = *sdom;
+domctl.u.scheduler_op.u.d.credit = *sdom;
return do_domctl(xch, &domctl);
}
@@ -57,7 +57,7 @@ xc_sched_credit_domain_get(
err = do_domctl(xch, &domctl);
if ( err == 0 )
-*sdom = domctl.u.scheduler_op.u.credit;
+*sdom = domctl.u.scheduler_op.u.d.credit;
return err;
}
diff --git a/tools/libxc/xc_csched2.c b/tools/libxc/xc_csched2.c
index 6da6a46..a604278 100644
--- a/tools/libxc/xc_csched2.c
+++ b/tools/libxc/xc_csched2.c
@@ -36,7 +36,7 @@ xc_sched_credit2_domain_set(
domctl.domain = (domid_t) domid;
domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_CREDIT2;
domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_putinfo;
-domctl.u.scheduler_op.u.credit2 = *sdom;
+domctl.u.scheduler_op.u.d.credit2 = *sdom;
return do_domctl(xch, &domctl);
}
@@ -57,7 +57,7 @@ xc_sched_credit2_domain_get(
err = do_domctl(xch, &domctl);
if ( err == 0 )
-*sdom = domctl.u.scheduler_op.u.credit2;
+*sdom = domctl.u.scheduler_op.u.d.credit2;
return err;
}
diff --git a/tools/libxc/xc_rt.c b/tools/libxc/xc_rt.c
index b2d1cc5..ee139b1 100644
--- a/tools/libxc/xc_rt.c
+++ b/tools/libxc/xc_rt.c
@@ -27,7 +27,7 @@
int xc_sched_rtds_domain_set(xc_interface *xch,
uint32_t domid,
- struct xen_domctl_sched_rtds *sdom)
+ xen_domctl_sched_rtds_t *sdom)
{
int rc;
DECLARE_DOMCTL;
@@ -36,8 +36,8 @@ int xc_sched_rtds_domain_set(xc_interface *xch,
domctl.domain = (domid_t) domid;
domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
domctl.u.scheduler_op.cmd = XEN_DOMCTL_SCHEDOP_putinfo;
-domctl.u.scheduler_op.u.rtds.period = sdom->period;
-domctl.u.scheduler_op.u.rtds.budget = sdom->budget;
+domctl.u.scheduler_op.u.d.rtds.period = sdom->period;
+domctl.u.scheduler_op.u.d.rtds.budget = sdom->budget;
rc = do_domctl(xch, &domctl);
@@ -46,7 +46,7 @@ int xc_sched_rtds_domain_set(xc_interface *xch,
int xc_sched_rtds_domain_get(xc_interface *xch,
uint32_t domid,
- struct xen_domctl_sched_rtds *sdom)
+ xen_domctl_sched_rtds_t *sdom)
{
int rc;
DECLARE_DOMCTL;
@@ -59,7 +59,61 @@ int xc_sched_rtds_domain_get(xc_interface *xch,
rc = do_domctl(xch, &domctl);
if ( rc == 0 )
-*sdom = domctl.u.scheduler_op.u.rtds;
+*sdom = domctl.u.scheduler_op.u.d.rtds;
+
+return rc;
+}
+
+int xc_sched_rtds_vcpu_set(xc_interface *xch,
+ uint32_t domid,
+ xen_domctl_schedparam_vcpu_t *vcpus,
+ uint16_t num_vcpus)
+{
+int rc;
+DECLARE_DOMCTL;
+DECLARE_HYPERCALL_BOUNCE(vcpus, sizeof(*vcpus) * num_vcpus,
+XC_HYPERCALL_BUFFER_BOUNCE_IN);
+
+if ( xc_hypercall_bounce_pre(xch, vcpus) )
+return -1;
+
+domctl.cmd = XEN_DOMCTL_scheduler_op;
+domctl.domain = (domid_t) domid;
+domctl.u.scheduler_op.sched_id = XEN_SCHEDULER_RTDS;
+domctl.u.schedule
[Xen-devel] [PATCH v3 for Xen 4.6 4/4] xl: enable per-VCPU parameter settings for RTDS scheduler
Change main_sched_rtds and related output functions to support per-VCPU
settings.
Changes on PATCH v2:
1) Remove per-domain output functions for RTDS scheduler.
2) Users now use '-v all' to specify all VCPUs.
3) Support outputting a subset of the parameters of the VCPUs of a specific
domain.
4) When setting all VCPUs with the same parameters (by only one command), no
per-domain function is invoked.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
CC:
CC:
CC:
CC:
CC:
CC:
---
docs/man/xl.pod.1 | 4 +
tools/libxl/xl_cmdimpl.c | 284 +++---
tools/libxl/xl_cmdtable.c | 10 +-
3 files changed, 250 insertions(+), 48 deletions(-)
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 4eb929d..d35e169 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -1093,6 +1093,10 @@ B
Specify domain for which scheduler parameters are to be modified or retrieved.
Mandatory for modifying scheduler parameters.
+=item B<-v VCPUID/all>, B<--vcpuid=VCPUID/all>
+
+Specify vcpu for which scheduler parameters are to be modified or retrieved.
+
=item B<-p PERIOD>, B<--period=PERIOD>
Period of time, in microseconds, over which to replenish the budget.
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index c858068..7f120af 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -5630,6 +5630,37 @@ static int sched_domain_set(int domid, const
libxl_domain_sched_params *scinfo)
return rc;
}
+static int sched_vcpu_get(libxl_scheduler sched, int domid,
+libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_get(ctx, domid, scinfo);
+if (rc) {
+fprintf(stderr, "libxl_vcpu_sched_params_get failed.\n");
+return rc;
+}
+if (scinfo->sched != sched) {
+fprintf(stderr, "libxl_vcpu_sched_params_get returned %s not %s.\n",
+libxl_scheduler_to_string(scinfo->sched),
+libxl_scheduler_to_string(sched));
+return ERROR_INVAL;
+}
+
+return 0;
+}
+
+static int sched_vcpu_set(int domid, const libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_set(ctx, domid, scinfo);
+if (rc)
+fprintf(stderr, "libxl_vcpu_sched_params_set failed.\n");
+
+return rc;
+}
+
static int sched_credit_params_set(int poolid, libxl_sched_credit_params
*scinfo)
{
int rc;
@@ -5756,33 +5787,35 @@ static int sched_sedf_domain_output(
return 0;
}
-static int sched_rtds_domain_output(
-int domid)
+static int sched_rtds_vcpu_output(
+int domid, libxl_vcpu_sched_params *scinfo)
{
char *domname;
-libxl_domain_sched_params scinfo;
int rc = 0;
+int i;
-if (domid < 0) {
-printf("%-33s %4s %9s %9s\n", "Name", "ID", "Period", "Budget");
+ if (domid < 0) {
+printf("%-33s %4s %4s %9s %9s\n", "Name", "ID",
+"VCPU", "Period", "Budget");
return 0;
}
-libxl_domain_sched_params_init(&scinfo);
-rc = sched_domain_get(LIBXL_SCHEDULER_RTDS, domid, &scinfo);
+rc = sched_vcpu_get(LIBXL_SCHEDULER_RTDS, domid, scinfo);
if (rc)
goto out;
domname = libxl_domid_to_name(ctx, domid);
-printf("%-33s %4d %9d %9d\n",
-domname,
-domid,
-scinfo.period,
-scinfo.budget);
+ for( i = 0; i < scinfo->num_vcpus; i++ ) {
+printf("%-33s %4d %4d %9"PRIu32" %9"PRIu32"\n",
+domname,
+domid,
+scinfo->vcpus[i].vcpuid,
+scinfo->vcpus[i].period,
+scinfo->vcpus[i].budget);
+}
free(domname);
out:
-libxl_domain_sched_params_dispose(&scinfo);
return rc;
}
@@ -5859,6 +5892,65 @@ static int sched_domain_output(libxl_scheduler sched,
int (*output)(int),
return 0;
}
+static int sched_vcpu_output(libxl_scheduler sched,
+ int (*output)(int, libxl_vcpu_sched_params *),
+ int (*pooloutput)(uint32_t), const char
*cpupool)
+{
+libxl_dominfo *info;
+libxl_cpupoolinfo *poolinfo = NULL;
+uint32_t poolid;
+int nb_domain, n_pools = 0, i, p;
+int rc = 0;
+
+if (cpupool) {
+if (libxl_cpupool_qualifier_to_cpupoolid(ctx, cpupool, &poolid, NULL)
||
+!libxl_cpupoolid_is_valid(ctx, poolid)) {
+fprintf(stderr, "unknown cpupool \'%s\'\n", cpupool);
+return -ERROR_FAIL;
+}
+}
+
+info = libxl_list_domain(ctx, &nb_domain);
+if (!info) {
+fprintf(stderr, "libxl_list_domain failed.\n");
+return 1;
+}
[Xen-devel] [PATCH v3 for Xen 4.6 0/4] Enable per-VCPU parameter settings for RTDS scheduler
[Goal] The current xl sched-rtds tool can only set the VCPUs of a domain to the same parameter although the scheduler supports VCPUs with different parameters. This patchset is to enable xl sched-rtds tool to configure the VCPUs of a domain with different parameters. This per-VCPU settings can be used in many scenarios. For example, based on Dario's statement in our pervious discussion(http://lists.xen.org/archives/html/xen-devel/2014-09/msg00423.html), if there are two real-time applications, which have different timing requirements, running in a multi-VCPU guest domain, it is beneficial to pin these two applications to two seperate VCPUs with different scheduling parameters. What this patchset includes is a wanted and planned feature for RTDS scheudler(http://wiki.xenproject.org/wiki/RTDS-Based-Scheduler) in Xen 4.6. The interface design of the xl sched-rtds tool is based on Meng's previous discussion with Dario, George and Wei(http://lists.xen.org/archives/html/xen-devel/2015-02/msg02606.html). Basically, there are two main changes: 1) in xl, we create an array that records all VCPUs whose parameters are about to modify or output. 2) in libxl, we receive the array and call different xc functions to handle it. 3) in xen and libxc, we use XEN_DOMCTL_SCHEDOP_getvcpuinfo/putvcpuinfo(introduced by this patchset) as the hypercall for per-VCPU operations(get/set method). [Usage] With this patchset in use, xl sched-rtds tool can: 1) show the budget and period of each VCPU of each domain, by using "xl sched-rtds -v all" command. An example would be like: # xl sched-rtds -v all Cpupool Pool-0: sched=RTDS NameID VCPUPeriodBudget Domain-0 00 1 4000 vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 vm2 20 1 4000 vm2 21 1 4000 2) show the budget and period of each VCPU of a specific domain, by using, e.g., "xl sched-rtds -d vm1 -v all" command. The output would be like: # xl sched-rtds -d vm1 -v all NameID VCPUPeriodBudget vm1 10 300 150 vm1 11 400 200 vm1 12 1 4000 vm1 13 1000 500 To show a subset of the parameters of the VCPUs of a specific domain, please use, e.g.,"xl sched-rtds -d vm1 -v 0 -v 3" command. The output would be: # xl sched-rtds -d vm1 -v 0 -v 3 NameID VCPUPeriodBudget vm1 10 300 150 vm1 13 1000 500 3) Users can set the budget and period of multiple VCPUs of a specific domain with only one command, e.g., "xl sched-rtds -d vm1 -v 0 -p 100 -b 50 -v 3 -p 300 -b 150". Users can set all VCPUs with the same parameters, by one command. e.g., "xl sched-rtds -d vm1 -v all -p 500 -b 250". --- CC: CC: CC: CC: CC: CC: CC: CC: CC: Chong Li (4): xen: enable per-VCPU parameter settings for RTDS scheduler libxc: enable per-VCPU parameter settings for RTDS scheduler libxl: enable per-VCPU parameter settings for RTDS scheduler xl: enable per-VCPU parameter settings for RTDS scheduler docs/man/xl.pod.1 | 4 + tools/libxc/include/xenctrl.h | 9 ++ tools/libxc/xc_csched.c | 4 +- tools/libxc/xc_csched2.c | 4 +- tools/libxc/xc_rt.c | 64 +- tools/libxc/xc_sedf.c | 4 +- tools/libxl/libxl.c | 209 +++ tools/libxl/libxl.h | 17 +++ tools/libxl/libxl_types.idl | 16 +++ tools/libxl/xl_cmdimpl.c | 284 +++--- tools/libxl/xl_cmdtable.c | 10 +- xen/common/Makefile | 1 - xen/common/domctl.c | 3 + xen/common/sched_credit.c | 14 +-- xen/common/sched_credit2.c| 6 +- xen/common/sched_rt.c | 80 +++- xen/common/schedule.c | 5 +- xen/include/public/domctl.h | 64 +++--- 18 files changed, 680 insertions(+), 118 deletions(-) -- 1.9.1 ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v3 for Xen 4.6 3/4] libxl: enable per-VCPU parameter settings for RTDS scheduler
Add libxl_vcpu_sched_params_get/set and sched_rtds_vcpu_get/set functions to
support
per-VCPU settings.
Changes on PATCH v2:
1) New data structure (libxl_vcpu_sched_params and libxl_sched_params) to help
per-VCPU settings.
2) sched_rtds_vcpu_get now can return a random subset of the parameters of the
VCPUs of a specific domain.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
CC:
CC:
CC:
CC:
CC:
CC:
CC:
CC:
---
tools/libxl/libxl.c | 209 +++-
tools/libxl/libxl.h | 17
tools/libxl/libxl_types.idl | 16
3 files changed, 220 insertions(+), 22 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index e9a2d26..9a9c59b 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -5854,6 +5854,144 @@ static int sched_sedf_domain_set(libxl__gc *gc,
uint32_t domid,
return 0;
}
+static int sched_rtds_validate_params(libxl__gc *gc, int period,
+ int budget, uint32_t *sdom_period,
+ uint32_t *sdom_budget)
+{
+if (period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT) {
+if (period < 1) {
+LOG(ERROR, "VCPU period is not set or out of range, "
+ "valid values are larger than or equal to 1");
+return 1;
+}
+*sdom_period = period;
+}
+
+if (budget != LIBXL_DOMAIN_SCHED_PARAM_BUDGET_DEFAULT) {
+if (budget < 1) {
+LOG(ERROR, "VCPU budget is not set or out of range, "
+ "valid values are larger than or equal to 1");
+return 1;
+}
+*sdom_budget = budget;
+}
+
+if (budget > period) {
+LOG(ERROR, "VCPU budget is larger than VCPU period");
+return 1;
+}
+
+return 0;
+}
+
+static int sched_rtds_vcpu_get(libxl__gc *gc, uint32_t domid,
+ libxl_vcpu_sched_params *scinfo)
+{
+uint16_t num_vcpus;
+int rc, i;
+xc_dominfo_t info;
+xen_domctl_schedparam_vcpu_t *vcpus;
+
+if (scinfo->num_vcpus == 0) {
+rc = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (rc < 0) {
+LOGE(ERROR, "getting domain info");
+return ERROR_FAIL;
+}
+num_vcpus = info.max_vcpu_id + 1;
+} else
+num_vcpus = scinfo->num_vcpus;
+
+GCNEW_ARRAY(vcpus, num_vcpus);
+
+if (scinfo->num_vcpus > 0)
+for(i=0; i < num_vcpus; i++)
+vcpus[i].vcpuid = scinfo->vcpus[i].vcpuid;
+else
+for(i=0; i < num_vcpus; i++)
+vcpus[i].vcpuid = i;
+
+rc = xc_sched_rtds_vcpu_get(CTX->xch, domid, vcpus, num_vcpus);
+if (rc != 0) {
+LOGE(ERROR, "getting vcpu sched rtds");
+return ERROR_FAIL;
+}
+
+scinfo->sched = LIBXL_SCHEDULER_RTDS;
+if (scinfo->num_vcpus == 0) {
+scinfo->num_vcpus = num_vcpus;
+scinfo->vcpus = libxl__calloc(NOGC, num_vcpus,
sizeof(libxl_sched_params));
+}
+for(i = 0; i < num_vcpus; i++) {
+scinfo->vcpus[i].period = vcpus[i].s.rtds.period;
+scinfo->vcpus[i].budget = vcpus[i].s.rtds.budget;
+scinfo->vcpus[i].vcpuid = vcpus[i].vcpuid;
+}
+
+return 0;
+}
+
+static int sched_rtds_vcpu_set(libxl__gc *gc, uint32_t domid,
+ const libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+int i;
+uint16_t max_vcpuid;
+xc_dominfo_t info;
+xen_domctl_schedparam_vcpu_t *vcpus;
+int num_vcpus;
+
+rc = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (rc < 0) {
+LOGE(ERROR, "getting domain info");
+return ERROR_FAIL;
+}
+max_vcpuid = info.max_vcpu_id;
+
+if (scinfo->num_vcpus > 0) {
+num_vcpus = scinfo->num_vcpus;
+GCNEW_ARRAY(vcpus, num_vcpus);
+for (i = 0; i < num_vcpus; i++) {
+if (scinfo->vcpus[i].vcpuid < 0 ||
+scinfo->vcpus[i].vcpuid > max_vcpuid) {
+LOG(ERROR, "VCPU index is out of range, "
+ "valid values are within range from 0 to %d",
+ max_vcpuid);
+return ERROR_INVAL;
+}
+vcpus[i].vcpuid = scinfo->vcpus[i].vcpuid;
+
+rc = sched_rtds_validate_params(gc,
+scinfo->vcpus[i].period, scinfo->vcpus[i].budget,
+&vcpus[i].s.rtds.period, &vcpus[i].s.rtds.budget);
+if (rc)
+return ERROR_INVAL;
+}
+} else {
+num_vcpus = max_vcpuid + 1;
+ GCNEW_ARRAY(vcpus, num_vcpus);
+rc = sched_rtds_validate_params(gc,
+scinfo->vcpus[0].period, scinfo->vcpus[0].budget,
+
[Xen-devel] [PATCH v3 for Xen 4.6 1/4] xen: enable per-VCPU parameter settings for RTDS scheduler
Add two hypercalls(XEN_DOMCTL_SCHEDOP_getvcpuinfo/putvcpuinfo) to get/set a
domain's
per-VCPU parameters.
Changes on PATCH v2:
1) Change struct xen_domctl_scheduler_op, for transferring per-vcpu parameters
between libxc and hypervisor.
2) Handler of XEN_DOMCTL_SCHEDOP_getinfo now just returns the default budget
and period values of RTDS scheduler.
3) Handler of XEN_DOMCTL_SCHEDOP_getvcpuinfo now can return a random subset of
the parameters of the VCPUs of a specific domain
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
CC:
CC:
CC:
CC:
CC:
CC:
---
xen/common/Makefile | 1 -
xen/common/domctl.c | 3 ++
xen/common/sched_credit.c | 14
xen/common/sched_credit2.c | 6 ++--
xen/common/sched_rt.c | 80 +
xen/common/schedule.c | 5 +--
xen/include/public/domctl.h | 64 ++--
7 files changed, 136 insertions(+), 37 deletions(-)
diff --git a/xen/common/Makefile b/xen/common/Makefile
index 1cddebc..3fdf931 100644
--- a/xen/common/Makefile
+++ b/xen/common/Makefile
@@ -31,7 +31,6 @@ obj-y += rbtree.o
obj-y += rcupdate.o
obj-y += sched_credit.o
obj-y += sched_credit2.o
-obj-y += sched_sedf.o
obj-y += sched_arinc653.o
obj-y += sched_rt.o
obj-y += schedule.o
diff --git a/xen/common/domctl.c b/xen/common/domctl.c
index 2a2d203..349f68b 100644
--- a/xen/common/domctl.c
+++ b/xen/common/domctl.c
@@ -839,6 +839,9 @@ long do_domctl(XEN_GUEST_HANDLE_PARAM(xen_domctl_t)
u_domctl)
case XEN_DOMCTL_scheduler_op:
ret = sched_adjust(d, &op->u.scheduler_op);
+if ( ret == -ERESTART )
+ret = hypercall_create_continuation(
+__HYPERVISOR_domctl, "h", u_domctl);
copyback = 1;
break;
diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
index 953ecb0..43b086b 100644
--- a/xen/common/sched_credit.c
+++ b/xen/common/sched_credit.c
@@ -1039,25 +1039,25 @@ csched_dom_cntl(
if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
{
-op->u.credit.weight = sdom->weight;
-op->u.credit.cap = sdom->cap;
+op->u.d.credit.weight = sdom->weight;
+op->u.d.credit.cap = sdom->cap;
}
else
{
ASSERT(op->cmd == XEN_DOMCTL_SCHEDOP_putinfo);
-if ( op->u.credit.weight != 0 )
+if ( op->u.d.credit.weight != 0 )
{
if ( !list_empty(&sdom->active_sdom_elem) )
{
prv->weight -= sdom->weight * sdom->active_vcpu_count;
-prv->weight += op->u.credit.weight * sdom->active_vcpu_count;
+prv->weight += op->u.d.credit.weight * sdom->active_vcpu_count;
}
-sdom->weight = op->u.credit.weight;
+sdom->weight = op->u.d.credit.weight;
}
-if ( op->u.credit.cap != (uint16_t)~0U )
-sdom->cap = op->u.credit.cap;
+if ( op->u.d.credit.cap != (uint16_t)~0U )
+sdom->cap = op->u.d.credit.cap;
}
diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index 75e0321..8992423 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -1438,20 +1438,20 @@ csched2_dom_cntl(
if ( op->cmd == XEN_DOMCTL_SCHEDOP_getinfo )
{
-op->u.credit2.weight = sdom->weight;
+op->u.d.credit2.weight = sdom->weight;
}
else
{
ASSERT(op->cmd == XEN_DOMCTL_SCHEDOP_putinfo);
-if ( op->u.credit2.weight != 0 )
+if ( op->u.d.credit2.weight != 0 )
{
struct list_head *iter;
int old_weight;
old_weight = sdom->weight;
-sdom->weight = op->u.credit2.weight;
+sdom->weight = op->u.d.credit2.weight;
/* Update weights for vcpus, and max_weight for runqueues on which
they reside */
list_for_each ( iter, &sdom->vcpu )
diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c
index 4372486..8d1740d 100644
--- a/xen/common/sched_rt.c
+++ b/xen/common/sched_rt.c
@@ -1137,18 +1137,19 @@ rt_dom_cntl(
struct list_head *iter;
unsigned long flags;
int rc = 0;
+xen_domctl_schedparam_vcpu_t local_sched;
+unsigned int index;
switch ( op->cmd )
{
case XEN_DOMCTL_SCHEDOP_getinfo:
spin_lock_irqsave(&prv->lock, flags);
-svc = list_entry(sdom->vcpu.next, struct rt_vcpu, sdom_elem);
-op->u.rtds.period = svc->period / MICROSECS(1); /* transfer to us */
-op->u.rtds.budget = svc->budget / MICROSECS(1);
+op->u.d.rtds.period = RTDS_DEFAULT_PERIOD / MICROSECS(1); /* transfer
to us */
+op->u.d.rtds.budget = RTDS_DEFAULT_BUDGET / MICROSECS(1);
spin_un
Re: [Xen-devel] [PATCH v2 for Xen 4.6 3/4] libxl: enabling XL to set per-VCPU parameters of a domain for RTDS scheduler
On Wed, Jun 17, 2015 at 7:32 AM, Dario Faggioli wrote: > On Wed, 2015-06-17 at 13:08 +0100, Ian Campbell wrote: >> On Tue, 2015-06-09 at 18:18 +0200, Dario Faggioli wrote: >> > On Mon, 2015-06-08 at 15:55 -0500, Chong Li wrote: >> > > On Mon, Jun 8, 2015 at 10:56 AM, Dario Faggioli >> > > >> For set overall I don't really think it matters too much if it sets >> everything (i..e all vcpus and the defaults) so long as is a documented >> effect of the API -- anyone who adds per-vcpu support would then be >> aware of this and can adjust their code accordingly. >> > Exactly. Personally, I think I'd make per-domain set behave > consistently, and hence use it for set the default. > >> For get I think saying it returns the defaults used for vcpus which >> don't have something more explicit set is perfectly fine and doesn't >> pose an upgrade wrinkle, since only those who are aware of the vcpu >> settings would care about the distinction. >> > As said, I like this, great idea. > > Chong, what do you think? Can you make this happen? I like this design, too. Will make it happen soon. Thanks for these feedbacks. Chong > > Regards, > Dario > -- > <> (Raistlin Majere) > ----- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v2 for Xen 4.6 3/4] libxl: enabling XL to set per-VCPU parameters of a domain for RTDS scheduler
If no more feedbacks, let me summarize the design for the next version.
For "get" operations, we will implement the following features:
1) Use " xl sched-rtds -v all " to output the per-dom parameters of
all domains. And use, e.g., " xl sched-rtds -d vm1 -v all ", to output
the per-dom parameters of one specific domain. When a domain (say vm1)
has vcpus with different scheduling parameters but meanwhile the user
uses "xl sched-rtds -d vm1 -v all " to show the per-dom parameters,
the actual output result is just the parameters of vcpu with ID=0
(which is pointless, and should be made clear to the users).
These two kinds of "get" operations would be implemented through
libxl_domain_sched_params_get() and other domain-related functions (no
changes to all these functions).
2) For example, use " xl sched-rtds -d vm1 -v 0 -v 2 -v 4 " to show
the per-vcpu parameters of vcpu"0", vcpu"2" and vcpu"4" of vm1.
This kind of "get" operation would be implemented through
libxl_vcpu_sched_params_get() and other newly-added vcpu-related
functions.
For "set" operations, no new feature is added, against patch v2.
We need some new data structures to support per-vcpu operations (for
all schedulers, not just RTDS).
1) In libxl, we will introduce:
libxl_vcpu_sched_params = Struct("vcpu_sched_params",[
("vcpuid", integer, { xxx some init val xxx}),
("weight", integer, {'init_val': 'LIBXL_PARAM_WEIGHT_DEFAULT'}),
("cap", integer, {'init_val': 'LIBXL_PARAM_CAP_DEFAULT'}),
("period", integer, {'init_val': 'LIBXL_PARAM_PERIOD_DEFAULT'}),
("slice",integer, {'init_val': 'LIBXL_PARAM_SLICE_DEFAULT'}),
("latency", integer, {'init_val': 'LIBXL_PARAM_LATENCY_DEFAULT'}),
("extratime",integer, {'init_val': 'LIBXL_PARAM_EXTRATIME_DEFAULT'}),
("budget", integer, {'init_val': 'LIBXL_PARAM_BUDGET_DEFAULT'}),
])
libxl_sched_params = Struct("sched_params",[
("sched",libxl_scheduler),
("vcpus",Array(libxl_sched_params, "num_vcpus")),
])
and use libxl_sched_params to store and transfer vcpu array with
parameters to change/output.
2) In xen, we will introduce:
struct xen_domctl_scheduler_op {
uint32_t sched_id; /* XEN_SCHEDULER_* */
uint32_t cmd; /* XEN_DOMCTL_SCHEDOP_* */
union {
xen_domctl_schedparam_t d;
struct {
XEN_GUEST_HANDLE_64(xen_domctl_schedparam_vcpu_t) vcpus;
uint16_t nr_vcpus;
} v;
} u;
};
typedef struct xen_domctl_scheduler_op xen_domctl_scheduler_op_t;
DEFINE_XEN_GUEST_HANDLE(xen_domctl_scheduler_op_t);
and some others (details can be found in
http://www.gossamer-threads.com/lists/xen/devel/380726?do=post_view_threaded
). Because of this new xen_domctl_scheduler_op_t, some changes have to
be done for credit and credit2 schedulers (for the
XEN_DOMCTL_scheduler_op processing there).
Please correct me if something is wrong.
Thanks,
Chong
On Tue, Jun 9, 2015 at 11:18 AM, Dario Faggioli
wrote:
> On Mon, 2015-06-08 at 15:55 -0500, Chong Li wrote:
>> On Mon, Jun 8, 2015 at 10:56 AM, Dario Faggioli
>
>> > So, Thoughts? What do you think the best way forward could be?
>>
>> I like option 2 more. But I think we may also need a 'vcpuid' field in
>> libxl_sched_params.
>>
> For sparse array support, yes. At which point, I would flip the names as
> well, i.e., something like this:
>
> libxl_vcpu_sched_params = Struct("vcpu_sched_params",[
> ("vcpuid", integer, { xxx some init val xxx}),
> ("weight", integer, {'init_val': 'LIBXL_PARAM_WEIGHT_DEFAULT'}),
> ("cap", integer, {'init_val': 'LIBXL_PARAM_CAP_DEFAULT'}),
> ("period", integer, {'init_val': 'LIBXL_PARAM_PERIOD_DEFAULT'}),
> ("slice",integer, {'init_val': 'LIBXL_PARAM_SLICE_DEFAULT'}),
> ("latency", integer, {'init_val': 'LIBXL_PARAM_LATENCY_DEFAULT'}),
> ("extratime",integer, {'init_val': 'LIBXL_PARAM_EXTRATIME_DEFAULT'}),
> ("budget", integer, {'init_val': 'LIBXL_PARAM_BUDGET_DEFAULT'}),
> ])
>
> libxl_sched_params = Struct("sched_params",[
> ("sched",libxl_scheduler),
> ("vcpus",Array(libxl_sched_params, "num_vcpus")),
> ])
>
Re: [Xen-devel] [PATCH v2 for Xen 4.6 4/4] xl: enabling XL to set per-VCPU parameters of a domain for RTDS scheduler
On Tue, Jun 9, 2015 at 3:01 AM, Dario Faggioli wrote: > On Mon, 2015-06-08 at 16:18 -0500, Chong Li wrote: >> On Mon, Jun 8, 2015 at 11:21 AM, Dario Faggioli >> wrote: >> > On Mon, 2015-05-25 at 19:11 -0500, Chong Li wrote: > >> > I appreciate just now that this probably affect other bits of the >> > interface, as well as other interfaces, and I think we should handle it >> > consistently... >> > >> > What do you think? >> > For example (although this belong to patch 3's review) what >> > libxl_domain_sched_params_get() does in the case I jus described? >> >> Here, "-o/--output" is used for the users who only do per-dom >> settings. In those cases, "-o" is provided to show per-dom parameters, >> and the output is just the same as what the old RTDS tool does. >> > Right, I saw that. And as far as xl is concerned, this is ok... I just > do not like the name "-o/--output". I'd rather go with something like > "-a/--all", or implement something like this: > > # xl sched-rtds -d 2 -v all > > This is a perhaps a bit more difficult to implement (but not so much, > unless I'm overlooking something), but I personally like it better, > considering it's similar to the syntax we already have for `xl > vcpu-pin'. How about making # xl sched-rtds -d vm1 output the per-dom parameters (of vm1), and meanwhile let # xl sched-rtds -d vm1 -v all output the per-vcpu parameters (of vm1). So, in this way, all the "per-vcpu" commands need to consistently specify "-v". > >> When "-o" is set, libxl_domain_sched_params_get() and >> sched_rtds_domain_get() (both two functions in libxl.c) are called. >> > OTOH, while in libxl, the thing is different. I mean, you can't assume > that a specific libxl function would be called _only_ in the way and > from the places where you're calling it in xl: other toolstack building > on top of libxl can try to do things differently! > > So, IIRC, in patch 3 you are just not touching > libxl_domain_sched_params_get(), nor sched_rtds_domain_get() and > (perhaps in another patch) xc_sched_rtds_domain_get(). What I'm asking > is, what does it mean "users who only do per-dom settings"? What happens > if I, for instance, do a bunch of: > > libxl_vcpu_sched_params_set(...) > > for various vcpus, each one with different parameters, and then call: > > libxl_domain_sched_params_get(...) > > Have you tested this case? What's the output? I described this use case in the cover letter. "This command should only be used when all vcpus of a domain have the same parameters, otherwise the output is pointless. The period and budget shown in the output are equal to the parameters of the first VCPU of each domain." Or, I think we can just output a warning message in this case. > > Dario > > -- > <> (Raistlin Majere) > - > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) Thanks, Chong -- Chong Li Department of Computer Science and Engineering Washington University in St.louis ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v2 for Xen 4.6 4/4] xl: enabling XL to set per-VCPU parameters of a domain for RTDS scheduler
On Mon, Jun 8, 2015 at 11:21 AM, Dario Faggioli
wrote:
> On Mon, 2015-05-25 at 19:11 -0500, Chong Li wrote:
>> Change main_sched_rtds and related output functions to support per-VCPU
>> settings
>> for xl sched-rtds tool.
>>
>> Signed-off-by: Chong Li
>> Signed-off-by: Meng Xu
>> Signed-off-by: Sisu Xi
>> ---
>> tools/libxl/xl_cmdimpl.c | 261
>> +--
>> 1 file changed, 230 insertions(+), 31 deletions(-)
>>
> You must be changing tools/libxl/xl_cmdtable.c as well, or new options
> won't work (at least they won't show up in the command's help).
Yes, you are right. Without this change, the complete command help
doesn't show up. It'll be fixed in v3.
>
>> --- a/tools/libxl/xl_cmdimpl.c
>> +++ b/tools/libxl/xl_cmdimpl.c
>
>> @@ -6120,76 +6186,209 @@ int main_sched_rtds(int argc, char **argv)
>> {
>> const char *dom = NULL;
>> const char *cpupool = NULL;
>> -int period = 0; /* period is in microsecond */
>> -int budget = 0; /* budget is in microsecond */
>> +
>> +int *vcpus = (int *)xmalloc(sizeof(int)); /* IDs of VCPUs that change */
>> +int *periods = (int *)xmalloc(sizeof(int)); /* period is in microsecond
>> */
>> +int *budgets = (int *)xmalloc(sizeof(int)); /* budget is in microsecond
>> */
>> +int vcpus_size = 1; /* size of vcpus array */
>> +int periods_size = 1; /* size of periods array */
>> +int budgets_size = 1; /* size of budgets array */
>> +int input_size = 0; /* number of the input param set (v, p, b) */
>> +bool flag_b = false;
>> +bool flag_p = false;
>> +bool flag_v = false;
>> bool opt_p = false;
>> bool opt_b = false;
>> -int opt, rc;
>> +bool opt_v = false;
>> +bool opt_o = false; /* get per-domain info instead of per-vcpu info */
>> +int opt, i;
>> +int rc = 0;
>> static struct option opts[] = {
>> {"domain", 1, 0, 'd'},
>> {"period", 1, 0, 'p'},
>> {"budget", 1, 0, 'b'},
>> +{"vcpu",1, 0, 'v'},
>> {"cpupool", 1, 0, 'c'},
>> +{"output", 1, 0, 'o'},
>> COMMON_LONG_OPTS,
>> {0, 0, 0, 0}
>> };
>>
> I don't like "-o/--output" as the name of the config switch for this.
>
> Also, what is the semantic of passing -o, in case one has given
> different parameters to each vcpus? What would the command print in that
> case?
>
> I think that, while it makes sense to have the interface in place for
> the setting part, as a shortcut of setting all the parameters of all the
> vcpus to the same values, for the getting and printing side of things,
> it make much less sense.
>
> I appreciate just now that this probably affect other bits of the
> interface, as well as other interfaces, and I think we should handle it
> consistently...
>
> What do you think?
> For example (although this belong to patch 3's review) what
> libxl_domain_sched_params_get() does in the case I jus described?
Here, "-o/--output" is used for the users who only do per-dom
settings. In those cases, "-o" is provided to show per-dom parameters,
and the output is just the same as what the old RTDS tool does.
When "-o" is set, libxl_domain_sched_params_get() and
sched_rtds_domain_get() (both two functions in libxl.c) are called.
Without "-o", then sched_rtds_domain_get() should be deleted because
it will never be touched. I'm also fine with that.
Chong
>
> Regards,
> Dario
>
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v2 for Xen 4.6 3/4] libxl: enabling XL to set per-VCPU parameters of a domain for RTDS scheduler
On Mon, Jun 8, 2015 at 10:56 AM, Dario Faggioli
wrote:
> On Fri, 2015-06-05 at 12:37 +0100, Ian Campbell wrote:
>> On Mon, 2015-05-25 at 19:09 -0500, Chong Li wrote:
>
>> > diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
>> > index 117b61d..d28d274 100644
>> > --- a/tools/libxl/libxl_types.idl
>> > +++ b/tools/libxl/libxl_types.idl
>> > @@ -347,6 +347,17 @@ libxl_domain_restore_params =
>> > Struct("domain_restore_params", [
>> > ("checkpointed_stream", integer),
>> > ])
>> >
>
> Alternatevily we can just add the per-vcpu stuff (as in 'option 0'), for
> all schedulers, and really leave libxl_domain_sched_params alone (let's
> call this 'option 2'):
>
> libxl_sched_params = Struct("sched_params",[
> ("weight", integer, {'init_val': 'LIBXL_PARAM_WEIGHT_DEFAULT'}),
> ("cap", integer, {'init_val': 'LIBXL_PARAM_CAP_DEFAULT'}),
> ("period", integer, {'init_val': 'LIBXL_PARAM_PERIOD_DEFAULT'}),
> ("slice",integer, {'init_val': 'LIBXL_PARAM_SLICE_DEFAULT'}),
> ("latency", integer, {'init_val': 'LIBXL_PARAM_LATENCY_DEFAULT'}),
> ("extratime",integer, {'init_val': 'LIBXL_PARAM_EXTRATIME_DEFAULT'}),
> ("budget", integer, {'init_val': 'LIBXL_PARAM_BUDGET_DEFAULT'}),
> ])
>
> libxl_vcpu_sched_params = Struct("vcpu_sched_params",[
> ("sched",libxl_scheduler),
> ("vcpus",Array(libxl_sched_params, "num_vcpus")),
> ])
>
> In this case the redundancy comes from having libxl_domain_sched_params
> and libxl_sched_params looking a lot similar, but not shared code in
> declaring them.
>
> Maybe we can also consider putting an union inside
> libl_domain_sched_params... but again, quite a severe breakage, and I
> wouldn't be sure about how to 'key it'...
>
> So, Thoughts? What do you think the best way forward could be?
I like option 2 more. But I think we may also need a 'vcpuid' field in
libxl_sched_params.
Chong
>
> Regards,
> Dario
>
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v2 for Xen 4.6 3/4] libxl: enabling XL to set per-VCPU parameters of a domain for RTDS scheduler
On Tue, Jun 2, 2015 at 7:53 AM, George Dunlap
wrote:
> On 05/26/2015 01:09 AM, Chong Li wrote:
>> Add libxl_vcpu_sched_params_get/set and sched_rtds_vcpu_get/set functions to
>> support
>> per-VCPU settings for RTDS scheduler.
>>
>> Add a new data structure (libxl_vcpu_sched_params) to help per-VCPU settings.
>>
>> Signed-off-by: Chong Li
>> Signed-off-by: Meng Xu
>> Signed-off-by: Sisu Xi
>
> This doesn't apply cleanly for me anymore -- can you refresh and resend?
>
> Thanks,
> -George
>
Yes, sure. I'm working on the comments that have already been posted
in this series and will send out a new version soon.
Chong
>> ---
>> tools/libxl/libxl.c | 189
>> ++--
>> tools/libxl/libxl.h | 19 +
>> tools/libxl/libxl_types.idl | 11 +++
>> 3 files changed, 196 insertions(+), 23 deletions(-)
>>
>> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
>> index feb3aa9..169901a 100644
>> --- a/tools/libxl/libxl.c
>> +++ b/tools/libxl/libxl.c
>> @@ -5797,6 +5797,120 @@ static int sched_sedf_domain_set(libxl__gc *gc,
>> uint32_t domid,
>> return 0;
>> }
>>
>> +static int sched_rtds_validate_params(libxl__gc *gc, uint32_t period,
>> + uint32_t budget, uint32_t *sdom_period,
>> + uint32_t *sdom_budget)
>> +{
>> +if (period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT) {
>> +if (period < 1) {
>> +LOG(ERROR, "VCPU period is not set or out of range, "
>> + "valid values are larger than 1");
>> +return ERROR_INVAL;
>> +}
>> +*sdom_period = period;
>> +}
>> +
>> +if (budget != LIBXL_DOMAIN_SCHED_PARAM_BUDGET_DEFAULT) {
>> +if (budget < 1) {
>> +LOG(ERROR, "VCPU budget is not set or out of range, "
>> + "valid values are larger than 1");
>> +return ERROR_INVAL;
>> +}
>> +*sdom_budget = budget;
>> +}
>> +
>> +if (budget > period) {
>> +LOG(ERROR, "VCPU budget is larger than VCPU period, "
>> + "VCPU budget should be no larger than VCPU period");
>> +return ERROR_INVAL;
>> +}
>> +
>> +return 0;
>> +}
>> +
>> +static int sched_rtds_vcpu_get(libxl__gc *gc, uint32_t domid,
>> + libxl_vcpu_sched_params *scinfo)
>> +{
>> +uint16_t num_vcpus;
>> +int rc, i;
>> +xc_dominfo_t info;
>> +
>> +rc = xc_domain_getinfo(CTX->xch, domid, 1, &info);
>> +if (rc < 0) {
>> +LOGE(ERROR, "getting domain info");
>> +return ERROR_FAIL;
>> +}
>> +num_vcpus = info.max_vcpu_id + 1;
>> +
>> +struct xen_domctl_sched_rtds_params *sdom = libxl__malloc(NOGC,
>> +sizeof(struct xen_domctl_sched_rtds_params) * num_vcpus);
>> +rc = xc_sched_rtds_vcpu_get(CTX->xch, domid, sdom, num_vcpus);
>> +if (rc != 0) {
>> +LOGE(ERROR, "getting vcpu sched rtds");
>> +return ERROR_FAIL;
>> +}
>> +
>> +libxl_vcpu_sched_params_init(scinfo);
>> +
>> +scinfo->sched = LIBXL_SCHEDULER_RTDS;
>> +scinfo->num_vcpus = num_vcpus;
>> +scinfo->vcpus = (libxl_rtds_vcpu *)
>> +libxl__malloc(NOGC, sizeof(libxl_rtds_vcpu) * num_vcpus);
>> +for(i = 0; i < num_vcpus; i++) {
>> +scinfo->vcpus[i].period = sdom[i].period;
>> +scinfo->vcpus[i].budget = sdom[i].budget;
>> +}
>> +
>> +return 0;
>> +}
>> +
>> +static int sched_rtds_vcpu_set(libxl__gc *gc, uint32_t domid,
>> + const libxl_vcpu_sched_params *scinfo)
>> +{
>> +int rc;
>> +int i;
>> +uint16_t num_vcpus;
>> +int vcpuid;
>> +uint32_t budget, period;
>> +xc_dominfo_t info;
>> +
>> +rc = xc_domain_getinfo(CTX->xch, domid, 1, &info);
>> +if (rc < 0) {
>> +LOGE(ERROR, "getting domain info");
>> +return ERROR_FAIL;
>> +}
>> +num_vcpus = info.max_vcpu_id + 1;
>> +
>> +struct xen_domctl_sched_rtds_params *sdom =
>> +libxl__malloc(NOGC, scinfo->num_vcpus);
>> +for (i = 0; i < scinfo->num
Re: [Xen-devel] [PATCH v2 for Xen 4.6 1/4] xen: enabling XL to set per-VCPU parameters of a domain for RTDS scheduler
On Fri, May 29, 2015 at 8:51 AM, Dario Faggioli
wrote:
> On Mon, 2015-05-25 at 19:05 -0500, Chong Li wrote:
>
>> diff --git a/xen/common/domctl.c b/xen/common/domctl.c
>> index 28aea55..8143c44 100644
>> --- a/xen/common/domctl.c
>> +++ b/xen/common/domctl.c
>
> /* Set or get info? */
> #define XEN_DOMCTL_SCHEDOP_putinfo 0
> #define XEN_DOMCTL_SCHEDOP_getinfo 1
> #define XEN_DOMCTL_SCHEDOP_putvcpuinfo 2
> #define XEN_DOMCTL_SCHEDOP_getvcpuinfo 3
> struct xen_domctl_scheduler_op {
> uint32_t sched_id; /* XEN_SCHEDULER_* */
> uint32_t cmd; /* XEN_DOMCTL_SCHEDOP_* */
> union {
> xen_domctl_schedparam_t d;
> struct {
> XEN_GUEST_HANDLE_64(xen_domctl_schedparam_vcpu_t) vcpus;
> uint16_t nr_vcpus;
> } v;
> } u;
> };
> typedef struct xen_domctl_scheduler_op xen_domctl_scheduler_op_t;
> DEFINE_XEN_GUEST_HANDLE(xen_domctl_scheduler_op_t);
>
> I'm also attaching a (build-tested only) patch to that effect (I'm
> killing SEDF in there, so I don't have to care about it, as it's going
> away anyway).
>
> Thoughts?
>
I see. So now we put per-dom params and per-vcpu params into the same
structure (xen_domctl_scheduler_op). This structure will be handled in
sched_adjust and there should be a "switch" in that function to
distinguish per-dom get/set and per-vcpu get/set. Right?
>
> --
> <> (Raistlin Majere)
> ---------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v2 for Xen 4.6 1/4] xen: enabling XL to set per-VCPU parameters of a domain for RTDS scheduler
On Wed, May 27, 2015 at 5:02 AM, Chao Peng wrote:
> On Mon, May 25, 2015 at 07:05:52PM -0500, Chong Li wrote:
>
> I didn't see any fields you need to copy back here ('vcpus' were copied back
> in rt_vcpu_cntl() already).
>
>> +{
>> +struct rt_private *prv = rt_priv(ops);
>> +struct rt_dom * const sdom = rt_dom(d);
>> +struct rt_vcpu *svc;
>> +struct list_head *iter;
>> +unsigned long flags;
>> +int rc = 0;
>> +xen_domctl_sched_rtds_params_t local_sched;
>> +unsigned int vcpuid;
>> +unsigned int i;
>
> 'vcpuid' is only used in 'get' path once while 'i' is used in 'set' path
> only, perhaps merge the two variables?
>
Yes, we can use vcpuid in both cases.
>
>> +}
>> +spin_unlock_irqrestore(&prv->lock, flags);
>> +break;
>
> 'nr_vcpus' is not actually used untile now but in xc side you do pass
> that in.
>
Right. nr_vcpus is just useful when it is 'set' case. I'll remove this
from the 'get' case.
> Regards
> Chao
Thanks,
Chong
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v2 for Xen 4.6 1/4] xen: enabling XL to set per-VCPU parameters of a domain for RTDS scheduler
On Tue, May 26, 2015 at 12:18 PM, Chong Li wrote:
> On Tue, May 26, 2015 at 4:08 AM, Jan Beulich wrote:
>>>>> On 26.05.15 at 02:05, wrote:
>>> Add two hypercalls(XEN_DOMCTL_SCHEDOP_getvcpuinfo/putvcpuinfo) to get/set a
>>> domain's
>>> per-VCPU parameters. Hypercalls are handled by newly added hook
>>> (.adjust_vcpu) in the
>>> scheduler interface.
>>>
>>> Add a new data structure (struct xen_domctl_scheduler_vcpu_op) for
>>> transferring data
>>> between tool and hypervisor.
>>>
>>> Signed-off-by: Chong Li
>>> Signed-off-by: Meng Xu
>>> Signed-off-by: Sisu Xi
>>> ---
>>
>> Missing brief description of changes in v2 here.
>
> Based on Dario's suggestion in PATCH v1, we think it would be better
> to make the per-vcpu get/set functionalities more general, rather than
> just serving for rtds scheduler. So, the changes in v2 are:
>
> 1) Add a new hypercall, XEN_DOMCTL_scheduler_vcpu_op. Any scheduler
> can use it for per-vcpu get/set. There is a new data structure (struct
> xen_domctl_scheduler_vcpu_op), which helps transferring data (per-vcpu
> params) between tool and hypervisor when the hypercall is invoked.
>
> 2) The new hypercall is handled by function sched_adjust_vcpu (in
> schedule.c), which would call a newly added hook (named as
> .adjust_vcpu, added to the scheduler interface (struct scheduler)).
>
> 3) In RTDS scheduler, .adjust_vcpu hooks a function defined in
> sched_rt.c, named as rt_vcpu_cntl. This function gets/sets the
> per-vcpu params.
>
>
>>
>>> --- a/xen/common/schedule.c
>>> +++ b/xen/common/schedule.c
>>> @@ -1104,6 +1104,30 @@ long sched_adjust(struct domain *d, struct
>>> xen_domctl_scheduler_op *op)
>>> return ret;
>>> }
>>>
>>> +/* Adjust scheduling parameter for the vcpus of a given domain. */
>>> +long sched_adjust_vcpu(
>>> +struct domain *d,
>>> +struct xen_domctl_scheduler_vcpu_op *op)
>>> +{
>>> +long ret;
>>
>> I see no reason for this and the function return type to be "long".
>
> The reason is stated in the begining.
Sorry for missing some details in the last reply.
sched_adjust_vcpu is added here because I need a seperate function to
handle XEN_DOMCTL_scheduler_vcpu_op, which is a new hypercall
dedicated for per-vcpu get/set. This function is actually based on
sched_adjust (which is dedicated for per-dom get/set in rtds), so the
return type is also the same.
>
>>
>> Jan
>>
>
> Best,
> Chong
>
> --
> Chong Li
> Department of Computer Science and Engineering
> Washington University in St.louis
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v2 for Xen 4.6 1/4] xen: enabling XL to set per-VCPU parameters of a domain for RTDS scheduler
On Tue, May 26, 2015 at 4:08 AM, Jan Beulich wrote:
>>>> On 26.05.15 at 02:05, wrote:
>> Add two hypercalls(XEN_DOMCTL_SCHEDOP_getvcpuinfo/putvcpuinfo) to get/set a
>> domain's
>> per-VCPU parameters. Hypercalls are handled by newly added hook
>> (.adjust_vcpu) in the
>> scheduler interface.
>>
>> Add a new data structure (struct xen_domctl_scheduler_vcpu_op) for
>> transferring data
>> between tool and hypervisor.
>>
>> Signed-off-by: Chong Li
>> Signed-off-by: Meng Xu
>> Signed-off-by: Sisu Xi
>> ---
>
> Missing brief description of changes in v2 here.
Based on Dario's suggestion in PATCH v1, we think it would be better
to make the per-vcpu get/set functionalities more general, rather than
just serving for rtds scheduler. So, the changes in v2 are:
1) Add a new hypercall, XEN_DOMCTL_scheduler_vcpu_op. Any scheduler
can use it for per-vcpu get/set. There is a new data structure (struct
xen_domctl_scheduler_vcpu_op), which helps transferring data (per-vcpu
params) between tool and hypervisor when the hypercall is invoked.
2) The new hypercall is handled by function sched_adjust_vcpu (in
schedule.c), which would call a newly added hook (named as
.adjust_vcpu, added to the scheduler interface (struct scheduler)).
3) In RTDS scheduler, .adjust_vcpu hooks a function defined in
sched_rt.c, named as rt_vcpu_cntl. This function gets/sets the
per-vcpu params.
>> +
>> +switch ( op->cmd )
>> +{
>> +case XEN_DOMCTL_SCHEDOP_getvcpuinfo:
>> +spin_lock_irqsave(&prv->lock, flags);
>> +list_for_each( iter, &sdom->vcpu )
>> +{
>> +svc = list_entry(iter, struct rt_vcpu, sdom_elem);
>> +vcpuid = svc->vcpu->vcpu_id;
>> +
>> +local_sched.budget = svc->budget / MICROSECS(1);
>> +local_sched.period = svc->period / MICROSECS(1);
>> +if ( copy_to_guest_offset(op->u.rtds.vcpus, vcpuid,
>
> What makes you think that the caller has provided enough slots?
This code works in a similar way as the one for XEN_SYSCTL_numainfo.
So if there is no enough slot in guest space, en error code is
returned.
>
>> +&local_sched, 1) )
>
> You're leaking hypervisor stack contents here, but by reading
> the structure from guest memory first to honor its vcpuid field
> (this making "get" match "put") you'd avoid this anyway.
>
The structure in guest memory has nothing, and it will hold per-vcpu
params after this XEN_DOMCTL_SCHEDOP_getvcpuinfo hypercall is well
served.
Does "leaking hypervisor stack" mean that I need to call
"local_sched.vcpuid = vcpuid" before copying this local_sched to the
guest memory?
>
>> +{
>> +spin_unlock_irqrestore(&prv->lock, flags);
>> +return -EFAULT;
>
> Setting rc to -EFAULT and break-ing would seem better for these
> error paths (avoiding the repeated spin_unlock_irqrestore()-s).
>
>> +}
>> +hypercall_preempt_check();
>
> ???
We've talked about this in patch v1
(http://lists.xenproject.org/archives/html/xen-devel/2015-05/msg01152.html).
When a domain has too many VCPUs, we need to make sure the spin lock
does not last for too long.
>
>> +}
>> +spin_unlock_irqrestore(&prv->lock, flags);
>> +break;
>> +case XEN_DOMCTL_SCHEDOP_putvcpuinfo:
>> +spin_lock_irqsave(&prv->lock, flags);
>> +for( i = 0; i < op->u.rtds.nr_vcpus; i++ )
>> +{
>> +if ( copy_from_guest_offset(&local_sched,
>> +op->u.rtds.vcpus, i, 1) )
>> +{
>> +spin_unlock_irqrestore(&prv->lock, flags);
>> +return -EFAULT;
>> +}
>> +if ( local_sched.period <= 0 || local_sched.budget <= 0 )
>> +{
>> +spin_unlock_irqrestore(&prv->lock, flags);
>> +return -EINVAL;
>> +}
>> +svc = rt_vcpu(d->vcpu[local_sched.vcpuid]);
>
> You mustn't assume local_sched.vcpuid to represent a valid array
> index.
Do you mean that I locate the vcpu by comparing the local_sched.vcpuid
with the IDs of each element in the vcpu array?
>
>
>> --- a/xen/common/schedule.c
>> +++ b/xen/common/schedule.c
>> @@ -1104,6 +1104,30 @@ long sched_adjust(struct domain *d, struct
>> xen_domctl_scheduler_op *op)
>> return ret;
>> }
>>
>> +/* Adjust scheduling parameter for the vcpus of a given domain. */
>> +long sched_adjust_vcpu(
>> +struct domain *d,
>> +struct xen_domctl_scheduler_vcpu_op *op)
>> +{
>> +long ret;
>
> I see no reason for this and the function return type to be "long".
The reason is stated in the begining.
>
> Jan
>
Best,
Chong
--
Chong Li
Department of Computer Science and Engineering
Washington University in St.louis
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v2 for Xen 4.6 4/4] xl: enabling XL to set per-VCPU parameters of a domain for RTDS scheduler
Change main_sched_rtds and related output functions to support per-VCPU settings
for xl sched-rtds tool.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
tools/libxl/xl_cmdimpl.c | 261 +--
1 file changed, 230 insertions(+), 31 deletions(-)
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 648ca08..a57f772 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -5577,6 +5577,37 @@ static int sched_domain_set(int domid, const
libxl_domain_sched_params *scinfo)
return rc;
}
+static int sched_vcpu_get(libxl_scheduler sched, int domid,
+libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_get(ctx, domid, scinfo);
+if (rc) {
+fprintf(stderr, "libxl_vcpu_sched_params_get failed.\n");
+return rc;
+}
+if (scinfo->sched != sched) {
+fprintf(stderr, "libxl_vcpu_sched_params_get returned %s not %s.\n",
+libxl_scheduler_to_string(scinfo->sched),
+libxl_scheduler_to_string(sched));
+return ERROR_INVAL;
+}
+
+return 0;
+}
+
+static int sched_vcpu_set(int domid, const libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+
+rc = libxl_vcpu_sched_params_set(ctx, domid, scinfo);
+if (rc)
+fprintf(stderr, "libxl_vcpu_sched_params_set failed.\n");
+
+return rc;
+}
+
static int sched_credit_params_set(int poolid, libxl_sched_credit_params
*scinfo)
{
int rc;
@@ -5733,6 +5764,41 @@ out:
return rc;
}
+static int sched_rtds_vcpu_output(
+int domid)
+{
+char *domname;
+libxl_vcpu_sched_params scinfo;
+int rc = 0;
+int i;
+
+ if (domid < 0) {
+printf("%-33s %4s %4s %9s %9s\n", "Name", "ID",
+"VCPU", "Period", "Budget");
+return 0;
+}
+
+libxl_vcpu_sched_params_init(&scinfo);
+rc = sched_vcpu_get(LIBXL_SCHEDULER_RTDS, domid, &scinfo);
+if (rc)
+goto out;
+
+domname = libxl_domid_to_name(ctx, domid);
+ for( i = 0; i < scinfo.num_vcpus; i++ ) {
+printf("%-33s %4d %4d %9"PRIu32" %9"PRIu32"\n",
+domname,
+domid,
+i,
+scinfo.vcpus[i].period,
+scinfo.vcpus[i].budget);
+}
+free(domname);
+
+out:
+libxl_vcpu_sched_params_dispose(&scinfo);
+return rc;
+}
+
static int sched_rtds_pool_output(uint32_t poolid)
{
char *poolname;
@@ -6120,76 +6186,209 @@ int main_sched_rtds(int argc, char **argv)
{
const char *dom = NULL;
const char *cpupool = NULL;
-int period = 0; /* period is in microsecond */
-int budget = 0; /* budget is in microsecond */
+
+int *vcpus = (int *)xmalloc(sizeof(int)); /* IDs of VCPUs that change */
+int *periods = (int *)xmalloc(sizeof(int)); /* period is in microsecond */
+int *budgets = (int *)xmalloc(sizeof(int)); /* budget is in microsecond */
+int vcpus_size = 1; /* size of vcpus array */
+int periods_size = 1; /* size of periods array */
+int budgets_size = 1; /* size of budgets array */
+int input_size = 0; /* number of the input param set (v, p, b) */
+bool flag_b = false;
+bool flag_p = false;
+bool flag_v = false;
bool opt_p = false;
bool opt_b = false;
-int opt, rc;
+bool opt_v = false;
+bool opt_o = false; /* get per-domain info instead of per-vcpu info */
+int opt, i;
+int rc = 0;
static struct option opts[] = {
{"domain", 1, 0, 'd'},
{"period", 1, 0, 'p'},
{"budget", 1, 0, 'b'},
+{"vcpu",1, 0, 'v'},
{"cpupool", 1, 0, 'c'},
+{"output", 1, 0, 'o'},
COMMON_LONG_OPTS,
{0, 0, 0, 0}
};
-SWITCH_FOREACH_OPT(opt, "d:p:b:c:h", opts, "sched-rtds", 0) {
+SWITCH_FOREACH_OPT(opt, "d:p:b:v:c:h:o", opts, "sched-rtds", 0) {
case 'd':
dom = optarg;
break;
case 'p':
-period = strtol(optarg, NULL, 10);
+if (flag_p == 1) { /* budget or vcpuID is missed */
+fprintf(stderr, "Must specify period, budget and vcpuID\n");
+rc = 1;
+goto out;
+}
+ if (input_size >= periods_size) {
+periods_size *= 2;
+periods = xrealloc(periods, periods_size);
+if (!periods) {
+fprintf(stderr, "Failed to realloc periods\n");
+rc = 1;
+goto out;
+}
+}
+periods[input_size] = strtol(optarg, NULL, 10);
opt_p = 1;
+flag_p = 1;
+if (flag_p
[Xen-devel] [PATCH v2 for Xen 4.6 3/4] libxl: enabling XL to set per-VCPU parameters of a domain for RTDS scheduler
Add libxl_vcpu_sched_params_get/set and sched_rtds_vcpu_get/set functions to
support
per-VCPU settings for RTDS scheduler.
Add a new data structure (libxl_vcpu_sched_params) to help per-VCPU settings.
Signed-off-by: Chong Li
Signed-off-by: Meng Xu
Signed-off-by: Sisu Xi
---
tools/libxl/libxl.c | 189 ++--
tools/libxl/libxl.h | 19 +
tools/libxl/libxl_types.idl | 11 +++
3 files changed, 196 insertions(+), 23 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index feb3aa9..169901a 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -5797,6 +5797,120 @@ static int sched_sedf_domain_set(libxl__gc *gc,
uint32_t domid,
return 0;
}
+static int sched_rtds_validate_params(libxl__gc *gc, uint32_t period,
+ uint32_t budget, uint32_t *sdom_period,
+ uint32_t *sdom_budget)
+{
+if (period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT) {
+if (period < 1) {
+LOG(ERROR, "VCPU period is not set or out of range, "
+ "valid values are larger than 1");
+return ERROR_INVAL;
+}
+*sdom_period = period;
+}
+
+if (budget != LIBXL_DOMAIN_SCHED_PARAM_BUDGET_DEFAULT) {
+if (budget < 1) {
+LOG(ERROR, "VCPU budget is not set or out of range, "
+ "valid values are larger than 1");
+return ERROR_INVAL;
+}
+*sdom_budget = budget;
+}
+
+if (budget > period) {
+LOG(ERROR, "VCPU budget is larger than VCPU period, "
+ "VCPU budget should be no larger than VCPU period");
+return ERROR_INVAL;
+}
+
+return 0;
+}
+
+static int sched_rtds_vcpu_get(libxl__gc *gc, uint32_t domid,
+ libxl_vcpu_sched_params *scinfo)
+{
+uint16_t num_vcpus;
+int rc, i;
+xc_dominfo_t info;
+
+rc = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (rc < 0) {
+LOGE(ERROR, "getting domain info");
+return ERROR_FAIL;
+}
+num_vcpus = info.max_vcpu_id + 1;
+
+struct xen_domctl_sched_rtds_params *sdom = libxl__malloc(NOGC,
+sizeof(struct xen_domctl_sched_rtds_params) * num_vcpus);
+rc = xc_sched_rtds_vcpu_get(CTX->xch, domid, sdom, num_vcpus);
+if (rc != 0) {
+LOGE(ERROR, "getting vcpu sched rtds");
+return ERROR_FAIL;
+}
+
+libxl_vcpu_sched_params_init(scinfo);
+
+scinfo->sched = LIBXL_SCHEDULER_RTDS;
+scinfo->num_vcpus = num_vcpus;
+scinfo->vcpus = (libxl_rtds_vcpu *)
+libxl__malloc(NOGC, sizeof(libxl_rtds_vcpu) * num_vcpus);
+for(i = 0; i < num_vcpus; i++) {
+scinfo->vcpus[i].period = sdom[i].period;
+scinfo->vcpus[i].budget = sdom[i].budget;
+}
+
+return 0;
+}
+
+static int sched_rtds_vcpu_set(libxl__gc *gc, uint32_t domid,
+ const libxl_vcpu_sched_params *scinfo)
+{
+int rc;
+int i;
+uint16_t num_vcpus;
+int vcpuid;
+uint32_t budget, period;
+xc_dominfo_t info;
+
+rc = xc_domain_getinfo(CTX->xch, domid, 1, &info);
+if (rc < 0) {
+LOGE(ERROR, "getting domain info");
+return ERROR_FAIL;
+}
+num_vcpus = info.max_vcpu_id + 1;
+
+struct xen_domctl_sched_rtds_params *sdom =
+libxl__malloc(NOGC, scinfo->num_vcpus);
+for (i = 0; i < scinfo->num_vcpus; i++) {
+vcpuid = scinfo->vcpus[i].vcpuid;
+budget = scinfo->vcpus[i].budget;
+period = scinfo->vcpus[i].period;
+if (vcpuid < 0 || vcpuid >= num_vcpus) {
+LOG(ERROR, "VCPU index is out of range, "
+ "valid values are within range from 0 to %d",
+ num_vcpus);
+return ERROR_INVAL;
+}
+sdom[i].vcpuid = vcpuid;
+
+rc = sched_rtds_validate_params(gc, period, budget,
+&sdom[i].period, &sdom[i].budget);
+if (rc == ERROR_INVAL)
+return rc;
+}
+
+rc = xc_sched_rtds_vcpu_set(CTX->xch, domid,
+sdom, scinfo->num_vcpus);
+if (rc != 0) {
+LOGE(ERROR, "setting vcpu sched rtds");
+return ERROR_FAIL;
+}
+
+return rc;
+}
+
static int sched_rtds_domain_get(libxl__gc *gc, uint32_t domid,
libxl_domain_sched_params *scinfo)
{
@@ -5830,29 +5944,10 @@ static int sched_rtds_domain_set(libxl__gc *gc,
uint32_t domid,
return ERROR_FAIL;
}
-if (scinfo->period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT) {
-if (scinfo->period < 1) {
-LOG(ERROR, "VCPU period is not set or out of range, "
- &
