Re: [Xen-devel] null scheduler bug

[Julien Grall]

On 09/27/2018 06:06 PM, Dario Faggioli wrote:

On Thu, 2018-09-27 at 16:09 +0100, Julien Grall wrote:
Hi,


Hi Dario,


On 09/27/2018 03:32 PM, Dario Faggioli wrote:

On Thu, 2018-09-27 at 15:15 +0200, Milan Boberic wrote:



In one of your e-mail, you wrote:

"Well, our implementation of RCU requires that, from time to time,
the
various physical CPUs of your box become idle, or get an interrupt,
or
go executing inside Xen (for hypercalls, vmexits, etc). In fact, a
CPU
going through Xen is what allow us to tell that it reached a so-
called
'quiescent state', which in turns is necessary for declaring a so-
called 'RCU grace period' over."

I don't quite agree with you on the definition of "quiescent state"
here.


Hehe... I was trying to be both quick and accurate. It's more than
possible that I failed. :-)


To take the domain example, we want to wait until all the CPU has
stopped using the pointer (an hypercall could race put_domain).


I'm not sure what you mean with "an hypercall could race put_domain".


I meant that another CPU get a pointer on the domain until the domain is 
effectively removed from the list by _domain_destroy.



What we want is to wait until all the CPUs that are involved in the
grace period, have gone through rcupdate.c:cpu_quiet(), or have become
idle.


Which is what I meant but in a more convoluted way.



Receiving an interrupt, or experiencing a context switch, or even going
idle, it's "just" how it happens that these CPUs have their chance to
go through cpu_quiet(). It is in this sense that I meant that those
events are used as markers of a quiescent state.

And "wfi=native" (in particular in combination with the null scheduler,
but I guess also with other ones, at least to a certain extent) makes
figuring out the "or have become idle" part tricky. That is the problem
here, isn't it?


That's correct.




That
pointer will not be in-use if the CPU is in kernel-mode/user-mode or
in
the idle loop. Am I correct?


Right.

So, we want that all the CPUs that were in Xen to have either left Xen
at least once or, if they're still there and have never left, that must
be because they've become idle.

And currently we treat all the CPUs that have not told the RCU
subsystem that they're idle (via rcu_idle_enter()) as busy, without
distinguishing between the ones that are busy in Xen from the one which
are busy in guest (kernel or user) mode.


So I am wondering whether we could:
- Mark any CPU in kernel-mode/user-mode quiet


Right. We'd need something like a rcu_guest_enter()/rcu_guest_exit()
(or a rcu_xen_exit()/rcu_xen_enter()), which works for all combination
of arches and guest types.

It looks to me too that this would help in this case, as the vCPU that
stays in guest mode because of wfi=idle would be counted as quiet, and
we won't have to wait for it.


- Raise a RCU_SOFTIRQ in call_rcu?


Mmm... what would be the point of this?


To check if the RCU has work to do. You may call_rcu with all the other 
CPUs quiet. Or do you expect this to be done in rcu_guest_{enter,exit}()?





With that solution, it may even be possible to avoid the timer in
the
idle loop.


Not sure. The timer is there to deal with the case when a CPU which has
a callback queued wants to go idle. It may have quiesced already, but
if there are others which have not, either:
1) we let it go idle, but then the callback will run only when it
wakes up from idle which, without the timer, could be far ahead in
time;
2) we don't let it go idle, but we waste resources;
3) we let it go idle and keep the timer. :-)


Oh right.



But anyway, even if it would not let us get rid of the timer, it seems
like it could be nicer than any other approaches. I accept
help/suggestions about the "let's intercept guest-Xen and Xen-guest
transitions, and track that inside RCU code.


The best place for Arm to have those calls would be in:
	- enter_hypervisor_head(): This is called after exiting the guest and 
before doing any other work
	- leave_hypervisor_head(): This is called just before returning to the 
guest.


I am CCing Stefano work e-mail. When I spoke with him he was interested 
to put some effort towards fixing the bug.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Milan Boberic]

Hi,
thank you for explanation, links and advices. I'm gonna go through all
that literature.

Best regards!
On Thu, Sep 27, 2018 at 7:06 PM Dario Faggioli  wrote:
>
> On Thu, 2018-09-27 at 16:09 +0100, Julien Grall wrote:
> > Hi Dario,
> >
> Hi,
>
> > On 09/27/2018 03:32 PM, Dario Faggioli wrote:
> > > On Thu, 2018-09-27 at 15:15 +0200, Milan Boberic wrote:
> > > >
> > In one of your e-mail, you wrote:
> >
> > "Well, our implementation of RCU requires that, from time to time,
> > the
> > various physical CPUs of your box become idle, or get an interrupt,
> > or
> > go executing inside Xen (for hypercalls, vmexits, etc). In fact, a
> > CPU
> > going through Xen is what allow us to tell that it reached a so-
> > called
> > 'quiescent state', which in turns is necessary for declaring a so-
> > called 'RCU grace period' over."
> >
> > I don't quite agree with you on the definition of "quiescent state"
> > here.
> >
> Hehe... I was trying to be both quick and accurate. It's more than
> possible that I failed. :-)
>
> > To take the domain example, we want to wait until all the CPU has
> > stopped using the pointer (an hypercall could race put_domain).
> >
> I'm not sure what you mean with "an hypercall could race put_domain".
> What we want is to wait until all the CPUs that are involved in the
> grace period, have gone through rcupdate.c:cpu_quiet(), or have become
> idle.
>
> Receiving an interrupt, or experiencing a context switch, or even going
> idle, it's "just" how it happens that these CPUs have their chance to
> go through cpu_quiet(). It is in this sense that I meant that those
> events are used as markers of a quiescent state.
>
> And "wfi=native" (in particular in combination with the null scheduler,
> but I guess also with other ones, at least to a certain extent) makes
> figuring out the "or have become idle" part tricky. That is the problem
> here, isn't it?
>
> > That
> > pointer will not be in-use if the CPU is in kernel-mode/user-mode or
> > in
> > the idle loop. Am I correct?
> >
> Right.
>
> So, we want that all the CPUs that were in Xen to have either left Xen
> at least once or, if they're still there and have never left, that must
> be because they've become idle.
>
> And currently we treat all the CPUs that have not told the RCU
> subsystem that they're idle (via rcu_idle_enter()) as busy, without
> distinguishing between the ones that are busy in Xen from the one which
> are busy in guest (kernel or user) mode.
>
> > So I am wondering whether we could:
> >   - Mark any CPU in kernel-mode/user-mode quiet
> >
> Right. We'd need something like a rcu_guest_enter()/rcu_guest_exit()
> (or a rcu_xen_exit()/rcu_xen_enter()), which works for all combination
> of arches and guest types.
>
> It looks to me too that this would help in this case, as the vCPU that
> stays in guest mode because of wfi=idle would be counted as quiet, and
> we won't have to wait for it.
>
> >   - Raise a RCU_SOFTIRQ in call_rcu?
> >
> Mmm... what would be the point of this?
>
> > With that solution, it may even be possible to avoid the timer in
> > the
> > idle loop.
> >
> Not sure. The timer is there to deal with the case when a CPU which has
> a callback queued wants to go idle. It may have quiesced already, but
> if there are others which have not, either:
> 1) we let it go idle, but then the callback will run only when it
>wakes up from idle which, without the timer, could be far ahead in
>time;
> 2) we don't let it go idle, but we waste resources;
> 3) we let it go idle and keep the timer. :-)
>
> But anyway, even if it would not let us get rid of the timer, it seems
> like it could be nicer than any other approaches. I accept
> help/suggestions about the "let's intercept guest-Xen and Xen-guest
> transitions, and track that inside RCU code.
>
> Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Software Engineer @ SUSE https://www.suse.com/

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Dario Faggioli]

On Thu, 2018-09-27 at 16:09 +0100, Julien Grall wrote:
> Hi Dario,
> 
Hi,

> On 09/27/2018 03:32 PM, Dario Faggioli wrote:
> > On Thu, 2018-09-27 at 15:15 +0200, Milan Boberic wrote:
> > > 
> In one of your e-mail, you wrote:
> 
> "Well, our implementation of RCU requires that, from time to time,
> the
> various physical CPUs of your box become idle, or get an interrupt,
> or
> go executing inside Xen (for hypercalls, vmexits, etc). In fact, a
> CPU
> going through Xen is what allow us to tell that it reached a so-
> called
> 'quiescent state', which in turns is necessary for declaring a so-
> called 'RCU grace period' over."
> 
> I don't quite agree with you on the definition of "quiescent state" 
> here. 
>
Hehe... I was trying to be both quick and accurate. It's more than
possible that I failed. :-)

> To take the domain example, we want to wait until all the CPU has 
> stopped using the pointer (an hypercall could race put_domain). 
>
I'm not sure what you mean with "an hypercall could race put_domain".
What we want is to wait until all the CPUs that are involved in the
grace period, have gone through rcupdate.c:cpu_quiet(), or have become
idle.

Receiving an interrupt, or experiencing a context switch, or even going
idle, it's "just" how it happens that these CPUs have their chance to
go through cpu_quiet(). It is in this sense that I meant that those
events are used as markers of a quiescent state.

And "wfi=native" (in particular in combination with the null scheduler,
but I guess also with other ones, at least to a certain extent) makes
figuring out the "or have become idle" part tricky. That is the problem
here, isn't it?

> That 
> pointer will not be in-use if the CPU is in kernel-mode/user-mode or
> in 
> the idle loop. Am I correct?
> 
Right.

So, we want that all the CPUs that were in Xen to have either left Xen
at least once or, if they're still there and have never left, that must
be because they've become idle.

And currently we treat all the CPUs that have not told the RCU
subsystem that they're idle (via rcu_idle_enter()) as busy, without
distinguishing between the ones that are busy in Xen from the one which
are busy in guest (kernel or user) mode.

> So I am wondering whether we could:
>   - Mark any CPU in kernel-mode/user-mode quiet
>
Right. We'd need something like a rcu_guest_enter()/rcu_guest_exit()
(or a rcu_xen_exit()/rcu_xen_enter()), which works for all combination
of arches and guest types.

It looks to me too that this would help in this case, as the vCPU that
stays in guest mode because of wfi=idle would be counted as quiet, and
we won't have to wait for it.

>   - Raise a RCU_SOFTIRQ in call_rcu?
> 
Mmm... what would be the point of this?

> With that solution, it may even be possible to avoid the timer in
> the 
> idle loop.
> 
Not sure. The timer is there to deal with the case when a CPU which has
a callback queued wants to go idle. It may have quiesced already, but
if there are others which have not, either:
1) we let it go idle, but then the callback will run only when it 
   wakes up from idle which, without the timer, could be far ahead in 
   time;
2) we don't let it go idle, but we waste resources;
3) we let it go idle and keep the timer. :-)

But anyway, even if it would not let us get rid of the timer, it seems
like it could be nicer than any other approaches. I accept
help/suggestions about the "let's intercept guest-Xen and Xen-guest
transitions, and track that inside RCU code.

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/


signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Julien Grall]

Hi Dario,

On 09/27/2018 03:32 PM, Dario Faggioli wrote:

On Thu, 2018-09-27 at 15:15 +0200, Milan Boberic wrote:

Hi,
I applied patch and vwfi=native and everything works fine, I can
create and destroy guest domain as many times as I want.


Ok, now that we know it works, what do you guys prefer?

Stefano? Julien? I know it's not strictly an ARM-only issue, but I'm
asking you because ARM is where it shows-up/harm the most.

I personally would be ok with:
- doing a patch adding qhimark, qlowmark and blimit boot command line
   parameters;
- doing a patch (similar to this one) forcing the parameters to a
   specific state (and printing a warning about that), if wfi=native is
   used.

Thoughts?


I know I first suggested this but I have been thinking about it and 
trying to find a different approach. With NULL scheduler, you end up 
partitioning your platform. I think may have have Xen to be there just 
for handling hypercall, emulation and guest interrupt. So I would like 
to avoid adding an interrupt when possible.


In one of your e-mail, you wrote:

"Well, our implementation of RCU requires that, from time to time, the
various physical CPUs of your box become idle, or get an interrupt, or
go executing inside Xen (for hypercalls, vmexits, etc). In fact, a CPU
going through Xen is what allow us to tell that it reached a so-called
'quiescent state', which in turns is necessary for declaring a so-
called 'RCU grace period' over."

I don't quite agree with you on the definition of "quiescent state" 
here. To take the domain example, we want to wait until all the CPU has 
stopped using the pointer (an hypercall could race put_domain). That 
pointer will not be in-use if the CPU is in kernel-mode/user-mode or in 
the idle loop. Am I correct?


So I am wondering whether we could:
- Mark any CPU in kernel-mode/user-mode quiet
- Raise a RCU_SOFTIRQ in call_rcu?

With that solution, it may even be possible to avoid the timer in the 
idle loop.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Dario Faggioli]

On Thu, 2018-09-27 at 15:15 +0200, Milan Boberic wrote:
> Hi,
> I applied patch and vwfi=native and everything works fine, I can
> create and destroy guest domain as many times as I want.
> 
Ok, now that we know it works, what do you guys prefer?

Stefano? Julien? I know it's not strictly an ARM-only issue, but I'm
asking you because ARM is where it shows-up/harm the most.

I personally would be ok with:
- doing a patch adding qhimark, qlowmark and blimit boot command line 
  parameters;
- doing a patch (similar to this one) forcing the parameters to a 
  specific state (and printing a warning about that), if wfi=native is 
  used.

Thoughts?

Oh, and I'm adding Juergen in case we consider this relevant for the
release (and in case we want to add a Jira issue, which I don't think I
can do).

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/


signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Dario Faggioli]

On Thu, 2018-09-27 at 15:15 +0200, Milan Boberic wrote:
> Hi,
> I applied patch and vwfi=native and everything works fine, I can
> create and destroy guest domain as many times as I want.
> 
> I have to ask, will this patch have any impact on performance (I will
> test it later, but I just need your opinions)?
>
Well, with a question like this, the only possible answer is "depends".
:-)

Basically, there is a little bit of overhead to be expected, with this
patch applied, every time that call_rcu() is invoked, inside Xen. Now,
you can look at when that happens, and you'll notice that this
basically never happen in an hot-path.

In your case, there is at least one call in the domain destruction
path. You can try to measure whether actually destroying the domain
takes more time _with_ "wfi=native" (plus this patch) as compared to
how long it takes _without_ "wfi=native" (and also without this patch).
I don't think you'll be able to appreciate any significant difference.

The point is more, I think, whether "wfi=native" helps your use case.
Have you measure that? I mean, have you checked what is the difference
in performance (or latency, or whatever you're interested in) between
the "wfi=native" case and the default?
If you have, and "wfi=native" helps, then you also need something like
this patch, or domain destruction won't work (in fact, I call the fact
that it takes 'around 7 seconds', not working). If "wfi=native" does
not help your use case, then you're better off not using neither it nor
this patch.

> And what this patch exactly do? I need to fully understand it because
> I need to document it in my master thesis which will be finished soon
> thanks to you people :D
> 
Have you heard about RCU? It's a very clever synchronization solution,
widely used in the Linux kernel. Xen has that too, but we use an old
version of the Linux code, and we don't use it that much.

This is, IMO, some good introductory material, but, really, just google
"RCU" or "RCU linux", and you'll hit tons of articles and docs:
https://lwn.net/Articles/262464/

Well, our implementation of RCU requires that, from time to time, the
various physical CPUs of your box become idle, or get an interrupt, or
go executing inside Xen (for hypercalls, vmexits, etc). In fact, a CPU
going through Xen is what allow us to tell that it reached a so-called
'quiescent state', which in turns is necessary for declaring a so-
called 'RCU grace period' over.

Usually, as soon as a guest (or dom0) vCPU become idle, the pCPU on
which it was running does go through Xen, to figure out whether or not
there is another vCPU, from the same or from another guest, to be run.
If not, the pCPU stays idle, but it stays idle _in_Xen_, and that is
good for RCU quiescence and grace period tracking.

Now, with the combination of "sched=null" and "wfi=native", when the
guest (or dom0) vCPU becomes idle, we _stay_in_the_guest_, until
something (typically an interrupt) comes. This means that the vCPU in
question never let Xen's RCU know that he has gone through a quiescent
state, and grace periods risk lasting very long, if not forever.

In fact, the reason why everything was working again with a printk()
was, as Julien noted, that an interrupt was being injected. Check the
old discussion on xen-devel about the RCU bug that I linked to in one
of my first messages in this thread to even more insights.

https://www.mail-archive.com/xen-devel@lists.xen.org/msg105388.html

https://lists.xenproject.org/archives/html/xen-devel/2017-07/msg02770.html

https://lists.xen.org/archives/html/xen-devel/2017-09/msg01855.html
https://lists.xen.org/archives/html/xen-devel/2017-09/msg03515.html
https://lists.xenproject.org/archives/html/xen-devel/2017-09/msg01855.html

Setting the qhimark, qlowmark and blimit to the values you see in the
patch, partially defeats the purpose of RCU, as the update of the data
structure is not deferred to some future point in time, but it is
basically always performed synchronously with the modification, and
that's why I dislike just doing it all the time, and I prefer limiting
to doing it when we're using "wfi=native".

For some more details about the meaning of the qhimark, qlowmark and
blimit values, check these:
https://www.systutorials.com/linux-kernels/132439/patch-rcu-batch-tuning-linux-2-6-16/
https://lwn.net/Articles/166647/

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/


signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Milan Boberic]

Hi,
I applied patch and vwfi=native and everything works fine, I can
create and destroy guest domain as many times as I want.

I have to ask, will this patch have any impact on performance (I will
test it later, but I just need your opinions)?
And what this patch exactly do? I need to fully understand it because
I need to document it in my master thesis which will be finished soon
thanks to you people :D

Thank you for taking your time to make this patch, best regards!
On Thu, Sep 27, 2018 at 11:51 AM Dario Faggioli  wrote:
>
> On Tue, 2018-09-25 at 19:49 +0200, Dario Faggioli wrote:
> > [Adding a few people to the Cc-list. See below...]
> > On Tue, 2018-09-25 at 12:15 +0100, Julien Grall wrote:
> > > On 09/25/2018 10:02 AM, Dario Faggioli wrote:
> > > > On Mon, 2018-09-24 at 22:46 +0100, Julien Grall wrote:
> > > > >
> > My knowledge of RCU themselves would need refreshing, though. I
> > managed
> > to getbecome reasonably familiar with how the implementation we
> > imported works back then, when working on the said issue, but I guess
> > I
> > better go check the code again.
> >
> > I'm Cc-ing the people that have reviewed the patches and helping with
> > the idle timer problem, in case anyone has bright ideas out of the
> > top
> > of his head.
> >
> > Perhaps we should "just" get away from using RCU for domain
> > destruction
> > (but I'm just tossing the idea around, without much consideration
> > about
> > whether it's the right solution, or about how hard/feasible it really
> > is).
> >
> > Or maybe we can still use the timer, in some special way, if we have
> > wfi=native (or equivalent)...
> >
> So, I've had a look (but only a quick one).
>
> If we want to do something specific within the domain destruction path,
> we can add an rcu_barrier() there (I mean in domain_destroy()).
> However, that does not feel right either. Also, how can we be sure that
> the CPU never going through idle (as far as Xen knows, at least), isn't
> going to be problem for other RCU calls as well?
>
> Another thing that we can do is to act on the parameters that control
> the threshold which decides when a quiescent state is forced. This was
> basically what Julien was suggesting, but I still would avoid to do
> that always.
>
> So, basically, in this hackish patch attached, I added a new boot
> command line argument, called rcu_force_quiesc. If set to true,
> thresholds are set so that quiescence is always forced at each
> invocation of call_rcu(). And even if the new param is not explicitly
> specified, I do tweak the threshold when "wfi=native" is.
>
> Milan, can you apply this patch, add "wfi=native" again, and re-test?
> If it works, we'll decide what to do next.
>
> E.g., we can expose the RCU threshold via the appropriate set of boot
> time parameters --like Linux, from where this code comes, did/does--
> and document how they should be set, if one wants to use "wfi=native".
>
> Thanks and Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Software Engineer @ SUSE https://www.suse.com/

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Dario Faggioli]

On Tue, 2018-09-25 at 19:49 +0200, Dario Faggioli wrote:
> [Adding a few people to the Cc-list. See below...]
> On Tue, 2018-09-25 at 12:15 +0100, Julien Grall wrote:
> > On 09/25/2018 10:02 AM, Dario Faggioli wrote:
> > > On Mon, 2018-09-24 at 22:46 +0100, Julien Grall wrote:
> > > > 
> My knowledge of RCU themselves would need refreshing, though. I
> managed
> to getbecome reasonably familiar with how the implementation we
> imported works back then, when working on the said issue, but I guess
> I
> better go check the code again.
> 
> I'm Cc-ing the people that have reviewed the patches and helping with
> the idle timer problem, in case anyone has bright ideas out of the
> top
> of his head.
> 
> Perhaps we should "just" get away from using RCU for domain
> destruction
> (but I'm just tossing the idea around, without much consideration
> about
> whether it's the right solution, or about how hard/feasible it really
> is).
> 
> Or maybe we can still use the timer, in some special way, if we have
> wfi=native (or equivalent)...
> 
So, I've had a look (but only a quick one).

If we want to do something specific within the domain destruction path,
we can add an rcu_barrier() there (I mean in domain_destroy()).
However, that does not feel right either. Also, how can we be sure that
the CPU never going through idle (as far as Xen knows, at least), isn't
going to be problem for other RCU calls as well?

Another thing that we can do is to act on the parameters that control
the threshold which decides when a quiescent state is forced. This was
basically what Julien was suggesting, but I still would avoid to do
that always.

So, basically, in this hackish patch attached, I added a new boot
command line argument, called rcu_force_quiesc. If set to true,
thresholds are set so that quiescence is always forced at each
invocation of call_rcu(). And even if the new param is not explicitly
specified, I do tweak the threshold when "wfi=native" is.

Milan, can you apply this patch, add "wfi=native" again, and re-test?
If it works, we'll decide what to do next.

E.g., we can expose the RCU threshold via the appropriate set of boot
time parameters --like Linux, from where this code comes, did/does--
and document how they should be set, if one wants to use "wfi=native".

Thanks and Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/
commit 0d2beb3d4125d65c415860d66974db9a5532ac84
Author: Dario Faggioli 
Date:   Wed Sep 26 11:47:06 2018 +0200

xen: RCU: bootparam to force quiescence at every call.

Signed-off-by: Dario Faggioli 

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 0f4b1f2a5d..536eb17017 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -110,7 +110,10 @@ static enum {
 static int __init parse_vwfi(const char *s)
 {
 	if ( !strcmp(s, "native") )
+	{
+		rcu_always_quiesc = true;
 		vwfi = NATIVE;
+	}
 	else
 		vwfi = TRAP;
 
diff --git a/xen/common/rcupdate.c b/xen/common/rcupdate.c
index 3517790913..219dd2884f 100644
--- a/xen/common/rcupdate.c
+++ b/xen/common/rcupdate.c
@@ -140,6 +140,9 @@ static int qhimark = 1;
 static int qlowmark = 100;
 static int rsinterval = 1000;
 
+bool rcu_always_quiesc = false;
+boolean_param("rcu_force_quiesc", rcu_always_quiesc);
+
 struct rcu_barrier_data {
 struct rcu_head head;
 atomic_t *cpu_count;
@@ -562,6 +565,13 @@ static void rcu_init_percpu_data(int cpu, struct rcu_ctrlblk *rcp,
 rdp->quiescbatch = rcp->completed;
 rdp->qs_pending = 0;
 rdp->cpu = cpu;
+if ( rcu_always_quiesc )
+{
+blimit = INT_MAX;
+qhimark = 0;
+qlowmark = 0;
+//rsinterval = 0;
+}
 rdp->blimit = blimit;
 init_timer(&rdp->idle_timer, rcu_idle_timer_handler, rdp, cpu);
 }
diff --git a/xen/include/xen/rcupdate.h b/xen/include/xen/rcupdate.h
index 3402eb5caf..274a01acf6 100644
--- a/xen/include/xen/rcupdate.h
+++ b/xen/include/xen/rcupdate.h
@@ -56,6 +56,8 @@ struct rcu_head {
 } while (0)
 
 
+extern bool rcu_always_quiesc;
+
 int rcu_pending(int cpu);
 int rcu_needs_cpu(int cpu);
 


signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Dario Faggioli]

[Adding a few people to the Cc-list. See below...]

On Tue, 2018-09-25 at 12:15 +0100, Julien Grall wrote:
> Hi Dario,
> 
Hi,

> On 09/25/2018 10:02 AM, Dario Faggioli wrote:
> > On Mon, 2018-09-24 at 22:46 +0100, Julien Grall wrote:
> > > 
> > > Per my understanding of call_rcu, the calls will be queued until
> > > the
> > > RCU
> > > reached a threshold. We don't have many place where call_rcu is
> > > called,
> > > so reaching the threeshold may just never happen. But nothing
> > > will
> > > tell
> > > that vCPU to go in Xen and say "I am done with RCU". Did I miss
> > > anything?
> > > 
> > Yeah, and in fact we added the timer _but_, in this case, it does
> > not
> > look that the timer is firing. It looks much more like "some random
> > interrupt happens", as you're suggesting. OTOH, in the case where
> > there
> > are no printk()s, it might be that the timer does fire, but the
> > vcpu
> > has not gone through Xen, so the grace period is, as far as we
> > know,
> > not expired yet (which is also in accordance with Julien's
> > analysis, as
> > far as I understood it).
> 
> The timer is only activated when sched_tick_suspend() is called.
> With 
> vwfi=native, you will never reach the idle_loop() and therefore
> never 
> setup a timer.
> 
Right.

> Milan confirmed that guest can be destroyed with vwfi=native removed.
> So 
> this is confirming my thinking. Trapping wfi will end up to switch
> to 
> idle vCPU and trigger the grace period.
> 
Indeed.

> I am not entirely sure you will be able to reproduce it on x86, but
> I 
> don't think it is a Xen Arm specific.
> 
Perhaps with something like idle=poll. But I'm not sure, as it's not
the same thing (that is for guests, while wfi=native is in Xen).

Well, IAC, I don't think we can call it an ARM issue either. It's an
issue of whatever combination of platform and configuration we support,
which we allow not to trap when the guest goes idle.

> When I looked at the code, I don't see any grace period in other
> context 
> than idle_loop. Rather than adding another grace period, I would
> just 
> force quiescence for every call_rcu.
> 
> This should not be have a big performance impact as we don't use
> much 
> call_rcu and it would allow domain to be fully destroyed in timely
> manner.
> 
I see your point, but this looks a little on the heavy side to me.
I'm not speaking of direct and measurable perf impact, but I think this
would make us diverge quite a bit from the idea of RCUs...

My knowledge of RCU themselves would need refreshing, though. I managed
to getbecome reasonably familiar with how the implementation we
imported works back then, when working on the said issue, but I guess I
better go check the code again.

I'm Cc-ing the people that have reviewed the patches and helping with
the idle timer problem, in case anyone has bright ideas out of the top
of his head.

Perhaps we should "just" get away from using RCU for domain destruction
(but I'm just tossing the idea around, without much consideration about
whether it's the right solution, or about how hard/feasible it really
is).

Or maybe we can still use the timer, in some special way, if we have
wfi=native (or equivalent)...

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/


signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Milan Boberic]

Reply for Julien,
yes, my platform have 4 CPUs it's UltraZed-EG board with carrier card.
I use only 2 CPUs, one for dom0 which is PetaLinux and one for domU
which is bare-metal application that blinks LED on the board (I use it
to measure jitter with oscilloscope), other two CPUs are unused (in
idle loop).

About command, commad is from xen-overlay.dtsi file which is included
in system-user.dtsi file in my project. Whole file is included in
atatchment in my earlier reply.

About this options:
maxvcpus=1
core_parking=performance
cpufreq=xen:performance
I was just testing them to see will I get any performance improvement,
I will remove them right away.

Best regards, Milan Boberic!
On Tue, Sep 25, 2018 at 1:15 PM Julien Grall  wrote:
>
> Hi Dario,
>
> On 09/25/2018 10:02 AM, Dario Faggioli wrote:
> > On Mon, 2018-09-24 at 22:46 +0100, Julien Grall wrote:
> >> On 09/21/2018 05:20 PM, Dario Faggioli wrote:
> >>>
> >>> What I'm after, is how log, after domain_destroy(),
> >>> complete_domain_destroy() is called, and whether/how it relates the
> >>> the
> >>> grace period idle timer we've added in the RCU code.
> >>
> >> NULL scheduler and vwfi=native will inevitably introduce a latency
> >> when
> >> destroying a domain. vwfi=native means the guest will not trap when
> >> it
> >> has nothing to do and switch to the idle vCPU. So, in such
> >> configuration, it is extremely unlikely the execute the idle_loop or
> >> even enter in the hypervisor unless there are an interrupt on that
> >> pCPU.
> >>
> > Ah! I'm not familiar with wfi=native --and in fact I was completely
> > ignoring it-- but this analysis makes sense to me.
> >
> >> Per my understanding of call_rcu, the calls will be queued until the
> >> RCU
> >> reached a threshold. We don't have many place where call_rcu is
> >> called,
> >> so reaching the threeshold may just never happen. But nothing will
> >> tell
> >> that vCPU to go in Xen and say "I am done with RCU". Did I miss
> >> anything?
> >>
> > Yeah, and in fact we added the timer _but_, in this case, it does not
> > look that the timer is firing. It looks much more like "some random
> > interrupt happens", as you're suggesting. OTOH, in the case where there
> > are no printk()s, it might be that the timer does fire, but the vcpu
> > has not gone through Xen, so the grace period is, as far as we know,
> > not expired yet (which is also in accordance with Julien's analysis, as
> > far as I understood it).
>
> The timer is only activated when sched_tick_suspend() is called. With
> vwfi=native, you will never reach the idle_loop() and therefore never
> setup a timer.
>
> Milan confirmed that guest can be destroyed with vwfi=native removed. So
> this is confirming my thinking. Trapping wfi will end up to switch to
> idle vCPU and trigger the grace period.
>
> I am not entirely sure you will be able to reproduce it on x86, but I
> don't think it is a Xen Arm specific.
>
> When I looked at the code, I don't see any grace period in other context
> than idle_loop. Rather than adding another grace period, I would just
> force quiescence for every call_rcu.
>
> This should not be have a big performance impact as we don't use much
> call_rcu and it would allow domain to be fully destroyed in timely manner.
>
> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Julien Grall]

Hi Dario,

On 09/25/2018 10:02 AM, Dario Faggioli wrote:

On Mon, 2018-09-24 at 22:46 +0100, Julien Grall wrote:

On 09/21/2018 05:20 PM, Dario Faggioli wrote:


What I'm after, is how log, after domain_destroy(),
complete_domain_destroy() is called, and whether/how it relates the
the
grace period idle timer we've added in the RCU code.


NULL scheduler and vwfi=native will inevitably introduce a latency
when
destroying a domain. vwfi=native means the guest will not trap when
it
has nothing to do and switch to the idle vCPU. So, in such
configuration, it is extremely unlikely the execute the idle_loop or
even enter in the hypervisor unless there are an interrupt on that
pCPU.


Ah! I'm not familiar with wfi=native --and in fact I was completely
ignoring it-- but this analysis makes sense to me.


Per my understanding of call_rcu, the calls will be queued until the
RCU
reached a threshold. We don't have many place where call_rcu is
called,
so reaching the threeshold may just never happen. But nothing will
tell
that vCPU to go in Xen and say "I am done with RCU". Did I miss
anything?


Yeah, and in fact we added the timer _but_, in this case, it does not
look that the timer is firing. It looks much more like "some random
interrupt happens", as you're suggesting. OTOH, in the case where there
are no printk()s, it might be that the timer does fire, but the vcpu
has not gone through Xen, so the grace period is, as far as we know,
not expired yet (which is also in accordance with Julien's analysis, as
far as I understood it).


The timer is only activated when sched_tick_suspend() is called. With 
vwfi=native, you will never reach the idle_loop() and therefore never 
setup a timer.


Milan confirmed that guest can be destroyed with vwfi=native removed. So 
this is confirming my thinking. Trapping wfi will end up to switch to 
idle vCPU and trigger the grace period.


I am not entirely sure you will be able to reproduce it on x86, but I 
don't think it is a Xen Arm specific.


When I looked at the code, I don't see any grace period in other context 
than idle_loop. Rather than adding another grace period, I would just 
force quiescence for every call_rcu.


This should not be have a big performance impact as we don't use much 
call_rcu and it would allow domain to be fully destroyed in timely manner.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Julien Grall]

On 09/25/2018 11:12 AM, Milan Boberic wrote:

Hello guys,


Hi Milan,


mapping on my system is:
dom0 have one vCPU and it is pinned on pCPU0
domU also have one vCPU and it's pinned for pCPU2


Your platform has 4 CPUs, right? What does the other do? Just sitting in 
the idle loop?




I removed only vwfi=native and everything works fine. I can destroy
and create a guest as many times as I want with out any error (still
using sched=null).


Thank you for testing, this is quite helpful to know that removing 
vwfi=native makes a difference.




These are xen bootargs in Xen-overlay.dtsi file:
  xen,xen-bootargs = "console=dtuart dtuart=serial0 dom0_mem=768M
bootscrub=0 maxcpus=1 dom0_max_vcpus=1 dom0_vcpus_pin=true
timer_slop=0 core_parking=performance cpufreq=xen:performance
sched=null";



Where does this command line comes from?

There are options that does not make sense for Arm:
- maxvcpus=1
- core_parking=performance
- cpufreq=xen:performance

All of those options are not supported on Arm. The first one is quite 
concerning because you request to limit the number of pCPUs used by Xen. 
This is not available today. If we ever add support, you would end to 
have only 1 pCPUs.




There is whole xen-overlay.dtsi file included in attachment.

Purpose of my work is implementing xen on UltraZed-EG board with
maximum performance (and lowest jitter, which is why I'm using null
scheduler and "hard" vcpu pinning) which is my master's thesis on
faculty. By removing vwfi=native I'm not getting the best performance,
right? vwfi=native should decrease interrupt latency by ~60% as I read
here:

https://blog.xenproject.org/author/stefano-stabellini/


IIRC the test was not done with NULL scheduler. So the interrupt latency 
may slightly be better. However, there will still be a performance 
impact as the scheduler may decide to switch to idle vCPU.


It is possible to reduce the overhead of switch to idle vCPU by 
optimizing the context switch.


Anyway, vwfi=native should not affect destroying guest. This should 
probably be fixed. I will answer on Dario's e-mail.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Milan Boberic]

Hello guys,
mapping on my system is:
dom0 have one vCPU and it is pinned on pCPU0
domU also have one vCPU and it's pinned for pCPU2

I removed only vwfi=native and everything works fine. I can destroy
and create a guest as many times as I want with out any error (still
using sched=null).

These are xen bootargs in Xen-overlay.dtsi file:
 xen,xen-bootargs = "console=dtuart dtuart=serial0 dom0_mem=768M
bootscrub=0 maxcpus=1 dom0_max_vcpus=1 dom0_vcpus_pin=true
timer_slop=0 core_parking=performance cpufreq=xen:performance
sched=null";

There is whole xen-overlay.dtsi file included in attachment.

Purpose of my work is implementing xen on UltraZed-EG board with
maximum performance (and lowest jitter, which is why I'm using null
scheduler and "hard" vcpu pinning) which is my master's thesis on
faculty. By removing vwfi=native I'm not getting the best performance,
right? vwfi=native should decrease interrupt latency by ~60% as I read
here:

https://blog.xenproject.org/author/stefano-stabellini/

Big thanks to Julien for having a look!
/ {
chosen {
#address-cells = <2>;
#size-cells = <1>;

xen,xen-bootargs = "console=dtuart dtuart=serial0 dom0_mem=768M 
bootscrub=0 maxcpus=1 dom0_max_vcpus=1 dom0_vcpus_pin=true timer_slop=0 
core_parking=performance cpufreq=xen:performance sched=null";
xen,dom0-bootargs = "console=hvc0 earlycon=xen earlyprintk=xen 
maxcpus=1 clk_ignore_unused";

dom0 {
compatible = "xen,linux-zimage", "xen,multiboot-module";
reg = <0x0 0x8 0x310>;
};
};

};

&smmu {
status = "okay";
mmu-masters = < &gem0 0x874
&gem1 0x875
&gem2 0x876
&gem3 0x877
&dwc3_0 0x860
&dwc3_1 0x861
&qspi 0x873
&lpd_dma_chan1 0x868
&lpd_dma_chan2 0x869
&lpd_dma_chan3 0x86a
&lpd_dma_chan4 0x86b
&lpd_dma_chan5 0x86c
&lpd_dma_chan6 0x86d
&lpd_dma_chan7 0x86e
&lpd_dma_chan8 0x86f
&fpd_dma_chan1 0x14e8
&fpd_dma_chan2 0x14e9
&fpd_dma_chan3 0x14ea
&fpd_dma_chan4 0x14eb
&fpd_dma_chan5 0x14ec
&fpd_dma_chan6 0x14ed
&fpd_dma_chan7 0x14ee
&fpd_dma_chan8 0x14ef
&sdhci0 0x870
&sdhci1 0x871
&nand0 0x872>;
};

&uart1 {
   xen,passthrough = <0x1>;
};

&gpio {
   xen,passthrough = <0x1>;
};___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Dario Faggioli]

On Mon, 2018-09-24 at 22:46 +0100, Julien Grall wrote:
> On 09/21/2018 05:20 PM, Dario Faggioli wrote:
> > 
> > What I'm after, is how log, after domain_destroy(),
> > complete_domain_destroy() is called, and whether/how it relates the
> > the
> > grace period idle timer we've added in the RCU code.
> 
> NULL scheduler and vwfi=native will inevitably introduce a latency
> when 
> destroying a domain. vwfi=native means the guest will not trap when
> it 
> has nothing to do and switch to the idle vCPU. So, in such 
> configuration, it is extremely unlikely the execute the idle_loop or 
> even enter in the hypervisor unless there are an interrupt on that
> pCPU.
> 
Ah! I'm not familiar with wfi=native --and in fact I was completely
ignoring it-- but this analysis makes sense to me.

> Per my understanding of call_rcu, the calls will be queued until the
> RCU 
> reached a threshold. We don't have many place where call_rcu is
> called, 
> so reaching the threeshold may just never happen. But nothing will
> tell 
> that vCPU to go in Xen and say "I am done with RCU". Did I miss
> anything?
> 
Yeah, and in fact we added the timer _but_, in this case, it does not
look that the timer is firing. It looks much more like "some random
interrupt happens", as you're suggesting. OTOH, in the case where there
are no printk()s, it might be that the timer does fire, but the vcpu
has not gone through Xen, so the grace period is, as far as we know,
not expired yet (which is also in accordance with Julien's analysis, as
far as I understood it).

> I have the feeling the problem might just be exacerbated the problem 
> (simlar to the idle bug with credit2) by vwfi=ative. Milan, would it
> be 
> possible to run the test without that option?
> 
Indeed. And thanks, Julien, for having a look! :-)

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/


signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Julien Grall]

Hi,

On 09/21/2018 05:20 PM, Dario Faggioli wrote:

[Adding Julien as well.

Julien, this seems related to the RCU issue we fought on ARM when using
Credit2, although this is null, but it's being even more weird...]

On Fri, 2018-09-21 at 16:14 +0200, Milan Boberic wrote:

Hey,
yes, I can see prink's outputs on console and in xl dmesg. Also added
timestamps, here are the results (created and destroyed domU a few
times, just to get more values), this is from xl dmesg:

NULL SCHEDULER - Not stressed PetaLinux host domain.

(XEN) t=218000327743:End of a domain_destroy function
(XEN) t=218000420874:End of a complete_domain_destroy function
(XEN) <3>memory_map:add: dom2 gfn=ff0a0 mfn=ff0a0 nr=1
...

Stressed PetaLinux host with command: yes > /dev/null &

(XEN) t=3247747255872:End of a domain_destroy function
(XEN) t=3247747349863:End of a complete_domain_destroy function
...

CREDIT SCHEDULER - not stressed PetaLinux host

(XEN) t=86245669606:End of a domain_destroy function
(XEN) t=86245761127:End of a complete_domain_destroy function
...

Stressed PetaLinux host with yes > /dev/null &

(XEN) t=331229997499:End of a domain_destroy function
(XEN) t=331230091770:End of a complete_domain_destroy function
...


Which, if I'm doing the math properly, tells us that
complete_domain_destroy() is called within ~90us, for both schedulers,
and in all stress/load conditions. That wouldn't be too bad, I think.

And in fact, if I remember correctly, you're saying that adding the
printk()s fixes the issue in null. I wonder why that is... Can you like
kill the printks, store 5 or so of the timestamps (or just the delta)
in a static array or something, and print it from somewhere else (like
a debug-key handler in the same file)?


I think you may end up to receive an interrupt on CPU0 when using 
printk. What is the mapping between vCPU and pCPU on your system?




What I'm after, is how log, after domain_destroy(),
complete_domain_destroy() is called, and whether/how it relates the the
grace period idle timer we've added in the RCU code.


NULL scheduler and vwfi=native will inevitably introduce a latency when 
destroying a domain. vwfi=native means the guest will not trap when it 
has nothing to do and switch to the idle vCPU. So, in such 
configuration, it is extremely unlikely the execute the idle_loop or 
even enter in the hypervisor unless there are an interrupt on that pCPU.


Per my understanding of call_rcu, the calls will be queued until the RCU 
reached a threshold. We don't have many place where call_rcu is called, 
so reaching the threeshold may just never happen. But nothing will tell 
that vCPU to go in Xen and say "I am done with RCU". Did I miss anything?


I have the feeling the problem might just be exacerbated the problem 
(simlar to the idle bug with credit2) by vwfi=ative. Milan, would it be 
possible to run the test without that option?


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Dario Faggioli]

[Adding Julien as well.

Julien, this seems related to the RCU issue we fought on ARM when using
Credit2, although this is null, but it's being even more weird...]

On Fri, 2018-09-21 at 16:14 +0200, Milan Boberic wrote:
> Hey,
> yes, I can see prink's outputs on console and in xl dmesg. Also added
> timestamps, here are the results (created and destroyed domU a few
> times, just to get more values), this is from xl dmesg:
> 
> NULL SCHEDULER - Not stressed PetaLinux host domain.
> 
> (XEN) t=218000327743:End of a domain_destroy function
> (XEN) t=218000420874:End of a complete_domain_destroy function
> (XEN) <3>memory_map:add: dom2 gfn=ff0a0 mfn=ff0a0 nr=1
> ...
> 
> Stressed PetaLinux host with command: yes > /dev/null &
> 
> (XEN) t=3247747255872:End of a domain_destroy function
> (XEN) t=3247747349863:End of a complete_domain_destroy function
> ...
>
> CREDIT SCHEDULER - not stressed PetaLinux host
> 
> (XEN) t=86245669606:End of a domain_destroy function
> (XEN) t=86245761127:End of a complete_domain_destroy function
> ...
> 
> Stressed PetaLinux host with yes > /dev/null &
> 
> (XEN) t=331229997499:End of a domain_destroy function
> (XEN) t=331230091770:End of a complete_domain_destroy function
> ...
>
Which, if I'm doing the math properly, tells us that
complete_domain_destroy() is called within ~90us, for both schedulers,
and in all stress/load conditions. That wouldn't be too bad, I think.

And in fact, if I remember correctly, you're saying that adding the
printk()s fixes the issue in null. I wonder why that is... Can you like
kill the printks, store 5 or so of the timestamps (or just the delta)
in a static array or something, and print it from somewhere else (like
a debug-key handler in the same file)?

What I'm after, is how log, after domain_destroy(),
complete_domain_destroy() is called, and whether/how it relates the the
grace period idle timer we've added in the RCU code.

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/


signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Milan Boberic]

Hey,
yes, I can see prink's outputs on console and in xl dmesg. Also added
timestamps, here are the results (created and destroyed domU a few
times, just to get more values), this is from xl dmesg:

NULL SCHEDULER - Not stressed PetaLinux host domain.

(XEN) t=218000327743:End of a domain_destroy function
(XEN) t=218000420874:End of a complete_domain_destroy function
(XEN) <3>memory_map:add: dom2 gfn=ff0a0 mfn=ff0a0 nr=1
(XEN) t=250216234232:End of a domain_destroy function
(XEN) t=250216326943:End of a complete_domain_destroy function
(XEN) <3>memory_map:add: dom3 gfn=ff0a0 mfn=ff0a0 nr=1
(XEN) t=842722811648:End of a domain_destroy function
(XEN) t=842722906089:End of a complete_domain_destroy function
(XEN) <3>memory_map:add: dom4 gfn=ff0a0 mfn=ff0a0 nr=1
(XEN) t=845884879948:End of a domain_destroy function
(XEN) t=845884972539:End of a complete_domain_destroy function
(XEN) <3>memory_map:add: dom5 gfn=ff0a0 mfn=ff0a0 nr=1
(XEN) t=847696699336:End of a domain_destroy function
(XEN) t=847696793097:End of a complete_domain_destroy function
(XEN) <3>memory_map:add: dom6 gfn=ff0a0 mfn=ff0a0 nr=1
(XEN) t=919260640576:End of a domain_destroy function
(XEN) t=919260732907:End of a complete_domain_destroy function

Stressed PetaLinux host with command: yes > /dev/null &

(XEN) t=3247747255872:End of a domain_destroy function
(XEN) t=3247747349863:End of a complete_domain_destroy function
(XEN) t=3253855263752:End of a domain_destroy function
(XEN) t=3253855357563:End of a complete_domain_destroy function
(XEN) <3>memory_map:add: dom10 gfn=ff0a0 mfn=ff0a0 nr=1
(XEN) t=3256797406444:End of a domain_destroy function
(XEN) t=3256797498584:End of a complete_domain_destroy function
(XEN) <3>memory_map:add: dom11 gfn=ff0a0 mfn=ff0a0 nr=1
(XEN) t=3259750219352:End of a domain_destroy function
(XEN) t=3259750313903:End of a complete_domain_destroy function
(XEN) <3>memory_map:add: dom12 gfn=ff0a0 mfn=ff0a0 nr=1
(XEN) t=3262418200662:End of a domain_destroy function
(XEN) t=3262418295912:End of a complete_domain_destroy function


CREDIT SCHEDULER - not stressed PetaLinux host

(XEN) t=86245669606:End of a domain_destroy function
(XEN) t=86245761127:End of a complete_domain_destroy function
(XEN) t=93862614736:End of a domain_destroy function
(XEN) t=93862702657:End of a complete_domain_destroy function
(XEN) t=96298736227:End of a domain_destroy function
(XEN) t=96298826618:End of a complete_domain_destroy function
(XEN) t=98042498304:End of a domain_destroy function
(XEN) t=98042590255:End of a complete_domain_destroy function
(XEN) t=99755209642:End of a domain_destroy function
(XEN) t=99755302643:End of a complete_domain_destroy function
(XEN) t=101441462434:End of a domain_destroy function
(XEN) t=101441553985:End of a complete_domain_destroy function

Stressed PetaLinux host with yes > /dev/null &

(XEN) t=331229997499:End of a domain_destroy function
(XEN) t=331230091770:End of a complete_domain_destroy function
(XEN) t=334493673726:End of a domain_destroy function
(XEN) t=334493765647:End of a complete_domain_destroy function
(XEN) t=336834521515:End of a domain_destroy function
(XEN) t=336834613326:End of a complete_domain_destroy function
(XEN) t=339215230042:End of a domain_destroy function
(XEN) t=339215321153:End of a complete_domain_destroy function

Also thank you for taking your time to help me, best regards!

On Thu, Sep 20, 2018 at 6:09 PM Dario Faggioli  wrote:
>
> Hey,
>
> Sorry for not having followed up. I was (and still am) planning to, but
> am also a bit busy.
>
> On Thu, 2018-09-20 at 15:04 +0200, Milan Boberic wrote:
> > I ran some more tests and managed to successfully create and destroy
> > domU as many times as I want, without any delay between destroy and
> > create.
> > I added:
> >  printk("End of a domain_destroy function");
> >  in domain_destroy function and
> > printk("End of a complete_domain_destroy function"); in
> > complete_domain_destroy function, at the end of the functions.
> > Those functions are in domain.c file.
> >
> Right. This is exactly the kind of debug patch I wanted to
> suggest/send. It is, in fact, what was being use to diagnose/fix the
> RCU issue, when it first came up, as you may have seen.
>
> > Now, after every xl create it prints:
> >
> > root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
> > Parsing config from bm1.cfg
> > <3>memory_map:add: dom2 gfn=ff0a0 mfn=ff0a0 nr=1
> >
> Mmm... So, you added a printk() (or, actually, two of them) in the
> domain destruction path, and are seeing (different) things being
> printed during domain creation? How nice! :-D
>
> I'm not sure how that happens, and whether/how this new output relates
> to the problem. However, what about the printk() you added? Do you see
> their output? It may be visible only on the console and/or in `xl
> dmesg'.
>
> While there, if you can, I'd add timestamsp, e.g.:
>
> printk("t=%"PRI_stime":End of a domain_destroy function", NOW());
> printk("t=%"PRI_stime":End of a complete_domain_destroy functio

Re: [Xen-devel] null scheduler bug

[Dario Faggioli]

Hey,

Sorry for not having followed up. I was (and still am) planning to, but
am also a bit busy.

On Thu, 2018-09-20 at 15:04 +0200, Milan Boberic wrote:
> I ran some more tests and managed to successfully create and destroy
> domU as many times as I want, without any delay between destroy and
> create.
> I added:
>  printk("End of a domain_destroy function");
>  in domain_destroy function and
> printk("End of a complete_domain_destroy function"); in
> complete_domain_destroy function, at the end of the functions.
> Those functions are in domain.c file.
>
Right. This is exactly the kind of debug patch I wanted to
suggest/send. It is, in fact, what was being use to diagnose/fix the
RCU issue, when it first came up, as you may have seen.

> Now, after every xl create it prints:
> 
> root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
> Parsing config from bm1.cfg
> <3>memory_map:add: dom2 gfn=ff0a0 mfn=ff0a0 nr=1
> 
Mmm... So, you added a printk() (or, actually, two of them) in the
domain destruction path, and are seeing (different) things being
printed during domain creation? How nice! :-D

I'm not sure how that happens, and whether/how this new output relates
to the problem. However, what about the printk() you added? Do you see
their output? It may be visible only on the console and/or in `xl
dmesg'.

While there, if you can, I'd add timestamsp, e.g.:

printk("t=%"PRI_stime":End of a domain_destroy function", NOW());
printk("t=%"PRI_stime":End of a complete_domain_destroy function", NOW());

and check what the outpur looks like both under credit or credit2, and
under null.

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/


signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Milan Boberic]

I ran some more tests and managed to successfully create and destroy
domU as many times as I want, without any delay between destroy and
create.
I added:
 printk("End of a domain_destroy function");
 in domain_destroy function and
printk("End of a complete_domain_destroy function"); in
complete_domain_destroy function, at the end of the functions.
Those functions are in domain.c file.
Now, after every xl create it prints:

root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
Parsing config from bm1.cfg
<3>memory_map:add: dom2 gfn=ff0a0 mfn=ff0a0 nr=1


This line never printed before but it doesn't affect anything:
<3>memory_map:add: dom2 gfn=ff0a0 mfn=ff0a0 nr=1

I tried removing printk from functions and I got same problem like before.

Do you guys have any idea what is going on here?
Thanks in advance, best regards!

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Milan Boberic]

Thank you for taking your time to deal with this problem.
I did more testing just to be sure and I also measured time (using
stopwatch on my phone which isn't precise at all, just wanted You to
get the feeling of what time intervals are we talking about).
Yes, I can confirm that that situation actually improves with Xen
4.10, which is why I'm going to continue to use it.

With Xen 4.9.2 after I create a guest and destroy it (note that it is
a guest with pass through which blinks GPIO PS LED) I can't re-create
it again. Never. Not even after 30 seconds, 2 minutes, 5 minutes,
etc...

These are testing results with Xen 4.10:

1.) I created a guest, destroyed it and immediately after that tried
to create it again (manualy, over keyboard, for that I need maybe half
a second or a second to hit twice "arrow up" and "enter" buttons on
keyboard) and this shows:

root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
root@uz3eg-iocc-2018-2:~# xl destroy bm1
root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
Parsing config from bm1.cfg
(XEN) IRQ 48 is already used by domain 27
libxl: error: libxl_create.c:1325:domcreate_launch_dm: Domain
28:failed give domain access to irq 48: Device or resource busy
libxl: error: libxl_domain.c:1000:libxl__destroy_domid: Domain
28:Non-existant domain
libxl: error: libxl_domain.c:959:domain_destroy_callback: Domain
28:Unable to destroy guest
libxl: error: libxl_domain.c:886:domain_destroy_cb: Domain
28:Destruction of domain failed

2.) Here I createed a guest, destroyed it and then immediately ran xl
create twice, fast. For that I also need like half a second or second.
Note that guest isn't in any state, is should be in "running" state
because I need that PS LED to blink.

root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
root@uz3eg-iocc-2018-2:~# xl destroy bm1
root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
Parsing config from bm1.cfg
(XEN) IRQ 48 is already used by domain 32
libxl: error: libxl_create.c:1325:domcreate_launch_dm: Domain
33:failed give domain access to irq 48: Device or resource busy
libxl: error: libxl_domain.c:1000:libxl__destroy_domid: Domain
33:Non-existant domain
libxl: error: libxl_domain.c:959:domain_destroy_callback: Domain
33:Unable to destroy guest
libxl: error: libxl_domain.c:886:domain_destroy_cb: Domain
33:Destruction of domain failed
root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
Parsing config from bm1.cfg
root@uz3eg-iocc-2018-2:~# xl list
NameID   Mem VCPUs  State   Time(s)
Domain-00 768 1
r-1936.2
bm1   34 8 1
 --  0.0

3.) Here I did same thing like in 2.) except I waited 6-7 seconds
after error pops and then ran xl create and guest worked fine (it is
in "running state"), so:

root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
Parsing config from bm1.cfg
root@uz3eg-iocc-2018-2:~# xl destroy bm1
root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
Parsing config from bm1.cfg
(XEN) IRQ 48 is already used by domain 57
libxl: error: libxl_create.c:1325:domcreate_launch_dm: Domain
58:failed give domain access to irq 48: Device or resource busy
libxl: error: libxl_domain.c:1000:libxl__destroy_domid: Domain
58:Non-existant domain
libxl: error: libxl_domain.c:959:domain_destroy_callback: Domain
58:Unable to destroy guest
libxl: error: libxl_domain.c:886:domain_destroy_cb: Domain
58:Destruction of domain failed

/* waited for approximately 6-7 seconds and then ran command bellow */

root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
Parsing config from bm1.cfg
root@uz3eg-iocc-2018-2:~# xl list
NameID   Mem VCPUs  State   Time(s)
Domain-0 0   768 1 r-3071.5
bm159   8 1
r-  8.2

4.) Here I createed a guest, destroyed it and then waited for
approximately 7 seconds and then ran xl create and everything worked
fine:

root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
Parsing config from bm1.cfg
root@uz3eg-iocc-2018-2:~# xl destroy bm1

/* waited for approximately 7 seconds and then ran command bellow */

root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
Parsing config from bm1.cfg
root@uz3eg-iocc-2018-2:~# xl list
NameID   Mem VCPUs  State   Time(s)
Domain-0 0   768 1 r-3641.1
bm170   8 1
r-  7.1

It looks like guest needs approximately 7 seconds to be fully
destroyed and to fully release IRQ.
And yes, if you menage to produce patch I will put it in my source
tree, build with it, test it and send you back the results.
In attachment I included dmesg, xl dmesg from xen 4.10.

On Thu, Sep 13, 2018 at 7:39 PM Dario Faggioli  wrote:
>
> On Thu, 2018-09-13 at 17:18 +0200, Milan Boberic wrote:
> > Commits are there and I will definitely continue with 4.10 ver

Re: [Xen-devel] null scheduler bug

[Dario Faggioli]

On Thu, 2018-09-13 at 17:18 +0200, Milan Boberic wrote:
> Commits are there and I will definitely continue with 4.10 version.
> But it didn't solve my problem entirely.
> 
> I create my bare-metal application (with xl create) and destroy it
> with xl destroy (it disappears from xl list) and when I try to create
> it again same error pops but if I immediately run xl create command
> again it creates it without error.
> If I run xl create twice fast sometimes bare-metal application isn't
> in any state (it should be in "running" state).
> If I wait some time (approximately between 30 and 90 seconds) after
> destruction of that bm app and then run xl create it will create it
> without error.
> 
Ok, thanks for trying and reporting back.

If possible, help me understand things a bit better.

So, can you confirm that the situation _actually_improves_ with Xen
4.10 ? 

Basically, as far as I've understood things, with Xen 4.9, you destroy
a guest, and you can _never_ re-create it, not even after 30 seconds,
90 seconds, 2 minutes, 1 hour, ecc. Is that correct?

With Xen 4.10, it may still fail, if you try to re-create it within ~30
to 90 seconds, but after that, it works. Is that also correct?

I need to know this, because I want to understand if the issue is, at
least partially, cured by the RCU fixes, although having to wait 30
seconds is definitely not what I was expecting (i.e., there might be
something else).

Another question, in case I manage to produce a debug patch, are you ok
to put it in your source tree, build with it, and tell us what you see?

Thanks again and Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/


signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Milan Boberic]

I'm sorry for html and dropping xen-devel and dropping other CCs,
missed to read the rules.
I tried 4.10 version and checked for commits you asked in earlier reply.

2b936ea7b "xen: RCU: avoid busy waiting until the end of grace period."
38ad8151f "xen: RCU: don't let a CPU with a callback go idle."

Commits are there and I will definitely continue with 4.10 version.
But it didn't solve my problem entirely.

I create my bare-metal application (with xl create) and destroy it
with xl destroy (it disappears from xl list) and when I try to create
it again same error pops but if I immediately run xl create command
again it creates it without error.
If I run xl create twice fast sometimes bare-metal application isn't
in any state (it should be in "running" state).
If I wait some time (approximately between 30 and 90 seconds) after
destruction of that bm app and then run xl create it will create it
without error.

This is github repository of Xen I use:
https://github.com/Xilinx/xen/tree/xilinx/stable-4.10

Is there anything else I can send that would be helpful?

Thanks in advance, best regards, Milan Boberic.

On Thu, Sep 13, 2018 at 10:24 AM Dario Faggioli  wrote:
>
> Hi,
>
> So, first of all:
> 1. use plaintext, not HTML
> 2. don't drop the xen-devel list (and other Cc-s) when replying.
>
> :-)
>
> That being said...
>
> On Thu, 2018-09-13 at 09:38 +0200, Milan Boberic wrote:
> > Hi Dario,
> > yes passtrhough is involved.
> >
> Ok.
>
> > This is everything I did so far:
> >
> > I implemented Xen Hypervisor 4.9.2 on UltraZed-EG board with carrier
> > card following these steps:
> > 1.) installed petalinux 2018.2 on Ubuntu 16.04
> > 2.) dowloaded  UltraZed-EG IO Carrier Card - PetaLinux 2018.2
> > Standard BSP
> > 3.) created project:   petalinux-create -t project –s  
> > 4.) copied xen-overlay.dtsi from ZCU102 project (also made with BSP)
> > to my project
> > 5.) built xen hypervisor following this guide link  (booting with SD
> > card)
> > 6.) made baremetal application that blinks PS LED with Vivado
> > 7.) measured signals jitted when application is on board with and
> > with out Xen
> >
> I am not familiar with building an SD-Card image for an ARM system with
> Xen on it. What I can tell, though, is that Xen 4.9.2 does not look to
> me to include the RCU subsystem fixes that I mentioned in my reply.
>
> I don't recall why we did not backport them. It may be that they're not
> entirely trivial, and they fix a behavior which manifests only in not
> fully supported cases. Or that we (well, I :-/) forgot.
>
> I can have a look, at how challenging it is to apply the patches to
> 4.9.x (but no hard feelings if anyone beats me at it, I promise
> :-D).
>
> In the meantime, if you have the chance of trying Xen 4.10 or 4.10.1,
> which has those fixes, that would be great. In fact, in case everything
> would work with such version, that would be another clue that the RCU
> issue is the actual culprit.
>
> > I menaged to decrease jitter adding sched=null vwfi=native in xen-
> > overlay.dtsi file in xen-bootargs.
> >
> > The problem is when I add sched=null vwfi=native, I can create my
> > bare-metal application with xl create and destroy it with xl destroy
> > (it disappears from xl list) but when I want to start it again (xl
> > create) this error pops:
> >
> > root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
> > Parsing config from bm1.cfg
> > (XEN) IRQ 48 is already used by domain 1
> > libxl: error: libxl_create.c:1278:domcreate_launch_dm: Domain
> > 2:failed give domain access to irq 48: Device or resource busy
> > libxl: error: libxl_domain.c:1003:libxl__destroy_domid: Domain 2:Non-
> > existant domain
> > libxl: error: libxl_domain.c:962:domain_destroy_callback: Domain
> > 2:Unable to destroy guest
> > libxl: error: libxl_domain.c:889:domain_destroy_cb: Domain
> > 2:Destruction of domain failed
> >
> > When I remove  sched=null vwfi=native and build project again
> > everything works fine, I can create and destroy bm app as many times
> > as I want.
> >
> Yes. If you look at the email (and to the full thread) I put links to,
> you'll see that the behavior is exactly the same. It mentions the
> Credit2 scheduler not working, rather than null, but the problem is the
> same, i.e., that there is no periodic timer tick neither in Credit2 nor
> in null.
>
> Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Software Engineer @ SUSE https://www.suse.com/

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] null scheduler bug

[Milan Boberic]

Hi,
I implemented Xen Hypervisor 4.9.2 on UltraZed-EG board with carrier card
following these steps:
1.) installed petalinux on Ubuntu 16.04
2.) dowloaded  UltraZed-EG IO Carrier Card - PetaLinux 2018.2 Standard BSP
3.) created project:   petalinux-create -t project –s  
4.) copied xen-overlay.dtsi from ZCU102 project (also made with BSP) to my
project
5.) built xen hypervisor following this guide link

 (booting
with SD card)
6.) made baremetal application that blinks PS LED with Vivado
7.) measured signals jitted when application is on board with and with out
Xen

I menaged to decrease jitter adding following in xen-overlay.dtsi file in
xen-bootargs: sched=null vwfi=native.

The problem is when I add sched=null vwfi=native I can create my bare-metal
application with xl create and destroy it with xl destroy (disapears from
xl list) but when I want to start it again (xl create) this error pops:

root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
Parsing config from bm1.cfg
(XEN) IRQ 48 is already used by domain 1
libxl: error: libxl_create.c:1278:domcreate_launch_dm: Domain 2:failed give
domain access to irq 48: Device or resource busy
libxl: error: libxl_domain.c:1003:libxl__destroy_domid: Domain
2:Non-existant domain
libxl: error: libxl_domain.c:962:domain_destroy_callback: Domain 2:Unable
to destroy guest
libxl: error: libxl_domain.c:889:domain_destroy_cb: Domain 2:Destruction of
domain failed

When I remove  sched=null vwfi=native and build project again everything
works fine, I can create and destroy bm app as many times as i want.
Added in attachment bare-metal application's configuration file, xl dmesg,
dmesg and xl -v create message, xen-overlay.dtsi file and system-user.dtsi
file.
Thank you in adance, best regards, Milan Boberic.
name = "bm1"
kernel = "ultraled.bin"
memory = 8
vcpus = 1
cpus = [2]
irqs = [ 48 ]
iomem = [ "0xff0a0,1" ]
hap=0[0.00] Booting Linux on physical CPU 0x0
[0.00] Linux version 4.14.0-xilinx-v2018.2 (oe-user@oe-host) (gcc 
version 7.2.0 (GCC)) #1 SMP Wed Sep 12 13:32:35 CEST 2018
[0.00] Boot CPU: AArch64 Processor [410fd034]
[0.00] Machine model: xlnx,zynqmp
[0.00] Xen 4.9 support found
[0.00] efi: Getting EFI parameters from FDT:
[0.00] efi: UEFI not found.
[0.00] cma: Reserved 256 MiB at 0x6000
[0.00] On node 0 totalpages: 196608
[0.00]   DMA zone: 2688 pages used for memmap
[0.00]   DMA zone: 0 pages reserved
[0.00]   DMA zone: 196608 pages, LIFO batch:31
[0.00] psci: probing for conduit method from DT.
[0.00] psci: PSCIv0.2 detected in firmware.
[0.00] psci: Using standard PSCI v0.2 function IDs
[0.00] psci: Trusted OS migration not required
[0.00] random: fast init done
[0.00] percpu: Embedded 21 pages/cpu @ffc03ffb7000 s46488 r8192 
d31336 u86016
[0.00] pcpu-alloc: s46488 r8192 d31336 u86016 alloc=21*4096
[0.00] pcpu-alloc: [0] 0
[0.00] Detected VIPT I-cache on CPU0
[0.00] CPU features: enabling workaround for ARM erratum 845719
[0.00] Built 1 zonelists, mobility grouping on.  Total pages: 193920
[0.00] Kernel command line: console=hvc0 earlycon=xen earlyprintk=xen 
maxcpus=1 clk_ignore_unused
[0.00] PID hash table entries: 4096 (order: 3, 32768 bytes)
[0.00] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[0.00] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[0.00] Memory: 423276K/786432K available (9980K kernel code, 644K 
rwdata, 3132K rodata, 512K init, 2168K bss, 101012K reserved, 262144K 
cma-reserved)
[0.00] Virtual kernel memory layout:
[0.00] modules : 0xff80 - 0xff800800   (   128 
MB)
[0.00] vmalloc : 0xff800800 - 0xffbebfff   (   250 
GB)
[0.00]   .text : 0xff800808 - 0xff8008a4   (  9984 
KB)
[0.00] .rodata : 0xff8008a4 - 0xff8008d6   (  3200 
KB)
[0.00]   .init : 0xff8008d6 - 0xff8008de   (   512 
KB)
[0.00]   .data : 0xff8008de - 0xff8008e81200   (   645 
KB)
[0.00].bss : 0xff8008e81200 - 0xff800909f2b0   (  2169 
KB)
[0.00] fixed   : 0xffbefe7fd000 - 0xffbefec0   (  4108 
KB)
[0.00] PCI I/O : 0xffbefee0 - 0xffbeffe0   (16 
MB)
[0.00] vmemmap : 0xffbf - 0xffc0   ( 4 
GB maximum)
[0.00]   0xffbf0070 - 0xffbf0188   (17 
MB actual)
[0.00] memory  : 0xffc02000 - 0xffc07000   (  1280 
MB)
[0.00] Hierarchical RCU implementation.
[0.00]  RCU event tracing is enabled.
[0.00]  RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=1.
[0.00

Re: [Xen-devel] null scheduler bug

[Dario Faggioli]

Hi,

So, first of all:
1. use plaintext, not HTML
2. don't drop the xen-devel list (and other Cc-s) when replying.

:-)

That being said...

On Thu, 2018-09-13 at 09:38 +0200, Milan Boberic wrote:
> Hi Dario, 
> yes passtrhough is involved. 
>
Ok.

> This is everything I did so far: 
> 
> I implemented Xen Hypervisor 4.9.2 on UltraZed-EG board with carrier
> card following these steps:
> 1.) installed petalinux 2018.2 on Ubuntu 16.04
> 2.) dowloaded  UltraZed-EG IO Carrier Card - PetaLinux 2018.2
> Standard BSP
> 3.) created project:   petalinux-create -t project –s  
> 4.) copied xen-overlay.dtsi from ZCU102 project (also made with BSP)
> to my project
> 5.) built xen hypervisor following this guide link  (booting with SD
> card)
> 6.) made baremetal application that blinks PS LED with Vivado
> 7.) measured signals jitted when application is on board with and
> with out Xen
> 
I am not familiar with building an SD-Card image for an ARM system with
Xen on it. What I can tell, though, is that Xen 4.9.2 does not look to
me to include the RCU subsystem fixes that I mentioned in my reply.

I don't recall why we did not backport them. It may be that they're not
entirely trivial, and they fix a behavior which manifests only in not
fully supported cases. Or that we (well, I :-/) forgot.

I can have a look, at how challenging it is to apply the patches to
4.9.x (but no hard feelings if anyone beats me at it, I promise
:-D).

In the meantime, if you have the chance of trying Xen 4.10 or 4.10.1,
which has those fixes, that would be great. In fact, in case everything
would work with such version, that would be another clue that the RCU
issue is the actual culprit.

> I menaged to decrease jitter adding sched=null vwfi=native in xen-
> overlay.dtsi file in xen-bootargs.
> 
> The problem is when I add sched=null vwfi=native, I can create my
> bare-metal application with xl create and destroy it with xl destroy
> (it disappears from xl list) but when I want to start it again (xl
> create) this error pops:
> 
> root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
> Parsing config from bm1.cfg
> (XEN) IRQ 48 is already used by domain 1
> libxl: error: libxl_create.c:1278:domcreate_launch_dm: Domain
> 2:failed give domain access to irq 48: Device or resource busy
> libxl: error: libxl_domain.c:1003:libxl__destroy_domid: Domain 2:Non-
> existant domain
> libxl: error: libxl_domain.c:962:domain_destroy_callback: Domain
> 2:Unable to destroy guest
> libxl: error: libxl_domain.c:889:domain_destroy_cb: Domain
> 2:Destruction of domain failed
> 
> When I remove  sched=null vwfi=native and build project again
> everything works fine, I can create and destroy bm app as many times
> as I want.
> 
Yes. If you look at the email (and to the full thread) I put links to,
you'll see that the behavior is exactly the same. It mentions the
Credit2 scheduler not working, rather than null, but the problem is the
same, i.e., that there is no periodic timer tick neither in Credit2 nor
in null.

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/


signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] null scheduler bug

[Dario Faggioli]

On Wed, 2018-09-12 at 16:03 -0700, Stefano Stabellini wrote:
> Hi Dario,
> 
> Milan has just found a bug in the null scheduler: apparently it is
> not
> possible to start a VM again after it has been destroyed.
> 
> My initial suspicion was that the VM wasn't properly destroyed, but I
> asked Milan to double check with xl list, and the VM doesn't show in
> the
> list anymore.
> 
Ok. With what version of Xen? Is PCI passthrough/device assignment
involved? I'm asking because this highly reminds me of this:

https://lists.xen.org/archives/html/xen-devel/2017-07/msg00501.html

Which then led to this:

https://www.mail-archive.com/xen-devel@lists.xen.org/msg105388.html

And to the following changes (related to RCU, which is how complete
destruction of a domain occurs):

2b936ea7b "xen: RCU: avoid busy waiting until the end of grace period."
38ad8151f "xen: RCU: don't let a CPU with a callback go idle."
...

Are these commit there?

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/


signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] null scheduler bug

[Stefano Stabellini]

Hi Dario,

Milan has just found a bug in the null scheduler: apparently it is not
possible to start a VM again after it has been destroyed.

My initial suspicion was that the VM wasn't properly destroyed, but I
asked Milan to double check with xl list, and the VM doesn't show in the
list anymore.

Any ideas?

Cheers,

Stefano

On Wed, 12 Sep 2018, Milan Boberic wrote:
> I double checked it, when I remove sched=null and vwfi=native I can create 
> and destroy bare-metal application as many times as I
> want. 
> When I include sched=null and vwfi=native I can create it once and when I 
> destroy it I can't create it again, again the same
> error.
>
> root@uz3eg-iocc-2018-2:~# xl create bm1.cfg
> Parsing config from bm1.cfg
> (XEN) IRQ 48 is already used by domain 1
> libxl: error: libxl_create.c:1278:domcreate_launch_dm: Domain 2:failed give 
> domain access to irq 48: Device or resource busy
> libxl: error: libxl_domain.c:1003:libxl__destroy_domid: Domain 2:Non-existant 
> domain
> libxl: error: libxl_domain.c:962:domain_destroy_callback: Domain 2:Unable to 
> destroy guest
> libxl: error: libxl_domain.c:889:domain_destroy_cb: Domain 2:Destruction of 
> domain failed___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel