Hi Rasty,
Please reply-to-all, then your reply will also pop up in my Inbox.
On 10/12/2022 08:52, Rasty Slutsker wrote:
Hi,
We did some performance measurements.
Added counters in 3 places (per Irq source)
1. entry to jailhouse ISR
2. dispatch of interrupt to particular vector to particular core
3. in RTOS isr.
Okay. How do you read and dump them? I hope after everything is done.
Take care that if you dump them immediately to the uart, this will
consume a lot of time and cause significant delay. ('heavy' printk logic
+ busy waiting for the uart)
*We see that all 3 counters have the same value*, but we measure time
huh? What counters do you access? There's something odd if they hold the
same value at different places.
gaps in RTOS in ISR invocation times, sometimes upto 60 uSec.
It means that either
a) interrupt request is lost. But, according to setup it is
edge-triggered, it cannot be lost, just delayed.
b) there is a delay of more than 60 usec in jailhouse ISR.
questions:
1. Is it possible that jailhouse interrupt dispatching routine enters
some loop that takes considerable amount of time?
If you use printk during dispatching for debugging - yes. Otherwise, I
guess no. Not 60µs.
2. What would be explanation of interrupt latency of 60 Secs? Even if we
take into account cache line refill we get much lower number, which do
not reach tens uSecs.
Ack, I would definitely not expect 60µs delay for IRQ reinjection.
Thanks
Ralf
Best regards
Rasty
On Tuesday, December 6, 2022 at 6:01:20 PM UTC+2 Ralf Ramsauer wrote:
Hi,
On 05/12/2022 17:30, Rasty Slutsker wrote:
> Hi Ralf,
> Thank you for the answer.
> We have periodic interrupt each 30 u(!)Sec. Linux cannot deal
with such
> rate, so we need hypervisor/RTOS.
I understand.
> We managed to read a code of hypervisor. It appears that all
interrupts
> to all cores are intercepted by hypervisor and then forwarded to
guests
> (per core).
Yes, exactly, that's the case if you don't have SDEI. If you have a
platform that would come with SDEI, then you have of course less
overhead.
> If we reduce interrupt priority of mentioned interrupt (as you
suggest)
> we lose even more interrupts, even without stress.
> Interrupt is defined as edge triggered, I assumed that it is
memorized
> by gic until serviced.
> Is it possible that Hypervisor acknowledges pending interrupt while
> servicing interrupt from another source ? Kind of race - 2
interrupts
> for 2 cores arrive nearly simultaneously. One is lost.
The EOIR and IAR registers of the GIC are core-local registers of the
GIC CPU interface (GICC), so I wonder how this should cause a race,
unless there isn't a hard logical mistake in the code, which I doubt.
What you could try to do for debugging purposes:
1. Slow down from 30µs to something sloooower, which you can handle
even under load. Say 100µs, 500µs, something like that.
2. Measure the jitter x between arrival of the interrupt, and final
acknowledgement in your RTOS. You can use performance monitoring
registers, or watch CPU cycle counters, whatever. Repeat the
measurement, w/ load and w/o load on Linux-side.
3. If max(x) >= 30µs, then you know where your IRQs go in case of a
periodic cycle of 30µs.
Reason: What I did some while ago, is measuring the Jitter of
Linux+Jailhouse on ARM systems with cyclictest. On a Jetson TX1
platform, for example, we saw Jitter up 50µs. So there's IRQ
reinjection, a full Linux stack and some userspace application
involved,
so three context switches and lots of code. You have probably two
context switches and less code, as you use a RTOS, but I think
there's a
certain chance to exceed 30µs.
What my gut feeling tells me is that you manage to hold those 30µs if
Linux is quiet. As soon as there's some stress on the system bus, and
even on shared caches, you exceed you deadline.
Thanks
Ralf
>
> Best regards
> Rasty
>
> On Monday, December 5, 2022 at 5:14:37 PM UTC+2 Ralf Ramsauer wrote:
>
> Hi Nir,
>
> On 29/11/2022 14:21, nirge...@gmail.com wrote:
> > Hi there,
> >
> > Our target is Sitara AM5726 , CortexA15 dual core on which we are
> > running Linux on A15 core0 and RTOS on core1.
> >
> > __
> >
> > RTOS gets periodic interrupt from external hardware via nirq1 pin
> > (dedicated input into ARM gic).____
> >
> > Under heavy load in Linux (core 0!), RTOS, which runs on core1
> misses
> > interrupts.____
>
> Uhm. Can you reconstruct that issue w/o Jailhouse under Linux?
>
> I mean, can you set the SMP affinity of that IRQ to core 1 under
Linux,
> and then write some test application running on core 1 that just
> receives the IRQ. If that issue happens under Linux as well, then
you
> know that the issue has probably nothing to do with Jailhouse.
>
>
> What also might happen: If there's enough pressure on the shared
system
> bus when Linux is under load, then you simply loose those IRQs as
the
> RTOS doesn't have enough time to handle it. You can test this
> hypothesis
> if you lower the frequency of the the periodic interrupt. If you
still
> loose IRQs, then this should not be the case.
>
> >
> > Questions____
> >
> > 1. Does linux/hypervisor participate in interrupt
> scheduling/forwarding
> > to cell on Core1____
>
> Linux: No, Linux does not participate in anything that is going
on on
> CPU 1. That's the idea behind Jailhouse.
>
> Jailhouse: Maybe. On ARM platforms, Jailhouse needs to reinject the
> Interrupt from the hypervisor to the guest, if SDEI is not
available.
> Does the Sitara come with support for SDEI support?
>
> (You can btw monitor the exits of the hypervisor with 'jailhouse
cell
> stats')
>
> Ralf
>
> > 2. Is there a description of interrupt forwarding/virtualization
> scheme
> > to cores (if exists)? Any pointer to document/source code would be
> > appreciated.
> >
> > Thanks a lot,
> >
> > Nir.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Jailhouse" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to jailhouse-de...@googlegroups.com
> > <mailto:jailhouse-de...@googlegroups.com>.
> > To view this discussion on the web visit
> >
>
https://groups.google.com/d/msgid/jailhouse-dev/fde55f66-2e83-4df2-8f5e-44b0fb831acbn%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/fde55f66-2e83-4df2-8f5e-44b0fb831acbn%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/fde55f66-2e83-4df2-8f5e-44b0fb831acbn%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/fde55f66-2e83-4df2-8f5e-44b0fb831acbn%40googlegroups.com>>
<https://groups.google.com/d/msgid/jailhouse-dev/fde55f66-2e83-4df2-8f5e-44b0fb831acbn%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/fde55f66-2e83-4df2-8f5e-44b0fb831acbn%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/fde55f66-2e83-4df2-8f5e-44b0fb831acbn%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/fde55f66-2e83-4df2-8f5e-44b0fb831acbn%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jailhouse" group.
> To unsubscribe from this group and stop receiving emails from it,
send
> an email to jailhouse-de...@googlegroups.com
> <mailto:jailhouse-de...@googlegroups.com>.
> To view this discussion on the web visit
>
https://groups.google.com/d/msgid/jailhouse-dev/9f1616c7-ac5f-49de-bc24-8bd8520f4c07n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/9f1616c7-ac5f-49de-bc24-8bd8520f4c07n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/9f1616c7-ac5f-49de-bc24-8bd8520f4c07n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/9f1616c7-ac5f-49de-bc24-8bd8520f4c07n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
--
You received this message because you are subscribed to the Google
Groups "Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to jailhouse-dev+unsubscr...@googlegroups.com
<mailto:jailhouse-dev+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/01dfb141-0e4f-4adc-939f-c7a0c145c055n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/01dfb141-0e4f-4adc-939f-c7a0c145c055n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jailhouse-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/8bf22a0b-ec26-5974-e942-c5a0ebc9a028%40oth-regensburg.de.