Re: [PATCH] arm64: KVM: hyp: debug-sr: Mark expected switch fall-throughs

[Gustavo A. R. Silva]

On 8/7/19 10:11 AM, Marc Zyngier wrote:

> 
> Already fixed (together with all the other KVM/arm warnings/bugs), and
> pull request sent to Paolo.
> 

Awesome. :)

Thanks, Marc.
--
Gustavo

Re: [PATCH 1/9] KVM: arm64: Document PV-time interface

[Steven Price]

On 07/08/2019 15:28, Christophe de Dinechin wrote:
> 
> 
>> On 7 Aug 2019, at 15:21, Steven Price > > wrote:
>>
>> On 05/08/2019 17:40, Christophe de Dinechin wrote:
>>>
>>> Steven Price writes:
>>>
 Introduce a paravirtualization interface for KVM/arm64 based on the
 "Arm Paravirtualized Time for Arm-Base Systems" specification DEN 0057A.

 This only adds the details about "Stolen Time" as the details of "Live
 Physical Time" have not been fully agreed.

>>> [...]
>>>
 +
 +Stolen Time
 +---
 +
 +The structure pointed to by the PV_TIME_ST hypercall is as follows:
 +
 +  Field   | Byte Length | Byte Offset | Description
 +  --- | --- | --- | --
 +  Revision    |  4  |  0  | Must be 0 for version 0.1
 +  Attributes  |  4  |  4  | Must be 0
 +  Stolen time |  8  |  8  | Stolen time in unsigned
 +  | | | nanoseconds indicating how
 +  | | | much time this VCPU thread
 +  | | | was involuntarily not
 +  | | | running on a physical CPU.
>>>
>>> I know very little about the topic, but I don't understand how the spec
>>> as proposed allows an accurate reading of the relation between physical
>>> time and stolen time simultaneously. In other words, could you draw
>>> Figure 1 of the spec from within the guest? Or is it a non-objective?
>>
>> Figure 1 is mostly attempting to explain Live Physical Time (LPT), which
>> is not part of this patch series. But it does touch on stolen time by
>> the difference between "live physical time" and "virtual time".
>>
>> I'm not sure what you mean by "from within the guest". From the
>> perspective of the guest the parts of the diagram where the guest isn't
>> running don't exist (therefore there are discontinuities in the
>> "physical time" and "live physical time" lines).
> 
> I meant: If I run code within the guest that attempts to draw Figure 1,
> race conditions may cause the diagram actually drawn by your guest
> program to look completely wrong on occasions.
> 
>> This patch series doesn't attempt to provide the guest with a view of
>> "physical time" (or LPT) - but it might be able to observe that by
>> consulting something external (e.g. an NTP server, or an emulated RTC
>> which reports wall-clock time).
> 
> … with what appear to be like a built-in race condition, as you correctly
> identified. I was wondering if the built-in race condition was deliberate
> and/or necessary, or if it was irrelevant for the planned uses of the value.
> 
>> What it does provide is a mechanism for obtaining the difference (as
>> reported by the host) between "live physical time" and "virtual time" -
>> this is reported in nanoseconds in the above structure.
>>
>>> For example, if you read the stolen time before you read CNTVCT_EL0,
>>> isn't it possible for a lengthy event like a migration to occur between
>>> the two reads, causing the stolen time to be obsolete and off by seconds?
>>
>> "Lengthy events" like migration are represented by the "paused" state in
>> the diagram - i.e. it's the difference between "physical time" and "live
>> physical time". So stolen time doesn't attempt to represent that.
>>
>> And yes, there is a race between reading CNTVCT_EL0 and reading stolen
>> time - but in practice this doesn't really matter. The usual pseudo-code
>> way of using stolen time is:
> 
> I’m assuming this is the guest scheduler you are talking about,

yes

> and I’m assuming virtualization can preempt that code anywhere.
> Maybe that’s where I’m wrong?

You are correct, the guest can be preempted at any point.

> 
> For the sake of the argument, assume there is a 1s pause.
> Not completely unreasonable in a migration scenario.

As I mentioned before, events like migration are not represented by
stolen time. They would be represented by CNTVCT_EL0 appearing to pause
during the migration (so showing a difference between "physical time"
and "live physical time"). The stolen time value would not be incremented.

>>  * scheduler captures stolen time from structure and CNTVCT_EL0:
>>  before_timer = CNTVCT_EL0
> 
> [insert optional 1s pause here, case A]
> 
>>  before_stolen = stolen
>>  * schedule in process
>>  * process is pre-empted (or blocked in some way)
>>  * scheduler captures stolen time from structure and CNTVCT_EL0:
>>  after_timer = CNTVCT_EL0
> 
> [insert optional 1s pause here, case B]
> 
>>  after_stolen = stolen
>>  time = to_nsecs(after_timer - before_timer) -
>> (after_stolen - before_stolen)
> 
> In case A, time is too big by one second. In case B, it is too small,
> to the point where your code might need to be ready for
> “time” unexpectedly showing up as negative.

So a 1 second 

Re: [PATCH] arm64: KVM: hyp: debug-sr: Mark expected switch fall-throughs

[Marc Zyngier]

On 07/08/2019 15:18, Gustavo A. R. Silva wrote:
> Mark switch cases where we are expecting to fall through.
> 
> This patch fixes the following warnings (Building: allmodconfig arm64):
> 
> arch/arm64/kvm/hyp/debug-sr.c:20:19: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:21:19: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:22:19: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:23:19: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:24:19: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:25:19: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:26:18: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:27:18: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:28:18: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:29:18: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:30:18: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:31:18: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:32:18: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:33:18: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> arch/arm64/kvm/hyp/debug-sr.c:34:18: warning: this statement may fall through 
> [-Wimplicit-fallthrough=]
> 
> Signed-off-by: Gustavo A. R. Silva 

Already fixed (together with all the other KVM/arm warnings/bugs), and
pull request sent to Paolo.

Thanks,

M.
-- 
Jazz is not dead, it just smells funny...

Re: [PATCH 1/9] KVM: arm64: Document PV-time interface

[Christophe de Dinechin]

> On 7 Aug 2019, at 15:21, Steven Price  wrote:
> 
> On 05/08/2019 17:40, Christophe de Dinechin wrote:
>> 
>> Steven Price writes:
>> 
>>> Introduce a paravirtualization interface for KVM/arm64 based on the
>>> "Arm Paravirtualized Time for Arm-Base Systems" specification DEN 0057A.
>>> 
>>> This only adds the details about "Stolen Time" as the details of "Live
>>> Physical Time" have not been fully agreed.
>>> 
>> [...]
>> 
>>> +
>>> +Stolen Time
>>> +---
>>> +
>>> +The structure pointed to by the PV_TIME_ST hypercall is as follows:
>>> +
>>> +  Field   | Byte Length | Byte Offset | Description
>>> +  --- | --- | --- | --
>>> +  Revision|  4  |  0  | Must be 0 for version 0.1
>>> +  Attributes  |  4  |  4  | Must be 0
>>> +  Stolen time |  8  |  8  | Stolen time in unsigned
>>> +  | | | nanoseconds indicating how
>>> +  | | | much time this VCPU thread
>>> +  | | | was involuntarily not
>>> +  | | | running on a physical CPU.
>> 
>> I know very little about the topic, but I don't understand how the spec
>> as proposed allows an accurate reading of the relation between physical
>> time and stolen time simultaneously. In other words, could you draw
>> Figure 1 of the spec from within the guest? Or is it a non-objective?
> 
> Figure 1 is mostly attempting to explain Live Physical Time (LPT), which
> is not part of this patch series. But it does touch on stolen time by
> the difference between "live physical time" and "virtual time".
> 
> I'm not sure what you mean by "from within the guest". From the
> perspective of the guest the parts of the diagram where the guest isn't
> running don't exist (therefore there are discontinuities in the
> "physical time" and "live physical time" lines).

I meant: If I run code within the guest that attempts to draw Figure 1,
race conditions may cause the diagram actually drawn by your guest
program to look completely wrong on occasions.

> This patch series doesn't attempt to provide the guest with a view of
> "physical time" (or LPT) - but it might be able to observe that by
> consulting something external (e.g. an NTP server, or an emulated RTC
> which reports wall-clock time).

… with what appear to be like a built-in race condition, as you correctly
identified. I was wondering if the built-in race condition was deliberate
and/or necessary, or if it was irrelevant for the planned uses of the value.

> What it does provide is a mechanism for obtaining the difference (as
> reported by the host) between "live physical time" and "virtual time" -
> this is reported in nanoseconds in the above structure.
> 
>> For example, if you read the stolen time before you read CNTVCT_EL0,
>> isn't it possible for a lengthy event like a migration to occur between
>> the two reads, causing the stolen time to be obsolete and off by seconds?
> 
> "Lengthy events" like migration are represented by the "paused" state in
> the diagram - i.e. it's the difference between "physical time" and "live
> physical time". So stolen time doesn't attempt to represent that.
> 
> And yes, there is a race between reading CNTVCT_EL0 and reading stolen
> time - but in practice this doesn't really matter. The usual pseudo-code
> way of using stolen time is:

I’m assuming this is the guest scheduler you are talking about,
and I’m assuming virtualization can preempt that code anywhere.
Maybe that’s where I’m wrong?

For the sake of the argument, assume there is a 1s pause.
Not completely unreasonable in a migration scenario.

>  * scheduler captures stolen time from structure and CNTVCT_EL0:
>  before_timer = CNTVCT_EL0

[insert optional 1s pause here, case A]

>  before_stolen = stolen
>  * schedule in process
>  * process is pre-empted (or blocked in some way)
>  * scheduler captures stolen time from structure and CNTVCT_EL0:
>  after_timer = CNTVCT_EL0

[insert optional 1s pause here, case B]

>  after_stolen = stolen
>  time = to_nsecs(after_timer - before_timer) -
> (after_stolen - before_stolen)

In case A, time is too big by one second. In case B, it is too small,
to the point where your code might need to be ready for
“time” unexpectedly showing up as negative.

> 
> The scheduler can then charge the process for "time" nanoseconds of
> time. This ensures that a process isn't unfairly penalised if the host
> doesn't schedule the VCPU while it is supposed to be running.
> 
> The race is very small in comparison to the time the process is running,
> and in the worst case just means the process is charged slightly more
> (or less) than it should be.

At this point, what I don’t understand is why the race would be
“very small” or why you would only be charged “slightly” more or less?

> I guess if you're really worried about 

[PATCH] arm64: KVM: hyp: debug-sr: Mark expected switch fall-throughs

[Gustavo A. R. Silva]

Mark switch cases where we are expecting to fall through.

This patch fixes the following warnings (Building: allmodconfig arm64):

arch/arm64/kvm/hyp/debug-sr.c:20:19: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:21:19: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:22:19: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:23:19: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:24:19: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:25:19: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:26:18: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:27:18: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:28:18: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:29:18: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:30:18: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:31:18: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:32:18: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:33:18: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
arch/arm64/kvm/hyp/debug-sr.c:34:18: warning: this statement may fall through 
[-Wimplicit-fallthrough=]

Signed-off-by: Gustavo A. R. Silva 
---
 arch/arm64/kvm/hyp/debug-sr.c | 30 ++
 1 file changed, 30 insertions(+)

diff --git a/arch/arm64/kvm/hyp/debug-sr.c b/arch/arm64/kvm/hyp/debug-sr.c
index 26781da3ad3e..0fc9872a1467 100644
--- a/arch/arm64/kvm/hyp/debug-sr.c
+++ b/arch/arm64/kvm/hyp/debug-sr.c
@@ -18,40 +18,70 @@
 #define save_debug(ptr,reg,nr) \
switch (nr) {   \
case 15:ptr[15] = read_debug(reg, 15);  \
+   /* Fall through */  \
case 14:ptr[14] = read_debug(reg, 14);  \
+   /* Fall through */  \
case 13:ptr[13] = read_debug(reg, 13);  \
+   /* Fall through */  \
case 12:ptr[12] = read_debug(reg, 12);  \
+   /* Fall through */  \
case 11:ptr[11] = read_debug(reg, 11);  \
+   /* Fall through */  \
case 10:ptr[10] = read_debug(reg, 10);  \
+   /* Fall through */  \
case 9: ptr[9] = read_debug(reg, 9);\
+   /* Fall through */  \
case 8: ptr[8] = read_debug(reg, 8);\
+   /* Fall through */  \
case 7: ptr[7] = read_debug(reg, 7);\
+   /* Fall through */  \
case 6: ptr[6] = read_debug(reg, 6);\
+   /* Fall through */  \
case 5: ptr[5] = read_debug(reg, 5);\
+   /* Fall through */  \
case 4: ptr[4] = read_debug(reg, 4);\
+   /* Fall through */  \
case 3: ptr[3] = read_debug(reg, 3);\
+   /* Fall through */  \
case 2: ptr[2] = read_debug(reg, 2);\
+   /* Fall through */  \
case 1: ptr[1] = read_debug(reg, 1);\
+   /* Fall through */  \
default:ptr[0] = read_debug(reg, 0);\
}
 
 #define restore_debug(ptr,reg,nr)  \
switch (nr) {   \
case 15:write_debug(ptr[15], reg, 15);  \
+   /* Fall through */  \
case 14:write_debug(ptr[14], reg, 14);  \
+   /* Fall through */  \
case 13:

Re: [PATCH 6/9] KVM: arm64: Provide a PV_TIME device to user space

[Marc Zyngier]

On 07/08/2019 14:39, Steven Price wrote:
> On 03/08/2019 18:34, Marc Zyngier wrote:
>> On Sat, 3 Aug 2019 13:51:13 +0100
>> Marc Zyngier  wrote:
>>
>> [forgot that one]
>>
>>> On Fri,  2 Aug 2019 15:50:14 +0100
>>> Steven Price  wrote:
>>
>> [...]
>>
 +static int __init kvm_pvtime_init(void)
 +{
 +  kvm_register_device_ops(_ops, KVM_DEV_TYPE_ARM_PV_TIME);
 +
 +  return 0;
 +}
 +
 +late_initcall(kvm_pvtime_init);
>>
>> Why is it an initcall? So far, the only initcall we've used is the one
>> that initializes KVM itself. Can't we just the device_ops just like we
>> do for the vgic?
> 
> So would you prefer a direct call from init_subsystems() in
> virt/kvm/arm/arm.c?

Yes. Consistency is important.

> The benefit of initcall is just that it keeps the code self-contained.
> In init_subsystems() I'd either need a #ifdef CONFIG_ARM64 or a dummy
> function for arm.

Having a dummy function for 32bit ARM is fine. Most of the code we add
to the 32bit port is made of empty stubs anyway.

Thanks,

M.
-- 
Jazz is not dead, it just smells funny...

Re: [PATCH 6/9] KVM: arm64: Provide a PV_TIME device to user space

[Steven Price]

On 03/08/2019 18:34, Marc Zyngier wrote:
> On Sat, 3 Aug 2019 13:51:13 +0100
> Marc Zyngier  wrote:
> 
> [forgot that one]
> 
>> On Fri,  2 Aug 2019 15:50:14 +0100
>> Steven Price  wrote:
> 
> [...]
> 
>>> +static int __init kvm_pvtime_init(void)
>>> +{
>>> +   kvm_register_device_ops(_ops, KVM_DEV_TYPE_ARM_PV_TIME);
>>> +
>>> +   return 0;
>>> +}
>>> +
>>> +late_initcall(kvm_pvtime_init);
> 
> Why is it an initcall? So far, the only initcall we've used is the one
> that initializes KVM itself. Can't we just the device_ops just like we
> do for the vgic?

So would you prefer a direct call from init_subsystems() in
virt/kvm/arm/arm.c?

The benefit of initcall is just that it keeps the code self-contained.
In init_subsystems() I'd either need a #ifdef CONFIG_ARM64 or a dummy
function for arm.

Steve

Re: [PATCH 1/9] KVM: arm64: Document PV-time interface

[Steven Price]

On 05/08/2019 17:40, Christophe de Dinechin wrote:
> 
> Steven Price writes:
> 
>> Introduce a paravirtualization interface for KVM/arm64 based on the
>> "Arm Paravirtualized Time for Arm-Base Systems" specification DEN 0057A.
>>
>> This only adds the details about "Stolen Time" as the details of "Live
>> Physical Time" have not been fully agreed.
>>
> [...]
> 
>> +
>> +Stolen Time
>> +---
>> +
>> +The structure pointed to by the PV_TIME_ST hypercall is as follows:
>> +
>> +  Field   | Byte Length | Byte Offset | Description
>> +  --- | --- | --- | --
>> +  Revision|  4  |  0  | Must be 0 for version 0.1
>> +  Attributes  |  4  |  4  | Must be 0
>> +  Stolen time |  8  |  8  | Stolen time in unsigned
>> +  | | | nanoseconds indicating how
>> +  | | | much time this VCPU thread
>> +  | | | was involuntarily not
>> +  | | | running on a physical CPU.
> 
> I know very little about the topic, but I don't understand how the spec
> as proposed allows an accurate reading of the relation between physical
> time and stolen time simultaneously. In other words, could you draw
> Figure 1 of the spec from within the guest? Or is it a non-objective?

Figure 1 is mostly attempting to explain Live Physical Time (LPT), which
is not part of this patch series. But it does touch on stolen time by
the difference between "live physical time" and "virtual time".

I'm not sure what you mean by "from within the guest". From the
perspective of the guest the parts of the diagram where the guest isn't
running don't exist (therefore there are discontinuities in the
"physical time" and "live physical time" lines).

This patch series doesn't attempt to provide the guest with a view of
"physical time" (or LPT) - but it might be able to observe that by
consulting something external (e.g. an NTP server, or an emulated RTC
which reports wall-clock time).

What it does provide is a mechanism for obtaining the difference (as
reported by the host) between "live physical time" and "virtual time" -
this is reported in nanoseconds in the above structure.

> For example, if you read the stolen time before you read CNTVCT_EL0,
> isn't it possible for a lengthy event like a migration to occur between
> the two reads, causing the stolen time to be obsolete and off by seconds?

"Lengthy events" like migration are represented by the "paused" state in
the diagram - i.e. it's the difference between "physical time" and "live
physical time". So stolen time doesn't attempt to represent that.

And yes, there is a race between reading CNTVCT_EL0 and reading stolen
time - but in practice this doesn't really matter. The usual pseudo-code
way of using stolen time is:

  * scheduler captures stolen time from structure and CNTVCT_EL0:
  before_timer = CNTVCT_EL0
  before_stolen = stolen
  * schedule in process
  * process is pre-empted (or blocked in some way)
  * scheduler captures stolen time from structure and CNTVCT_EL0:
  after_timer = CNTVCT_EL0
  after_stolen = stolen
  time = to_nsecs(after_timer - before_timer) -
 (after_stolen - before_stolen)

The scheduler can then charge the process for "time" nanoseconds of
time. This ensures that a process isn't unfairly penalised if the host
doesn't schedule the VCPU while it is supposed to be running.

The race is very small in comparison to the time the process is running,
and in the worst case just means the process is charged slightly more
(or less) than it should be.

I guess if you're really worried about it, you could do a dance like:

do {
before = stolen
timer = CNTVCT_EL0
after = stolen
} while (before != after);

But I don't see the need to have such an accurate view of elapsed time
that the VCPU was scheduled. And of course at the moment (without this
series) the guest has no idea about time stolen by the host.

Steve
___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

[PATCH] KVM: arm/arm64: vgic: Reevaluate level sensitive interrupts on enable

[Alexandru Elisei]

A HW mapped level sensitive interrupt asserted by a device will not be put
into the ap_list if it is disabled at the VGIC level. When it is enabled
again, it will be inserted into the ap_list and written to a list register
on guest entry regardless of the state of the device.

We could argue that this can also happen on real hardware, when the command
to enable the interrupt reached the GIC before the device had the chance to
de-assert the interrupt signal; however, we emulate the distributor and
redistributors in software and we can do better than that.

Signed-off-by: Alexandru Elisei 
---
 virt/kvm/arm/vgic/vgic-mmio.c | 16 
 1 file changed, 16 insertions(+)

diff --git a/virt/kvm/arm/vgic/vgic-mmio.c b/virt/kvm/arm/vgic/vgic-mmio.c
index 3ba7278fb533..44efc2ff863f 100644
--- a/virt/kvm/arm/vgic/vgic-mmio.c
+++ b/virt/kvm/arm/vgic/vgic-mmio.c
@@ -113,6 +113,22 @@ void vgic_mmio_write_senable(struct kvm_vcpu *vcpu,
struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
 
raw_spin_lock_irqsave(>irq_lock, flags);
+   if (vgic_irq_is_mapped_level(irq)) {
+   bool was_high = irq->line_level;
+
+   /*
+* We need to update the state of the interrupt because
+* the guest might have changed the state of the device
+* while the interrupt was disabled at the VGIC level.
+*/
+   irq->line_level = vgic_get_phys_line_level(irq);
+   /*
+* Deactivate the physical interrupt so the GIC will let
+* us know when it is asserted again.
+*/
+   if (!irq->active && was_high && !irq->line_level)
+   vgic_irq_set_phys_active(irq, false);
+   }
irq->enabled = true;
vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
 
-- 
2.7.4

___
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm