Re: [Xen-devel] unable start xen in hikey

2016-08-18 Thread Chenxiao Zhao 
It seems that you are using a dtb file that is not compatible with XEN
hypervisor on hikey. I strongly suggest you compile the kernel from source
in 96boards's repository. When you compile the kernel it will compile the
dtb from dts as well.

On Thu, Aug 18, 2016 at 6:30 AM Kamenee Arumugam <kame...@seas.upenn.edu>
wrote:

> Hi Chen,
>
> My previous issue was resolved when i used hi6220-hikey.dtb from this
> source: https://github.com/kuscsik/device-linaro-hikey-kernel. When I
> download linux kernel, there doesn't seems to contain hi6220-hikey.dtb.
>
> But now it got stuck while loading Dom0 and below are log traces:
>
>
> (XEN) *** LOADING DOMAIN 0 ***
> (XEN) Loading kernel from boot module @ 7a037000
> (XEN) Loading ramdisk from boot module @ 0ae0
> (XEN) Allocating 1:1 mappings totalling 512MB for dom0:
> (XEN) BANK[0] 0x004000-0x006000 (512MB)
> (XEN) Grant table range: 0x0005c0-0x0005c54000
> (XEN) Loading zImage from 7a037000 to
> 4008-40ce8c00
> (XEN) Loading dom0 initrd from 0ae0 to
> 0x4820-0x48a0
> (XEN) Allocating PPI 16 for event channel interrupt
> (XEN) Loading dom0 DTB to 0x4800-0x4800af11
> (XEN) Scrubbing Free RAM on 1 nodes using 8 CPUs
> (XEN) ..done.
> (XEN) Initial low memory virq threshold set at 0x4000 pages.
> (XEN) Std. Loglevel: Errors and warnings
> (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
> (XEN) *** Serial input -> DOM0 (type 'CTRL-x' three times to switch input
> to Xen)
> (XEN) Freed 284kB init memory.
> (XEN) d0v0: vGICD: unhandled word write 0x to ICACTIVER4
> (XEN) d0v0: vGICD: unhandled word write 0x to ICACTIVER8
> (XEN) d0v0: vGICD: unhandled word write 0x to ICACTIVER12
> (XEN) d0v0: vGICD: unhandled word write 0x to ICACTIVER16
> (XEN) d0v0: vGICD: unhandled word write 0x to ICACTIVER0
> (XEN) d0v1: vGICD: unhandled word write 0x to ICACTIVER0
> (XEN) d0v2: vGICD: unhandled word write 0x to ICACTIVER0
> (XEN) d0v3: vGICD: unhandled word write 0x to ICACTIVER0
>
> It just hang there after the last line above.
> I have looked into those messages " d0v3: vGICD: unhandled word write
> 0x to ICACTIVER0"  and linux kernel already taking care to ignore
> it. Therefore, I believe this is not contributing to there hang here. Any
> idea on this issue?
>
> Thanks,
> Kamenee
>
>
> On Wed, Aug 10, 2016 at 9:11 PM, Chenxiao Zhao <chenxiao.z...@gmail.com>
> wrote:
>
>> You should compile kernel first. Please follow the "Getting Started" page
>> on 96boards
>>
>> On 8/10/2016 7:07 AM, Kamenee Arumugam wrote:
>>
>>> Hi Chen,
>>>
>>> I have tried to flash UEFI in my board with latest one but still facing
>>> the same issue. Regarding to use dtb file from kernel source, I couldnt
>>> find hi6220-hikey.dtb in my source kernel. i am downloading my kernel
>>> source from
>>> here: https://github.com/96boards/linux/tree/android-hikey-linaro-4.1
>>>
>>> I only could find hi6220-hikey.dts file in the kernel source. Therefore
>>> download from other github but which is correct one that i need to use?
>>> In case, I am using incorrect kernel source, pls point to me to correct
>>> one if you aware of?
>>>
>>> Thanks,
>>> Kamenee
>>>
>>> On Thu, Aug 4, 2016 at 11:06 PM, Chenxiao Zhao <chenxiao.z...@gmail.com
>>> <mailto:chenxiao.z...@gmail.com>> wrote:
>>>
>>> The xen.cfg file seems fine to me. I suggest you try to use the dtb
>>> file compiled from the kernel source and update UEFI bootloader on
>>> hikey board to the latest version.
>>>
>>> hope this could help.
>>>
>>> On 8/4/2016 11:02 PM, Kamenee Arumugam wrote:
>>>
>>> I have already pass hi6220-key.dtb in my xen.cfg:
>>>
>>> options=console=dtuart dom0_mem=512M dom0_max_vcpus=4 conswitch=x
>>> dtuart=/smb/uart@f7113000
>>> kernel=Image console=hvc0 root=/dev/mmcblk1p4 rootwait rw
>>> dtb=hi6220-hikey.dtb
>>>
>>> I download the hi6220-hikey.dtb from this site:
>>>
>>> https://android.googlesource.com/device/linaro/hikey-kernel/+/f117fa12746b59aa0a1a4d6335145e58935c422b/hi6220-hikey.dtb
>>> <
>>> https://android.googlesource.com/device/linaro/hikey-kernel/+/f117fa12746b59aa0a1a4d6335145e58935c422b/hi6220-hikey.dtb
>>> >
>>>

Re: [Xen-devel] unable start xen in hikey

2016-08-10 Thread Chenxiao Zhao 
You should compile kernel first. Please follow the "Getting Started" 
page on 96boards


On 8/10/2016 7:07 AM, Kamenee Arumugam wrote:

Hi Chen,

I have tried to flash UEFI in my board with latest one but still facing
the same issue. Regarding to use dtb file from kernel source, I couldnt
find hi6220-hikey.dtb in my source kernel. i am downloading my kernel
source from
here: https://github.com/96boards/linux/tree/android-hikey-linaro-4.1

I only could find hi6220-hikey.dts file in the kernel source. Therefore
download from other github but which is correct one that i need to use?
In case, I am using incorrect kernel source, pls point to me to correct
one if you aware of?

Thanks,
Kamenee

On Thu, Aug 4, 2016 at 11:06 PM, Chenxiao Zhao <chenxiao.z...@gmail.com
<mailto:chenxiao.z...@gmail.com>> wrote:

The xen.cfg file seems fine to me. I suggest you try to use the dtb
file compiled from the kernel source and update UEFI bootloader on
hikey board to the latest version.

hope this could help.

On 8/4/2016 11:02 PM, Kamenee Arumugam wrote:

I have already pass hi6220-key.dtb in my xen.cfg:

options=console=dtuart dom0_mem=512M dom0_max_vcpus=4 conswitch=x
dtuart=/smb/uart@f7113000
kernel=Image console=hvc0 root=/dev/mmcblk1p4 rootwait rw
dtb=hi6220-hikey.dtb

I download the hi6220-hikey.dtb from this site:

https://android.googlesource.com/device/linaro/hikey-kernel/+/f117fa12746b59aa0a1a4d6335145e58935c422b/hi6220-hikey.dtb

<https://android.googlesource.com/device/linaro/hikey-kernel/+/f117fa12746b59aa0a1a4d6335145e58935c422b/hi6220-hikey.dtb>


Firstly, I build kernel using the command below:

make -j24 Image hi6220-hikey.dtb

I placed hikey6220-hikey.dtb in linux directory level. I also
copied the
dtb into sd before booting up. Please correct me if my steps are
wrong.


On Thu, Aug 4, 2016 at 3:27 AM, Chenxiao Zhao
<chenxiao.z...@gmail.com <mailto:chenxiao.z...@gmail.com>
<mailto:chenxiao.z...@gmail.com
<mailto:chenxiao.z...@gmail.com>>> wrote:

you have to pass hi6220-key.dtb to xen.

On Thu, Aug 4, 2016 at 2:15 AM Kamenee Arumugam
<kame...@seas.upenn.edu <mailto:kame...@seas.upenn.edu>
<mailto:kame...@seas.upenn.edu <mailto:kame...@seas.upenn.edu>>>
wrote:

Hi all,

I am unable to start xen in hikey successfully but
getting to
this prompt message:


>/Xen 4.7.0 (c/s Mon Jun 20 11:38:15 2016 +0100
git:9a6cc4f) EFI
loader/
>/Using configuration file 'xen.cfg'/
>/Image: 0x79fd-0x7acb8c00
/
/> Unable to create new FDT
/
/Shell

/
I am following this wiki (http://wiki.xen.org/wiki/HiKey
<http://wiki.xen.org/wiki/HiKey>) to
setup xen in hikey. I am not sure what could cause this
issue
here. Appreciate all of you input and advice here.

Thanks,
Kamenee//
/
/
___
Xen-devel mailing list
Xen-devel@lists.xen.org <mailto:Xen-devel@lists.xen.org>
<mailto:Xen-devel@lists.xen.org <mailto:Xen-devel@lists.xen.org>>
https://lists.xen.org/xen-devel
<https://lists.xen.org/xen-devel>





___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] unable start xen in hikey

2016-08-04 Thread Chenxiao Zhao 
The xen.cfg file seems fine to me. I suggest you try to use the dtb file 
compiled from the kernel source and update UEFI bootloader on hikey 
board to the latest version.


hope this could help.

On 8/4/2016 11:02 PM, Kamenee Arumugam wrote:

I have already pass hi6220-key.dtb in my xen.cfg:

options=console=dtuart dom0_mem=512M dom0_max_vcpus=4 conswitch=x
dtuart=/smb/uart@f7113000
kernel=Image console=hvc0 root=/dev/mmcblk1p4 rootwait rw
dtb=hi6220-hikey.dtb

I download the hi6220-hikey.dtb from this site:
https://android.googlesource.com/device/linaro/hikey-kernel/+/f117fa12746b59aa0a1a4d6335145e58935c422b/hi6220-hikey.dtb


Firstly, I build kernel using the command below:

make -j24 Image hi6220-hikey.dtb

I placed hikey6220-hikey.dtb in linux directory level. I also copied the
dtb into sd before booting up. Please correct me if my steps are wrong.


On Thu, Aug 4, 2016 at 3:27 AM, Chenxiao Zhao <chenxiao.z...@gmail.com
<mailto:chenxiao.z...@gmail.com>> wrote:

you have to pass hi6220-key.dtb to xen.

On Thu, Aug 4, 2016 at 2:15 AM Kamenee Arumugam
<kame...@seas.upenn.edu <mailto:kame...@seas.upenn.edu>> wrote:

Hi all,

I am unable to start xen in hikey successfully but getting to
this prompt message:


>/Xen 4.7.0 (c/s Mon Jun 20 11:38:15 2016 +0100 git:9a6cc4f) EFI
loader/
>/Using configuration file 'xen.cfg'/
>/Image: 0x79fd-0x7acb8c00
/
/> Unable to create new FDT
/
/Shell

/
I am following this wiki (http://wiki.xen.org/wiki/HiKey) to
setup xen in hikey. I am not sure what could cause this issue
here. Appreciate all of you input and advice here.

Thanks,
Kamenee//
/
/
___
Xen-devel mailing list
Xen-devel@lists.xen.org <mailto:Xen-devel@lists.xen.org>
https://lists.xen.org/xen-devel




___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH RFC 03/04] tools/libxc: arm: Implement save ops arm_setup

2016-08-04 Thread Chenxiao Zhao 
arm_setup need to return number of pages that vm is allocated. Code is 
copied from x64 save ops.


Signed-off-by: Chenxiao Zhao <chenxiao.z...@gmail.com>

diff --git a/tools/libxc/xc_sr_save_arm.c b/tools/libxc/xc_sr_save_arm.c
index 611f99a..a2ef2db 100644
--- a/tools/libxc/xc_sr_save_arm.c
+++ b/tools/libxc/xc_sr_save_arm.c
@@ -122,7 +122,23 @@ static int arm_normalise_page(struct xc_sr_context 
*ctx,


 static int arm_setup(struct xc_sr_context *ctx)
 {
-/* no-op */
+xc_interface *xch = ctx->xch;
+xen_pfn_t nr_pfns;
+
+if (xc_domain_nr_gpfns(xch, ctx->domid, _pfns) < 0 )
+{
+PERROR("Unable to obtain the guest p2m size");
+return -1;
+}
+if ( nr_pfns > ~XEN_DOMCTL_PFINFO_LTAB_MASK )
+{
+errno = E2BIG;
+PERROR("Cannot save this big a guest");
+return -1;
+}
+
+ctx->save.p2m_size = nr_pfns;
+
 return 0;
 }

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH RFC 04/04] xen: arm64: save/restore domain type

2016-08-04 Thread Chenxiao Zhao 
On arm64 VM uses d->arch.type to indicate if the domain is 32-bit or 
64-bit. This value is set while parsing kernel image. while restoring 
VCPU state, it will use this value to decide the VCPU type. I can not 
find a proper place to save/restore this value in state file, this is 
surely not a good implementation.


Signed-off-by: Chenxiao Zhao <chenxiao.z...@gmail.com>

diff --git a/xen/arch/arm/hvm.c b/xen/arch/arm/hvm.c
index 3073b17..4702f60 100644
--- a/xen/arch/arm/hvm.c
+++ b/xen/arch/arm/hvm.c
@@ -108,6 +108,7 @@ static int cpu_save(struct domain *d, 
hvm_domain_context_t *h)

 continue;

 memset(, 0, sizeof(ctxt));
+   ctxt.is_64bit = is_64bit_domain(d);
 ctxt.sctlr = v->arch.sctlr;
 ctxt.ttbr0 = v->arch.ttbr0;
 ctxt.ttbr1 = v->arch.ttbr1;
@@ -188,6 +189,9 @@ static int cpu_load(struct domain *d, 
hvm_domain_context_t *h)

 if ( hvm_load_entry(VCPU, h, ) != 0 )
 return -EINVAL;

+if(ctxt.is_64bit)
+   v->domain->arch.type = DOMAIN_64BIT;
+
 v->arch.sctlr = ctxt.sctlr;
 v->arch.ttbr0 = ctxt.ttbr0;
 v->arch.ttbr1 = ctxt.ttbr1;
diff --git a/xen/include/public/arch-arm/hvm/save.h 
b/xen/include/public/arch-arm/hvm/save.h

index 80330dc..68b6136 100644
--- a/xen/include/public/arch-arm/hvm/save.h
+++ b/xen/include/public/arch-arm/hvm/save.h
@@ -53,6 +53,10 @@ struct hvm_hw_cpu
 uint64_t vfp[66];
 #endif

+#ifdef CONFIG_ARM_64
+uint32_t is_64bit;
+#endif
+
 /* Guest core registers */
 struct vcpu_guest_core_regs core_regs;

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH RFC 02/04] tools/libxc: arm: Add missing save ops

2016-08-04 Thread Chenxiao Zhao 
Add arm_check_vm_state function, .check_vm_state can not be NULL in arch 
specific ops.


Signed-off-by: Chenxiao Zhao <chenxiao.z...@gmail.com>

diff --git a/tools/libxc/xc_sr_save_arm.c b/tools/libxc/xc_sr_save_arm.c
index 1442679..611f99a 100644
--- a/tools/libxc/xc_sr_save_arm.c
+++ b/tools/libxc/xc_sr_save_arm.c
@@ -161,6 +161,12 @@ static int arm_cleanup(struct xc_sr_context *ctx)
 return 0;
 }

+static int arm_check_vm_state(struct xc_sr_context *ctx)
+{
+   /* no-op */
+   return 0;
+}
+
 struct xc_sr_save_ops save_ops_arm =
 {
 .pfn_to_gfn  = arm_pfn_to_gfn,
@@ -169,6 +175,7 @@ struct xc_sr_save_ops save_ops_arm =
 .start_of_stream = arm_start_of_stream,
 .start_of_checkpoint = arm_start_of_checkpoint,
 .end_of_checkpoint   = arm_end_of_checkpoint,
+.check_vm_state  = arm_check_vm_state,
 .cleanup = arm_cleanup,
 };

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH RFC 00/04] xen: arm64: Support VM save/restore on arm64

2016-08-04 Thread Chenxiao Zhao 

Hi all,

This patch added support VM save/restore for arm64. It is based on 
previous work done by Ian Campbell [1] with some bug fixes to make it 
work on stable-4.7. You should apply Ian's patch first.


There still some known issues that have not fixed yet.

* GIC v2 not support
* No live migration
* does not support VM with multiple CPU cores


[1] 
https://lists.xenproject.org/archives/html/xen-devel/2015-12/msg01053.html


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH RFC 01/04] xen: arm64: save/restore VCPU state for arm64

2016-08-04 Thread Chenxiao Zhao 

added support to save/restore arm64 registers

Signed-off-by: Chenxiao Zhao <chenxiao.z...@gmail.com>

diff --git a/xen/arch/arm/hvm.c b/xen/arch/arm/hvm.c
index aee3353..3073b17 100644
--- a/xen/arch/arm/hvm.c
+++ b/xen/arch/arm/hvm.c
@@ -120,7 +120,8 @@ static int cpu_save(struct domain *d, 
hvm_domain_context_t *h)

 ctxt.dfar = v->arch.dfar;
 ctxt.dfsr = v->arch.dfsr;
 #else
-/* XXX 64-bit */
+   ctxt.far = v->arch.far;
+   ctxt.esr = v->arch.esr;
 #endif

 #ifdef CONFIG_ARM_32
@@ -199,7 +200,8 @@ static int cpu_load(struct domain *d, 
hvm_domain_context_t *h)

 v->arch.dfar = ctxt.dfar;
 v->arch.dfsr = ctxt.dfsr;
 #else
-/* XXX 64-bit */
+v->arch.far = ctxt.far;
+v->arch.esr = ctxt.esr;
 #endif

 #ifdef CONFIG_ARM_32
diff --git a/xen/include/public/arch-arm/hvm/save.h 
b/xen/include/public/arch-arm/hvm/save.h

index db916b1..80330dc 100644
--- a/xen/include/public/arch-arm/hvm/save.h
+++ b/xen/include/public/arch-arm/hvm/save.h
@@ -46,16 +46,26 @@ DECLARE_HVM_SAVE_TYPE(HEADER, 1, struct 
hvm_save_header);


 struct hvm_hw_cpu
 {
+#ifdef CONFIG_ARM_32
 uint64_t vfp[34]; /* Vector floating pointer */
 /* VFP v3 state is 34x64 bit, VFP v4 is not yet supported */
+#else
+uint64_t vfp[66];
+#endif

 /* Guest core registers */
 struct vcpu_guest_core_regs core_regs;

-uint32_t sctlr, ttbcr;
+uint32_t sctlr;
+#ifdef CONFIG_ARM_32
+uint32_t ttbcr;
+#else
+uint64_t ttbcr;
+#endif
 uint64_t ttbr0, ttbr1;

 uint32_t ifar, dfar;
+uint64_t far, esr;
 uint32_t ifsr, dfsr;
 uint32_t dacr;
 uint64_t par;

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] unable start xen in hikey

2016-08-04 Thread Chenxiao Zhao 
you have to pass hi6220-key.dtb to xen.

On Thu, Aug 4, 2016 at 2:15 AM Kamenee Arumugam 
wrote:

> Hi all,
>
> I am unable to start xen in hikey successfully but getting to this prompt
> message:
>
>
> >* Xen 4.7.0 (c/s Mon Jun 20 11:38:15 2016 +0100 git:9a6cc4f) EFI loader*
> >* Using configuration file 'xen.cfg'*
> >
> * Image: 0x79fd-0x7acb8c00*
>
> *> Unable to create new FDT*
>
>
> *Shell*
> I am following this wiki (http://wiki.xen.org/wiki/HiKey) to setup xen in
> hikey. I am not sure what could cause this issue here. Appreciate all of
> you input and advice here.
>
> Thanks,
> Kamenee
>
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
>
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] questions of vm save/restore on arm64

2016-06-12 Thread Chenxiao Zhao 



On 6/12/2016 11:31 PM, Julien Grall wrote:

On 12/06/2016 10:46, Chenxiao Zhao wrote:

Hi all,


Hello,


I finally got save/restore working on arm64, but it only works when I
assign only one vCPU to VM. If I set vcpus=4 in configure file, the
restored VM does not work properly.


Can you describe what you mean by "does not work properly"? What are the
symptoms?


After restoring VM with more than one vCPU, the VM keeps in "b" state.

I'm running Centos on guest and listed the console log after restore.

[   32.530490] Xen: initializing cpu0
[   32.530490] xen:grant_table: Grant tables using version 1 layout
[   32.531034] PM: noirq restore of devices complete after 0.382 msecs
[   32.531382] PM: early restore of devices complete after 0.300 msecs
[   32.531430] Xen: initializing cpu1
[   32.569028] PM: restore of devices complete after 24.663 msecs
[   32.569304] Restarting tasks ...
[   32.569903] systemd-journal[800]: undefined instruction: 
pc=007fa37dd4c8

[   32.569975] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   32.571530] done.
[   32.571631] systemd[1]: undefined instruction: pc=007f8a9ea4c8
[   32.571650] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   32.573527] auditd[1365]: undefined instruction: pc=007f8aca24c8
[   32.573553] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   32.636573] systemd-cgroups[2210]: undefined instruction: 
pc=007f99ad14c8

[   32.636633] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   32.636726] audit: *NO* daemon at audit_pid=1365
[   32.636741] audit: audit_lost=1 audit_rate_limit=0 
audit_backlog_limit=320

[   32.636755] audit: auditd disappeared
[   32.638545] systemd-logind[1387]: undefined instruction: 
pc=007f86e5b4c8

[   32.638594] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   32.638673] audit: type=1701 audit(68.167:214): auid=4294967295 uid=0 
gid=0 s
es=4294967295 subj=system_u:system_r:systemd_logind_t:s0 pid=1387 
comm="systemd-

logind" exe="/usr/lib/systemd/systemd-logind" sig=4
[   32.647972] systemd-cgroups[2211]: undefined instruction: 
pc=007fa7f414c8

[   32.648017] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   32.648087] audit: type=1701 audit(68.177:215): auid=4294967295 uid=0 
gid=0 s
es=4294967295 subj=system_u:system_r:init_t:s0 pid=2211 
comm="systemd-cgroups" e

xe="/usr/lib/systemd/systemd-cgroups-agent" sig=4
[   61.401838] do_undefinstr: 8 callbacks suppressed
[   61.401882] crond[1550]: undefined instruction: pc=007f8d15d4c8
[   61.401903] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   61.402077] audit: type=1701 audit(96.947:218): auid=4294967295 uid=0 
gid=0 s
es=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 pid=1550 
comm="crond

" exe="/usr/sbin/crond" sig=4
[   61.407024] dbus-daemon[1390]: undefined instruction: pc=007f87fae4c8
[   61.407064] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   61.407212] audit: type=1701 audit(96.947:219): auid=4294967295 
uid=81 gid=81
 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 
pid=1390 co

mm="dbus-daemon" exe="/usr/bin/dbus-daemon" sig=4
[   61.408311] systemd-cgroups-agent[2214]: Failed to process message 
[type=erro
r sender=org.freedesktop.DBus path=n/a interface=n/a member=n/a 
signature=s]: Co

nnection timed out
[   61.416815] systemd-cgroups-agent[2216]: Failed to get D-Bus 
connection: Conn

ection refused
[   61.421499] systemd-cgroups-agent[2215]: Failed to get D-Bus 
connection: Conn

ection refused
[   61.429413] systemd-cgroups-agent[2217]: Failed to get D-Bus 
connection: Conn

ection refused
[   61.434301] systemd-cgroups-agent[2218]: Failed to get D-Bus 
connection: Conn

ection refused
[   61.435016] systemd-cgroups-agent[2219]: Failed to get D-Bus 
connection: Conn

ection refused

[  110.095570] audit: type=1701 audit(145.637:220): auid=0 uid=0 gid=0 
ses=1 sub
j=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=2189 
comm="bash" exe

="/usr/bin/bash" sig=4
[  110.098120] audit: type=1104 audit(145.637:221): pid=1602 uid=0 
auid=0 ses=1
subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:setcred 
grantors
=pam_securetty,pam_unix acct="root" exe="/usr/bin/login" hostname=? 
addr=? termi

nal=hvc0 res=success'
[  110.102730] audit: type=1106 audit(145.637:222): pid=1602 uid=0 
auid=0 ses=1
subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 
msg='op=PAM:session_close gr
antors=? acct="root" exe="/usr/bin/login" hostname=? addr=? 
terminal=hvc0 res=fa

iled'
[  110.112341] systemd-cgroups-agent[2220]: Failed to get D-Bus 
connection: Conn

ection refused



Also, I would start by debugging with 2 vCPUs and then increasing the
number step by step.


It's the same issue when restoring VM with more than one vCPUS. What I 
see i

Re: [Xen-devel] questions of vm save/restore on arm64

2016-06-12 Thread Chenxiao Zhao 



On 6/7/2016 9:17 AM, Chenxiao Zhao wrote:



On 6/6/2016 7:58 PM, Stefano Stabellini wrote:

On Fri, 3 Jun 2016, Chenxiao Zhao wrote:

On 6/3/2016 4:02 AM, Julien Grall wrote:

Hello,

First thing, the time in the mail headers seems to be wrong. Maybe
because of a wrong timezone?

I got: 04/06/16 02:32 however we are still the 3rd in my timezone.

On 04/06/16 02:32, Chenxiao Zhao wrote:



On 6/3/2016 3:16 AM, Julien Grall wrote:

Hello,

On 03/06/16 18:05, Chenxiao Zhao wrote:

I finally found out that the problem is that the toolstack did
not get
corret p2m_size while sending all pages on save(always be zero).
After I
fixed that, the guest could be restored but guest kernel caught
handle_mm_fault().

where do you think I'm going to investigate, guest kernel
hibernation
restore or xen?


The hibernation support for ARM64 has only been merged recently in
the
kernel. Which kernel are you using?


Hi Julien,

I'm using a linaro ported Linux kernel 4.1 for hikey from this link.

https://github.com/96boards/linux/tree/android-hikey-linaro-4.1

I also applied following patches to make the kernel support
hibernation.


This looks the wrong way to do it as this series may requires some
patches which have been upstreamed before hand.

Linux upstream seems support to the hikey board [1]. Any reason to not
using it?


I tried a newer version of kernel 4.4, but got no luck to start dom0
with xen.
so I decide to stay in 4.1 for now.




[1] http://www.spinics.net/lists/arm-kernel/msg477769.html
[2] http://lists.xen.org/archives/html/xen-devel/2015-12/msg01068.html



Also, what are the modifications you have made to support Xen
suspend/resume for ARM64?


I believe I have posted my modifications on xen in the first mail of
this thread.


I mean in Linux. The patch from Ian Campbell does not have any kind of
support for ARM64.

For instance arch/arm/xen/suspend.c needs to be built for ARM64. So
I am
wondering if your kernel has support of hibernation...


Oh, yes, I most forgot I added this file in arch/arm64/xen/Makefile
to let it
build for arm64.




 From my understanding, a kernel hibernation will cause kernel to save
memories to disk(swap partition). But on guest save progress, the
hibernation for domU does not make the guest save memories to disk.
it's
more like suspend all processes in guest, and memors actually
depends on
xen toolstack to save the pages to file. Am I correct?


You are using an older tree with a patch series based on a newer tree.

So I would recommend you to move to a newer tree. If it is not
possible,
please test that hibernation works on baremetal.


I think the suspend/resume in guest is working, cause I can use
pause/unpause
command in toolstack to suspend/resume guest without problem. I can
also see
the suspend/resume kernel messages from guest's console. The only
problem is
it's can not resume from restore.


But can you still connect to the guest after resume, maybe over the
network?
If you cannot, then something is likely wrong.


Hi Stefano,

I can connect to the guest after resume from xen console. It responds by
'return' key, but I can not run any other commands, e.g. ls or ps. I
think the guest is not 'fully' restored.





One thing that confused me is that the kernel's hibernation means the
guest
kernel will save the memory state to disk and power off VM at last.
The guest
will also take care of the memory restore itself. But I do not see the
save/restore on xen works that way. So my question is why it requires
hibernation (aka. suspend to disk) instead of the real suspend (aka.
suspend
to RAM and standby)?


Xen suspend/resume has nothing to do with guest suspend to RAM or guest
hibernation.

Xen suspend/resume is a way for the hypervisor to save to file the
entire state of the VM, including RAM and the state of any devices.
Guest suspend to RAM and guest hibernation are two guest driven
technologies to save the state of the operating system to RAM or to
disk. The only link between Xen suspend and guest suspend is that when
Xen issues a domain suspend, it notifies the guest of it so that it can
ease the process.  The code in Linux to support Xen suspend/resume is:

drivers/xen/manage.c:do_suspend

and makes use of some of the Linux internal hooks provided for
hibernations (see CONFIG_HIBERNATE_CALLBACKS). But that's just for
better integration with the rest of Linux: hibernation is NOT what is
happening.

I hope that this clarifies things a bit, I realize that it is confusing.



Thanks for your explanation, It clear enough and just as my
understanding from the code. I think the problem might caused by
incompatible of arm p2m and xen save/restore mechanism. I'll try a
core-dump and compare the memory after save and restore. I suppose this
two dumps should be identical but there already pages are different.
I'll let you know if I got some progress.

Regards.


Hi all,

I finally got save/restore working on arm64, but it only works when I 
assign only one vCPU to VM. If I set

Re: [Xen-devel] questions of vm save/restore on arm64

2016-06-06 Thread Chenxiao Zhao 



On 6/6/2016 7:58 PM, Stefano Stabellini wrote:

On Fri, 3 Jun 2016, Chenxiao Zhao wrote:

On 6/3/2016 4:02 AM, Julien Grall wrote:

Hello,

First thing, the time in the mail headers seems to be wrong. Maybe
because of a wrong timezone?

I got: 04/06/16 02:32 however we are still the 3rd in my timezone.

On 04/06/16 02:32, Chenxiao Zhao wrote:



On 6/3/2016 3:16 AM, Julien Grall wrote:

Hello,

On 03/06/16 18:05, Chenxiao Zhao wrote:

I finally found out that the problem is that the toolstack did not get
corret p2m_size while sending all pages on save(always be zero).
After I
fixed that, the guest could be restored but guest kernel caught
handle_mm_fault().

where do you think I'm going to investigate, guest kernel hibernation
restore or xen?


The hibernation support for ARM64 has only been merged recently in the
kernel. Which kernel are you using?


Hi Julien,

I'm using a linaro ported Linux kernel 4.1 for hikey from this link.

https://github.com/96boards/linux/tree/android-hikey-linaro-4.1

I also applied following patches to make the kernel support hibernation.


This looks the wrong way to do it as this series may requires some
patches which have been upstreamed before hand.

Linux upstream seems support to the hikey board [1]. Any reason to not
using it?


I tried a newer version of kernel 4.4, but got no luck to start dom0 with xen.
so I decide to stay in 4.1 for now.




[1] http://www.spinics.net/lists/arm-kernel/msg477769.html
[2] http://lists.xen.org/archives/html/xen-devel/2015-12/msg01068.html



Also, what are the modifications you have made to support Xen
suspend/resume for ARM64?


I believe I have posted my modifications on xen in the first mail of
this thread.


I mean in Linux. The patch from Ian Campbell does not have any kind of
support for ARM64.

For instance arch/arm/xen/suspend.c needs to be built for ARM64. So I am
wondering if your kernel has support of hibernation...


Oh, yes, I most forgot I added this file in arch/arm64/xen/Makefile to let it
build for arm64.




 From my understanding, a kernel hibernation will cause kernel to save
memories to disk(swap partition). But on guest save progress, the
hibernation for domU does not make the guest save memories to disk. it's
more like suspend all processes in guest, and memors actually depends on
xen toolstack to save the pages to file. Am I correct?


You are using an older tree with a patch series based on a newer tree.

So I would recommend you to move to a newer tree. If it is not possible,
please test that hibernation works on baremetal.


I think the suspend/resume in guest is working, cause I can use pause/unpause
command in toolstack to suspend/resume guest without problem. I can also see
the suspend/resume kernel messages from guest's console. The only problem is
it's can not resume from restore.


But can you still connect to the guest after resume, maybe over the network?
If you cannot, then something is likely wrong.


Hi Stefano,

I can connect to the guest after resume from xen console. It responds by 
'return' key, but I can not run any other commands, e.g. ls or ps. I 
think the guest is not 'fully' restored.






One thing that confused me is that the kernel's hibernation means the guest
kernel will save the memory state to disk and power off VM at last. The guest
will also take care of the memory restore itself. But I do not see the
save/restore on xen works that way. So my question is why it requires
hibernation (aka. suspend to disk) instead of the real suspend (aka. suspend
to RAM and standby)?


Xen suspend/resume has nothing to do with guest suspend to RAM or guest
hibernation.

Xen suspend/resume is a way for the hypervisor to save to file the
entire state of the VM, including RAM and the state of any devices.
Guest suspend to RAM and guest hibernation are two guest driven
technologies to save the state of the operating system to RAM or to
disk. The only link between Xen suspend and guest suspend is that when
Xen issues a domain suspend, it notifies the guest of it so that it can
ease the process.  The code in Linux to support Xen suspend/resume is:

drivers/xen/manage.c:do_suspend

and makes use of some of the Linux internal hooks provided for
hibernations (see CONFIG_HIBERNATE_CALLBACKS). But that's just for
better integration with the rest of Linux: hibernation is NOT what is
happening.

I hope that this clarifies things a bit, I realize that it is confusing.



Thanks for your explanation, It clear enough and just as my 
understanding from the code. I think the problem might caused by 
incompatible of arm p2m and xen save/restore mechanism. I'll try a 
core-dump and compare the memory after save and restore. I suppose this 
two dumps should be identical but there already pages are different. 
I'll let you know if I got some progress.


Regards.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] questions of vm save/restore on arm64

2016-06-03 Thread Chenxiao Zhao 



On 6/3/2016 4:02 AM, Julien Grall wrote:

Hello,

First thing, the time in the mail headers seems to be wrong. Maybe
because of a wrong timezone?

I got: 04/06/16 02:32 however we are still the 3rd in my timezone.

On 04/06/16 02:32, Chenxiao Zhao wrote:



On 6/3/2016 3:16 AM, Julien Grall wrote:

Hello,

On 03/06/16 18:05, Chenxiao Zhao wrote:

I finally found out that the problem is that the toolstack did not get
corret p2m_size while sending all pages on save(always be zero).
After I
fixed that, the guest could be restored but guest kernel caught
handle_mm_fault().

where do you think I'm going to investigate, guest kernel hibernation
restore or xen?


The hibernation support for ARM64 has only been merged recently in the
kernel. Which kernel are you using?


Hi Julien,

I'm using a linaro ported Linux kernel 4.1 for hikey from this link.

https://github.com/96boards/linux/tree/android-hikey-linaro-4.1

I also applied following patches to make the kernel support hibernation.


This looks the wrong way to do it as this series may requires some
patches which have been upstreamed before hand.

Linux upstream seems support to the hikey board [1]. Any reason to not
using it?


I tried a newer version of kernel 4.4, but got no luck to start dom0 
with xen. so I decide to stay in 4.1 for now.





[1] http://www.spinics.net/lists/arm-kernel/msg477769.html
[2] http://lists.xen.org/archives/html/xen-devel/2015-12/msg01068.html



Also, what are the modifications you have made to support Xen
suspend/resume for ARM64?


I believe I have posted my modifications on xen in the first mail of
this thread.


I mean in Linux. The patch from Ian Campbell does not have any kind of
support for ARM64.

For instance arch/arm/xen/suspend.c needs to be built for ARM64. So I am
wondering if your kernel has support of hibernation...


Oh, yes, I most forgot I added this file in arch/arm64/xen/Makefile to 
let it build for arm64.




 From my understanding, a kernel hibernation will cause kernel to save
memories to disk(swap partition). But on guest save progress, the
hibernation for domU does not make the guest save memories to disk. it's
more like suspend all processes in guest, and memors actually depends on
xen toolstack to save the pages to file. Am I correct?


You are using an older tree with a patch series based on a newer tree.

So I would recommend you to move to a newer tree. If it is not possible,
please test that hibernation works on baremetal.


I think the suspend/resume in guest is working, cause I can use 
pause/unpause command in toolstack to suspend/resume guest without 
problem. I can also see the suspend/resume kernel messages from guest's 
console. The only problem is it's can not resume from restore.


One thing that confused me is that the kernel's hibernation means the 
guest kernel will save the memory state to disk and power off VM at 
last. The guest will also take care of the memory restore itself. But I 
do not see the save/restore on xen works that way. So my question is why 
it requires hibernation (aka. suspend to disk) instead of the real 
suspend (aka. suspend to RAM and standby)?





Regards,

[1] https://lists.96boards.org/pipermail/dev/2016-May/000933.html



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] questions of vm save/restore on arm64

2016-06-03 Thread Chenxiao Zhao 



On 6/3/2016 3:16 AM, Julien Grall wrote:

Hello,

On 03/06/16 18:05, Chenxiao Zhao wrote:

I finally found out that the problem is that the toolstack did not get
corret p2m_size while sending all pages on save(always be zero). After I
fixed that, the guest could be restored but guest kernel caught
handle_mm_fault().

where do you think I'm going to investigate, guest kernel hibernation
restore or xen?


The hibernation support for ARM64 has only been merged recently in the
kernel. Which kernel are you using?


Hi Julien,

I'm using a linaro ported Linux kernel 4.1 for hikey from this link.

https://github.com/96boards/linux/tree/android-hikey-linaro-4.1

I also applied following patches to make the kernel support hibernation.

[1] http://www.spinics.net/lists/arm-kernel/msg477769.html
[2] http://lists.xen.org/archives/html/xen-devel/2015-12/msg01068.html



Also, what are the modifications you have made to support Xen
suspend/resume for ARM64?


I believe I have posted my modifications on xen in the first mail of 
this thread.


From my understanding, a kernel hibernation will cause kernel to save 
memories to disk(swap partition). But on guest save progress, the 
hibernation for domU does not make the guest save memories to disk. it's 
more like suspend all processes in guest, and memors actually depends on 
xen toolstack to save the pages to file. Am I correct?


Looking forward for your advice.

Thanks.



Regards,



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] questions of vm save/restore on arm64

2016-06-02 Thread Chenxiao Zhao 



On 6/2/2016 5:29 AM, Julien Grall wrote:

Hello,

On 01/06/16 01:28, Chenxiao Zhao wrote:



On 5/30/2016 4:40 AM, Stefano Stabellini wrote:

On Fri, 27 May 2016, Chenxiao Zhao wrote:

Hi,

My board is Hikey on which have octa-core of arm cortex-a53. I have
applied patches [1] to try vm save/restore on arm.
These patches originally do not working on arm64. I have made some
changes based on patch set [2].


Hello Chenxiao,

thanks for your interest in Xen on ARM save/restore.


Hi Stefano,

Thanks for your advice.

I found a possible reason that cause the restore failure is that xen
always failed on p2m_lookup for guest domain.


Who is calling p2m_lookup?


It call by handle_hvm_params(tools/libxc/xc_sr_restore_arm.c) while 
restoring HVM_PARAM_STORE_PFN.






I called dump_p2m_lookup in p2m_look() and get the output like below:

(XEN) dom1 IPA 0x39001000


Looking at the memory layout (see include/public/arch-arm.h) 0x39001000
is part of the magic region. It contains pages for the console,
xenstore, memaccess (see the list in tools/libxc/xc_dom_arm.c).


yes, I also noticed that.



0x39001000 is the base address of the xenstore page.


(XEN) P2M @ 000801e7ce80 mfn:0x79f3a
(XEN) Using concatenated root table 0
(XEN) 1ST[0x0] = 0x004079f3c77f
(XEN) 2ND[0x1c8] = 0x

My question is:

1. who is responsible for restoring p2m table, Xen or guest kernel?


AFAIK, the toolstack is restoring the most of the memory.


2. After restore, the vm always get zero memory space, but there is no


What do you mean by "zero memory space"?


I mean xl does not assign any memory to the restored VM.

NameID   Mem VCPUs  State 
Time(s)
Domain-0 0  1024 8 r- 
  76.3
guest2 0 1 r- 
   1.9






error reported on the restore progress. Does the memory requested by the
guest kernel or should be allocated early by hypervisor?


Bear in mind that the patch series you are working on is an RFC and the
ARM64 was not supported. There might be some issue in the save/restore
path.

For instance xc_clear_domain_page (tools/libxc/xc_sr_restore_arm.c) main
return an error if the memory is not mapped. But the caller does not
check the return value.

Regards,


I finally found out that the problem is that the toolstack did not get 
corret p2m_size while sending all pages on save(always be zero). After I 
fixed that, the guest could be restored but guest kernel caught 
handle_mm_fault().


where do you think I'm going to investigate, guest kernel hibernation 
restore or xen?


Best regards.





___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.7-rc4 on HiKey

2016-06-02 Thread Chenxiao Zhao 



On 6/1/2016 6:09 PM, Tamas K Lengyel wrote:

On Wed, Jun 1, 2016 at 3:49 PM, Julien Grall  wrote:

On 01/06/2016 18:10, Tamas K Lengyel wrote:


Hi all,



Hi Tamas,


following the steps from http://wiki.xen.org/wiki/HiKey I'm unable to
get the board to boot Xen. I'm using the Debian reference image from
linaro as base
(https://builds.96boards.org/releases/hikey/linaro/debian/latest/)
and that boots fine both from the SD card directly and through
startup.nsh. However, when I try to boot Xen I see only the following:

Xen 4.7.0-rc (c/s Mon May 23 12:07:20 2016 +0100 git:8478c94-dirty) EFI
loader
Using configuration file 'xen.cfg'
Image: 0x7a121000-0x7acb8a70

After that no output and dom0 never comes online on the network
either. I tried compiling Xen on the device itself in case it was a
problem with my cross-compile setup but no difference. Also with and
without debug=y. Any tips on what might be going wrong here?



I gave a quick look at the upstream device-tree of the hikey, the path
/smb/uart@f7113000.

If you use the upstream device-tree, it should be able to find the correct
UART via "stdout-path" and therefore dtuart=... should not be necessary.

Let me know if it works for you, and I will update the wiki page.


No, I removed the dtuart=... part from xen.cfg but there is no
difference in the output and dom0 never comes up on the network
either.


Hi Tamas,

My xen.cfg for Hikey is like this:

options=dom0_mem=1024M dom0_max_vcpus=8 conswitch=x console=dtuart 
dtuart=/smb/uart@f7113000

kernel=Image console=hvc root=/dev/mmcblk0p9 rootwait rw 3
dtb=hi6220-hikey.dtb

hope this could help you.



Tamas

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] questions of vm save/restore on arm64

2016-05-31 Thread Chenxiao Zhao 



On 5/30/2016 4:40 AM, Stefano Stabellini wrote:

On Fri, 27 May 2016, Chenxiao Zhao wrote:

Hi,

My board is Hikey on which have octa-core of arm cortex-a53. I have applied 
patches [1] to try vm save/restore on arm.
These patches originally do not working on arm64. I have made some changes 
based on patch set [2].


Hello Chenxiao,

thanks for your interest in Xen on ARM save/restore.


Hi Stefano,

Thanks for your advice.

I found a possible reason that cause the restore failure is that xen 
always failed on p2m_lookup for guest domain.


I called dump_p2m_lookup in p2m_look() and get the output like below:

(XEN) dom1 IPA 0x39001000
(XEN) P2M @ 000801e7ce80 mfn:0x79f3a
(XEN) Using concatenated root table 0
(XEN) 1ST[0x0] = 0x004079f3c77f
(XEN) 2ND[0x1c8] = 0x

My question is:

1. who is responsible for restoring p2m table, Xen or guest kernel?
2. After restore, the vm always get zero memory space, but there is no 
error reported on the restore progress. Does the memory requested by the 
guest kernel or should be allocated early by hypervisor?



Name  ID   Mem VCPUs  State   Time(s)
Domain-0   0  1024 8 r-  15.0
guest  1 0 1 --p---   0.0







What I have got so far is

1. if I run 'xl save -p guest memState' to leave guest in suspend state, then 
run 'xl unpause guest'.
the guest can resume successfully. so I suppose the guest works find on 
suspend/resume.

2. if I run 'xl restore -p memState' to restore guest and use xenctx to dump 
all vcpu's registers.
all the registers are identical to the state on save. After I run 'xl 
unpause guest', I got no error but can not connect to console.
After restore the guest's PC is at a function called 
user_disable_single_step(), which is called by single_step_handler().

My question is

1. How could I debug guest on restore progress? are there any tools available?


Nothing special. You can use ctrl-AAA on the console to switch to the
hypervisor console and see the state of the guest. You can also add some
debug printks; if the console doesn't work you can use
dom0_write_console in Linux to get messages out of your guest (you need
to compile Xen with debug=y for that to work).



2. From my understanding, the restore not working is because some status is 
missing when saving.
 e.g. on cpu_save, it know the domain is 64bit, but on cpu_load, it always 
think it is a 32bit domain. so I have hard coded the domain type to
DOMAIN_64BIT.
Am I correct?


If Xen thinks the domain is 32-bit at restore, it must be a bug.



3. How could I dump all VM's status? I only found xenctx can dump vcpu's 
registers.


You can use the hypervisor console via the ctrl-aaa menu.



I have attached my patch and log below.

Looking forward for your feedback.
Thanks

xl list
NameID   Mem VCPUs  State   Time(s)
Domain-0 0  1024 8 r-  11.7
root@linaro-alip:~# xl create guest.cfg
Parsing config from guest.cfg
[   39.238723] xen-blkback: ring-ref 8, event-channel 3, protocol 1 (arm-abi) 
persistent grants

root@linaro-alip:~# xl save -p guest memState
Saving to memState new xl format (info 0x3/0x0/931)
xc: info: Saving domain 1, type ARM
(XEN) HVM1 save: VCPU
(XEN) HVM1 save: A15_TIMER
(XEN) HVM1 save: GICV2_GICD
(XEN) HVM1 save: GICV2_GICC
(XEN) HVM1 save: GICV3
root@linaro-alip:~# /usr/lib/xen/bin/xenctx -a 1
PC:   ffcab028
LR:   ffc00050458c
ELR_EL1:  ffc86b34
CPSR: 21c5
SPSR_EL1: 6145
SP_EL0:   007ff6f2a850
SP_EL1:   ffc0140a7ca0

 x0: 0001x1: deadbeefx2: 0002
 x3: 0002x4: 0004x5: 
 x6: 001bx7: 0001x8: 00618e589e00
 x9:    x10:    x11: 
x12: 01a3   x13: 1911a7d9   x14: 2ee0
x15: 0005   x16: deadbeef   x17: 0001
x18: 0007   x19:    x20: ffc014163d58
x21: ffc014163cd8   x22: 0001   x23: 0140
x24: ffc000d5bb18   x25: ffc014163cd8   x26: 
x27:    x28:    x29: ffc0140a7ca0

SCTLR: 34d5d91d
TTBCR: 0032b5193519
TTBR0: 002d54876000
TTBR1: 40dcf000
root@linaro-alip:~# xl destroy guest
(XEN) mm.c:1265:d0v1 gnttab_mark_dirty not implemented yet
root@linaro-alip:~# xl restore -p memState
Loading new save file memState (new xl fmt info 0x3/0x0/931)
 Savefile contains xl domain config in JSON format
Parsing config from 
xc: info: (XEN) HVM2 restore: VCPU 0
Found ARM domain from Xen 4.7
xc: info: Restoring domain
(XEN) HVM2 restore: A15_TIMER 0
(XEN) HVM2 restore: A15_TIMER 0
(XEN) HVM2 restore: GICV2_GICD 0
(XEN) HVM2 restore: GICV2_GICC 0
(XEN

[Xen-devel] questions of vm save/restore on arm64

2016-05-27 Thread Chenxiao Zhao 
Hi,

My board is Hikey on which have octa-core of arm cortex-a53. I have applied
patches [1] to try vm save/restore on arm.
These patches originally do not working on arm64. I have made some changes
based on patch set [2].

What I have got so far is

1. if I run 'xl save -p guest memState' to leave guest in suspend state,
then run 'xl unpause guest'.
the guest can resume successfully. so I suppose the guest works find on
suspend/resume.

2. if I run 'xl restore -p memState' to restore guest and use xenctx to
dump all vcpu's registers.
all the registers are identical to the state on save. After I run 'xl
unpause guest', I got no error but can not connect to console.
After restore the guest's PC is at a function
called user_disable_single_step(), which is called
by single_step_handler().

My question is

1. How could I debug guest on restore progress? are there any tools
available?
2. From my understanding, the restore not working is because some status is
missing when saving.
 e.g. on cpu_save, it know the domain is 64bit, but on cpu_load, it always
think it is a 32bit domain. so I have hard coded the domain type to
DOMAIN_64BIT.
Am I correct?
3. How could I dump all VM's status? I only found xenctx can dump vcpu's
registers.

I have attached my patch and log below.

Looking forward for your feedback.
Thanks

xl list
NameID   Mem VCPUs  State
Time(s)
Domain-0 0  1024 8 r-
 11.7
root@linaro-alip:~# xl create guest.cfg
Parsing config from guest.cfg
[   39.238723] xen-blkback: ring-ref 8, event-channel 3, protocol 1
(arm-abi) persistent grants

root@linaro-alip:~# xl save -p guest memState
Saving to memState new xl format (info 0x3/0x0/931)
xc: info: Saving domain 1, type ARM
(XEN) HVM1 save: VCPU
(XEN) HVM1 save: A15_TIMER
(XEN) HVM1 save: GICV2_GICD
(XEN) HVM1 save: GICV2_GICC
(XEN) HVM1 save: GICV3
root@linaro-alip:~# /usr/lib/xen/bin/xenctx -a 1
PC:   ffcab028
LR:   ffc00050458c
ELR_EL1:  ffc86b34
CPSR: 21c5
SPSR_EL1: 6145
SP_EL0:   007ff6f2a850
SP_EL1:   ffc0140a7ca0

 x0: 0001x1: deadbeefx2: 0002
 x3: 0002x4: 0004x5: 
 x6: 001bx7: 0001x8: 00618e589e00
 x9:    x10:    x11: 
x12: 01a3   x13: 1911a7d9   x14: 2ee0
x15: 0005   x16: deadbeef   x17: 0001
x18: 0007   x19:    x20: ffc014163d58
x21: ffc014163cd8   x22: 0001   x23: 0140
x24: ffc000d5bb18   x25: ffc014163cd8   x26: 
x27:    x28:    x29: ffc0140a7ca0

SCTLR: 34d5d91d
TTBCR: 0032b5193519
TTBR0: 002d54876000
TTBR1: 40dcf000
root@linaro-alip:~# xl destroy guest
(XEN) mm.c:1265:d0v1 gnttab_mark_dirty not implemented yet
root@linaro-alip:~# xl restore -p memState
Loading new save file memState (new xl fmt info 0x3/0x0/931)
 Savefile contains xl domain config in JSON format
Parsing config from 
xc: info: (XEN) HVM2 restore: VCPU 0
Found ARM domain from Xen 4.7
xc: info: Restoring domain
(XEN) HVM2 restore: A15_TIMER 0
(XEN) HVM2 restore: A15_TIMER 0
(XEN) HVM2 restore: GICV2_GICD 0
(XEN) HVM2 restore: GICV2_GICC 0
(XEN) GICH_LRs (vcpu 0) mask=0
(XEN)VCPU_LR[0]=0
(XEN)VCPU_LR[1]=0
(XEN)VCPU_LR[2]=0
(XEN)VCPU_LR[3]=0
xc: info: Restore successful
xc: info: XenStore: mfn 0x39001, dom 0, evt 1
xc: info: Console: mfn 0x39000, dom 0, evt 2
root@linaro-alip:~# /usr/lib/xen/bin/xenctx -a 2
PC:   ffcab028
LR:   ffc00050458c
ELR_EL1:  ffc86b34
CPSR: 21c5
SPSR_EL1: 6145
SP_EL0:   007ff6f2a850
SP_EL1:   ffc0140a7ca0

 x0: x1: deadbeefx2: 0002
 x3: 0002x4: 0004x5: 
 x6: 001bx7: 0001x8: 00618e589e00
 x9:    x10:    x11: 
x12: 01a3   x13: 1911a7d9   x14: 2ee0
x15: 0005   x16: deadbeef   x17: 0001
x18: 0007   x19:    x20: ffc014163d58
x21: ffc014163cd8   x22: 0001   x23: 0140
x24: ffc000d5bb18   x25: ffc014163cd8   x26: 
x27:    x28:    x29: ffc0140a7ca0

SCTLR: 34d5d91d
TTBCR: b5193519
TTBR0: 002d54876000
TTBR1: 40dcf000
root@linaro-alip:~# xl unpause guest
root@linaro-alip:~# xl list
NameID   Mem VCPUs  State
Time(s)
Domain-0 0  1024 8 r-
 22.2
guest2 0 1 r-
4.8

Re: [Xen-devel] [PATCH v2 12/15] xen/arm: arm64: Add Cortex-A53 cache errata workaround

2016-05-23 Thread Chenxiao Zhao 
On Mon, May 23, 2016 at 7:22 AM Julien Grall  wrote:

> The ARM errata 819472, 827319 and 824069 define the same workaround for
> these hardware issues in certain Cortex-A53 parts.
>
> The cache instructions "dc cvac" and "dc cvau" need to be upgraded to
> "dc civac".
>
> Use the alternative framework to replace those instructions only on
> affected cores.
>
> Whilst the errata affect cache instructions issued at any exception
> level, it is not necessary to trap EL1/EL0 data cache instructions
> access in order to upgrade them. Indeed the data cache corruption would
> always be at the address used by the data cache instructions. Note that
> this address could point to a shared memory between guests and the
> hypervisors, however all the information present in it are be validated
> before any use.
>
> Therefore a malicious guest could only hurt itself. Note that all the
> guests should implement/enable the workaround for the affected cores.
>
> Signed-off-by: Julien Grall 
> ---
>  docs/misc/arm/silicon-errata.txt |  3 ++
>  xen/arch/arm/Kconfig | 71
> 
>  xen/arch/arm/arm64/cache.S   | 54 ++
>  xen/arch/arm/cpuerrata.c | 17 ++
>  xen/include/asm-arm/arm64/page.h |  7 +++-
>  xen/include/asm-arm/cpufeature.h |  4 ++-
>  xen/include/asm-arm/processor.h  |  6 
>  7 files changed, 160 insertions(+), 2 deletions(-)
>
> diff --git a/docs/misc/arm/silicon-errata.txt
> b/docs/misc/arm/silicon-errata.txt
> index 3f0d32b..9a2983b 100644
> --- a/docs/misc/arm/silicon-errata.txt
> +++ b/docs/misc/arm/silicon-errata.txt
> @@ -42,4 +42,7 @@ stable hypervisors.
>  | Implementor| Component   | Erratum ID  | Kconfig
>  |
>
>  
> ++-+-+-+
>  | ARM| Cortex-A15  | #766422 | N/A
>  |
> +| ARM| Cortex-A53  | #827319 |
> ARM64_ERRATUM_827319|
> +| ARM| Cortex-A53  | #824069 |
> ARM64_ERRATUM_824069|
> +| ARM| Cortex-A53  | #819472 |
> ARM64_ERRATUM_819472|
>  | ARM| Cortex-A57  | #852523 | N/A
>  |
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index 0141bd9..a473d9c 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -53,6 +53,77 @@ config ALTERNATIVE
>
>  endmenu
>
> +menu "ARM errata workaround via the alternative framework"
> +   depends on ALTERNATIVE
> +
> +config ARM64_ERRATUM_827319
> +   bool "Cortex-A53: 827319: Data cache clean instructions might
> cause overlapping transactions to the interconnect"
> +   default y
> +   depends on ARM_64
> +   help
> + This option adds an alternative code sequence to work around ARM
> + erratum 827319 on Cortex-A53 parts up to r0p2 with an AMBA 5 CHI
> + master interface and an L2 cache.
> +
> + Under certain conditions this erratum can cause a clean line
> eviction
> + to occur at the same time as another transaction to the same
> address
> + on the AMBA 5 CHI interface, which can cause data corruption if
> the
> + interconnect reorders the two transactions.
> +
> + The workaround promotes data cache clean instructions to
> + data cache clean-and-invalidate.
> + Please note that this does not necessarily enable the workaround,
> + as it depends on the alternative framework, which will only patch
> + the kernel if an affected CPU is detected.
> +
> + If unsure, say Y.
> +
> +config ARM64_ERRATUM_824069
> +   bool "Cortex-A53: 824069: Cache line might not be marked as clean
> after a CleanShared snoop"
> +   default y
> +   depends on ARM_64
> +   help
> + This option adds an alternative code sequence to work around ARM
> + erratum 824069 on Cortex-A53 parts up to r0p2 when it is
> connected
> + to a coherent interconnect.
> +
> + If a Cortex-A53 processor is executing a store or prefetch for
> + write instruction at the same time as a processor in another
> + cluster is executing a cache maintenance operation to the same
> + address, then this erratum might cause a clean cache line to be
> + incorrectly marked as dirty.
> +
> + The workaround promotes data cache clean instructions to
> + data cache clean-and-invalidate.
> + Please note that this option does not necessarily enable the
> + workaround, as it depends on the alternative framework, which
> will
> + only patch the kernel if an affected CPU is detected.
> +
> + If unsure, say Y.
> +
> +config ARM64_ERRATUM_819472
> +   bool "Cortex-A53: 819472: Store exclusive instructions might cause
> data corruption"
> +   default y
> +   depends on ARM_64
> +   help
> +

[Xen-devel] run xl toolstack on hikey board

2016-05-10 Thread Chenxiao Zhao 
Hi,

I have followed xen wiki to get xen hypervisor and dom0 run on hikey board.
when I try to run xl command to list all the domain in dom0, the command
line get stuck and even ctrl+c can not terminate it. I used strace and
found the last system call is write data to xenbus and it does not return.

How could I determine what the issue is and how to solve it?

Thanks.

Attached the log, last line was the place where got stuck.

root@linaro-alip:~# xl info
host   : linaro-alip
release: 4.1.15+
version: #5 SMP PREEMPT Tue May 10 16:05:38 CST 2016
machine: aarch64
nr_cpus: 8
max_cpu_id : 127
nr_nodes   : 1
cores_per_socket   : 1
threads_per_core   : 1
cpu_mhz: 1
hw_caps:
:::::::
virt_caps  :
total_memory   : 1998
free_memory: 953
sharing_freed_memory   : 0
sharing_used_memory: 0
outstanding_claims : 0
free_cpus  : 0
xen_major  : 4
xen_minor  : 7
xen_extra  : .0-rc
xen_version: 4.7.0-rc
xen_caps   : xen-3.0-aarch64 xen-3.0-armv7l
xen_scheduler  : credit
xen_pagesize   : 4096
platform_params: virt_start=0x20
xen_changeset  : Wed May 4 08:54:07 2016 -0700 git:f8d4c1d-dirty
xen_commandline: xen dom0_mem=1024M dom0_max_vcpus=8 conswitch=x
console
=dtuart dtuart=/smb/uart@f7113000
cc_compiler: aarch64-linux-gnu-gcc (crosstool-NG
linaro-1.13.1-4.9-2
014.09 -
cc_compile_by  : uxum
cc_compile_domain  :
cc_compile_date: Mon May  9 11:54:02 CST 2016
build_id   : 3f708087e6d8eda47f581d576372e08f782fccc1
xend_config_format : 4
root@linaro-alip:~# strace xl list
execve("/usr/local/sbin/xl", ["xl", "list"], [/* 14 vars */]) = 0
brk(0)  = 0x2db5a000
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file
or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f98297000
faccessat(AT_FDCWD, "/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file
or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=53481, ...}) = 0
mmap(NULL, 53481, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f98262000
close(3)= 0
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file
or directory)
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/tls/aarch64/libxlutil.so.4.6",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib/aarch64-linux-gnu/tls/aarch64", 0x7fc6d86ba0, 0)
= -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/tls/libxlutil.so.4.6",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib/aarch64-linux-gnu/tls", 0x7fc6d86ba0, 0) = -1
ENOENT(No such file or directory)
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/aarch64/libxlutil.so.4.6",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib/aarch64-linux-gnu/aarch64", 0x7fc6d86ba0, 0) =
-1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libxlutil.so.4.6",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib/aarch64-linux-gnu", {st_mode=S_IFDIR|0755,
st_size=8192, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib/aarch64-linux-gnu/tls/aarch64/libxlutil.so.4.6",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib/aarch64-linux-gnu/tls/aarch64",
0x7fc6d86ba0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/aarch64-linux-gnu/tls/libxlutil.so.4.6",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib/aarch64-linux-gnu/tls", 0x7fc6d86ba0, 0) =
-1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/aarch64-linux-gnu/aarch64/libxlutil.so.4.6",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib/aarch64-linux-gnu/aarch64", 0x7fc6d86ba0, 0)
= -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/aarch64-linux-gnu/libxlutil.so.4.6",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib/aarch64-linux-gnu", {st_mode=S_IFDIR|0755,
st_size=28672, ...}, 0) = 0
openat(AT_FDCWD, "/lib/tls/aarch64/libxlutil.so.4.6", O_RDONLY|O_CLOEXEC) =
-1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib/tls/aarch64", 0x7fc6d86ba0, 0) = -1 ENOENT (No
such file or directory)
openat(AT_FDCWD, "/lib/tls/libxlutil.so.4.6", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib/tls", 0x7fc6d86ba0, 0) = -1 ENOENT (No such file
or directory)
openat(AT_FDCWD, "/lib/aarch64/libxlutil.so.4.6", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or