Re: [Xen-devel] xl create failure on arm64 with XEN 4.9rc6

[Julien Grall]

Hi,

On 30/05/17 18:41, Feng Kan wrote:

Yap, worked like a charm. Much thanks.

Tested on Centos 7 arm64 system with 4.12rc2 64K page size kernel
(dom0 and domU) & Xen 4.9rc6
Tested-by: Feng Kan 


I have sent a slightly different version, so I didn't keep your tested-by.

Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] xl create failure on arm64 with XEN 4.9rc6

[Feng Kan]

Yap, worked like a charm. Much thanks.

Tested on Centos 7 arm64 system with 4.12rc2 64K page size kernel
(dom0 and domU) & Xen 4.9rc6
Tested-by: Feng Kan 

On Sun, May 28, 2017 at 10:12 AM, Julien Grall  wrote:
> Hi,
>
> On 05/26/2017 11:22 PM, Feng Kan wrote:
>> On Fri, May 26, 2017 at 5:40 AM, Julien Grall  wrote:
>>>
>>>
>>> On 26/05/17 01:37, Feng Kan wrote:

 On Thu, May 25, 2017 at 12:56 PM, Julien Grall 
 wrote:
>
> (CC toolstack maintainers)
>
> On 25/05/2017 19:58, Feng Kan wrote:
>>
>>
>> Hi All:
>
>
>
> Hello,
>
>> This is not specifically against the XEN 4.9. I am using 4.12rc2
>> kernel on arm64 platform. Started dom0 fine with ACPI enabled, but
>> failed when creating the domU guest. Xen is built natively on the
>> arm64 platform. Using the same kernel and ramdisk as dom0. Any idea as
>> why it is stuck here
>> would be greatly appreciated?
>
>
>
> The first step would to try a stable release if you can. Also, it would
> be
> useful if you provide information about the guest (i.e the configuration)
> and your .config for the kernel.

 I am using the default xen_defconfig in the arm64 directory.
>>>
>>>
>>> I am confused. There are no xen_defconfig in the arm64 directory of the
>>> kernel. So which one are you talking about?
>> Sorry, my mistake.
>>>
 This is
 very early on
 in building the domain, would the guest configuration matter?
>>>
>>>
>>> The configuration of DOM0 kernel matters when you want to build the guest.
>>> That's why I wanted to know what options you enabled.
>> I see. I am using the default centos 7.2 kernel config plus enabling
>> the XEN option. (Attached below)
>
> Looking at the .config, Linux is using 64KB page granularity.
>
> I managed to reproduce the failure (though different error) by using
> an initramfs > 32MB (smaller works). The patch below should fix the
> error, can you give it a try?
>
> commit c4684b425552a8330f00d7703f3175d721992ab0
> Author: Julien Grall 
> Date:   Sun May 28 17:50:07 2017 +0100
>
> xen/privcmd: Support correctly 64KB page granularity when mapping memory
>
> Commit 5995a68 "xen/privcmd: Add support for Linux 64KB page granularity" 
> did
> not go far enough to support 64KB in mmap_batch_fn.
>
> The variable 'nr' is the number of 4KB chunk to map. However, when Linux
> is using 64KB page granularity the array of pages (vma->vm_private_data)
> contain one page per 64KB. Fix it by incrementing st->index correctly.
>
> Furthermore, st->va is not correctly incremented as PAGE_SIZE !=
> XEN_PAGE_SIZE.
>
> Fixes: 5995a68 ("xen/privcmd: Add support for Linux 64KB page 
> granularity")
> CC: sta...@vger.kernel.org
> Reported-by: Feng Kan 
> Signed-off-by: Julien Grall 
>
> diff --git a/drivers/xen/privcmd.c b/drivers/xen/privcmd.c
> index 7a92a5e..38d9a43 100644
> --- a/drivers/xen/privcmd.c
> +++ b/drivers/xen/privcmd.c
> @@ -345,7 +345,7 @@ static int mmap_batch_fn(void *data, int nr, void *state)
> int ret;
>
> if (xen_feature(XENFEAT_auto_translated_physmap))
> -   cur_pages = [st->index];
> +   cur_pages = [st->index / XEN_PFN_PER_PAGE];
>
> BUG_ON(nr < 0);
> ret = xen_remap_domain_gfn_array(st->vma, st->va & PAGE_MASK, gfnp, 
> nr,
> @@ -362,7 +362,7 @@ static int mmap_batch_fn(void *data, int nr, void *state)
> st->global_error = 1;
> }
> }
> -   st->va += PAGE_SIZE * nr;
> +   st->va += XEN_PAGE_SIZE * nr;
> st->index += nr;
>
> return 0;
>
> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] xl create failure on arm64 with XEN 4.9rc6

[Julien Grall]

Hi,

On 05/26/2017 11:22 PM, Feng Kan wrote:
> On Fri, May 26, 2017 at 5:40 AM, Julien Grall  wrote:
>>
>>
>> On 26/05/17 01:37, Feng Kan wrote:
>>>
>>> On Thu, May 25, 2017 at 12:56 PM, Julien Grall 
>>> wrote:

 (CC toolstack maintainers)

 On 25/05/2017 19:58, Feng Kan wrote:
>
>
> Hi All:



 Hello,

> This is not specifically against the XEN 4.9. I am using 4.12rc2
> kernel on arm64 platform. Started dom0 fine with ACPI enabled, but
> failed when creating the domU guest. Xen is built natively on the
> arm64 platform. Using the same kernel and ramdisk as dom0. Any idea as
> why it is stuck here
> would be greatly appreciated?



 The first step would to try a stable release if you can. Also, it would
 be
 useful if you provide information about the guest (i.e the configuration)
 and your .config for the kernel.
>>>
>>> I am using the default xen_defconfig in the arm64 directory.
>>
>>
>> I am confused. There are no xen_defconfig in the arm64 directory of the
>> kernel. So which one are you talking about?
> Sorry, my mistake.
>>
>>> This is
>>> very early on
>>> in building the domain, would the guest configuration matter?
>>
>>
>> The configuration of DOM0 kernel matters when you want to build the guest.
>> That's why I wanted to know what options you enabled.
> I see. I am using the default centos 7.2 kernel config plus enabling
> the XEN option. (Attached below)

Looking at the .config, Linux is using 64KB page granularity.

I managed to reproduce the failure (though different error) by using
an initramfs > 32MB (smaller works). The patch below should fix the
error, can you give it a try?

commit c4684b425552a8330f00d7703f3175d721992ab0
Author: Julien Grall 
Date:   Sun May 28 17:50:07 2017 +0100

xen/privcmd: Support correctly 64KB page granularity when mapping memory

Commit 5995a68 "xen/privcmd: Add support for Linux 64KB page granularity" 
did
not go far enough to support 64KB in mmap_batch_fn.

The variable 'nr' is the number of 4KB chunk to map. However, when Linux
is using 64KB page granularity the array of pages (vma->vm_private_data)
contain one page per 64KB. Fix it by incrementing st->index correctly.

Furthermore, st->va is not correctly incremented as PAGE_SIZE !=
XEN_PAGE_SIZE.

Fixes: 5995a68 ("xen/privcmd: Add support for Linux 64KB page granularity")
CC: sta...@vger.kernel.org
Reported-by: Feng Kan 
Signed-off-by: Julien Grall 

diff --git a/drivers/xen/privcmd.c b/drivers/xen/privcmd.c
index 7a92a5e..38d9a43 100644
--- a/drivers/xen/privcmd.c
+++ b/drivers/xen/privcmd.c
@@ -345,7 +345,7 @@ static int mmap_batch_fn(void *data, int nr, void *state)
int ret;
 
if (xen_feature(XENFEAT_auto_translated_physmap))
-   cur_pages = [st->index];
+   cur_pages = [st->index / XEN_PFN_PER_PAGE];
 
BUG_ON(nr < 0);
ret = xen_remap_domain_gfn_array(st->vma, st->va & PAGE_MASK, gfnp, nr,
@@ -362,7 +362,7 @@ static int mmap_batch_fn(void *data, int nr, void *state)
st->global_error = 1;
}
}
-   st->va += PAGE_SIZE * nr;
+   st->va += XEN_PAGE_SIZE * nr;
st->index += nr;
 
return 0;

Cheers,

-- 
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] xl create failure on arm64 with XEN 4.9rc6

[Julien Grall]

On 26/05/17 01:37, Feng Kan wrote:

On Thu, May 25, 2017 at 12:56 PM, Julien Grall  wrote:

(CC toolstack maintainers)

On 25/05/2017 19:58, Feng Kan wrote:


Hi All:



Hello,


This is not specifically against the XEN 4.9. I am using 4.12rc2
kernel on arm64 platform. Started dom0 fine with ACPI enabled, but
failed when creating the domU guest. Xen is built natively on the
arm64 platform. Using the same kernel and ramdisk as dom0. Any idea as
why it is stuck here
would be greatly appreciated?



The first step would to try a stable release if you can. Also, it would be
useful if you provide information about the guest (i.e the configuration)
and your .config for the kernel.

I am using the default xen_defconfig in the arm64 directory.


I am confused. There are no xen_defconfig in the arm64 directory of the 
kernel. So which one are you talking about?



This is
very early on
in building the domain, would the guest configuration matter?


The configuration of DOM0 kernel matters when you want to build the 
guest. That's why I wanted to know what options you enabled.




I gave a try on Linux 4.12rc2 and I was not able to reproduce your error.

Thanks, I started with 4.11 and work my way up. I have the same
problem in both cases.


I cannot rule out a problem in your .config until you sent me the full one.







xc: error: panic: xc_dom_boot.c:178: xc_dom_boot_domU_map: failed to
mmap domU pages 0x450c2+0x2f3d [mmap, errno=22 (Invalid ar
gument), chunksize 0x1000]: Internal error
libxl: error: libxl_dom.c:679:libxl__build_dom: xc_dom_build_image
failed: Invalid argument
domainbuilder: detail: xc_dom_release: called
libxl: error: libxl_create.c:1217:domcreate_rebuild_done: Domain
1:cannot (re-)build domain: -3
libxl: debug: libxl_domain.c:1140:devices_destroy_cb: Domain 1:Forked
pid 2477 for destroy of domain
libxl: debug: libxl_create.c:1646:do_domain_create: Domain 0:ao
0x1ae10cb0: inprogress: poller=0x1ae10d40, flags=i
libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x1ae10cb0:
complete, rc=-3
libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x1ae10cb0:
destroy
libxl: debug: libxl_domain.c:871:libxl_domain_destroy: Domain 1:ao
0x1ae10cb0: create: how=(nil) callback=(nil) poller=0x1ae10d
40

Seem to failed when mmapping pages for the ramdisk. I did some digging
and the failure
occurs during the  IOCTL_PRIVCMD_MMAPBATCH_V2 call. It seems the
8192's page had a err code of -22.



-22 is -EINVAL. There are quite a few path return -EINVAL, did you try to
narrow the failure in the kernel?

I dug a bit deeper, in privcmd_ioctl_mmap_batch, a global error was detected for
mmap_batch_fn during the mapping of the second half of  0x2f3d of pages. I am
still trying to track down why it is the 8192th element that cause the
problem. seem
like too much of coincidence its the first element of the second half.


Can you explain what you mean by "global error"?





 The system have plenty of memory.


Afterward, a null guest is created.
As a side note, how do I get rid of it?



Normally the domain should be destroyed by the tools if the building failed.

You should be able to destroy it using 'xl domain domid' where domid is the
domain ID of the domain. If it does not work, then it means dom0 is holding
reference on some page belonging to that domain.

Make sense, since I can rename it but destroy it still left it as null.


This is likely because someone (such as dom0) kept a reference on a page 
belonging to the domain.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] xl create failure on arm64 with XEN 4.9rc6

[Feng Kan]

On Thu, May 25, 2017 at 12:56 PM, Julien Grall  wrote:
> (CC toolstack maintainers)
>
> On 25/05/2017 19:58, Feng Kan wrote:
>>
>> Hi All:
>
>
> Hello,
>
>> This is not specifically against the XEN 4.9. I am using 4.12rc2
>> kernel on arm64 platform. Started dom0 fine with ACPI enabled, but
>> failed when creating the domU guest. Xen is built natively on the
>> arm64 platform. Using the same kernel and ramdisk as dom0. Any idea as
>> why it is stuck here
>> would be greatly appreciated?
>
>
> The first step would to try a stable release if you can. Also, it would be
> useful if you provide information about the guest (i.e the configuration)
> and your .config for the kernel.
I am using the default xen_defconfig in the arm64 directory. This is
very early on
in building the domain, would the guest configuration matter?
>
> I gave a try on Linux 4.12rc2 and I was not able to reproduce your error.
Thanks, I started with 4.11 and work my way up. I have the same
problem in both cases.

>
>>
>> xc: error: panic: xc_dom_boot.c:178: xc_dom_boot_domU_map: failed to
>> mmap domU pages 0x450c2+0x2f3d [mmap, errno=22 (Invalid ar
>> gument), chunksize 0x1000]: Internal error
>> libxl: error: libxl_dom.c:679:libxl__build_dom: xc_dom_build_image
>> failed: Invalid argument
>> domainbuilder: detail: xc_dom_release: called
>> libxl: error: libxl_create.c:1217:domcreate_rebuild_done: Domain
>> 1:cannot (re-)build domain: -3
>> libxl: debug: libxl_domain.c:1140:devices_destroy_cb: Domain 1:Forked
>> pid 2477 for destroy of domain
>> libxl: debug: libxl_create.c:1646:do_domain_create: Domain 0:ao
>> 0x1ae10cb0: inprogress: poller=0x1ae10d40, flags=i
>> libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x1ae10cb0:
>> complete, rc=-3
>> libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x1ae10cb0:
>> destroy
>> libxl: debug: libxl_domain.c:871:libxl_domain_destroy: Domain 1:ao
>> 0x1ae10cb0: create: how=(nil) callback=(nil) poller=0x1ae10d
>> 40
>>
>> Seem to failed when mmapping pages for the ramdisk. I did some digging
>> and the failure
>> occurs during the  IOCTL_PRIVCMD_MMAPBATCH_V2 call. It seems the
>> 8192's page had a err code of -22.
>
>
> -22 is -EINVAL. There are quite a few path return -EINVAL, did you try to
> narrow the failure in the kernel?
I dug a bit deeper, in privcmd_ioctl_mmap_batch, a global error was detected for
mmap_batch_fn during the mapping of the second half of  0x2f3d of pages. I am
still trying to track down why it is the 8192th element that cause the
problem. seem
like too much of coincidence its the first element of the second half.

>
>  The system have plenty of memory.
>>
>> Afterward, a null guest is created.
>> As a side note, how do I get rid of it?
>
>
> Normally the domain should be destroyed by the tools if the building failed.
>
> You should be able to destroy it using 'xl domain domid' where domid is the
> domain ID of the domain. If it does not work, then it means dom0 is holding
> reference on some page belonging to that domain.
Make sense, since I can rename it but destroy it still left it as null.
>
> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] xl create failure on arm64 with XEN 4.9rc6

[Julien Grall]

(CC toolstack maintainers)

On 25/05/2017 19:58, Feng Kan wrote:

Hi All:


Hello,


This is not specifically against the XEN 4.9. I am using 4.12rc2
kernel on arm64 platform. Started dom0 fine with ACPI enabled, but
failed when creating the domU guest. Xen is built natively on the
arm64 platform. Using the same kernel and ramdisk as dom0. Any idea as
why it is stuck here
would be greatly appreciated?


The first step would to try a stable release if you can. Also, it would 
be useful if you provide information about the guest (i.e the 
configuration) and your .config for the kernel.


I gave a try on Linux 4.12rc2 and I was not able to reproduce your error.



xc: error: panic: xc_dom_boot.c:178: xc_dom_boot_domU_map: failed to
mmap domU pages 0x450c2+0x2f3d [mmap, errno=22 (Invalid ar
gument), chunksize 0x1000]: Internal error
libxl: error: libxl_dom.c:679:libxl__build_dom: xc_dom_build_image
failed: Invalid argument
domainbuilder: detail: xc_dom_release: called
libxl: error: libxl_create.c:1217:domcreate_rebuild_done: Domain
1:cannot (re-)build domain: -3
libxl: debug: libxl_domain.c:1140:devices_destroy_cb: Domain 1:Forked
pid 2477 for destroy of domain
libxl: debug: libxl_create.c:1646:do_domain_create: Domain 0:ao
0x1ae10cb0: inprogress: poller=0x1ae10d40, flags=i
libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x1ae10cb0:
complete, rc=-3
libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x1ae10cb0: destroy
libxl: debug: libxl_domain.c:871:libxl_domain_destroy: Domain 1:ao
0x1ae10cb0: create: how=(nil) callback=(nil) poller=0x1ae10d
40

Seem to failed when mmapping pages for the ramdisk. I did some digging
and the failure
occurs during the  IOCTL_PRIVCMD_MMAPBATCH_V2 call. It seems the
8192's page had a err code of -22.


-22 is -EINVAL. There are quite a few path return -EINVAL, did you try 
to narrow the failure in the kernel?


 The system have plenty of memory.

Afterward, a null guest is created.
As a side note, how do I get rid of it?


Normally the domain should be destroyed by the tools if the building failed.

You should be able to destroy it using 'xl domain domid' where domid is 
the domain ID of the domain. If it does not work, then it means dom0 is 
holding reference on some page belonging to that domain.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] xl create failure on arm64 with XEN 4.9rc6

[Feng Kan]

Hi All:

This is not specifically against the XEN 4.9. I am using 4.12rc2
kernel on arm64 platform. Started dom0 fine with ACPI enabled, but
failed when creating the domU guest. Xen is built natively on the
arm64 platform. Using the same kernel and ramdisk as dom0. Any idea as
why it is stuck here
would be greatly appreciated?

xc: error: panic: xc_dom_boot.c:178: xc_dom_boot_domU_map: failed to
mmap domU pages 0x450c2+0x2f3d [mmap, errno=22 (Invalid ar
gument), chunksize 0x1000]: Internal error
libxl: error: libxl_dom.c:679:libxl__build_dom: xc_dom_build_image
failed: Invalid argument
domainbuilder: detail: xc_dom_release: called
libxl: error: libxl_create.c:1217:domcreate_rebuild_done: Domain
1:cannot (re-)build domain: -3
libxl: debug: libxl_domain.c:1140:devices_destroy_cb: Domain 1:Forked
pid 2477 for destroy of domain
libxl: debug: libxl_create.c:1646:do_domain_create: Domain 0:ao
0x1ae10cb0: inprogress: poller=0x1ae10d40, flags=i
libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x1ae10cb0:
complete, rc=-3
libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x1ae10cb0: destroy
libxl: debug: libxl_domain.c:871:libxl_domain_destroy: Domain 1:ao
0x1ae10cb0: create: how=(nil) callback=(nil) poller=0x1ae10d
40

Seem to failed when mmapping pages for the ramdisk. I did some digging
and the failure
occurs during the  IOCTL_PRIVCMD_MMAPBATCH_V2 call. It seems the
8192's page had a err code of -22. The system have plenty of memory.
Afterward, a null guest is created.
As a side note, how do I get rid of it?


[root@localhost xen-guest]# xl list
NameID   Mem VCPUs  State   Time(s)
Domain-0 0 6553632 r-  16.3
(null)   1 0 0 --p--d   0.0

[root@localhost xen-guest]# xl info
host   : localhost.localdomain
release: 4.12.0-rc2
version: #1 SMP Thu May 25 13:17:35 EDT 2017
machine: aarch64
nr_cpus: 32
max_cpu_id : 31
nr_nodes   : 1
cores_per_socket   : 1
threads_per_core   : 1
cpu_mhz: 50
hw_caps:
:::::::
virt_caps  :
total_memory   : 130700
free_memory: 61637
sharing_freed_memory   : 0
sharing_used_memory: 0
outstanding_claims : 0
free_cpus  : 0
xen_major  : 4
xen_minor  : 9
xen_extra  : -rc
xen_version: 4.9-rc
xen_caps   : xen-3.0-aarch64 xen-3.0-armv7l
xen_scheduler  : credit
xen_pagesize   : 4096
platform_params: virt_start=0x20
xen_changeset  : Tue May 23 19:21:04 2017 -0400 git:4108ded-dirty
xen_commandline: xen.efi dom0_mem=64G loglvl=all
guest_loglvl=all conswitch=x acpi=force
cc_compiler: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)
cc_compile_by  : root
cc_compile_domain  :
cc_compile_date: Thu May 25 14:38:25 EDT 2017
build_id   : 22732b38d4e0ea99d541519eb49a7d50115717da
xend_config_format : 4

Thanks in advance.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel