Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-10-06 Thread Stefano Stabellini
On Tue, 6 Oct 2015, Roger Pau Monné wrote:
> Do KVM guests also need such extensive set of modifications in order to
> run using 64KB pages? I know it's not fair to compare KVM and Xen in
> this regard, because the interfaces are very different, but that's what
> developers are going to look at.

QEMU doesn't need extensive modifications to run 64K guests on a 4K
host, but on the other hand KVM needs its own set of non-trivial
modifications to do it.___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-10-06 Thread Ian Campbell
On Tue, 2015-10-06 at 11:28 +0200, Roger Pau Monné wrote:
> El 22/09/15 a les 12.59, Ian Campbell ha escrit:
> > On Mon, 2015-09-14 at 17:23 +0200, Roger Pau Monné wrote:
> > > I'm not saying that we shouldn't take those patches, I'm just saying
> > > that IMHO this is a workaround, and I would like to see a plan and
> > > somebody committed to have it fixed in a proper way, by introducing a
> > > 64KB PV block protocol.
> > 
> > In my view the basic unit of operation for all Xen interfaces (on x86
> > and
> > arm at least[0]) is 4K. The peer at either end of a PV protocol should
> > therefore be entitled to assume that at a minimum the other end
> > supports
> > operation in 4K mode (i.e. 4K is the baseline).
> > 
> > Operations in larger sizes (which would necessarily be multiples of 4K)
> > are
> > then an extension which is subject to a negotiation between the two
> > ends,
> > it doesn't really matter if those larger sizes arise because of the use
> > of
> > superpages or because the guest is internally using some larger basic
> > page
> > size (which ARM calls a "granule", and where 64K comes from here).
> > 
> > I think this line of reasoning applies just as strongly to the
> > hypercall
> > ABI as well BTW, they all use 4K as their basic unit, but might support
> > extensions to operation on multiples of that (negotiated either via a
> > specific error return and fallback or via the use of XENFEAT).
> 
> Yes, I completely agree that the current Xen interface is based on 4KB
> chunks. What I'm trying to say is that the approach taken, which is
> "let's not modify Xen" puts all the burden on the guest and it is going
> to hurt us in the long run.
> 
> Are we expecting all guests that want to use 64KB page to implement all
> the modifications that are needed? IMHO, those modifications are very
> far from trivial, and we are putting the bar for Xen guest support very
> high, and that is wrong. We are already struggling to have a decent set
> of PV drivers on guests different than Linux, and this is certainly not
> making things easier.

I think you are underestimating the impact/burden of not having a common
lowest common denominator interface which all front and backends should be
expected to provide in order to interact successfully with each other at a
basic level.

With your approach of requiring a 64KB guest to use native 64K interfaces
only you end up with guests containing front and backends which simply
cannot communicate at all because they do not have a common denominator or
a combinatorial explosion of guests supporting different sets of mutually
incompatible interfaces with no common ground.

I think that fragmentation is a far worse danger than of developers being
unwilling or unable to provide helpers which iterate over pages in whatever
size the guest happens to have (either larger granule or superpages) and
performing however many of whatever ops size has been negotiated with the
peer.

I'm also not convinced that Julien's patches are any less trivial than the
equivalent code to select the correct operations based on the kernels page
size without breaking them up, the developer still needs to be aware of the
4K vs large distinction if they want to support larger than 4K pages.

Developers can still choose to only support 4K pages in their OS, if they
feel this scheme of supporting larger pages is too intrusive or hard I
doubt they are going to want to implement a scheme based on an alternative
large-page protocol either.

> Do KVM guests also need such extensive set of modifications in order to
> run using 64KB pages? I know it's not fair to compare KVM and Xen in
> this regard, because the interfaces are very different, but that's what
> developers are going to look at.

I'm not sure that virtio cares about pages at all, just addresses which
have no particular alignment or granularity constraints, since they don't
have grant tables nor any requirement to mediate accesses to guest memory
by the host at any granularity at all.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-10-06 Thread Roger Pau Monné
El 22/09/15 a les 12.59, Ian Campbell ha escrit:
> On Mon, 2015-09-14 at 17:23 +0200, Roger Pau Monné wrote:
>> I'm not saying that we shouldn't take those patches, I'm just saying
>> that IMHO this is a workaround, and I would like to see a plan and
>> somebody committed to have it fixed in a proper way, by introducing a
>> 64KB PV block protocol.
> 
> In my view the basic unit of operation for all Xen interfaces (on x86 and
> arm at least[0]) is 4K. The peer at either end of a PV protocol should
> therefore be entitled to assume that at a minimum the other end supports
> operation in 4K mode (i.e. 4K is the baseline).
> 
> Operations in larger sizes (which would necessarily be multiples of 4K) are
> then an extension which is subject to a negotiation between the two ends,
> it doesn't really matter if those larger sizes arise because of the use of
> superpages or because the guest is internally using some larger basic page
> size (which ARM calls a "granule", and where 64K comes from here).
> 
> I think this line of reasoning applies just as strongly to the hypercall
> ABI as well BTW, they all use 4K as their basic unit, but might support
> extensions to operation on multiples of that (negotiated either via a
> specific error return and fallback or via the use of XENFEAT).

Yes, I completely agree that the current Xen interface is based on 4KB
chunks. What I'm trying to say is that the approach taken, which is
"let's not modify Xen" puts all the burden on the guest and it is going
to hurt us in the long run.

Are we expecting all guests that want to use 64KB page to implement all
the modifications that are needed? IMHO, those modifications are very
far from trivial, and we are putting the bar for Xen guest support very
high, and that is wrong. We are already struggling to have a decent set
of PV drivers on guests different than Linux, and this is certainly not
making things easier.

Do KVM guests also need such extensive set of modifications in order to
run using 64KB pages? I know it's not fair to compare KVM and Xen in
this regard, because the interfaces are very different, but that's what
developers are going to look at.

Roger.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-29 Thread David Vrabel
On 29/09/15 17:33, Julien Grall wrote:
> Hi David,
> 
> On 29/09/15 17:27, David Vrabel wrote:
>> On 07/09/15 16:33, Julien Grall wrote:
>>>
>>> A branch based on the latest xentip/for-linus-4.3 can be found here:
>>>
>>> git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4
>>
>> This has too many conflicts with the
>> git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.4
>> branch.
>>
>> Please rebase.
> 
> I was about to send a new version rebased (no changes in it so far
> except for rebasing purpose). I will do it on top of for-linus-4.4.
> I will also include the swiotlb changes at the end as it was reviewed by
> Stefano.

Thanks.

> BTW, some of the patches should, strictly speaking, go via the block and
> network subsystem. Although, they heavily depends on other patches in
> this series. Would it be fine to send all through xentip?

Yes, if they've been acked by the appropriate maintainer.

David

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-29 Thread Julien Grall
Hi David,

On 29/09/15 17:27, David Vrabel wrote:
> On 07/09/15 16:33, Julien Grall wrote:
>>
>> A branch based on the latest xentip/for-linus-4.3 can be found here:
>>
>> git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4
> 
> This has too many conflicts with the
> git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.4
> branch.
> 
> Please rebase.

I was about to send a new version rebased (no changes in it so far
except for rebasing purpose). I will do it on top of for-linus-4.4.
I will also include the swiotlb changes at the end as it was reviewed by
Stefano.

BTW, some of the patches should, strictly speaking, go via the block and
network subsystem. Although, they heavily depends on other patches in
this series. Would it be fine to send all through xentip?

Regards,

-- 
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-29 Thread David Vrabel
On 07/09/15 16:33, Julien Grall wrote:
> 
> A branch based on the latest xentip/for-linus-4.3 can be found here:
> 
> git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4

This has too many conflicts with the
git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.4
branch.

Please rebase.

David

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-22 Thread Ian Campbell
On Mon, 2015-09-14 at 17:23 +0200, Roger Pau Monné wrote:
> I'm not saying that we shouldn't take those patches, I'm just saying
> that IMHO this is a workaround, and I would like to see a plan and
> somebody committed to have it fixed in a proper way, by introducing a
> 64KB PV block protocol.

In my view the basic unit of operation for all Xen interfaces (on x86 and
arm at least[0]) is 4K. The peer at either end of a PV protocol should
therefore be entitled to assume that at a minimum the other end supports
operation in 4K mode (i.e. 4K is the baseline).

Operations in larger sizes (which would necessarily be multiples of 4K) are
then an extension which is subject to a negotiation between the two ends,
it doesn't really matter if those larger sizes arise because of the use of
superpages or because the guest is internally using some larger basic page
size (which ARM calls a "granule", and where 64K comes from here).

I think this line of reasoning applies just as strongly to the hypercall
ABI as well BTW, they all use 4K as their basic unit, but might support
extensions to operation on multiples of that (negotiated either via a
specific error return and fallback or via the use of XENFEAT).

Ian.

[0] It's not implausible that some other architecture may exsit which
doesn't use 4K as the basic page szie)

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-18 Thread Julien Grall
Hi Roger,

On 14/09/15 13:08, Roger Pau Monné wrote:
> Well, absolute numbers together with the standard deviation are IMHO the
> best way to provide those figures (ie: see ministat(1) output for
> example), but percentages should also be fine.
> 
> I'm just interested in knowing the performance difference between having
> this patches applied or not when using 4KB pages on the frontend and the
> backend.

I did some benchmark:
DOM0: 1 VCPU, 4G of RAM based on 4.3-rc1 without this series
GUEST: 4 VPUs, 4G of RAM, second disk associate to a nullbk device

I used fio with the following options

42sh> fio --name=test --ioengine=libaio --rw=read --numjobs=8 \
--iodepth=32 --time_based=1 --runtime=30 --bs=4KB \
--filename=/dev/xvdb--direct=1 --group_reporting=1 --iodepth_batch=16

The guest is also based on 4.3-rc1 with and without the series.

The overhead with my series is about 0.56%.

Regards,

-- 
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-15 Thread Arnd Bergmann
On Tuesday 15 September 2015 14:14:09 David Vrabel wrote:
> On 14/09/15 12:32, Arnd Bergmann wrote:
> > On Monday 14 September 2015 13:04:59 Roger Pau Monné wrote:
> >>> TBH, I'm expecting a small impact to the performance. It would be hard
> >>> to get the exactly the same performance as today if we keep the helpers
> >>> to avoid the backend dealing himself with the splitting and page
> >>> granularity.
> >>>
> >>> Although, if the performance impact is not acceptable, it may be
> >>> possible to optimize gnttab_foreach_grant_in_range by moving the
> >>> function inline. The current way to the loop is the fastest I've found
> >>> (I've wrote a small program to test different way) and we will need it
> >>> when different of size will be supported.
> >>
> >> I don't expect the performance to drop massively with this patches
> >> applied, but it would be good to al least have an idea of the impact.
> > 
> > Note that using 64kb pages in Linux tends to destroy performance
> > in Linux in any case, as the memory consumption for most workloads
> > explodes. In a virtualized environment you already tend to be
> > memory constrained, so any measurement should take that into account
> > and put the extra overhead into perspective to the massive overhead
> > of running 64kb pages when RAM is tight.
> 
> If this is the case, why are some distros using 64 KiB pages then?

I believe that IBM started this on PowerPC64 when they found that it
performs better in microbenchmarks and for certain database workloads
when you do have enough memory and do not work with small files or
many processes.

I also believe that the AIX memory management code is designed differently
enough that it suffers less from large pages (or suffers more from
small pages), so the hardware designers pushed Linux to do the same as AIX.

Why you'd do it on ARM64 I have no idea at all, other than copying what
IBM did on PowerPC. If your TLB is too small, hardware designers may
of course try to push the OS to using pages as large as possible for the
low-level benchmarks to suck less, but in general, you'd be better off
with transparent hugepages and 4kb or 8kb pages for real workloads.

Arnd

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-15 Thread David Vrabel
On 14/09/15 12:32, Arnd Bergmann wrote:
> On Monday 14 September 2015 13:04:59 Roger Pau Monné wrote:
>>> TBH, I'm expecting a small impact to the performance. It would be hard
>>> to get the exactly the same performance as today if we keep the helpers
>>> to avoid the backend dealing himself with the splitting and page
>>> granularity.
>>>
>>> Although, if the performance impact is not acceptable, it may be
>>> possible to optimize gnttab_foreach_grant_in_range by moving the
>>> function inline. The current way to the loop is the fastest I've found
>>> (I've wrote a small program to test different way) and we will need it
>>> when different of size will be supported.
>>
>> I don't expect the performance to drop massively with this patches
>> applied, but it would be good to al least have an idea of the impact.
> 
> Note that using 64kb pages in Linux tends to destroy performance
> in Linux in any case, as the memory consumption for most workloads
> explodes. In a virtualized environment you already tend to be
> memory constrained, so any measurement should take that into account
> and put the extra overhead into perspective to the massive overhead
> of running 64kb pages when RAM is tight.

If this is the case, why are some distros using 64 KiB pages then?

David

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-14 Thread Roger Pau Monné
El 14/09/15 a les 16.54, Stefano Stabellini ha escrit:
> On Mon, 14 Sep 2015, Roger Pau Monné wrote:
>> IMHO this splitting is just a workaround for the fact that we don't have
>> a 64KB PV block protocol, and this is the real problem that should be
>> solved.
> 
> 64K is a pure one guest kernel configuration option, not a platform wide
> option. The hypervisor interfaces are still the same, the ABI is the
> same and all the other guests are still the same, the Xen binary is
> still the same.

Yes, I understand that, but the PV block protocol is missing 64KB page
support, and that's a fact that cannot be ignored. To put an example, is
there a hardware SATA controller on ARM that doesn't support 64KB pages
and needs a similar workaround?

> A 64K block protocol could be a good performance imprevement, but should
> not be required to run kernels which have different config options.
> 
>> In the long term this will put a burden on all blkfronts (if 64KB pages
>> are also used by other OSes), while introducing a 64KB PV block protocol
>> will make the blkfront implementation in all OSes very similar to what
>> we have now, without replicating the splitting code amongst all the
>> possible blkfront implementations.
>>
>> Granted that some changes to blkback will be needed in order to support
>> mapping 64KB grants, but there are much fewer blkback implementations
>> out there than blkfronts.
> 
> I don't think we can rely on blkback having something in order to run
> new guests, otherwise we break compatibility: new guests won't run on
> old hypervisors.

I agree that this is far from ideal, but I don't think it's so
outrageous. For example Linux PVOPS Dom0 kernels require Xen 4.0.1 at
least in order to run, because previous versions lack the necessary
IOAPIC setup hypercall. Also, it won't prevent guests from booting, it
would just prevent them from using blkfront, but you can still get a
root filesystem using iSCSI, NFS or other means.

I'm not saying that we shouldn't take those patches, I'm just saying
that IMHO this is a workaround, and I would like to see a plan and
somebody committed to have it fixed in a proper way, by introducing a
64KB PV block protocol.

Roger.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-14 Thread Stefano Stabellini
On Mon, 14 Sep 2015, Roger Pau Monné wrote:
> IMHO this splitting is just a workaround for the fact that we don't have
> a 64KB PV block protocol, and this is the real problem that should be
> solved.

64K is a pure one guest kernel configuration option, not a platform wide
option. The hypervisor interfaces are still the same, the ABI is the
same and all the other guests are still the same, the Xen binary is
still the same.

A 64K block protocol could be a good performance imprevement, but should
not be required to run kernels which have different config options.


> In the long term this will put a burden on all blkfronts (if 64KB pages
> are also used by other OSes), while introducing a 64KB PV block protocol
> will make the blkfront implementation in all OSes very similar to what
> we have now, without replicating the splitting code amongst all the
> possible blkfront implementations.
>
> Granted that some changes to blkback will be needed in order to support
> mapping 64KB grants, but there are much fewer blkback implementations
> out there than blkfronts.

I don't think we can rely on blkback having something in order to run
new guests, otherwise we break compatibility: new guests won't run on
old hypervisors.___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-14 Thread Julien Grall
On 14/09/15 15:29, Roger Pau Monné wrote:
>> To give you an example, Centos 7, which will support Xen and only 64KB
>> page granularity, will be supported for years. Dropping any splitting in
>> a short future (3-5 years) will just break those guests to boot on Xen.
> 
> Can't the patches to support 64KB grants be backported to CentOS? In the
> past you said that distros had no problem in picking the needed Xen
> patches for 64KB support, I don't see how that would be different.

Centos 7 is not yet released so it's more easy to get patch in. I don't
know what will be position when the guest is shipped.

I bet they will still want to support 64KB with older Xen (i.e where
64KB grant is not supported).

Anyway, I don't expect to see the 64KB grants support before next year
as my TODO list is already pretty full. So let's talk about it when I
(or someone else) will send an RFC to extend the grant size.

Regards,

-- 
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-14 Thread Roger Pau Monné
Hello,

El 14/09/15 a les 14.47, Julien Grall ha escrit:
> On 14/09/15 13:08, Roger Pau Monné wrote:
>>> I'd like to see a basic support of 64KB page granularity upstream before
>>> starting to think about performance improvement. And there is still a
>>> lot to do.
>>
>> I wasn't actually thinking of this as a performance improvement, but
>> rather as a way of fixing this properly and removing the complexity
>> added to xen-blkfront, thus providing a PV block protocol that natively
>> supports 64KB pages. This would be like hardware components already do
>> AFAIK, because I don't think there are other drivers in the Linux tree
>> that do this kind of splitting.
>>
>>> Although, having 64KB grants won't remove the splitting as we would
>>> still have to support frontend/backend which only handle 4KB grant.
>>
>> IMHO, given enough time I would just drop those.
> 
> The PV protocol has always been backward compatible.
> 
> Introducing the support of 64KB grant will require some changes in Xen
> which won't surely be backported to older version. We still have to be
> able to run new Linux guest on older Xen and vice versa.

OK, so let's wait a couple of Xen and Linux releases before dropping
this from the Linux kernel once 64KB grant support is added. Guests
should still be able to boot, it's just the disk that won't be attached.

IMHO this splitting is just a workaround for the fact that we don't have
a 64KB PV block protocol, and this is the real problem that should be
solved.

In the long term this will put a burden on all blkfronts (if 64KB pages
are also used by other OSes), while introducing a 64KB PV block protocol
will make the blkfront implementation in all OSes very similar to what
we have now, without replicating the splitting code amongst all the
possible blkfront implementations.

Granted that some changes to blkback will be needed in order to support
mapping 64KB grants, but there are much fewer blkback implementations
out there than blkfronts.

> To give you an example, Centos 7, which will support Xen and only 64KB
> page granularity, will be supported for years. Dropping any splitting in
> a short future (3-5 years) will just break those guests to boot on Xen.

Can't the patches to support 64KB grants be backported to CentOS? In the
past you said that distros had no problem in picking the needed Xen
patches for 64KB support, I don't see how that would be different.

Also, a blkfront with the splitting code should also be able to work
perfectly fine once we have a 64KB PV block protocol, it will just be
treated like it's using 4KB pages, that's all.

> AFAIK, we never took a such radical decision on Xen based on the
> complexity of the code.

We never had to deal with a problem like this I'm afraid (mixing guests
with different page sizes), so I don't think we can make statements
about that.

And it's not just about the complexity, it's because as stated above
this IMHO is just sweeping the real problem under the carpet instead of
solving it.

Roger.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-14 Thread Julien Grall
On 14/09/15 13:08, Roger Pau Monné wrote:
>> I'd like to see a basic support of 64KB page granularity upstream before
>> starting to think about performance improvement. And there is still a
>> lot to do.
> 
> I wasn't actually thinking of this as a performance improvement, but
> rather as a way of fixing this properly and removing the complexity
> added to xen-blkfront, thus providing a PV block protocol that natively
> supports 64KB pages. This would be like hardware components already do
> AFAIK, because I don't think there are other drivers in the Linux tree
> that do this kind of splitting.
> 
>> Although, having 64KB grants won't remove the splitting as we would
>> still have to support frontend/backend which only handle 4KB grant.
> 
> IMHO, given enough time I would just drop those.

The PV protocol has always been backward compatible.

Introducing the support of 64KB grant will require some changes in Xen
which won't surely be backported to older version. We still have to be
able to run new Linux guest on older Xen and vice versa.

To give you an example, Centos 7, which will support Xen and only 64KB
page granularity, will be supported for years. Dropping any splitting in
a short future (3-5 years) will just break those guests to boot on Xen.

AFAIK, we never took a such radical decision on Xen based on the
complexity of the code.

Regards,

-- 
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-14 Thread Roger Pau Monné
El 14/09/15 a les 13.21, Julien Grall ha escrit:
> On 14/09/15 12:04, Roger Pau Monné wrote:
>> Hello,
>>
>> El 14/09/15 a les 12.40, Julien Grall ha escrit:
>>> Hi Roger,
>>>
>>> On 14/09/15 09:56, Roger Pau Monné wrote:
 El 07/09/15 a les 17.33, Julien Grall ha escrit:
> Hi all,
>
> ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, 
> Xen
> hypercall interface and PV protocol are always based on 4KB page 
> granularity.
>
> Any attempt to boot a Linux guest with 64KB pages enabled will result to a
> guest crash.
>
> This series is a first attempt to allow those Linux running with the 
> current
> hypercall interface and PV protocol.
>
> This solution has been chosen because we want to run Linux 64KB in 
> released
> Xen ARM version or/and platform using an old version of Linux DOM0.
>
> There is room for improvement, such as support of 64KB grant, modification
> of PV protocol to support different page size... They will be explored in 
> a
> separate patch series later.
>
> TODO list:
> - Convert swiotlb to 64KB
> - Convert xenfb to 64KB
> - Support for multiple page ring support
> - Support for 64KB in gnttdev
> - Support of non-indirect grant with 64KB frontend
> - It may be possible to move some common define between
> netback/netfront and blkfront/blkback in an header
>
> I've got most of the patches for the TODO items. I'm planning to send 
> them as
> a follow-up as it's not a requirement for a basic guests.
>
> All patches has been built tested for ARM32, ARM64, x86. But I haven't 
> tested
> to run it on x86 as I don't have a box with Xen x86 running. I would be
> happy if someone give a try and see possible regression for x86.

 Do you have figures regarding if/how much performance penalty do the
 blkfront/blkback patches add to the traditional 4KB scenario (frontend
 and backend running on guests using 4KB pages)?
>>>
>>> Which benchmark do you advice? Although, I don't have SSD on this
>>> platform so it may be possible that the bottleneck will be the hard drive.
>>
>> I've normally used a ramdisk in order to test performance, but it seems
>> nullblk would be a better option (it wasn't available before) and it
>> doesn't reduce the size of RAM available to the system:
>>
>> https://www.kernel.org/doc/Documentation/block/null_blk.txt
> 
> I will give a look to this.
> 

 Since there's been no design document about this and the TODO list
 doesn't contain it, I would like to know which plans do we have in order
 to fix this properly.
>>>
>>> Can you explain what kind of design document you were expecting? The
>>> support of 64KB page granularity is pretty straight-forward and there is
>>> not many way to do it. We have to split the page in 4KB chunk to feed
>>> the ring.
>>
>> I was thinking about adding support for 64KB grants with the aim of
>> eventually removing the splitting done in xen-blkfront and the
>> grant-table code in general.
> 
> I'd like to see a basic support of 64KB page granularity upstream before
> starting to think about performance improvement. And there is still a
> lot to do.

I wasn't actually thinking of this as a performance improvement, but
rather as a way of fixing this properly and removing the complexity
added to xen-blkfront, thus providing a PV block protocol that natively
supports 64KB pages. This would be like hardware components already do
AFAIK, because I don't think there are other drivers in the Linux tree
that do this kind of splitting.

> Although, having 64KB grants won't remove the splitting as we would
> still have to support frontend/backend which only handle 4KB grant.

IMHO, given enough time I would just drop those.

>>> TBH, I'm expecting a small impact to the performance. It would be hard
>>> to get the exactly the same performance as today if we keep the helpers
>>> to avoid the backend dealing himself with the splitting and page
>>> granularity.
>>>
>>> Although, if the performance impact is not acceptable, it may be
>>> possible to optimize gnttab_foreach_grant_in_range by moving the
>>> function inline. The current way to the loop is the fastest I've found
>>> (I've wrote a small program to test different way) and we will need it
>>> when different of size will be supported.
>>
>> I don't expect the performance to drop massively with this patches
>> applied, but it would be good to al least have an idea of the impact.
> 
> I will only be able to give percentage. I guess this would be sufficient
> for you?

Well, absolute numbers together with the standard deviation are IMHO the
best way to provide those figures (ie: see ministat(1) output for
example), but percentages should also be fine.

I'm just interested in knowing the performance difference between having
this patches applied or not when using 4K

Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-14 Thread Arnd Bergmann
On Monday 14 September 2015 13:04:59 Roger Pau Monné wrote:
> > TBH, I'm expecting a small impact to the performance. It would be hard
> > to get the exactly the same performance as today if we keep the helpers
> > to avoid the backend dealing himself with the splitting and page
> > granularity.
> > 
> > Although, if the performance impact is not acceptable, it may be
> > possible to optimize gnttab_foreach_grant_in_range by moving the
> > function inline. The current way to the loop is the fastest I've found
> > (I've wrote a small program to test different way) and we will need it
> > when different of size will be supported.
> 
> I don't expect the performance to drop massively with this patches
> applied, but it would be good to al least have an idea of the impact.

Note that using 64kb pages in Linux tends to destroy performance
in Linux in any case, as the memory consumption for most workloads
explodes. In a virtualized environment you already tend to be
memory constrained, so any measurement should take that into account
and put the extra overhead into perspective to the massive overhead
of running 64kb pages when RAM is tight.

Arnd

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-14 Thread Julien Grall
On 14/09/15 12:04, Roger Pau Monné wrote:
> Hello,
> 
> El 14/09/15 a les 12.40, Julien Grall ha escrit:
>> Hi Roger,
>>
>> On 14/09/15 09:56, Roger Pau Monné wrote:
>>> El 07/09/15 a les 17.33, Julien Grall ha escrit:
 Hi all,

 ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen
 hypercall interface and PV protocol are always based on 4KB page 
 granularity.

 Any attempt to boot a Linux guest with 64KB pages enabled will result to a
 guest crash.

 This series is a first attempt to allow those Linux running with the 
 current
 hypercall interface and PV protocol.

 This solution has been chosen because we want to run Linux 64KB in released
 Xen ARM version or/and platform using an old version of Linux DOM0.

 There is room for improvement, such as support of 64KB grant, modification
 of PV protocol to support different page size... They will be explored in a
 separate patch series later.

 TODO list:
 - Convert swiotlb to 64KB
 - Convert xenfb to 64KB
 - Support for multiple page ring support
 - Support for 64KB in gnttdev
 - Support of non-indirect grant with 64KB frontend
 - It may be possible to move some common define between
 netback/netfront and blkfront/blkback in an header

 I've got most of the patches for the TODO items. I'm planning to send them 
 as
 a follow-up as it's not a requirement for a basic guests.

 All patches has been built tested for ARM32, ARM64, x86. But I haven't 
 tested
 to run it on x86 as I don't have a box with Xen x86 running. I would be
 happy if someone give a try and see possible regression for x86.
>>>
>>> Do you have figures regarding if/how much performance penalty do the
>>> blkfront/blkback patches add to the traditional 4KB scenario (frontend
>>> and backend running on guests using 4KB pages)?
>>
>> Which benchmark do you advice? Although, I don't have SSD on this
>> platform so it may be possible that the bottleneck will be the hard drive.
> 
> I've normally used a ramdisk in order to test performance, but it seems
> nullblk would be a better option (it wasn't available before) and it
> doesn't reduce the size of RAM available to the system:
> 
> https://www.kernel.org/doc/Documentation/block/null_blk.txt

I will give a look to this.

>>>
>>> Since there's been no design document about this and the TODO list
>>> doesn't contain it, I would like to know which plans do we have in order
>>> to fix this properly.
>>
>> Can you explain what kind of design document you were expecting? The
>> support of 64KB page granularity is pretty straight-forward and there is
>> not many way to do it. We have to split the page in 4KB chunk to feed
>> the ring.
> 
> I was thinking about adding support for 64KB grants with the aim of
> eventually removing the splitting done in xen-blkfront and the
> grant-table code in general.

I'd like to see a basic support of 64KB page granularity upstream before
starting to think about performance improvement. And there is still a
lot to do.

Although, having 64KB grants won't remove the splitting as we would
still have to support frontend/backend which only handle 4KB grant.


>> TBH, I'm expecting a small impact to the performance. It would be hard
>> to get the exactly the same performance as today if we keep the helpers
>> to avoid the backend dealing himself with the splitting and page
>> granularity.
>>
>> Although, if the performance impact is not acceptable, it may be
>> possible to optimize gnttab_foreach_grant_in_range by moving the
>> function inline. The current way to the loop is the fastest I've found
>> (I've wrote a small program to test different way) and we will need it
>> when different of size will be supported.
> 
> I don't expect the performance to drop massively with this patches
> applied, but it would be good to al least have an idea of the impact.

I will only be able to give percentage. I guess this would be sufficient
for you?

Regards,

-- 
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-14 Thread Roger Pau Monné
Hello,

El 14/09/15 a les 12.40, Julien Grall ha escrit:
> Hi Roger,
> 
> On 14/09/15 09:56, Roger Pau Monné wrote:
>> El 07/09/15 a les 17.33, Julien Grall ha escrit:
>>> Hi all,
>>>
>>> ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen
>>> hypercall interface and PV protocol are always based on 4KB page 
>>> granularity.
>>>
>>> Any attempt to boot a Linux guest with 64KB pages enabled will result to a
>>> guest crash.
>>>
>>> This series is a first attempt to allow those Linux running with the current
>>> hypercall interface and PV protocol.
>>>
>>> This solution has been chosen because we want to run Linux 64KB in released
>>> Xen ARM version or/and platform using an old version of Linux DOM0.
>>>
>>> There is room for improvement, such as support of 64KB grant, modification
>>> of PV protocol to support different page size... They will be explored in a
>>> separate patch series later.
>>>
>>> TODO list:
>>> - Convert swiotlb to 64KB
>>> - Convert xenfb to 64KB
>>> - Support for multiple page ring support
>>> - Support for 64KB in gnttdev
>>> - Support of non-indirect grant with 64KB frontend
>>> - It may be possible to move some common define between
>>> netback/netfront and blkfront/blkback in an header
>>>
>>> I've got most of the patches for the TODO items. I'm planning to send them 
>>> as
>>> a follow-up as it's not a requirement for a basic guests.
>>>
>>> All patches has been built tested for ARM32, ARM64, x86. But I haven't 
>>> tested
>>> to run it on x86 as I don't have a box with Xen x86 running. I would be
>>> happy if someone give a try and see possible regression for x86.
>>
>> Do you have figures regarding if/how much performance penalty do the
>> blkfront/blkback patches add to the traditional 4KB scenario (frontend
>> and backend running on guests using 4KB pages)?
> 
> Which benchmark do you advice? Although, I don't have SSD on this
> platform so it may be possible that the bottleneck will be the hard drive.

I've normally used a ramdisk in order to test performance, but it seems
nullblk would be a better option (it wasn't available before) and it
doesn't reduce the size of RAM available to the system:

https://www.kernel.org/doc/Documentation/block/null_blk.txt

>>
>> Since there's been no design document about this and the TODO list
>> doesn't contain it, I would like to know which plans do we have in order
>> to fix this properly.
> 
> Can you explain what kind of design document you were expecting? The
> support of 64KB page granularity is pretty straight-forward and there is
> not many way to do it. We have to split the page in 4KB chunk to feed
> the ring.

I was thinking about adding support for 64KB grants with the aim of
eventually removing the splitting done in xen-blkfront and the
grant-table code in general.

Some modifications would be needed in xen-blkback in order to support
mapping 64KB grants, but this could also be used by x86 if we ever
enable 2MB grants and allow them to be used in the block PV protocol,
while the current code can only be used by ARM.

> TBH, I'm expecting a small impact to the performance. It would be hard
> to get the exactly the same performance as today if we keep the helpers
> to avoid the backend dealing himself with the splitting and page
> granularity.
> 
> Although, if the performance impact is not acceptable, it may be
> possible to optimize gnttab_foreach_grant_in_range by moving the
> function inline. The current way to the loop is the fastest I've found
> (I've wrote a small program to test different way) and we will need it
> when different of size will be supported.

I don't expect the performance to drop massively with this patches
applied, but it would be good to al least have an idea of the impact.

Roger.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-14 Thread Julien Grall
Hi Roger,

On 14/09/15 09:56, Roger Pau Monné wrote:
> El 07/09/15 a les 17.33, Julien Grall ha escrit:
>> Hi all,
>>
>> ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen
>> hypercall interface and PV protocol are always based on 4KB page granularity.
>>
>> Any attempt to boot a Linux guest with 64KB pages enabled will result to a
>> guest crash.
>>
>> This series is a first attempt to allow those Linux running with the current
>> hypercall interface and PV protocol.
>>
>> This solution has been chosen because we want to run Linux 64KB in released
>> Xen ARM version or/and platform using an old version of Linux DOM0.
>>
>> There is room for improvement, such as support of 64KB grant, modification
>> of PV protocol to support different page size... They will be explored in a
>> separate patch series later.
>>
>> TODO list:
>> - Convert swiotlb to 64KB
>> - Convert xenfb to 64KB
>> - Support for multiple page ring support
>> - Support for 64KB in gnttdev
>> - Support of non-indirect grant with 64KB frontend
>> - It may be possible to move some common define between
>> netback/netfront and blkfront/blkback in an header
>>
>> I've got most of the patches for the TODO items. I'm planning to send them as
>> a follow-up as it's not a requirement for a basic guests.
>>
>> All patches has been built tested for ARM32, ARM64, x86. But I haven't tested
>> to run it on x86 as I don't have a box with Xen x86 running. I would be
>> happy if someone give a try and see possible regression for x86.
> 
> Do you have figures regarding if/how much performance penalty do the
> blkfront/blkback patches add to the traditional 4KB scenario (frontend
> and backend running on guests using 4KB pages)?

Which benchmark do you advice? Although, I don't have SSD on this
platform so it may be possible that the bottleneck will be the hard drive.

> 
> Since there's been no design document about this and the TODO list
> doesn't contain it, I would like to know which plans do we have in order
> to fix this properly.

Can you explain what kind of design document you were expecting? The
support of 64KB page granularity is pretty straight-forward and there is
not many way to do it. We have to split the page in 4KB chunk to feed
the ring.

TBH, I'm expecting a small impact to the performance. It would be hard
to get the exactly the same performance as today if we keep the helpers
to avoid the backend dealing himself with the splitting and page
granularity.

Although, if the performance impact is not acceptable, it may be
possible to optimize gnttab_foreach_grant_in_range by moving the
function inline. The current way to the loop is the fastest I've found
(I've wrote a small program to test different way) and we will need it
when different of size will be supported.

Regards,

-- 
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-14 Thread Roger Pau Monné
El 07/09/15 a les 17.33, Julien Grall ha escrit:
> Hi all,
> 
> ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen
> hypercall interface and PV protocol are always based on 4KB page granularity.
> 
> Any attempt to boot a Linux guest with 64KB pages enabled will result to a
> guest crash.
> 
> This series is a first attempt to allow those Linux running with the current
> hypercall interface and PV protocol.
> 
> This solution has been chosen because we want to run Linux 64KB in released
> Xen ARM version or/and platform using an old version of Linux DOM0.
> 
> There is room for improvement, such as support of 64KB grant, modification
> of PV protocol to support different page size... They will be explored in a
> separate patch series later.
> 
> TODO list:
> - Convert swiotlb to 64KB
> - Convert xenfb to 64KB
> - Support for multiple page ring support
> - Support for 64KB in gnttdev
> - Support of non-indirect grant with 64KB frontend
> - It may be possible to move some common define between
> netback/netfront and blkfront/blkback in an header
> 
> I've got most of the patches for the TODO items. I'm planning to send them as
> a follow-up as it's not a requirement for a basic guests.
> 
> All patches has been built tested for ARM32, ARM64, x86. But I haven't tested
> to run it on x86 as I don't have a box with Xen x86 running. I would be
> happy if someone give a try and see possible regression for x86.

Do you have figures regarding if/how much performance penalty do the
blkfront/blkback patches add to the traditional 4KB scenario (frontend
and backend running on guests using 4KB pages)?

Since there's been no design document about this and the TODO list
doesn't contain it, I would like to know which plans do we have in order
to fix this properly.

Roger.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-11 Thread Julien Grall
Hi,

A quick update on the TODO.

On 07/09/15 16:33, Julien Grall wrote:
> ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen
> hypercall interface and PV protocol are always based on 4KB page granularity.
> 
> Any attempt to boot a Linux guest with 64KB pages enabled will result to a
> guest crash.
> 
> This series is a first attempt to allow those Linux running with the current
> hypercall interface and PV protocol.
> 
> This solution has been chosen because we want to run Linux 64KB in released
> Xen ARM version or/and platform using an old version of Linux DOM0.
> 
> There is room for improvement, such as support of 64KB grant, modification
> of PV protocol to support different page size... They will be explored in a
> separate patch series later.
> 
> TODO list:
> - Convert swiotlb to 64KB

Sent http://lists.xen.org/archives/html/xen-devel/2015-09/msg01292.html

> - Convert xenfb to 64KB
> - Support for multiple page ring support
> - Support for 64KB in gnttdev
> - Support of non-indirect grant with 64KB frontend

Sent http://lists.xen.org/archives/html/xen-devel/2015-09/msg01577.html

> - It may be possible to move some common define between
> netback/netfront and blkfront/blkback in an header
> 
> I've got most of the patches for the TODO items. I'm planning to send them as
> a follow-up as it's not a requirement for a basic guests.
> 
> All patches has been built tested for ARM32, ARM64, x86. But I haven't tested
> to run it on x86 as I don't have a box with Xen x86 running. I would be
> happy if someone give a try and see possible regression for x86.
> 
> I know that Konrad as a test-suite for x86. Konrand, would it be possible to
> give a run to for this series?
> 
> A branch based on the latest xentip/for-linus-4.3 can be found here:
> 
> git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4

I will resend a new version when 4.3-rc1 is out in order to fix any
possible conflict with linux/master. I already know that the netback
patch (#18) is conflicting with 1d5d48523900a4b0f25d6b52f1a93c84bd671186
"xen-netback: require fewer guest Rx slots when not using GSO".

If the 2 series above are completely acked/reviewed, I will fold them in
this series.

Regards,

-- 
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


[Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

2015-09-07 Thread Julien Grall
Hi all,

ARM64 Linux is supporting both 4KB and 64KB page granularity. Although, Xen
hypercall interface and PV protocol are always based on 4KB page granularity.

Any attempt to boot a Linux guest with 64KB pages enabled will result to a
guest crash.

This series is a first attempt to allow those Linux running with the current
hypercall interface and PV protocol.

This solution has been chosen because we want to run Linux 64KB in released
Xen ARM version or/and platform using an old version of Linux DOM0.

There is room for improvement, such as support of 64KB grant, modification
of PV protocol to support different page size... They will be explored in a
separate patch series later.

TODO list:
- Convert swiotlb to 64KB
- Convert xenfb to 64KB
- Support for multiple page ring support
- Support for 64KB in gnttdev
- Support of non-indirect grant with 64KB frontend
- It may be possible to move some common define between
netback/netfront and blkfront/blkback in an header

I've got most of the patches for the TODO items. I'm planning to send them as
a follow-up as it's not a requirement for a basic guests.

All patches has been built tested for ARM32, ARM64, x86. But I haven't tested
to run it on x86 as I don't have a box with Xen x86 running. I would be
happy if someone give a try and see possible regression for x86.

I know that Konrad as a test-suite for x86. Konrand, would it be possible to
give a run to for this series?

A branch based on the latest xentip/for-linus-4.3 can be found here:

git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4

Comments, suggestions are welcomed.

Sincerely yours,

Cc: david.vra...@citrix.com
Cc: konrad.w...@oracle.com
Cc: boris.ostrov...@oracle.com
Cc: wei.l...@citrix.com
Cc: roger@citrix.com

Status of each patch:

A: Reviewed-by - Acked-by
M: Patch modified in this series
m: Minor changes in this series (i.e renaming due to previous patches, typoes)
L: Missing Acked-by from a Linux maintainers (Boris, David, Konrad)
N: Missing Acked-by from a Netback maintainers (Ian or Wei)

Julien Grall (20):
A   net/xen-netback: xenvif_gop_frag_copy: move GSO check out of the loop
A   arm/xen: Drop pte_mfn and mfn_pte
A M L   xen: Add Xen specific page definition
A M xen/grant: Introduce helpers to split a page into grant
A   xen/grant: Add helper gnttab_page_grant_foreign_access_ref_one
A   block/xen-blkfront: Split blkif_queue_request in 2
A m block/xen-blkfront: Store a page rather a pfn in the grant structure
A   block/xen-blkfront: split get_grant in 2
A m L   xen/biomerge: Don't allow biovec's to be merged when Linux is not
  using 4KB pages
A   xen/xenbus: Use Xen page definition
A m L   tty/hvc: xen: Use xen page definition
  M L   xen/balloon: Don't rely on the page granularity is the same for Xen
  and Linux
A   xen/events: fifo: Make it running on 64KB granularity
A   xen/grant-table: Make it running on 64KB granularity
A m block/xen-blkfront: Make it running on 64KB page granularity
A   block/xen-blkback: Make it running on 64KB page granularity
A m net/xen-netfront: Make it running on 64KB page granularity
  m  N  net/xen-netback: Make it running on 64KB page granularity
A m xen/privcmd: Add support for Linux 64KB page granularity
A   arm/xen: Add support for 64KB page granularity

 arch/arm/include/asm/xen/page.h |  18 +-
 arch/arm/xen/enlighten.c|   6 +-
 arch/arm/xen/p2m.c  |   6 +-
 arch/x86/include/asm/xen/page.h |   2 +-
 drivers/block/xen-blkback/blkback.c |   5 +-
 drivers/block/xen-blkback/common.h  |  17 +-
 drivers/block/xen-blkback/xenbus.c  |   9 +-
 drivers/block/xen-blkfront.c| 552 +++-
 drivers/net/xen-netback/common.h|  18 +-
 drivers/net/xen-netback/netback.c   | 163 +++
 drivers/net/xen-netfront.c  | 122 +---
 drivers/tty/hvc/hvc_xen.c   |   4 +-
 drivers/xen/balloon.c   |  59 +++-
 drivers/xen/biomerge.c  |   8 +
 drivers/xen/events/events_base.c|   2 +-
 drivers/xen/events/events_fifo.c|   2 +-
 drivers/xen/grant-table.c   |  32 ++-
 drivers/xen/privcmd.c   |   8 +-
 drivers/xen/xenbus/xenbus_client.c  |   6 +-
 drivers/xen/xenbus/xenbus_probe.c   |   3 +-
 drivers/xen/xlate_mmu.c | 124 +---
 include/xen/grant_table.h   |  51 
 include/xen/page.h  |  27 +-
 23 files changed, 855 insertions(+), 389 deletions(-)

-- 
2.1.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel