Shared caches in multi-core CPU architectures represent a problem for
predictability of memory access latency. This jeopardizes applicability
of many Arm platform in real-time critical and mixed-criticality
scenarios. We introduce support for cache partitioning with page
coloring, a transparent software technique that enables isolation
between domains and Xen, and thus avoids cache interference.
When creating a domain, a simple syntax (e.g. `0-3` or `4-11`) allows
the user to define assignments of cache partitions ids, called colors,
where assigning different colors guarantees no mutual eviction on cache
will ever happen. This instructs the Xen memory allocator to provide
the i-th color assignee only with pages that maps to color i, i.e. that
are indexed in the i-th cache partition.
The proposed implementation supports the dom0less experimental feature.
The solution has been tested in several scenarios, including Xilinx Zynq
MPSoCs.
Overview of implementation and commits structure
------------------------------------------------
- Coloring support is added for dom0 and domU by defining the core
logic, as well as the hardware inspection functionalities used for
getting needed coloring information [4-17].
- A new memory page allocator that implement the cache coloring
mechanism is introduced. The allocation algorithm follows the given
coloring scheme specified for each domain, and maximizes contiguity in
the page selection [18-21].
- Coloring support is added to Xen .text region [22-29], as well as to
dom0less domains [30].
- Extensive documentation details the technique and gently explains
usage [33-36].
Known limitations
-----------------
- We need to bring back [1-3] the relocation feature in order to move
Xen memory to a colored space where the hypervisor could be isolated
from VMs interference.
- When cache coloring is used, static memory assignment is disabled to
avoid incompatibility. [31]
- Due to assert failure [32], the number of supported colors is
currently limited at 64, which should be satisfactory for most chips.
In particular, the problem lies in the cache coloring configuration
data structure that belongs to each domain. We are aware that this is
not a clean solution but we hope that this could be discussed and
solved within this pull request.
Acknowledgements
----------------
This work is sponsored by Xilinx Inc., and supported by University of
Modena and Reggio Emilia and Minerva Systems.
***
Luca Miccio (36):
Revert "xen/arm: setup: Add Xen as boot module before printing all
boot modules"
Revert "xen/arm: mm: Initialize page-tables earlier"
xen/arm: restore xen_paddr argument in setup_pagetables
xen/arm: add parsing function for cache coloring configuration
xen/arm: compute LLC way size by hardware inspection
xen/arm: add coloring basic initialization
xen/arm: add coloring data to domains
xen/arm: add colored flag to page struct
xen/arch: add default colors selection function
xen/arch: check color selection function
xen/include: define hypercall parameter for coloring
xen/arm: initialize cache coloring data for Dom0/U
xen/arm: A domain is not direct mapped when coloring is enabled
xen/arch: add dump coloring info for domains
tools: add support for cache coloring configuration
xen/color alloc: implement color_from_page for ARM64
xen/arm: add get_max_color function
Alloc: introduce page_list_for_each_reverse
xen/arch: introduce cache-coloring allocator
xen/common: introduce buddy required reservation
xen/common: add colored allocator initialization
xen/arch: init cache coloring conf for Xen
xen/arch: coloring: manually calculate Xen physical addresses
xen/arm: enable consider_modules for coloring
xen/arm: bring back get_xen_paddr
xen/arm: add argument to remove_early_mappings
xen/arch: add coloring support for Xen
xen/arm: introduce xen_map_text_rw
xen/arm: add dump function for coloring info
xen/arm: add coloring support to dom0less
Disable coloring if static memory support is selected
xen/arm: reduce the number of supported colors
doc, xen-command-line: introduce coloring options
doc, xl.cfg: introduce coloring configuration option
doc, device-tree: introduce 'colors' property
doc, arm: add usage documentation for cache coloring support
docs/man/xl.cfg.5.pod.in | 14 +
docs/misc/arm/cache_coloring.rst | 191 +++++++++++
docs/misc/arm/device-tree/booting.txt | 3 +
docs/misc/xen-command-line.pandoc | 51 ++-
tools/libs/light/libxl_arm.c | 11 +
tools/libs/light/libxl_types.idl | 1 +
tools/xl/xl_parse.c | 59 +++-
xen/arch/arm/Kconfig | 6 +
xen/arch/arm/Makefile | 2 +-
xen/arch/arm/alternative.c | 8 +-
xen/arch/arm/coloring.c | 469 ++++++++++++++++++++++++++
xen/arch/arm/domain.c | 56 +++
xen/arch/arm/domain_build.c | 42 ++-
xen/arch/arm/include/asm/coloring.h | 98 ++++++
xen/arch/arm/include/asm/mm.h | 18 +-
xen/arch/arm/mm.c | 245 +++++++++++++-
xen/arch/arm/psci.c | 4 +-
xen/arch/arm/setup.c | 94 +++++-
xen/arch/arm/smpboot.c | 19 +-
xen/common/page_alloc.c | 321 +++++++++++++++++-
xen/common/vmap.c | 4 +-
xen/include/public/arch-arm.h | 8 +
xen/include/xen/mm.h | 7 +
xen/include/xen/sched.h | 4 +
xen/include/xen/vmap.h | 2 +
25 files changed, 1689 insertions(+), 48 deletions(-)
create mode 100644 docs/misc/arm/cache_coloring.rst
create mode 100644 xen/arch/arm/coloring.c
create mode 100644 xen/arch/arm/include/asm/coloring.h
--
2.30.2
