On 8/12/19 12:58 PM, Fabiano Rosas wrote: > Claudio Carvalho <cclau...@linux.ibm.com> writes: > > Some small suggestions below: > >> + >> + * The privilege of a process is now determined by three MSR bits, >> + MSR(S, HV, PR). In each of the tables below the modes are listed >> + from least privilege to highest privilege. The higher privilege >> + modes can access all the resources of the lower privilege modes. >> + >> + **Secure Mode MSR Settings** >> + >> + +---+---+---+---------------+ >> + | S | HV| PR|Privilege | >> + +===+===+===+===============+ >> + | 1 | 0 | 1 | Problem | >> + +---+---+---+---------------+ >> + | 1 | 0 | 0 | Privileged(OS)| >> + +---+---+---+---------------+ >> + | 1 | 1 | 0 | Ultravisor | >> + +---+---+---+---------------+ >> + | 1 | 1 | 1 | Reserved | >> + +---+---+---+---------------+ >> + >> + **Normal Mode MSR Settings** >> + >> + +---+---+---+---------------+ >> + | S | HV| PR|Privilege | >> + +===+===+===+===============+ >> + | 0 | 0 | 1 | Problem | >> + +---+---+---+---------------+ >> + | 0 | 0 | 0 | Privileged(OS)| >> + +---+---+---+---------------+ >> + | 0 | 1 | 0 | Hypervisor | >> + +---+---+---+---------------+ >> + | 0 | 1 | 1 | Problem (HV) | >> + +---+---+---+---------------+ > I find the use of '(HV)' in this last line a bit ambiguous. Since we are > already using 'HV' to refer to MSR(HV). I'd suggest using '(Host)' or > simply leaving it out. > >> + >> + * Memory is partitioned into secure and normal memory. Only processes >> + that are running in secure mode can access secure memory. >> + >> + * The hardware does not allow anything that is not running secure to >> + access secure memory. This means that the Hypervisor cannot access >> + the memory of the SVM without using an ultracall (asking the >> + Ultravisor). The Ultravisor will only allow the hypervisor to see >> + the SVM memory encrypted. >> + >> + * I/O systems are not allowed to directly address secure memory. This >> + limits the SVMs to virtual I/O only. >> + >> + * The architecture allows the SVM to share pages of memory with the >> + hypervisor that are not protected with encryption. However, this >> + sharing must be initiated by the SVM. >> + >> + * When a process is running in secure mode all hypercalls >> + (syscall lev=1) are reflected to the Ultravisor. > Here 'reflected' refers to the Ultravisor. Later on, it is used as > meaning the Ultravisor reflects hypercalls/interrupts to the > Hypervisor. I suggest we use this term to mean the latter only. > >> + >> + * When a process is in secure mode all interrupts go to the >> + Ultravisor. >> + >> + * The following resources have become Ultravisor privileged and >> + require an Ultravisor interface to manipulate: >> + >> + * Processor configurations registers (SCOMs). >> + >> + * Stop state information. >> + >> + * The debug registers CIABR, DAWR, and DAWRX become Ultravisor >> + resources when SMFCTRL(D) is set. If SMFCTRL(D) is not set they do > It looks like you could go without "become Ultravisor resources" since > it is already mentioned in the parent item above (The following...). > >> + not work in secure mode. When set, reading and writing requires >> + an Ultravisor call, otherwise that will cause a Hypervisor Emulation >> + Assistance interrupt. >> + >> + * PTCR and partition table entries (partition table is in secure >> + memory). An attempt to write to PTCR will cause a Hypervisor >> + Emulation Assitance interrupt. >> + >> + * LDBAR (LD Base Address Register) and IMC (In-Memory Collection) >> + non-architected registers. An attempt to write to them will cause a >> + Hypervisor Emulation Assistance interrupt. >> + >> + * Paging for an SVM, sharing of memory with Hypervisor for an SVM. >> + (Including Virtual Processor Area (VPA) and virtual I/O). >> + >> + >> +Software/Microcode >> +================== >> + >> + The software changes include: >> + >> + * SVMs are created from normal VM using (open source) tooling supplied >> + by IBM. >> + >> + * All SVMs start as normal VMs and utilize an ultracall, UV_ESM >> + (Enter Secure Mode), to make the transition. >> + >> + * When the UV_ESM ultracall is made the Ultravisor copies the VM into >> + secure memory, decrypts the verification information, and checks the >> + integrity of the SVM. If the integrity check passes the Ultravisor >> + passes control in secure mode. >> + >> + * For external interrupts the Ultravisor saves the state of the SVM, >> + and reflects the interrupt to the hypervisor for processing. >> + For hypercalls, the Ultravisor inserts neutral state into all >> + registers not needed for the hypercall then reflects the call to >> + the hypervisor for processing. The H_RANDOM hypercall is performed >> + by the Ultravisor and not reflected. >> + >> + * The verification information includes the pass phrase for the >> + encrypted disk associated with the SVM. This pass phrase is given >> + to the SVM when requested. > This is the second mention of 'verification information'. Could we > perhaps move this up after "SMVs are created..."
> and rephrase it in a way > that introduces the concept? What/where/in what state is the > verification information? Thanks for reviewing the patch Fabiano. I addressed all your suggestions in the next version, except the one above because the "verification information" concept is big and it is not related to the kernel. Claudio. >> + >> + * The Ultravisor is not involved in protecting the encrypted disk of >> + the SVM while at rest. >> + >> + * For virtual I/O to work bounce buffering must be done. >> + >> + * The Ultravisor uses AES (IAPM) for protection of SVM memory. IAPM >> + is a mode of AES that provides integrity and secrecy concurrently. >> + >> + * The movement of data between normal and secure pages is coordinated >> + with the Ultravisor by a new HMM plug-in in the Hypervisor. >> + >> + The Ultravisor offers new services to the hypervisor and SVMs. These >> + are accessed through ultracalls. >> + >> +Terminology >> +=========== >> + >> + * Hypercalls: special system calls used to request services from >> + Hypervisor. >> + >> + * Normal memory: Memory that is accessible to Hypervisor. >> + >> + * Normal page: Page backed by normal memory and available to >> + Hypervisor. >> + >> + * Shared page: A page backed by normal memory and available to both >> + the Hypervisor/QEMU and the SVM (i.e page has mappings in SVM and >> + Hypervisor/QEMU). >> + >> + * Secure memory: Memory that is accessible only to Ultravisor and >> + SVMs. >> + >> + * Secure page: Page backed by secure memory and only available to >> + Ultravisor and SVM. >> + >> + * SVM: Secure Virtual Machine. >> + >> + * Ultracalls: special system calls used to request services from >> + Ultravisor. >> + >> + >> +Ultravisor calls API >> +#################### >> + >> + This section describes Ultravisor calls (ultracalls) needed to >> + support Secure Virtual Machines (SVM)s and Paravirtalized KVM. The > Paravirtualized > >> + ultracalls allow the SVMs and Hypervisor to request services from the >> + Ultravisor such as accessing a register or memory region that can only >> + be accessed when running in Ultravisor-privileged mode. >> + >> + The specific service needed from an ultracall is specified in register >> + R3 (the first parameter to the ultracall). Other parameters to the >> + ultracall, if any, are specified in registers R4 through R12. >> + >> + Return value of all ultracalls is in register R3. Other output values >> + from the ultracall, if any, are returned in registers R4 through R12. >> + The only exception to this register usage is the ``UV_RETURN`` >> + ultracall described below. >> + >> + Each ultracall returns specific error codes, applicable in the context >> + of the ultracall. However, like with the PowerPC Architecture Platform >> + Reference (PAPR), if no specific error code is defined for a >> + particular situation, then the ultracall will fallback to an erroneous >> + parameter-position based code. i.e U_PARAMETER, U_P2, U_P3 etc >> + depending on the ultracall parameter that may have caused the error. >> + >> + Some ultracalls involve transferring a page of data between Ultravisor >> + and Hypervisor. Secure pages that are transferred from secure memory >> + to normal memory may be encrypted using dynamically generated keys. >> + When the secure pages are transferred back to secure memory, they may >> + be decrypted using the same dynamically generated keys. Generation and >> + management of these keys will be covered in a separate document. >> + >> + For now this only covers ultracalls currently implemented and being >> + used by Hypervisor and SVMs but others can be added here when it >> + makes sense. >> + >> + The full specification for all hypercalls/ultracalls will eventually >> + be made available in the public/OpenPower version of the PAPR >> + specification. >> + >> + **Note** >> + >> + If PEF is not enabled, the ultracalls will be redirected to the >> + Hypervisor which must handle/fail the calls. >> + >> +Ultracalls used by Hypervisor >> +============================= >> + >> + This section describes the virtual memory management ultracalls used >> + by the Hypervisor to manage SVMs. >> + >> +UV_PAGE_OUT >> +----------- >> + >> + Encrypt and move the contents of a page from secure memory to normal >> + memory. >> + >> +Syntax >> +~~~~~~ >> + >> +.. code-block:: c >> + >> + uint64_t ultracall(const uint64_t UV_PAGE_OUT, >> + uint16_t lpid, /* LPAR ID */ >> + uint64_t dest_ra, /* real address of destination page */ >> + uint64_t src_gpa, /* source guest-physical-address */ >> + uint8_t flags, /* flags */ >> + uint64_t order) /* page size order */ >> + >> +Return values >> +~~~~~~~~~~~~~ >> + >> + One of the following values: >> + >> + * U_SUCCESS on success. >> + * U_PARAMETER if ``lpid`` is invalid. >> + * U_P2 if ``dest_ra`` is invalid. >> + * U_P3 if the ``src_gpa`` address is invalid. >> + * U_P4 if any bit in the ``flags`` is unrecognized >> + * U_P5 if the ``order`` parameter is unsupported. >> + * U_FUNCTION if functionality is not supported. >> + * U_BUSY if page cannot be currently paged-out. >> + >> +Description >> +~~~~~~~~~~~ >> + >> + Encrypt the contents of a secure-page and make it available to >> + Hypervisor in a normal page. >> + >> + By default, the source page is unmapped from the SVM's partition- >> + scoped page table. But the Hypervisor can provide a hint to the >> + Ultravisor to retain the page mapping by setting the ``UV_SNAPSHOT`` >> + flag in ``flags`` parameter. >> + >> + If the source page is already a shared page the call returns >> + U_SUCCESS, without doing anything. >> + >> +Use cases >> +~~~~~~~~~ >> + >> + #. QEMU attempts to access an address belonging to the SVM but the >> + page frame for that address is not mapped into QEMU's address >> + space. In this case, the Hypervisor will allocate a page frame, >> + map it into QEMU's address space and issue the ``UV_PAGE_OUT`` >> + call to retrieve the encrypted contents of the page. >> + >> + #. When Ultravisor runs low on secure memory and it needs to page-out >> + an LRU page. In this case, Ultravisor will issue the >> + ``H_SVM_PAGE_OUT`` hypercall to the Hypervisor. The Hypervisor will >> + then allocate a normal page and issue the ``UV_PAGE_OUT`` ultracall >> + and the Ultravisor will encrypt and move the contents of the secure >> + page into the normal page. >> + >> + >> +UV_PAGE_IN >> +---------- >> + >> + Move the contents of a page from normal memory to secure memory. >> + >> +Syntax >> +~~~~~~ >> + >> +.. code-block:: c >> + >> + uint64_t ultracall(const uint64_t UV_PAGE_IN, >> + uint16_t lpid, /* the LPAR ID */ >> + uint64_t src_ra, /* source real address of page */ >> + uint64_t dest_gpa, /* destination guest physical address */ >> + uint64_t flags, /* flags */ >> + uint64_t order) /* page size order */ >> + >> +Return values >> +~~~~~~~~~~~~~ >> + >> + One of the following values: >> + >> + * U_SUCCESS on success. >> + * U_BUSY if page cannot be currently paged-in. >> + * U_FUNCTION if functionality is not supported >> + * U_PARAMETER if ``lpid`` is invalid. >> + * U_P2 if ``src_ra`` is invalid. >> + * U_P3 if the ``dest_gpa`` address is invalid. >> + * U_P4 if any bit in the ``flags`` is unrecognized >> + * U_P5 if the ``order`` parameter is unsupported. >> + >> +Description >> +~~~~~~~~~~~ >> + >> + Move the contents of the page identified by ``src_ra`` from normal >> + memory to secure memory and map it to the guest physical address >> + ``dest_gpa``. >> + >> + If `dest_gpa` refers to a shared address, map the page into the >> + partition-scoped page-table of the SVM. If `dest_gpa` is not shared, >> + copy the contents of the page into the corresponding secure page. >> + Depending on the context, decrypt the page before being copied. >> + >> + The caller provides the attributes of the page through the ``flags`` >> + parameter. Valid values for ``flags`` are: >> + >> + * CACHE_INHIBITED >> + * CACHE_ENABLED >> + * WRITE_PROTECTION >> + >> + The Hypervisor must pin the page in memory before making >> + ``UV_PAGE_IN`` ultracall. >> + >> +Use cases >> +~~~~~~~~~ >> + >> + #. When a normal VM switches to secure mode, all its pages residing >> + in normal memory, are moved into secure memory. >> + >> + #. When an SVM requests to share a page with Hypervisor the Hypervisor >> + allocates a page and informs the Ultravisor. >> + >> + #. When an SVM accesses a secure page that has been paged-out, >> + Ultravisor invokes the Hypervisor to locate the page. After >> + locating the page, the Hypervisor uses UV_PAGE_IN to make the >> + page available to Ultravisor. >> + >> + #. When Hypervisor accesses SVM data, the Hypervisor requests the >> + Ultravisor to transfer the corresponding page into a insecure page, >> + which the Hypervisor can access. The data in the normal page will >> + be encrypted though. > This looks like it should be under UV_PAGE_OUT instead. >
