Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-18 Thread Haitao Huang
On Wed, 18 Oct 2023 10:52:23 -0500, Michal Koutný wrote: On Wed, Oct 18, 2023 at 08:37:25AM -0700, Dave Hansen wrote: 1. Admin sets a limit 2. Enclave is created 3. Enclave hits limit, allocation fails I was actually about to suggest reorganizing the series to a part implementing this

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-18 Thread Michal Koutný
On Wed, Oct 18, 2023 at 08:37:25AM -0700, Dave Hansen wrote: > 1. Admin sets a limit > 2. Enclave is created > 3. Enclave hits limit, allocation fails I was actually about to suggest reorganizing the series to a part implementing this simple limiting and a subsequent part with the reclaim stuff

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-18 Thread Dave Hansen
On 10/18/23 08:26, Haitao Huang wrote: > Maybe not in sense of killing something. My understanding memory.reclaim > does not necessarily invoke the OOM killer. But what I really intend to > say is we can have a separate knob for user to express the need for > reducing the current usage explicitly

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-18 Thread Haitao Huang
On Wed, 18 Oct 2023 08:55:12 -0500, Dave Hansen wrote: On 10/17/23 21:37, Haitao Huang wrote: Yes we can introduce misc.reclaim to give user a knob to forcefully reducing usage if that is really needed in real usage. The semantics would make force-kill VMs explicit to user. Do any other

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-18 Thread Dave Hansen
On 10/17/23 21:37, Haitao Huang wrote: > Yes we can introduce misc.reclaim to give user a knob to forcefully > reducing usage if that is really needed in real usage. The semantics > would make force-kill VMs explicit to user. Do any other controllers do something like this? It seems odd.

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-17 Thread Haitao Huang
On Tue, 17 Oct 2023 14:13:22 -0500, Michal Koutný wrote: On Tue, Oct 17, 2023 at 08:54:48PM +0200, Michal Koutný wrote: Is this distinction between preemptability of EPC pages mandated by the HW implementation? (host/"process" enclaves vs VM enclaves) Or do have users an option to lock

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-17 Thread Haitao Huang
Hi Michal, On Tue, 17 Oct 2023 13:54:46 -0500, Michal Koutný wrote: Hello Haitao. On Tue, Oct 17, 2023 at 07:58:02AM -0500, Haitao Huang wrote: AFAIK, before we introducing max_write() callback in this series, no misc controller would possibly enforce the limit when misc.max is reduced.

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-17 Thread Michal Koutný
On Tue, Oct 17, 2023 at 08:54:48PM +0200, Michal Koutný wrote: > Is this distinction between preemptability of EPC pages mandated by the > HW implementation? (host/"process" enclaves vs VM enclaves) Or do have > users an option to lock certain pages in memory that yields this > difference?

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-17 Thread Michal Koutný
Hello Haitao. On Tue, Oct 17, 2023 at 07:58:02AM -0500, Haitao Huang wrote: > AFAIK, before we introducing max_write() callback in this series, no misc > controller would possibly enforce the limit when misc.max is reduced. e.g. I > don't think CVMs be killed when ASID limit is reduced and the

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-17 Thread Sean Christopherson
On Mon, Oct 16, 2023, Haitao Huang wrote: > Hi Sean > > On Mon, 16 Oct 2023 16:32:31 -0500, Sean Christopherson > wrote: > > > On Mon, Oct 16, 2023, Haitao Huang wrote: > > > From this perspective, I think the current implementation is > > > "well-defined": > > > EPC cgroup limits for VMs are

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-17 Thread Haitao Huang
On Mon, 16 Oct 2023 20:34:57 -0500, Huang, Kai wrote: On Mon, 2023-10-16 at 19:10 -0500, Haitao Huang wrote: On Mon, 16 Oct 2023 16:09:52 -0500, Huang, Kai wrote: [...] > still need to fix the bug mentioned above here. > > I really think you should just go this simple way: > > When you

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-17 Thread Mikko Ylinen
On Mon, Oct 16, 2023 at 02:32:31PM -0700, Sean Christopherson wrote: > Genuinely curious, who is asking for EPC cgroup support that *isn't* running > VMs? People who work with containers: [1], [2]. > AFAIK, these days, SGX is primarily targeted at cloud. I assume virtual EPC > is > the

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-16 Thread Huang, Kai
On Mon, 2023-10-16 at 19:10 -0500, Haitao Huang wrote: > On Mon, 16 Oct 2023 16:09:52 -0500, Huang, Kai wrote: > [...] > > > still need to fix the bug mentioned above here. > > > > I really think you should just go this simple way: > > > > When you want to take EPC back from VM, kill the VM. >

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-16 Thread Haitao Huang
On Mon, 16 Oct 2023 16:09:52 -0500, Huang, Kai wrote: [...] still need to fix the bug mentioned above here. I really think you should just go this simple way: When you want to take EPC back from VM, kill the VM. My only concern is that this is a compromise due to current limitation (no

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-16 Thread Haitao Huang
Hi Sean On Mon, 16 Oct 2023 16:32:31 -0500, Sean Christopherson wrote: On Mon, Oct 16, 2023, Haitao Huang wrote: From this perspective, I think the current implementation is "well-defined": EPC cgroup limits for VMs are only enforced at VM launch time, not runtime. In practice, SGX VM

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-16 Thread Sean Christopherson
On Mon, Oct 16, 2023, Haitao Huang wrote: > From this perspective, I think the current implementation is "well-defined": > EPC cgroup limits for VMs are only enforced at VM launch time, not runtime. > In practice, SGX VM can be launched only with fixed EPC size and all those > EPCs are fully

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-16 Thread Huang, Kai
> > > From this perspective, I think the current implementation is > "well-defined": EPC cgroup limits for VMs are only enforced at VM launch > time, not runtime. In practice, SGX VM can be launched only with fixed > EPC size and all those EPCs are fully committed to the VM once

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-16 Thread Haitao Huang
On Mon, 16 Oct 2023 05:57:36 -0500, Huang, Kai wrote: On Thu, 2023-10-12 at 08:27 -0500, Haitao Huang wrote: On Tue, 10 Oct 2023 19:51:17 -0500, Huang, Kai wrote: [...] > (btw, even you track VA/SECS pages in unreclaimable list, given they > both have > 'enclave' as the owner, do you

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-16 Thread Huang, Kai
On Wed, 2023-10-11 at 01:14 +, Huang, Kai wrote: > On Tue, 2023-10-10 at 11:49 -0500, Haitao Huang wrote: > > > > > > This patch adds SGX_ENCL_NO_MEMORY.  I guess we can use it for virtual  > > > EPC too? > > > > > > > That flag is set for enclaves, do you mean we set similar flag in vepc 

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-16 Thread Huang, Kai
On Thu, 2023-10-12 at 08:27 -0500, Haitao Huang wrote: > On Tue, 10 Oct 2023 19:51:17 -0500, Huang, Kai wrote: > [...] > > (btw, even you track VA/SECS pages in unreclaimable list, given they > > both have > > 'enclave' as the owner, do you still need SGX_EPC_OWNER_ENCL and > >

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-12 Thread Haitao Huang
On Tue, 10 Oct 2023 19:51:17 -0500, Huang, Kai wrote: [...] (btw, even you track VA/SECS pages in unreclaimable list, given they both have 'enclave' as the owner, do you still need SGX_EPC_OWNER_ENCL and SGX_EPC_OWNER_PAGE ?) Let me think about it, there might be also a way just track encl

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-11 Thread Haitao Huang
On Tue, 10 Oct 2023 19:31:19 -0500, Huang, Kai wrote: On Tue, 2023-10-10 at 12:05 -0500, Haitao Huang wrote: On Mon, 09 Oct 2023 21:12:27 -0500, Huang, Kai wrote: > > > > > > > > > > Later the hosting process could migrated/reassigned to another > > cgroup? > > > > What to do when the new

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-11 Thread Haitao Huang
On Tue, 10 Oct 2023 19:01:25 -0500, Sean Christopherson wrote: On Tue, Oct 10, 2023, Haitao Huang wrote: On Mon, 09 Oct 2023 21:23:12 -0500, Huang, Kai wrote: > On Mon, 2023-10-09 at 20:42 -0500, Haitao Huang wrote: > > Hi Sean > > > > On Mon, 09 Oct 2023 19:23:04 -0500, Sean

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-10 Thread Huang, Kai
On Tue, 2023-10-10 at 11:49 -0500, Haitao Huang wrote: > > > > This patch adds SGX_ENCL_NO_MEMORY.  I guess we can use it for virtual  > > EPC too? > > > > That flag is set for enclaves, do you mean we set similar flag in vepc  > struct? Yes.

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-10 Thread Huang, Kai
On Tue, 2023-10-10 at 11:49 -0500, Haitao Huang wrote: > On Mon, 09 Oct 2023 20:34:29 -0500, Huang, Kai wrote: > > > On Tue, 2023-10-10 at 00:50 +, Huang, Kai wrote: > > > On Mon, 2023-10-09 at 17:23 -0700, Sean Christopherson wrote: > > > > On Mon, Oct 09, 2023, Kai Huang wrote: > > > > >

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-10 Thread Huang, Kai
On Tue, 2023-10-10 at 12:05 -0500, Haitao Huang wrote: > On Mon, 09 Oct 2023 21:12:27 -0500, Huang, Kai wrote: > > > > > > > > > > > > > > Later the hosting process could migrated/reassigned to another > > > cgroup? > > > > > What to do when the new cgroup is OOM? > > > > > > > > > > > > >

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-10 Thread Sean Christopherson
On Tue, Oct 10, 2023, Haitao Huang wrote: > On Mon, 09 Oct 2023 21:23:12 -0500, Huang, Kai wrote: > > > On Mon, 2023-10-09 at 20:42 -0500, Haitao Huang wrote: > > > Hi Sean > > > > > > On Mon, 09 Oct 2023 19:23:04 -0500, Sean Christopherson > > > wrote: > > > > I can see userspace wanting to

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-10 Thread Haitao Huang
On Mon, 09 Oct 2023 21:12:27 -0500, Huang, Kai wrote: > > > > > Later the hosting process could migrated/reassigned to another cgroup? > > What to do when the new cgroup is OOM? > > > > You addressed in the documentation, no? > > +Migration > +- > + > +Once an EPC page is charged

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-10 Thread Haitao Huang
On Mon, 09 Oct 2023 20:34:29 -0500, Huang, Kai wrote: On Tue, 2023-10-10 at 00:50 +, Huang, Kai wrote: On Mon, 2023-10-09 at 17:23 -0700, Sean Christopherson wrote: > On Mon, Oct 09, 2023, Kai Huang wrote: > > On Fri, 2023-09-22 at 20:06 -0700, Haitao Huang wrote: > > > +/** > > > + *

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-10 Thread Haitao Huang
On Mon, 09 Oct 2023 21:23:12 -0500, Huang, Kai wrote: On Mon, 2023-10-09 at 20:42 -0500, Haitao Huang wrote: Hi Sean On Mon, 09 Oct 2023 19:23:04 -0500, Sean Christopherson wrote: > On Mon, Oct 09, 2023, Kai Huang wrote: > > On Fri, 2023-09-22 at 20:06 -0700, Haitao Huang wrote: > > > +/**

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-09 Thread Huang, Kai
On Mon, 2023-10-09 at 20:42 -0500, Haitao Huang wrote: > Hi Sean > > On Mon, 09 Oct 2023 19:23:04 -0500, Sean Christopherson > wrote: > > > On Mon, Oct 09, 2023, Kai Huang wrote: > > > On Fri, 2023-09-22 at 20:06 -0700, Haitao Huang wrote: > > > > +/** > > > > + * sgx_epc_oom() - invoke EPC

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-09 Thread Huang, Kai
> > > > > > > Later the hosting process could migrated/reassigned to another cgroup? > > > What to do when the new cgroup is OOM? > > > > > > > You addressed in the documentation, no? > > > > +Migration > > +- > > + > > +Once an EPC page is charged to a cgroup (during allocation), it

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-09 Thread Haitao Huang
Hi Sean On Mon, 09 Oct 2023 19:23:04 -0500, Sean Christopherson wrote: On Mon, Oct 09, 2023, Kai Huang wrote: On Fri, 2023-09-22 at 20:06 -0700, Haitao Huang wrote: > +/** > + * sgx_epc_oom() - invoke EPC out-of-memory handling on target LRU > + * @lru: LRU that is low > + * > + *

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-09 Thread Haitao Huang
On Mon, 09 Oct 2023 20:18:00 -0500, Huang, Kai wrote: On Mon, 2023-10-09 at 20:04 -0500, Haitao Huang wrote: On Mon, 09 Oct 2023 18:45:06 -0500, Huang, Kai wrote: > On Fri, 2023-09-22 at 20:06 -0700, Haitao Huang wrote: > > From: Sean Christopherson > > > > Introduce the OOM path for

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-09 Thread Huang, Kai
On Tue, 2023-10-10 at 00:50 +, Huang, Kai wrote: > On Mon, 2023-10-09 at 17:23 -0700, Sean Christopherson wrote: > > On Mon, Oct 09, 2023, Kai Huang wrote: > > > On Fri, 2023-09-22 at 20:06 -0700, Haitao Huang wrote: > > > > +/** > > > > + * sgx_epc_oom() - invoke EPC out-of-memory handling on

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-09 Thread Huang, Kai
On Mon, 2023-10-09 at 20:04 -0500, Haitao Huang wrote: > On Mon, 09 Oct 2023 18:45:06 -0500, Huang, Kai wrote: > > > On Fri, 2023-09-22 at 20:06 -0700, Haitao Huang wrote: > > > From: Sean Christopherson > > > > > > Introduce the OOM path for killing an enclave with a reclaimer that is > > >

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-09 Thread Haitao Huang
On Mon, 09 Oct 2023 18:45:06 -0500, Huang, Kai wrote: On Fri, 2023-09-22 at 20:06 -0700, Haitao Huang wrote: From: Sean Christopherson Introduce the OOM path for killing an enclave with a reclaimer that is no longer able to reclaim enough EPC pages. Find a victim enclave, which will be

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-09 Thread Huang, Kai
On Mon, 2023-10-09 at 17:23 -0700, Sean Christopherson wrote: > On Mon, Oct 09, 2023, Kai Huang wrote: > > On Fri, 2023-09-22 at 20:06 -0700, Haitao Huang wrote: > > > +/** > > > + * sgx_epc_oom() - invoke EPC out-of-memory handling on target LRU > > > + * @lru: LRU that is low > > > + * > > > + *

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-09 Thread Sean Christopherson
On Mon, Oct 09, 2023, Kai Huang wrote: > On Fri, 2023-09-22 at 20:06 -0700, Haitao Huang wrote: > > +/** > > + * sgx_epc_oom() - invoke EPC out-of-memory handling on target LRU > > + * @lru: LRU that is low > > + * > > + * Return: %true if a victim was found and kicked. > > + */ > > +bool

Re: [PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-10-09 Thread Huang, Kai
On Fri, 2023-09-22 at 20:06 -0700, Haitao Huang wrote: > From: Sean Christopherson > > Introduce the OOM path for killing an enclave with a reclaimer that is no > longer able to reclaim enough EPC pages. Find a victim enclave, which > will be an enclave with only "unreclaimable" EPC pages left

[PATCH v5 12/18] x86/sgx: Add EPC OOM path to forcefully reclaim EPC

2023-09-22 Thread Haitao Huang
From: Sean Christopherson Introduce the OOM path for killing an enclave with a reclaimer that is no longer able to reclaim enough EPC pages. Find a victim enclave, which will be an enclave with only "unreclaimable" EPC pages left in the cgroup LRU lists. Once a victim is identified, mark the