On Tue, Jan 12, 2021 at 07:35:50PM +0100, Borislav Petkov wrote:
> + paulmck.
>
> On Tue, Jan 12, 2021 at 02:08:10AM +0200, Jarkko Sakkinen wrote:
> > On Tue, Jan 05, 2021 at 03:57:49PM +0100, Borislav Petkov wrote:
> > > On Wed, Dec 16, 2020 at 03:49:20PM +0200, Jarkko Sakkinen wrote:
> > > > Add synchronize_srcu_expedited() to sgx_encl_release() to catch a grace
> > > > period initiated by sgx_mmu_notifier_release().
> > > >
> > > > A trivial example of a failing sequence with tasks A and B:
> > > >
> > > > 1. A: -> sgx_release()
> > > > 2. B: -> sgx_mmu_notifier_release()
> > > > 3. B: -> list_del_rcu()
> > > > 3. A: -> sgx_encl_release()
> > > > 4. A: -> cleanup_srcu_struct()
> > > >
> > > > The loop in sgx_release() observes an empty list because B has removed
> > > > its
> > > > entry in the middle, and calls cleanup_srcu_struct() before B has a
> > > > chance
> > > > to calls synchronize_srcu().
> > >
> > > Leading to what? NULL ptr?
> > >
> > > https://lkml.kernel.org/r/x9e2jowz1hfxv...@google.com
> > >
> > > already suggested that you should explain the bug better and add the
> > > splat but I'm still missing that explanation.
> >
> > OK, I'll try to explain it how I understand the issue.
> >
> > Consider this loop in the VFS release hook (sgx_release):
> >
> > /*
> > * Drain the remaining mm_list entries. At this point the list contains
> > * entries for processes, which have closed the enclave file but have
> > * not exited yet. The processes, which have exited, are gone from the
> > * list by sgx_mmu_notifier_release().
> > */
> > for ( ; ; ) {
> > spin_lock(&encl->mm_lock);
> >
> > if (list_empty(&encl->mm_list)) {
> > encl_mm = NULL;
> > } else {
> > encl_mm = list_first_entry(&encl->mm_list,
> > struct sgx_encl_mm, list);
> > list_del_rcu(&encl_mm->list);
> > }
> >
> > spin_unlock(&encl->mm_lock);
> >
> > /* The enclave is no longer mapped by any mm. */
> > if (!encl_mm)
> > break;
> >
> > synchronize_srcu(&encl->srcu);
> > mmu_notifier_unregister(&encl_mm->mmu_notifier, encl_mm->mm);
> > kfree(encl_mm);
> > }
> >
> >
> > At this point all processes have closed the enclave file, but that doesn't
> > mean that they all have exited yet.
> >
> > Now, let's imagine that there is exactly one entry in the encl->mm_list.
> > and sgx_release() execution gets scheduled right after returning from
> > synchronize_srcu().
> >
> > With some bad luck, some process comes and removes that last entry befoe
> > sgx_release() acquires mm_lock. The loop in sgx_release() just leaves
> >
> > /* The enclave is no longer mapped by any mm. */
> > if (!encl_mm)
> > break;
> >
> > No synchronize_srcu().
> >
> > After writing this, I think that the placement for synchronize_srcu()
> > in this patch is not best possible. It should be rather that the
> > above loop would also call synchronize_srcu() when leaving.
> >
> > I.e. the code change would result:
> >
> > for ( ; ; ) {
> > spin_lock(&encl->mm_lock);
> >
> > if (list_empty(&encl->mm_list)) {
> > encl_mm = NULL;
> > } else {
> > encl_mm = list_first_entry(&encl->mm_list,
> > struct sgx_encl_mm, list);
> > list_del_rcu(&encl_mm->list);
> > }
> >
> > spin_unlock(&encl->mm_lock);
> >
> > /*
> > * synchronize_srcu() is mandatory *even* when the list was
> > * empty, in order make sure that grace periods stays in
> > * sync even when another task took away the last entry
> > * (i.e. exiting process when it deletes its mm_list).
> > */
> > synchronize_srcu(&encl->srcu);
> >
> > /* The enclave is no longer mapped by any mm. */
> > if (!encl_mm)
> > break;
> >
> > mmu_notifier_unregister(&encl_mm->mmu_notifier, encl_mm->mm);
> > kfree(encl_mm);
> > }
> >
> > What do you think? Does this start to make more sense now?
> > I don't have logs for this but the bug can be also reasoned.
>
> It does. Now you need to write it up in a detailed form so that it is
> clear to readers months/years from now what exactly can happen. You can
> use a two-column format like
>
> CPU A CPU B
>
> Bla
> Blu
>
> This happens now here
> But this needs to happen there
>
> and so on.
>
> Also, from reading up a bit on this, Documentation/RCU/checklist.rst says
>
> "Use of the expedited primitives should be restricted to rare
> configuration-change operations that would not normally be undertaken
> while a real-time workload is running."
>
> so why are you using synchronize_srcu_expedited()? Grepping the tree
> reveals only a couple of call sites only... but I've almost no clue of
> RCU so lemme CC Paul.
The SRCU expedited grace periods are easier on real-time workloads than
synchronize_rcu_expedited(), but the SRCU variant still burns more CPU
time on a given grace period. But either way, as the document says
further down "However, real-time workloads can use rcupdate.rcu_normal
kernel boot parameter to completely disable expedited grace periods,
though this might have performance implications."
So what are the performance implications in this case?
Thanx, Paul
