Re: [lustre-discuss] Are there any performance hits with the https://access.redhat.com/security/vulnerabilities/speculativeexecution?

[Patrick Farrell]

Note though that since the servers live in kernel space they are also going to 
be affected only minimally.  The Lustre server code itself will see zero 
effect, since it’s entirely kernel code.  Other things running on those servers 
may see impact, and if there’s enough user space stuff, increased usage there 
could reduce resources available for Lustre.

Note also it’s important to distinguish here: the issue is not context switches 
(which is scheduling a different process), it’s syscalls, which do not require 
a context switch.  Context switches already had this sort of overhead.  A 
syscall is not a context switch.  (But the KPTI changes make the effective 
difference smaller.)



From: lustre-discuss  on behalf of 
E.S. Rosenberg 
Sent: Monday, January 8, 2018 7:05:48 AM
To: Arman Khalatyan
Cc: Lustre discussion
Subject: Re: [lustre-discuss] Are there any performance hits with the 
https://access.redhat.com/security/vulnerabilities/speculativeexecution?

The hit is mainly for things that do context switches (which IO is the biggest 
thing in.

On Mon, Jan 8, 2018 at 1:23 PM, Arman Khalatyan 
> wrote:
Ok, We did some tests with the new lustre clients(no patch on servers)
I can confirm like Marek: maximum downgrade is about 40% by rsync with
small files, lfs find on large folders 45% performance penalty:(
We found terrible performance on the test system with zfs+compression+lustre.
Good news: the compute node flops are about 1% or even none. So only
IO intensive applications are impacted.

Cheers,
Arman.

On Mon, Jan 8, 2018 at 11:45 AM, Marek Magryś 
> wrote:
> Hi all,
>
>> I wonder if any performance impacts on lustre with the new security
>> patches for the Intel?
>
> According to our initial tests on 3.10.0-693.11.6.el7.x86_64 kernel
> (Centos 7.4) with Lustre 2.10.2, there is a penalty of ca. 10% in nice
> workloads (1MB IO) up to 40% in 4k IOs. Tested with IOR.
>
> It looks bad, however probably we don't need to patch the servers, as
> Lustre lives in kernelspace anyway. Some kind of advisory from Intel
> HPDD would be nice here.
>
> Cheers,
> Marek
>
> --
> Marek Magrys
> ACC Cyfronet AGH-UST
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Are there any performance hits with the https://access.redhat.com/security/vulnerabilities/speculativeexecution?

[E.S. Rosenberg]

The hit is mainly for things that do context switches (which IO is the
biggest thing in.

On Mon, Jan 8, 2018 at 1:23 PM, Arman Khalatyan  wrote:

> Ok, We did some tests with the new lustre clients(no patch on servers)
> I can confirm like Marek: maximum downgrade is about 40% by rsync with
> small files, lfs find on large folders 45% performance penalty:(
> We found terrible performance on the test system with
> zfs+compression+lustre.
> Good news: the compute node flops are about 1% or even none. So only
> IO intensive applications are impacted.
>
> Cheers,
> Arman.
>
> On Mon, Jan 8, 2018 at 11:45 AM, Marek Magryś 
> wrote:
> > Hi all,
> >
> >> I wonder if any performance impacts on lustre with the new security
> >> patches for the Intel?
> >
> > According to our initial tests on 3.10.0-693.11.6.el7.x86_64 kernel
> > (Centos 7.4) with Lustre 2.10.2, there is a penalty of ca. 10% in nice
> > workloads (1MB IO) up to 40% in 4k IOs. Tested with IOR.
> >
> > It looks bad, however probably we don't need to patch the servers, as
> > Lustre lives in kernelspace anyway. Some kind of advisory from Intel
> > HPDD would be nice here.
> >
> > Cheers,
> > Marek
> >
> > --
> > Marek Magrys
> > ACC Cyfronet AGH-UST
> > ___
> > lustre-discuss mailing list
> > lustre-discuss@lists.lustre.org
> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Are there any performance hits with the https://access.redhat.com/security/vulnerabilities/speculativeexecution?

[Arman Khalatyan]

Ok, We did some tests with the new lustre clients(no patch on servers)
I can confirm like Marek: maximum downgrade is about 40% by rsync with
small files, lfs find on large folders 45% performance penalty:(
We found terrible performance on the test system with zfs+compression+lustre.
Good news: the compute node flops are about 1% or even none. So only
IO intensive applications are impacted.

Cheers,
Arman.

On Mon, Jan 8, 2018 at 11:45 AM, Marek Magryś  wrote:
> Hi all,
>
>> I wonder if any performance impacts on lustre with the new security
>> patches for the Intel?
>
> According to our initial tests on 3.10.0-693.11.6.el7.x86_64 kernel
> (Centos 7.4) with Lustre 2.10.2, there is a penalty of ca. 10% in nice
> workloads (1MB IO) up to 40% in 4k IOs. Tested with IOR.
>
> It looks bad, however probably we don't need to patch the servers, as
> Lustre lives in kernelspace anyway. Some kind of advisory from Intel
> HPDD would be nice here.
>
> Cheers,
> Marek
>
> --
> Marek Magrys
> ACC Cyfronet AGH-UST
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Are there any performance hits with the https://access.redhat.com/security/vulnerabilities/speculativeexecution?

[Marek Magryś]

Hi all,

> I wonder if any performance impacts on lustre with the new security
> patches for the Intel?

According to our initial tests on 3.10.0-693.11.6.el7.x86_64 kernel
(Centos 7.4) with Lustre 2.10.2, there is a penalty of ca. 10% in nice
workloads (1MB IO) up to 40% in 4k IOs. Tested with IOR.

It looks bad, however probably we don't need to patch the servers, as
Lustre lives in kernelspace anyway. Some kind of advisory from Intel
HPDD would be nice here.

Cheers,
Marek

-- 
Marek Magrys
ACC Cyfronet AGH-UST
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Are there any performance hits with the https://access.redhat.com/security/vulnerabilities/speculativeexecution?

[E.S. Rosenberg]

There is absolutely no argument from me that the Client side *has* to be
patched immediately, my question was only about server side which seems to
me to be at a mitigated risk due to the nature of the server.
I think we'll be switching to vanilla kernel on client side and seeing how
that works for us (at least until we migrate to server 2.10.x or 2.11).

Regards,
Eli

On Sat, Jan 6, 2018 at 12:29 AM, Marion Hakanson  wrote:

> We may not need to apply these mitigations to Lustre servers,
> but a lot of Lustre code runs on the client systems.
>
> Let's say you run a multi-user research cluster;  Lab group A says
> that their data must not be seen by any user except those in Lab A, so
> user, group, and filesystem permissions are set to implement that policy.
>
> Lab groups B and C may not have malicious users, but they do download,
> compile, and run programs from collaborators, or from the Internet
> at large.  So they may inadvertently install and run some malicious
> code on that research cluster, and potentially expose Lab group A's
> data even though B and C users wouldn't normally have permissions
> to do so.
>
> Do you analyze every bit of code that runs on your research cluster?
> We don't have the resources to do so.
>
>
> A possible related issue:  In addition to the kernel-vs-user address space
> changes needed for Meltdown, there are also some code changes needed to
> prevent the Spectre type of attacks.  Those changes (function call/return
> conventions) need to happen in user-space code, but also in the kernel.
> I imagine that Lustre code itself could need these mods too, in order
> to be protected from attack code on client systems.
>
> https://newsroom.intel.com/wp-content/uploads/sites/11/2018/
> 01/Intel-Analysis-o
> f-Speculative-Execution-Side-Channels.pdf
>
> I didn't find any items matching "meltdown" or "spectre" in the HPDD
> Lustre JIRA just now, so perhaps work hasn't started on this yet.
>
> Regards,
>
> Marion
>
>
>
> > Date: Fri, 5 Jan 2018 13:31:23 -0500
> > From: Mark Hahn 
> > To: Lustre discussion 
> > Subject: Re: [lustre-discuss] Are there any performance hits with the
> >
> > > Also to what extent would a Lustre system that is essentially a filer
> be at
> > > risk? It's not running user code and you're not browsing from it...
> >
> > to be vulnerable, attack code must run on the system.
> > ___
> > lustre-discuss mailing list
> > lustre-discuss@lists.lustre.org
> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> >
>
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Are there any performance hits with the https://access.redhat.com/security/vulnerabilities/speculativeexecution?

[Marion Hakanson]

We may not need to apply these mitigations to Lustre servers,
but a lot of Lustre code runs on the client systems.

Let's say you run a multi-user research cluster;  Lab group A says
that their data must not be seen by any user except those in Lab A, so
user, group, and filesystem permissions are set to implement that policy.

Lab groups B and C may not have malicious users, but they do download,
compile, and run programs from collaborators, or from the Internet
at large.  So they may inadvertently install and run some malicious
code on that research cluster, and potentially expose Lab group A's
data even though B and C users wouldn't normally have permissions
to do so.

Do you analyze every bit of code that runs on your research cluster?
We don't have the resources to do so.


A possible related issue:  In addition to the kernel-vs-user address space
changes needed for Meltdown, there are also some code changes needed to
prevent the Spectre type of attacks.  Those changes (function call/return
conventions) need to happen in user-space code, but also in the kernel.
I imagine that Lustre code itself could need these mods too, in order
to be protected from attack code on client systems.

https://newsroom.intel.com/wp-content/uploads/sites/11/2018/01/Intel-Analysis-o
f-Speculative-Execution-Side-Channels.pdf

I didn't find any items matching "meltdown" or "spectre" in the HPDD
Lustre JIRA just now, so perhaps work hasn't started on this yet.

Regards,

Marion



> Date: Fri, 5 Jan 2018 13:31:23 -0500
> From: Mark Hahn 
> To: Lustre discussion 
> Subject: Re: [lustre-discuss] Are there any performance hits with the
> 
> > Also to what extent would a Lustre system that is essentially a filer be at
> > risk? It's not running user code and you're not browsing from it...
> 
> to be vulnerable, attack code must run on the system.
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> 


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Are there any performance hits with the https://access.redhat.com/security/vulnerabilities/speculativeexecution?

[Mark Hahn]

Also to what extent would a Lustre system that is essentially a filer be at
risk? It's not running user code and you're not browsing from it...


to be vulnerable, attack code must run on the system.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Are there any performance hits with the https://access.redhat.com/security/vulnerabilities/speculativeexecution?

[E.S. Rosenberg]

If Phoronix FS-Mark is in any way like Lustre we should be ready for a
serious hit it would seem:
https://www.phoronix.com/scan.php?page=article=linux-415-x86pti=2

Has anyone built the server against the new kernel yet?

Also to what extent would a Lustre system that is essentially a filer be at
risk? It's not running user code and you're not browsing from it...

HTH and Thanks,
Eli

On Fri, Jan 5, 2018 at 11:16 AM, Arman Khalatyan  wrote:

> Hello List!
> I wonder if any performance impacts on lustre with the new security
> patches for the Intel?
>
> Thanks,
> Arman.
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org