Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On 4/19/21 2:33 PM, Len Brown via Libc-alpha wrote: the AI guys are super excited about matrix multiplication, but I have a hard time imagining why grep(1) would find a use for it. I don't. Matrix multiplication is used in modern string-searching algorithms that could be useful in running 'grep' on CPUs that have relevant hardware support. See, for example: Susanina Y, Yaveyn A, Grigorev S. Modification of Valiant’s Parsing Algorithm for the String-Searching Problem. CIBB 2019. https://doi.org/10.1007/978-3-030-63061-4_17 Although nowadays this technology is typically proposed for bioinformatics (DNA pattern matching, etc.), it's not that much of a stretch to imagine a future 'grep' or 'diff' that does matrix multiplication. After all, GNU 'diff' currently uses an algorithm designed by a DNA expert. (We now return you to the regular AMX debates. :-)
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Apr 19, 2021 at 05:33:03PM -0400, Len Brown wrote: > For this to happen, every thread would not only have to include/link-with > code that uses AMX, but that code would have to *run*. It looks like either I'm not expressing myself clearly enough or you're not reading my text: the *library* does that decision automatically! Which means *every* possible thread on the system. Which means, *every* thread has a fat 8K buffer attached to it because the library uses AMX on its behalf by *default*. > I'm sure that the AI guys are super excited about matrix multiplication, > but I have a hard time imagining why grep(1) would find a use for it. It doesn't matter if you're imagining it or not - what matters is if the decision whether the thread uses AMX or not is put in the hands of the thread and *NOT* in the hands of the library. Which means, majority of the threads should not allow AMX and only a handful who do, will have to explicitly state that. And the library will have to comply. Not the library decides for every thread itself because the feature's there. > Indeed, if anyone expected AMX to be used by every task, we would have > never gone to the trouble of inventing the XFD hardware to support the > kernel's lazy 8KB buffer allocation. If it gives me fat-buffers-off-by-default and on only for a handful of threads which really want it and *request* it *explicitly*, sure, whatever gets the job done. -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Apr 19, 2021 at 3:15 PM Borislav Petkov wrote: > All of a sudden you have *every* thread sporting a fat 8K buffer because > the library decided to use a fat buffer feature for it. > > Nope, I don't want that to happen. For this to happen, every thread would not only have to include/link-with code that uses AMX, but that code would have to *run*. I'm sure that the AI guys are super excited about matrix multiplication, but I have a hard time imagining why grep(1) would find a use for it. Indeed, if anyone expected AMX to be used by every task, we would have never gone to the trouble of inventing the XFD hardware to support the kernel's lazy 8KB buffer allocation. cheers, Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Apr 19, 2021 at 02:18:51PM -0400, Len Brown wrote: > Yes, we could invent a new system call and mandate that it be called > between #2 and #3. However, we'd still do #4 in response, so I don't > see any value to that system call. Lemme refresh your memory: there was a time when the kernel did lazy FPU switching because tasks which really wanted to do that, would use FPU insns and from the first use onwards, the kernel would shuffle an FPU state buffer back'n'forth for the task, for the length of its lifetime. Then glibc decided to use FPU in memcpy or whatever, leading up to *every* task using the FPU which practically made us remove all that lazy FPU switching logic and do eager FPU. Back then that state was what, dunno, 1-2 KB tops. Now imagine the same lazy => eager switch but with AVX or AMX or . All of a sudden you have *every* thread sporting a fat 8K buffer because the library decided to use a fat buffer feature for it. Nope, I don't want that to happen. -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Apr 19, 2021 at 10:15 AM Borislav Petkov wrote: > > Tasks are created without an 8KB AMX buffer. > > Tasks have to actually touch the AMX TILE registers for us to allocate > > one for them. > > When tasks do that it doesn't matter too much - for the library it does! > > If the library does that by default and the processes which comprise > of that pipe I mentioned earlier, get all 8K buffers because the > underlying library decided so and swinging those buffers around when > saving/restoring contexts turns out to be a performance penalty, then we > have lost. > > Lost because if that thing goes upstream in this way of use of AMX is > allowed implicitly, there ain't fixing it anymore once it becomes an > ABI. > > So, that library should ask the kernel whether it supports AMX and only > use it if has gotten a positive answer. Right, the library *does* ask the kernel whether it supports AMX (below). > And by default that answer > should be "no" because the majority of processes - that same pipe I keep > mentioning - don't need it. Indeed, the default is "no" because most libraries will *not* ask the system for AMX support (below). However, if they *did* probe for it, and they *did* use it, the kernel would not stand in the way of any of those requests. > I have no good idea yet how granulary that should be - per process, per > thread, whatever, but there should be a way for the kernel to control > whether the library uses AMX, AVX512 or whatever fat state is out there > available. > > Then, if a process wants the library to use AMX on its behalf, then it > can say so and the library can do that but only after having asked for > explicitly. The ABI works like this: 0. App or library author decides AMX is useful at build-time. 1. App checks CPUID for AMX CPU feature 2. App checks XCR0 for AMX OS support (if app touches AMX without these two being TRUE, it will suffer the consequence of a #UD when it touches an AMX instruction) This ABI is how AVX works today. What is new with AMX is the ability of the hardware and the OS to delay the allocation of the context switch buffer until if/when it is actually needed. This is transparent, and thus not part of the ABI, unless you count the absence of a mandated system call to be an ABI. 3. the application then touches an AMX register, triggering... 4. #NM handled by the kernel, which allocates a context switch buffer for that task, and dis-arms XFD. Yes, we could invent a new system call and mandate that it be called between #2 and #3. However, we'd still do #4 in response, so I don't see any value to that system call. Indeed, I would advocate that glibc replace it with a return statement. So back to the example: | grep | awk | sed ... Sure, if grep grows support for some AI feature that we haven't imaged yet, then something in its code flow is fully empowered to probe for AMX and use AMX on AMX hardware. Sort of hard to imagine with the programs above that we know today, but future programs certainly could do this if they chose to. thanks, Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Fri, Apr 16, 2021 at 06:05:10PM -0400, Len Brown wrote: > I'm not aware of any intent to transparently use AMX for bcopy, like > what happened > with AVX-512. (didn't they undo that mistake?) No clue, did they? > Tasks are created without an 8KB AMX buffer. > Tasks have to actually touch the AMX TILE registers for us to allocate > one for them. When tasks do that it doesn't matter too much - for the library it does! If the library does that by default and the processes which comprise of that pipe I mentioned earlier, get all 8K buffers because the underlying library decided so and swinging those buffers around when saving/restoring contexts turns out to be a performance penalty, then we have lost. Lost because if that thing goes upstream in this way of use of AMX is allowed implicitly, there ain't fixing it anymore once it becomes an ABI. So, that library should ask the kernel whether it supports AMX and only use it if has gotten a positive answer. And by default that answer should be "no" because the majority of processes - that same pipe I keep mentioning - don't need it. I have no good idea yet how granulary that should be - per process, per thread, whatever, but there should be a way for the kernel to control whether the library uses AMX, AVX512 or whatever fat state is out there available. Then, if a process wants the library to use AMX on its behalf, then it can say so and the library can do that but only after having asked for explicitly. Thx. -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Fri, Apr 16, 2021 at 6:14 PM Andy Lutomirski wrote: > My point is that every ... I encourage you to continue to question everything and trust nobody. While it may cost you a lot in counseling, it is certainly valuable, at least to me! :-) I do request, however, that feedback stay specific, stay technical, and stay on-topic. We all have plenty of real challenges we can be tackling with our limited time. > Is there any authoritative guidance at all on what actually happens, > performance-wise, when someone does AMX math? Obviously, I can't speak to the performance of AMX itself pre-production, and somebody who does that for a living will release stuff on or before release day. What I've told you about the performance side-effects on the system (and lack thereof) from running AMX code is an authoritative answer, and is as much as I can tell you today. If I failed to answer a question about AMX, my apologies, please re-ask it. And if we learn something new between now and release day that is relevant to this discussion, I will certainly request to share it. Our team (Intel Open Source Technology Center) advocated getting the existing public AMX documentation published as early as possible. However, if you are really nto the details of how AMX works, you may also be interested to know that the AMX hardware patent filings are fully public ;-) cheers, Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Fri, Apr 16, 2021 at 3:11 PM Len Brown wrote: > > > I get it. That does not explain why LDMXCSR and VLDMXCSR cause > > pipelines stalls. > > Sorry, I thought this thread was about AMX. > I don't know the answer to your LDMXCSR and VLDMXCSR question. My point is that every single major math extension since the original XMM extensions (SSE, etc) has come with performance gotchas. Given Intel's general unwillingness to document the gotchas in hardware that is actually shipping, I'm sceptical that AMX is as delightfully gotcha-free as you are making it out to be. Is there any authoritative guidance at all on what actually happens, performance-wise, when someone does AMX math?
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
> I get it. That does not explain why LDMXCSR and VLDMXCSR cause > pipelines stalls. Sorry, I thought this thread was about AMX. I don't know the answer to your LDMXCSR and VLDMXCSR question.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Thu, Apr 15, 2021 at 1:47 AM Borislav Petkov wrote: > What I'd like to see is 0-overhead for current use cases and only > overhead for those who want to use it. If that can't be done > automagically, then users should request it explicitly. So basically you > blow up the xsave buffer only for processes which want to do AMX. Indeed, expanding the xsave buffer happens only for tasks that touch AMX TILE registers. > And this brings the question about libraries which, if they start using > AMX by default - which doesn't sound like they will want to because AMX > reportedly will have only a limited? set of users - if libraries start > using it by default, then it better be worth the handling of the 8kb > buffer per process. I'm not aware of any intent to transparently use AMX for bcopy, like what happened with AVX-512. (didn't they undo that mistake?) > If not, this should also be requestable per process so that a simple > pipe in Linux: > > | grep | awk | sed ... > > and so on is not penalized to allocate and handle by default 8kb for > *each* process' buffer in that pipe just because each is linking against > glibc which has detected AMX support in CPUID and is using it too for > some weird reason like some microbenchmark saying so. Tasks are created without an 8KB AMX buffer. Tasks have to actually touch the AMX TILE registers for us to allocate one for them. > But my initial question was on the "establishing" part and was asking > where we have established anything wrt AMX. The patch set on LKML establishes working AMX Linux support in public. I am thankful for your and other public review and feedback on that series. I can think of 3 actual bugs that were found in the process. thanks, Len Brown Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Fri, Apr 16, 2021 at 2:54 PM Len Brown wrote: > > On Thu, Apr 15, 2021 at 12:24 PM Andy Lutomirski wrote: > > On Wed, Apr 14, 2021 at 2:48 PM Len Brown wrote: > > > > > ... the transition penalty into and out of AMX code > > The concept of 'transition' exists between AVX and SSE instructions > because it is possible to mix both instruction sets and touch different > parts of the same registers. The "unused" parts of those registers > need to be tracked to assure that data is not lost when mixing. I get it. That does not explain why LDMXCSR and VLDMXCSR cause pipelines stalls. > > This concept is moot with AMX, which has its own dedicated registers. > > > What is the actual impact of a trivial function that initializes the > > tile config, does one tiny math op, and then does TILERELEASE? "does one tiny math op" AVX-512 *also* has sort-of-dedicated registers: ZMM16 and up. I still can't find any conclusive evidence as to whether that avoids the performance hit. Intel's track record at actually explaining what operations cause what particular performance disasters is poor, and your explanation is not helping the situation. Sorry.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Thu, Apr 15, 2021 at 12:24 PM Andy Lutomirski wrote: > On Wed, Apr 14, 2021 at 2:48 PM Len Brown wrote: > > > ... the transition penalty into and out of AMX code The concept of 'transition' exists between AVX and SSE instructions because it is possible to mix both instruction sets and touch different parts of the same registers. The "unused" parts of those registers need to be tracked to assure that data is not lost when mixing. This concept is moot with AMX, which has its own dedicated registers. > What is the actual impact of a trivial function that initializes the > tile config, does one tiny math op, and then does TILERELEASE? 1. Task takes #NM on first touch of TILE registers 2. Kernel allocates 8KB for that task and dis-arms XFD 3. Kernel context switches XFD with task state If the task takes a signal *before* TILERELEASE 4. XSAVE transfers AMX state to signal stack, XRESTOR the reverse. If the task context switches *before* TILERELEASE 5. kernel context switch XSAVES the AMX state to 8KB context switch buffer, XRESTORE the reverse. If the task takes a signal *after* TILERELEASE 4. XSAVE does NOT transfer AMX state (or zeros) to signal stack, 8KB is consumed on signal stack but not touched. XRESTOR, the reverse. If the task context switches *after* TILERELEASE 5. kernel contexts switch ignores INIT=1 AMX state. 8KB buffer is quiescent. As we discussed previously, there is no impact to frequency from either INIT=0 or INIT=1 AMX state. Frequency is impacted by *compute*, and since there isn't any compute this scenario, there is no frequency impact. As we discussed previously, for INIT=1 (which the kernel guarantees, there is also no impact on power. thanks, Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
> On Apr 15, 2021, at 10:00 AM, Dave Hansen wrote: > > On 4/15/21 9:24 AM, Andy Lutomirski wrote: >> In the patches, *as submitted*, if you trip the XFD #NM *once* and you >> are the only thread on the system to do so, you will eat the cost of a >> WRMSR on every subsequent context switch. > > I think you're saying: If a thread trips XFD #NM *once*, every switch to > and from *that* thread will incur the WRMSR cost. Indeed. My sentence was missing a few words at the end. > > The first time I read this, I thought you were saying that all threads > would incur a WRMSR cost on every context switch. If that's the case, I > grossly misread the patches. :)
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On 4/15/21 9:24 AM, Andy Lutomirski wrote: > In the patches, *as submitted*, if you trip the XFD #NM *once* and you > are the only thread on the system to do so, you will eat the cost of a > WRMSR on every subsequent context switch. I think you're saying: If a thread trips XFD #NM *once*, every switch to and from *that* thread will incur the WRMSR cost. The first time I read this, I thought you were saying that all threads would incur a WRMSR cost on every context switch. If that's the case, I grossly misread the patches. :)
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Wed, Apr 14, 2021 at 2:48 PM Len Brown wrote:
>
>
> > Then I take the transition penalty into and out of AMX code (I'll
> > believe there is no penalty when I see it -- we've had a penalty with
> > VEX and with AVX-512) and my program runs *slower*.
>
> If you have a clear definition of what "transition penalty" is, please share
> it.
Given the generally awful state of Intel's documentation about these
issues, it's quite hard to tell for real. But here are some examples.
VEX: Figures 11-1 ("AVX-SSE Transitions in the Broadwell, and Prior
Generation Microarchitectures") and 11-2 ("AVX-SSE Transitions in the
Skylake Microarchitecture"). We *still* have a performance regression
in the upstream kernel because, despite all common sense, the CPUs
consider LDMXCSR to be an SSE instruction and VLDMXCSR to be an AVX
instruction despite the fact that neither one of them touch the XMM or
YMM state at all.
AVX-512:
https://lore.kernel.org/linux-crypto/calcetru06cuvuf5ndsm8--dy3dokxyq88cgwaakoque4vkz...@mail.gmail.com/
https://travisdowns.github.io/blog/2020/01/17/avxfreq1.html
>
> Lacking one, I'll assume you are referring to the
> impact on turbo frequency of using AMX hardware?
>
> Again...
>
> On the hardware that supports AMX, there is zero impact on frequency
> due to the presence of AMX state, whether modified or unmodified.
>
> We resolved on another thread that Linux will never allow entry
> into idle with modified AMX state, and so AMX will have zero impact
> on the ability of the process to enter deep power-saving C-states.
>
> It is true that AMX activity is considered when determining max turbo.
> (as it must be)
> However, the *release* of the turbo credits consumed by AMX is
> "several orders of magnitude" faster on this generation
> than it was for AVX-512 on pre-AMX hardware.
What is the actual impact of a trivial function that initializes the
tile config, does one tiny math op, and then does TILERELEASE?
> Yes, the proposal, and the working patch set on the list, context
> switches XFD -- which is exactly what that hardware was designed to do.
> If the old and new tasks have the same value of XFD, the MSR write is skipped.
>
> I'm not aware of any serious proposal to context-switch XCR0,
> as it would break the current programming model, where XCR0
> advertises what the OS supports. It would also impact performance,
> as every write to XCR0 necessarily provokes a VMEXIT.
You're arguing against a nonsensical straw man.
In the patches, *as submitted*, if you trip the XFD #NM *once* and you
are the only thread on the system to do so, you will eat the cost of a
WRMSR on every subsequent context switch. This is not free. If we
use XCR0 (I'm not saying we will -- I'm just mentioning at a
possibility), then the penalty is presumably worse due to the VMX
issue.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Thu, Apr 15, 2021 at 07:29:38AM +0200, Willy Tarreau wrote: > What Len is saying is that not being interested in a feature is not an > argument for rejecting its adoption, Oh, I'm not rejecting its adoption - no, don't mean that. > which I'm perfectly fine with. But conversely not being interested in > a feature is also an argument for insisting that its adoption doesn't > harm other use cases (generally speaking, not this specific case > here). Pretty much. What I'd like to see is 0-overhead for current use cases and only overhead for those who want to use it. If that can't be done automagically, then users should request it explicitly. So basically you blow up the xsave buffer only for processes which want to do AMX. And this brings the question about libraries which, if they start using AMX by default - which doesn't sound like they will want to because AMX reportedly will have only a limited? set of users - if libraries start using it by default, then it better be worth the handling of the 8kb buffer per process. If not, this should also be requestable per process so that a simple pipe in Linux: | grep | awk | sed ... and so on is not penalized to allocate and handle by default 8kb for *each* process' buffer in that pipe just because each is linking against glibc which has detected AMX support in CPUID and is using it too for some weird reason like some microbenchmark saying so. All AFAIU, ofc. But my initial question was on the "establishing" part and was asking where we have established anything wrt AMX. Thx. -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Thu, Apr 15, 2021 at 06:43:43AM +0200, Borislav Petkov wrote: > On Wed, Apr 14, 2021 at 05:57:22PM -0400, Len Brown wrote: > > I'm pretty sure that the "it isn't my use case of interest, so it > > doesn't matter" line of reasoning has long been established as -EINVAL > > ;-) > > I have only a very faint idea what you're trying to say here. Please > explain properly and more verbosely what exactly has been established > where? What Len is saying is that not being interested in a feature is not an argument for rejecting its adoption, which I'm perfectly fine with. But conversely not being interested in a feature is also an argument for insisting that its adoption doesn't harm other use cases (generally speaking, not this specific case here). Willy
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Wed, Apr 14, 2021 at 05:57:22PM -0400, Len Brown wrote: > I'm pretty sure that the "it isn't my use case of interest, so it > doesn't matter" line of reasoning has long been established as -EINVAL > ;-) I have only a very faint idea what you're trying to say here. Please explain properly and more verbosely what exactly has been established where? Thx. -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Wed, Apr 14, 2021 at 5:58 AM Borislav Petkov wrote: > > On Tue, Apr 13, 2021 at 03:51:50PM -0400, Len Brown wrote: > > AMX does the type of matrix multiplication that AI algorithms use. In > > the unlikely event that you or one of the libraries you call are doing > > the same, then you will be very happy with AMX. Otherwise, you'll > > probably not use it. > > Which sounds to me like AMX is something which should not be enabled > automatically but explicitly requested. I don't see the majority of the > processes on the majority of the Linux machines out there doing AI with > AMX - at least not anytime soon. If it becomes ubiquitous later, we can > make it automatic then. I'm pretty sure that the "it isn't my use case of interest, so it doesn't matter" line of reasoning has long been established as -EINVAL ;-)
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Tue, Apr 13, 2021 at 6:59 PM Andy Lutomirski wrote: > Suppose I write some user code and call into a library that uses AMX > because the library authors benchmarked it and determined that using > AMX is faster when called in a loop. But I don't call it in a loop. Again... AMX registers are volatile. That means if a routine uses them and expects them to persist across a call, the caller must save them. It also means that they can't be used as globals, they can't be used for passing parameters, and they can't be used for static information in a called routine. And so a routine that uses AMX starts from scratch, and finishes with no state in registers. Everything gets loaded from memory into AMX, joyous number crunching proceeds, and the answer is stored in memory when the routine is complete. Could somebody write a routine that uses AMX to perform a single multiply, and then call that routine from a loop? Yes, that is possible, but I would challenge them to demonstrate it is a good idea under *any* conditions. More realistic, perhaps would be a routine that does a matrix multiply and is passed matrices of variable size. It would be easy to demonstrate that is a performance win on a big matrix, but then what happens when somebody calls that routine with a matrix size of 1? Surely that would be a net loss. This is pretty much exactly what Willy described. It is faster to copy a 5-byte structure by hand, than to call bcopy(). Indeed, dedicated co-processors have been built to do what AMX does, but the work has to be big enough to out-weigh the overhead of invoking them. This trade-off is as old as the hills, and yes, it is possible to screw up. > Then I take the transition penalty into and out of AMX code (I'll > believe there is no penalty when I see it -- we've had a penalty with > VEX and with AVX-512) and my program runs *slower*. If you have a clear definition of what "transition penalty" is, please share it. Lacking one, I'll assume you are referring to the impact on turbo frequency of using AMX hardware? Again... On the hardware that supports AMX, there is zero impact on frequency due to the presence of AMX state, whether modified or unmodified. We resolved on another thread that Linux will never allow entry into idle with modified AMX state, and so AMX will have zero impact on the ability of the process to enter deep power-saving C-states. It is true that AMX activity is considered when determining max turbo. (as it must be) However, the *release* of the turbo credits consumed by AMX is "several orders of magnitude" faster on this generation than it was for AVX-512 on pre-AMX hardware. I respect your right to not believe me about performance until you have this hardware. But proposing a new ABI based on concern of a problem that hasn't been shown to exist would be folly. > And, to top it > off, I've just permanently allocated 8kB of extra FPU state buffer, > *and* I'm taking either an XCR0 or an XFD write penalty on every > future context switch. Again... We allocate an 8kB FPU state buffer for tasks that use AMX. We do not allocate that buffer for tasks that do not use AMX. If it turns out to be common that a long running task touches AMX once and never again, it would not be difficult to optimize for that case and free the buffer. Again... Yes, the proposal, and the working patch set on the list, context switches XFD -- which is exactly what that hardware was designed to do. If the old and new tasks have the same value of XFD, the MSR write is skipped. I'm not aware of any serious proposal to context-switch XCR0, as it would break the current programming model, where XCR0 advertises what the OS supports. It would also impact performance, as every write to XCR0 necessarily provokes a VMEXIT. > Someone or something needs to make a decision as to whether AMX should > actually be used for a given algorithm. The user library community > has swept this under the rug by declaring that libraries should use > the best-in-a-tight-loop code for the entire existence of extensions > beyond XMM, and the cost keeps getting higher. Is this a plea for library writers to run the simplest microbenchmarks to determine if their code makes any sense at all before releasing it? If so, I agree. > Perhaps some form of decision should be reached before AMX lands? > Landing AMX in its current form is a decision, and we should make a > credible effort to decide if it's the right one. Three questions come to mind: 1. Do we have a power or performance issue? 2. Do we have ABI breakage? 3. Can we do better in the long term? 1. Power or Performance issue? Per above, and multiple other threads, I'm not aware of any unresolved power or performance issues on AMX-capable hardware. 2. ABI Breakage? We all recognize that the signal.h hard-coded alt-sig-stack size ABI was ill conceived, and overlooked -- for decades. We all recognize that there are a non-zero number of applications that fail
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Wed, Apr 14, 2021 at 12:06:39PM +0200, Willy Tarreau wrote: > And change jobs :-) I think by the time that happens, we'll be ready to go to the eternal vacation. Which means: not my problem. :-))) -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Wed, Apr 14, 2021 at 11:58:04AM +0200, Borislav Petkov wrote: > On Tue, Apr 13, 2021 at 03:51:50PM -0400, Len Brown wrote: > > AMX does the type of matrix multiplication that AI algorithms use. In > > the unlikely event that you or one of the libraries you call are doing > > the same, then you will be very happy with AMX. Otherwise, you'll > > probably not use it. > > Which sounds to me like AMX is something which should not be enabled > automatically but explicitly requested. I don't see the majority of the > processes on the majority of the Linux machines out there doing AI with > AMX - at least not anytime soon. If it becomes ubiquitous later, we can > make it automatic then. And change jobs :-) Willy
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Tue, Apr 13, 2021 at 03:51:50PM -0400, Len Brown wrote: > AMX does the type of matrix multiplication that AI algorithms use. In > the unlikely event that you or one of the libraries you call are doing > the same, then you will be very happy with AMX. Otherwise, you'll > probably not use it. Which sounds to me like AMX is something which should not be enabled automatically but explicitly requested. I don't see the majority of the processes on the majority of the Linux machines out there doing AI with AMX - at least not anytime soon. If it becomes ubiquitous later, we can make it automatic then. Thx. -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Tue, Apr 13, 2021 at 3:47 PM Len Brown wrote: > > On Tue, Apr 13, 2021 at 4:16 PM Andy Lutomirski wrote: > > > > On Mon, Apr 12, 2021 at 4:46 PM Len Brown wrote: > > > > > > On Mon, Apr 12, 2021 at 11:21 AM Andy Lutomirski wrote: > > > > > > > AMX: Multiplying a 4x4 matrix probably looks *great* in a > > > > microbenchmark. Do it once and you permanently allocate 8kB (is that > > > > even a constant? can it grow in newer parts?), potentially hurts all > > > > future context switches, and does who-knows-what to Turbo licenses and > > > > such. > > > > > > Intel expects that AMX will be extremely valuable to key workloads. > > > It is true that you may never run that kind of workload on the machine > > > in front of you, > > > and so you have every right to be doubtful about the value of AMX. > > > > I fully believe that AMX will be amazing when used for the right > > workload. The problem is that a library may have no way to tell > > whether a workload is the type of computationally intensive workload > > for which it makes sense. Imagine you have a little function: > > > > int matrix_times_vector(int dim, float *out, const float *matrix, > > const float *vector); > > > > A clever library might use AMX for this. If dim == 4 and the caller > > is planning to call it in a long, tight loop, maybe this even makes > > sense. If dim == 4 and it's being called once, AMX is probably a > > losing proposition. With previous technologies, at least the impact > > was limited to the function itself and maybe once per call to the > > caller. But now, with AMX, the program that invoked this takes a > > performance and memory hit *forever* if it uses AMX once. > > Again... > > As this is a "clever" library, built with a clever toolchain, and the > result is that > TILERELEASE was properly issued at the end of computation. > Thus the hardware knows that the (volatile) AMX registers are no longer live. My argument has *nothing* to do with TILERELEASE. Let me try again. Suppose I write some user code an call into a library that uses AMX because the library authors benchmarked it and determined that using AMX is faster when called in a loop. But I don't call it in a loop. Then I take the transition penalty into and out of AMX code (I'll believe there is no penalty when I see it -- we've had a penalty with VEX and with AVX-512) and my program runs *slower*. And, to top it off, I've just permanently allocated 8kB of extra FPU state buffer, *and* I'm taking either an XCR0 or an XFD write penalty on every future context switch. Someone or something needs to make a decision as to whether AMX should actually be used for a given algorithm. The user library community has swept this under the rug by declaring that libraries should use the best-in-a-tight-loop code for the entire existence of extensions beyond XMM, and the cost keeps getting higher. > > Beyond that, we have the signal handling issue. > > I'm unaware of any unresolved feedback on the signal handling series > other than a wistful "wouldn't a new SIGFAIL be more clear (for future apps) > than the existing SIGSEGV?" I agree with this sentiment, but I don't > think we should hold up a patch to prevent corrupting user data > because a new signal number to describe the scenario doesn't exit. > Particularly since the new code that knows about the new SIGFAIL > will also be new code that has been compiled with the new glibc > that for most cases will prevent this scenario in the first place... > > > One solution, going > > off of what WIlly mentioned, is: > > > > bool amx_begin(void *signal_save_buffer); > > void amx_end(); > > > > In the amx_begin() region, if you get a signal, the AMX state is saved > > in the buffer. Outside the region, if you get a signal and AMX is in > > use, the kernel will either unceremoniously kill the task or will > > deliver SIGYOUBLEWIT. [0] > > I think it is clear that if a new signal ABI is going to be invented, > that it should be opt-in on state, so that it can run fast on machines > far into the future by not choosing to opt-in on anything. > > It isn't clear that changing the signal save state around critical regions > (in multiple threads) so that a single (per process definition) of a signal > handler gets a different result at different times is going to make that > (new) signal handler author especially happy. More likely they > either always want the state, or they do not. Perhaps some form of decision should be reached before AMX lands? Landing AMX in its current form is a decision, and we should make a credible effort to decide if it's the right one. --Andy
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Tue, Apr 13, 2021 at 4:16 PM Andy Lutomirski wrote: > > On Mon, Apr 12, 2021 at 4:46 PM Len Brown wrote: > > > > On Mon, Apr 12, 2021 at 11:21 AM Andy Lutomirski wrote: > > > > > AMX: Multiplying a 4x4 matrix probably looks *great* in a > > > microbenchmark. Do it once and you permanently allocate 8kB (is that > > > even a constant? can it grow in newer parts?), potentially hurts all > > > future context switches, and does who-knows-what to Turbo licenses and > > > such. > > > > Intel expects that AMX will be extremely valuable to key workloads. > > It is true that you may never run that kind of workload on the machine > > in front of you, > > and so you have every right to be doubtful about the value of AMX. > > I fully believe that AMX will be amazing when used for the right > workload. The problem is that a library may have no way to tell > whether a workload is the type of computationally intensive workload > for which it makes sense. Imagine you have a little function: > > int matrix_times_vector(int dim, float *out, const float *matrix, > const float *vector); > > A clever library might use AMX for this. If dim == 4 and the caller > is planning to call it in a long, tight loop, maybe this even makes > sense. If dim == 4 and it's being called once, AMX is probably a > losing proposition. With previous technologies, at least the impact > was limited to the function itself and maybe once per call to the > caller. But now, with AMX, the program that invoked this takes a > performance and memory hit *forever* if it uses AMX once. Again... As this is a "clever" library, built with a clever toolchain, and the result is that TILERELEASE was properly issued at the end of computation. Thus the hardware knows that the (volatile) AMX registers are no longer live. The XSAVE hardware recognizes this INIT=1 condition and transfers NO DATA "*forever*". This is true both on context switch (compacted) where it is automatic, and on (uncompacted) signal delivery, where we check for this case. Was that the "performance hit" of concern, or did I miss something? Yes, it is true that the kernel allocated a context switch buffer for the lifetime of that task, and it will not be freed until that task exits. If this proves to be an issue, there is nothing preventing us from implementing a re-claim scheme for a rarely used buffer. After recognizing this situation, we'd simply arm XFD, free the buffer, and from then onwards, the task behaves exactly as if had never touched AMX. However, nobody has yet suggested that would be a common situation worth an optimization to reclaim that task's 8KB. > Beyond that, we have the signal handling issue. I'm unaware of any unresolved feedback on the signal handling series other than a wistful "wouldn't a new SIGFAIL be more clear (for future apps) than the existing SIGSEGV?" I agree with this sentiment, but I don't think we should hold up a patch to prevent corrupting user data because a new signal number to describe the scenario doesn't exit. Particularly since the new code that knows about the new SIGFAIL will also be new code that has been compiled with the new glibc that for most cases will prevent this scenario in the first place... > One solution, going > off of what WIlly mentioned, is: > > bool amx_begin(void *signal_save_buffer); > void amx_end(); > > In the amx_begin() region, if you get a signal, the AMX state is saved > in the buffer. Outside the region, if you get a signal and AMX is in > use, the kernel will either unceremoniously kill the task or will > deliver SIGYOUBLEWIT. [0] I think it is clear that if a new signal ABI is going to be invented, that it should be opt-in on state, so that it can run fast on machines far into the future by not choosing to opt-in on anything. It isn't clear that changing the signal save state around critical regions (in multiple threads) so that a single (per process definition) of a signal handler gets a different result at different times is going to make that (new) signal handler author especially happy. More likely they either always want the state, or they do not. > I'm really hoping some userspace people can chime in. Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Apr 12, 2021 at 4:46 PM Len Brown wrote: > > On Mon, Apr 12, 2021 at 11:21 AM Andy Lutomirski wrote: > > > AMX: Multiplying a 4x4 matrix probably looks *great* in a > > microbenchmark. Do it once and you permanently allocate 8kB (is that > > even a constant? can it grow in newer parts?), potentially hurts all > > future context switches, and does who-knows-what to Turbo licenses and > > such. > > Intel expects that AMX will be extremely valuable to key workloads. > It is true that you may never run that kind of workload on the machine > in front of you, > and so you have every right to be doubtful about the value of AMX. I fully believe that AMX will be amazing when used for the right workload. The problem is that a library may have no way to tell whether a workload is the type of computationally intensive workload for which it makes sense. Imagine you have a little function: int matrix_times_vector(int dim, float *out, const float *matrix, const float *vector); A clever library might use AMX for this. If dim == 4 and the caller is planning to call it in a long, tight loop, maybe this even makes sense. If dim == 4 and it's being called once, AMX is probably a losing proposition. With previous technologies, at least the impact was limited to the function itself and maybe once per call to the caller. But now, with AMX, the program that invoked this takes a performance and memory hit *forever* if it uses AMX once. Beyond that, we have the signal handling issue. One solution, going off of what WIlly mentioned, is: bool amx_begin(void *signal_save_buffer); void amx_end(); In the amx_begin() region, if you get a signal, the AMX state is saved in the buffer. Outside the region, if you get a signal and AMX is in use, the kernel will either unceremoniously kill the task or will deliver SIGYOUBLEWIT. [0] I'm really hoping some userspace people can chime in. [0] We really ought to have a SIGSIGNALFAILURE or something for the case where normal signal delivery fails. This is the userspace equivalent of #DF. SIGYOUBLEWIT could be folded in. There would be a flag in the signal frame saying "don't even try to sigreturn".
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
Thanks for sharing your perspective, Willy. I agree that if your application is so sensitive that you need to hand-code your own memcmp, then linking with (any) new version of (any) dynamic library is a risk you must consider carefully. AMX does the type of matrix multiplication that AI algorithms use. In the unlikely event that you or one of the libraries you call are doing the same, then you will be very happy with AMX. Otherwise, you'll probably not use it. I acknowledge the issue with the toolchain transparently using AVX-512 for copying data, and how that approach impacted systems with a poor AVX-512 hardware implementation. FWIW. I'm not aware of any plans to implicitly use AMX this way, and I'm not aware of any non-Xeon AMX implementations in the near future. cheers, Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Apr 12, 2021 at 07:46:06PM -0400, Len Brown wrote: > On Mon, Apr 12, 2021 at 11:21 AM Andy Lutomirski wrote: > > > AMX: Multiplying a 4x4 matrix probably looks *great* in a > > microbenchmark. Do it once and you permanently allocate 8kB (is that > > even a constant? can it grow in newer parts?), potentially hurts all > > future context switches, and does who-knows-what to Turbo licenses and > > such. > > Intel expects that AMX will be extremely valuable to key workloads. > It is true that you may never run that kind of workload on the machine > in front of you, > and so you have every right to be doubtful about the value of AMX. > > The AMX architectural state size is not expected to change. > Rather, if a "new AMX" has a different state size, it is expected to > use a new feature bit, different from AMX. > > The AMX context switch buffer is allocated only if and when a task > touches AMX registers. > > Yes, there will be data transfer to and from that buffer when three > things all happen. > 1. the data is valid > 2. hardware interrupts the application > 3. the kernel decides to context switch. As a userspace developer of a proxy, my code is extremely sensitive to syscall cost and works in environments where 1 million interrupts/s is not uncommon. Additionally the data I process are small HTTP headers and I already had to reimplement my own byte-level memcmp because the overhead of some libc to decide what variant to use to compare 5 bytes was higher than the time to iterate over them. So I'm among those userspace developers who grumble each time new technology is automatically adopted by the compiler and libs, because that tends to make me figure what the impact is and how to work around it. I have no idea what AMX could bring me but reading this above makes me think that it has a great potential of significantly hurting the performance if one lib decides to occasionally make use of it. It would possibly be similar if a lib decided to use AVX-512 to copy data and if it resulted in the CPU quickly reaching its TDP and starting to throttle like crazy :-/ Thus I think that the first thing to think about before introducing possibly cost-sensitive optimizations is : how do I allow easily user-space to easily disable them for a task, and how do I allow an admin to easily disable them system-wide. "echo !foobar > cpuinfo" could be a nice way to mask a flag system-wide for example. prctl() would be nice for a task (as long as it's not too late already). Maybe the API should be surrounded by __amx_begin() / __amx_end() and the calls having undefined behavior outside of these. These flags would put a flag somewhere asking to extend the stacks, or __amx_begin() could even point itself to the specific stack to be used. This way it could possibly allow some userspace libraries to use it for small stuff without definitely impacting the rest of the process. > At the risk of stating the obvious... > Intel's view is that libraries that deliver the most value from the > hardware are a "good thing", > and that anything preventing libraries from getting the most value > from the hardware is a "bad thing":-) As a developer I have a different view. Anything that requires to build using different libraries depending on the systems is a real hassle, and I want to focus on the same code to run everywhere. I'm fine with some #ifdef in the code if I know that a specific part must run as fast as possible, and even some runtime detection at various points but do not want to have to deal with extra dependencies that further increase the test matrix and combinations in bug reports. Just my two cents, Willy
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Apr 12, 2021 at 8:17 PM Thomas Gleixner wrote: > I'm fine with that as long the kernel has a way to detect that and can > kill the offending application/library combo with an excplicit > -SIG_NICE_TRY. Agreed. The new run-time check for altsigstack overflow is one place we can do that. Let me know if you think of others, thanks, Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Apr 12 2021 at 19:46, Len Brown wrote: > On Mon, Apr 12, 2021 at 11:21 AM Andy Lutomirski wrote: >> Even putting aside all kernel and ABI issues, is it actually a good >> idea for user libraries to transparently use these new features? I'm >> not really convinced. I think that serious discussion among userspace >> people is needed. > > At the risk of stating the obvious... > Intel's view is that libraries that deliver the most value from the > hardware are a "good thing", > and that anything preventing libraries from getting the most value > from the hardware is a "bad thing":-) Sure, and as a consequence the kernel is the problem when creative libraries cause wreckage along the way. I'm fine with that as long the kernel has a way to detect that and can kill the offending application/library combo with an excplicit -SIG_NICE_TRY. Thanks, tglx
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Apr 12, 2021 at 11:21 AM Andy Lutomirski wrote: > AMX: Multiplying a 4x4 matrix probably looks *great* in a > microbenchmark. Do it once and you permanently allocate 8kB (is that > even a constant? can it grow in newer parts?), potentially hurts all > future context switches, and does who-knows-what to Turbo licenses and > such. Intel expects that AMX will be extremely valuable to key workloads. It is true that you may never run that kind of workload on the machine in front of you, and so you have every right to be doubtful about the value of AMX. The AMX architectural state size is not expected to change. Rather, if a "new AMX" has a different state size, it is expected to use a new feature bit, different from AMX. The AMX context switch buffer is allocated only if and when a task touches AMX registers. Yes, there will be data transfer to and from that buffer when three things all happen. 1. the data is valid 2. hardware interrupts the application 3. the kernel decides to context switch. There will be no data transfer of AMX architectural state when it is in INIT state. As AMX registers are volatile, correct software will always have them in INIT state before calls, including system calls. I've addressed turbo licenses already. > Even putting aside all kernel and ABI issues, is it actually a good > idea for user libraries to transparently use these new features? I'm > not really convinced. I think that serious discussion among userspace > people is needed. At the risk of stating the obvious... Intel's view is that libraries that deliver the most value from the hardware are a "good thing", and that anything preventing libraries from getting the most value from the hardware is a "bad thing":-) cheers, Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Sun, Apr 11, 2021, Len Brown wrote: > On Fri, Apr 9, 2021 at 5:44 PM Andy Lutomirski wrote: > > > > On Fri, Apr 9, 2021 at 1:53 PM Len Brown wrote: > > > > > > On Wed, Mar 31, 2021 at 6:45 PM Andy Lutomirski wrote: > > > > > > > > On Wed, Mar 31, 2021 at 3:28 PM Len Brown wrote: > > > > > We've also established that when running in a VMM, every update to > > > > > XCR0 causes a VMEXIT. > > > > > > > > This is true, it sucks, and Intel could fix it going forward. > > > > > > What hardware fix do you suggest? > > > If a guest is permitted to set XCR0 bits without notifying the VMM, > > > what happens when it sets bits that the VMM doesn't know about? > > > > The VM could have a mask of allowed XCR0 bits that don't exist. > > > > TDX solved this problem *somehow* -- XSETBV doesn't (visibly?) exit on > > TDX. Surely plain VMX could fix it too. > > There are two cases. > > 1. Hardware that exists today and in the foreseeable future. > > VM modification of XCR0 results in VMEXIT to VMM. > The VMM sees bits set by the guest, and so it can accept what > it supports, or send the VM a fault for non-support. > > Here it is not possible for the VMM to change XCR0 without the VMM knowing. > > 2. Future Hardware that allows guests to write XCR0 w/o VMEXIT. > > Not sure I follow your proposal. > > Yes, the VM effectively has a mask of what is supported, > because it can issue CPUID. > > The VMM virtualizes CPUID, and needs to know it must not > expose to the VM any state features it doesn't support. > Also, the VMM needs to audit XCR0 before it uses XSAVE, > else the guest could attack or crash the VMM through > buffer overrun. The VMM already needs to context switch XCR0 and XSS, so this is a non-issue. > Is this what you suggest? Yar. In TDX, XSETBV exits, but only to the TDX module. I.e. TDX solves the problem in software by letting the VMM tell the TDX module what features the guest can set in XCR0/XSS via the XFAM (Extended Features Allowed Mask). But, that software "fix" can also be pushed into ucode, e.g. add an XFAM VMCS field, the guest can set any XCR0 bits that are '1' in VMCS.XFAM without exiting. Note, SGX has similar functionality in the form of XFRM (XSAVE-Feature Request Mask). The enclave author can specify what features will be enabled in XCR0 when the enclave is running. Not that relevant, other than to reinforce that this is a solvable problem. > If yes, what do you suggest in the years between now and when > that future hardware and VMM exist? Burn some patch space? :-)
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Apr 12, 2021 at 7:19 AM Florian Weimer wrote: > > * Andy Lutomirski: > > Maybe we could have done this in 2016 when I reported this for the first > time. Now it is too late, as more and more software is using > CPUID-based detection for AVX-512. Our users have been using AVX-512 > hardware for quite some time now, and I haven't seen *that* many issues > resulting from the context size. That isn't to say that problems do not > exist, but they are more of the kind that the increased stack usage > means that areas of the stack that used to be zero no longer are, so > users encounter different side effects from uninitialized-variable bugs. > > How much software depends on the signal handler data layout? The %zmm > state does not appear to be exposed today, so perhaps some savings could > be had there. The fact that including is barely functional in glibc probably helps keep software from touching the state. :) > > The suggestion to make CPUID trap doesn't sound workable to me. At > least in the past, it's been suggested as a serializing instruction to > be used alongside RDTSC, which makes it rather time-critical for some > applications. > > Even today, signal handlers do not really compose well in the sense that > multiple libraries can use them and collaborate without being aware of > each other (like they can divide up TLS memory with the help of the > dynamic linker, or carve out address space using mmap). Proposals to > set special process-wide flags only make that situation worse. Code > that installs a signal handler often does not have control on which > thread an asynchronous signal is delivered, or which code it interrupts. > A single process-wide flag cannot capture that accurately, even if it is > per signal number. I think this would want to be a per-signal-handler flag, not per process. It's entirely possible to write a signal handler callback that doesn't touch AVX512 or AMX state, even if the toolchain may make it annoying. That specific handler could set the "make me fast" flag. > > The rseq extension might work conceptually, but it requires to make > operations idempotent, with periodic checkpoint, and of course > inline/flatten all calls. And it requires compiler work, the present > model based on inline asm hacks doesn't look workable. Maybe that works > for AMX. I have not checked if there is yet any public documentation of > the programming model. I tend to think that the rseq model will be unworkable. People trying to use the new instructions will hate it. > > I think someone expressed the sentiment (maybe on another thread) that > the current CPU feature enablement process does not work. I do not > agree. Currently it is only necessary to upgrade the kernel and maybe > glibc (but not in all cases), and then you are good to go. You can keep > using your old libraries, old compilers, and even old assemblers if you > are okay with .byte hacks. You do not need special userspace libraries, > new compilers for different languages, special firmware or binary blobs. > Overall, it just works. > > On x86, we are really bad about actually using CPU features pervasively, > but that is a different story. > "Just works" is different from "is a good idea", though. With SSE2 and other non-VEX xmm extensions, just using them in userspace seems quite reasonable. If a function could run faster using xmm, then it might as well use xmm. But this model starts to break down with newer features: VEX: ymm (AFAIK) performs just fine, at least on most CPUs, except that mixing VEX and non-VEX code has big penalties. Copying that 64-bit data structure using ymm is not necessarily wise even if it microbenchmarks well. Heck, mixing translation units using normal C floating point code that were compiled with different flags can be quite slow. AVX-512: Intel has still not responded to my requests for detailed documentation of the performance issues. The internet is full of various reports and various voodoo ideas. VZEROALL does not do what one would naively expect, and the implications are unclear. AVX-512 code, even used just once, is likely to permanently bloat the signal state. Even ignoring the unknowns here, on most current non-Xeon-phi parts AFAICT, using small bits of AVX-512 code has *huge* performance impacts. Libraries automatically using AVX-512 just because it's there is not necessarily a good idea, even if it microbenchmarks well. AMX: Multiplying a 4x4 matrix probably looks *great* in a microbenchmark. Do it once and you permanently allocate 8kB (is that even a constant? can it grow in newer parts?), potentially hurts all future context switches, and does who-knows-what to Turbo licenses and such. Even putting aside all kernel and ABI issues, is it actually a good idea for user libraries to transparently use these new features? I'm not really convinced. I think that serious discussion among userspace people is needed. --Andy
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
> On Apr 12, 2021, at 7:38 AM, Florian Weimer wrote: > > * Borislav Petkov: > >>> On Mon, Apr 12, 2021 at 04:19:29PM +0200, Florian Weimer wrote: >>> Maybe we could have done this in 2016 when I reported this for the first >>> time. Now it is too late, as more and more software is using >>> CPUID-based detection for AVX-512. >> >> So as I said on another mail today, I don't think a library should rely >> solely on CPUID-based detection of features especially if those features >> need kernel support too. IOW, it should ask whether the kernel can >> handle those too, first. > > Yes, that's why we have the XGETBV handshake. I was imprecise. It's > CPUID + XGETBV of course. Or even AT_HWCAP2 (for FSGSBASE). > >> And the CPUID-faulting thing would solve stuff like that because then >> the kernel can *actually* get involved into answering something where it >> has a say in, too. > > But why wouldn't we use a syscall or an entry in the auxiliary vector > for that? Why fault a potentially performance-critical instruction? > CPUID is horrifically slow in various virt scenarios. If user code needs to serialize, use IRET or SERIALIZE.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Apr 12, 2021 at 04:38:15PM +0200, Florian Weimer wrote: > Yes, that's why we have the XGETBV handshake. I was imprecise. It's > CPUID + XGETBV of course. Or even AT_HWCAP2 (for FSGSBASE). Ok, that sounds better. So looking at glibc sources, I see something like this: init_cpu_features |-> update_usable |-> CPU_FEATURE_SET_USABLE (cpu_features, XGETBV_ECX_1); so I'm guessing this happens when the library gets loaded per process, right? Which means, once the detection has taken place and the library has gotten XCR0, it is going to use it and won't re-ask the kernel or so? I.e., I'm trying to imagine how a per-process thing would work at all. If at all. And this sounds especially "fun": > Code that installs a signal handler often does not have control on > which thread an asynchronous signal is delivered, or which code it > interrupts. In my simplistic approach I'm thinking about something along the lines of: Library: hey kernel, can you handle AVX512? Kernel: yes Library: ok, I will use that in the signal handler And since kernel has said yes, kernel is going to take care of handling AVX512 state and library can assume that. All those old processes which cannot be recompiled, for them I guess the kernel should have to say no. Dunno how much sense that makes... > > And the CPUID-faulting thing would solve stuff like that because then > > the kernel can *actually* get involved into answering something where it > > has a say in, too. > > But why wouldn't we use a syscall or an entry in the auxiliary vector > for that? Why fault a potentially performance-critical instruction? Oh sure, CPUID faulting was just an example. I think the intent is to have this important aspect of userspace asking the kernel first what kind of features it can handle and then do accordingly. IOW, legacy stuff can work unchanged and new libraries and kernels can support fancier features and bigger buffers. Methinks. Thx. -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
* Borislav Petkov: > On Mon, Apr 12, 2021 at 04:19:29PM +0200, Florian Weimer wrote: >> Maybe we could have done this in 2016 when I reported this for the first >> time. Now it is too late, as more and more software is using >> CPUID-based detection for AVX-512. > > So as I said on another mail today, I don't think a library should rely > solely on CPUID-based detection of features especially if those features > need kernel support too. IOW, it should ask whether the kernel can > handle those too, first. Yes, that's why we have the XGETBV handshake. I was imprecise. It's CPUID + XGETBV of course. Or even AT_HWCAP2 (for FSGSBASE). > And the CPUID-faulting thing would solve stuff like that because then > the kernel can *actually* get involved into answering something where it > has a say in, too. But why wouldn't we use a syscall or an entry in the auxiliary vector for that? Why fault a potentially performance-critical instruction? Thanks, Florian
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Apr 12, 2021 at 04:19:29PM +0200, Florian Weimer wrote: > Maybe we could have done this in 2016 when I reported this for the first > time. Now it is too late, as more and more software is using > CPUID-based detection for AVX-512. So as I said on another mail today, I don't think a library should rely solely on CPUID-based detection of features especially if those features need kernel support too. IOW, it should ask whether the kernel can handle those too, first. And the CPUID-faulting thing would solve stuff like that because then the kernel can *actually* get involved into answering something where it has a say in, too. Thx. -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Sun, Apr 11, 2021 at 03:07:29PM -0400, Len Brown wrote: > If it is the program, how does it know that the library wants to use > what instructions? > > If it is the library, then you have just changed XCR0 at run-time and > you expose breakage of the thread library that has computed XSAVE size. So, when old programs which cannot possibly know about the arch_prctl() extension we're proposing here, link against that library, then that library should not be allowed to go use "fat" states. Unless the library can "tell" the process which links to it, that it has dynamically enlarged the save state. If it can and the process can handle that, then all is fine, save state gets enlarged dynamically and it all continues merrily. Also, in order for the library to use fat states, it needs to ask the kernel for such support - not CPUID - because the kernel is doing the state handling for everybody and doing all the CR4.OSXSAVE setup etc. Which also means that the kernel can help here by telling the library: - No, you cannot use fat states with this process because it hasn't called arch_prctl() so it cannot handle them properly. - Yes, this process allowes me to handle fat states for it so you can use those states and thus those instructions when doing operations for it. So the kernel becomes the arbiter in all this - as it should be - and then all should work fine. -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
RE: Candidate Linux ABI for Intel AMX and hypothetical new related features
From: Len Brown > Sent: 11 April 2021 20:07 ... > Granted, if you have an application that is statically linked and run > on new hardware > and new OS, it can still fail. That also includes anything compiled and released as a program binary that must run on older Linux installations. Such programs have to be compiled with old copies of the system headers (and probably with an of gcc). While such programs themselves won't use AVX without checking for OS support, the glibc code on the installed system might. Such programs can be modified to run-time detect the required signal stack size - but cannot rely on glibc to convert SIGSTKSZ into a function call. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Fri, Apr 9, 2021 at 5:44 PM Andy Lutomirski wrote: > > On Fri, Apr 9, 2021 at 1:53 PM Len Brown wrote: > > > > On Wed, Mar 31, 2021 at 6:45 PM Andy Lutomirski wrote: > > > > > > On Wed, Mar 31, 2021 at 3:28 PM Len Brown wrote: > > > > > > > We added compiler annotation for user-level interrupt handlers. > > > > I'm not aware of it failing, or otherwise being confused. > > > > > > I followed your link and found nothing. Can you elaborate? In the > > > kernel, we have noinstr, and gcc gives approximately no help toward > > > catching problems. > > > > A search for the word "interrupt" on this page > > https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.html#x86-Function-Attributes > > comes to the description of this attribute: > > > > __attribute__ ((interrupt)) > > > > I read that and I see no mention of anything saying "this will > generate code that does not touch extended state". Instead I see, > paraphrasing, "this will generate code with an ABI that is completely > inappropriate for use in a user space signal handler". Am I missing > something? Again... An analogous annotation could be created for fast signals. gcc can be told exactly what registers and instructions it can use for that routine. If somebody can suggest a way to make fast signal handers faster than saving only the state that they-themselves actually use, I'm all ears. > > > > dynamic XCR0 breaks the installed base, I thought we had established > > > > that. > > > > > > I don't think this is at all established. If some code thinks it > > > knows the uncompacted XSTATE size and XCR0 changes, it crashes. This > > > is not necessarily a showstopper. > > > > My working assumption is that crashing applications actually *is* a > > showstopper. > > Please clarify. > > I think you're presuming that some program actually does this. If no > program does this, it's not an ABI break. So you agree that for a program that uses xgetbv to read XCR0 and compute XSTATE size for user-space use of XSAVE can break if XCR0 changes during its lifetime. But you don't believe such software exists? > More relevantly, this can only happen in a process that uses XSAVE and > thinks it knows the size that *also* does the prctl to change XCR0. > By construction, existing programs can't break unless they load new > dynamic libraries that break them. Let's say that a program does math. It calls a library to do that math. It doesn't know or care what instructions the library uses to do math. eg. the library uses SSE on an Atom, and uses AVX512 on a Xeon. Who calls the new prctl, the program, or the library? If it is the program, how does it know that the library wants to use what instructions? If it is the library, then you have just changed XCR0 at run-time and you expose breakage of the thread library that has computed XSAVE size. > > > > We've also established that when running in a VMM, every update to > > > > XCR0 causes a VMEXIT. > > > > > > This is true, it sucks, and Intel could fix it going forward. > > > > What hardware fix do you suggest? > > If a guest is permitted to set XCR0 bits without notifying the VMM, > > what happens when it sets bits that the VMM doesn't know about? > > The VM could have a mask of allowed XCR0 bits that don't exist. > > TDX solved this problem *somehow* -- XSETBV doesn't (visibly?) exit on > TDX. Surely plain VMX could fix it too. There are two cases. 1. Hardware that exists today and in the foreseeable future. VM modification of XCR0 results in VMEXIT to VMM. The VMM sees bits set by the guest, and so it can accept what it supports, or send the VM a fault for non-support. Here it is not possible for the VMM to change XCR0 without the VMM knowing. 2. Future Hardware that allows guests to write XCR0 w/o VMEXIT. Not sure I follow your proposal. Yes, the VM effectively has a mask of what is supported, because it can issue CPUID. The VMM virtualizes CPUID, and needs to know it must not expose to the VM any state features it doesn't support. Also, the VMM needs to audit XCR0 before it uses XSAVE, else the guest could attack or crash the VMM through buffer overrun. Is this what you suggest? If yes, what do you suggest in the years between now and when that future hardware and VMM exist? > > > > I thought the goal was to allow new programs to have fast signal > > > > handlers. > > > > By default, those fast signal handlers would have a stable state > > > > image, and would > > > > not inherit large architectural state on their stacks, and could thus > > > > have minimal overhead on all hardware. > > > > > > That is *a* goal, but not necessarily the only goal. > > > > I fully support coming up with a scheme for fast future-proof signal > > handlers, > > and I'm willing to back that up by putting work into it. > > > > I don't see any other goals articulated in this thread. > > Before we get too carried away with *fast* signal handlers, something > that works with existing programs is also a
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Fri, Apr 9, 2021 at 1:53 PM Len Brown wrote: > > On Wed, Mar 31, 2021 at 6:45 PM Andy Lutomirski wrote: > > > > On Wed, Mar 31, 2021 at 3:28 PM Len Brown wrote: > > > > > We added compiler annotation for user-level interrupt handlers. > > > I'm not aware of it failing, or otherwise being confused. > > > > I followed your link and found nothing. Can you elaborate? In the > > kernel, we have noinstr, and gcc gives approximately no help toward > > catching problems. > > A search for the word "interrupt" on this page > https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.html#x86-Function-Attributes > comes to the description of this attribute: > > __attribute__ ((interrupt)) > I read that and I see no mention of anything saying "this will generate code that does not touch extended state". Instead I see, paraphrasing, "this will generate code with an ABI that is completely inappropriate for use in a user space signal handler". Am I missing something? > > > dynamic XCR0 breaks the installed base, I thought we had established that. > > > > I don't think this is at all established. If some code thinks it > > knows the uncompacted XSTATE size and XCR0 changes, it crashes. This > > is not necessarily a showstopper. > > My working assumption is that crashing applications actually *is* a > showstopper. > Please clarify. I think you're presuming that some program actually does this. If no program does this, it's not an ABI break. More relevantly, this can only happen in a process that uses XSAVE and thinks it knows the size that *also* does the prctl to change XCR0. By construction, existing programs can't break unless they load new dynamic libraries that break them. > > > > We've also established that when running in a VMM, every update to > > > XCR0 causes a VMEXIT. > > > > This is true, it sucks, and Intel could fix it going forward. > > What hardware fix do you suggest? > If a guest is permitted to set XCR0 bits without notifying the VMM, > what happens when it sets bits that the VMM doesn't know about? The VM could have a mask of allowed XCR0 bits that don't exist. TDX solved this problem *somehow* -- XSETBV doesn't (visibly?) exit on TDX. Surely plain VMX could fix it too. > > > > I thought the goal was to allow new programs to have fast signal handlers. > > > By default, those fast signal handlers would have a stable state > > > image, and would > > > not inherit large architectural state on their stacks, and could thus > > > have minimal overhead on all hardware. > > > > That is *a* goal, but not necessarily the only goal. > > I fully support coming up with a scheme for fast future-proof signal handlers, > and I'm willing to back that up by putting work into it. > > I don't see any other goals articulated in this thread. Before we get too carried away with *fast* signal handlers, something that works with existing programs is also a pretty strong goal. RIght now AVX-512 breaks existing programs, even if they don't use AVX-512.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Wed, Mar 31, 2021 at 6:54 PM Borislav Petkov wrote: > > On Wed, Mar 31, 2021 at 06:28:27PM -0400, Len Brown wrote: > > dynamic XCR0 breaks the installed base, I thought we had established > > that. > > We should do a clear cut and have legacy stuff which has its legacy > expectations on the XSTATE layout and not touch those at all. > > And then all new apps which will use these new APIs can go and request > whatever fancy new state constellations we support. Including how they > want their signals handled, etc. > > Fat states like avx512, amx etc will be off by default and apps > explicitly requesting those, can get them. > > That's it. 100% agreement from me! (does anybody disagree?) thanks, Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Wed, Mar 31, 2021 at 6:45 PM Andy Lutomirski wrote: > > On Wed, Mar 31, 2021 at 3:28 PM Len Brown wrote: > > > We added compiler annotation for user-level interrupt handlers. > > I'm not aware of it failing, or otherwise being confused. > > I followed your link and found nothing. Can you elaborate? In the > kernel, we have noinstr, and gcc gives approximately no help toward > catching problems. A search for the word "interrupt" on this page https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.html#x86-Function-Attributes comes to the description of this attribute: __attribute__ ((interrupt)) > > dynamic XCR0 breaks the installed base, I thought we had established that. > > I don't think this is at all established. If some code thinks it > knows the uncompacted XSTATE size and XCR0 changes, it crashes. This > is not necessarily a showstopper. My working assumption is that crashing applications actually *is* a showstopper. Please clarify. > > We've also established that when running in a VMM, every update to > > XCR0 causes a VMEXIT. > > This is true, it sucks, and Intel could fix it going forward. What hardware fix do you suggest? If a guest is permitted to set XCR0 bits without notifying the VMM, what happens when it sets bits that the VMM doesn't know about? > > I thought the goal was to allow new programs to have fast signal handlers. > > By default, those fast signal handlers would have a stable state > > image, and would > > not inherit large architectural state on their stacks, and could thus > > have minimal overhead on all hardware. > > That is *a* goal, but not necessarily the only goal. I fully support coming up with a scheme for fast future-proof signal handlers, and I'm willing to back that up by putting work into it. I don't see any other goals articulated in this thread. thanks, Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Wed, Mar 31, 2021 at 06:28:27PM -0400, Len Brown wrote: > dynamic XCR0 breaks the installed base, I thought we had established > that. We should do a clear cut and have legacy stuff which has its legacy expectations on the XSTATE layout and not touch those at all. And then all new apps which will use these new APIs can go and request whatever fancy new state constellations we support. Including how they want their signals handled, etc. Fat states like avx512, amx etc will be off by default and apps explicitly requesting those, can get them. That's it. Thx. -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Wed, Mar 31, 2021 at 3:28 PM Len Brown wrote:
>
> On Wed, Mar 31, 2021 at 12:53 PM Andy Lutomirski wrote:
>
> > But this whole annotation thing will require serious compiler support.
> > We already have problems with compilers inlining functions and getting
> > confused about attributes.
>
> We added compiler annotation for user-level interrupt handlers.
> I'm not aware of it failing, or otherwise being confused.
I followed your link and found nothing. Can you elaborate? In the
kernel, we have noinstr, and gcc gives approximately no help toward
catching problems.
>
> Why would compiler support for fast-signals be any more "serious"?
>
> > An API like:
> >
> > if (get_amx()) {
> > use AMX;
> > } else {
> > don’t;
> > }
> >
> > Avoids this problem. And making XCR0 dynamic, for all its faults, at least
> > helps force a degree of discipline on user code.
>
> dynamic XCR0 breaks the installed base, I thought we had established that.
I don't think this is at all established. If some code thinks it
knows the uncompacted XSTATE size and XCR0 changes, it crashes. This
is not necessarily a showstopper.
>
> We've also established that when running in a VMM, every update to
> XCR0 causes a VMEXIT.
This is true, it sucks, and Intel could fix it going forward.
>
> I thought the goal was to allow new programs to have fast signal handlers.
> By default, those fast signal handlers would have a stable state
> image, and would
> not inherit large architectural state on their stacks, and could thus
> have minimal overhead on all hardware.
That is *a* goal, but not necessarily the only goal.
--Andy
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Wed, Mar 31, 2021 at 12:53 PM Andy Lutomirski wrote:
> But this whole annotation thing will require serious compiler support.
> We already have problems with compilers inlining functions and getting
> confused about attributes.
We added compiler annotation for user-level interrupt handlers.
I'm not aware of it failing, or otherwise being confused.
Why would compiler support for fast-signals be any more "serious"?
> An API like:
>
> if (get_amx()) {
> use AMX;
> } else {
> don’t;
> }
>
> Avoids this problem. And making XCR0 dynamic, for all its faults, at least
> helps force a degree of discipline on user code.
dynamic XCR0 breaks the installed base, I thought we had established that.
We've also established that when running in a VMM, every update to
XCR0 causes a VMEXIT.
I thought the goal was to allow new programs to have fast signal handlers.
By default, those fast signal handlers would have a stable state
image, and would
not inherit large architectural state on their stacks, and could thus
have minimal overhead on all hardware.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Wed, Mar 31, 2021 at 5:42 PM Robert O'Callahan wrote: > > For the record, the benefits of dynamic XCR0 for rr recording > portability still apply. I guess it'd be useful for CRIU too. We would > also benefit from anything that incentivizes increased support for > CPUID faulting. As previously mentioned, today we don't have an architectural way to trap a user into the kernel on CPUID, even though we can do this for a VMM. But spoofing CPUID isn't a solution to all problems. The feature really needs to be OFF to prevent users from using it, even if the supported mechanisms of discovering that feature say "NOT PRESENT". Today there are plenty of users who will opportunistically try everything in the cloud and choose the machine that allows them to do something that other machines will not -- even if it is not officially supported. If something is not enumerated, it really needs to also be turned off. cheers, --Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
For the record, the benefits of dynamic XCR0 for rr recording portability still apply. I guess it'd be useful for CRIU too. We would also benefit from anything that incentivizes increased support for CPUID faulting. Rob -- Su ot deraeppa sah dna Rehtaf eht htiw saw hcihw, efil lanrete eht uoy ot mialcorp ew dna, ti ot yfitset dna ti nees evah ew; deraeppa efil eht. Efil fo Drow eht gninrecnoc mialcorp ew siht - dehcuot evah sdnah ruo dna ta dekool evah ew hcihw, seye ruo htiw nees evah ew hcihw, draeh evah ew hcihw, gninnigeb eht morf saw hcihw taht.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
> On Mar 31, 2021, at 9:31 AM, Len Brown wrote:
>
> On Tue, Mar 30, 2021 at 6:01 PM David Laight wrote:
>
>>> Can we leave it in live registers? That would be the speed-of-light
>>> signal handler approach. But we'd need to teach the signal handler to
>>> not clobber it. Perhaps that could be part of the contract that a
>>> fast signal handler signs? INIT=0 AMX state could simply sit
>>> patiently in the AMX registers for the duration of the signal handler.
>>> You can't get any faster than doing nothing :-)
>>>
>>> Of course part of the contract for the fast signal handler is that it
>>> knows that it can't possibly use XRESTOR of the stuff on the stack to
>>> necessarily get back to the state of the signaled thread (assuming we
>>> even used XSTATE format on the fast signal handler stack, it would
>>> forget the contents of the AMX registers, in this example)
>>
>> gcc will just use the AVX registers for 'normal' code within
>> the signal handler.
>> So it has to have its own copy of all the registers.
>> (Well, maybe you could make the TMX instructions fault,
>> but that would need a nested signal delivered.)
>
> This is true, by default, but it doesn't have to be true.
>
> Today, gcc has an annotation for user-level interrupts
> https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.html#x86-Function-Attributes
>
> An analogous annotation could be created for fast signals.
> gcc can be told exactly what registers and instructions it can use for
> that routine.
>
> Of course, this begs the question about what routines that handler calls,
> and that would need to be constrained too.
>
> Today signal-safety(7) advises programmers to limit what legacy signal
> handlers
> can call. There is no reason that a fast-signal-safety(7) could not be
> created
> for the fast path.
>
>> There is also the register save buffer that you need in order
>> to long-jump out of a signal handler.
>> Unfortunately that is required to work.
>> I'm pretty sure the original setjmp/longjmp just saved the stack
>> pointer - but that really doesn't work any more.
>>
>> OTOH most signal handlers don't care - but there isn't a flag
>> to sigset() (etc) so ask for a specific register layout.
>
> Right, the idea is to optimize for *most* signal handlers,
> since making any changes to *all* signal handlers is intractable.
>
> So the idea is that opting-in to a fast signal handler would opt-out
> of some legacy signal capibilities. Complete state is one of them,
> and thus long-jump is not supported, because the complete state
> may not automatically be available.
Long jump is probably the easiest problem of all: sigsetjmp() is a *function*,
following ABI, so sigsetjmp() is expected to clobber most or all of the
extended state.
But this whole annotation thing will require serious compiler support. We
already have problems with compilers inlining functions and getting confused
about attributes.
An API like:
if (get_amx()) {
use AMX;
} else {
don’t;
}
Avoids this problem. And making XCR0 dynamic, for all its faults, at least
helps force a degree of discipline on user code.
>
> thanks,
> Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Tue, Mar 30, 2021 at 6:01 PM David Laight wrote: > > Can we leave it in live registers? That would be the speed-of-light > > signal handler approach. But we'd need to teach the signal handler to > > not clobber it. Perhaps that could be part of the contract that a > > fast signal handler signs? INIT=0 AMX state could simply sit > > patiently in the AMX registers for the duration of the signal handler. > > You can't get any faster than doing nothing :-) > > > > Of course part of the contract for the fast signal handler is that it > > knows that it can't possibly use XRESTOR of the stuff on the stack to > > necessarily get back to the state of the signaled thread (assuming we > > even used XSTATE format on the fast signal handler stack, it would > > forget the contents of the AMX registers, in this example) > > gcc will just use the AVX registers for 'normal' code within > the signal handler. > So it has to have its own copy of all the registers. > (Well, maybe you could make the TMX instructions fault, > but that would need a nested signal delivered.) This is true, by default, but it doesn't have to be true. Today, gcc has an annotation for user-level interrupts https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.html#x86-Function-Attributes An analogous annotation could be created for fast signals. gcc can be told exactly what registers and instructions it can use for that routine. Of course, this begs the question about what routines that handler calls, and that would need to be constrained too. Today signal-safety(7) advises programmers to limit what legacy signal handlers can call. There is no reason that a fast-signal-safety(7) could not be created for the fast path. > There is also the register save buffer that you need in order > to long-jump out of a signal handler. > Unfortunately that is required to work. > I'm pretty sure the original setjmp/longjmp just saved the stack > pointer - but that really doesn't work any more. > > OTOH most signal handlers don't care - but there isn't a flag > to sigset() (etc) so ask for a specific register layout. Right, the idea is to optimize for *most* signal handlers, since making any changes to *all* signal handlers is intractable. So the idea is that opting-in to a fast signal handler would opt-out of some legacy signal capibilities. Complete state is one of them, and thus long-jump is not supported, because the complete state may not automatically be available. thanks, Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Fri, Mar 26, 2021 at 04:12:25PM -0700, Andy Lutomirski wrote: > To detect features and control XCR0, we add some new arch_prctls: > > arch_prctl(ARCH_GET_XCR0_SUPPORT, 0, ...); > > returns the set of XCR0 bits supported on the current kernel. > > arch_prctl(ARCH_GET_XCR0_LAZY_SUPPORT, 0, ...); > > returns 0. See below. > > arch_prctl(ARCH_SET_XCR0, xcr0, lazy_states, sigsave_states, > sigclear_states, 0); Right, but I'd simply replace that "XCR0" arch detail, more or less, with "XSTATE": ARCH_GET_XSTATE_SUPPORT ARCH_GET_XSTATE_LAZY_SUPPORT ARCH_SET_XSTATE or ARCH_ENABLE_XSTATE or so to denote that this is controlling XSTATE handling while the XCR0 thing is the control register and when in the future something else does control that (XCR0[63] is one provision for that) then we're still on-point with the naming. > Thoughts? FTR, I really like the aspect of apps *requesting* handling of non-legacy, fat states and latter remaining off otherwise in order to keep the sanity of everyone involved. Thx. -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
RE: Candidate Linux ABI for Intel AMX and hypothetical new related features
From: Len Brown > Sent: 30 March 2021 21:42 > > On Tue, Mar 30, 2021 at 4:20 PM Andy Lutomirski wrote: > > > > > > > On Mar 30, 2021, at 12:12 PM, Dave Hansen wrote: > > > > > > On 3/30/21 10:56 AM, Len Brown wrote: > > >> On Tue, Mar 30, 2021 at 1:06 PM Andy Lutomirski > > >> wrote: > > On Mar 30, 2021, at 10:01 AM, Len Brown wrote: > > Is it required (by the "ABI") that a user program has everything > > on the stack for user-space XSAVE/XRESTOR to get back > > to the state of the program just before receiving the signal? > > >>> The current Linux signal frame format has XSTATE in uncompacted format, > > >>> so everything has to be there. > > >>> Maybe we could have an opt in new signal frame format, but the details > > >>> would need to be worked > out. > > >>> > > >>> It is certainly the case that a signal should be able to be delivered, > > >>> run “async-signal-safe” > code, > > >>> and return, without corrupting register contents. > > >> And so an an acknowledgement: > > >> > > >> We can't change the legacy signal stack format without breaking > > >> existing programs. The legacy is uncompressed XSTATE. It is a > > >> complete set of architectural state -- everything necessary to > > >> XRESTOR. Further, the sigreturn flow allows the signal handler to > > >> *change* any of that state, so that it becomes active upon return from > > >> signal. > > > > > > One nit with this: XRSTOR itself can work with the compacted format or > > > uncompacted format. Unlike the XSAVE/XSAVEC side where compaction is > > > explicit from the instruction itself, XRSTOR changes its behavior by > > > reading XCOMP_BV. There's no XRSTORC. > > > > > > The issue with using the compacted format is when legacy software in the > > > signal handler needs to go access the state. *That* is what can't > > > handle a change in the XSAVE buffer format (either optimized/XSAVEOPT, > > > or compacted/XSAVEC). > > > > The compacted format isn’t compact enough anyway. If we want to keep AMX > > and AVX512 enabled in XCR0 > then we need to further muck with the format to omit the not-in-use features. > I *think* we can pull > this off in a way that still does the right thing wrt XRSTOR. > > Agreed. Compacted format doesn't save any space when INIT=0, so it is > only a half-step forward. > > > If we go this route, I think we want a way for sigreturn to understand a > > pointer to the state > instead of inline state to allow programs to change the state. Or maybe just > to have a way to ask > sigreturn to skip the restore entirely. > > The legacy approach puts all architectural state on the signal stack > in XSTATE format. > > If we make the signal stack smaller with a new fast-signal scheme, we > need to find another place for that state to live. > > It can't live in the task context switch buffer. If we put it there > and then take an interrupt while running the signal handler, then we'd > overwrite the signaled thread's state with the signal handler's state. > > Can we leave it in live registers? That would be the speed-of-light > signal handler approach. But we'd need to teach the signal handler to > not clobber it. Perhaps that could be part of the contract that a > fast signal handler signs? INIT=0 AMX state could simply sit > patiently in the AMX registers for the duration of the signal handler. > You can't get any faster than doing nothing :-) > > Of course part of the contract for the fast signal handler is that it > knows that it can't possibly use XRESTOR of the stuff on the stack to > necessarily get back to the state of the signaled thread (assuming we > even used XSTATE format on the fast signal handler stack, it would > forget the contents of the AMX registers, in this example) gcc will just use the AVX registers for 'normal' code within the signal handler. So it has to have its own copy of all the registers. (Well, maybe you could make the TMX instructions fault, but that would need a nested signal delivered.) There is also the register save buffer that you need in order to long-jump out of a signal handler. Unfortunately that is required to work. I'm pretty sure the original setjmp/longjmp just saved the stack pointer - but that really doesn't work any more. OTOH most signal handlers don't care - but there isn't a flag to sigset() (etc) so ask for a specific register layout. I did have 'fun' changing the x86 segment registers so that the 'return to user' faulted in kernel during the last bit of the 'return to user' path - and then fixing the fallout. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Tue, Mar 30, 2021 at 4:20 PM Andy Lutomirski wrote: > > > > On Mar 30, 2021, at 12:12 PM, Dave Hansen wrote: > > > > On 3/30/21 10:56 AM, Len Brown wrote: > >> On Tue, Mar 30, 2021 at 1:06 PM Andy Lutomirski > >> wrote: > On Mar 30, 2021, at 10:01 AM, Len Brown wrote: > Is it required (by the "ABI") that a user program has everything > on the stack for user-space XSAVE/XRESTOR to get back > to the state of the program just before receiving the signal? > >>> The current Linux signal frame format has XSTATE in uncompacted format, > >>> so everything has to be there. > >>> Maybe we could have an opt in new signal frame format, but the details > >>> would need to be worked out. > >>> > >>> It is certainly the case that a signal should be able to be delivered, > >>> run “async-signal-safe” code, > >>> and return, without corrupting register contents. > >> And so an an acknowledgement: > >> > >> We can't change the legacy signal stack format without breaking > >> existing programs. The legacy is uncompressed XSTATE. It is a > >> complete set of architectural state -- everything necessary to > >> XRESTOR. Further, the sigreturn flow allows the signal handler to > >> *change* any of that state, so that it becomes active upon return from > >> signal. > > > > One nit with this: XRSTOR itself can work with the compacted format or > > uncompacted format. Unlike the XSAVE/XSAVEC side where compaction is > > explicit from the instruction itself, XRSTOR changes its behavior by > > reading XCOMP_BV. There's no XRSTORC. > > > > The issue with using the compacted format is when legacy software in the > > signal handler needs to go access the state. *That* is what can't > > handle a change in the XSAVE buffer format (either optimized/XSAVEOPT, > > or compacted/XSAVEC). > > The compacted format isn’t compact enough anyway. If we want to keep AMX and > AVX512 enabled in XCR0 then we need to further muck with the format to omit > the not-in-use features. I *think* we can pull this off in a way that still > does the right thing wrt XRSTOR. Agreed. Compacted format doesn't save any space when INIT=0, so it is only a half-step forward. > If we go this route, I think we want a way for sigreturn to understand a > pointer to the state instead of inline state to allow programs to change the > state. Or maybe just to have a way to ask sigreturn to skip the restore > entirely. The legacy approach puts all architectural state on the signal stack in XSTATE format. If we make the signal stack smaller with a new fast-signal scheme, we need to find another place for that state to live. It can't live in the task context switch buffer. If we put it there and then take an interrupt while running the signal handler, then we'd overwrite the signaled thread's state with the signal handler's state. Can we leave it in live registers? That would be the speed-of-light signal handler approach. But we'd need to teach the signal handler to not clobber it. Perhaps that could be part of the contract that a fast signal handler signs? INIT=0 AMX state could simply sit patiently in the AMX registers for the duration of the signal handler. You can't get any faster than doing nothing :-) Of course part of the contract for the fast signal handler is that it knows that it can't possibly use XRESTOR of the stuff on the stack to necessarily get back to the state of the signaled thread (assuming we even used XSTATE format on the fast signal handler stack, it would forget the contents of the AMX registers, in this example) Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
> On Mar 30, 2021, at 12:12 PM, Dave Hansen wrote: > > On 3/30/21 10:56 AM, Len Brown wrote: >> On Tue, Mar 30, 2021 at 1:06 PM Andy Lutomirski wrote: On Mar 30, 2021, at 10:01 AM, Len Brown wrote: Is it required (by the "ABI") that a user program has everything on the stack for user-space XSAVE/XRESTOR to get back to the state of the program just before receiving the signal? >>> The current Linux signal frame format has XSTATE in uncompacted format, >>> so everything has to be there. >>> Maybe we could have an opt in new signal frame format, but the details >>> would need to be worked out. >>> >>> It is certainly the case that a signal should be able to be delivered, run >>> “async-signal-safe” code, >>> and return, without corrupting register contents. >> And so an an acknowledgement: >> >> We can't change the legacy signal stack format without breaking >> existing programs. The legacy is uncompressed XSTATE. It is a >> complete set of architectural state -- everything necessary to >> XRESTOR. Further, the sigreturn flow allows the signal handler to >> *change* any of that state, so that it becomes active upon return from >> signal. > > One nit with this: XRSTOR itself can work with the compacted format or > uncompacted format. Unlike the XSAVE/XSAVEC side where compaction is > explicit from the instruction itself, XRSTOR changes its behavior by > reading XCOMP_BV. There's no XRSTORC. > > The issue with using the compacted format is when legacy software in the > signal handler needs to go access the state. *That* is what can't > handle a change in the XSAVE buffer format (either optimized/XSAVEOPT, > or compacted/XSAVEC). The compacted format isn’t compact enough anyway. If we want to keep AMX and AVX512 enabled in XCR0 then we need to further muck with the format to omit the not-in-use features. I *think* we can pull this off in a way that still does the right thing wrt XRSTOR. If we go this route, I think we want a way for sigreturn to understand a pointer to the state instead of inline state to allow programs to change the state. Or maybe just to have a way to ask sigreturn to skip the restore entirely.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On 3/30/21 10:56 AM, Len Brown wrote: > On Tue, Mar 30, 2021 at 1:06 PM Andy Lutomirski wrote: >>> On Mar 30, 2021, at 10:01 AM, Len Brown wrote: >>> Is it required (by the "ABI") that a user program has everything >>> on the stack for user-space XSAVE/XRESTOR to get back >>> to the state of the program just before receiving the signal? >> The current Linux signal frame format has XSTATE in uncompacted format, >> so everything has to be there. >> Maybe we could have an opt in new signal frame format, but the details would >> need to be worked out. >> >> It is certainly the case that a signal should be able to be delivered, run >> “async-signal-safe” code, >> and return, without corrupting register contents. > And so an an acknowledgement: > > We can't change the legacy signal stack format without breaking > existing programs. The legacy is uncompressed XSTATE. It is a > complete set of architectural state -- everything necessary to > XRESTOR. Further, the sigreturn flow allows the signal handler to > *change* any of that state, so that it becomes active upon return from > signal. One nit with this: XRSTOR itself can work with the compacted format or uncompacted format. Unlike the XSAVE/XSAVEC side where compaction is explicit from the instruction itself, XRSTOR changes its behavior by reading XCOMP_BV. There's no XRSTORC. The issue with using the compacted format is when legacy software in the signal handler needs to go access the state. *That* is what can't handle a change in the XSAVE buffer format (either optimized/XSAVEOPT, or compacted/XSAVEC).
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Tue, Mar 30, 2021 at 1:06 PM Andy Lutomirski wrote: > > On Mar 30, 2021, at 10:01 AM, Len Brown wrote: > > Is it required (by the "ABI") that a user program has everything > > on the stack for user-space XSAVE/XRESTOR to get back > > to the state of the program just before receiving the signal? > > The current Linux signal frame format has XSTATE in uncompacted format, > so everything has to be there. > Maybe we could have an opt in new signal frame format, but the details would > need to be worked out. > > It is certainly the case that a signal should be able to be delivered, run > “async-signal-safe” code, > and return, without corrupting register contents. And so an an acknowledgement: We can't change the legacy signal stack format without breaking existing programs. The legacy is uncompressed XSTATE. It is a complete set of architectural state -- everything necessary to XRESTOR. Further, the sigreturn flow allows the signal handler to *change* any of that state, so that it becomes active upon return from signal. And a proposal: Future programs, which know that they don't need the full-blown legacy signal stack format, can opt-in to a new format. That new format, can be minimal (fast) by default. Perhaps, as Noah suggests, it could have some sort of mechanism where the program can explicitly select which state components they would want included on their signal stack, and restored by sigreturn. If the new fast-signal format is successful, in a number of years, it will have spread to have taken over the world. thoughts? Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
> On Mar 30, 2021, at 10:01 AM, Len Brown wrote: > > Andy, > > I agree, completely, with your description of the challenge, > thank you for focusing the discussion on that problem statement. > > Question: > > Is it required (by the "ABI") that a user program has everything > on the stack for user-space XSAVE/XRESTOR to get back > to the state of the program just before receiving the signal? The current Linux signal frame format has XSTATE in uncompacted format, so everything has to be there. Maybe we could have an opt in new signal frame format, but the details would need to be worked out. It is certainly the case that a signal should be able to be delivered, run “async-signal-safe” code, and return, without corrupting register contents.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
Andy, I agree, completely, with your description of the challenge, thank you for focusing the discussion on that problem statement. Question: Is it required (by the "ABI") that a user program has everything on the stack for user-space XSAVE/XRESTOR to get back to the state of the program just before receiving the signal? My understanding is that there are programs that do this. However, if it is not guaranteed to work, that could greatly simplify what we are required to put on the signal stack. thanks, -Len
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
Forgive if this is silly but would it be possible to do something simliar to rseq where the user can register a set of features for a program counter region and then on interrupt check that to determine what needs to be saved? For example if a user doesn't use any AMX but loads a library that does, for all ip in the users code AMX state won't be saved but an interrupt in ip range of the library will save AMX state. One advantage of this is it would be pretty easy silently do this right with compiler support and to preserve old code the "ip not found in table" case could default to the worst case the CPU supports. On Tue, Mar 30, 2021 at 1:09 AM Andy Lutomirski via Libc-alpha wrote: > > On Mon, Mar 29, 2021 at 3:38 PM Len Brown wrote: > > > > On Mon, Mar 29, 2021 at 2:16 PM Andy Lutomirski wrote: > > > > > > Hi Andy, > > > > Can you provide a concise definition of the exact problemI(s) this thread > > is attempting to address? > > The AVX-512 state, all by itself, is more than 2048 bytes. Quoting > the POSIX sigaltstack page (man 3p sigaltstack): > >The value SIGSTKSZ is a system default specifying the number of bytes >that would be used to cover the usual case when manually allocating an >alternate stack area. The value MINSIGSTKSZ is defined to be the mini‐ >mum stack size for a signal handler. In computing an alternate stack >size, a program should add that amount to its stack requirements to al‐ >low for the system implementation overhead. The constants SS_ONSTACK, >SS_DISABLE, SIGSTKSZ, and MINSIGSTKSZ are defined in . > > arch/x86/include/uapi/asm/signal.h:#define MINSIGSTKSZ2048 > arch/x86/include/uapi/asm/signal.h:#define SIGSTKSZ8192 > > Regrettably, the Linux signal frame format is the uncompacted format > and, also regrettably, the uncompacted format has the nasty property > that its format depends on XCR0 but not on the set of registers that > are actually used or wanted, so, with the current ABI, the signal > frame is stuck being quite large for all programs on a machine that > supports avx512 and has it enabled by the kernel. And it's even > larger for AMX and violates SIGSTKSZ as well as MINSTKSZ. > > There are apparently real programs that break as a result. We need to > find a way to handle new, large extended states without breaking user > ABI. We should also find a way to handle them without consuming silly > amounts of stack space for programs that don't use them. > > Sadly, if the solution we settle on involves context switching XCR0, > performance on first-generation hardware will suffer because VMX does > not have any way to allow guests to write XCR0 without exiting. I > don't consider this to be a showstopper -- if we end up having this > problem, fixing it in subsequent CPUs is straightforward. > > > > > Thank ahead-of-time for excluding "blow up power consumption", > > since that paranoia is not grounded in fact. > > > > I will gladly exclude power consumption from this discussion, since > that's a separate issue that has nothing to do with the user<->kernel > ABI.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Mar 29, 2021 at 3:38 PM Len Brown wrote: > > On Mon, Mar 29, 2021 at 2:16 PM Andy Lutomirski wrote: > > > Hi Andy, > > Can you provide a concise definition of the exact problemI(s) this thread > is attempting to address? The AVX-512 state, all by itself, is more than 2048 bytes. Quoting the POSIX sigaltstack page (man 3p sigaltstack): The value SIGSTKSZ is a system default specifying the number of bytes that would be used to cover the usual case when manually allocating an alternate stack area. The value MINSIGSTKSZ is defined to be the mini‐ mum stack size for a signal handler. In computing an alternate stack size, a program should add that amount to its stack requirements to al‐ low for the system implementation overhead. The constants SS_ONSTACK, SS_DISABLE, SIGSTKSZ, and MINSIGSTKSZ are defined in . arch/x86/include/uapi/asm/signal.h:#define MINSIGSTKSZ2048 arch/x86/include/uapi/asm/signal.h:#define SIGSTKSZ8192 Regrettably, the Linux signal frame format is the uncompacted format and, also regrettably, the uncompacted format has the nasty property that its format depends on XCR0 but not on the set of registers that are actually used or wanted, so, with the current ABI, the signal frame is stuck being quite large for all programs on a machine that supports avx512 and has it enabled by the kernel. And it's even larger for AMX and violates SIGSTKSZ as well as MINSTKSZ. There are apparently real programs that break as a result. We need to find a way to handle new, large extended states without breaking user ABI. We should also find a way to handle them without consuming silly amounts of stack space for programs that don't use them. Sadly, if the solution we settle on involves context switching XCR0, performance on first-generation hardware will suffer because VMX does not have any way to allow guests to write XCR0 without exiting. I don't consider this to be a showstopper -- if we end up having this problem, fixing it in subsequent CPUs is straightforward. > > Thank ahead-of-time for excluding "blow up power consumption", > since that paranoia is not grounded in fact. > I will gladly exclude power consumption from this discussion, since that's a separate issue that has nothing to do with the user<->kernel ABI.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Mon, Mar 29, 2021 at 2:16 PM Andy Lutomirski wrote: > > > > On Mar 29, 2021, at 8:47 AM, Len Brown wrote: > > > > On Sat, Mar 27, 2021 at 5:58 AM Greg KH wrote: > >>> On Fri, Mar 26, 2021 at 11:39:18PM -0400, Len Brown wrote: > >>> Hi Andy, > >>> Say a mainline links with a math library that uses AMX without the > >>> knowledge of the mainline. > > > > sorry for the confusion. > > > > mainline = main(). > > > > ie. the part of the program written by you, and not the library you linked > > with. > > > > In particular, the library may use instructions that main() doesn't know > > exist. > > If we pretend for a bit that AMX were a separate device instead of a part of > the CPU, this would be a no brainer: something would be responsible for > opening a device node or otherwise requesting access to the device. > > Real AMX isn’t so different. Programs acquire access either by syscall or by > a fault, they use it, and (hopefully) they release it again using > TILERELEASE. The only thing special about it is that, supposedly, acquiring > and releasing access (at least after the first time) is quite fast. But > holding access is *not* free — despite all your assertions to the contrary, > the kernel *will* correctly context switch it to avoid blowing up power > consumption, and this will have overhead. > > We’ve seen the pattern of programs thinking that, just because something is a > CPU insn, it’s free and no thought is needed before using it. This happened > with AVX and AVX512, and it will happen again with AMX. We *still* have a > significant performance regression in the kernel due to screwing up the AVX > state machine, and the only way I know about any of the details is that I > wrote silly test programs to try to reverse engineer the nonsensical behavior > of the CPUs. > > I might believe that Intel has figured out how to make a well behaved XSTATE > feature after Intel demonstrates at least once that it’s possible. That > means full documentation of all the weird issues, no new special cases, and > the feature actually making sense in the context of XSTATE. This has not > happened. Let’s list all of them: > > - SSE. Look for all the MXCSR special cases in the pseudocode and tell me > with a straight face that this one works sensibly. > > - AVX. Also has special cases in the pseudocode. And has transition issues > that are still problems and still not fully documented. L > > - AVX2. Horrible undocumented performance issues. Otherwise maybe okay? > > - MPX: maybe the best example, but the compat mode part got flubbed and it’s > MPX. > > - PKRU: Should never have been in XSTATE. (Also, having WRPKRU in the ISA was > a major mistake, now unfixable, that seriously limits the usefulness of the > whole feature. I suppose Intel could release PKRU2 with a better ISA and > deprecate the original PKRU, but I’m not holding my breath.) > > - AVX512: Yet more uarch-dependent horrible performance issues, and Intel has > still not responded about documentation. The web is full of people > speculating differently about when, exactly, using AVX512 breaks performance. > This is NAKked in kernel until docs arrive. Also, it broke old user programs. > If we had noticed a few years ago, AVX512 enablement would have been > reverted. > > - AMX: This mess. > > The current system of automatic user enablement does not work. We need > something better. Hi Andy, Can you provide a concise definition of the exact problemI(s) this thread is attempting to address? Thank ahead-of-time for excluding "blow up power consumption", since that paranoia is not grounded in fact. thanks, -Len
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
> On Mar 29, 2021, at 8:47 AM, Len Brown wrote: > > On Sat, Mar 27, 2021 at 5:58 AM Greg KH wrote: >>> On Fri, Mar 26, 2021 at 11:39:18PM -0400, Len Brown wrote: >>> Hi Andy, >>> Say a mainline links with a math library that uses AMX without the >>> knowledge of the mainline. > > sorry for the confusion. > > mainline = main(). > > ie. the part of the program written by you, and not the library you linked > with. > > In particular, the library may use instructions that main() doesn't know > exist. If we pretend for a bit that AMX were a separate device instead of a part of the CPU, this would be a no brainer: something would be responsible for opening a device node or otherwise requesting access to the device. Real AMX isn’t so different. Programs acquire access either by syscall or by a fault, they use it, and (hopefully) they release it again using TILERELEASE. The only thing special about it is that, supposedly, acquiring and releasing access (at least after the first time) is quite fast. But holding access is *not* free — despite all your assertions to the contrary, the kernel *will* correctly context switch it to avoid blowing up power consumption, and this will have overhead. We’ve seen the pattern of programs thinking that, just because something is a CPU insn, it’s free and no thought is needed before using it. This happened with AVX and AVX512, and it will happen again with AMX. We *still* have a significant performance regression in the kernel due to screwing up the AVX state machine, and the only way I know about any of the details is that I wrote silly test programs to try to reverse engineer the nonsensical behavior of the CPUs. I might believe that Intel has figured out how to make a well behaved XSTATE feature after Intel demonstrates at least once that it’s possible. That means full documentation of all the weird issues, no new special cases, and the feature actually making sense in the context of XSTATE. This has not happened. Let’s list all of them: - SSE. Look for all the MXCSR special cases in the pseudocode and tell me with a straight face that this one works sensibly. - AVX. Also has special cases in the pseudocode. And has transition issues that are still problems and still not fully documented. L - AVX2. Horrible undocumented performance issues. Otherwise maybe okay? - MPX: maybe the best example, but the compat mode part got flubbed and it’s MPX. - PKRU: Should never have been in XSTATE. (Also, having WRPKRU in the ISA was a major mistake, now unfixable, that seriously limits the usefulness of the whole feature. I suppose Intel could release PKRU2 with a better ISA and deprecate the original PKRU, but I’m not holding my breath.) - AVX512: Yet more uarch-dependent horrible performance issues, and Intel has still not responded about documentation. The web is full of people speculating differently about when, exactly, using AVX512 breaks performance. This is NAKked in kernel until docs arrive. Also, it broke old user programs. If we had noticed a few years ago, AVX512 enablement would have been reverted. - AMX: This mess. The current system of automatic user enablement does not work. We need something better.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
> On Mar 29, 2021, at 9:39 AM, Len Brown wrote: > > >> >> In particular, the library may use instructions that main() doesn't know >> exist. > > And so I'll ask my question another way. > > How is it okay to change the value of XCR0 during the run time of a program? > > I submit that it is not, and that is a deal-killer for a request/release API. > > eg. main() doesn't know that the math library wants to use AMX, > and neither does the threading library. So main() doesn't know to > call the API before either library is invoked. The threading library starts > up > and creates user-space threads based on the initial value from XCR0. > Then the math library calls the API, which adds bits to XCRO, > and then the user-space context switch in the threading library corrupts data > because the new XCR0 size doesn't match the initial size. > In the most extreme case, userspace could require that every loaded DSO be tagged with a new ELF note indicating support for dynamic XCR0 before changing XCR0. I would like to remind everyone that kernel enablement of AVX512 *already* broke old userspace. AMX will further break something. At least with dynamic XCR0 we can make the breakage opt-in. The ISA could have helped here by allowing the non-compacted XSTATE format to be frozen even in the face of changing XCR0. But it didn’t. At the end of the day, we are faced with the fact that XSTATE is a poor design, and we have to make the best of it.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
* Len Brown via Libc-alpha: >> In particular, the library may use instructions that main() doesn't know >> exist. > > And so I'll ask my question another way. > > How is it okay to change the value of XCR0 during the run time of a > program? > > I submit that it is not, and that is a deal-killer for a > request/release API. > > eg. main() doesn't know that the math library wants to use AMX, and > neither does the threading library. So main() doesn't know to call > the API before either library is invoked. The threading library > starts up and creates user-space threads based on the initial value > from XCR0. Then the math library calls the API, which adds bits to > XCRO, and then the user-space context switch in the threading > library corrupts data because the new XCR0 size doesn't match the > initial size. I agree that this doesn't quite work. (Today, it's not the thread library, but the glibc dynamic loader trampoline.) I disagree that CPU feature enablement has been a failure. I think we are pretty good at enabling new CPU features on older operating systems, not just bleeding edge mainline kernels. Part of that is that anything but the kernel stays out of the way, and most features are available directly via inline assembly (you can even use .byte hacks if you want). There is no need to switch to new userspace libraries, compile out-of-tree kernel drivers that have specific firmware requirements, and so on. If the operations that need a huge context can be made idempotent, with periodic checkpoints, it might be possible to avoid saving the context completely by some rseq-like construct.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
> In particular, the library may use instructions that main() doesn't know > exist. And so I'll ask my question another way. How is it okay to change the value of XCR0 during the run time of a program? I submit that it is not, and that is a deal-killer for a request/release API. eg. main() doesn't know that the math library wants to use AMX, and neither does the threading library. So main() doesn't know to call the API before either library is invoked. The threading library starts up and creates user-space threads based on the initial value from XCR0. Then the math library calls the API, which adds bits to XCRO, and then the user-space context switch in the threading library corrupts data because the new XCR0 size doesn't match the initial size. -Len
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Sat, Mar 27, 2021 at 5:58 AM Greg KH wrote: > > On Fri, Mar 26, 2021 at 11:39:18PM -0400, Len Brown wrote: > > Hi Andy, > > > > Say a mainline links with a math library that uses AMX without the > > knowledge of the mainline. sorry for the confusion. mainline = main(). ie. the part of the program written by you, and not the library you linked with. In particular, the library may use instructions that main() doesn't know exist. -- Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On 3/27/21 5:53 PM, Thomas Gleixner wrote: > Making it solely depend on XCR0 and fault if not requested upfront is > bringing you into the situation that you broke 'legacy code' which > relied on the CPUID bit and that worked until now which gets you > in the no-regression trap. Trying to find the right place to jump into this thread... :) I don't know what apps do in practice. But, the enumeration of the features in the SDM describes three steps: 1. Check for XGETBV support 2. Use XGETBV[0] to check that the OS is aware of the feature and is context-switching it 3. Detect the feature itself So, apps *are* supposed to be checking XCR0 via XGETBV. If they don't, they run the risk of a feature being supported by the CPU and the registers "working" but not being context-switched. Zeroing out bits in XCR0 will have the effect of telling the app that the OS isn't context-switching the state. I think this means that apps will see the same thing in both situations: 1. If they run an old (say pre-AVX-512) kernel on new AVX-512-enabled hardware, or 2. They run a new kernel with this fancy proposed XCR0-switching mechanism I _think_ that gets us off the hook for an ABI break, at least for AVX-512.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Sun, Mar 28, 2021 at 01:53:15AM +0100, Thomas Gleixner wrote: > Though the little devil in my head tells me, that making AMX support > depend on the CPUID faulting capability might be not the worst thing. > > Then we actually enforce CPUID faulting (finally) on CPUs which support > it, which would be a first step into the right direction simply because > then random library X has to go to the kernel and ask for it explicitely > or just shrug and use whatever the kernel is willing to hand out in > CPUID. > > Now take that one step further. When the first part of some user space > application asks for it, then you can register that with the process and > make sane decisions for all other requesters which come after it, which > is an important step into the direction of having a common orchestration > for this. I wrote something like that at least one... https://lore.kernel.org/lkml/20190212164833.gk32...@hirez.programming.kicks-ass.net/ we just need to make sure AMD implements that before it ships a chip with AVX512 on.
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
Andy, On Fri, Mar 26 2021 at 16:18, Andy Lutomirski wrote: > arch_prctl(ARCH_SET_XCR0, xcr0, lazy_states, sigsave_states, > sigclear_states, 0); > > Sets xcr0. All states are preallocated except that states in > lazy_states may be unallocated in the kernel until used. (Not > supported at all in v1. lazy_states & ~xcr0 != 0 is illegal.) States > in sigsave_states are saved in the signal frame. States in > sigclear_states are reset to the init state on signal delivery. > States in sigsave_states are restored by sigreturn, and states not in > sigsave_states are left alone by sigreturn. I like the idea in principle. > Optionally we come up with a new format for new features in the signal > frame, since the current format is showing its age. Taking 8kB for a > signal with AMX is one thing. Taking another 8kB for a nested signal > if AMX is not in use is worse. I don't think that we should make that optional to begin with. Sizing sigaltstack is lottery as of today and making it more so does not help at all. > Optionally we make AVX-512 also default off, which fixes what is > arguably a serious ABI break with AVX-512: lots of programs, following > POSIX (!), seem to think that they know much much space to allocate > for sigaltstack(). AVX-512 is too big. I really wish we could do that. That AVX512 disaster is not trivial to sort. Let's focus on AMX first. That ship at least has not sailed yet, but if it does without a proper resolution then it's going to sail deep south. Maybe we end up with some ideas about the AVX512 issue as well that way. The main problem I see is simply historical. Every other part of the user stack space from libraries to applications tries to be "smart" about utilizing the assumed best instruction set, feature extensions which are detected when something is initialized. I can sing a song of that because I was casually involved porting debian to an unsupported architecture. Magic all over the place. Now add the whole pile of proprietary software stacks, libraries on top of that picture and things get completely out of control. Why? Simply because user space has absolutely no concept about orchestrating these things at all. That worked for a while by some definition of works and this model is still proliferated today even by players who should know better. Even if you expected that some not so distant events and the experience with fleet consistency would have stopped the 'performance first, features first' chorus in some way, that's not what reality is. Linux is not necessarily innocent. For years we just crammed features into the kernel without thinking too hard about the big picture. But, yes we realized the hard way that there is a problem and just adding yet another magic 'make it work' hack for AMX is definitely the wrong approach. What are the possible problems when we make it a hard requirement for AMX to be requested by an application/task in order to use it? For the kernel itself. Not really any consequence I can think off aside of unhappy campers in user space. For user space this is disruptive and we have at least to come up with some reasonable model how all involved components with different ideas of how to best utilize a given CPU can be handled. That starts at the very simple problem of feature enumeration. Up to now CPUID is non-priviledged and a large amount of user space just takes that as the ultimate reference. We can change that when CPUID faulting in CPL3 is supported by the CPU which we can't depend on because it is not architectural. Though the little devil in my head tells me, that making AMX support depend on the CPUID faulting capability might be not the worst thing. Then we actually enforce CPUID faulting (finally) on CPUs which support it, which would be a first step into the right direction simply because then random library X has to go to the kernel and ask for it explicitely or just shrug and use whatever the kernel is willing to hand out in CPUID. Now take that one step further. When the first part of some user space application asks for it, then you can register that with the process and make sane decisions for all other requesters which come after it, which is an important step into the direction of having a common orchestration for this. Sure you can do that via XCR0 as well to some extent, but that CPUID fault would solve a whole class of other problems which people who care about feature consistency face today at least to some extent. And contrary to XCR0, which is orthogonal and obviously still required for the AMX (and hint AVX512) problem, CPUID faulting would just hand out the feature bits which the kernel want's to hand out. If the app, library or whatever still tries to use them, then they get the #UD, #GP or whatever penalty is associated to that particular XCR0 disabled piece. It's not there, you tried, keep the pieces. Making it solely depend on XCR0 and fault if not requested upfront is bringing you into the
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Fri, Mar 26, 2021 at 11:39:18PM -0400, Len Brown wrote: > Hi Andy, > > Say a mainline links with a math library that uses AMX without the > knowledge of the mainline. What does this mean? What happened to the context here? > Say the mainline is also linked with a userspace threading library > that thinks it has a concept of XSAVE area size. How can the kernel (what I think you mean by "mainline" here) be linked with a userspace library at all? > Wouldn't the change in XCR0, resulting in XSAVE size change, risk > confusing the threading library? Shouldn't that be the job of the kernel and not userspace? totally confused, greg k-h
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
On Fri, Mar 26, 2021 at 11:39:18PM -0400, Len Brown wrote: > Say a mainline links with a math library that uses AMX without the > knowledge of the mainline. What is a "mainline"? -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
Hi Andy, Say a mainline links with a math library that uses AMX without the knowledge of the mainline. Say the mainline is also linked with a userspace threading library that thinks it has a concept of XSAVE area size. Wouldn't the change in XCR0, resulting in XSAVE size change, risk confusing the threading library? thanks, Len Brown, Intel Open Source Technology Center
Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
Sigh, cc linux-api, not linux-abi. On Fri, Mar 26, 2021 at 4:12 PM Andy Lutomirski wrote: > > Hi all- > > After some discussion on IRC, I have a proposal for a Linux ABI for > using Intel AMX and other similar features. It works like this: > > First, we make XCR0 dynamic. This looks a lot like Keno's patch but > with a different API, outlined below. Different tasks can have > different XCR0 values. The default XCR0 for new tasks does not > include big features like AMX. XMM and YMM are still there. The AVX2 > states are debatable -- see below. > > To detect features and control XCR0, we add some new arch_prctls: > > arch_prctl(ARCH_GET_XCR0_SUPPORT, 0, ...); > > returns the set of XCR0 bits supported on the current kernel. > > arch_prctl(ARCH_GET_XCR0_LAZY_SUPPORT, 0, ...); > > returns 0. See below. > > arch_prctl(ARCH_SET_XCR0, xcr0, lazy_states, sigsave_states, > sigclear_states, 0); > > Sets xcr0. All states are preallocated except that states in > lazy_states may be unallocated in the kernel until used. (Not > supported at all in v1. lazy_states & ~xcr0 != 0 is illegal.) States > in sigsave_states are saved in the signal frame. States in > sigclear_states are reset to the init state on signal delivery. > States in sigsave_states are restored by sigreturn, and states not in > sigsave_states are left alone by sigreturn. > > Optionally we do not support PKRU at all in XCR0 -- it doesn't make > that much sense as an XSAVE feature, and I'm not convinced that trying > to correctly context switch XINUSE[PKRU] is worthwhile. I doubt we > get it right today. > > Optionally we come up with a new format for new features in the signal > frame, since the current format is showing its age. Taking 8kB for a > signal with AMX is one thing. Taking another 8kB for a nested signal > if AMX is not in use is worse. > > Optionally we make AVX-512 also default off, which fixes what is > arguably a serious ABI break with AVX-512: lots of programs, following > POSIX (!), seem to think that they know much much space to allocate > for sigaltstack(). AVX-512 is too big. > > Thoughts? > > --Andy
