On Fri, Sep 06, 2019, Borislav Petkov <b...@alien8.de> wrote: > On Fri, Sep 06, 2019 at 08:46:18AM -0700, Johannes Erdfelt wrote: > > That said, we very much rely on late microcode loading and it has helped > > us and our customers significantly. > > You do realize that you rely on an update method which *won't* work in > all possible cases and then you *will* have to reboot if the microcode > patching *must* happen early, do you?
Yeah. I even explained some cases where it wouldn't work. If we get a microcode update that isn't late loadable, then yes, we have to do something different. That doesn't mean that late loading isn't still useful. In practice, it can be very valuable. It isn't bad or dangerous in vast majority of cases. We haven't had a microcode update across all of the models of Intel CPUs we have (going back a handful of generations) in the past almost two years that wasn't safely late loadable. Just as I can't know for sure that every future microcode update will be safely late loadable, you can't know for sure that every future microcode update won't be safely late loadable. > > It's really easy to say "fix your infrastructure" when you're not > > running that infrastructure. > > I'm not saying you should fix your infrastructure now - I'm saying you > should keep that in mind when thinking whether to rely more on late > loading or not. Who knows, maybe newer generation machines in the fleet > could do load balancing, live migration, whatever fancy new cloud stuff > it is, to facilitate a proper reboot. We are well aware of the downsides of late microcode loading and live patching. However, like I said, it's currently the best tools available. We are certainly looking at other options, but some aren't feasible for mitigating security vulnerabilities where time to getting patched is very much important. > > The more reboots we can avoid, the better it is for us and our > > customers. > > So how do you update the kernels on those machines? Or you live-patch in > the new functionality too? Depends on what kind of patch is needed. Livepatching, while having it's own set of problems, has been very valuable for us. We do use other techniques as well particularly when it's not time sensitive. > > I understand that it could be unsafe to late load some rare microcode > > updates (theoretical or not). However, that is certainly the exception. > > We have done this multiple times on our fleet and we plan to continue > > doing so in the future. > > The fact that it has worked for you does not make it right. It won't > magically become safe, as tglx said. It very much makes it right because it's still a tool that can be used safely in the right cases. Just because it can't be used 100% of the time (even if it is close to that in practice) doesn't make it magically unsafe either. > Practically speaking, late loading probably won't disappear as it is > being used apparently. Just don't expect that it will get "extended" if > that extension brings with itself fallout and duct tape fixes left and > right. I don't have a particular use case for the patchset at hand and I'm certainly not arguing for or against this patchset. But I do get concerned when there is talk about removing a feature we currently use extensively. I'm happy to hear it will likely not be removed and I hope it was partly because I spoke up to show that is actively being used and it's important to us. JE