Re: revert scope for 5.8, was Re: dma-pool fixes
On Mon, 3 Aug 2020, Christoph Hellwig wrote: > On Sun, Aug 02, 2020 at 09:14:41PM -0700, David Rientjes wrote: > > Christoph: since we have atomic DMA coherent pools in 5.8 now, recall our > > previous discussions, including Greg KH, regarding backports to stable > > trees (we are interested in 5.4 and 5.6) of this support for the purposes > > of confidential VM users who wish to run their own SEV-enabled guests in > > cloud. > > > > Would you have any objections to backports being offered for this support > > now? We can prepare them immediately. Or, if you would prefer we hold > > off for a while, please let me know and we can delay it until you are more > > comfortable. (For confidential VM users, the ability to run your own > > unmodified stable kernel is a much different security story than a guest > > modified by the cloud provider.) > > Before we start backporting I think we need the two fixes from > the "dma pool fixes" series. Can you start reviewing them? I also > think we should probably wait at least until -rc2 for any fallout > before starting the backport. > Thanks Christoph, this makes perfect sense. I see Nicolas has refreshed the two patches: dma-pool: fix coherent pool allocations for IOMMU mappings dma-pool: Only allocate from CMA when in same memory zone and posted them again today on LKML for review. I'll review those and we'll send a full series of backports for stable consideration for 5.4 LTS and 5.6 around 5.9-rc2 timeframe. Thanks! ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: revert scope for 5.8, was Re: dma-pool fixes
On Sun, Aug 02, 2020 at 09:14:41PM -0700, David Rientjes wrote: > Christoph: since we have atomic DMA coherent pools in 5.8 now, recall our > previous discussions, including Greg KH, regarding backports to stable > trees (we are interested in 5.4 and 5.6) of this support for the purposes > of confidential VM users who wish to run their own SEV-enabled guests in > cloud. > > Would you have any objections to backports being offered for this support > now? We can prepare them immediately. Or, if you would prefer we hold > off for a while, please let me know and we can delay it until you are more > comfortable. (For confidential VM users, the ability to run your own > unmodified stable kernel is a much different security story than a guest > modified by the cloud provider.) Before we start backporting I think we need the two fixes from the "dma pool fixes" series. Can you start reviewing them? I also think we should probably wait at least until -rc2 for any fallout before starting the backport. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: revert scope for 5.8, was Re: dma-pool fixes
On Sun, 2 Aug 2020, Amit Pundir wrote: > > > > Hi, I found the problematic memory region. It was a memory > > > > chunk reserved/removed in the downstream tree but was > > > > seemingly reserved upstream for different drivers. I failed to > > > > calculate the length of the total region reserved downstream > > > > correctly. And there was still a portion of memory left unmarked, > > > > which I should have marked as reserved in my testing earlier > > > > today. > > > > > > > > Sorry for all the noise and thanks Nicolas, Christoph and David > > > > for your patience. > > > > > > So you'll need to patch the upstream DTS to fix this up? Do you also > > > need my two fixes? What about the Oneplus phones? Can you send a > > > mail with a summary of the status? > > > > Poco's DTS is not upstreamed yet. I have updated it for this fix > > and sent out a newer version for review. > > https://lkml.org/lkml/2020/8/1/184 > > > > I didn't need to try your two add-on fixes. I'll give them a spin > > later today. > > Hi Christoph, > > I see no obvious regressions with your twin dma-pool fixes on my > PocoF1 phone. > > Caleb also confirmed that a similar reserved-memory region fix in his > One Plus 6 device-tree worked for him too. > Thanks very much for the update, Amit. Christoph: since we have atomic DMA coherent pools in 5.8 now, recall our previous discussions, including Greg KH, regarding backports to stable trees (we are interested in 5.4 and 5.6) of this support for the purposes of confidential VM users who wish to run their own SEV-enabled guests in cloud. Would you have any objections to backports being offered for this support now? We can prepare them immediately. Or, if you would prefer we hold off for a while, please let me know and we can delay it until you are more comfortable. (For confidential VM users, the ability to run your own unmodified stable kernel is a much different security story than a guest modified by the cloud provider.) ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: revert scope for 5.8, was Re: dma-pool fixes
On Sun, 2 Aug 2020 at 10:16, Amit Pundir wrote: > > On Sat, 1 Aug 2020 at 23:09, Christoph Hellwig wrote: > > > > On Sat, Aug 01, 2020 at 05:27:04PM +0530, Amit Pundir wrote: > > > Hi, I found the problematic memory region. It was a memory > > > chunk reserved/removed in the downstream tree but was > > > seemingly reserved upstream for different drivers. I failed to > > > calculate the length of the total region reserved downstream > > > correctly. And there was still a portion of memory left unmarked, > > > which I should have marked as reserved in my testing earlier > > > today. > > > > > > Sorry for all the noise and thanks Nicolas, Christoph and David > > > for your patience. > > > > So you'll need to patch the upstream DTS to fix this up? Do you also > > need my two fixes? What about the Oneplus phones? Can you send a > > mail with a summary of the status? > > Poco's DTS is not upstreamed yet. I have updated it for this fix > and sent out a newer version for review. > https://lkml.org/lkml/2020/8/1/184 > > I didn't need to try your two add-on fixes. I'll give them a spin > later today. Hi Christoph, I see no obvious regressions with your twin dma-pool fixes on my PocoF1 phone. Caleb also confirmed that a similar reserved-memory region fix in his One Plus 6 device-tree worked for him too. > > I'm sure One Plus 6 and 6T will be running into similar problem. > I'll check with Caleb and send out a status mail with the summary. > > Regards, > Amit Pundir ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: revert scope for 5.8, was Re: dma-pool fixes
On Sat, 1 Aug 2020 at 23:09, Christoph Hellwig wrote: > > On Sat, Aug 01, 2020 at 05:27:04PM +0530, Amit Pundir wrote: > > Hi, I found the problematic memory region. It was a memory > > chunk reserved/removed in the downstream tree but was > > seemingly reserved upstream for different drivers. I failed to > > calculate the length of the total region reserved downstream > > correctly. And there was still a portion of memory left unmarked, > > which I should have marked as reserved in my testing earlier > > today. > > > > Sorry for all the noise and thanks Nicolas, Christoph and David > > for your patience. > > So you'll need to patch the upstream DTS to fix this up? Do you also > need my two fixes? What about the Oneplus phones? Can you send a > mail with a summary of the status? Poco's DTS is not upstreamed yet. I have updated it for this fix and sent out a newer version for review. https://lkml.org/lkml/2020/8/1/184 I didn't need to try your two add-on fixes. I'll give them a spin later today. I'm sure One Plus 6 and 6T will be running into similar problem. I'll check with Caleb and send out a status mail with the summary. Regards, Amit Pundir ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: revert scope for 5.8, was Re: dma-pool fixes
On Sat, 1 Aug 2020 at 23:58, Linus Torvalds wrote: > > On Sat, Aug 1, 2020 at 4:57 AM Amit Pundir wrote: > > > > Hi, I found the problematic memory region. It was a memory > > chunk reserved/removed in the downstream tree but was > > seemingly reserved upstream for different drivers. > > Is this happening with a clean tree, or are there external drivers > involved that trigger the problem? > > Because if it's a clean tree, I guess I need to do an rc8 anyway, just > to get whatever workaround you then added to devicetree and/or some > driver to make it work again. > No, this is not on a clean tree. The phone's device-tree is not upstreamed yet. That is the only change I carry. I have updated the device-tree for this fix and sent out a newer version for review. https://lkml.org/lkml/2020/8/1/184 Regards, Amit Pundir > Or is there a quick fix that I can get today or early tomorrow? We had > some confusion this week due to a nasty include header mess, but > despite that there hasn't been anything else that has come up (so > far), so.. > >Linus ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: revert scope for 5.8, was Re: dma-pool fixes
On Sat, Aug 1, 2020 at 4:57 AM Amit Pundir wrote: > > Hi, I found the problematic memory region. It was a memory > chunk reserved/removed in the downstream tree but was > seemingly reserved upstream for different drivers. Is this happening with a clean tree, or are there external drivers involved that trigger the problem? Because if it's a clean tree, I guess I need to do an rc8 anyway, just to get whatever workaround you then added to devicetree and/or some driver to make it work again. Or is there a quick fix that I can get today or early tomorrow? We had some confusion this week due to a nasty include header mess, but despite that there hasn't been anything else that has come up (so far), so.. Linus ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: revert scope for 5.8, was Re: dma-pool fixes
On Sat, Aug 01, 2020 at 05:27:04PM +0530, Amit Pundir wrote: > Hi, I found the problematic memory region. It was a memory > chunk reserved/removed in the downstream tree but was > seemingly reserved upstream for different drivers. I failed to > calculate the length of the total region reserved downstream > correctly. And there was still a portion of memory left unmarked, > which I should have marked as reserved in my testing earlier > today. > > Sorry for all the noise and thanks Nicolas, Christoph and David > for your patience. So you'll need to patch the upstream DTS to fix this up? Do you also need my two fixes? What about the Oneplus phones? Can you send a mail with a summary of the status? ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: revert scope for 5.8, was Re: dma-pool fixes
On Sat, 2020-08-01 at 17:27 +0530, Amit Pundir wrote: [...] > > I'm between a rock and a hard place here. If we simply want to revert > > commits as-is to make sure both the Raspberry Pi 4 and thone phone do > > not regress we'll have to go all the way back and revert the whole SEV > > pool support. I could try to manual revert of the multiple pool > > support, but it is very late for that. > > Hi, I found the problematic memory region. It was a memory > chunk reserved/removed in the downstream tree but was > seemingly reserved upstream for different drivers. I failed to > calculate the length of the total region reserved downstream > correctly. And there was still a portion of memory left unmarked, > which I should have marked as reserved in my testing earlier > today. > > Sorry for all the noise and thanks Nicolas, Christoph and David > for your patience. That's great news, thanks for persevering! Regards, Nicolas ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: revert scope for 5.8, was Re: dma-pool fixes
On Sat, 1 Aug 2020 at 14:27, Christoph Hellwig wrote: > > On Sat, Aug 01, 2020 at 01:20:07AM -0700, David Rientjes wrote: > > To follow-up on this, the introduction of the DMA atomic pools in 5.8 > > fixes an issue for any AMD SEV enabled guest that has a driver that > > requires atomic DMA allocations (for us, nvme) because runtime decryption > > of memory allocated through the DMA API may block. This manifests itself > > as "sleeping in invalid context" BUGs for any confidential VM user in > > cloud. > > > > I unfortunately don't have Amit's device to be able to independently debug > > this issue and certainly could not have done a better job at working the > > bug than Nicolas and Christoph have done so far. I'm as baffled by the > > results as anybody else. > > > > I fully understand the no regressions policy. I'd also ask that we > > consider that *all* SEV guests are currently broken if they use nvme or > > any other driver that does atomic DMA allocations. It's an extremely > > serious issue for cloud. If there is *anything* that I can do to make > > forward progress on this issue for 5.8, including some of the workarounds > > above that Amit requested, I'd be very happy to help. Christoph will make > > the right decision for DMA in 5.8, but I simply wanted to state how > > critical working SEV guests are to users. > > I'm between a rock and a hard place here. If we simply want to revert > commits as-is to make sure both the Raspberry Pi 4 and thone phone do > not regress we'll have to go all the way back and revert the whole SEV > pool support. I could try to manual revert of the multiple pool > support, but it is very late for that. Hi, I found the problematic memory region. It was a memory chunk reserved/removed in the downstream tree but was seemingly reserved upstream for different drivers. I failed to calculate the length of the total region reserved downstream correctly. And there was still a portion of memory left unmarked, which I should have marked as reserved in my testing earlier today. Sorry for all the noise and thanks Nicolas, Christoph and David for your patience. Regards, Amit Pundir > > Or maybe Linus has decided to cut a -rc8 which would give us a little > more time. > - ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
revert scope for 5.8, was Re: dma-pool fixes
On Sat, Aug 01, 2020 at 01:20:07AM -0700, David Rientjes wrote: > To follow-up on this, the introduction of the DMA atomic pools in 5.8 > fixes an issue for any AMD SEV enabled guest that has a driver that > requires atomic DMA allocations (for us, nvme) because runtime decryption > of memory allocated through the DMA API may block. This manifests itself > as "sleeping in invalid context" BUGs for any confidential VM user in > cloud. > > I unfortunately don't have Amit's device to be able to independently debug > this issue and certainly could not have done a better job at working the > bug than Nicolas and Christoph have done so far. I'm as baffled by the > results as anybody else. > > I fully understand the no regressions policy. I'd also ask that we > consider that *all* SEV guests are currently broken if they use nvme or > any other driver that does atomic DMA allocations. It's an extremely > serious issue for cloud. If there is *anything* that I can do to make > forward progress on this issue for 5.8, including some of the workarounds > above that Amit requested, I'd be very happy to help. Christoph will make > the right decision for DMA in 5.8, but I simply wanted to state how > critical working SEV guests are to users. I'm between a rock and a hard place here. If we simply want to revert commits as-is to make sure both the Raspberry Pi 4 and thone phone do not regress we'll have to go all the way back and revert the whole SEV pool support. I could try to manual revert of the multiple pool support, but it is very late for that. Or maybe Linus has decided to cut a -rc8 which would give us a little more time. - ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
