iommu_iova slab eats too much memory

2020-04-23 Thread Bin
Hey, guys: I'm running a batch of CoreOS boxes, the lsb_release is: ``` # cat /etc/lsb-release DISTRIB_ID="Container Linux by CoreOS" DISTRIB_RELEASE=2303.3.0 DISTRIB_CODENAME="Rhyolite" DISTRIB_DESCRIPTION="Container Linux by CoreOS 2303.3.0 (Rhyolite)" ``` ``` # uname -a Linux cloud-worker-25

Re: iommu_iova slab eats too much memory

2020-04-23 Thread Bin
Forget to mention, I've already disabled the slab merge, so this is what it is. Bin 于2020年4月23日周四 下午5:11写道: > Hey, guys: > > I'm running a batch of CoreOS boxes, the lsb_release is: > > ``` > # cat /etc/lsb-release > DISTRIB_ID="Container Linux by CoreOS" > DISTRIB_RELEASE=2303.3.0 > DISTRIB_COD

Re: iommu_iova slab eats too much memory

2020-04-23 Thread Bin
Hello? anyone there? Bin 于2020年4月23日周四 下午5:14写道: > Forget to mention, I've already disabled the slab merge, so this is what > it is. > > Bin 于2020年4月23日周四 下午5:11写道: > >> Hey, guys: >> >> I'm running a batch of CoreOS boxes, the lsb_release is: >> >> ``` >> # cat /etc/lsb-release >> DISTRIB_ID="

Re: iommu_iova slab eats too much memory

2020-04-24 Thread Robin Murphy
On 2020-04-24 1:40 am, Bin wrote: Hello? anyone there? Bin 于2020年4月23日周四 下午5:14写道: Forget to mention, I've already disabled the slab merge, so this is what it is. Bin 于2020年4月23日周四 下午5:11写道: Hey, guys: I'm running a batch of CoreOS boxes, the lsb_release is: ``` # cat /etc/lsb-release D

Re: iommu_iova slab eats too much memory

2020-04-24 Thread Bin
Well, that's the problem! I'm assuming the iommu kernel module is leaking memory. But I don't know why and how. Do you have any idea about it? Or any further information is needed? Robin Murphy 于 2020年4月24日周五 19:20写道: > On 2020-04-24 1:40 am, Bin wrote: > > Hello? anyone there? > > > > Bin 于20

Re: iommu_iova slab eats too much memory

2020-04-24 Thread Bin
I'm not familiar with the mmu stuff, so what you mean by "some driver leaking DMA mappings", is it possible that some other kernel module like KVM or NIC driver leads to the leaking problem instead of the iommu module itself? Bin 于 2020年4月24日周五 20:00写道: > Well, that's the problem! I'm assuming t

Re: iommu_iova slab eats too much memory

2020-04-24 Thread Robin Murphy
On 2020-04-24 1:06 pm, Bin wrote: I'm not familiar with the mmu stuff, so what you mean by "some driver leaking DMA mappings", is it possible that some other kernel module like KVM or NIC driver leads to the leaking problem instead of the iommu module itself? Yes - I doubt that intel-iommu itse

Re: iommu_iova slab eats too much memory

2020-04-24 Thread Bin
Dear Robin: Thank you for your explanation. Now, I understand that this could be NIC driver's fault, but how could I confirm it? Do I have to debug the driver myself? Robin Murphy 于2020年4月24日周五 下午8:15写道: > On 2020-04-24 1:06 pm, Bin wrote: > > I'm not familiar with the mmu stuff, so what you

Re: iommu_iova slab eats too much memory

2020-04-24 Thread Robin Murphy
On 2020-04-24 2:20 pm, Bin wrote: Dear Robin: Thank you for your explanation. Now, I understand that this could be NIC driver's fault, but how could I confirm it? Do I have to debug the driver myself? I'd start with CONFIG_DMA_API_DEBUG - of course it will chew through memory about an ord

Re: iommu_iova slab eats too much memory

2020-04-24 Thread John Garry
On 24/04/2020 17:30, Robin Murphy wrote: On 2020-04-24 2:20 pm, Bin wrote: Dear Robin: Thank you for your explanation. Now, I understand that this could be NIC driver's fault, but how could I confirm it? Do I have to debug the driver myself? I'd start with CONFIG_DMA_API_DEBUG - of cours

Re: iommu_iova slab eats too much memory

2020-04-25 Thread Bin
Dear John: Thank you for your reply. The case you mentioned is a typical performance regression issue, there's no need for the kernel to oom kill any random process even in the worst case. But in my observations, the iommu_iova slab could consume up to 40G memory, and the kernel have to kill my

RE: iommu_iova slab eats too much memory

2020-04-28 Thread Salil Mehta
age- > From: iommu [mailto:iommu-boun...@lists.linux-foundation.org] On Behalf Of > Robin Murphy > Sent: Friday, April 24, 2020 5:31 PM > To: Bin > Cc: iommu@lists.linux-foundation.org > Subject: Re: iommu_iova slab eats too much memory > > On 2020-04-24 2:20 pm

Re: iommu_iova slab eats too much memory

2020-04-28 Thread Bin
iommu-boun...@lists.linux-foundation.org] On Behalf > Of > > Robin Murphy > > Sent: Friday, April 24, 2020 5:31 PM > > To: Bin > > Cc: iommu@lists.linux-foundation.org > > Subject: Re: iommu_iova slab eats too much memory > > > > On 2020-04-24 2:20 pm

RE: iommu_iova slab eats too much memory

2020-04-29 Thread Salil Mehta
- > > From: iommu [mailto:iommu-boun...@lists.linux-foundation.org] On Behalf Of > > Robin Murphy > > Sent: Friday, April 24, 2020 5:31 PM > > To: Bin > > Cc: iommu@lists.linux-foundation.org > > Subject: Re: iommu_iova slab eats too much memory > > >

Re: iommu_iova slab eats too much memory

2020-04-29 Thread Robin Murphy
Robin Murphy Sent: Friday, April 24, 2020 5:31 PM To: Bin Cc: iommu@lists.linux-foundation.org Subject: Re: iommu_iova slab eats too much memory On 2020-04-24 2:20 pm, Bin wrote: Dear Robin: Thank you for your explanation. Now, I understand that this could be NIC driver's fault, but

RE: iommu_iova slab eats too much memory

2020-04-29 Thread Salil Mehta
DMA buffer leak not DMA > mapping > >> leak. I doubt VM will get killed due to exhaustion of the DMA mappings in > the > >> IOMMU > >> Layer for a transient reason or even due to mapping/unmapping leak. > >> > >> 2. Could you check if you have TSO offload e