Hello Ritesh,
Thanks a lot for reviewing this patch. My responses are inline
On 7/30/26 10:09 AM, Ritesh Harjani (IBM) wrote:
Hi Gaurav,
Thanks for the patch. Few observation, inputs and queries -
Gaurav Batra <[email protected]> writes:
In PowerPC (pseries) a non-virtualized adapter will have 2 DMA windows -
2GB default and a larger Dynamic DMA Window (DDW). DDW is large enough to
map total RAM to a device.
During normal functioning of OS, since RAM is pre-mapped, 2GB default
window is not used. The only scenario it might get used is when buffers in
pmemory are mapped to the device for DMA.
As of today, during kdump, during early device discovery, pci_dma_find()
finds that the device has 2 DMA windows. It selects to use DDW. This is a
kdump path and DMA window is needed for IO to the device.
Since, in the previous life of the LPAR (before panic), RAM was pre-mapped
via DDW, the DDW is completely full. So, in the iommu table initialization
code, iommu_table_clear() frees KDUMP_MIN_TCE_ENTRIES (2K) number of
TCEs.
But, it seems these are not enough for NVMe over Fibre-channel. When kdump
is trying to save vmcore on storage device, which is NVMe-FC, the
TCE usage is much more than 2K number of entries. The driver is mapping
a lot more buffers for DMA. After all the TCEs are consumed, iommu returns
iommu_alloc failures and the driver is not able to further map buffers for
IO. kdump fails to copy vmcore to NVMe-FC storage device.
Some context I collected while reviewing this patch -
1. kdump environment generally prefers configurations so that we could
avoid issues like memory allocation failures. I guess, we don't want to
be running the system with max configurations - that is also the reason
why distros keep nr_cpus to a lower value during kdump case.
For e.g. see this [1]:
scsi: lpfc: Limit xri count for kdump environment
scsi-mq operation inherently performs pre-allocation of resources for
blk-mq request queues. Even though the kdump environment reduces the
configuration to a single CPU, thus 1 hardware queue, which helps
significantly, the resources are still rather large due to the per
request
allocations.
<...>
[1]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=31f06d2e73726160645f8d9976a0b3f42e136da7
2. While looking more into this call stack path I also saw -
/*
* If a crashdump is active, then we are potentially in a very
- * memory constrained environment. Limit us to 1 queue and
- * 64 tags to prevent using too much memory.
+ * memory constrained environment. Limit us to 64 tags to
prevent
+ * using too much memory.
*/
- if (is_kdump_kernel()) {
- set->nr_hw_queues = 1;
- set->nr_maps = 1;
+ if (is_kdump_kernel())
set->queue_depth = min(64U, set->queue_depth);
- }
+
@@ -4515,7 +4513,7 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set
*set)
GFP_KERNEL, set->numa_node);
if (!set->map[i].mq_map)
goto out_free_mq_map;
- set->map[i].nr_queues = is_kdump_kernel() ? 1 :
set->nr_hw_queues;
+ set->map[i].nr_queues = set->nr_hw_queues;
}
blk_mq_update_queue_map(set);
So looks like we already reduce blk-mq queue_depth to 64 in
blk_mq_alloc_tag_set(), no matter what the queue_depth is passed to
us by the driver.
3. Also with above patch from v6.9 onwards, we made nr_queues as
set->nr_hw_queues, whereas earlier it was clamped to 1. Because the
code assumes that in kdump kernel we boot with nr_cpus=1..
blk-mq: don't change nr_hw_queues and nr_maps for kdump kernel
For most of ARCHs, 'nr_cpus=1' is passed for kdump kernel, so
nr_hw_queues for each mapping is supposed to be 1 already.
More importantly, this way may cause trouble for driver, because
blk-mq and
driver see different queue mapping since driver should setup hardware
queue setting before calling into allocating blk-mq tagset.
So not overriding nr_hw_queues and nr_maps for kdump kernel.
[1]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ec30b461f3d067bd322a6c3a6c5105746ed9bf14
I agree with your assessment that kdump executes in a very limited
environment
and drivers make an effort to use less resources. The goal is to dump
memory and
not performance.
So looking at above, I had few questions -
1. In your kdump kernel how many nr_cpus you are booting up with?
Recently what I heard rhel/sles might be using nr_cpus=16/32. @Sourabh?
So the calculation in nvme-fc driver then becomes:
32(nr_cpus) * 64(queue_depth) * 2(cmd+resp) = 4096
If you are able to reproduce this issue 100% of the time - then can
you try kdump with nr_cpus=1 and see whether it fixes your iommu alloc
failure?
Initially, when I reproduced this issue, I was not passing any value to
nr_cpus to the kdump kernel. I made changes to /etc/sysconfig/kdump to pass
nr_cpus=1. With this the kdump was successful. I tried with
nr_cpus=16/32. These
were successful as well. Though, in these cases, I did notice a few
iommu_alloc failures
(maybe < 10), vmcore was gathered successfully.
I started to see the issue with nr_cpus=64. My LPAR is configured with
max cpus = 64. So,
earlier, when I was not specifying nr_cpus in the /etc/sysconfig/kdump,
kdump could be
defaulting to 64 CPUs and hence allocating more resources during kdump.
2. Also were there more than 1 controller attached? Which can change the
above calculation then.
Only 1 controller. Here is the output of lscpi
ltcd41-lp11:~ # lspci
0153:70:00.0 Fibre Channel: Emulex Corporation LPe37000/LPe38000 Series
32Gb/64Gb Fibre Channel Adapter (rev 10)
0153:70:00.1 Fibre Channel: Emulex Corporation LPe37000/LPe38000 Series
32Gb/64Gb Fibre Channel Adapter (rev 10)
0153:70:00.2 Fibre Channel: Emulex Corporation LPe37000/LPe38000 Series
32Gb/64Gb Fibre Channel Adapter (rev 10)
0153:70:00.3 Fibre Channel: Emulex Corporation LPe37000/LPe38000 Series
32Gb/64Gb Fibre Channel Adapter (rev 10)
Looking at the lpfc and nvme-fc driver - a lot of the calculation are
based on nr_online_cpus. I somehow think if we clamp that value of
nr_online_cpus, we should stop seeing these alloc failures.
Thoughts?
Hopefully, if you can work on some above points further, it will also
explain why are we seeing this failures only now. Is this something that
has caused an issue after RHEL/SLES moved to nr_cpus=16/32?
it seems this got exposed because in my test LPAR, nr_cpus=1/16/32 was not
getting passed to the kdump kernel.
Or was it after this commit from v6.9?
ec30b461f3d: ("blk-mq: don't change nr_hw_queues and nr_maps for kdump
kernel")
Here are the driver logs and stack
lpfc 0153:70:00.0: iommu_alloc failed,
tbl 0000000034ebcf5e vaddr 00000000d814df0b npages 1
lpfc 0153:70:00.0: FCP Op failed - cmdiu dma mapping failed.
lpfc 0153:70:00.0: iommu_alloc failed,
tbl 0000000034ebcf5e vaddr 000000009779e4d2 npages 1
lpfc 0153:70:00.0: FCP Op failed - cmdiu dma mapping failed.
iommu_map_phys+0x1c4/0x1f0 (unreliable)
dma_iommu_map_phys+0x54/0xa0
dma_map_phys+0x3f8/0x590
__nvme_fc_init_request+0x110/0x300 [nvme_fc]
nvme_fc_init_request+0x60/0xb8 [nvme_fc]
blk_mq_alloc_map_and_rqs+0x388/0x510
blk_mq_alloc_tag_set+0x2a4/0x5f0
nvme_alloc_io_tag_set+0xe0/0x1e0 [nvme_core]
nvme_fc_connect_ctrl_work+0x85c/0xdac [nvme_fc]
process_one_work+0x1e4/0x5a0
worker_thread+0x1ec/0x3e0
Increasing the number of free TCE entries in iommu_table_clear() will
increase the probability of hitting EEH since there could still be some
active IOs from the previous life of the kernel.
Instead, during kdump, we can switch to default 2GB DMA window. This window
will mostly be empty. Or, could be slightly used if buffers in pmemory
were mapped for IO.
This will still remain a problem when we have SR-IOV adapter attached
correct? Because in that case we only get 1 window, so we anyway can't
use default window in kdump case. Correct?
you are right. The patch is fixing the dedicated adapter path only by
switching to
default window for kdump. Before I submitted the patch, I did try SR-IOV
path as well.
Here, I assigned a virtualized adapter to LPAR and gathered kdump over
NFS. I checked the
footprint of DMA buffers in this path. They were not much. I think, I
did sent these details
in my emails (the discussion/advice).
As of now SR-IOV path doesn't seems to be of concern. But, I think, the
correct overall fix
should be to maintain the DDW state --> if it is pre-mapped DDW,
transfer this knowledge to kdump.
With this, the DDW will be intact and buffers pre-mapped, as before.
But, this requires more work
and thorough testing by FVT/ISST. So, I kept this for later. For now,
switching to default window
seems to me the least invasive fix for this very narrow problem.
Your insight and thoughts?
Thanks a lot
Gaurav
-ritesh
