From: Sultan Alsawaf <sul...@kerneltoast.com> Typically, drivers allocate sg lists of sizes up to a few MiB in size. The current algorithm deals with large sg lists by splitting them into several smaller arrays and chaining them together. But if the sg list allocation is large, and we know the size ahead of time, sg chaining is both inefficient and unnecessary.
Rather than calling kmalloc hundreds of times in a loop for chaining tiny arrays, we can simply do it all at once with kvmalloc, which has the proper tradeoff on when to stop using kmalloc and instead use vmalloc. Abusing repeated kmallocs to produce a large allocation puts strain on the slab allocator, when kvmalloc can be used instead. The single kvmalloc allocation for all sg lists reduces the burden on the slab and page allocators, since for large sg list allocations, this commit replaces numerous kmalloc calls with one kvmalloc call. The sg chaining is effectively disabled by changing SG_MAX_SINGLE_ALLOC to UINT_MAX, which causes sg list allocations to never be split into chains, since no allocation is larger than UINT_MAX. We then plumb kvmalloc into the allocation functions so that it is used. Signed-off-by: Sultan Alsawaf <sul...@kerneltoast.com> --- include/linux/scatterlist.h | 2 +- lib/scatterlist.c | 23 ++--------------------- 2 files changed, 3 insertions(+), 22 deletions(-) diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h index 6eec50fb36c8..e2e26c53c441 100644 --- a/include/linux/scatterlist.h +++ b/include/linux/scatterlist.h @@ -310,7 +310,7 @@ size_t sg_zero_buffer(struct scatterlist *sgl, unsigned int nents, * Maximum number of entries that will be allocated in one piece, if * a list larger than this is required then chaining will be utilized. */ -#define SG_MAX_SINGLE_ALLOC (PAGE_SIZE / sizeof(struct scatterlist)) +#define SG_MAX_SINGLE_ALLOC (UINT_MAX) /* * The maximum number of SG segments that we will put inside a diff --git a/lib/scatterlist.c b/lib/scatterlist.c index c2cf2c311b7d..bf76854a34aa 100644 --- a/lib/scatterlist.c +++ b/lib/scatterlist.c @@ -148,31 +148,12 @@ EXPORT_SYMBOL(sg_init_one); */ static struct scatterlist *sg_kmalloc(unsigned int nents, gfp_t gfp_mask) { - if (nents == SG_MAX_SINGLE_ALLOC) { - /* - * Kmemleak doesn't track page allocations as they are not - * commonly used (in a raw form) for kernel data structures. - * As we chain together a list of pages and then a normal - * kmalloc (tracked by kmemleak), in order to for that last - * allocation not to become decoupled (and thus a - * false-positive) we need to inform kmemleak of all the - * intermediate allocations. - */ - void *ptr = (void *) __get_free_page(gfp_mask); - kmemleak_alloc(ptr, PAGE_SIZE, 1, gfp_mask); - return ptr; - } else - return kmalloc_array(nents, sizeof(struct scatterlist), - gfp_mask); + return kvmalloc_array(nents, sizeof(struct scatterlist), gfp_mask); } static void sg_kfree(struct scatterlist *sg, unsigned int nents) { - if (nents == SG_MAX_SINGLE_ALLOC) { - kmemleak_free(sg); - free_page((unsigned long) sg); - } else - kfree(sg); + kvfree(sg); } /** -- 2.22.0