[ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc
> +queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL); How big might this buffer be? Any chance of allocation failure due to memory fragmentation? - R. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc
Hi Roland, thanks for the quick review. I was hoping you could apply these changes for 2.6.30 because this will be the codebase for the next OFED release. The patch is well tested in HPC environment and we haven't seen any issues. Regarding Antons patch you are right. If a user allocates an unrealistically large queue pair it could happen that kmalloc() is not able to allocate the memory. In this case we will return ENOMEM to the user so the kernel will not be affected at all. We plan to add vmalloc() call in case kmalloc() fails for the next kernel release. Mit freundlichen Grüßen / Kind regards Stefan Roscher eHCA/eHEA Linux Driver Development IBM Systems &Technology Group, Systems Software Development / FW I/O Firmware Entwicklung 2 --- IBM Deutschland Schoenaicher Str. 220 71032 Boeblingen Phone: +49-7031-16-2015 E-Mail: stefan.rosc...@de.ibm.com --- IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Geschäftsführung: Herbert Kircher Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: Roland Dreier To: Stefan Roscher Cc: "LinuxPPC-Dev" , LKML , "OF-EWG" , Roland Dreier , Joachim Fenkes/Germany/i...@ibmde, Christoph Raisch/Germany/i...@ibmde, Alexander Schmidt1/Germany/i...@ibmde, Stefan Roscher/Germany/i...@ibmde, Hoang-Nam Nguyen/Germany/i...@ibmde Date: 21.04.2009 19:34 Subject: Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc > + queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL); How big might this buffer be? Any chance of allocation failure due to memory fragmentation? - R. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc
In case of large queue pairs there is the possibillity of allocation failures
due to memory fragmentationo with kmalloc().To ensure the memory is allocated
even
if kmalloc() can not find chunks which are big enough, we try to allocate the
memory
with vmalloc().
Signed-off-by: Stefan Roscher
---
On Tuesday 21 April 2009 07:34:30 pm Roland Dreier wrote:
> > + queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
>
> How big might this buffer be? Any chance of allocation failure due to
> memory fragmentation?
>
> - R.
Hey Roland,
yes you are right and here is the patch to circumvent the described problem.
It will apply on top of the patchset.
regards Stefan
drivers/infiniband/hw/ehca/ipz_pt_fn.c | 17 +
1 files changed, 13 insertions(+), 4 deletions(-)
diff --git a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
index a260559..1227c59 100644
--- a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
+++ b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
@@ -222,8 +222,11 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue
*queue,
/* allocate queue page pointers */
queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
if (!queue->queue_pages) {
- ehca_gen_err("Couldn't allocate queue page list");
- return 0;
+ queue->queue_pages = vmalloc(nr_of_pages * sizeof(void *));
+ if (!queue->queue_pages) {
+ ehca_gen_err("Couldn't allocate queue page list");
+ return 0;
+ }
}
memset(queue->queue_pages, 0, nr_of_pages * sizeof(void *));
@@ -240,7 +243,10 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue
*queue,
ipz_queue_ctor_exit0:
ehca_gen_err("Couldn't alloc pages queue=%p "
"nr_of_pages=%x", queue, nr_of_pages);
- kfree(queue->queue_pages);
+ if (is_vmalloc_addr(queue->queue_pages))
+ vfree(queue->queue_pages);
+ else
+ kfree(queue->queue_pages);
return 0;
}
@@ -262,7 +268,10 @@ int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue
*queue)
free_page((unsigned long)queue->queue_pages[i]);
}
- kfree(queue->queue_pages);
+ if (is_vmalloc_addr(queue->queue_pages))
+ vfree(queue->queue_pages);
+ else
+ kfree(queue->queue_pages);
return 1;
}
--
1.5.5
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc
On Wednesday 22 April 2009 04:10:18 pm michael wrote: > Hi, > > I don't take the point, if it is not import use the vmalloc. Why you try > with a kmalloc > alloc first? and why do not use kzalloc? Because kmalloc() is faster than vmalloc() causing a huge performance win when someone allocates a large number of queue pairs. We fall back to vmalloc() only if kmalloc() can't deliver the memory chunk. We don't need kzalloc because we fill the list right after the alloc. regards Stefan ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc
On Tue, 2009-04-21 at 17:16 +0200, Stefan Roscher wrote: > From: Anton Blanchard > > To improve performance of driver ressource allocation, > replace the vmalloc() call with kmalloc(). Just curious, but how big are these allocations? Why was vmalloc() even ever used if we know they'll be small? -- Dave ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc
thanks, applied. > From: Anton Blanchard > Signed-off-by: Stefan Roscher please use '@' signs so these are real email addresses. - R. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc
Hi Roland,
did you have a chance to take a look at the patchset and will you apply it, or
are there any outstanding issues we need to address?
Regards,
Alex
On Wed, 22 Apr 2009 16:02:28 +0200
Stefan Roscher wrote:
> In case of large queue pairs there is the possibillity of allocation failures
> due to memory fragmentationo with kmalloc().To ensure the memory is allocated
> even
> if kmalloc() can not find chunks which are big enough, we try to allocate the
> memory
> with vmalloc().
>
> Signed-off-by: Stefan Roscher
> ---
>
> On Tuesday 21 April 2009 07:34:30 pm Roland Dreier wrote:
> > > +queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *),
> > GFP_KERNEL);
> >
> > How big might this buffer be? Any chance of allocation failure due to
> > memory fragmentation?
> >
> > - R.
> Hey Roland,
> yes you are right and here is the patch to circumvent the described problem.
> It will apply on top of the patchset.
> regards Stefan
>
>
>
> drivers/infiniband/hw/ehca/ipz_pt_fn.c | 17 +
> 1 files changed, 13 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> index a260559..1227c59 100644
> --- a/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> +++ b/drivers/infiniband/hw/ehca/ipz_pt_fn.c
> @@ -222,8 +222,11 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue
> *queue,
> /* allocate queue page pointers */
> queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
> if (!queue->queue_pages) {
> - ehca_gen_err("Couldn't allocate queue page list");
> - return 0;
> + queue->queue_pages = vmalloc(nr_of_pages * sizeof(void *));
> + if (!queue->queue_pages) {
> + ehca_gen_err("Couldn't allocate queue page list");
> + return 0;
> + }
> }
> memset(queue->queue_pages, 0, nr_of_pages * sizeof(void *));
>
> @@ -240,7 +243,10 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue
> *queue,
> ipz_queue_ctor_exit0:
> ehca_gen_err("Couldn't alloc pages queue=%p "
>"nr_of_pages=%x", queue, nr_of_pages);
> - kfree(queue->queue_pages);
> + if (is_vmalloc_addr(queue->queue_pages))
> + vfree(queue->queue_pages);
> + else
> + kfree(queue->queue_pages);
>
> return 0;
> }
> @@ -262,7 +268,10 @@ int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue
> *queue)
> free_page((unsigned long)queue->queue_pages[i]);
> }
>
> - kfree(queue->queue_pages);
> + if (is_vmalloc_addr(queue->queue_pages))
> + vfree(queue->queue_pages);
> + else
> + kfree(queue->queue_pages);
>
> return 1;
> }
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc
> did you have a chance to take a look at the patchset and will you apply it, > or > are there any outstanding issues we need to address? I guess it's OK, but definitely 2.6.31 material. I guess I'll stick it linux-next soon. - R. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc
On Tue, 28 Apr 2009 07:01:32 -0700 Roland Dreier wrote: > > did you have a chance to take a look at the patchset and will you apply > it, or > > are there any outstanding issues we need to address? > > I guess it's OK, but definitely 2.6.31 material. I guess I'll stick it > linux-next soon. > > - R. Okay with us, thank you very much! Alex ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
