[PATCH] kernel/fork.c: VM accounting bugfix (2.6.11-rc3-bk5)

[Mark F. Haigh]

Unless I'm missing something, in kernel/fork.c, dup_mmap():
                        if (security_vm_enough_memory(len))
                                goto fail_nomem;
/* ... */
fail_nomem:
        retval = -ENOMEM;
        vm_unacct_memory(charge);
/* ... */
If security_vm_enough_memory() fails there, then we vm_unacct_memory() 
that we never accounted (if security_vm_enough_memory() fails, no memory 
is accounted).

If it is in fact a bug, a simple but largely untested patch (against 
2.6.11-rc3-bk5) is included.

Mark F. Haigh
[EMAIL PROTECTED]
--- linux-2.6.11-rc3-bk5/kernel/fork.c.orig     2005-02-08 19:12:26.254589504 
-0800
+++ linux-2.6.11-rc3-bk5/kernel/fork.c  2005-02-08 19:16:30.756419576 -0800
@@ -193,8 +193,10 @@
                charge = 0;
                if (mpnt->vm_flags & VM_ACCOUNT) {
                        unsigned int len = (mpnt->vm_end - mpnt->vm_start) >> 
PAGE_SHIFT;
-                       if (security_vm_enough_memory(len))
-                               goto fail_nomem;
+                       if (security_vm_enough_memory(len)) {
+                               retval = -ENOMEM;
+                               goto out;
+                       }
                        charge = len;
                }
                tmp = kmem_cache_alloc(vm_area_cachep, SLAB_KERNEL);