Re: [PATCH] bitmap: Fix optimization of bitmap_set/clear for big-endian machines

[Michael Ellerman]

Matthew Wilcox  writes:

> (I don't think I can reliably send patches from outlook; sorry for
> breaking the threading)
>
> I see where we're not incrementing the failure count ... try this patch!
>
> --- 8< ---
>
> Subject: Fix bitmap optimisation tests to report errors correctly
> From: Matthew Wilcox 
>
> I had neglected to increment the error counter when the tests failed,
> which made the tests noisy when they fail, but not actually return an
> error code.
>
> Reported-by: Michael Ellerman 
> Signed-off-by: Matthew Wilcox 
> Cc: sta...@kernel.org

Thanks, that works for me.

  test_bitmap: failed 31840 out of 460506 tests

cheers

Re: [PATCH] bitmap: Fix optimization of bitmap_set/clear for big-endian machines

[Michael Ellerman]

Matthew Wilcox  writes:

> (I don't think I can reliably send patches from outlook; sorry for
> breaking the threading)
>
> I see where we're not incrementing the failure count ... try this patch!
>
> --- 8< ---
>
> Subject: Fix bitmap optimisation tests to report errors correctly
> From: Matthew Wilcox 
>
> I had neglected to increment the error counter when the tests failed,
> which made the tests noisy when they fail, but not actually return an
> error code.
>
> Reported-by: Michael Ellerman 
> Signed-off-by: Matthew Wilcox 
> Cc: sta...@kernel.org

Thanks, that works for me.

  test_bitmap: failed 31840 out of 460506 tests

cheers

Re: [PATCH] bitmap: Fix optimization of bitmap_set/clear for big-endian machines

[Matthew Wilcox]

(I don't think I can reliably send patches from outlook; sorry for
breaking the threading)

I see where we're not incrementing the failure count ... try this patch!

--- 8< ---

Subject: Fix bitmap optimisation tests to report errors correctly
From: Matthew Wilcox 

I had neglected to increment the error counter when the tests failed,
which made the tests noisy when they fail, but not actually return an
error code.

Reported-by: Michael Ellerman 
Signed-off-by: Matthew Wilcox 
Cc: sta...@kernel.org

diff --git a/lib/test_bitmap.c b/lib/test_bitmap.c
index aa1f2669bdd5..ae8a830e4e54 100644
--- a/lib/test_bitmap.c
+++ b/lib/test_bitmap.c
@@ -430,23 +430,32 @@ static void noinline __init test_mem_optimisations(void)
unsigned int start, nbits;
 
for (start = 0; start < 1024; start += 8) {
-   memset(bmap1, 0x5a, sizeof(bmap1));
-   memset(bmap2, 0x5a, sizeof(bmap2));
for (nbits = 0; nbits < 1024 - start; nbits += 8) {
+   memset(bmap1, 0x5a, sizeof(bmap1));
+   memset(bmap2, 0x5a, sizeof(bmap2));
+
bitmap_set(bmap1, start, nbits);
__bitmap_set(bmap2, start, nbits);
-   if (!bitmap_equal(bmap1, bmap2, 1024))
+   if (!bitmap_equal(bmap1, bmap2, 1024)) {
printk("set not equal %d %d\n", start, nbits);
-   if (!__bitmap_equal(bmap1, bmap2, 1024))
+   failed_tests++;
+   }
+   if (!__bitmap_equal(bmap1, bmap2, 1024)) {
printk("set not __equal %d %d\n", start, nbits);
+   failed_tests++;
+   }
 
bitmap_clear(bmap1, start, nbits);
__bitmap_clear(bmap2, start, nbits);
-   if (!bitmap_equal(bmap1, bmap2, 1024))
+   if (!bitmap_equal(bmap1, bmap2, 1024)) {
printk("clear not equal %d %d\n", start, nbits);
-   if (!__bitmap_equal(bmap1, bmap2, 1024))
+   failed_tests++;
+   }
+   if (!__bitmap_equal(bmap1, bmap2, 1024)) {
printk("clear not __equal %d %d\n", start,
nbits);
+   failed_tests++;
+   }
}
}
 }

Re: [PATCH] bitmap: Fix optimization of bitmap_set/clear for big-endian machines

[Matthew Wilcox]

(I don't think I can reliably send patches from outlook; sorry for
breaking the threading)

I see where we're not incrementing the failure count ... try this patch!

--- 8< ---

Subject: Fix bitmap optimisation tests to report errors correctly
From: Matthew Wilcox 

I had neglected to increment the error counter when the tests failed,
which made the tests noisy when they fail, but not actually return an
error code.

Reported-by: Michael Ellerman 
Signed-off-by: Matthew Wilcox 
Cc: sta...@kernel.org

diff --git a/lib/test_bitmap.c b/lib/test_bitmap.c
index aa1f2669bdd5..ae8a830e4e54 100644
--- a/lib/test_bitmap.c
+++ b/lib/test_bitmap.c
@@ -430,23 +430,32 @@ static void noinline __init test_mem_optimisations(void)
unsigned int start, nbits;
 
for (start = 0; start < 1024; start += 8) {
-   memset(bmap1, 0x5a, sizeof(bmap1));
-   memset(bmap2, 0x5a, sizeof(bmap2));
for (nbits = 0; nbits < 1024 - start; nbits += 8) {
+   memset(bmap1, 0x5a, sizeof(bmap1));
+   memset(bmap2, 0x5a, sizeof(bmap2));
+
bitmap_set(bmap1, start, nbits);
__bitmap_set(bmap2, start, nbits);
-   if (!bitmap_equal(bmap1, bmap2, 1024))
+   if (!bitmap_equal(bmap1, bmap2, 1024)) {
printk("set not equal %d %d\n", start, nbits);
-   if (!__bitmap_equal(bmap1, bmap2, 1024))
+   failed_tests++;
+   }
+   if (!__bitmap_equal(bmap1, bmap2, 1024)) {
printk("set not __equal %d %d\n", start, nbits);
+   failed_tests++;
+   }
 
bitmap_clear(bmap1, start, nbits);
__bitmap_clear(bmap2, start, nbits);
-   if (!bitmap_equal(bmap1, bmap2, 1024))
+   if (!bitmap_equal(bmap1, bmap2, 1024)) {
printk("clear not equal %d %d\n", start, nbits);
-   if (!__bitmap_equal(bmap1, bmap2, 1024))
+   failed_tests++;
+   }
+   if (!__bitmap_equal(bmap1, bmap2, 1024)) {
printk("clear not __equal %d %d\n", start,
nbits);
+   failed_tests++;
+   }
}
}
 }

Re: [PATCH] bitmap: Fix optimization of bitmap_set/clear for big-endian machines

[Michael Ellerman]

Paul Mackerras  writes:

> On Wed, Oct 25, 2017 at 07:39:48AM +, Matthew Wilcox wrote:
>> Hang on, don't tell me you found this by inspection. Are you not
>> running the bitmap testcase, enabled by CONFIG_TEST_BITMAP? Either
>> that should be producing an error, or there's a missing test case, or
>> your inspection is wrong ...

Oops.

  [1.907533] test_bitmap: test 13: input is '0-2047:128/256' OK, Time: 1359
  [1.907596] set not equal 0 8
  [1.907640] set not __equal 0 8
  [1.907684] clear not equal 0 8
  [1.907728] clear not __equal 0 8
  [1.907772] set not equal 0 16
  [1.907816] set not __equal 0 16
  [1.907861] clear not equal 0 16
  [1.907905] clear not __equal 0 16
  [1.907949] set not equal 0 24
  [1.907993] set not __equal 0 24
  [1.908038] clear not equal 0 24
  [1.908082] clear not __equal 0 24
  ...
  [ snip ~30,000 lines ]
  ...
  [3.345729] set not equal 1008 8
  [3.345773] set not __equal 1008 8
  [3.345818] clear not equal 1008 8
  [3.345863] clear not __equal 1008 8

  [3.345909] test_bitmap: all 460506 tests passed
  ^^^
  O_o

With the patch:

  [1.904393] test_bitmap: test 13: input is '0-2047:128/256' OK, Time: 1359
  [1.916270] test_bitmap: all 460506 tests passed

cheers

Re: [PATCH] bitmap: Fix optimization of bitmap_set/clear for big-endian machines

[Michael Ellerman]

Paul Mackerras  writes:

> On Wed, Oct 25, 2017 at 07:39:48AM +, Matthew Wilcox wrote:
>> Hang on, don't tell me you found this by inspection. Are you not
>> running the bitmap testcase, enabled by CONFIG_TEST_BITMAP? Either
>> that should be producing an error, or there's a missing test case, or
>> your inspection is wrong ...

Oops.

  [1.907533] test_bitmap: test 13: input is '0-2047:128/256' OK, Time: 1359
  [1.907596] set not equal 0 8
  [1.907640] set not __equal 0 8
  [1.907684] clear not equal 0 8
  [1.907728] clear not __equal 0 8
  [1.907772] set not equal 0 16
  [1.907816] set not __equal 0 16
  [1.907861] clear not equal 0 16
  [1.907905] clear not __equal 0 16
  [1.907949] set not equal 0 24
  [1.907993] set not __equal 0 24
  [1.908038] clear not equal 0 24
  [1.908082] clear not __equal 0 24
  ...
  [ snip ~30,000 lines ]
  ...
  [3.345729] set not equal 1008 8
  [3.345773] set not __equal 1008 8
  [3.345818] clear not equal 1008 8
  [3.345863] clear not __equal 1008 8

  [3.345909] test_bitmap: all 460506 tests passed
  ^^^
  O_o

With the patch:

  [1.904393] test_bitmap: test 13: input is '0-2047:128/256' OK, Time: 1359
  [1.916270] test_bitmap: all 460506 tests passed

cheers

RE: [PATCH] bitmap: Fix optimization of bitmap_set/clear for big-endian machines

[Matthew Wilcox]

From: Paul Mackerras [mailto:pau...@ozlabs.org]
> On Wed, Oct 25, 2017 at 07:39:48AM +, Matthew Wilcox wrote:
> > Hang on, don't tell me you found this by inspection.  Are you not running 
> > the
> bitmap testcase, enabled by CONFIG_TEST_BITMAP?  Either that should be
> producing an error, or there's a missing test case, or your inspection is 
> wrong ...
> 
> I did find it by inspection.  I was looking for a version of the
> bitmap_* API that does little-endian style bitmaps on all systems, and
> the inline bitmap_set() does that in the case where it calls memset,
> but not in the case where it calls __bitmap_set.

I do believe that you noticed it by inspection, but you shouldn't've had to.  I 
thought we had a comprehensive set of tests for exactly this, which means that 
either 01.org isn't running the right set of tests on a BE system or the tests 
are broken.

> I'll fire up a big-endian system tomorrow when I get to work to run
> the test case.  (PPC64 is almost entirely little-endian these days as
> far as the IBM POWER systems are concerned.)
> 
> In any case, it's pretty clearly wrong as it is.  On a big-endian
> 64-bit system, bitmap_set(p, 56, 16) should set bytes 0 and 15 to
> 0xff, and there's no way a single memset can do that.

So ... this loop should include that case, right?

for (start = 0; start < 1024; start += 8) {
memset(bmap1, 0x5a, sizeof(bmap1));
memset(bmap2, 0x5a, sizeof(bmap2));
for (nbits = 0; nbits < 1024 - start; nbits += 8) {
bitmap_set(bmap1, start, nbits);
__bitmap_set(bmap2, start, nbits);
if (!bitmap_equal(bmap1, bmap2, 1024))
printk("set not equal %d %d\n", start, nbits);
if (!__bitmap_equal(bmap1, bmap2, 1024))
printk("set not __equal %d %d\n", start, nbits);

RE: [PATCH] bitmap: Fix optimization of bitmap_set/clear for big-endian machines

[Matthew Wilcox]

From: Paul Mackerras [mailto:pau...@ozlabs.org]
> On Wed, Oct 25, 2017 at 07:39:48AM +, Matthew Wilcox wrote:
> > Hang on, don't tell me you found this by inspection.  Are you not running 
> > the
> bitmap testcase, enabled by CONFIG_TEST_BITMAP?  Either that should be
> producing an error, or there's a missing test case, or your inspection is 
> wrong ...
> 
> I did find it by inspection.  I was looking for a version of the
> bitmap_* API that does little-endian style bitmaps on all systems, and
> the inline bitmap_set() does that in the case where it calls memset,
> but not in the case where it calls __bitmap_set.

I do believe that you noticed it by inspection, but you shouldn't've had to.  I 
thought we had a comprehensive set of tests for exactly this, which means that 
either 01.org isn't running the right set of tests on a BE system or the tests 
are broken.

> I'll fire up a big-endian system tomorrow when I get to work to run
> the test case.  (PPC64 is almost entirely little-endian these days as
> far as the IBM POWER systems are concerned.)
> 
> In any case, it's pretty clearly wrong as it is.  On a big-endian
> 64-bit system, bitmap_set(p, 56, 16) should set bytes 0 and 15 to
> 0xff, and there's no way a single memset can do that.

So ... this loop should include that case, right?

for (start = 0; start < 1024; start += 8) {
memset(bmap1, 0x5a, sizeof(bmap1));
memset(bmap2, 0x5a, sizeof(bmap2));
for (nbits = 0; nbits < 1024 - start; nbits += 8) {
bitmap_set(bmap1, start, nbits);
__bitmap_set(bmap2, start, nbits);
if (!bitmap_equal(bmap1, bmap2, 1024))
printk("set not equal %d %d\n", start, nbits);
if (!__bitmap_equal(bmap1, bmap2, 1024))
printk("set not __equal %d %d\n", start, nbits);

Re: [PATCH] bitmap: Fix optimization of bitmap_set/clear for big-endian machines

[Paul Mackerras]

On Wed, Oct 25, 2017 at 07:39:48AM +, Matthew Wilcox wrote:
> Hang on, don't tell me you found this by inspection.  Are you not running the 
> bitmap testcase, enabled by CONFIG_TEST_BITMAP?  Either that should be 
> producing an error, or there's a missing test case, or your inspection is 
> wrong ...

I did find it by inspection.  I was looking for a version of the
bitmap_* API that does little-endian style bitmaps on all systems, and
the inline bitmap_set() does that in the case where it calls memset,
but not in the case where it calls __bitmap_set.

I'll fire up a big-endian system tomorrow when I get to work to run
the test case.  (PPC64 is almost entirely little-endian these days as
far as the IBM POWER systems are concerned.)

In any case, it's pretty clearly wrong as it is.  On a big-endian
64-bit system, bitmap_set(p, 56, 16) should set bytes 0 and 15 to
0xff, and there's no way a single memset can do that.

Paul.

(and yes, I stuffed up the address for lkml)

Re: [PATCH] bitmap: Fix optimization of bitmap_set/clear for big-endian machines

[Paul Mackerras]

On Wed, Oct 25, 2017 at 07:39:48AM +, Matthew Wilcox wrote:
> Hang on, don't tell me you found this by inspection.  Are you not running the 
> bitmap testcase, enabled by CONFIG_TEST_BITMAP?  Either that should be 
> producing an error, or there's a missing test case, or your inspection is 
> wrong ...

I did find it by inspection.  I was looking for a version of the
bitmap_* API that does little-endian style bitmaps on all systems, and
the inline bitmap_set() does that in the case where it calls memset,
but not in the case where it calls __bitmap_set.

I'll fire up a big-endian system tomorrow when I get to work to run
the test case.  (PPC64 is almost entirely little-endian these days as
far as the IBM POWER systems are concerned.)

In any case, it's pretty clearly wrong as it is.  On a big-endian
64-bit system, bitmap_set(p, 56, 16) should set bytes 0 and 15 to
0xff, and there's no way a single memset can do that.

Paul.

(and yes, I stuffed up the address for lkml)