Re: BaseLineAndCompareQueryPerfJUnitTest failure

[Dan Smith]

We do have support for running micro benchmarks with jmh - see the
geode-benchmarks project. But we don't have anything to watch and fail the
build if the performance gets worse. That may be kinda hard to do since
we're running on shared infrastructure.

-Dan

On Thu, Feb 9, 2017 at 6:13 PM, Anilkumar Gingade 
wrote:

> It was added to identify performance issues sooner than later...The test
> run queries for multiple times so that, jvm hot-spot is kicked-in and
> stabilized...
>
> You are right, functional-testing is not the right place for this...
>
> We may need to think about performance test framework/category for Geode...
>
> -Anil.
>
>
> On Thu, Feb 9, 2017 at 3:53 PM, Kirk Lund  wrote:
>
> > I just had a precheckin fail in BaseLineAndCompareQueryPerfJUnitTest.
> I'm
> > not sure a junit test is really a good place for perf testing like this.
> > Between JVM optimizing at runtime and possibility for machine CPU
> > pollution, I'm surprised we haven't seen this fail before. Just wanted to
> > mention this in case others see it or someone wants to take a closer look
> > at the test...
> >
> > :geode-core:integrationTest
> >
> > org.apache.geode.cache.query.BaseLineAndCompareQueryPerfJUnitTest >
> > testPerformanceForRangeQueries FAILED
> > java.lang.AssertionError: Avg. Time taken to query with index should
> be
> > less than time taken without index
> > at org.junit.Assert.fail(Assert.java:88)
> > at org.junit.Assert.assertTrue(Assert.java:41)
> > at
> > org.apache.geode.cache.query.BaseLineAndCompareQueryPerfJUnitTest.
> > testPerformanceForRangeQueries(BaseLineAndCompareQueryPerfJUn
> > itTest.java:299)
> >
> > 3360 tests completed, 1 failed, 165 skipped
> > :geode-core:integrationTest FAILED
> >
>

Re: BaseLineAndCompareQueryPerfJUnitTest failure

[Anilkumar Gingade]

It was added to identify performance issues sooner than later...The test
run queries for multiple times so that, jvm hot-spot is kicked-in and
stabilized...

You are right, functional-testing is not the right place for this...

We may need to think about performance test framework/category for Geode...

-Anil.


On Thu, Feb 9, 2017 at 3:53 PM, Kirk Lund  wrote:

> I just had a precheckin fail in BaseLineAndCompareQueryPerfJUnitTest. I'm
> not sure a junit test is really a good place for perf testing like this.
> Between JVM optimizing at runtime and possibility for machine CPU
> pollution, I'm surprised we haven't seen this fail before. Just wanted to
> mention this in case others see it or someone wants to take a closer look
> at the test...
>
> :geode-core:integrationTest
>
> org.apache.geode.cache.query.BaseLineAndCompareQueryPerfJUnitTest >
> testPerformanceForRangeQueries FAILED
> java.lang.AssertionError: Avg. Time taken to query with index should be
> less than time taken without index
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.geode.cache.query.BaseLineAndCompareQueryPerfJUnitTest.
> testPerformanceForRangeQueries(BaseLineAndCompareQueryPerfJUn
> itTest.java:299)
>
> 3360 tests completed, 1 failed, 165 skipped
> :geode-core:integrationTest FAILED
>

BaseLineAndCompareQueryPerfJUnitTest failure

[Kirk Lund]

I just had a precheckin fail in BaseLineAndCompareQueryPerfJUnitTest. I'm
not sure a junit test is really a good place for perf testing like this.
Between JVM optimizing at runtime and possibility for machine CPU
pollution, I'm surprised we haven't seen this fail before. Just wanted to
mention this in case others see it or someone wants to take a closer look
at the test...

:geode-core:integrationTest

org.apache.geode.cache.query.BaseLineAndCompareQueryPerfJUnitTest >
testPerformanceForRangeQueries FAILED
java.lang.AssertionError: Avg. Time taken to query with index should be
less than time taken without index
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.geode.cache.query.BaseLineAndCompareQueryPerfJUnitTest.testPerformanceForRangeQueries(BaseLineAndCompareQueryPerfJUnitTest.java:299)

3360 tests completed, 1 failed, 165 skipped
:geode-core:integrationTest FAILED