GitHub user samarthjain opened a pull request:
https://github.com/apache/incubator-phoenix/pull/33
Version two for making phoenix tests run faster
Following is the overview of changes in this commit:
1) We now have integration tests divided in three categories:
HBaseTimeManaged, ClientTimeManaged and TestThatNeedsOwnMiniCluster.
HBaseTimeManaged and ClientTimeManaged tests use one mini cluster isolated
for each category of tests. We don't need to any longer start and shutdown
mini-cluster at the beginning and end of tests.
Tests categorized as TestThatNeedsOwnMiniCluster spin up their own
mini-clusters.
2) The number of threads used by a mini cluster has been reduced from
around 200 to 130. Also, the max heap memory allocated while running
integration tests is now reduced to 2GB from 3GB. The two changes let us run
more number of forked JVMs running integration tests. For example - on my 24GB
8 core machine, I can easily run up to 10 forked JVMs. Reducing the number of
threads used by mini-cluster also helps us in preventing hitting the number of
threads limit imposed by OS.
3) All the tests have been now annotated as belonging to one of the three
categories. Once, Junit4.12 is out, we hopefully won't need to add annotations
for every new test class. Just assigning the Category annotation to the base
test class will be sufficient.
4) QueryIT has been broken up into various tests. This helps in parallelism
as otherwise this test class would alone take around 4min 30 seconds to finish.
Breaking up into different sub-classes helped reduce the execution time by
around a minute.
5) Timing numbers:
On my dev box, running 8 concurrent JVMS using mvn install -DnumForkedIT=8,
tests under phoenix-core module finished in approximately 6 min 20 seconds. The
same used to take around 50 mins before the optimization work. v1 of faster
tests that spun up and down a mini cluster for every test used to take around
10 mins for the phoenix-core module.
The files with the major changes are:
pom.xml
PhoenixTestDriver.java
ConnectionQueryServicesTestImpl.java
BaseTest.java - a lot of code form BaseConnectedQueryIT.java has been moved
here.
QueryIT.java
ClientManagedTimeTest.java
HBaseManagedTimeTest.java
NeedsOwnMinitClusterTest.java
BaseClientManagedTimeIT.java
BaseHBaseManagedTimeIT.java
BaseConnectedQueryIT.java - deleted
Shadower.java - method level annotation to help tell the developers that a
public static method is being shadowed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/samarthjain/incubator-phoenix master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-phoenix/pull/33.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #33
----
commit 6bc4380c38e37a5e2fa0203218498db2c9f49505
Author: samarthjain <[email protected]>
Date: 2014-05-16T01:58:00Z
Faster tests v2
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
