[ https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163555#comment-16163555 ]
ASF GitHub Bot commented on DRILL-5752: --------------------------------------- GitHub user ilooner opened a pull request: https://github.com/apache/drill/pull/940 DRILL-5752 Speed Up Unit Tests add Test Categories This PR does the following. **Increased Concurrency:** Previously tests only executed in 2 simultaneous processes, now one process per core is used to execute tests. This PR also fixed a few issues that popped up when concurrency was increased. **Test Categories:** Tests can now be executed according to their categories. There are two main categories of tests **Fast Tests** and **Secondary Tests**. **Fast Tests** are fast and are executed by default with the following command: ``` mvn -T 1C clean install ``` **Note:** -T 1C increased the number of threads used to compile classes, and allows us to run tests for sub modules in parallel. This command takes about 15 minutes to run on my laptop. **Secondary Tests** are slower. To run both fast tests and **Secondary Tests** you can use the following command. ``` mvn -T 1C clean install -DexcludedGroups="" ``` In addition to **Fast** and **Secondary** tests there are more categories like: - **JdbcTest** - **ParquetTest** - **HiveStorageTest** - And many more All of these categories are in the **drill-common** sub modules in **org.apahce.drill.categories**. All the tests are marked with one or more of these categories, and we can use these categories to select the tests we run. To run all the **Fast** tests that belong to the **JdbcTest** category we can use the following command: ``` mvn -T 1C clean install -Dgroups="org.apache.drill.categories.JdbcTest" ``` In order to run all the **Fast** AND **Secondary** JdbcTests you can use this command ``` mvn -T 1C clean install -Dgroups="org.apache.drill.categories.JdbcTest" -DexcludedGroups="" ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ilooner/drill DRILL-5752 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/940.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #940 ---- commit ba3d1ed53d7cabc5f5ac860531f6e989f567bc65 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-30T19:53:49Z - Removed redundant exclusion from java-exec pom - Marked the slow tests in contrib as secondary tests commit 9a3b7d2e1de85ce4397311e9ff2433432982fb03 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-30T22:02:23Z - Made slow unit tests SecondaryTests - Changed the number of forked test processes from just 2 processes to 1.5 process per core commit 4159bf8f3ad4966751b9176843e240bb4885cec5 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T00:10:38Z - Added JdbcTest category commit 0ab8ba8474b68bae8175f4b3c4a9b372f20eef8a Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T00:24:03Z - Removed unused imports from tests commit bf3748c6ddd96b303762d8fde9046215cb243c02 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T00:34:22Z - Added JdbcStorageTest Category commit 749fa9b1c4ae113a010de33bf359a38a1d56833e Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T18:15:18Z - Created the HiveStorageTest category - Made excludedGroups overrideable commit 50ddbb63f05eb08ac06d46dcf3c078e61dd3d042 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T18:39:28Z - Added HbaseStorageTest category commit 10097f61cb2df8898c2e7d8ff20cfb63fddcb6c5 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T18:50:33Z - Added the KuduStorageTest category commit 4dbad7edd5f479abc7db698e3ee2a2dad62abf20 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T19:06:34Z - Added the MongoStorageTest Category commit 5077b7beeccddcb0c84c70645c879cd3168f1bc1 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T19:42:09Z - Added a MemoryTest category commit a0e3832511041aeb18590d65ebb7166e40be045f Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T19:53:42Z - Add another secondary test. commit efc9b8bb5af3ce891b890217ed1595d0db26cbbb Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T20:03:08Z - Added a SecurityTest category - Made more of the security tests secondary tests commit 8dfacb0715252e063eca7d7a51895b8da191b273 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T20:05:05Z - Removed unused import for security test commit 72c3db7f1774fb0e37dd61a25ef1215f0391da63 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T20:05:57Z - Added another memory test commit 3fd24f8062887253a27bbdb8f31f12a06aa97270 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T20:22:14Z - Added an OperatorTest category commit 11290f9d48d223f8ca8b9c2df56e458616b51614 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T20:29:50Z - Added license headers - Added a ParquetTest category commit 3b03bb385466a2796578dd8ad96f2596a2b0f95e Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T20:33:35Z - Added a VectorTest category commit 4d242150cb8228ba044ad96706f09524664773bb Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T20:55:00Z - Added the SqlTest category commit 3cb3e895a9eb5171b88085c6984dbcb2685bbe58 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T20:59:14Z - Added more Vector tests commit 5f6ef77b65966b430a3b77ec577d3f2627133b59 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T21:01:07Z - Added more security tests commit 900911b2845e1a2b08d1f587afc505436ce6424e Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T21:03:26Z - Added another sql test commit 0fc54d716a9920bfd768d28314f9cdc30bf8d0d5 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T21:04:50Z - Added another security test commit 8e8c74b23646a3c6f203ae31468bf1e51787face Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T21:07:13Z - Added another vector test commit 0b0f42a85d327b5ef723aa4ba58b30a5854c8ca6 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T21:14:33Z - Added more operator and sql tests. commit f62ad15d3e0063032e55e857c898e08af7dc6084 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T21:39:01Z - Added the PlannerTest category - Categorized more tests commit 012e4a7b4048846e7ccca3b263c065daa1156d77 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T22:26:40Z - Added the SqlFunctionsTest category - Categorized more unit tests commit 04024ee2ae01d04070638e5e67f5dcc9cbc9305b Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-08-31T22:49:57Z - Fixed checkstyle errors commit c2d63579aaeac88ede9e369dee67db49a9595fc4 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-09-01T04:21:21Z - Moved test categories into their own package - Fixed race condition with jdbc storage plugin tests - Fixed prepared statement timeout with the drill-jdbc tests commit 47173b8616e5a08929f5a178206fbcb3431e1628 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-09-05T16:55:29Z - Cleaned up jdbc storage plugin pom - Increased the timeout of some unit tests - Increased the prepared statement creation timeout for some unit tests commit e0c9a361c7e089bec5713defbd41228369336325 Author: Timothy Farkas <timothyfar...@apache.org> Date: 2017-09-12T17:55:29Z - Decrease to one test process per core ---- > Speed Up Unit Tests > ------------------- > > Key: DRILL-5752 > URL: https://issues.apache.org/jira/browse/DRILL-5752 > Project: Apache Drill > Issue Type: Improvement > Reporter: Timothy Farkas > Assignee: Timothy Farkas > > Tests can be split into categories. > High-level categories: > * Fast > * Slow > Low-level categories: > * Vector > * WebUI > * Planner > * Operator > * Storage > * Hive > * JDBC > * Kudu > * Mongo > * Hbase > After the tests are categorized the Travis build can just run the fast tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)