[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138506769 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java --- @@ -54,11 +56,13 @@ import org.junit.Ignore; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.TemporaryFolder; import org.junit.runner.RunWith; import org.junit.runners.Parameterized; @RunWith(Parameterized.class) +@Category({SecondaryTest.class, ParquetTest.class}) --- End diff -- Please see Terminology update in comment above. ---
[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138506701 --- Diff: common/src/test/java/org/apache/drill/categories/package-info.java --- @@ -0,0 +1,23 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * This package stores the JUnit test categories. + */ --- End diff -- Terminology update. I'm renaming **SecondaryTest** to **SlowTest** and instead of **BugFixTest** I will create a category called **UnlikelyTest**. **UnlikelyTest** will be a category for tests that excercise specific bug fixes, or are for tests that you're unlikely to break. ---
[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138505403 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/xsort/TestSimpleExternalSort.java --- @@ -148,10 +149,10 @@ public void sortOneKeyDescendingExternalSortLegacy() throws Throwable { private void sortOneKeyDescendingExternalSort(boolean testLegacy) throws Throwable { FixtureBuilder builder = ClusterFixture.builder() -.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4) -.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_GROUP_SIZE, 4) -.configProperty(ExecConstants.EXTERNAL_SORT_BATCH_LIMIT, 4) -.configProperty(ExecConstants.EXTERNAL_SORT_DISABLE_MANAGED, false) +.configProperty(OptionValidator.OPTION_DEFAULTS_ROOT + ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4) --- End diff -- Done, thanks for catching. ---
[jira] [Created] (DRILL-5785) Query Error on a large PCAP file
Takeo Ogawara created DRILL-5785: Summary: Query Error on a large PCAP file Key: DRILL-5785 URL: https://issues.apache.org/jira/browse/DRILL-5785 Project: Apache Drill Issue Type: Bug Components: Storage - Other Affects Versions: 1.11.0 Reporter: Takeo Ogawara Priority: Minor Query on a very large PCAP file (larger than 100GB) failed with following error message. > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a > > Fragment 1:169 > > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] > (state=,code=0) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] drill issue #936: DRILL-5772: Add unit tests to indicate how utf-8 support c...
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/936 @arina-ielchiieva, thanks for the explanation. Drill's runtime framework assumes that data is either: 1. ASCII (or, at least, single-byte character set based on ASCII), or 2. UTF-8 (data is converted to/from UTF-8 when converting VarChar to a Java String.) Since Drill's code seems to assume ASCII (when it cares about character format), then one could claim that Drill does not have an encoding: it just treats characters as bytes. But, things such as determining string length, doing pattern matching, and so on must be aware of the character set -- if only to know which bytes are continuations of a multi-byte character. (That is, a three-byte sequence in UTF-8 might be one, two or three characters, depending.) Now, if the planner assumes ISO-8859-1, but the Drill execution engine assumes UTF-8, then string constants passed from one to the other can become corrupted in the case where a particular byte sequence in ISO-8859-1 represents a different character than that same byte sequence in UTF-8. Where would this occur? Look at the [ISO-8859-1](https://en.wikipedia.org/wiki/ISO/IEC_8859-1) definition. ISO-8859 is a single-byte character set with meanings associated to the bytes in the range 0x40 to 0x7f. But, in UTF-8, the high bit indicates a prefix character. So, 0xF7 is a valid single-byte character in ISO-8859, but is a lead-in character in UTF-8. The point here is that setting the character set would seem to be a global setting. If the Saffron setting is purely for the parser (how to interpret incoming text), and the parser always produces Java strings in UTF-16 (which are then encoded into UTF-8 for execution), then we're fine. But, if the parser encoding is written as bytes sent to the execution engine, we're in trouble. Further, Drill has a web UI. The typical web character set is UTF-8, so queries coming from the web UI are encoded in UTF-8. All this suggests two things: 1. Drill should either always accept UTF-8 (the Saffron property should always be set.) or 2. The property is specified by the client and used to decode the bytes within a Protobuf message to produce a character stream given to the parser. It appears that UTF-8 is the default Protobuf String type encoding; sender and receiver would have to agree on another format. Does Drill have such an RPC property? I've not seen it, but I'm not an expert. In short, if this change ensures that the parser *always* uses UTF-8, then this is good. If the character encoding is an option, then we have to consider all the above issues to have a fully working, end-to-end solution. ---
[GitHub] drill issue #895: DRILL-5704: Improve error message on client side when quer...
Github user sohami commented on the issue: https://github.com/apache/drill/pull/895 This PR was already merged by @arina-ielchiieva on 08/24 ---
[GitHub] drill pull request #895: DRILL-5704: Improve error message on client side wh...
Github user sohami closed the pull request at: https://github.com/apache/drill/pull/895 ---
[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138499857 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java --- @@ -54,11 +56,13 @@ import org.junit.Ignore; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.TemporaryFolder; import org.junit.runner.RunWith; import org.junit.runners.Parameterized; @RunWith(Parameterized.class) +@Category({SecondaryTest.class, ParquetTest.class}) --- End diff -- To me the main distinction between smoke and secondary tests should be runtime. It's hard in my mind to distinguish based on importance, because the relevance of a test depends on the change being done. And you would end up running all the tests before finishing your change no matter what. The workflow I had in mind was the following: 1. Make a change 1. Run Fast tests for my category of interest (e.g. **OperatorTest**). Oops there's a failure 1. Fix the failure 1. Run Fast tests for category **OperatorTest** again. Oops another failure 1. Fix the failure 1. Run Fast tests for category **OperatorTest** again. Yay they pass 1. Run all the tests 1. Done Let me know what you think. ---
[GitHub] drill issue #939: DRILL-5550: Missing CSV column value set to null
Github user prasadns14 commented on the issue: https://github.com/apache/drill/pull/939 @paul-rogers Can you please review the PR? ---
[jira] [Created] (DRILL-5784) SYSTEM ERROR: IndexOutOfBoundsException: index: 512, length: 4 (expected: range(0, 512))
Vlad Rozov created DRILL-5784: - Summary: SYSTEM ERROR: IndexOutOfBoundsException: index: 512, length: 4 (expected: range(0, 512)) Key: DRILL-5784 URL: https://issues.apache.org/jira/browse/DRILL-5784 Project: Apache Drill Issue Type: Bug Environment: planner.slice_target > 10 planner.enable_nljoin_for_scalar_only = false Reporter: Vlad Rozov Assignee: Vlad Rozov The following query causes IndexOutOfBoundsException: {code} SELECT `t1`.`one` `one` FROM ( SELECT 1 `one` FROM dfs.`/drill/exec/java-exec/src/test/resources/join/j1` INNER JOIN ( SELECT 314 `c_integer` FROM dfs.`/drill/exec/java-exec/src/test/resources/join/j1` ) `t0` ON ( `/drill/exec/java-exec/src/test/resources/join/j1`.c_integer IS NOT DISTINCT FROM `t0`.`c_integer` ) GROUP BY `one` ) `t1` INNER JOIN ( SELECT count(1) `measure` FROM dfs.`/drill/exec/java-exec/src/test/resources/join/j1` ) `t5` ON TRUE {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138497914 --- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java --- @@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException { // Connection--and other JDBC objects--on test method failure, but this test // class uses some objects across methods.) Driver.load(); -connection = DriverManager.getConnection( "jdbc:drill:zk=local" ); +Properties properties = new Properties(); +properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3"); --- End diff -- The Properties class only allows strings to be set as values. The conversion to an integer is done somewhere inside the DriverManager. ---
[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138497894 --- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java --- @@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException { // Connection--and other JDBC objects--on test method failure, but this test // class uses some objects across methods.) Driver.load(); -connection = DriverManager.getConnection( "jdbc:drill:zk=local" ); +Properties properties = new Properties(); +properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3"); --- End diff -- I made ExecConstants a file class with a private constructor and added the static method there. ---
[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...
Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/938#discussion_r138495914 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java --- @@ -158,19 +158,17 @@ public BatchHolder(int idx) { } finally { if (!success) { htContainer.clear(); - if (links != null) { -links.clear(); - } + if (links != null) { links.clear();} } } } private void init(IntVector links, IntVector hashValues, int size) { for (int i = 0; i < size; i++) { -links.getMutator().setSafe(i, EMPTY_SLOT); +links.getMutator().set(i, EMPTY_SLOT); --- End diff -- This init() method is not used looks like leftover old code ---
[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...
Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/938#discussion_r138495296 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTable.java --- @@ -58,7 +59,7 @@ public int getHashCode(int incomingRowIdx) throws SchemaChangeException; - public PutStatus put(int incomingRowIdx, IndexPointer htIdxHolder, int hashCode) throws SchemaChangeException; + public PutStatus put(int incomingRowIdx, IndexPointer htIdxHolder, int hashCode) throws SchemaChangeException, RetryAfterSpillException; --- End diff -- Done. ---
[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...
Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/938#discussion_r138495164 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -1178,20 +1273,38 @@ private void checkGroupAndAggrValues(int incomingRowIdx) { hashCode >>>= bitsInMask; HashTable.PutStatus putStatus = null; long allocatedBeforeHTput = allocator.getAllocatedMemory(); - // == // Insert the key columns into the hash table // == +boolean noReserveMem = reserveValueBatchMemory == 0; try { + if ( noReserveMem && canSpill ) { throw new RetryAfterSpillException();} // proactive spill, skip put() + putStatus = htables[currentPartition].put(incomingRowIdx, htIdxHolder, hashCode); + +} catch (RetryAfterSpillException re) { + if ( ! canSpill ) { throw new OutOfMemoryException(getOOMErrorMsg("Can not spill")); } --- End diff -- The method getOOMErrorMsg() does all this explanation ... ---
[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...
Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/938#discussion_r138494777 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -1178,20 +1273,38 @@ private void checkGroupAndAggrValues(int incomingRowIdx) { hashCode >>>= bitsInMask; HashTable.PutStatus putStatus = null; long allocatedBeforeHTput = allocator.getAllocatedMemory(); - // == // Insert the key columns into the hash table // == +boolean noReserveMem = reserveValueBatchMemory == 0; try { + if ( noReserveMem && canSpill ) { throw new RetryAfterSpillException();} // proactive spill, skip put() + putStatus = htables[currentPartition].put(incomingRowIdx, htIdxHolder, hashCode); + +} catch (RetryAfterSpillException re) { --- End diff -- Done. ---
[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...
Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/938#discussion_r138492663 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -646,6 +687,46 @@ public AggOutcome doWork() { } /** + * Use reserved values memory (if available) to try and preemp an OOM + */ + private void useReservedValuesMemory() { +// try to preempt an OOM by using the reserved memory +long reservedMemory = reserveValueBatchMemory; +if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() + reservedMemory); } + +reserveValueBatchMemory = 0; + } + /** + * Use reserved outgoing output memory (if available) to try and preemp an OOM + */ + private void useReservedOutgoingMemory() { +// try to preempt an OOM by using the reserved memory +long reservedMemory = reserveOutgoingMemory; +if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() + reservedMemory); } + +reserveOutgoingMemory = 0; + } + /** + * Restore the reserve memory (both) + * + */ + private void restoreReservedMemory() { +if ( 0 == reserveOutgoingMemory ) { // always restore OutputValues first (needed for spilling) + long memAvail = allocator.getLimit() - allocator.getAllocatedMemory(); + if ( memAvail > estOutgoingAllocSize) { +allocator.setLimit(allocator.getLimit() - estOutgoingAllocSize); --- End diff -- Can never add twice, as the code only adds to an empty ( == 0 ) reserve. And these are not "relative deltas", but the actual expected batch size (so that the following allocation would not OOM). ---
[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138489633 --- Diff: common/src/test/java/org/apache/drill/categories/package-info.java --- @@ -0,0 +1,23 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * This package stores the JUnit test categories. + */ --- End diff -- Currently there are two broad categories: **Fast** and **Secondary**. The **Fast** tests are smoke tests. The **Secondary** tests are for slow tests. I can add a third **BugFix**category to represent tests for specific bugs. The rest of the categories represent broad groups of tests. Some of those categories have tests which span multiple class files and multiple packages. An example of such a category is **OperatorTest**, **PlannerTest**, or **SqlTest**. These categories are helpful when you are making a change in a particular area since they allow you to run all the tests for that area easily, and also effectively serve as a form of documentation. For example it would have been helpful to have an OptionsTest category when I was making the option system changes, since it took me a while to find all the relevant test classes and I had to construct a long maven command to run the tests. Also the structure of the tests as they are now is somewhat counter intuitive because tests can span different modules. For example consider the vector submodule. One would think all the relevant tests for vectors are in that submodule, but currently they are not. In fact there are no unit tests in the vector submodule and all relevant vector tests are in java-exec. Also the memory submodule has tests in it and a relevant test in the java-exec module as well. Some categories I agree are redundant like HiveStorageTest. Those tests have a clear one to one mapping to their submodule, however, I included a category for them as well for the sake of consistency. This way we can have a uniform mechanism for running subsets of tests instead of one mechanism for some groups of tests and another mechanism for other tests. To summarize, I can add a **BugFixTest** category to add another level of distinction between **Secondary** tests. But I am compelled to keep the other test categories intact for the reasons outlined above. Let me know what you think. ---
[GitHub] drill issue #895: DRILL-5704: Improve error message on client side when quer...
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/895 @sohami, please rebase on latest master and resolve conflicts. ---
[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138473108 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/xsort/TestSimpleExternalSort.java --- @@ -148,10 +149,10 @@ public void sortOneKeyDescendingExternalSortLegacy() throws Throwable { private void sortOneKeyDescendingExternalSort(boolean testLegacy) throws Throwable { FixtureBuilder builder = ClusterFixture.builder() -.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4) -.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_GROUP_SIZE, 4) -.configProperty(ExecConstants.EXTERNAL_SORT_BATCH_LIMIT, 4) -.configProperty(ExecConstants.EXTERNAL_SORT_DISABLE_MANAGED, false) +.configProperty(OptionValidator.OPTION_DEFAULTS_ROOT + ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4) --- End diff -- These options are boot options, not runtime options, so no need for the prefix (unless someone changed something.) ---
[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138466695 --- Diff: common/src/test/java/org/apache/drill/categories/package-info.java --- @@ -0,0 +1,23 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * This package stores the JUnit test categories. + */ --- End diff -- Very cool. One thing that is not intuitively obvious here is the purpose and usage of the categories. On the one hand, we already have modules for memory, vectors, Hive, etc. One can easily run just those tests by cd'ing into that module and running Maven from there. So, one would imagine categories to provide another dimension, orthogonal to modules. For example, the classic "fast" vs. "slow." One could also imagine a core set of "smoke" tests that run quickly and do a quick validation. Then a set of standard tests that are more thorough. Then, very in-depth tests that either take a long time, or validate very specific bugs or regressions. Can you add a bit of commentary to advise how you envision the categories to be used? ---
[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138472376 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestCorruptParquetDateCorrection.java --- @@ -59,6 +61,7 @@ * Use of this option is assumed to be extremely unlikely, but it is included * for completeness. */ +@Category(ParquetTest.class) --- End diff -- Secondary? Would not seem that this test need to be run on each build... ---
[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138472834 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java --- @@ -54,11 +56,13 @@ import org.junit.Ignore; import org.junit.Rule; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.junit.rules.TemporaryFolder; import org.junit.runner.RunWith; import org.junit.runners.Parameterized; @RunWith(Parameterized.class) +@Category({SecondaryTest.class, ParquetTest.class}) --- End diff -- On the other hand, the Parquet writer seems pretty important and we'd want to test it more often. This is why, in my earlier comment, I asked about the purpose of categorization: those that are worth running frequently ("smoke tests") vs. those that give a more thorough test, but will typically not find issues. ---
[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138475152 --- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java --- @@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException { // Connection--and other JDBC objects--on test method failure, but this test // class uses some objects across methods.) Driver.load(); -connection = DriverManager.getConnection( "jdbc:drill:zk=local" ); +Properties properties = new Properties(); +properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3"); --- End diff -- Also, since properties don't have to be strings, why is this property stored as a string when the value is an int? ---
[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/940#discussion_r138475004 --- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java --- @@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException { // Connection--and other JDBC objects--on test method failure, but this test // class uses some objects across methods.) Driver.load(); -connection = DriverManager.getConnection( "jdbc:drill:zk=local" ); +Properties properties = new Properties(); +properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3"); --- End diff -- I wonder, should we define a function for this pattern? ``` ExecConstants.bootDefaultFor( ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS) ``` But, we use Java 7, so the static function must reside elsewhere. Actually, the function might exist; I seem to recall making the same comment in the PR that externalized the defaults. ---
[GitHub] drill issue #940: DRILL-5752 Speed Up Unit Tests add Test Categories
Github user ilooner commented on the issue: https://github.com/apache/drill/pull/940 @paul-rogers This is an implementation of the Test Categories suggestion you made on the dev list. It's ready for review. ---
[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories
GitHub user ilooner opened a pull request: https://github.com/apache/drill/pull/940 DRILL-5752 Speed Up Unit Tests add Test Categories This PR does the following. **Increased Concurrency:** Previously tests only executed in 2 simultaneous processes, now one process per core is used to execute tests. This PR also fixed a few issues that popped up when concurrency was increased. **Test Categories:** Tests can now be executed according to their categories. There are two main categories of tests **Fast Tests** and **Secondary Tests**. **Fast Tests** are fast and are executed by default with the following command: ``` mvn -T 1C clean install ``` **Note:** -T 1C increased the number of threads used to compile classes, and allows us to run tests for sub modules in parallel. This command takes about 15 minutes to run on my laptop. **Secondary Tests** are slower. To run both fast tests and **Secondary Tests** you can use the following command. ``` mvn -T 1C clean install -DexcludedGroups="" ``` In addition to **Fast** and **Secondary** tests there are more categories like: - **JdbcTest** - **ParquetTest** - **HiveStorageTest** - And many more All of these categories are in the **drill-common** sub modules in **org.apahce.drill.categories**. All the tests are marked with one or more of these categories, and we can use these categories to select the tests we run. To run all the **Fast** tests that belong to the **JdbcTest** category we can use the following command: ``` mvn -T 1C clean install -Dgroups="org.apache.drill.categories.JdbcTest" ``` In order to run all the **Fast** AND **Secondary** JdbcTests you can use this command ``` mvn -T 1C clean install -Dgroups="org.apache.drill.categories.JdbcTest" -DexcludedGroups="" ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ilooner/drill DRILL-5752 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/940.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #940 commit ba3d1ed53d7cabc5f5ac860531f6e989f567bc65 Author: Timothy Farkas Date: 2017-08-30T19:53:49Z - Removed redundant exclusion from java-exec pom - Marked the slow tests in contrib as secondary tests commit 9a3b7d2e1de85ce4397311e9ff2433432982fb03 Author: Timothy Farkas Date: 2017-08-30T22:02:23Z - Made slow unit tests SecondaryTests - Changed the number of forked test processes from just 2 processes to 1.5 process per core commit 4159bf8f3ad4966751b9176843e240bb4885cec5 Author: Timothy Farkas Date: 2017-08-31T00:10:38Z - Added JdbcTest category commit 0ab8ba8474b68bae8175f4b3c4a9b372f20eef8a Author: Timothy Farkas Date: 2017-08-31T00:24:03Z - Removed unused imports from tests commit bf3748c6ddd96b303762d8fde9046215cb243c02 Author: Timothy Farkas Date: 2017-08-31T00:34:22Z - Added JdbcStorageTest Category commit 749fa9b1c4ae113a010de33bf359a38a1d56833e Author: Timothy Farkas Date: 2017-08-31T18:15:18Z - Created the HiveStorageTest category - Made excludedGroups overrideable commit 50ddbb63f05eb08ac06d46dcf3c078e61dd3d042 Author: Timothy Farkas Date: 2017-08-31T18:39:28Z - Added HbaseStorageTest category commit 10097f61cb2df8898c2e7d8ff20cfb63fddcb6c5 Author: Timothy Farkas Date: 2017-08-31T18:50:33Z - Added the KuduStorageTest category commit 4dbad7edd5f479abc7db698e3ee2a2dad62abf20 Author: Timothy Farkas Date: 2017-08-31T19:06:34Z - Added the MongoStorageTest Category commit 5077b7beeccddcb0c84c70645c879cd3168f1bc1 Author: Timothy Farkas Date: 2017-08-31T19:42:09Z - Added a MemoryTest category commit a0e3832511041aeb18590d65ebb7166e40be045f Author: Timothy Farkas Date: 2017-08-31T19:53:42Z - Add another secondary test. commit efc9b8bb5af3ce891b890217ed1595d0db26cbbb Author: Timothy Farkas Date: 2017-08-31T20:03:08Z - Added a SecurityTest category - Made more of the security tests secondary tests commit 8dfacb0715252e063eca7d7a51895b8da191b273 Author: Timothy Farkas Date: 2017-08-31T20:05:05Z - Removed unused import for security test commit 72c3db7f1774fb0e37dd61a25ef1215f0391da63 Author: Timothy Farkas Date: 2017-08-31T20:05:57Z - Added another memory test commit 3fd24f8062887253a27bbdb8f31f12a06aa97270 Author: Timothy Farkas Date: 2017-08-31T20:22:14Z - Added an OperatorTest category commit 11290f9d48d223f8ca8b9c2df56e458616b51614 Author: Timothy Farkas Date: 2017-08-31T20:29:50Z - Added license headers - Added a ParquetTest category commit 3b03bb385466a2796578dd8ad96f2596a2b0f95e Author: Timothy Farkas Date: 2017-08-31T20:33:3
[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...
Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/938#discussion_r138437442 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -646,6 +687,46 @@ public AggOutcome doWork() { } /** + * Use reserved values memory (if available) to try and preemp an OOM + */ + private void useReservedValuesMemory() { +// try to preempt an OOM by using the reserved memory +long reservedMemory = reserveValueBatchMemory; +if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() + reservedMemory); } + +reserveValueBatchMemory = 0; + } + /** + * Use reserved outgoing output memory (if available) to try and preemp an OOM + */ + private void useReservedOutgoingMemory() { +// try to preempt an OOM by using the reserved memory +long reservedMemory = reserveOutgoingMemory; +if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() + reservedMemory); } --- End diff -- Because the first uncontrolled memory allocation happens when inserting into the hash table (i.e. put()). Given this "uncontrollability", better OOM there (which we can handle, by spilling and retrying). Now if all the memory was "given" in the limit, the put() may not OOM, but leave too little available memory to continue (i.e. to create a values batch, or an outgoing batch) -- these situations we can not handle. By subtracting from the limit a "reserve" for these two batches, we may force a put() OOM early (but that's OK). But we also ensure that the following two batches could be allocated. In some way this is similar to having multiple dedicated allocators, only simpler. Adding or subtracting is just an operation on a local field; no performance effect in any way. Also by using a single allocator we can handle cases like a "bump" in a batch size (which may exceed pre-allocation in a dedicated separate allocator). ---
[GitHub] drill issue #905: DRILL-1162: Fix OOM for hash join operator when the right ...
Github user jinfengni commented on the issue: https://github.com/apache/drill/pull/905 @vvysotskyi , what I'm thinking is not just comparing the ratio of column count vs ratio of row count. Let's take a step back. This SwapHashJoinVisitor is trying to correct the join decision made by either VolcanoPlanner or HepPlanner, in the sense the original join order might need put bigger dataset on the build side. The swap is needed because we want to avoid high memory requirement on hash join operator. The question is how do we define "big dataset". For now, SwapHashJoinVisitor simply uses rowCount, which is not sufficient. In stead, we probably should add a method `getSelfMemCost` to all `Prel` node. For non-blocking operator, it's simply returning either 0 or some constant (to hold one single batch). For non-blocking operator such as HashJoin, it will return a value proportional to rowCount X columnCount (more precisely, total number of bytes per row, considering different column data type). Same as existing method of `computeSelfCost`, we need `getCumulativeMemCost` which will return the cumulative cost for child nodes rooted at one `Prel` node. With this `getSelfMemCost` and `getCumulativeMemCost` defined for HashJoin, and a HashJoin with input1, input2 as inputs, we could estimate cumulative memory cost for HashJoin(input1, input2), and HashJoin(input2, input1), and use that as criteria to decide whether we have to switch them. This idea is not trying to adjust the row count estimation. In stead, it's trying to change the criteria where we may think it's necessary to swap, based on the observation that we want to do swap only when we want to reduce memory requirement for a query. Will the above idea work? If it could not address this issue, it's probably fine to go with what you proposed. Before we go that option, please give some thoughts about the above idea. ---
[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...
Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/938#discussion_r138435187 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -382,19 +390,25 @@ private void delayedSetup() { final boolean fallbackEnabled = context.getOptions().getOption(ExecConstants.HASHAGG_FALLBACK_ENABLED_KEY).bool_val; // Set the number of partitions from the configuration (raise to a power of two, if needed) -numPartitions = context.getConfig().getInt(ExecConstants.HASHAGG_NUM_PARTITIONS); -if ( numPartitions == 1 ) { +numPartitions = (int)context.getOptions().getOption(ExecConstants.HASHAGG_NUM_PARTITIONS_VALIDATOR); +if ( numPartitions == 1 && is2ndPhase ) { // 1st phase can still do early return with 1 partition canSpill = false; logger.warn("Spilling is disabled due to configuration setting of num_partitions to 1"); } numPartitions = BaseAllocator.nextPowerOfTwo(numPartitions); // in case not a power of 2 -if ( schema == null ) { estMaxBatchSize = 0; } // incoming was an empty batch +if ( schema == null ) { estValuesBatchSize = estOutgoingAllocSize = estMaxBatchSize = 0; } // incoming was an empty batch else { // Estimate the max batch size; should use actual data (e.g. lengths of varchars) updateEstMaxBatchSize(incoming); } -long memAvail = memoryLimit - allocator.getAllocatedMemory(); +// create "reserved memory" and adjust the memory limit down +reserveValueBatchMemory = reserveOutgoingMemory = estValuesBatchSize ; +long newMemoryLimit = allocator.getLimit() - reserveValueBatchMemory - reserveOutgoingMemory ; +long memAvail = newMemoryLimit - allocator.getAllocatedMemory(); +if ( memAvail <= 0 ) { throw new OutOfMemoryException("Too little memory available"); } +allocator.setLimit(newMemoryLimit); + --- End diff -- Yes indeed. The only attempt to force a code path is the new testing option *use_memory_prediction* which can disable the estimate based prediction (when to spill), hence forcing the code path that relies on an OOM (for hash table put() ) to take place (one unit test was added for that). Getting a full code coverage would be ideal, but hard. ---
[jira] [Created] (DRILL-5783) Make code generation in the TopN operator more modular and test it
Timothy Farkas created DRILL-5783: - Summary: Make code generation in the TopN operator more modular and test it Key: DRILL-5783 URL: https://issues.apache.org/jira/browse/DRILL-5783 Project: Apache Drill Issue Type: Improvement Reporter: Timothy Farkas Assignee: Timothy Farkas -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...
Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/938#discussion_r138433967 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -500,22 +516,45 @@ private void initializeSetup(RecordBatch newIncoming) throws SchemaChangeExcepti */ private void updateEstMaxBatchSize(RecordBatch incoming) { if ( estMaxBatchSize > 0 ) { return; } // no handling of a schema (or varchar) change +// Use the sizer to get the input row width and the length of the longest varchar column RecordBatchSizer sizer = new RecordBatchSizer(incoming); logger.trace("Incoming sizer: {}",sizer); // An empty batch only has the schema, can not tell actual length of varchars // else use the actual varchars length, each capped at 50 (to match the space allocation) -estRowWidth = sizer.rowCount() == 0 ? sizer.stdRowWidth() : sizer.netRowWidthCap50(); -estMaxBatchSize = estRowWidth * MAX_BATCH_SIZE; +long estInputRowWidth = sizer.rowCount() == 0 ? sizer.stdRowWidth() : sizer.netRowWidthCap50(); // Get approx max (varchar) column width to get better memory allocation -maxColumnWidth = Math.max(sizer.maxSize(), VARIABLE_MIN_WIDTH_VALUE_SIZE); +maxColumnWidth = Math.max(sizer.maxAvgColumnSize(), VARIABLE_MIN_WIDTH_VALUE_SIZE); maxColumnWidth = Math.min(maxColumnWidth, VARIABLE_MAX_WIDTH_VALUE_SIZE); -logger.trace("{} phase. Estimated row width: {} batch size: {} memory limit: {} max column width: {}", - isTwoPhase?(is2ndPhase?"2nd":"1st"):"Single",estRowWidth,estMaxBatchSize,memoryLimit,maxColumnWidth); +// +// Calculate the estimated max (internal) batch (i.e. Keys batch + Values batch) size +// (which is used to decide when to spill) +// Also calculate the values batch size (used as a reserve to overcome an OOM) +// +Iterator> outgoingIter = outContainer.iterator(); +int fieldId = 0; +while (outgoingIter.hasNext()) { + ValueVector vv = outgoingIter.next().getValueVector(); + MaterializedField mr = vv.getField(); + int fieldSize = vv instanceof VariableWidthVector ? maxColumnWidth : + TypeHelper.getSize(mr.getType()); + estRowWidth += fieldSize; + estOutputRowWidth += fieldSize; + if ( fieldId < numGroupByOutFields ) { fieldId++; } + else { estValuesRowWidth += fieldSize; } +} +// multiply by the max number of rows in a batch to get the final estimated max size +estMaxBatchSize = Math.max(estRowWidth, estInputRowWidth) * MAX_BATCH_SIZE; --- End diff -- Most of these estimates are for internal "worst case". Only the "outgoing" one is for the outgoing batch (which is also for spilling - which is internal). Anyway all these estimates have nothing to do with _throttling_ the outgoing batch size; that logic was not changed from the original code (likely up to MAX_BATCH_SIZE). Making such a change should be a separate project. ---
[GitHub] drill issue #905: DRILL-1162: Fix OOM for hash join operator when the right ...
Github user vvysotskyi commented on the issue: https://github.com/apache/drill/pull/905 @jinfengni thanks for looking into this. Completely agree with you that it would be better to consider both row and column count. Unfortunately, it does not help to fix this issue, since the ratio of column count of tables very often much smaller than the ratio of row count (actual row count). (`c1/c2 << r2/r1`) So I guess it makes sense to implement your proposal to consider the column count together with the row count in another pull request. ---
[GitHub] drill pull request #930: DRILL-5761: Disable Lilith ClassicMultiplexSocketAp...
Github user vrozov commented on a diff in the pull request: https://github.com/apache/drill/pull/930#discussion_r138408349 --- Diff: common/src/test/resources/logback-test.xml --- @@ -0,0 +1,92 @@ + + + + + + + +true +1 +true +${LILITH_HOSTNAME:-localhost} + + + + + + + + %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n + + + +
[GitHub] drill pull request #930: DRILL-5761: Disable Lilith ClassicMultiplexSocketAp...
Github user vrozov commented on a diff in the pull request: https://github.com/apache/drill/pull/930#discussion_r138407993 --- Diff: common/src/test/resources/logback-test.xml --- @@ -0,0 +1,111 @@ + + + + + + + +true +1 +true +${LILITH_HOSTNAME:-localhost} + + + + + + + + %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n + + + + + + + --- End diff -- Can this logger be defined between line 21 and 22 in a single if condition? ---
[GitHub] drill issue #936: DRILL-5772: Add unit tests to indicate how utf-8 support c...
Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/936 @paul-rogers, the unit tests just show which influence saffron property has on Drill (since people in the community where asking how to enable support UTF-8 in Drill), a long with this PR we'll also add description to Drill documentation. So far Drill relies on Calcite saffron property to determine which charset it supports. By default, it's ISO-8859-1. So currently if customer wants to query UTF-8 data in Drill, he will get an error (one of the unit tests shows it) and needs to override saffron property to proceed. I guess, we don't support UTF-8 by default since there are some issues and Drill did not fully tested UTF-8 support. ---
[jira] [Created] (DRILL-5782) Web UI: do not attempt build visualized plan when plan is absent
Arina Ielchiieva created DRILL-5782: --- Summary: Web UI: do not attempt build visualized plan when plan is absent Key: DRILL-5782 URL: https://issues.apache.org/jira/browse/DRILL-5782 Project: Apache Drill Issue Type: Bug Affects Versions: 1.11.0 Reporter: Arina Ielchiieva Attachments: error_when_creating_visualized_plan.JPG When trying to execute query with incorrect syntax, there is an error on query profile page (screenshot attached). Error occurs when we try to build visualized plan when plan is absent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (DRILL-5339) Web UI: flaws in query profile display on error
[ https://issues.apache.org/jira/browse/DRILL-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-5339. - Resolution: Fixed Points 1-4 fixed in the scope of DRILL-5766. > Web UI: flaws in query profile display on error > --- > > Key: DRILL-5339 > URL: https://issues.apache.org/jira/browse/DRILL-5339 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.12.0 > > Attachments: error_shift.JPG, fragment_profiles.JPG, > verbose_error.JPG, visualized_plan.JPG > > > 1. Error is shifted to the right (screenshot - error_shift.jpg) > 2. When Fragment Profiles doesn't have any data table, there is offset in > overview tab (screenshot - fragment_profiles.jpg) > 3. Verbose error message tab can be designed like overview tab to be more > concise (screenshot - verbose_error.jpg) > 4. Remove three dots in the end of "Verbose error message..." . > 5. When visualized plan is empty, there is big offset (screenshot - > visualized_plan.jpg) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (DRILL-5338) Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile that they are part of total time duration
[ https://issues.apache.org/jira/browse/DRILL-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-5338. - Resolution: Fixed Fixed in the scope of DRILL-5766 > Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query > Profile that they are part of total time duration > --- > > Key: DRILL-5338 > URL: https://issues.apache.org/jira/browse/DRILL-5338 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: 1.12.0 > > Attachments: QueryProfile.JPG > > > Add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile > that they are part of total time duration > Screenshot is attached. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (DRILL-5341) Web UI: remove duplicating link to documentation in Options page
[ https://issues.apache.org/jira/browse/DRILL-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-5341. - Resolution: Fixed Fixed in the scope of DRILL-5766 > Web UI: remove duplicating link to documentation in Options page > > > Key: DRILL-5341 > URL: https://issues.apache.org/jira/browse/DRILL-5341 > Project: Apache Drill > Issue Type: Bug >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.12.0 > > Attachments: doc_link.JPG > > > Remove link to Documentation on Options page (http://localhost:8047/options) > as we have the same link on page header. Screenshot - doc_link.JPG -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (DRILL-5346) Web UI: remove link on user in query profile list
[ https://issues.apache.org/jira/browse/DRILL-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-5346. - Resolution: Fixed Fixed in the scope of DRILL-5766 > Web UI: remove link on user in query profile list > - > > Key: DRILL-5346 > URL: https://issues.apache.org/jira/browse/DRILL-5346 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.12.0 > > Attachments: query_profile_link.JPG > > > Remove link on user column that leads to query profile. Since we don't have > user page information, it's better to leave user name without link to avoid > confusion. Plus we already have link that leads to query profile on query > itself. > Page - http://localhost:8047/profiles > Screenshot - query_profile_link.JPG -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] drill pull request #935: DRILL-5766: Fix XSS vulnerabilities in Drill
Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/935 ---
[jira] [Created] (DRILL-5781) Fix unit test failures to use tests config even if default config is available
Volodymyr Vysotskyi created DRILL-5781: -- Summary: Fix unit test failures to use tests config even if default config is available Key: DRILL-5781 URL: https://issues.apache.org/jira/browse/DRILL-5781 Project: Apache Drill Issue Type: Bug Affects Versions: 1.11.0 Reporter: Volodymyr Vysotskyi Assignee: Volodymyr Vysotskyi Unit tests fail when they are run with the mapr profile. Tests failures, connected with the Zookeeper configuration that differs from expected: {noformat} DrillClientTest>TestWithZookeeper.setUp:32 » Runtime java.io.IOException: Coul... TestZookeeperClient.testPutWithMatchingVersion » IO Could not configure server... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testStartingClientEnablesCacheAndEnsuresRootNodeExists » IO TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testHasPathThrowsDrillRuntimeException » IO Could not conf... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testHasPathFalseWithVersion » IO Could not configure serve... TestZookeeperClient.tearDown:86 NullPointer TestEphemeralStore.testPutAndGetWorksAntagonistacally » IO Could not configure... TestEphemeralStore.tearDown:132 NullPointer TestZookeeperClient.testGetWithVersion » IO Could not configure server because... TestZookeeperClient.tearDown:86 NullPointer TestEphemeralStore.testStoreRegistersDispatcherAndStartsItsClient » IO Could n... TestEphemeralStore.tearDown:132 NullPointer TestZookeeperClient.testPutWithNonMatchingVersion » IO Could not configure ser... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testGetWithEventualConsistencyHitsCache » IO Could not con... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testPutIfAbsentWhenPresent » IO Could not configure server... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testHasPathTrueWithVersion » IO Could not configure server... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testPutAndGetWorks » IO Could not configure server because... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testPutIfAbsentWhenAbsent » IO Could not configure server ... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testHasPathWithEventualConsistencyHitsCache » IO Could not... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testCreate » IO Could not configure server because SASL co... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testDelete » IO Could not configure server because SASL co... TestZookeeperClient.tearDown:86 NullPointer TestZookeeperClient.testEntriesReturnsRelativePaths » IO Could not configure s... TestZookeeperClient.tearDown:86 NullPointer TestPStoreProviders>TestWithZookeeper.setUp:32 » Runtime java.io.IOException: ... TestPauseInjection.pauseOnSpecificBit:151 » Runtime java.io.IOException: Could... TestExceptionInjection.injectionOnSpecificBit:217 » Runtime java.io.IOExceptio... HBaseTestsSuite.initCluster:110 » IO No JAAS configuration section named 'Serv... {noformat} Test failures, connected with Hadoop configuration that differs from expected: {noformat} TestInboundImpersonation.setup:58->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 » ClassCast TestImpersonationMetadata.setup:58->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 » ClassCast TestImpersonationDisabledWithMiniDFS.setup:37->BaseTestImpersonation.startMiniDfsCluster:106 » Runtime TestImpersonationQueries.setup:46->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 » ClassCast TestHiveStorage>HiveTestBase.generateHive:34 » Runtime java.lang.RuntimeExcept... TestInfoSchemaOnHiveStorage>HiveTestBase.generateHive:34 » Runtime java.lang.R... TestInbuiltHiveUDFs>HiveTestBase.generateHive:35 » ExecutionSetup Failure sett... TestSampleHiveUDFs>HiveTestBase.generateHive:35 » ExecutionSetup Failure setti... TestStorageBasedHiveAuthorization.setup:109->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 » ClassCast TestSqlStdBasedAuthorization.setup:72->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111 » ClassCast TestHivePartitionPruning>HiveTestBase.generateHive:35 » ExecutionSetup Failure... TestViewSupportOnHiveTables.generateHive:35 » ExecutionSetup Failure setting u... TestHiveProjectPushDown>HiveTestBase.generateHive:35 » ExecutionSetup Failure ... {noformat} Test specific configuration should be added to fix these failures. -- This message was sent by Atlassian JIRA (v6.4.14#64029)