[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables
[
https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268363#comment-16268363
]
Hive QA commented on HIVE-17361:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899527/HIVE-17361.24.patch
{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 11409 tests
executed
*Failed tests:*
{noformat}
TestTriggersWorkloadManager - did not produce a TEST-*.xml file (likely timed
out) (batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=159)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerializeWithDeltas
(batchId=268)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerializeWithDeltas
(batchId=268)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8015/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8015/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8015/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899527 - PreCommit-HIVE-Build
> Support LOAD DATA for transactional tables
> --
>
> Key: HIVE-17361
> URL: https://issues.apache.org/jira/browse/HIVE-17361
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Wei Zheng
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch,
> HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch,
> HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch,
> HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch,
> HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.21.patch,
> HIVE-17361.23.patch, HIVE-17361.24.patch, HIVE-17361.3.patch,
> HIVE-17361.4.patch
>
>
> LOAD DATA was not supported since ACID was introduced. Need to fill this gap
> between ACID table and regular hive table.
> Current Documentation is under [DML
> Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations]
> and [Loading files into
> tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]:
> \\
> * Load Data performs very limited validations of the data, in particular it
> uses the input file name which may not be in 0_0 which can break some
> read logic. (Certainly will for Acid).
> * It does not check the schema of the file. This may be a non issue for Acid
> which requires ORC which is self describing so Schema Evolution may handle
> this seamlessly. (Assuming Schema is not too different).
> * It does check that _InputFormat_S are compatible.
> * Bucketed (and thus sorted) tables don't support Load Data (but only if
> hive.strict.checks.bucketing=true (default)). Will keep this restriction for
> Acid.
> * Load Data supports OVERWRITE clause
> * What happens to file permissions/ownership: rename vs copy differences
> \\
> The implementation will follow the same idea as in HIVE-14988 and use a
> base_N/ dir for OVERWRITE clause.
> \\
> How is minor compaction going to handle delta/base with original files?
> Since delta_8_8/_meta_data is created before files are moved, delta_8_8
> becomes visible before it's populated. Is that an issue?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18160) Jar localization during session initialization is slow
[
https://issues.apache.org/jira/browse/HIVE-18160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-18160:
> Jar localization during session initialization is slow
> --
>
> Key: HIVE-18160
> URL: https://issues.apache.org/jira/browse/HIVE-18160
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> Same Jar getting localized multiple times resulting in SHA256 computation
> several times causes slow session initialization time. Also, the default
> sha256 implementation from commons-codec uses 1KB buffer to read jar file
> which is slow (buffer size not configurable).
> {code}
> 2017-11-28T00:40:55,795 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,105 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> e20986f3a422f8fa5eb61c5a2756cd6f7d2b779dbcab49eae6f2c8dfff7ad2a2 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-llap-tez-3.0.0-SNAPSHOT.jar
> of length: 109.53KB in 1 ms
> 2017-11-28T00:40:56,353 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 231 ms
> 2017-11-28T00:40:56,602 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,612 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 686d66b825fdc4fc241e0591e7646a1bbca1c7114a7224c41da7f4795cf9477a for file:
> file:/work/hadoop/hadoop/hadoop-dist/target/hadoop-2.9.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-registry-2.9.0-SNAPSHOT.jar
> of length: 122.72KB in 2 ms
> {code}
> From above logs, sha256 is computed 3 times for hive-exec jar and each
> invocation takes around 240ms.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17528) Add more q-tests for Hive-on-Spark with Parquet vectorized reader
[
https://issues.apache.org/jira/browse/HIVE-17528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268412#comment-16268412
]
Hive QA commented on HIVE-17528:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
32s{color} | {color:red} root: The patch generated 2 new + 191 unchanged - 1
fixed = 193 total (was 192) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/util: The patch generated 2 new + 191 unchanged
- 1 fixed = 193 total (was 192) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8016/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8016/yetus/diff-checkstyle-itests_util.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8016/yetus/patch-asflicense-problems.txt
|
| modules | C: ql . itests itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8016/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add more q-tests for Hive-on-Spark with Parquet vectorized reader
> -
>
> Key: HIVE-17528
> URL: https://issues.apache.org/jira/browse/HIVE-17528
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
> Attachments: HIVE-17528.1.patch, HIVE-17528.2.patch,
> HIVE-17528.3.patch, HIVE-17528.4.patch, HIVE-17528.5.patch,
> HIVE-17528.7.patch, HIVE-17528.8.patch, HIVE-17528.patch
>
>
> Most of the vectorization related q-tests operate on ORC tables using Tez. It
> would be good to add more coverage on a different combination of engine and
> file-format. We can model existing q-tests using parquet tables and run it
> using TestSparkCliDriver
--
This message was sent by Atla
[jira] [Updated] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18157: -- Attachment: HIVE-18157.2.patch Fixed the testconfiguration file. The other failing tests passed locally. > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18157.1.patch, HIVE-18157.2.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17528) Add more q-tests for Hive-on-Spark with Parquet vectorized reader
[
https://issues.apache.org/jira/browse/HIVE-17528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268437#comment-16268437
]
Hive QA commented on HIVE-17528:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899531/HIVE-17528.8.patch
{color:green}SUCCESS:{color} +1 due to 30 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 11468 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_div0]
(batchId=140)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=227)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=233)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=233)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=233)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8016/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8016/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8016/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899531 - PreCommit-HIVE-Build
> Add more q-tests for Hive-on-Spark with Parquet vectorized reader
> -
>
> Key: HIVE-17528
> URL: https://issues.apache.org/jira/browse/HIVE-17528
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
> Attachments: HIVE-17528.1.patch, HIVE-17528.2.patch,
> HIVE-17528.3.patch, HIVE-17528.4.patch, HIVE-17528.5.patch,
> HIVE-17528.7.patch, HIVE-17528.8.patch, HIVE-17528.patch
>
>
> Most of the vectorization related q-tests operate on ORC tables using Tez. It
> would be good to add more coverage on a different combination of engine and
> file-format. We can model existing q-tests using parquet tables and run it
> using TestSparkCliDriver
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18157: -- Attachment: (was: HIVE-18157.2.patch) > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18157.1.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18157: -- Attachment: HIVE-18157.3.patch Uploaded wrong file for v2 > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18157.1.patch, HIVE-18157.3.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18160) Jar localization during session initialization is slow
[
https://issues.apache.org/jira/browse/HIVE-18160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18160:
-
Description:
Same Jar getting localized multiple times resulting in SHA256 computation
several times causes slow session initialization time.
{code}
2017-11-28T00:40:55,795 INFO [main]: tez.TezSessionState
(TezSessionState.java:createJarLocalResource(716)) - Computed sha:
55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
of length: 35.68MB in 241 ms
2017-11-28T00:40:56,105 INFO [main]: tez.TezSessionState
(TezSessionState.java:createJarLocalResource(716)) - Computed sha:
e20986f3a422f8fa5eb61c5a2756cd6f7d2b779dbcab49eae6f2c8dfff7ad2a2 for file:
file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-llap-tez-3.0.0-SNAPSHOT.jar
of length: 109.53KB in 1 ms
2017-11-28T00:40:56,353 INFO [main]: tez.TezSessionState
(TezSessionState.java:createJarLocalResource(716)) - Computed sha:
55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
of length: 35.68MB in 231 ms
2017-11-28T00:40:56,602 INFO [main]: tez.TezSessionState
(TezSessionState.java:createJarLocalResource(716)) - Computed sha:
55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
of length: 35.68MB in 241 ms
2017-11-28T00:40:56,612 INFO [main]: tez.TezSessionState
(TezSessionState.java:createJarLocalResource(716)) - Computed sha:
686d66b825fdc4fc241e0591e7646a1bbca1c7114a7224c41da7f4795cf9477a for file:
file:/work/hadoop/hadoop/hadoop-dist/target/hadoop-2.9.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-registry-2.9.0-SNAPSHOT.jar
of length: 122.72KB in 2 ms
{code}
>From above logs, sha256 is computed 3 times for hive-exec jar and each
>invocation takes around 240ms.
was:
Same Jar getting localized multiple times resulting in SHA256 computation
several times causes slow session initialization time. Also, the default sha256
implementation from commons-codec uses 1KB buffer to read jar file which is
slow (buffer size not configurable).
{code}
2017-11-28T00:40:55,795 INFO [main]: tez.TezSessionState
(TezSessionState.java:createJarLocalResource(716)) - Computed sha:
55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
of length: 35.68MB in 241 ms
2017-11-28T00:40:56,105 INFO [main]: tez.TezSessionState
(TezSessionState.java:createJarLocalResource(716)) - Computed sha:
e20986f3a422f8fa5eb61c5a2756cd6f7d2b779dbcab49eae6f2c8dfff7ad2a2 for file:
file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-llap-tez-3.0.0-SNAPSHOT.jar
of length: 109.53KB in 1 ms
2017-11-28T00:40:56,353 INFO [main]: tez.TezSessionState
(TezSessionState.java:createJarLocalResource(716)) - Computed sha:
55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
of length: 35.68MB in 231 ms
2017-11-28T00:40:56,602 INFO [main]: tez.TezSessionState
(TezSessionState.java:createJarLocalResource(716)) - Computed sha:
55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
of length: 35.68MB in 241 ms
2017-11-28T00:40:56,612 INFO [main]: tez.TezSessionState
(TezSessionState.java:createJarLocalResource(716)) - Computed sha:
686d66b825fdc4fc241e0591e7646a1bbca1c7114a7224c41da7f4795cf9477a for file:
file:/work/hadoop/hadoop/hadoop-dist/target/hadoop-2.9.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-registry-2.9.0-SNAPSHOT.jar
of length: 122.72KB in 2 ms
{code}
>From above logs, sha256 is computed 3 times for hive-exec jar and each
>invocation takes around 240ms.
> Jar localization during session initialization is slow
> --
>
> Key: HIVE-18160
> URL: https://issues.apache.org/jira/browse/HIVE-18160
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> Same Jar getting localized multiple t
[jira] [Commented] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills
[
https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268466#comment-16268466
]
Hive QA commented on HIVE-18076:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
35s{color} | {color:red} ql: The patch generated 28 new + 192 unchanged - 24
fixed = 220 total (was 216) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8017/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8017/yetus/patch-asflicense-problems.txt
|
| modules | C: llap-tez ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8017/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> killquery doesn't actually work for non-trigger WM kills
>
>
> Key: HIVE-18076
> URL: https://issues.apache.org/jira/browse/HIVE-18076
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18076.01.patch, HIVE-18076.02.patch,
> HIVE-18076.03.patch, HIVE-18076.patch
>
>
> Not sure what's wrong with it, need to take a look.
> It dumps a lot of info about everything being cancelled, instead of a nice
> message like triggers do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18160) Jar localization during session initialization is slow
[
https://issues.apache.org/jira/browse/HIVE-18160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18160:
-
Attachment: HIVE-18160.1.patch
> Jar localization during session initialization is slow
> --
>
> Key: HIVE-18160
> URL: https://issues.apache.org/jira/browse/HIVE-18160
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18160.1.patch
>
>
> Same Jar getting localized multiple times resulting in SHA256 computation
> several times causes slow session initialization time.
> {code}
> 2017-11-28T00:40:55,795 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,105 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> e20986f3a422f8fa5eb61c5a2756cd6f7d2b779dbcab49eae6f2c8dfff7ad2a2 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-llap-tez-3.0.0-SNAPSHOT.jar
> of length: 109.53KB in 1 ms
> 2017-11-28T00:40:56,353 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 231 ms
> 2017-11-28T00:40:56,602 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,612 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 686d66b825fdc4fc241e0591e7646a1bbca1c7114a7224c41da7f4795cf9477a for file:
> file:/work/hadoop/hadoop/hadoop-dist/target/hadoop-2.9.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-registry-2.9.0-SNAPSHOT.jar
> of length: 122.72KB in 2 ms
> {code}
> From above logs, sha256 is computed 3 times for hive-exec jar and each
> invocation takes around 240ms.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18160) Jar localization during session initialization is slow
[
https://issues.apache.org/jira/browse/HIVE-18160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18160:
-
Status: Patch Available (was: Open)
> Jar localization during session initialization is slow
> --
>
> Key: HIVE-18160
> URL: https://issues.apache.org/jira/browse/HIVE-18160
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18160.1.patch
>
>
> Same Jar getting localized multiple times resulting in SHA256 computation
> several times causes slow session initialization time.
> {code}
> 2017-11-28T00:40:55,795 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,105 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> e20986f3a422f8fa5eb61c5a2756cd6f7d2b779dbcab49eae6f2c8dfff7ad2a2 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-llap-tez-3.0.0-SNAPSHOT.jar
> of length: 109.53KB in 1 ms
> 2017-11-28T00:40:56,353 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 231 ms
> 2017-11-28T00:40:56,602 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,612 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 686d66b825fdc4fc241e0591e7646a1bbca1c7114a7224c41da7f4795cf9477a for file:
> file:/work/hadoop/hadoop/hadoop-dist/target/hadoop-2.9.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-registry-2.9.0-SNAPSHOT.jar
> of length: 122.72KB in 2 ms
> {code}
> From above logs, sha256 is computed 3 times for hive-exec jar and each
> invocation takes around 240ms.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18149) Stats: rownum estimation from datasize underestimates in most cases
[ https://issues.apache.org/jira/browse/HIVE-18149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268492#comment-16268492 ] Zoltan Haindrich commented on HIVE-18149: - unfortunately these changes are starting to stick togetherbecause of this problem; some table stats are demoted to PARTIAL in HIVE-18108 because the estimated rowsize is greater than the whole dataset size... > Stats: rownum estimation from datasize underestimates in most cases > --- > > Key: HIVE-18149 > URL: https://issues.apache.org/jira/browse/HIVE-18149 > Project: Hive > Issue Type: Sub-task > Components: Statistics >Reporter: Zoltan Haindrich > > rownum estimation is based on the following fact as of now: > * datasize being used from the following sources: > ** basicstats aggregates the loaded "on-heap" row sizes ; other readers are > able to give "raw size" estimation - I've checked orc; but I'm sure others > will do the sameapi docs are a bit vague about the methods purpose... > ** if the basicstats level info is not available; the filesystem level > "file-size-sums" are used as the "raw data size" ; which is multiplied by the > [deserialization > ratio|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L261] > ; which is currently 1. > the problem with all of this is that deser factor is 1; and that rowsize > counts in the online object headers.. > example; 20 rows are loaded into a partition > [columnstats_partlvl_dp.q|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L7] > after HIVE-18108 [this > explain|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L25] > will estimate the rowsize of the table to be 404 bytes; however the 20 rows > of text is only 169 bytes...so it ends up with 0 rows... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268524#comment-16268524 ] Matt McCline commented on HIVE-18157: - [~djaiswal] I looked at the code changes on the RB -- thanks for analyzing this issue and fixing it. +1 LGTM tests pending. > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18157.1.patch, HIVE-18157.3.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills
[
https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268526#comment-16268526
]
Hive QA commented on HIVE-18076:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899536/HIVE-18076.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11412 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=159)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8017/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8017/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8017/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899536 - PreCommit-HIVE-Build
> killquery doesn't actually work for non-trigger WM kills
>
>
> Key: HIVE-18076
> URL: https://issues.apache.org/jira/browse/HIVE-18076
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18076.01.patch, HIVE-18076.02.patch,
> HIVE-18076.03.patch, HIVE-18076.patch
>
>
> Not sure what's wrong with it, need to take a look.
> It dumps a lot of info about everything being cancelled, instead of a nice
> message like triggers do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (HIVE-14572) Investigate jenkins test report timings
[ https://issues.apache.org/jira/browse/HIVE-14572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich resolved HIVE-14572. - Resolution: Fixed this is now fixed by upgrading the plugin to the latest version; which contains the fix. https://builds.apache.org/job/PreCommit-HIVE-Build/7988/testReport/org.apache.hadoop.hive.cli/TestCliDriver/ reports 4min but https://builds.apache.org/job/PreCommit-HIVE-Build/8017/testReport/junit/org.apache.hadoop.hive.cli/TestCliDriver/ reports 5hours which is correct > Investigate jenkins test report timings > --- > > Key: HIVE-14572 > URL: https://issues.apache.org/jira/browse/HIVE-14572 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > > [~sseth] have noticed some odd timings in the jenkins reports > I've created a sample project, to emulate a clidriver run during qtest: > the testclass: > * 1 sec beforeclass > * 3x 0.2s test > created using junit4 parameterized. > Double checkout; second project runs different tests...or at least they have > different name. > here are my preliminary findings: > || thing || expected || 2.16 || 2.19.1 > | total time | ~3.4s | 1.2s | 3.4s > | package time | ~3.4s | 0.61s | 1.7s > | class time | ~3.4s | 0.61s | 1.7s > | testcase times | ~.2s | ~.2s | ~.2s > notes: > * using 2.16 beforeclass timngs are totally hidden or lost > * 2.19.1 does account for beforeclass but still fails to correctly aggregate > the two runs of the similary named testclasses > it might worth a try to look at the bleeding edge of this jenkins plugin... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18134) some alter resource plan fixes
[
https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268564#comment-16268564
]
Hive QA commented on HIVE-18134:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} standalone-metastore: The patch generated 1 new + 477
unchanged - 0 fixed = 478 total (was 477) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8018/yetus/diff-checkstyle-standalone-metastore.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8018/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8018/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> some alter resource plan fixes
> --
>
> Key: HIVE-18134
> URL: https://issues.apache.org/jira/browse/HIVE-18134
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18134.01.patch, HIVE-18134.02.patch,
> HIVE-18134.patch
>
>
> Part of HIVE-18075
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18134) some alter resource plan fixes
[
https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268610#comment-16268610
]
Hive QA commented on HIVE-18134:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899537/HIVE-18134.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11412 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_isops] (batchId=76)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8018/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8018/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8018/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899537 - PreCommit-HIVE-Build
> some alter resource plan fixes
> --
>
> Key: HIVE-18134
> URL: https://issues.apache.org/jira/browse/HIVE-18134
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18134.01.patch, HIVE-18134.02.patch,
> HIVE-18134.patch
>
>
> Part of HIVE-18075
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268635#comment-16268635
]
Hive QA commented on HIVE-17856:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} common: The patch generated 2 new + 8 unchanged - 1
fixed = 10 total (was 9) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 41 new + 1756 unchanged - 7
fixed = 1797 total (was 1763) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 6 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8019/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8019/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8019/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8019/yetus/whitespace-tabs.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8019/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8019/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch,
> HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch,
> HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.2.patch,
> HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17
[jira] [Commented] (HIVE-18146) Vectorization: VectorMapJoinOperator Decimal64ColumnVector key/value cast bug
[ https://issues.apache.org/jira/browse/HIVE-18146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268641#comment-16268641 ] Teddy Choi commented on HIVE-18146: --- +1 LTGM. :) > Vectorization: VectorMapJoinOperator Decimal64ColumnVector key/value cast bug > - > > Key: HIVE-18146 > URL: https://issues.apache.org/jira/browse/HIVE-18146 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-18146.01.patch, HIVE-18146.02.patch > > > Need to convert automatically convert Decimal64ColumnVector key/value > expressions to DecimalColumnVector. > vector_decimal_mapjoin.q was changed to use text format and have join values > for the small decimal queries to trigger the issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268672#comment-16268672
]
Hive QA commented on HIVE-17856:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899538/HIVE-17856.15.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 11422 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=47)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=159)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueing
(batchId=282)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8019/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8019/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8019/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899538 - PreCommit-HIVE-Build
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch,
> HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch,
> HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.2.patch,
> HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch,
> HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18149) Stats: rownum estimation from datasize underestimates in most cases
[ https://issues.apache.org/jira/browse/HIVE-18149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-18149: --- Assignee: Zoltan Haindrich > Stats: rownum estimation from datasize underestimates in most cases > --- > > Key: HIVE-18149 > URL: https://issues.apache.org/jira/browse/HIVE-18149 > Project: Hive > Issue Type: Sub-task > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > > rownum estimation is based on the following fact as of now: > * datasize being used from the following sources: > ** basicstats aggregates the loaded "on-heap" row sizes ; other readers are > able to give "raw size" estimation - I've checked orc; but I'm sure others > will do the sameapi docs are a bit vague about the methods purpose... > ** if the basicstats level info is not available; the filesystem level > "file-size-sums" are used as the "raw data size" ; which is multiplied by the > [deserialization > ratio|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L261] > ; which is currently 1. > the problem with all of this is that deser factor is 1; and that rowsize > counts in the online object headers.. > example; 20 rows are loaded into a partition > [columnstats_partlvl_dp.q|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L7] > after HIVE-18108 [this > explain|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L25] > will estimate the rowsize of the table to be 404 bytes; however the 20 rows > of text is only 169 bytes...so it ends up with 0 rows... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17988) Replace patch utility usage with git apply in ptest
[
https://issues.apache.org/jira/browse/HIVE-17988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268682#comment-16268682
]
Zoltan Haindrich commented on HIVE-17988:
-
+1 looks good to me
[~spena], [~pvary] could you please also take a look? I think you are more
familiar with the inner workings of these things :D
> Replace patch utility usage with git apply in ptest
> ---
>
> Key: HIVE-17988
> URL: https://issues.apache.org/jira/browse/HIVE-17988
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Daniel Voros
> Attachments: HIVE-17988.1.patch, HIVE-17988.2.patch
>
>
> It would be great to replace the standard diff util because {{git}} can do a
> 3-way merge - which in most cases successfull.
> This could reduce the ptest results which are erroring out because of build
> failure.
> {code}
> error: patch failed:
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7003
> Falling back to three-way merge...
> Applied patch to
> 'ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java' cleanly.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18138) Fix columnstats problem in case schema evolution
[
https://issues.apache.org/jira/browse/HIVE-18138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268688#comment-16268688
]
Hive QA commented on HIVE-18138:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} standalone-metastore: The patch generated 3 new + 577
unchanged - 2 fixed = 580 total (was 579) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8020/yetus/diff-checkstyle-standalone-metastore.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8020/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8020/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix columnstats problem in case schema evolution
>
>
> Key: HIVE-18138
> URL: https://issues.apache.org/jira/browse/HIVE-18138
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18138.01.patch, HIVE-18138.01.patch
>
>
> column stats are kept in case the main table schema is altered; and this
> causes all kind of problems.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18163) stats: create materialized view should also collect stats
[
https://issues.apache.org/jira/browse/HIVE-18163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-18163:
---
Assignee: Zoltan Haindrich
> stats: create materialized view should also collect stats
> -
>
> Key: HIVE-18163
> URL: https://issues.apache.org/jira/browse/HIVE-18163
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>
> not having basic stats on the materialized view may cause it to be "ruled
> out" as a viable alternative by the cbo.
> repro set {{set hive.stats.deserialization.factor=10.0}} in
> {{ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q}}
> blocks: HIVE-18149
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18138) Fix columnstats problem in case schema evolution
[
https://issues.apache.org/jira/browse/HIVE-18138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268747#comment-16268747
]
Hive QA commented on HIVE-18138:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899069/HIVE-18138.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 135 failed/errored test(s), 11413 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_stats]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_numbuckets_partitioned_table2_h23]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_numbuckets_partitioned_table_h23]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_change_col]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_rename_partition_authorization]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_serde2]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_6]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_explain]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_2]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_disable_cbo_3]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join11] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_comments]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_date] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_partitioned]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ba_table3] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin10]
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin11]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin12]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin13]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin8]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin9]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_1]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[compute_stats_decimal]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constGby] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_like_view]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_or_replace_view]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_with_constraints]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_5] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[database_drop]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_whole_partition]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[describe_table]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[display_colstats_tbllvl]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic1] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic3]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_topn] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[equal_ns] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_04_evolved_parts]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[extract] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_cond_pushdown]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_limit]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby4_noskew]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby7_noskew_multi_single_reducer]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets_grouping]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[hook_order] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[implicit_cast1
[jira] [Commented] (HIVE-18140) Partitioned tables statistics can go wrong in basic stats mixed case
[
https://issues.apache.org/jira/browse/HIVE-18140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268782#comment-16268782
]
Hive QA commented on HIVE-18140:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} ql: The patch generated 15 new + 125 unchanged - 5
fixed = 140 total (was 130) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
57s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8021/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8021/yetus/whitespace-eol.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8021/yetus/diff-javadoc-javadoc-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8021/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8021/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Partitioned tables statistics can go wrong in basic stats mixed case
>
>
> Key: HIVE-18140
> URL: https://issues.apache.org/jira/browse/HIVE-18140
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18140.01wip01.patch
>
>
> suppose the following scenario:
> * part1 has basic stats {{RC=10,DS=1K}}
> * all other partition has no basic stats (and a bunch of rows)
> then
> [this|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L378]
> condition would be false; which in turn produces estimations for the whole
> partitioned table: {{RC=10,DS=1K}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18164) Hive2 select with group by error if transactional = true
[ https://issues.apache.org/jira/browse/HIVE-18164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitro-Vasilenko updated HIVE-18164: Description: Connected to: Apache Hive (version 1.2.1000.2.6.3.0-235) Driver: Hive JDBC (version 1.2.1000.2.6.3.0-235) 0: jdbc:hive2://serv01:2181,ks-> select sum(destination),messagetype from t1.cdr where hday='2017-09-14' group by messagetype; INFO : Session is already open INFO : Dag name: select sum(destination),messag...messagetype(Stage-1) ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1511771679762_0301_2_00, diagnostics=[Vertex vertex_1511771679762_0301_2_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: cdr initializer failed, vertex=vertex_1511771679762_0301_2_00 [Map 1], java.lang.RuntimeException: serious problem at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1277) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1304) at org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat.getSplits(BucketizedHiveInputFormat.java:141) at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:448) at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:300) at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:266) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: delta_16881612_29766798 does not start with base_ at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1254) ... 16 more Caused by: java.lang.IllegalArgumentException: delta_16881612_29766798 does not start with base_ at org.apache.hadoop.hive.ql.io.AcidUtils.parseBase(AcidUtils.java:190) at org.apache.hadoop.hive.ql.io.AcidUtils.parseBaseBucketFilename(AcidUtils.java:221) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.callInternal(OrcInputFormat.java:804) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.access$600(OrcInputFormat.java:747) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:772) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:769) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:769) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:747) ... 4 more ] Error occur if delta_* present : << [serv01]$ hdfs dfs -ls /warehouse/t1/cdr/hday=2017-09-14 Found 15 items drwxrwxrwx - hive hdfs 0 2017-09-16 11:29 /warehouse/t1/cdr/hday=2017-09-14/base_16881497 drwxrwxrwx - hive hdfs 0 2017-10-21 18:42 /warehouse/t1/cdr/hday=2017-09-14/delta_16881612_29766798 drwxr-xr-x - hive hdfs 0 2017-10-22 17:48 /warehouse/t1/cdr/hday=2017-09-14/delta_30628231_30628231_ drwxr-xr-x - hive hdfs 0 2017-10-26 18:06 /warehouse/t1/cdr/hday=2017-09-14/delta_33418590_33418590_ drwxr-xr-x - hive hdfs 0 2017-10-27 16:23 /warehouse/t1/cdr/hday=2017-09-14/delta_33540229_33540229_ drwxr-xr-x - hive hdfs 0 2017-10-27 16:33 /warehouse/t1/cdr/hday=2017-09-14/delta_33541
[jira] [Updated] (HIVE-18163) Stats: create materialized view should also collect stats
[
https://issues.apache.org/jira/browse/HIVE-18163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18163:
Summary: Stats: create materialized view should also collect stats (was:
stats: create materialized view should also collect stats)
> Stats: create materialized view should also collect stats
> -
>
> Key: HIVE-18163
> URL: https://issues.apache.org/jira/browse/HIVE-18163
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>
> not having basic stats on the materialized view may cause it to be "ruled
> out" as a viable alternative by the cbo.
> repro set {{set hive.stats.deserialization.factor=10.0}} in
> {{ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q}}
> blocks: HIVE-18149
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18140) Partitioned tables statistics can go wrong in basic stats mixed case
[
https://issues.apache.org/jira/browse/HIVE-18140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268857#comment-16268857
]
Hive QA commented on HIVE-18140:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899061/HIVE-18140.01wip01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 11414 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_table_null_partition]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_part]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_partlvl_dp]
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_var_samp]
(batchId=84)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_reordering_no_stats]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=159)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats8]
(batchId=134)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8021/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8021/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8021/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899061 - PreCommit-HIVE-Build
> Partitioned tables statistics can go wrong in basic stats mixed case
>
>
> Key: HIVE-18140
> URL: https://issues.apache.org/jira/browse/HIVE-18140
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18140.01wip01.patch
>
>
> suppose the following scenario:
> * part1 has basic stats {{RC=10,DS=1K}}
> * all other partition has no basic stats (and a bunch of rows)
> then
> [this|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L378]
> condition would be false; which in turn produces estimations for the whole
> partitioned table: {{RC=10,DS=1K}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18163) Stats: create materialized view should also collect stats
[
https://issues.apache.org/jira/browse/HIVE-18163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18163:
Attachment: HIVE-18163.01.patch
#1)
* collect stats for {{create materialized view}} as well; AFAIK its not
possible to do an update on a materialized view ; so every materialized view
operation can be considered as a rewrite w.r.t stats
* added a small collection to delay the construction of the view objects in
{{MaterializedViewRegistry}}; the reason this was needed is:
** {{StatsTask}} runs after {{DDLTask}} :
*** {{DDLTask}} invoked {{MaterializedViewRegistry}} to put the view in cache
*** {{StatsTask}} filled out the basicStats info in the metastore...
*** next query used the *out-dated* cached table object (which was available at
the time {{MaterializedViewRegistry}} built the scanner.
* I've rerun all the "materialized_view" tests. and the results look good to me
* in materialized_view_create_rewrite_2.q.out the usage of {{cmv_mat_view_5}}
appeared; which looks good to me
[~jcamachorodriguez] could you take a look?
> Stats: create materialized view should also collect stats
> -
>
> Key: HIVE-18163
> URL: https://issues.apache.org/jira/browse/HIVE-18163
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18163.01.patch
>
>
> not having basic stats on the materialized view may cause it to be "ruled
> out" as a viable alternative by the cbo.
> repro set {{set hive.stats.deserialization.factor=10.0}} in
> {{ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q}}
> blocks: HIVE-18149
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18163) Stats: create materialized view should also collect stats
[
https://issues.apache.org/jira/browse/HIVE-18163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18163:
Status: Patch Available (was: Open)
> Stats: create materialized view should also collect stats
> -
>
> Key: HIVE-18163
> URL: https://issues.apache.org/jira/browse/HIVE-18163
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18163.01.patch
>
>
> not having basic stats on the materialized view may cause it to be "ruled
> out" as a viable alternative by the cbo.
> repro set {{set hive.stats.deserialization.factor=10.0}} in
> {{ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q}}
> blocks: HIVE-18149
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18164) Hive2 select with group by error if transactional = true
[ https://issues.apache.org/jira/browse/HIVE-18164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18164: -- Component/s: Transactions > Hive2 select with group by error if transactional = true > - > > Key: HIVE-18164 > URL: https://issues.apache.org/jira/browse/HIVE-18164 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Transactions >Affects Versions: 2.3.0 > Environment: Hortonworks HDP-2.6.3.0 >Reporter: Dmitro-Vasilenko >Priority: Critical > > Connected to: Apache Hive (version 1.2.1000.2.6.3.0-235) > Driver: Hive JDBC (version 1.2.1000.2.6.3.0-235) > 0: jdbc:hive2://serv01:2181,ks-> select sum(destination),messagetype from > t1.cdr where hday='2017-09-14' group by messagetype; > INFO : Session is already open > INFO : Dag name: select sum(destination),messag...messagetype(Stage-1) > ERROR : Status: Failed > ERROR : Vertex failed, vertexName=Map 1, > vertexId=vertex_1511771679762_0301_2_00, diagnostics=[Vertex > vertex_1511771679762_0301_2_00 [Map 1] killed/failed due > to:ROOT_INPUT_INIT_FAILURE, Vertex Input: cdr initializer failed, > vertex=vertex_1511771679762_0301_2_00 [Map 1], java.lang.RuntimeException: > serious problem > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1277) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1304) > at > org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat.getSplits(BucketizedHiveInputFormat.java:141) > at > org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:448) > at > org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:300) > at > org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:266) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: > java.lang.IllegalArgumentException: delta_16881612_29766798 does not start > with base_ > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1254) > ... 16 more > Caused by: java.lang.IllegalArgumentException: delta_16881612_29766798 does > not start with base_ > at > org.apache.hadoop.hive.ql.io.AcidUtils.parseBase(AcidUtils.java:190) > at > org.apache.hadoop.hive.ql.io.AcidUtils.parseBaseBucketFilename(AcidUtils.java:221) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.callInternal(OrcInputFormat.java:804) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.access$600(OrcInputFormat.java:747) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:772) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:769) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:769) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:747) > ... 4 more > ] > > Error occur if delta_* present : << > > [serv01]$ hdfs dfs -ls /warehouse/t1/cdr/hday=2017-09-14 > Found 15 items > drwxrwxrwx
[jira] [Commented] (HIVE-13567) Enable auto-gather column stats by default
[
https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268966#comment-16268966
]
Hive QA commented on HIVE-13567:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} standalone-metastore: The patch generated 3 new + 582
unchanged - 2 fixed = 585 total (was 584) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
53s{color} | {color:red} root: The patch generated 3 new + 1531 unchanged - 2
fixed = 1534 total (was 1533) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8023/yetus/diff-checkstyle-standalone-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8023/yetus/diff-checkstyle-root.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8023/yetus/patch-asflicense-problems.txt
|
| modules | C: common standalone-metastore ql accumulo-handler contrib
hbase-handler . itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8023/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enable auto-gather column stats by default
> --
>
> Key: HIVE-13567
> URL: https://issues.apache.org/jira/browse/HIVE-13567
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Zoltan Haindrich
> Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch,
> HIVE-13567.03.patch, HIVE-13567.04.patch, HIVE-13567.05.patch,
> HIVE-13567.06.patch, HIVE-13567.07.patch, HIVE-13567.08.patch,
> HIVE-13567.09.patch, HIVE-13567.10.patch, HIVE-13567.11.patch,
> HIVE-13567.12.patch, HIVE-13567.13.patch, HIVE-13567.14.patch,
> HIVE-13567.15.patch, HIVE-13567.16.patch, HIVE-13567.17.patch,
> HIVE-13567.18.patch, HIVE-13567.19.patch, HIVE
[jira] [Commented] (HIVE-13567) Enable auto-gather column stats by default
[
https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268972#comment-16268972
]
Hive QA commented on HIVE-13567:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899036/HIVE-13567.23wip07.patch
{color:green}SUCCESS:{color} +1 due to 44 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 143 failed/errored test(s), 11413 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_stats]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_numbuckets_partitioned_table2_h23]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_numbuckets_partitioned_table_h23]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_change_col]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_rename_partition_authorization]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_serde2]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_6]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_explain]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_2]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_disable_cbo_3]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join11] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_comments]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_date] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_partitioned]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ba_table3] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin10]
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin11]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin12]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin13]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin8]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin9]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_1]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[compute_stats_decimal]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constGby] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_like_view]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_or_replace_view]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_with_constraints]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_5] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[database_drop]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_whole_partition]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[describe_table]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[display_colstats_tbllvl]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic1] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic3]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_topn] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[equal_ns] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_04_evolved_parts]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[extract] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_cond_pushdown]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_limit]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby4_noskew]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby7_noskew_multi_single_reducer]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets_grouping]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[hoo
[jira] [Commented] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[
https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269002#comment-16269002
]
Hive QA commented on HIVE-18157:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8024/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8024/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization : Insert in bucketed table is broken with vectorization
> -
>
> Key: HIVE-18157
> URL: https://issues.apache.org/jira/browse/HIVE-18157
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-18157.1.patch, HIVE-18157.3.patch
>
>
> create temporary table foo (x int) clustered by (x) into 4 buckets;
> insert overwrite table foo values(1),(2),(3),(4),(9);
> select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo;
> OK
> 9 00_0
> 4 00_0
> 3 00_0
> 2 00_0
> 1 00_0
> set hive.vectorized.execution.enabled=false;
> insert overwrite table foo values(1),(2),(3),(4),(9);
> select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo;
> OK
> 4 00_0
> 9 01_0
> 1 01_0
> 2 02_0
> 3 03_0
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[ https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269040#comment-16269040 ] Janaki Lahorani commented on HIVE-18147: [~vihangk1] The new ASF License Warnings are not caused by this patch. The originally reported checkstyle issues are resolved. > Tests can fail with java.net.BindException: Address already in use > -- > > Key: HIVE-18147 > URL: https://issues.apache.org/jira/browse/HIVE-18147 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch, > HIVE-18147.3.patch, HIVE-18147.4.patch > > > Between the time a new port is picked up to start HMS or HS2, and the time > when HMS or HS2 is started, the port can get used by some other service > resulting in test failure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[
https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269073#comment-16269073
]
Hive QA commented on HIVE-18157:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899580/HIVE-18157.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11413 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=157)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8024/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8024/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8024/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899580 - PreCommit-HIVE-Build
> Vectorization : Insert in bucketed table is broken with vectorization
> -
>
> Key: HIVE-18157
> URL: https://issues.apache.org/jira/browse/HIVE-18157
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-18157.1.patch, HIVE-18157.3.patch
>
>
> create temporary table foo (x int) clustered by (x) into 4 buckets;
> insert overwrite table foo values(1),(2),(3),(4),(9);
> select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo;
> OK
> 9 00_0
> 4 00_0
> 3 00_0
> 2 00_0
> 1 00_0
> set hive.vectorized.execution.enabled=false;
> insert overwrite table foo values(1),(2),(3),(4),(9);
> select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo;
> OK
> 4 00_0
> 9 01_0
> 1 01_0
> 2 02_0
> 3 03_0
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18166) Result of hive.query.string is encoded.
[ https://issues.apache.org/jira/browse/HIVE-18166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani reassigned HIVE-18166: -- > Result of hive.query.string is encoded. > --- > > Key: HIVE-18166 > URL: https://issues.apache.org/jira/browse/HIVE-18166 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > > set hive.query.string returns encoded string. > hive.query.string=%0A%0Aselect+*+from+t1 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17980) Move HiveMetaStoreClient plus a few remaining classes.
[ https://issues.apache.org/jira/browse/HIVE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-17980: -- Attachment: HIVE-17980.2.patch Second version of the patch that fixes unit test failures. > Move HiveMetaStoreClient plus a few remaining classes. > -- > > Key: HIVE-17980 > URL: https://issues.apache.org/jira/browse/HIVE-17980 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates > Labels: pull-request-available > Attachments: HIVE-17980.2.patch, HIVE-17980.patch > > > This moves over HiveMetaStoreClient a few more remaining things that haven't > been moved yet. It also cleans up the mess I made of MetaStoreUtils. I > renamed the class in metastore to HiveMetaStoreUtils and left a few methods > in there that could be moved. Everything else has been moved to > standalone-metastore/.../MetaStoreUtils. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18160) Jar localization during session initialization is slow
[
https://issues.apache.org/jira/browse/HIVE-18160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269120#comment-16269120
]
Hive QA commented on HIVE-18160:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 7 new + 43 unchanged - 3 fixed
= 50 total (was 46) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8025/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8025/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8025/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Jar localization during session initialization is slow
> --
>
> Key: HIVE-18160
> URL: https://issues.apache.org/jira/browse/HIVE-18160
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18160.1.patch
>
>
> Same Jar getting localized multiple times resulting in SHA256 computation
> several times causes slow session initialization time.
> {code}
> 2017-11-28T00:40:55,795 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,105 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> e20986f3a422f8fa5eb61c5a2756cd6f7d2b779dbcab49eae6f2c8dfff7ad2a2 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-llap-tez-3.0.0-SNAPSHOT.jar
> of length: 109.53KB in 1 ms
> 2017-11-28T00:40:56,353 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSH
[jira] [Commented] (HIVE-17954) Implement pool, user, group and trigger to pool management API's.
[ https://issues.apache.org/jira/browse/HIVE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269172#comment-16269172 ] Vihang Karajgaonkar commented on HIVE-17954: Looks like this patch introduces some files which don't have ASF license headers and its been showing up on other patches as a Yetus -1. Can you please help fix them? Thanks! Lines that start with ? in the ASF License report indicate files that do not have an Apache license header: !? /data/hiveptest/working/yetus/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateOrDropTriggerToPoolMappingDesc.java !? /data/hiveptest/working/yetus/ql/src/java/org/apache/hadoop/hive/ql/plan/DropWMMappingDesc.java !? /data/hiveptest/working/yetus/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateOrAlterWMMappingDesc.java !? /data/hiveptest/working/yetus/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateOrAlterWMPoolDesc.java !? /data/hiveptest/working/yetus/ql/src/java/org/apache/hadoop/hive/ql/plan/DropWMPoolDesc.java > Implement pool, user, group and trigger to pool management API's. > - > > Key: HIVE-17954 > URL: https://issues.apache.org/jira/browse/HIVE-17954 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Fix For: 3.0.0 > > Attachments: HIVE-17954.01.patch, HIVE-17954.02.patch, > HIVE-17954.03.patch, HIVE-17954.04.patch, HIVE-17954.05.patch, > HIVE-17954.06.patch, HIVE-17954.07.patch, HIVE-17954.08.patch, > HIVE-17954.09.patch, HIVE-17954.10.patch > > > Implement the following commands: > -- Pool management. > CREATE POOL `resource_plan`.`pool_path` WITH > ALLOC_FRACTION=`fraction`, > QUERY_PARALLELISM=`parallelism`, > SCHEDULING_POLICY=`policy`; > ALTER POOL `resource_plan`.`pool_path` SET > PATH = `new_path`, > ALLOC_FRACTION = `fraction`, > QUERY_PARALLELISM = `parallelism`, > SCHEDULING_POLICY = `policy`; > DROP POOL `resource_plan`.`pool_path`; > -- Adding triggers to pools. > ALTER POOL `resource_plan`.`pool_path` ADD TRIGGER `trigger_name`; > ALTER POOL `resource_plan`.`pool_path` DROP TRIGGER `trigger_name`; > -- User/Group to pool mappings. > CREATE USER|GROUP MAPPING `resource_plan`.`group_or_user_name` > TO `pool_path` WITH ORDERING `order_no`; > DROP USER|GROUP MAPPING `resource_plan`.`group_or_user_name`; -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[ https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269178#comment-16269178 ] Vihang Karajgaonkar commented on HIVE-18147: I see .. Thanks for confirming [~janulatha] It seems to be originating from HIVE-17954. I have commented on that JIRA about the same. Hopefully they should be fixed soon. org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_decimal] seems to be new failures between version 3 and version 4 patch. Is that related? > Tests can fail with java.net.BindException: Address already in use > -- > > Key: HIVE-18147 > URL: https://issues.apache.org/jira/browse/HIVE-18147 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch, > HIVE-18147.3.patch, HIVE-18147.4.patch > > > Between the time a new port is picked up to start HMS or HS2, and the time > when HMS or HS2 is started, the port can get used by some other service > resulting in test failure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269180#comment-16269180 ] Vihang Karajgaonkar commented on HIVE-18157: Thanks for confirming [~djaiswal] Linking the break JIRA here so that its easier to track. > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18157.1.patch, HIVE-18157.3.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18140) Partitioned tables statistics can go wrong in basic stats mixed case
[
https://issues.apache.org/jira/browse/HIVE-18140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269196#comment-16269196
]
Zoltan Haindrich commented on HIVE-18140:
-
I've merged HIVE-18108 locally into this...
> Partitioned tables statistics can go wrong in basic stats mixed case
>
>
> Key: HIVE-18140
> URL: https://issues.apache.org/jira/browse/HIVE-18140
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18140.01wip01.patch
>
>
> suppose the following scenario:
> * part1 has basic stats {{RC=10,DS=1K}}
> * all other partition has no basic stats (and a bunch of rows)
> then
> [this|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L378]
> condition would be false; which in turn produces estimations for the whole
> partitioned table: {{RC=10,DS=1K}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18005) Improve size estimation for array() to be not 0
[
https://issues.apache.org/jira/browse/HIVE-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18005:
Attachment: HIVE-18005.01.patch
> Improve size estimation for array() to be not 0
> ---
>
> Key: HIVE-18005
> URL: https://issues.apache.org/jira/browse/HIVE-18005
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18005.01.patch, HIVE-18005.01.patch
>
>
> happens only in case the array is not from a column; and the array contains
> no column references
> {code}
> EXPLAIN
> SELECT sort_array(array("b", "d", "c", "a")),array("1","2") FROM t
> ...
> Statistics: Num rows: 1 Data size: 0 Basic stats: COMPLETE
> Column stats: COMPLETE
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18160) Jar localization during session initialization is slow
[
https://issues.apache.org/jira/browse/HIVE-18160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269203#comment-16269203
]
Hive QA commented on HIVE-18160:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899585/HIVE-18160.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11412 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=159)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8025/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8025/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8025/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899585 - PreCommit-HIVE-Build
> Jar localization during session initialization is slow
> --
>
> Key: HIVE-18160
> URL: https://issues.apache.org/jira/browse/HIVE-18160
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18160.1.patch
>
>
> Same Jar getting localized multiple times resulting in SHA256 computation
> several times causes slow session initialization time.
> {code}
> 2017-11-28T00:40:55,795 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,105 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> e20986f3a422f8fa5eb61c5a2756cd6f7d2b779dbcab49eae6f2c8dfff7ad2a2 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-llap-tez-3.0.0-SNAPSHOT.jar
> of length: 109.53KB in 1 ms
> 2017-11-28T00:40:56,353 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 231 ms
> 2017-11-28T00:40:56,602 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,612 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 686d66b825fdc4fc241e0591e7646a1bbca1c7114a7224c41da7f4795cf9477a for file:
> file:/work/hadoop/hadoop/hadoop-dist/target/hadoop-2.9.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-registry-2.9.0-SNAPSHOT.jar
> of length: 122.72KB in 2 ms
> {code}
> From above logs, sha256 is computed 3 times for hive-exec jar and each
> invocation takes around 240ms.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18149) Stats: rownum estimation from datasize underestimates in most cases
[
https://issues.apache.org/jira/browse/HIVE-18149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18149:
Attachment: HIVE-18149.01wip01.patch
#1)
* set deser factor to {{10}}
> Stats: rownum estimation from datasize underestimates in most cases
> ---
>
> Key: HIVE-18149
> URL: https://issues.apache.org/jira/browse/HIVE-18149
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18149.01wip01.patch
>
>
> rownum estimation is based on the following fact as of now:
> * datasize being used from the following sources:
> ** basicstats aggregates the loaded "on-heap" row sizes ; other readers are
> able to give "raw size" estimation - I've checked orc; but I'm sure others
> will do the sameapi docs are a bit vague about the methods purpose...
> ** if the basicstats level info is not available; the filesystem level
> "file-size-sums" are used as the "raw data size" ; which is multiplied by the
> [deserialization
> ratio|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L261]
> ; which is currently 1.
> the problem with all of this is that deser factor is 1; and that rowsize
> counts in the online object headers..
> example; 20 rows are loaded into a partition
> [columnstats_partlvl_dp.q|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L7]
> after HIVE-18108 [this
> explain|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L25]
> will estimate the rowsize of the table to be 404 bytes; however the 20 rows
> of text is only 169 bytes...so it ends up with 0 rows...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[ https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269228#comment-16269228 ] Janaki Lahorani commented on HIVE-18147: [~vihangk1]These test failures are not related to this patch. > Tests can fail with java.net.BindException: Address already in use > -- > > Key: HIVE-18147 > URL: https://issues.apache.org/jira/browse/HIVE-18147 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch, > HIVE-18147.3.patch, HIVE-18147.4.patch > > > Between the time a new port is picked up to start HMS or HS2, and the time > when HMS or HS2 is started, the port can get used by some other service > resulting in test failure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18163) Stats: create materialized view should also collect stats
[
https://issues.apache.org/jira/browse/HIVE-18163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269240#comment-16269240
]
Jesus Camacho Rodriguez commented on HIVE-18163:
[~kgyrtkirk], thanks for the patch.
Could we follow a different approach: would it be possible to put the view in
the materialized view registry after StatsTask has run?
The problem with the approach in the patch is that we will load the
materialized views in the cache at query planning time (when we call
_getRewritingMaterializedViews_): we do not want to do that as it will put an
additional burden on compilation time for queries (loading the query into the
cache in fact parses the query, creates the Calcite logical plan, etc.). The
assumption is that it is acceptable that the MV is not available immediately
after creation for query rewriting.
> Stats: create materialized view should also collect stats
> -
>
> Key: HIVE-18163
> URL: https://issues.apache.org/jira/browse/HIVE-18163
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18163.01.patch
>
>
> not having basic stats on the materialized view may cause it to be "ruled
> out" as a viable alternative by the cbo.
> repro set {{set hive.stats.deserialization.factor=10.0}} in
> {{ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q}}
> blocks: HIVE-18149
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18163) Stats: create materialized view should also collect stats
[
https://issues.apache.org/jira/browse/HIVE-18163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269253#comment-16269253
]
Hive QA commented on HIVE-18163:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 13 unchanged - 0 fixed
= 14 total (was 13) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8026/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8026/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8026/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Stats: create materialized view should also collect stats
> -
>
> Key: HIVE-18163
> URL: https://issues.apache.org/jira/browse/HIVE-18163
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18163.01.patch
>
>
> not having basic stats on the materialized view may cause it to be "ruled
> out" as a viable alternative by the cbo.
> repro set {{set hive.stats.deserialization.factor=10.0}} in
> {{ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q}}
> blocks: HIVE-18149
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18167) Optimizer: Rewrite Constant BETWEEN col1 and col2 for PPD
[
https://issues.apache.org/jira/browse/HIVE-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18167:
---
Labels: newbie++ (was: )
> Optimizer: Rewrite Constant BETWEEN col1 and col2 for PPD
> -
>
> Key: HIVE-18167
> URL: https://issues.apache.org/jira/browse/HIVE-18167
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Reporter: Gopal V
> Labels: newbie++
>
> {code}
> CURRENT_DATE between start_date and end_date
> {code}
> is not pushed down to the predicate layer, while the inclusive comparison ops
> {code}
> start_date <= CURRENT_DATE and end_date >= CURRENT_DATE
> {code}
> are evaluated at the row-index level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-18163) Stats: create materialized view should also collect stats
[
https://issues.apache.org/jira/browse/HIVE-18163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269240#comment-16269240
]
Jesus Camacho Rodriguez edited comment on HIVE-18163 at 11/28/17 6:51 PM:
--
[~kgyrtkirk], thanks for the patch.
Could we follow a different approach: would it be possible to put the view in
the materialized view registry after StatsTask has run?
The problem with the approach in the patch is that we will load the
materialized views in the cache at query planning time (when we call
_getRewritingMaterializedViews_): we do not want to do that as it will put an
additional burden on compilation time for queries (loading the query into the
cache in fact parses the query, creates the Calcite logical plan, etc.).
was (Author: jcamachorodriguez):
[~kgyrtkirk], thanks for the patch.
Could we follow a different approach: would it be possible to put the view in
the materialized view registry after StatsTask has run?
The problem with the approach in the patch is that we will load the
materialized views in the cache at query planning time (when we call
_getRewritingMaterializedViews_): we do not want to do that as it will put an
additional burden on compilation time for queries (loading the query into the
cache in fact parses the query, creates the Calcite logical plan, etc.). The
assumption is that it is acceptable that the MV is not available immediately
after creation for query rewriting.
> Stats: create materialized view should also collect stats
> -
>
> Key: HIVE-18163
> URL: https://issues.apache.org/jira/browse/HIVE-18163
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18163.01.patch
>
>
> not having basic stats on the materialized view may cause it to be "ruled
> out" as a viable alternative by the cbo.
> repro set {{set hive.stats.deserialization.factor=10.0}} in
> {{ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q}}
> blocks: HIVE-18149
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269269#comment-16269269 ] Deepak Jaiswal commented on HIVE-18157: --- Only failing test sysdb passed repeatedly locally. > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18157.1.patch, HIVE-18157.3.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Yeom updated HIVE-17856:
--
Attachment: HIVE-17856.16.patch
Please ignore this patch 16, which is purely for debugging purpose.
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch,
> HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch,
> HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.16.patch,
> HIVE-17856.2.patch, HIVE-17856.3.patch, HIVE-17856.4.patch,
> HIVE-17856.5.patch, HIVE-17856.6.patch, HIVE-17856.7.patch,
> HIVE-17856.8.patch, HIVE-17856.9.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18152) Better error message when activating an already active resource plan
[
https://issues.apache.org/jira/browse/HIVE-18152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-18152:
Assignee: Prasanth Jayachandran
> Better error message when activating an already active resource plan
>
>
> Key: HIVE-18152
> URL: https://issues.apache.org/jira/browse/HIVE-18152
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> {code}
> show resource plans;
> +--+--++
> | rp_name | status | query_parallelism |
> +--+--++
> | llap | ACTIVE | 1 |
> | global | ENABLED | 1 |
> +--+--++
> ALTER RESOURCE PLAN llap ACTIVATE;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot get a resource plan to
> apply (state=08S01,code=1)
> {code}
> This error message can be improved to say the plan is already active.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18157: Fix Version/s: 3.0.0 > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Fix For: 3.0.0 > > Attachments: HIVE-18157.1.patch, HIVE-18157.3.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269329#comment-16269329 ] Matt McCline commented on HIVE-18157: - Committed to master. > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Fix For: 3.0.0 > > Attachments: HIVE-18157.1.patch, HIVE-18157.3.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18157: Resolution: Fixed Status: Resolved (was: Patch Available) > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Fix For: 3.0.0 > > Attachments: HIVE-18157.1.patch, HIVE-18157.3.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269333#comment-16269333 ] Deepak Jaiswal commented on HIVE-18157: --- Thanks [~mmccline] for the review and commit. > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Fix For: 3.0.0 > > Attachments: HIVE-18157.1.patch, HIVE-18157.3.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18163) Stats: create materialized view should also collect stats
[
https://issues.apache.org/jira/browse/HIVE-18163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269342#comment-16269342
]
Hive QA commented on HIVE-18163:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899636/HIVE-18163.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11412 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=244)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=159)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8026/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8026/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8026/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899636 - PreCommit-HIVE-Build
> Stats: create materialized view should also collect stats
> -
>
> Key: HIVE-18163
> URL: https://issues.apache.org/jira/browse/HIVE-18163
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18163.01.patch
>
>
> not having basic stats on the materialized view may cause it to be "ruled
> out" as a viable alternative by the cbo.
> repro set {{set hive.stats.deserialization.factor=10.0}} in
> {{ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q}}
> blocks: HIVE-18149
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18152) Idempotent state change for resource plan
[
https://issues.apache.org/jira/browse/HIVE-18152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18152:
-
Summary: Idempotent state change for resource plan (was: Better error
message when activating an already active resource plan)
> Idempotent state change for resource plan
> -
>
> Key: HIVE-18152
> URL: https://issues.apache.org/jira/browse/HIVE-18152
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> {code}
> show resource plans;
> +--+--++
> | rp_name | status | query_parallelism |
> +--+--++
> | llap | ACTIVE | 1 |
> | global | ENABLED | 1 |
> +--+--++
> ALTER RESOURCE PLAN llap ACTIVATE;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot get a resource plan to
> apply (state=08S01,code=1)
> {code}
> This error message can be improved to say the plan is already active.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[ https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269349#comment-16269349 ] Vihang Karajgaonkar commented on HIVE-18147: org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_decimal] worked for me as well with the patch. Commited the patch to master. > Tests can fail with java.net.BindException: Address already in use > -- > > Key: HIVE-18147 > URL: https://issues.apache.org/jira/browse/HIVE-18147 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch, > HIVE-18147.3.patch, HIVE-18147.4.patch > > > Between the time a new port is picked up to start HMS or HS2, and the time > when HMS or HS2 is started, the port can get used by some other service > resulting in test failure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18152) Idempotent state change for resource plan
[
https://issues.apache.org/jira/browse/HIVE-18152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18152:
-
Description:
{code}
show resource plans;
+--+--++
| rp_name | status | query_parallelism |
+--+--++
| llap | ACTIVE | 1 |
| global | ENABLED | 1 |
+--+--++
ALTER RESOURCE PLAN llap ACTIVATE;
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot get a resource plan to
apply (state=08S01,code=1)
{code}
It is better not to throw an error when current state is same as the altered
state.
was:
{code}
show resource plans;
+--+--++
| rp_name | status | query_parallelism |
+--+--++
| llap | ACTIVE | 1 |
| global | ENABLED | 1 |
+--+--++
ALTER RESOURCE PLAN llap ACTIVATE;
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot get a resource plan to
apply (state=08S01,code=1)
{code}
This error message can be improved to say the plan is already active.
> Idempotent state change for resource plan
> -
>
> Key: HIVE-18152
> URL: https://issues.apache.org/jira/browse/HIVE-18152
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> {code}
> show resource plans;
> +--+--++
> | rp_name | status | query_parallelism |
> +--+--++
> | llap | ACTIVE | 1 |
> | global | ENABLED | 1 |
> +--+--++
> ALTER RESOURCE PLAN llap ACTIVATE;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot get a resource plan to
> apply (state=08S01,code=1)
> {code}
> It is better not to throw an error when current state is same as the altered
> state.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18152) Idempotent state change for resource plan
[
https://issues.apache.org/jira/browse/HIVE-18152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18152:
-
Status: Patch Available (was: Open)
> Idempotent state change for resource plan
> -
>
> Key: HIVE-18152
> URL: https://issues.apache.org/jira/browse/HIVE-18152
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18152.1.patch
>
>
> {code}
> show resource plans;
> +--+--++
> | rp_name | status | query_parallelism |
> +--+--++
> | llap | ACTIVE | 1 |
> | global | ENABLED | 1 |
> +--+--++
> ALTER RESOURCE PLAN llap ACTIVATE;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot get a resource plan to
> apply (state=08S01,code=1)
> {code}
> It is better not to throw an error when current state is same as the altered
> state.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18152) Idempotent state change for resource plan
[
https://issues.apache.org/jira/browse/HIVE-18152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18152:
-
Attachment: HIVE-18152.1.patch
> Idempotent state change for resource plan
> -
>
> Key: HIVE-18152
> URL: https://issues.apache.org/jira/browse/HIVE-18152
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18152.1.patch
>
>
> {code}
> show resource plans;
> +--+--++
> | rp_name | status | query_parallelism |
> +--+--++
> | llap | ACTIVE | 1 |
> | global | ENABLED | 1 |
> +--+--++
> ALTER RESOURCE PLAN llap ACTIVATE;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot get a resource plan to
> apply (state=08S01,code=1)
> {code}
> It is better not to throw an error when current state is same as the altered
> state.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17361: -- Attachment: HIVE-17361.25.patch > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Wei Zheng >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch, > HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch, > HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch, > HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch, > HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.21.patch, > HIVE-17361.23.patch, HIVE-17361.24.patch, HIVE-17361.25.patch, > HIVE-17361.3.patch, HIVE-17361.4.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. > Current Documentation is under [DML > Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] > and [Loading files into > tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: > \\ > * Load Data performs very limited validations of the data, in particular it > uses the input file name which may not be in 0_0 which can break some > read logic. (Certainly will for Acid). > * It does not check the schema of the file. This may be a non issue for Acid > which requires ORC which is self describing so Schema Evolution may handle > this seamlessly. (Assuming Schema is not too different). > * It does check that _InputFormat_S are compatible. > * Bucketed (and thus sorted) tables don't support Load Data (but only if > hive.strict.checks.bucketing=true (default)). Will keep this restriction for > Acid. > * Load Data supports OVERWRITE clause > * What happens to file permissions/ownership: rename vs copy differences > \\ > The implementation will follow the same idea as in HIVE-14988 and use a > base_N/ dir for OVERWRITE clause. > \\ > How is minor compaction going to handle delta/base with original files? > Since delta_8_8/_meta_data is created before files are moved, delta_8_8 > becomes visible before it's populated. Is that an issue? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17361: -- Description: LOAD DATA was not supported since ACID was introduced. Need to fill this gap between ACID table and regular hive table. Current Documentation is under [DML Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] and [Loading files into tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: \\ * Load Data performs very limited validations of the data, in particular it uses the input file name which may not be in 0_0 which can break some read logic. (Certainly will for Acid). * It does not check the schema of the file. This may be a non issue for Acid which requires ORC which is self describing so Schema Evolution may handle this seamlessly. (Assuming Schema is not too different). * It does check that _InputFormat_S are compatible. * Bucketed (and thus sorted) tables don't support Load Data (but only if hive.strict.checks.bucketing=true (default)). Will keep this restriction for Acid. * Load Data supports OVERWRITE clause * What happens to file permissions/ownership: rename vs copy differences \\ The implementation will follow the same idea as in HIVE-14988 and use a base_N/ dir for OVERWRITE clause. \\ How is minor compaction going to handle delta/base with original files? Since delta_8_8/_meta_data is created before files are moved, delta_8_8 becomes visible before it's populated. Is that an issue? It's not since txn 8 is not committed. Implementation Notes/Limitations (patch 25) * bucketed/sorted tables are not supported * input files names must be of the form 0_0/0_0_copy_1 - enforced. was: LOAD DATA was not supported since ACID was introduced. Need to fill this gap between ACID table and regular hive table. Current Documentation is under [DML Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] and [Loading files into tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: \\ * Load Data performs very limited validations of the data, in particular it uses the input file name which may not be in 0_0 which can break some read logic. (Certainly will for Acid). * It does not check the schema of the file. This may be a non issue for Acid which requires ORC which is self describing so Schema Evolution may handle this seamlessly. (Assuming Schema is not too different). * It does check that _InputFormat_S are compatible. * Bucketed (and thus sorted) tables don't support Load Data (but only if hive.strict.checks.bucketing=true (default)). Will keep this restriction for Acid. * Load Data supports OVERWRITE clause * What happens to file permissions/ownership: rename vs copy differences \\ The implementation will follow the same idea as in HIVE-14988 and use a base_N/ dir for OVERWRITE clause. \\ How is minor compaction going to handle delta/base with original files? Since delta_8_8/_meta_data is created before files are moved, delta_8_8 becomes visible before it's populated. Is that an issue? > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Wei Zheng >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch, > HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch, > HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch, > HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch, > HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.21.patch, > HIVE-17361.23.patch, HIVE-17361.24.patch, HIVE-17361.25.patch, > HIVE-17361.3.patch, HIVE-17361.4.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. > Current Documentation is under [DML > Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] > and [Loading files into > tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: > \\ > * Load Data performs very limited validations of the data, in particular it > uses the input file name which may not be in 0_0 which can break some > read logic. (Certainly will for Acid). > * It does not check the schema of the file. This may be a non issue for Acid > which requires ORC which is self describing so Schema Evolution may handle > this seamlessly. (Assuming Sc
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17361: -- Description: LOAD DATA was not supported since ACID was introduced. Need to fill this gap between ACID table and regular hive table. Current Documentation is under [DML Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] and [Loading files into tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: \\ * Load Data performs very limited validations of the data, in particular it uses the input file name which may not be in 0_0 which can break some read logic. (Certainly will for Acid). * It does not check the schema of the file. This may be a non issue for Acid which requires ORC which is self describing so Schema Evolution may handle this seamlessly. (Assuming Schema is not too different). * It does check that _InputFormat_S are compatible. * Bucketed (and thus sorted) tables don't support Load Data (but only if hive.strict.checks.bucketing=true (default)). Will keep this restriction for Acid. * Load Data supports OVERWRITE clause * What happens to file permissions/ownership: rename vs copy differences \\ The implementation will follow the same idea as in HIVE-14988 and use a base_N/ dir for OVERWRITE clause. \\ How is minor compaction going to handle delta/base with original files? Since delta_8_8/_meta_data is created before files are moved, delta_8_8 becomes visible before it's populated. Is that an issue? It's not since txn 8 is not committed. h3. Implementation Notes/Limitations (patch 25) * bucketed/sorted tables are not supported * input files names must be of the form 0_0/0_0_copy_1 - enforced. (HIVE-18125) * Load Data creates a delta_x_x/ that contains new files * Load Data w/Overwrite creates a base_x/ that contains new files * The input files must be 'plain' ORC files, i.e. not contain acid metadata columns as would be the case if these files were copied from another Acid table. In the latter case, the ROW_IDs embedded in the data may not make sense in the target table (if it's in a different cluster, for example). Such files may also have a mix of committed and aborted data. ** this could be relaxed later by adding info to the _metadata_acid file to ignore existing ROW_IDs on read. * ROW_IDs are attached dynamically at read time and made permanent by compaction. This is done the same way has handling of files that were written to a table before it was converted to Acid. * Vectorization is supported was: LOAD DATA was not supported since ACID was introduced. Need to fill this gap between ACID table and regular hive table. Current Documentation is under [DML Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] and [Loading files into tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: \\ * Load Data performs very limited validations of the data, in particular it uses the input file name which may not be in 0_0 which can break some read logic. (Certainly will for Acid). * It does not check the schema of the file. This may be a non issue for Acid which requires ORC which is self describing so Schema Evolution may handle this seamlessly. (Assuming Schema is not too different). * It does check that _InputFormat_S are compatible. * Bucketed (and thus sorted) tables don't support Load Data (but only if hive.strict.checks.bucketing=true (default)). Will keep this restriction for Acid. * Load Data supports OVERWRITE clause * What happens to file permissions/ownership: rename vs copy differences \\ The implementation will follow the same idea as in HIVE-14988 and use a base_N/ dir for OVERWRITE clause. \\ How is minor compaction going to handle delta/base with original files? Since delta_8_8/_meta_data is created before files are moved, delta_8_8 becomes visible before it's populated. Is that an issue? It's not since txn 8 is not committed. Implementation Notes/Limitations (patch 25) * bucketed/sorted tables are not supported * input files names must be of the form 0_0/0_0_copy_1 - enforced. > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Wei Zheng >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch, > HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch, > HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch, > HIVE-17361.
[jira] [Commented] (HIVE-17954) Implement pool, user, group and trigger to pool management API's.
[ https://issues.apache.org/jira/browse/HIVE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269408#comment-16269408 ] Sergey Shelukhin commented on HIVE-17954: - Fixed, thanks for the pointer > Implement pool, user, group and trigger to pool management API's. > - > > Key: HIVE-17954 > URL: https://issues.apache.org/jira/browse/HIVE-17954 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Fix For: 3.0.0 > > Attachments: HIVE-17954.01.patch, HIVE-17954.02.patch, > HIVE-17954.03.patch, HIVE-17954.04.patch, HIVE-17954.05.patch, > HIVE-17954.06.patch, HIVE-17954.07.patch, HIVE-17954.08.patch, > HIVE-17954.09.patch, HIVE-17954.10.patch > > > Implement the following commands: > -- Pool management. > CREATE POOL `resource_plan`.`pool_path` WITH > ALLOC_FRACTION=`fraction`, > QUERY_PARALLELISM=`parallelism`, > SCHEDULING_POLICY=`policy`; > ALTER POOL `resource_plan`.`pool_path` SET > PATH = `new_path`, > ALLOC_FRACTION = `fraction`, > QUERY_PARALLELISM = `parallelism`, > SCHEDULING_POLICY = `policy`; > DROP POOL `resource_plan`.`pool_path`; > -- Adding triggers to pools. > ALTER POOL `resource_plan`.`pool_path` ADD TRIGGER `trigger_name`; > ALTER POOL `resource_plan`.`pool_path` DROP TRIGGER `trigger_name`; > -- User/Group to pool mappings. > CREATE USER|GROUP MAPPING `resource_plan`.`group_or_user_name` > TO `pool_path` WITH ORDERING `order_no`; > DROP USER|GROUP MAPPING `resource_plan`.`group_or_user_name`; -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17361: -- Description: LOAD DATA was not supported since ACID was introduced. Need to fill this gap between ACID table and regular hive table. Current Documentation is under [DML Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] and [Loading files into tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: \\ * Load Data performs very limited validations of the data, in particular it uses the input file name which may not be in 0_0 which can break some read logic. (Certainly will for Acid). * It does not check the schema of the file. This may be a non issue for Acid which requires ORC which is self describing so Schema Evolution may handle this seamlessly. (Assuming Schema is not too different). * It does check that _InputFormat_S are compatible. * Bucketed (and thus sorted) tables don't support Load Data (but only if hive.strict.checks.bucketing=true (default)). Will keep this restriction for Acid. * Load Data supports OVERWRITE clause * What happens to file permissions/ownership: rename vs copy differences \\ The implementation will follow the same idea as in HIVE-14988 and use a base_N/ dir for OVERWRITE clause. \\ How is minor compaction going to handle delta/base with original files? Since delta_8_8/_meta_data is created before files are moved, delta_8_8 becomes visible before it's populated. Is that an issue? It's not since txn 8 is not committed. h3. Implementation Notes/Limitations (patch 25) * bucketed/sorted tables are not supported * input files names must be of the form 0_0/0_0_copy_1 - enforced. (HIVE-18125) * Load Data creates a delta_x_x/ that contains new files * Load Data w/Overwrite creates a base_x/ that contains new files * A '_metadata_acid' file is placed in the target directory to indicate it requires special handling on read * The input files must be 'plain' ORC files, i.e. not contain acid metadata columns as would be the case if these files were copied from another Acid table. In the latter case, the ROW_IDs embedded in the data may not make sense in the target table (if it's in a different cluster, for example). Such files may also have a mix of committed and aborted data. ** this could be relaxed later by adding info to the _metadata_acid file to ignore existing ROW_IDs on read. * ROW_IDs are attached dynamically at read time and made permanent by compaction. This is done the same way has handling of files that were written to a table before it was converted to Acid. * Vectorization is supported was: LOAD DATA was not supported since ACID was introduced. Need to fill this gap between ACID table and regular hive table. Current Documentation is under [DML Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] and [Loading files into tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: \\ * Load Data performs very limited validations of the data, in particular it uses the input file name which may not be in 0_0 which can break some read logic. (Certainly will for Acid). * It does not check the schema of the file. This may be a non issue for Acid which requires ORC which is self describing so Schema Evolution may handle this seamlessly. (Assuming Schema is not too different). * It does check that _InputFormat_S are compatible. * Bucketed (and thus sorted) tables don't support Load Data (but only if hive.strict.checks.bucketing=true (default)). Will keep this restriction for Acid. * Load Data supports OVERWRITE clause * What happens to file permissions/ownership: rename vs copy differences \\ The implementation will follow the same idea as in HIVE-14988 and use a base_N/ dir for OVERWRITE clause. \\ How is minor compaction going to handle delta/base with original files? Since delta_8_8/_meta_data is created before files are moved, delta_8_8 becomes visible before it's populated. Is that an issue? It's not since txn 8 is not committed. h3. Implementation Notes/Limitations (patch 25) * bucketed/sorted tables are not supported * input files names must be of the form 0_0/0_0_copy_1 - enforced. (HIVE-18125) * Load Data creates a delta_x_x/ that contains new files * Load Data w/Overwrite creates a base_x/ that contains new files * A '_metadata_acid' files is placed in the target directory to indicate it requires special handling on read * The input files must be 'plain' ORC files, i.e. not contain acid metadata columns as would be the case if these files were copied from another Acid table. In the latter case, the ROW_IDs embedded in the data may not make sense in the target table
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17361: -- Description: LOAD DATA was not supported since ACID was introduced. Need to fill this gap between ACID table and regular hive table. Current Documentation is under [DML Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] and [Loading files into tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: \\ * Load Data performs very limited validations of the data, in particular it uses the input file name which may not be in 0_0 which can break some read logic. (Certainly will for Acid). * It does not check the schema of the file. This may be a non issue for Acid which requires ORC which is self describing so Schema Evolution may handle this seamlessly. (Assuming Schema is not too different). * It does check that _InputFormat_S are compatible. * Bucketed (and thus sorted) tables don't support Load Data (but only if hive.strict.checks.bucketing=true (default)). Will keep this restriction for Acid. * Load Data supports OVERWRITE clause * What happens to file permissions/ownership: rename vs copy differences \\ The implementation will follow the same idea as in HIVE-14988 and use a base_N/ dir for OVERWRITE clause. \\ How is minor compaction going to handle delta/base with original files? Since delta_8_8/_meta_data is created before files are moved, delta_8_8 becomes visible before it's populated. Is that an issue? It's not since txn 8 is not committed. h3. Implementation Notes/Limitations (patch 25) * bucketed/sorted tables are not supported * input files names must be of the form 0_0/0_0_copy_1 - enforced. (HIVE-18125) * Load Data creates a delta_x_x/ that contains new files * Load Data w/Overwrite creates a base_x/ that contains new files * A '_metadata_acid' files is placed in the target directory to indicate it requires special handling on read * The input files must be 'plain' ORC files, i.e. not contain acid metadata columns as would be the case if these files were copied from another Acid table. In the latter case, the ROW_IDs embedded in the data may not make sense in the target table (if it's in a different cluster, for example). Such files may also have a mix of committed and aborted data. ** this could be relaxed later by adding info to the _metadata_acid file to ignore existing ROW_IDs on read. * ROW_IDs are attached dynamically at read time and made permanent by compaction. This is done the same way has handling of files that were written to a table before it was converted to Acid. * Vectorization is supported was: LOAD DATA was not supported since ACID was introduced. Need to fill this gap between ACID table and regular hive table. Current Documentation is under [DML Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] and [Loading files into tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: \\ * Load Data performs very limited validations of the data, in particular it uses the input file name which may not be in 0_0 which can break some read logic. (Certainly will for Acid). * It does not check the schema of the file. This may be a non issue for Acid which requires ORC which is self describing so Schema Evolution may handle this seamlessly. (Assuming Schema is not too different). * It does check that _InputFormat_S are compatible. * Bucketed (and thus sorted) tables don't support Load Data (but only if hive.strict.checks.bucketing=true (default)). Will keep this restriction for Acid. * Load Data supports OVERWRITE clause * What happens to file permissions/ownership: rename vs copy differences \\ The implementation will follow the same idea as in HIVE-14988 and use a base_N/ dir for OVERWRITE clause. \\ How is minor compaction going to handle delta/base with original files? Since delta_8_8/_meta_data is created before files are moved, delta_8_8 becomes visible before it's populated. Is that an issue? It's not since txn 8 is not committed. h3. Implementation Notes/Limitations (patch 25) * bucketed/sorted tables are not supported * input files names must be of the form 0_0/0_0_copy_1 - enforced. (HIVE-18125) * Load Data creates a delta_x_x/ that contains new files * Load Data w/Overwrite creates a base_x/ that contains new files * The input files must be 'plain' ORC files, i.e. not contain acid metadata columns as would be the case if these files were copied from another Acid table. In the latter case, the ROW_IDs embedded in the data may not make sense in the target table (if it's in a different cluster, for example). Such files may also have a mix of committed and aborted data
[jira] [Commented] (HIVE-17980) Move HiveMetaStoreClient plus a few remaining classes.
[
https://issues.apache.org/jira/browse/HIVE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269428#comment-16269428
]
Hive QA commented on HIVE-17980:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
27s{color} | {color:red} standalone-metastore: The patch generated 1146 new +
448 unchanged - 6 fixed = 1594 total (was 454) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} metastore: The patch generated 5 new + 11 unchanged -
1177 fixed = 16 total (was 1188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch accumulo-handler passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch druid-handler passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hbase-handler: The patch generated 1 new + 43
unchanged - 0 fixed = 44 total (was 43) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hcatalog/core: The patch generated 1 new + 197
unchanged - 2 fixed = 198 total (was 199) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch server-extensions passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hcatalog/streaming: The patch generated 0 new + 65
unchanged - 1 fixed = 65 total (was 66) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} hcatalog/webhcat/java-client: The patch generated 1
new + 65 unchanged - 0 fixed = 66 total (was 65) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch svr passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hive-blobstore passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hive-minikdc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch hive-unit-hadoop2 passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch qtest passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} The patch qtest-accumulo passed checkstyle {color} |
|
[jira] [Updated] (HIVE-18142) Data corruption can cause SerializationUtils.readRemainingLongs() function hang
[
https://issues.apache.org/jira/browse/HIVE-18142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dustin updated HIVE-18142:
--
Description:
Similar to the SerializationUtils.readLongBE() function reported in
[HIVE-13397|https://issues.apache.org/jira/browse/HIVE-13397], when Instream is
corrupted, the following loop can become infinite, too.
{code:java}
private void readRemainingLongs(long[] buffer, int offset, InStream input,
int remainder,
int numBytes) throws IOException {
final int toRead = remainder * numBytes;
// bulk read to buffer
int bytesRead = input.read(readBuffer, 0, toRead);
while (bytesRead != toRead) {
bytesRead += input.read(readBuffer, bytesRead, toRead - bytesRead);
}
...
}
{code}
was:
Similar to the SerializationUtils.readLongBE() function reported in
[HIVE-13255|https://issues.apache.org/jira/browse/HIVE-13255], when Instream is
corrupted, the following loop can become infinite, too.
{code:java}
private void readRemainingLongs(long[] buffer, int offset, InStream input,
int remainder,
int numBytes) throws IOException {
final int toRead = remainder * numBytes;
// bulk read to buffer
int bytesRead = input.read(readBuffer, 0, toRead);
while (bytesRead != toRead) {
bytesRead += input.read(readBuffer, bytesRead, toRead - bytesRead);
}
...
}
{code}
> Data corruption can cause SerializationUtils.readRemainingLongs() function
> hang
> ---
>
> Key: HIVE-18142
> URL: https://issues.apache.org/jira/browse/HIVE-18142
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Dustin
> Fix For: 2.0.1, 2.1.0
>
>
> Similar to the SerializationUtils.readLongBE() function reported in
> [HIVE-13397|https://issues.apache.org/jira/browse/HIVE-13397], when Instream
> is corrupted, the following loop can become infinite, too.
> {code:java}
> private void readRemainingLongs(long[] buffer, int offset, InStream input,
> int remainder,
> int numBytes) throws IOException {
> final int toRead = remainder * numBytes;
> // bulk read to buffer
> int bytesRead = input.read(readBuffer, 0, toRead);
> while (bytesRead != toRead) {
> bytesRead += input.read(readBuffer, bytesRead, toRead - bytesRead);
> }
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17980) Move HiveMetaStoreClient plus a few remaining classes.
[
https://issues.apache.org/jira/browse/HIVE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269431#comment-16269431
]
Hive QA commented on HIVE-17980:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899651/HIVE-17980.2.patch
{color:green}SUCCESS:{color} +1 due to 29 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11413 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=47)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=157)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=229)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=229)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8027/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8027/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8027/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899651 - PreCommit-HIVE-Build
> Move HiveMetaStoreClient plus a few remaining classes.
> --
>
> Key: HIVE-17980
> URL: https://issues.apache.org/jira/browse/HIVE-17980
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Labels: pull-request-available
> Attachments: HIVE-17980.2.patch, HIVE-17980.patch
>
>
> This moves over HiveMetaStoreClient a few more remaining things that haven't
> been moved yet. It also cleans up the mess I made of MetaStoreUtils. I
> renamed the class in metastore to HiveMetaStoreUtils and left a few methods
> in there that could be moved. Everything else has been moved to
> standalone-metastore/.../MetaStoreUtils.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18160) Jar localization during session initialization is slow
[
https://issues.apache.org/jira/browse/HIVE-18160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269438#comment-16269438
]
Sergey Shelukhin commented on HIVE-18160:
-
Some minor comments on RB
> Jar localization during session initialization is slow
> --
>
> Key: HIVE-18160
> URL: https://issues.apache.org/jira/browse/HIVE-18160
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18160.1.patch
>
>
> Same Jar getting localized multiple times resulting in SHA256 computation
> several times causes slow session initialization time.
> {code}
> 2017-11-28T00:40:55,795 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,105 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> e20986f3a422f8fa5eb61c5a2756cd6f7d2b779dbcab49eae6f2c8dfff7ad2a2 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-llap-tez-3.0.0-SNAPSHOT.jar
> of length: 109.53KB in 1 ms
> 2017-11-28T00:40:56,353 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 231 ms
> 2017-11-28T00:40:56,602 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,612 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 686d66b825fdc4fc241e0591e7646a1bbca1c7114a7224c41da7f4795cf9477a for file:
> file:/work/hadoop/hadoop/hadoop-dist/target/hadoop-2.9.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-registry-2.9.0-SNAPSHOT.jar
> of length: 122.72KB in 2 ms
> {code}
> From above logs, sha256 is computed 3 times for hive-exec jar and each
> invocation takes around 240ms.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18005) Improve size estimation for array() to be not 0
[
https://issues.apache.org/jira/browse/HIVE-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269461#comment-16269461
]
Hive QA commented on HIVE-18005:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
9s{color} | {color:green} storage-api generated 0 new + 2 unchanged - 1 fixed =
2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} storage-api: The patch generated 0 new + 35
unchanged - 4 fixed = 35 total (was 39) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 1 new + 93 unchanged - 0 fixed
= 94 total (was 93) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 5add3a7 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8028/yetus/diff-checkstyle-ql.txt
|
| modules | C: storage-api ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8028/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve size estimation for array() to be not 0
> ---
>
> Key: HIVE-18005
> URL: https://issues.apache.org/jira/browse/HIVE-18005
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18005.01.patch, HIVE-18005.01.patch
>
>
> happens only in case the array is not from a column; and the array contains
> no column references
> {code}
> EXPLAIN
> SELECT sort_array(array("b", "d", "c", "a")),array("1","2") FROM t
> ...
> Statistics: Num rows: 1 Data size: 0 Basic stats: COMPLETE
> Column stats: COMPLETE
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[ https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-18147: --- Attachment: HIVE-18147-branch-2.01.patch > Tests can fail with java.net.BindException: Address already in use > -- > > Key: HIVE-18147 > URL: https://issues.apache.org/jira/browse/HIVE-18147 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-18147-branch-2.01.patch, HIVE-18147.1.patch, > HIVE-18147.2.patch, HIVE-18147.3.patch, HIVE-18147.4.patch > > > Between the time a new port is picked up to start HMS or HS2, and the time > when HMS or HS2 is started, the port can get used by some other service > resulting in test failure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18134) some alter resource plan fixes
[ https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269486#comment-16269486 ] Prasanth Jayachandran commented on HIVE-18134: -- For ordering, adding SORT_QUERY_RESULTS to q file should help? looks good otherwise, +1 > some alter resource plan fixes > -- > > Key: HIVE-18134 > URL: https://issues.apache.org/jira/browse/HIVE-18134 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18134.01.patch, HIVE-18134.02.patch, > HIVE-18134.patch > > > Part of HIVE-18075 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18160) Jar localization during session initialization is slow
[
https://issues.apache.org/jira/browse/HIVE-18160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269514#comment-16269514
]
Sergey Shelukhin commented on HIVE-18160:
-
+1 pending tests
> Jar localization during session initialization is slow
> --
>
> Key: HIVE-18160
> URL: https://issues.apache.org/jira/browse/HIVE-18160
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18160.1.patch, HIVE-18160.2.patch
>
>
> Same Jar getting localized multiple times resulting in SHA256 computation
> several times causes slow session initialization time.
> {code}
> 2017-11-28T00:40:55,795 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,105 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> e20986f3a422f8fa5eb61c5a2756cd6f7d2b779dbcab49eae6f2c8dfff7ad2a2 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-llap-tez-3.0.0-SNAPSHOT.jar
> of length: 109.53KB in 1 ms
> 2017-11-28T00:40:56,353 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 231 ms
> 2017-11-28T00:40:56,602 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,612 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 686d66b825fdc4fc241e0591e7646a1bbca1c7114a7224c41da7f4795cf9477a for file:
> file:/work/hadoop/hadoop/hadoop-dist/target/hadoop-2.9.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-registry-2.9.0-SNAPSHOT.jar
> of length: 122.72KB in 2 ms
> {code}
> From above logs, sha256 is computed 3 times for hive-exec jar and each
> invocation takes around 240ms.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18160) Jar localization during session initialization is slow
[
https://issues.apache.org/jira/browse/HIVE-18160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18160:
-
Attachment: HIVE-18160.2.patch
Addressed review comments.
> Jar localization during session initialization is slow
> --
>
> Key: HIVE-18160
> URL: https://issues.apache.org/jira/browse/HIVE-18160
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18160.1.patch, HIVE-18160.2.patch
>
>
> Same Jar getting localized multiple times resulting in SHA256 computation
> several times causes slow session initialization time.
> {code}
> 2017-11-28T00:40:55,795 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,105 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> e20986f3a422f8fa5eb61c5a2756cd6f7d2b779dbcab49eae6f2c8dfff7ad2a2 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-llap-tez-3.0.0-SNAPSHOT.jar
> of length: 109.53KB in 1 ms
> 2017-11-28T00:40:56,353 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 231 ms
> 2017-11-28T00:40:56,602 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 55aa783d2dda0599fb89a37daae2a2efebf0eed0d4f6e99e3ce140d2fa2f0c30 for file:
> file:/work/hive/hive-git/packaging/target/apache-hive-3.0.0-SNAPSHOT-bin/apache-hive-3.0.0-SNAPSHOT-bin/lib/hive-exec-3.0.0-SNAPSHOT.jar
> of length: 35.68MB in 241 ms
> 2017-11-28T00:40:56,612 INFO [main]: tez.TezSessionState
> (TezSessionState.java:createJarLocalResource(716)) - Computed sha:
> 686d66b825fdc4fc241e0591e7646a1bbca1c7114a7224c41da7f4795cf9477a for file:
> file:/work/hadoop/hadoop/hadoop-dist/target/hadoop-2.9.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-registry-2.9.0-SNAPSHOT.jar
> of length: 122.72KB in 2 ms
> {code}
> From above logs, sha256 is computed 3 times for hive-exec jar and each
> invocation takes around 240ms.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills
[ https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269522#comment-16269522 ] Prasanth Jayachandran commented on HIVE-18076: -- +1. optional comment in RB. > killquery doesn't actually work for non-trigger WM kills > > > Key: HIVE-18076 > URL: https://issues.apache.org/jira/browse/HIVE-18076 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18076.01.patch, HIVE-18076.02.patch, > HIVE-18076.03.patch, HIVE-18076.patch > > > Not sure what's wrong with it, need to take a look. > It dumps a lot of info about everything being cancelled, instead of a nice > message like triggers do. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18005) Improve size estimation for array() to be not 0
[
https://issues.apache.org/jira/browse/HIVE-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269525#comment-16269525
]
Hive QA commented on HIVE-18005:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899659/HIVE-18005.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11415 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[select_dummy_source]
(batchId=244)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_onview2]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_onview]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[select_dummy_source]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_array] (batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_sort_array]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_split] (batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=157)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.hcatalog.templeton.TestConcurrentJobRequestsThreadsAndTimeout.ConcurrentListJobsTimeOutException
(batchId=186)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8028/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8028/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8028/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899659 - PreCommit-HIVE-Build
> Improve size estimation for array() to be not 0
> ---
>
> Key: HIVE-18005
> URL: https://issues.apache.org/jira/browse/HIVE-18005
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18005.01.patch, HIVE-18005.01.patch
>
>
> happens only in case the array is not from a column; and the array contains
> no column references
> {code}
> EXPLAIN
> SELECT sort_array(array("b", "d", "c", "a")),array("1","2") FROM t
> ...
> Statistics: Num rows: 1 Data size: 0 Basic stats: COMPLETE
> Column stats: COMPLETE
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18170) User mapping not initialized correctly on start
[ https://issues.apache.org/jira/browse/HIVE-18170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-18170: > User mapping not initialized correctly on start > --- > > Key: HIVE-18170 > URL: https://issues.apache.org/jira/browse/HIVE-18170 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > User mapping throws NPE as it is not initialized during HS2 start. > Initial RP is notified in WM c'tor but wm thread has not started yet > resulting in NPE accessing user-pool mapping. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18171) User mapping not initialized correctly on start
[ https://issues.apache.org/jira/browse/HIVE-18171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-18171: > User mapping not initialized correctly on start > --- > > Key: HIVE-18171 > URL: https://issues.apache.org/jira/browse/HIVE-18171 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > User mapping throws NPE as it is not initialized during HS2 start. > Initial RP is notified in WM c'tor but wm thread has not started yet > resulting in NPE accessing user-pool mapping. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HIVE-18171) User mapping not initialized correctly on start
[ https://issues.apache.org/jira/browse/HIVE-18171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran resolved HIVE-18171. -- Resolution: Duplicate Duplicate of HIVE-18170. > User mapping not initialized correctly on start > --- > > Key: HIVE-18171 > URL: https://issues.apache.org/jira/browse/HIVE-18171 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > User mapping throws NPE as it is not initialized during HS2 start. > Initial RP is notified in WM c'tor but wm thread has not started yet > resulting in NPE accessing user-pool mapping. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17900) analyze stats on columns triggered by Compactor generates malformed SQL with > 1 partition column
[
https://issues.apache.org/jira/browse/HIVE-17900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-17900:
-
Assignee: Eugene Koifman
> analyze stats on columns triggered by Compactor generates malformed SQL with
> > 1 partition column
> -
>
> Key: HIVE-17900
> URL: https://issues.apache.org/jira/browse/HIVE-17900
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> {noformat}
> 2017-10-16 09:01:51,255 ERROR [haddl0007.mycenterpointenergy.com-51]:
> ql.Driver (SessionState.java:printError(993)) - FAILED: ParseException line
> 1:70 mismatched input 'dates' expecting ) near ''201608'' in analyze statement
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:70 mismatched input
> 'dates' expecting ) near ''201608'' in analyze statement
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:205)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:438)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:321)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1221)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1158)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1148)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker$StatsUpdater.gatherStats(Worker.java:294)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:265)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:168)
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - start=1508162511253 end=1508162511255 duration=2
> from=org.apache.hadoop.hive.ql.Driver>
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> ql.Driver (Driver.java:compile(559)) - We are resetting the hadoop caller
> context to
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> log.PerfLogger (PerfLogger.java:PerfLogBegin(149)) - method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - method=releaseLocks start=1508162511255 end=1508162511255 duration=0
> from=org.apache.hadoop.hive.ql.Driver>
> 2017-10-16 09:01:51,256 INFO [haddl0007.mycenterpointenergy.com-51]:
> tez.TezSessionPoolManager (TezSessionPoolManager.java:close(183)) - Closing
> tez session default? false
> 2017-10-16 09:01:51,256 INFO [haddl0007.mycenterpointenergy.com-51]:
> tez.TezSessionState (TezSessionState.java:close(294)) - Closing Tez Session
> 2017-10-16 09:01:51,256 INFO [haddl0007.mycenterpointenergy.com-51]:
> client.TezClient (TezClient.java:stop(518)) - Shutting down Tez Session,
> sessionName=HIVE-ae652f03-72c7-4ca8-a2d8-05dcc7392f4f,
> applicationId=application_1507779664083_0159
> 2017-10-16 09:01:51,279 ERROR [haddl0007.mycenterpointenergy.com-51]:
> compactor.Worker (Worker.java:run(191)) - Caught exception while trying to
> compact
> id:3723,dbname:mobiusad,tableName:zces_img_data_small_pt,partName:month=201608/dates=9,state:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking failed to avoid repeated failures, java.io.IOException: Could not
> update stats for table mobiusad.zces_img_data_small_pt/month=201608/dates=9
> due to: (4,FAILED: ParseException line 1:70 mismatched input 'dates'
> expecting ) near ''201608'' in analyze statement,42000line 1:70 mismatched
> input 'dates' expecting ) near ''201608'' in analyze statement)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker$StatsUpdater.gatherStats(Worker.java:296)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:265)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:168)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18172) Make query triggers work with fetch task
[ https://issues.apache.org/jira/browse/HIVE-18172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-18172: > Make query triggers work with fetch task > > > Key: HIVE-18172 > URL: https://issues.apache.org/jira/browse/HIVE-18172 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > Currently triggers work only with counters from cluster tasks. This is to > support some cases where triggers can be applied for fetch task as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18170) User mapping not initialized correctly on start
[ https://issues.apache.org/jira/browse/HIVE-18170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18170: - Attachment: HIVE-18170.1.patch > User mapping not initialized correctly on start > --- > > Key: HIVE-18170 > URL: https://issues.apache.org/jira/browse/HIVE-18170 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-18170.1.patch > > > User mapping throws NPE as it is not initialized during HS2 start. > Initial RP is notified in WM c'tor but wm thread has not started yet > resulting in NPE accessing user-pool mapping. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18170) User mapping not initialized correctly on start
[
https://issues.apache.org/jira/browse/HIVE-18170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18170:
-
Description:
User mapping throws NPE as it is not initialized during HS2 start.
Initial RP is notified in WM c'tor but wm thread has not started yet resulting
in NPE accessing user-pool mapping.
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.tez.WorkloadManager.isManaged(WorkloadManager.java:1866)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.tez.WorkloadManagerFederation.getSession(WorkloadManagerFederation.java:43)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:169)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2230)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1882)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1613)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1358)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1351)
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:252)
~[hive-service-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:91)
~[hive-service-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:344)
~[hive-service-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_121]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_121]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
~[hadoop-common-2.8.1.jar:?]
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:357)
~[hive-service-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_121]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_121]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
{code}
was:
User mapping throws NPE as it is not initialized during HS2 start.
Initial RP is notified in WM c'tor but wm thread has not started yet resulting
in NPE accessing user-pool mapping.
> User mapping not initialized correctly on start
> ---
>
> Key: HIVE-18170
> URL: https://issues.apache.org/jira/browse/HIVE-18170
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18170.1.patch
>
>
> User mapping throws NPE as it is not initialized during HS2 start.
> Initial RP is notified in WM c'tor but wm thread has not started yet
> resulting in NPE accessing user-pool mapping.
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.WorkloadManager.isManaged(WorkloadManager.java:1866)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.WorkloadManagerFederation.getSession(WorkloadManagerFederation.java:43)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:169)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2230)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1882)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1613)
[jira] [Updated] (HIVE-18170) User mapping not initialized correctly on start
[ https://issues.apache.org/jira/browse/HIVE-18170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18170: - Status: Patch Available (was: Open) > User mapping not initialized correctly on start > --- > > Key: HIVE-18170 > URL: https://issues.apache.org/jira/browse/HIVE-18170 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-18170.1.patch > > > User mapping throws NPE as it is not initialized during HS2 start. > Initial RP is notified in WM c'tor but wm thread has not started yet > resulting in NPE accessing user-pool mapping. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.
[
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269613#comment-16269613
]
Aihua Xu commented on HIVE-14792:
-
Avro table does suffer with lots of repeated reading on the same schema file.
+1.
> AvroSerde reads the remote schema-file at least once per mapper, per table
> reference.
> -
>
> Key: HIVE-14792
> URL: https://issues.apache.org/jira/browse/HIVE-14792
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-14792.1.patch
>
>
> Avro tables that use "external" schema files stored on HDFS can cause
> excessive calls to {{FileSystem::open()}}, especially for queries that spawn
> large numbers of mappers.
> This is because of the following code in {{AvroSerDe::initialize()}}:
> {code:title=AvroSerDe.java|borderStyle=solid}
> public void initialize(Configuration configuration, Properties properties)
> throws SerDeException {
> // ...
> if (hasExternalSchema(properties)
> || columnNameProperty == null || columnNameProperty.isEmpty()
> || columnTypeProperty == null || columnTypeProperty.isEmpty()) {
> schema = determineSchemaOrReturnErrorSchema(configuration, properties);
> } else {
> // Get column names and sort order
> columnNames = Arrays.asList(columnNameProperty.split(","));
> columnTypes =
> TypeInfoUtils.getTypeInfosFromTypeString(columnTypeProperty);
> schema = getSchemaFromCols(properties, columnNames, columnTypes,
> columnCommentProperty);
>
> properties.setProperty(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName(),
> schema.toString());
> }
> // ...
> }
> {code}
> For tables using {{avro.schema.url}}, every time the SerDe is initialized
> (i.e. at least once per mapper), the schema file is read remotely. For
> queries with thousands of mappers, this leads to a stampede to the handful
> (3?) datanodes that host the schema-file. In the best case, this causes
> slowdowns.
> It would be preferable to distribute the Avro-schema to all mappers as part
> of the job-conf. The alternatives aren't exactly appealing:
> # One can't rely solely on the {{column.list.types}} stored in the Hive
> metastore. (HIVE-14789).
> # {{avro.schema.literal}} might not always be usable, because of the
> size-limit on table-parameters. The typical size of the Avro-schema file is
> between 0.5-3MB, in my limited experience. Bumping the max table-parameter
> size isn't a great solution.
> If the {{avro.schema.file}} were read during query-planning, and made
> available as part of table-properties (but not serialized into the
> metastore), the downstream logic will remain largely intact. I have a patch
> that does this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269611#comment-16269611
]
Hive QA commented on HIVE-17856:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
34s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} common: The patch generated 2 new + 8 unchanged - 1
fixed = 10 total (was 9) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
50s{color} | {color:red} ql: The patch generated 41 new + 1756 unchanged - 7
fixed = 1797 total (was 1763) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
3s{color} | {color:red} root: The patch generated 43 new + 1764 unchanged - 8
fixed = 1807 total (was 1772) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 6 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 5add3a7 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8029/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8029/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8029/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8029/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8029/yetus/whitespace-tabs.txt
|
| modules | C: common ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8029/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.pa
[jira] [Resolved] (HIVE-18151) LLAP external client: Better error message propagation during submission failures
[ https://issues.apache.org/jira/browse/HIVE-18151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere resolved HIVE-18151. --- Resolution: Fixed Fix Version/s: 3.0.0 Committed to master > LLAP external client: Better error message propagation during submission > failures > - > > Key: HIVE-18151 > URL: https://issues.apache.org/jira/browse/HIVE-18151 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: 3.0.0 > > Attachments: HIVE-18151.1.patch > > > During failed submissions, the original error message should be added to the > error message that is eventually propagated to the user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269645#comment-16269645
]
Hive QA commented on HIVE-17856:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899665/HIVE-17856.16.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11423 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_25]
(batchId=86)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=157)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementParallel
(batchId=228)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8029/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8029/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8029/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899665 - PreCommit-HIVE-Build
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch,
> HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch,
> HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.16.patch,
> HIVE-17856.2.patch, HIVE-17856.3.patch, HIVE-17856.4.patch,
> HIVE-17856.5.patch, HIVE-17856.6.patch, HIVE-17856.7.patch,
> HIVE-17856.8.patch, HIVE-17856.9.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_m
[jira] [Resolved] (HIVE-18020) print per query workload management trace after query execution
[ https://issues.apache.org/jira/browse/HIVE-18020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-18020. - Resolution: Duplicate > print per query workload management trace after query execution > --- > > Key: HIVE-18020 > URL: https://issues.apache.org/jira/browse/HIVE-18020 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran > > For debugging and testing purposes, print some summary (pool, cluster > fraction traces) post query execution to console. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18152) Idempotent state change for resource plan
[
https://issues.apache.org/jira/browse/HIVE-18152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269712#comment-16269712
]
Hive QA commented on HIVE-18152:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 8d39a08 |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8030/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Idempotent state change for resource plan
> -
>
> Key: HIVE-18152
> URL: https://issues.apache.org/jira/browse/HIVE-18152
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18152.1.patch
>
>
> {code}
> show resource plans;
> +--+--++
> | rp_name | status | query_parallelism |
> +--+--++
> | llap | ACTIVE | 1 |
> | global | ENABLED | 1 |
> +--+--++
> ALTER RESOURCE PLAN llap ACTIVATE;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot get a resource plan to
> apply (state=08S01,code=1)
> {code}
> It is better not to throw an error when current state is same as the altered
> state.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills
[ https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18076: Attachment: HIVE-18076.04.patch Updated. > killquery doesn't actually work for non-trigger WM kills > > > Key: HIVE-18076 > URL: https://issues.apache.org/jira/browse/HIVE-18076 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18076.01.patch, HIVE-18076.02.patch, > HIVE-18076.03.patch, HIVE-18076.04.patch, HIVE-18076.patch > > > Not sure what's wrong with it, need to take a look. > It dumps a lot of info about everything being cancelled, instead of a nice > message like triggers do. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18134) some alter resource plan fixes
[ https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18134: Attachment: HIVE-18134.03.patch Updated with SORT_QUERY_RESULTS. > some alter resource plan fixes > -- > > Key: HIVE-18134 > URL: https://issues.apache.org/jira/browse/HIVE-18134 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18134.01.patch, HIVE-18134.02.patch, > HIVE-18134.03.patch, HIVE-18134.patch > > > Part of HIVE-18075 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18170) User mapping not initialized correctly on start
[
https://issues.apache.org/jira/browse/HIVE-18170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18170:
-
Attachment: HIVE-18170.2.patch
[~sershe] can you please take a look? small patch
> User mapping not initialized correctly on start
> ---
>
> Key: HIVE-18170
> URL: https://issues.apache.org/jira/browse/HIVE-18170
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18170.1.patch, HIVE-18170.2.patch
>
>
> User mapping throws NPE as it is not initialized during HS2 start.
> Initial RP is notified in WM c'tor but wm thread has not started yet
> resulting in NPE accessing user-pool mapping.
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.WorkloadManager.isManaged(WorkloadManager.java:1866)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.WorkloadManagerFederation.getSession(WorkloadManagerFederation.java:43)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:169)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2230)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1882)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1613)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1358)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1351)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:252)
> ~[hive-service-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:91)
> ~[hive-service-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:344)
> ~[hive-service-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> ~[hadoop-common-2.8.1.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:357)
> ~[hive-service-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_121]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17900) analyze stats on columns triggered by Compactor generates malformed SQL with > 1 partition column
[
https://issues.apache.org/jira/browse/HIVE-17900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-17900:
--
Status: Patch Available (was: Open)
> analyze stats on columns triggered by Compactor generates malformed SQL with
> > 1 partition column
> -
>
> Key: HIVE-17900
> URL: https://issues.apache.org/jira/browse/HIVE-17900
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-17900.01.patch
>
>
> {noformat}
> 2017-10-16 09:01:51,255 ERROR [haddl0007.mycenterpointenergy.com-51]:
> ql.Driver (SessionState.java:printError(993)) - FAILED: ParseException line
> 1:70 mismatched input 'dates' expecting ) near ''201608'' in analyze statement
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:70 mismatched input
> 'dates' expecting ) near ''201608'' in analyze statement
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:205)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:438)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:321)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1221)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1158)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1148)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker$StatsUpdater.gatherStats(Worker.java:294)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:265)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:168)
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - start=1508162511253 end=1508162511255 duration=2
> from=org.apache.hadoop.hive.ql.Driver>
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> ql.Driver (Driver.java:compile(559)) - We are resetting the hadoop caller
> context to
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> log.PerfLogger (PerfLogger.java:PerfLogBegin(149)) - method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - method=releaseLocks start=1508162511255 end=1508162511255 duration=0
> from=org.apache.hadoop.hive.ql.Driver>
> 2017-10-16 09:01:51,256 INFO [haddl0007.mycenterpointenergy.com-51]:
> tez.TezSessionPoolManager (TezSessionPoolManager.java:close(183)) - Closing
> tez session default? false
> 2017-10-16 09:01:51,256 INFO [haddl0007.mycenterpointenergy.com-51]:
> tez.TezSessionState (TezSessionState.java:close(294)) - Closing Tez Session
> 2017-10-16 09:01:51,256 INFO [haddl0007.mycenterpointenergy.com-51]:
> client.TezClient (TezClient.java:stop(518)) - Shutting down Tez Session,
> sessionName=HIVE-ae652f03-72c7-4ca8-a2d8-05dcc7392f4f,
> applicationId=application_1507779664083_0159
> 2017-10-16 09:01:51,279 ERROR [haddl0007.mycenterpointenergy.com-51]:
> compactor.Worker (Worker.java:run(191)) - Caught exception while trying to
> compact
> id:3723,dbname:mobiusad,tableName:zces_img_data_small_pt,partName:month=201608/dates=9,state:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking failed to avoid repeated failures, java.io.IOException: Could not
> update stats for table mobiusad.zces_img_data_small_pt/month=201608/dates=9
> due to: (4,FAILED: ParseException line 1:70 mismatched input 'dates'
> expecting ) near ''201608'' in analyze statement,42000line 1:70 mismatched
> input 'dates' expecting ) near ''201608'' in analyze statement)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker$StatsUpdater.gatherStats(Worker.java:296)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:265)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:168)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
