[jira] [Commented] (HIVE-20025) Clean-up of event files created by HiveProtoLoggingHook.
[
https://issues.apache.org/jira/browse/HIVE-20025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532357#comment-16532357
]
Hive QA commented on HIVE-20025:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12930097/HIVE-20025.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12368/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12368/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12368/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12930097/HIVE-20025.04.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12930097 - PreCommit-HIVE-Build
> Clean-up of event files created by HiveProtoLoggingHook.
>
>
> Key: HIVE-20025
> URL: https://issues.apache.org/jira/browse/HIVE-20025
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: Hive, hooks, pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20025.01.patch, HIVE-20025.02.patch,
> HIVE-20025.03.patch, HIVE-20025.04.patch
>
>
> Currently, HiveProtoLoggingHook write event data to hdfs. The number of files
> can grow to very large numbers.
> Since the files are created under a folder with Date being a part of the
> path, hive should have a way to clean up data older than a certain configured
> time / date. This can be a job that can run with as little frequency as just
> once a day.
> This time should be set to 1 week default. There should also be a sane upper
> bound of # of files so that when a large cluster generates a lot of files
> during a spike, we don't force the cluster fall over.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20076) Delete on a partitioned table removes more rows than expected
[ https://issues.apache.org/jira/browse/HIVE-20076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532352#comment-16532352 ] Teddy Choi commented on HIVE-20076: --- The second patch includes test data and better fix for RecordReaderImpl. > Delete on a partitioned table removes more rows than expected > - > > Key: HIVE-20076 > URL: https://issues.apache.org/jira/browse/HIVE-20076 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-20076.2.patch, HIVE-20076.patch > > > Delete on a partitioned table removes more rows than expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20025) Clean-up of event files created by HiveProtoLoggingHook.
[
https://issues.apache.org/jira/browse/HIVE-20025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532351#comment-16532351
]
Hive QA commented on HIVE-20025:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12930097/HIVE-20025.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14637 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12367/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12367/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12367/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12930097 - PreCommit-HIVE-Build
> Clean-up of event files created by HiveProtoLoggingHook.
>
>
> Key: HIVE-20025
> URL: https://issues.apache.org/jira/browse/HIVE-20025
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: Hive, hooks, pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20025.01.patch, HIVE-20025.02.patch,
> HIVE-20025.03.patch, HIVE-20025.04.patch
>
>
> Currently, HiveProtoLoggingHook write event data to hdfs. The number of files
> can grow to very large numbers.
> Since the files are created under a folder with Date being a part of the
> path, hive should have a way to clean up data older than a certain configured
> time / date. This can be a job that can run with as little frequency as just
> once a day.
> This time should be set to 1 week default. There should also be a sane upper
> bound of # of files so that when a large cluster generates a lot of files
> during a spike, we don't force the cluster fall over.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20076) Delete on a partitioned table removes more rows than expected
[ https://issues.apache.org/jira/browse/HIVE-20076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20076: -- Attachment: HIVE-20076.2.patch > Delete on a partitioned table removes more rows than expected > - > > Key: HIVE-20076 > URL: https://issues.apache.org/jira/browse/HIVE-20076 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-20076.2.patch, HIVE-20076.patch > > > Delete on a partitioned table removes more rows than expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20076) Delete on a partitioned table removes more rows than expected
[ https://issues.apache.org/jira/browse/HIVE-20076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20076: -- Attachment: (was: BUG-105962.2.patch) > Delete on a partitioned table removes more rows than expected > - > > Key: HIVE-20076 > URL: https://issues.apache.org/jira/browse/HIVE-20076 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-20076.patch > > > Delete on a partitioned table removes more rows than expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20076) Delete on a partitioned table removes more rows than expected
[ https://issues.apache.org/jira/browse/HIVE-20076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20076: -- Attachment: BUG-105962.2.patch > Delete on a partitioned table removes more rows than expected > - > > Key: HIVE-20076 > URL: https://issues.apache.org/jira/browse/HIVE-20076 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Attachments: BUG-105962.2.patch, HIVE-20076.patch > > > Delete on a partitioned table removes more rows than expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20025) Clean-up of event files created by HiveProtoLoggingHook.
[
https://issues.apache.org/jira/browse/HIVE-20025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532320#comment-16532320
]
Hive QA commented on HIVE-20025:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
3s{color} | {color:blue} standalone-metastore in master has 228 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
58s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} standalone-metastore: The patch generated 1 new + 75
unchanged - 0 fixed = 76 total (was 75) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} metastore: The patch generated 4 new + 0 unchanged - 0
fixed = 4 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 11 new + 0
unchanged - 0 fixed = 11 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12367/dev-support/hive-personality.sh
|
| git revision | master / e972122 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12367/yetus/diff-checkstyle-standalone-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12367/yetus/diff-checkstyle-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12367/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: common standalone-metastore metastore ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12367/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean-up of event files created by HiveProto
[jira] [Commented] (HIVE-20072) Write access being requested when performing select on a table
[
https://issues.apache.org/jira/browse/HIVE-20072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532316#comment-16532316
]
mahesh kumar behera commented on HIVE-20072:
[~thejas] [~vgumashta]
Can you please review the patch
> Write access being requested when performing select on a table
> --
>
> Key: HIVE-20072
> URL: https://issues.apache.org/jira/browse/HIVE-20072
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Critical
> Attachments: HIVE-20072.01.patch
>
>
> hive.exec.pre.hooks=org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec
> along with hive doAs=true
> which is causing to perform Alter table operation internally when accessed
> Hive table, and results in HiveException with Write access to table directory.
> Select access is provided to a table and when performing select * for that
> table, below exception is seen.
>
> Steps to reproduce. (with ranger enabled)
> 1. Create test table
> create table lastaccesstime (id int);
> 2. Enable Hive doAs
> 3. update
> hive.exec.pre.hooks=org.apache.hadoop.hive.ql.hooks.ATSHook,org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec
> 4. Perform select on table lastaccesstime as non-hive user and exception will
> be thrown.
> {code:java}
> 0: jdbc:hive2://c470-node4.squadron-labs.com:> select * from lastaccesstime;
> Error: Error while processing statement: FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.metadata.HiveException(Unable to alter table.
> java.security.AccessControlException: Permission denied: user=kafka,
> access=WRITE,
> inode="/apps/hive/warehouse/testdb.db/lastaccesstime":hive:hadoop:drwxr-x---
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:252)
> at
> org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:427)
> at
> org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:303)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1908)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAccess(FSNamesystem.java:8800)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.checkAccess(NameNodeRpcServer.java:2089)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.checkAccess(ClientNamenodeProtocolServerSideTranslatorPB.java:1466)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
> ) (state=08S01,code=12)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19938) Upgrade scripts for information schema
[ https://issues.apache.org/jira/browse/HIVE-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-19938: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 4.0.0 3.1.0 Status: Resolved (was: Patch Available) Thanks [~kgyrtkirk] fixing the test failure. Patch pushed to master/branch-3/branch-3.1. > Upgrade scripts for information schema > -- > > Key: HIVE-19938 > URL: https://issues.apache.org/jira/browse/HIVE-19938 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19938.1.patch, HIVE-19938.2.patch, > HIVE-19938.3.patch, HIVE-19938.4.patch, HIVE-19938.5.patch, > HIVE-19938.6.patch, HIVE-19938.7.patch, HIVE-19938.8.patch, > HIVE-19938.8.patch, HIVE-19938.9.patch > > > To make schematool -upgradeSchema work for information schema. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19938) Upgrade scripts for information schema
[
https://issues.apache.org/jira/browse/HIVE-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532271#comment-16532271
]
Hive QA commented on HIVE-19938:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12930082/HIVE-19938.9.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12366/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12366/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12366/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12930082/HIVE-19938.9.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12930082 - PreCommit-HIVE-Build
> Upgrade scripts for information schema
> --
>
> Key: HIVE-19938
> URL: https://issues.apache.org/jira/browse/HIVE-19938
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19938.1.patch, HIVE-19938.2.patch,
> HIVE-19938.3.patch, HIVE-19938.4.patch, HIVE-19938.5.patch,
> HIVE-19938.6.patch, HIVE-19938.7.patch, HIVE-19938.8.patch,
> HIVE-19938.8.patch, HIVE-19938.9.patch
>
>
> To make schematool -upgradeSchema work for information schema.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20049) hive.groupby.limit.extrastep should be false by default
[
https://issues.apache.org/jira/browse/HIVE-20049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532270#comment-16532270
]
Hive QA commented on HIVE-20049:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12930083/HIVE-20049.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 79 failed/errored test(s), 14636 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_select]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_SortUnionTransposeRule]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_input26] (batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[char_union1] (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop_1]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_colname]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby1_limit]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input11_limit]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input14_limit]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input1_limit]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input25] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input26] (batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input4_limit]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_part10] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_into3]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_into4]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_into5]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_into6]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_noalias]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_onview]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit_pushdown_negative]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part14]
(batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[merge4] (batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonreserved_keywords_insert_into1]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_avro_partition_uniontype]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_mixed_partition_formats2]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[script_pipe] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[select_column_pruning]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udtf_explode]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union3] (batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_25]
(batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[varchar_union1]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_simple]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_varchar_simple]
(batchId=80)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_masking]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_nullscan]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_dpp]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ctas]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lateral_view]
(batchId=175)
org.apache.hadoop.hive.
[jira] [Commented] (HIVE-20049) hive.groupby.limit.extrastep should be false by default
[
https://issues.apache.org/jira/browse/HIVE-20049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532230#comment-16532230
]
Hive QA commented on HIVE-20049:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12365/dev-support/hive-personality.sh
|
| git revision | master / e972122 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12365/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> hive.groupby.limit.extrastep should be false by default
> ---
>
> Key: HIVE-20049
> URL: https://issues.apache.org/jira/browse/HIVE-20049
> Project: Hive
> Issue Type: Task
> Components: Query Planning
>Reporter: Ashutosh Chauhan
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-20049.01.patch, HIVE-20049.02.patch,
> HIVE-20049.03.patch
>
>
> In fact this flag is not needed since FetchTask can enforce limit there is
> never a reason to have another vertex purely for limit.
>
> It was introduced by HIVE-12963
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20025) Clean-up of event files created by HiveProtoLoggingHook.
[ https://issues.apache.org/jira/browse/HIVE-20025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532229#comment-16532229 ] anishek commented on HIVE-20025: +1 > Clean-up of event files created by HiveProtoLoggingHook. > > > Key: HIVE-20025 > URL: https://issues.apache.org/jira/browse/HIVE-20025 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Hive, hooks, pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20025.01.patch, HIVE-20025.02.patch, > HIVE-20025.03.patch, HIVE-20025.04.patch > > > Currently, HiveProtoLoggingHook write event data to hdfs. The number of files > can grow to very large numbers. > Since the files are created under a folder with Date being a part of the > path, hive should have a way to clean up data older than a certain configured > time / date. This can be a job that can run with as little frequency as just > once a day. > This time should be set to 1 week default. There should also be a sane upper > bound of # of files so that when a large cluster generates a lot of files > during a spike, we don't force the cluster fall over. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20041) ResultsCache: Improve loggng for concurrent queries
[
https://issues.apache.org/jira/browse/HIVE-20041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532213#comment-16532213
]
Hive QA commented on HIVE-20041:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929974/HIVE-20041.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14636 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testDynamicPartitionsMerge2
(batchId=306)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12364/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12364/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12364/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929974 - PreCommit-HIVE-Build
> ResultsCache: Improve loggng for concurrent queries
> ---
>
> Key: HIVE-20041
> URL: https://issues.apache.org/jira/browse/HIVE-20041
> Project: Hive
> Issue Type: Improvement
> Components: Diagnosability
>Reporter: Gopal V
>Assignee: Laszlo Bodor
>Priority: Minor
> Attachments: HIVE-20041.01.patch, HIVE-20041.02.patch
>
>
> The logging for QueryResultsCache ends up printing information without
> context, like
> {code}
> 2018-06-30T17:48:45,502 INFO [HiveServer2-Background-Pool: Thread-166]
> results.QueryResultsCache: Waiting on pending cacheEntry
> {code}
> {code}
> 2018-06-30T17:50:17,963 INFO [HiveServer2-Background-Pool: Thread-145]
> ql.Driver: savedToCache: true
> {code}
> The previous lines for this are in DEBUG level, so the logging ends up being
> useless at INFO level to debug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20001) With doas set to true, running select query as hrt_qa user on external table fails due to permission denied to read /warehouse/tablespace/managed directory.
[
https://issues.apache.org/jira/browse/HIVE-20001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M updated HIVE-20001:
---
Attachment: HIVE-20001.4.patch
Status: Patch Available (was: In Progress)
> With doas set to true, running select query as hrt_qa user on external table
> fails due to permission denied to read /warehouse/tablespace/managed
> directory.
>
>
> Key: HIVE-20001
> URL: https://issues.apache.org/jira/browse/HIVE-20001
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20001.1.patch, HIVE-20001.1.patch,
> HIVE-20001.2.patch, HIVE-20001.3.patch, HIVE-20001.4.patch
>
>
> Hive: With doas set to true, running select query as hrt_qa user on external
> table fails due to permission denied to read /warehouse/tablespace/managed
> directory.
> Steps:
> 1. Create a external table.
> 2. Set doas to true.
> 3. run select count(*) using user hrt_qa.
> Table creation query.
> {code}
> beeline -n hrt_qa -p pwd -u
> "jdbc:hive2://ctr-e138-1518143905142-375925-01-06.hwx.site:2181,ctr-e138-1518143905142-375925-01-05.hwx.site:2181,ctr-e138-1518143905142-375925-01-07.hwx.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit"
> --outputformat=tsv -e "drop table if exists test_table purge;
> create external table test_table(id int, age int) row format delimited fields
> terminated by '|' stored as textfile;
> load data inpath '/tmp/table1.dat' overwrite into table test_table;
> {code}
> select count(*) query execution fails
> {code}
> beeline -n hrt_qa -p pwd -u
> "jdbc:hive2://ctr-e138-1518143905142-375925-01-06.hwx.site:2181,ctr-e138-1518143905142-375925-01-05.hwx.site:2181,ctr-e138-1518143905142-375925-01-07.hwx.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit"
> --outputformat=tsv -e "select count(*) from test_table where age>30 and
> id<10100;"
> 2018-06-22 10:22:29,328|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|SLF4J: Class path contains
> multiple SLF4J bindings.
> 2018-06-22 10:22:29,330|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|SLF4J: See
> http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> 2018-06-22 10:22:29,335|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|SLF4J: Actual binding is of
> type [org.apache.logging.slf4j.Log4jLoggerFactory]
> 2018-06-22 10:22:31,408|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|Format tsv is deprecated,
> please use tsv2
> 2018-06-22 10:22:31,529|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|Connecting to
> jdbc:hive2://ctr-e138-1518143905142-375925-01-06.hwx.site:2181,ctr-e138-1518143905142-375925-01-05.hwx.site:2181,ctr-e138-1518143905142-375925-01-07.hwx.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit
> 2018-06-22 10:22:32,031|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|18/06/22 10:22:32 [main]:
> INFO jdbc.HiveConnection: Connected to
> ctr-e138-1518143905142-375925-01-04.hwx.site:10001
> 2018-06-22 10:22:34,130|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|18/06/22 10:22:34 [main]:
> WARN jdbc.HiveConnection: Failed to connect to
> ctr-e138-1518143905142-375925-01-04.hwx.site:10001
> 2018-06-22 10:22:34,244|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|18/06/22 10:22:34 [main]:
> WARN jdbc.HiveConnection: Could not open client transport with JDBC Uri:
> jdbc:hive2://ctr-e138-1518143905142-375925-01-04.hwx.site:10001/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit:
> Failed to open new session:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:java.security.AccessControlExc
[jira] [Updated] (HIVE-20001) With doas set to true, running select query as hrt_qa user on external table fails due to permission denied to read /warehouse/tablespace/managed directory.
[
https://issues.apache.org/jira/browse/HIVE-20001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M updated HIVE-20001:
---
Status: In Progress (was: Patch Available)
> With doas set to true, running select query as hrt_qa user on external table
> fails due to permission denied to read /warehouse/tablespace/managed
> directory.
>
>
> Key: HIVE-20001
> URL: https://issues.apache.org/jira/browse/HIVE-20001
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20001.1.patch, HIVE-20001.1.patch,
> HIVE-20001.2.patch, HIVE-20001.3.patch, HIVE-20001.4.patch
>
>
> Hive: With doas set to true, running select query as hrt_qa user on external
> table fails due to permission denied to read /warehouse/tablespace/managed
> directory.
> Steps:
> 1. Create a external table.
> 2. Set doas to true.
> 3. run select count(*) using user hrt_qa.
> Table creation query.
> {code}
> beeline -n hrt_qa -p pwd -u
> "jdbc:hive2://ctr-e138-1518143905142-375925-01-06.hwx.site:2181,ctr-e138-1518143905142-375925-01-05.hwx.site:2181,ctr-e138-1518143905142-375925-01-07.hwx.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit"
> --outputformat=tsv -e "drop table if exists test_table purge;
> create external table test_table(id int, age int) row format delimited fields
> terminated by '|' stored as textfile;
> load data inpath '/tmp/table1.dat' overwrite into table test_table;
> {code}
> select count(*) query execution fails
> {code}
> beeline -n hrt_qa -p pwd -u
> "jdbc:hive2://ctr-e138-1518143905142-375925-01-06.hwx.site:2181,ctr-e138-1518143905142-375925-01-05.hwx.site:2181,ctr-e138-1518143905142-375925-01-07.hwx.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit"
> --outputformat=tsv -e "select count(*) from test_table where age>30 and
> id<10100;"
> 2018-06-22 10:22:29,328|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|SLF4J: Class path contains
> multiple SLF4J bindings.
> 2018-06-22 10:22:29,330|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|SLF4J: See
> http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> 2018-06-22 10:22:29,335|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|SLF4J: Actual binding is of
> type [org.apache.logging.slf4j.Log4jLoggerFactory]
> 2018-06-22 10:22:31,408|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|Format tsv is deprecated,
> please use tsv2
> 2018-06-22 10:22:31,529|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|Connecting to
> jdbc:hive2://ctr-e138-1518143905142-375925-01-06.hwx.site:2181,ctr-e138-1518143905142-375925-01-05.hwx.site:2181,ctr-e138-1518143905142-375925-01-07.hwx.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit
> 2018-06-22 10:22:32,031|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|18/06/22 10:22:32 [main]:
> INFO jdbc.HiveConnection: Connected to
> ctr-e138-1518143905142-375925-01-04.hwx.site:10001
> 2018-06-22 10:22:34,130|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|18/06/22 10:22:34 [main]:
> WARN jdbc.HiveConnection: Failed to connect to
> ctr-e138-1518143905142-375925-01-04.hwx.site:10001
> 2018-06-22 10:22:34,244|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|18/06/22 10:22:34 [main]:
> WARN jdbc.HiveConnection: Could not open client transport with JDBC Uri:
> jdbc:hive2://ctr-e138-1518143905142-375925-01-04.hwx.site:10001/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit:
> Failed to open new session:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:java.security.AccessControlException: Permission
> denied: user=hrt_
[jira] [Commented] (HIVE-20076) Delete on a partitioned table removes more rows than expected
[ https://issues.apache.org/jira/browse/HIVE-20076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532200#comment-16532200 ] Teddy Choi commented on HIVE-20076: --- Explanation: org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.nextBatch needs to update rowInBatch value even if it uses the fast path. However, I still need to check whether it makes side effects or not. > Delete on a partitioned table removes more rows than expected > - > > Key: HIVE-20076 > URL: https://issues.apache.org/jira/browse/HIVE-20076 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-20076.patch > > > Delete on a partitioned table removes more rows than expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20076) Delete on a partitioned table removes more rows than expected
[ https://issues.apache.org/jira/browse/HIVE-20076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532198#comment-16532198 ] Teddy Choi commented on HIVE-20076: --- I guess that it's alllowed to. I will make the patch with the test data set I have. > Delete on a partitioned table removes more rows than expected > - > > Key: HIVE-20076 > URL: https://issues.apache.org/jira/browse/HIVE-20076 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-20076.patch > > > Delete on a partitioned table removes more rows than expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20076) Delete on a partitioned table removes more rows than expected
[ https://issues.apache.org/jira/browse/HIVE-20076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532197#comment-16532197 ] Teddy Choi commented on HIVE-20076: --- I have some test data, but I was not sure whether it's allowed to share. So I will find other data set which already is in Hive. It will make the difference more clear. Thanks for feedback, [~sershe]. > Delete on a partitioned table removes more rows than expected > - > > Key: HIVE-20076 > URL: https://issues.apache.org/jira/browse/HIVE-20076 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-20076.patch > > > Delete on a partitioned table removes more rows than expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532189#comment-16532189
]
Ganesha Shreedhara commented on HIVE-19850:
---
Can someone review this patch please?
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.0.0
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-19850.patch
>
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> set hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
>
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>
> CREATE VIEW `view1` AS select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1` union all select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2`;
> CREATE VIEW `view2` AS select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2` union all select
> `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
> from `t3`;
> create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union
> all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1
> v1 join view2 v2 on v1.dt=v2.dt;
> CREATE VIEW `view3` AS select
> `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
> from `t4` union all select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1`;
>
> select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work
> found for tablescan error
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19937) Intern JobConf objects in Spark tasks
[
https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532187#comment-16532187
]
Misha Dmitriev commented on HIVE-19937:
---
Thank you for sharing the jxray report, [~stakiar].
If it reflects the situation in real-life applications accurately enough, then
it looks like the sources of duplicate strings are not as much {{JobConf}}
tables as various other things, that you can easily see if you expand the
"Expensive fields" and "Full reference chains" in section 7:
# Most of the duplicate strings (~9% out of 13.5% total) come from data fields
of {{java.net.URI}}. All these URIs, in turn, come from
{{org.apache.hadoop.fs.Path.uri}}. {{Path}}s come from more than one sources,
but the biggest one is this reference chain:
{code:java}
↖java.net.URI.schemeSpecificPart
↖org.apache.hadoop.fs.Path.uri
↖{j.u.LinkedHashMap}.keys
↖org.apache.hadoop.hive.ql.plan.MapWork.pathToAliases{code}
It turns out that in the past I have already taken care of interning strings in
such URIs, see e.g. this method in MapWork.java:
{code:java}
public void setPathToAliases(final LinkedHashMap>
pathToAliases) {
for (Path p : pathToAliases.keySet()) {
StringInternUtils.internUriStringsInPath(p);
}
this.pathToAliases = pathToAliases;
}{code}
but it turns out that there are also other methods that can add {{Path}}s to
{{pathToAliases}}: two flavors of {{addPathToAlias()}} and maybe something
else. I think we need to modify all these methods so that they also call
{{StringInternUtils.internUriStringsInPath()}} for {{Path}}s that are passed to
them. This will remove the said 9% of duplicate strings.
# One other source of duplicate strings in URIs referenced by {{Path}}s is the
map in {{ProjectionPusher.pathToPartitionInfo}}. I think this would be fixed if
in the following line in this class
{code:java}
pathToPartitionInfo.put(Path.getPathWithoutSchemeAndAuthority(entry.getKey()),
...{code}
you insert the {{StringInternUtils.internUriStringsInPath()}} call.
# The very first line in the "Full reference chains" says that 2% of memory is
wasted by duplicate strings that are values in {{CopyOnFirstWriteProperties}}
tables, that are reachable via this reference chain
{code:java}
org.apache.hadoop.hive.common.CopyOnFirstWriteProperties.table
↖org.apache.hadoop.hive.ql.plan.PartitionDesc.properties
↖{j.u.LinkedHashMap}.values
↖org.apache.hadoop.hive.ql.plan.MapWork.pathToPartitionInfo{code}
This is a bit unexpected, given that, as you noticed before, we already take
care of interning this table's values in {{PartitionDesc#internProperties.
}}{{}}Probably some uninterned string values are later added to this table,
probably by the code that obtains this table by calling {{getProperties()}}. I
hope with your knowledge of Hive code you will manage to determine the culprit
here. One more clue is the contents of the duplicate strings coming from these
tables, e.g.
||*Num strings* || *String value* ||
|
36|"hdfs://vc0501.halxg.cloudera.com:8020/user/systest/tpcds_1000_decimal_parquet/store_sales/ss_sold_date_sk=2452497"|
|
36|"hdfs://vc0501.halxg.cloudera.com:8020/user/systest/tpcds_1000_decimal_parquet/store_sales/ss_sold_date_sk=2452422"|
# There are several other sources of duplicate strings that jxray reports.
They cause much less overhead, but some may be still worth fixing. Let me know
if you need help with them. Interestingly, as far as I can see, strings coming
from {{JobConf}} waste just about 0.2% of memory.
Also, as far as I can see in section 2, {{java.util.Properties}} objects
together consume 8.5% of memory, which is significant. But most of that comes
from {{TableDesc#properties}}. {{JobConf#properties}} uses just 0.8% of memory,
so probably not worth optimizing.
> Intern JobConf objects in Spark tasks
> -
>
> Key: HIVE-19937
> URL: https://issues.apache.org/jira/browse/HIVE-19937
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19937.1.patch, report.html
>
>
> When fixing HIVE-16395, we decided that each new Spark task should clone the
> {{JobConf}} object to prevent any {{ConcurrentModificationException}} from
> being thrown. However, setting this variable comes at a cost of storing a
> duplicate {{JobConf}} object for each Spark task. These objects can take up a
> significant amount of memory, we should intern them so that Spark tasks
> running in the same JVM don't store duplicate copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20082) HiveDecimal to string conversion doesn't format the decimal correctly - master
[ https://issues.apache.org/jira/browse/HIVE-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20082: Description: Example: LPAD on a decimal(7,1) values of 0 returns "0" (plus padding) but it should be "0.0" (plus padding) (was: Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be "0.0") > HiveDecimal to string conversion doesn't format the decimal correctly - master > -- > > Key: HIVE-20082 > URL: https://issues.apache.org/jira/browse/HIVE-20082 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Example: LPAD on a decimal(7,1) values of 0 returns "0" (plus padding) but it > should be "0.0" (plus padding) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20083) HiveDecimal to string conversion doesn't format the decimal correctly - branch-1
[ https://issues.apache.org/jira/browse/HIVE-20083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20083: --- Assignee: Jason Dere > HiveDecimal to string conversion doesn't format the decimal correctly - > branch-1 > > > Key: HIVE-20083 > URL: https://issues.apache.org/jira/browse/HIVE-20083 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Jason Dere >Priority: Major > > Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be > "0.0". > HiveDecimal impl is significantly different between branch-1 and master. This > is for the pre-FastDecimal fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20083) HiveDecimal to string conversion doesn't format the decimal correctly - branch-1
[ https://issues.apache.org/jira/browse/HIVE-20083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20083: Description: Example: LPAD on a decimal(7,1) values of 0 returns "0" (plus padding) but it should be "0.0" (plus padding) HiveDecimal impl is significantly different between branch-1 and master. This is for the pre-FastDecimal fix. was: Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be "0.0". HiveDecimal impl is significantly different between branch-1 and master. This is for the pre-FastDecimal fix. > HiveDecimal to string conversion doesn't format the decimal correctly - > branch-1 > > > Key: HIVE-20083 > URL: https://issues.apache.org/jira/browse/HIVE-20083 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Jason Dere >Priority: Major > > Example: LPAD on a decimal(7,1) values of 0 returns "0" (plus padding) but it > should be "0.0" (plus padding) > HiveDecimal impl is significantly different between branch-1 and master. This > is for the pre-FastDecimal fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20083) HiveDecimal to string conversion doesn't format the decimal correctly - branch-1
[ https://issues.apache.org/jira/browse/HIVE-20083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532181#comment-16532181 ] Sergey Shelukhin commented on HIVE-20083: - cc [~jdere] [~ashutoshc] I might take a brief look to see if it's just a simple fix. > HiveDecimal to string conversion doesn't format the decimal correctly - > branch-1 > > > Key: HIVE-20083 > URL: https://issues.apache.org/jira/browse/HIVE-20083 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Jason Dere >Priority: Major > > Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be > "0.0". > HiveDecimal impl is significantly different between branch-1 and master. This > is for the pre-FastDecimal fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20083) HiveDecimal to string conversion doesn't format the decimal correctly - branch-1
[ https://issues.apache.org/jira/browse/HIVE-20083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20083: Summary: HiveDecimal to string conversion doesn't format the decimal correctly - branch-1 (was: HiveDecimal to string conversion doesn't format the decimal correctly - branch-2, branch-1) > HiveDecimal to string conversion doesn't format the decimal correctly - > branch-1 > > > Key: HIVE-20083 > URL: https://issues.apache.org/jira/browse/HIVE-20083 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be > "0.0". > HiveDecimal impl is significantly different between branch-1/2 and master. > This is for the pre-FastDecimal fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20083) HiveDecimal to string conversion doesn't format the decimal correctly - branch-1
[ https://issues.apache.org/jira/browse/HIVE-20083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20083: Description: Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be "0.0". HiveDecimal impl is significantly different between branch-1 and master. This is for the pre-FastDecimal fix. was: Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be "0.0". HiveDecimal impl is significantly different between branch-1/2 and master. This is for the pre-FastDecimal fix. > HiveDecimal to string conversion doesn't format the decimal correctly - > branch-1 > > > Key: HIVE-20083 > URL: https://issues.apache.org/jira/browse/HIVE-20083 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be > "0.0". > HiveDecimal impl is significantly different between branch-1 and master. This > is for the pre-FastDecimal fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20083) HiveDecimal to string conversion doesn't format the decimal correctly - branch-2
[ https://issues.apache.org/jira/browse/HIVE-20083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20083: Description: Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be "0.0". HiveDecimal impl is significantly different between branch-1/2 and master. This is for a branch-2 fix. was: Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be "0.0". HiveDecimal is different between branch-2 and master. This is for a branch-2 fix. > HiveDecimal to string conversion doesn't format the decimal correctly - > branch-2 > > > Key: HIVE-20083 > URL: https://issues.apache.org/jira/browse/HIVE-20083 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be > "0.0". > HiveDecimal impl is significantly different between branch-1/2 and master. > This is for a branch-2 fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20083) HiveDecimal to string conversion doesn't format the decimal correctly - branch-2, branch-1
[ https://issues.apache.org/jira/browse/HIVE-20083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20083: Description: Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be "0.0". HiveDecimal impl is significantly different between branch-1/2 and master. This is for the pre-FastDecimal fix. was: Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be "0.0". HiveDecimal impl is significantly different between branch-1/2 and master. This is for a branch-2 fix. > HiveDecimal to string conversion doesn't format the decimal correctly - > branch-2, branch-1 > -- > > Key: HIVE-20083 > URL: https://issues.apache.org/jira/browse/HIVE-20083 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be > "0.0". > HiveDecimal impl is significantly different between branch-1/2 and master. > This is for the pre-FastDecimal fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20083) HiveDecimal to string conversion doesn't format the decimal correctly - branch-2, branch-1
[ https://issues.apache.org/jira/browse/HIVE-20083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20083: Summary: HiveDecimal to string conversion doesn't format the decimal correctly - branch-2, branch-1 (was: HiveDecimal to string conversion doesn't format the decimal correctly - branch-2) > HiveDecimal to string conversion doesn't format the decimal correctly - > branch-2, branch-1 > -- > > Key: HIVE-20083 > URL: https://issues.apache.org/jira/browse/HIVE-20083 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be > "0.0". > HiveDecimal impl is significantly different between branch-1/2 and master. > This is for a branch-2 fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20083) HiveDecimal to string conversion doesn't format the decimal correctly - branch-2
[ https://issues.apache.org/jira/browse/HIVE-20083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20083: Description: Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be "0.0". HiveDecimal is different between branch-2 and master. This is for a branch-2 fix. was:Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be "0.0" > HiveDecimal to string conversion doesn't format the decimal correctly - > branch-2 > > > Key: HIVE-20083 > URL: https://issues.apache.org/jira/browse/HIVE-20083 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Example: LPAD on a decimal(7,1) values of 0 returns "0" but it should be > "0.0". > HiveDecimal is different between branch-2 and master. This is for a branch-2 > fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20041) ResultsCache: Improve loggng for concurrent queries
[
https://issues.apache.org/jira/browse/HIVE-20041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532168#comment-16532168
]
Hive QA commented on HIVE-20041:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
55s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12364/dev-support/hive-personality.sh
|
| git revision | master / e972122 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12364/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ResultsCache: Improve loggng for concurrent queries
> ---
>
> Key: HIVE-20041
> URL: https://issues.apache.org/jira/browse/HIVE-20041
> Project: Hive
> Issue Type: Improvement
> Components: Diagnosability
>Reporter: Gopal V
>Assignee: Laszlo Bodor
>Priority: Minor
> Attachments: HIVE-20041.01.patch, HIVE-20041.02.patch
>
>
> The logging for QueryResultsCache ends up printing information without
> context, like
> {code}
> 2018-06-30T17:48:45,502 INFO [HiveServer2-Background-Pool: Thread-166]
> results.QueryResultsCache: Waiting on pending cacheEntry
> {code}
> {code}
> 2018-06-30T17:50:17,963 INFO [HiveServer2-Background-Pool: Thread-145]
> ql.Driver: savedToCache: true
> {code}
> The previous lines for this are in DEBUG level, so the logging ends up being
> useless at INFO level to debug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[ https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532163#comment-16532163 ] Junjie Chen commented on HIVE-17593: It's my fault, will update llap side as well. > DataWritableWriter strip spaces for CHAR type before writing, but predicate > generator doesn't do same thing. > > > Key: HIVE-17593 > URL: https://issues.apache.org/jira/browse/HIVE-17593 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 3.0.0 >Reporter: Junjie Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-17593.2.patch, HIVE-17593.3.patch, HIVE-17593.patch > > > DataWritableWriter strip spaces for CHAR type before writing. While when > generating predicate, it does NOT do same striping which should cause data > missing! > In current version, it doesn't cause data missing since predicate is not well > push down to parquet due to HIVE-17261. > Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as > same which will build a predicate with tail spaces. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19820) add ACID stats support to background stats updater
[ https://issues.apache.org/jira/browse/HIVE-19820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532161#comment-16532161 ] Sergey Shelukhin commented on HIVE-19820: - Backup for the patch; not done yet. The problem now without txn list is that stats cannot be written without write ID, yet that is what analyze does. So in the paradigm where we treat stats as data for ACID, we'd need to allocate write ID for analyze. cc [~ekoifman] > add ACID stats support to background stats updater > -- > > Key: HIVE-19820 > URL: https://issues.apache.org/jira/browse/HIVE-19820 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19820.01-master-txnstats.patch, > HIVE-19820.02-master-txnstats.patch, HIVE-19820.03-master-txnstats.patch, > HIVE-19820.04-master-txnstats.patch > > > Follow-up from HIVE-19418. > Right now it checks whether stats are valid in an old-fashioned way... and > also gets ACID state, and discards it without using. > When ACID stats are implemented, ACID state needs to be used to do > version-aware valid stats checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19820) add ACID stats support to background stats updater
[ https://issues.apache.org/jira/browse/HIVE-19820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19820: Attachment: HIVE-19820.04-master-txnstats.patch > add ACID stats support to background stats updater > -- > > Key: HIVE-19820 > URL: https://issues.apache.org/jira/browse/HIVE-19820 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19820.01-master-txnstats.patch, > HIVE-19820.02-master-txnstats.patch, HIVE-19820.03-master-txnstats.patch, > HIVE-19820.04-master-txnstats.patch > > > Follow-up from HIVE-19418. > Right now it checks whether stats are valid in an old-fashioned way... and > also gets ACID state, and discards it without using. > When ACID stats are implemented, ACID state needs to be used to do > version-aware valid stats checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[ https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532156#comment-16532156 ] Ferdinand Xu commented on HIVE-17593: - My fault. You should update LLAP's output file. See "org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet_types] (batchId=172)" > DataWritableWriter strip spaces for CHAR type before writing, but predicate > generator doesn't do same thing. > > > Key: HIVE-17593 > URL: https://issues.apache.org/jira/browse/HIVE-17593 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 3.0.0 >Reporter: Junjie Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-17593.2.patch, HIVE-17593.3.patch, HIVE-17593.patch > > > DataWritableWriter strip spaces for CHAR type before writing. While when > generating predicate, it does NOT do same striping which should cause data > missing! > In current version, it doesn't cause data missing since predicate is not well > push down to parquet due to HIVE-17261. > Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as > same which will build a predicate with tail spaces. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[ https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532153#comment-16532153 ] Ferdinand Xu edited comment on HIVE-17593 at 7/4/18 2:09 AM: - I see. The test report is a little bit out of date since your patch should already updated the output file. And could you also update Tez and LLAP output file as well? LGTM +1 pending on the refreshed test report. was (Author: ferd): I see. The test report is a little bit out of date since your patch should already updated the output file. LGTM +1 pending on the refreshed test report. > DataWritableWriter strip spaces for CHAR type before writing, but predicate > generator doesn't do same thing. > > > Key: HIVE-17593 > URL: https://issues.apache.org/jira/browse/HIVE-17593 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 3.0.0 >Reporter: Junjie Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-17593.2.patch, HIVE-17593.3.patch, HIVE-17593.patch > > > DataWritableWriter strip spaces for CHAR type before writing. While when > generating predicate, it does NOT do same striping which should cause data > missing! > In current version, it doesn't cause data missing since predicate is not well > push down to parquet due to HIVE-17261. > Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as > same which will build a predicate with tail spaces. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[ https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532155#comment-16532155 ] Junjie Chen commented on HIVE-17593: Thanks [~Ferd], I think the last test report already cover my latest HIVE-17593.3.patch. > DataWritableWriter strip spaces for CHAR type before writing, but predicate > generator doesn't do same thing. > > > Key: HIVE-17593 > URL: https://issues.apache.org/jira/browse/HIVE-17593 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 3.0.0 >Reporter: Junjie Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-17593.2.patch, HIVE-17593.3.patch, HIVE-17593.patch > > > DataWritableWriter strip spaces for CHAR type before writing. While when > generating predicate, it does NOT do same striping which should cause data > missing! > In current version, it doesn't cause data missing since predicate is not well > push down to parquet due to HIVE-17261. > Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as > same which will build a predicate with tail spaces. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[ https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532153#comment-16532153 ] Ferdinand Xu commented on HIVE-17593: - I see. The test report is a little bit out of date since your patch should already updated the output file. LGTM +1 pending on the refreshed test report. > DataWritableWriter strip spaces for CHAR type before writing, but predicate > generator doesn't do same thing. > > > Key: HIVE-17593 > URL: https://issues.apache.org/jira/browse/HIVE-17593 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 3.0.0 >Reporter: Junjie Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-17593.2.patch, HIVE-17593.3.patch, HIVE-17593.patch > > > DataWritableWriter strip spaces for CHAR type before writing. While when > generating predicate, it does NOT do same striping which should cause data > missing! > In current version, it doesn't cause data missing since predicate is not well > push down to parquet due to HIVE-17261. > Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as > same which will build a predicate with tail spaces. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19938) Upgrade scripts for information schema
[
https://issues.apache.org/jira/browse/HIVE-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532151#comment-16532151
]
Hive QA commented on HIVE-19938:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12930082/HIVE-19938.9.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14635 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12363/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12363/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12363/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12930082 - PreCommit-HIVE-Build
> Upgrade scripts for information schema
> --
>
> Key: HIVE-19938
> URL: https://issues.apache.org/jira/browse/HIVE-19938
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19938.1.patch, HIVE-19938.2.patch,
> HIVE-19938.3.patch, HIVE-19938.4.patch, HIVE-19938.5.patch,
> HIVE-19938.6.patch, HIVE-19938.7.patch, HIVE-19938.8.patch,
> HIVE-19938.8.patch, HIVE-19938.9.patch
>
>
> To make schematool -upgradeSchema work for information schema.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19821) Distributed HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-19821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532142#comment-16532142 ] Kai Zheng commented on HIVE-19821: -- Hi Sahil, This is a nice proposal. Having the heavy HS2 taking be separated out to container level for isolation and scalability looks promising. Not digging into the details yet, some questions: # In HoS, the spark context resides in separate JVM, now since the main work of HS2 is to be in a container/JVM per session/user, would it be good to combine the two together, consolidating the spark context back into the new HS2 container? For efficiency. # I like the architect picture in [Apache Livy|https://livy.incubator.apache.org/], would be good to have some similar one in the design. # Will this approach affect security, like auth and authorization? # Considerations like backward compatibility and interfaces/tools. Thank you. > Distributed HiveServer2 > --- > > Key: HIVE-19821 > URL: https://issues.apache.org/jira/browse/HIVE-19821 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-19821.1.WIP.patch, HIVE-19821.2.WIP.patch, > HIVE-19821_ Distributed HiveServer2.pdf > > > HS2 deployments often hit OOM issues due to a number of factors: (1) too many > concurrent connections, (2) query that scan a large number of partitions have > to pull a lot of metadata into memory (e.g. a query reading thousands of > partitions requires loading thousands of partitions into memory), (3) very > large queries can take up a lot of heap space, especially during query > parsing. There are a number of other factors that cause HiveServer2 to run > out of memory, these are just some of the more commons ones. > Distributed HS2 proposes to do all query parsing, compilation, planning, and > execution coordination inside a dedicated container. This should > significantly decrease memory pressure on HS2 and allow HS2 to scale to a > larger number of concurrent users. > For HoS (and I think Hive-on-Tez) this just requires moving all query > compilation, planning, etc. inside the application master for the > corresponding Hive session. > The main benefit here is isolation. A poorly written Hive query cannot bring > down an entire HiveServer2 instance and force all other queries to fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[ https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17751: -- Attachment: HIVE-17751.10.patch > Separate HMS Client and HMS server into separate sub-modules > > > Key: HIVE-17751 > URL: https://issues.apache.org/jira/browse/HIVE-17751 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-17751.01.patch, HIVE-17751.02.patch, > HIVE-17751.03.patch, HIVE-17751.04.patch, > HIVE-17751.06-standalone-metastore.patch, HIVE-17751.07.patch, > HIVE-17751.08.patch, HIVE-17751.09.patch, HIVE-17751.10.patch > > > external applications which are interfacing with HMS should ideally only > include HMSClient library instead of one big library containing server as > well. We should ideally have a thin client library so that cross version > support for external applications is easier. We should sub-divide the > standalone module into possibly 3 modules (one for common classes, one for > client classes and one for server) or 2 sub-modules (one for client and one > for server) so that we can generate separate jars for HMS client and server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[ https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532138#comment-16532138 ] Junjie Chen commented on HIVE-17593: [~Ferd], yes, previous qtest result uses stripped value for char type verification, and I change to use padding value according to char definition. > DataWritableWriter strip spaces for CHAR type before writing, but predicate > generator doesn't do same thing. > > > Key: HIVE-17593 > URL: https://issues.apache.org/jira/browse/HIVE-17593 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 3.0.0 >Reporter: Junjie Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-17593.2.patch, HIVE-17593.3.patch, HIVE-17593.patch > > > DataWritableWriter strip spaces for CHAR type before writing. While when > generating predicate, it does NOT do same striping which should cause data > missing! > In current version, it doesn't cause data missing since predicate is not well > push down to parquet due to HIVE-17261. > Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as > same which will build a predicate with tail spaces. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[
https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532135#comment-16532135
]
Alexander Kolbasov commented on HIVE-17751:
---
09.patch is merged with
{code}
* commit e972122e0cad69a0a77110bb5ad11990b7bc5747 (origin/master, origin/HEAD,
master)
| Author: Deepak Jaiswal
| Date: Tue Jul 3 17:22:02 2018 -0700
|
| HIVE-20039 : Bucket pruning: Left Outer Join on bucketed table gives
wrong result (Deepak Jaiswal, reviewed by Gopal V)
|
{code}
> Separate HMS Client and HMS server into separate sub-modules
>
>
> Key: HIVE-17751
> URL: https://issues.apache.org/jira/browse/HIVE-17751
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-17751.01.patch, HIVE-17751.02.patch,
> HIVE-17751.03.patch, HIVE-17751.04.patch,
> HIVE-17751.06-standalone-metastore.patch, HIVE-17751.07.patch,
> HIVE-17751.08.patch, HIVE-17751.09.patch
>
>
> external applications which are interfacing with HMS should ideally only
> include HMSClient library instead of one big library containing server as
> well. We should ideally have a thin client library so that cross version
> support for external applications is easier. We should sub-divide the
> standalone module into possibly 3 modules (one for common classes, one for
> client classes and one for server) or 2 sub-modules (one for client and one
> for server) so that we can generate separate jars for HMS client and server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[ https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17751: -- Attachment: HIVE-17751.09.patch > Separate HMS Client and HMS server into separate sub-modules > > > Key: HIVE-17751 > URL: https://issues.apache.org/jira/browse/HIVE-17751 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-17751.01.patch, HIVE-17751.02.patch, > HIVE-17751.03.patch, HIVE-17751.04.patch, > HIVE-17751.06-standalone-metastore.patch, HIVE-17751.07.patch, > HIVE-17751.08.patch, HIVE-17751.09.patch > > > external applications which are interfacing with HMS should ideally only > include HMSClient library instead of one big library containing server as > well. We should ideally have a thin client library so that cross version > support for external applications is easier. We should sub-divide the > standalone module into possibly 3 modules (one for common classes, one for > client classes and one for server) or 2 sub-modules (one for client and one > for server) so that we can generate separate jars for HMS client and server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[
https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532134#comment-16532134
]
Ferdinand Xu commented on HIVE-17593:
-
Is this diff caused by padding?
{code:java}
214,217c214,217
< a 1 a 3 48.88 1
< ab2 ab 3 8.72 1
< abc 3 abc 3 90.21 1
< abcd 4 abcd 4 3.89 1
---
> a 5 a 3 48.88 1
> ab5 ab 3 8.72 1
> abc 5 abc 3 90.21 1
> abcd 5 abcd 4 3.89 1
555,558c555,558
< a 1 a 3 48.88 1
< ab2 ab 3 8.72 1
< abc 3 abc 3 90.21 1
< abcd 4 abcd 4 3.89 1
---
> a 5 a 3 48.88 1
> ab5 ab 3 8.72 1
> abc 5 abc 3 90.21 1
> abcd 5 abcd 4 3.89 1
{code}
> DataWritableWriter strip spaces for CHAR type before writing, but predicate
> generator doesn't do same thing.
>
>
> Key: HIVE-17593
> URL: https://issues.apache.org/jira/browse/HIVE-17593
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.0, 3.0.0
>Reporter: Junjie Chen
>Assignee: Junjie Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-17593.2.patch, HIVE-17593.3.patch, HIVE-17593.patch
>
>
> DataWritableWriter strip spaces for CHAR type before writing. While when
> generating predicate, it does NOT do same striping which should cause data
> missing!
> In current version, it doesn't cause data missing since predicate is not well
> push down to parquet due to HIVE-17261.
> Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as
> same which will build a predicate with tail spaces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19820) add ACID stats support to background stats updater
[ https://issues.apache.org/jira/browse/HIVE-19820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532123#comment-16532123 ] Sergey Shelukhin commented on HIVE-19820: - [~steveyeom2017] fyi the API change from this patch is going to be undone, it's not needed for stats updater after switching to single write ID. So it shouldn't conflict with CachedStore patch > add ACID stats support to background stats updater > -- > > Key: HIVE-19820 > URL: https://issues.apache.org/jira/browse/HIVE-19820 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19820.01-master-txnstats.patch, > HIVE-19820.02-master-txnstats.patch, HIVE-19820.03-master-txnstats.patch > > > Follow-up from HIVE-19418. > Right now it checks whether stats are valid in an old-fashioned way... and > also gets ACID state, and discards it without using. > When ACID stats are implemented, ACID state needs to be used to do > version-aware valid stats checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19938) Upgrade scripts for information schema
[
https://issues.apache.org/jira/browse/HIVE-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532111#comment-16532111
]
Hive QA commented on HIVE-19938:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
11s{color} | {color:blue} standalone-metastore in master has 228 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} standalone-metastore: The patch generated 1 new + 32
unchanged - 1 fixed = 33 total (was 33) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml findbugs checkstyle
compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12363/dev-support/hive-personality.sh
|
| git revision | master / 19676a3 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12363/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: standalone-metastore metastore ql packaging U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12363/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade scripts for information schema
> --
>
> Key: HIVE-19938
> URL: https://issues.apache.org/jira/browse/HIVE-19938
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19938.1.patch, HIVE-19938.2.patch,
> HIVE-19938.3.patch, HIVE-19938.4.patch, HIVE-19938.5.patch,
> HIVE-19938.6.patch, HIVE-19938.7.patch, HIVE-19938.8.patch,
> HIVE-19938.8.patch, HIVE-19938.9.patch
>
>
> To make schematool -upgradeSchema work for information schema.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[ https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532092#comment-16532092 ] Junjie Chen commented on HIVE-17593: the failed tests are not related. [~Ferd], take a look? > DataWritableWriter strip spaces for CHAR type before writing, but predicate > generator doesn't do same thing. > > > Key: HIVE-17593 > URL: https://issues.apache.org/jira/browse/HIVE-17593 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 3.0.0 >Reporter: Junjie Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-17593.2.patch, HIVE-17593.3.patch, HIVE-17593.patch > > > DataWritableWriter strip spaces for CHAR type before writing. While when > generating predicate, it does NOT do same striping which should cause data > missing! > In current version, it doesn't cause data missing since predicate is not well > push down to parquet due to HIVE-17261. > Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as > same which will build a predicate with tail spaces. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20039: -- Fix Version/s: 3.2.0 4.0.0 > Bucket pruning: Left Outer Join on bucketed table gives wrong result > - > > Key: HIVE-20039 > URL: https://issues.apache.org/jira/browse/HIVE-20039 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 2.3.2 >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20039.01-branch-3.patch, HIVE-20039.1.patch, > HIVE-20039.2.patch, HIVE-20039.3.patch, HIVE-20039.4.patch > > > Left outer join on bucketed table on certain cases gives wrong results. > Depending on the order in which the table-scans are walked through, the > FilterPruner might end up using the wrong table scan's table properties on > the other table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532087#comment-16532087 ] Deepak Jaiswal commented on HIVE-20039: --- Committed to master. Thanks [~gopalv] for the review. Preparing for branch-3. > Bucket pruning: Left Outer Join on bucketed table gives wrong result > - > > Key: HIVE-20039 > URL: https://issues.apache.org/jira/browse/HIVE-20039 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 2.3.2 >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20039.01-branch-3.patch, HIVE-20039.1.patch, > HIVE-20039.2.patch, HIVE-20039.3.patch, HIVE-20039.4.patch > > > Left outer join on bucketed table on certain cases gives wrong results. > Depending on the order in which the table-scans are walked through, the > FilterPruner might end up using the wrong table scan's table properties on > the other table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20039: -- Attachment: HIVE-20039.01-branch-3.patch > Bucket pruning: Left Outer Join on bucketed table gives wrong result > - > > Key: HIVE-20039 > URL: https://issues.apache.org/jira/browse/HIVE-20039 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 2.3.2 >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20039.01-branch-3.patch, HIVE-20039.1.patch, > HIVE-20039.2.patch, HIVE-20039.3.patch, HIVE-20039.4.patch > > > Left outer join on bucketed table on certain cases gives wrong results. > Depending on the order in which the table-scans are walked through, the > FilterPruner might end up using the wrong table scan's table properties on > the other table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18852) Misleading error message in alter table validation
[
https://issues.apache.org/jira/browse/HIVE-18852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18852:
--
Attachment: HIVE-18852.1.patch
Status: Patch Available (was: Open)
> Misleading error message in alter table validation
> --
>
> Key: HIVE-18852
> URL: https://issues.apache.org/jira/browse/HIVE-18852
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.4.0
>Reporter: Dan Burkert
>Assignee: Andrew Sherman
>Priority: Major
> Attachments: HIVE-18852.1.patch
>
>
> The metastore's validation error message when attempting to rename a table to
> a non-existent database is wrong. For instance, attempting to alter table
> 'db.table' to 'non_existent_database.table' results in the Thrift error:
> {{TException - service has thrown: InvalidOperationException(message=Unable
> to change partition or table. Database db does not exist Check metastore logs
> for detailed stack.non_existent_database)}}
> I believe the offending line of code is
> [here|https://github.com/apache/hive/blob/branch-2/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java?utf8=%E2%9C%93#L331-L333],
> notice that {{dbname}} is used in the message, not {{newDbName}}. I don't
> know if switching that would cause the case of a non-existing {{dbname}} case
> to regress, though.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20047) consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20047: --- Assignee: (was: Sergey Shelukhin) > consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20029) add more tests
[ https://issues.apache.org/jira/browse/HIVE-20029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20029: Summary: add more tests (was: add negative test for parallel insert) > add more tests > -- > > Key: HIVE-20029 > URL: https://issues.apache.org/jira/browse/HIVE-20029 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > > We need a few tests, esp. for parallel case, where we verify that stats are > NOT used. > Right now many code paths don't fail but return -1, null or whatever when > something else is not present, so positive tests might pass because they skip > the check, not because the check passes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20029) add more tests
[ https://issues.apache.org/jira/browse/HIVE-20029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20029: Description: 1) We need a few tests, esp. for parallel case, where we verify that stats are NOT used. Right now many code paths don't fail but return -1, null or whatever when something else is not present, so positive tests might pass because they skip the check, not because the check passes. 2) Analyze table needs a test, esp analyze table after parallel insert, and also analyze table after an invalid transaction. was: We need a few tests, esp. for parallel case, where we verify that stats are NOT used. Right now many code paths don't fail but return -1, null or whatever when something else is not present, so positive tests might pass because they skip the check, not because the check passes. > add more tests > -- > > Key: HIVE-20029 > URL: https://issues.apache.org/jira/browse/HIVE-20029 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > > 1) We need a few tests, esp. for parallel case, where we verify that stats > are NOT used. > Right now many code paths don't fail but return -1, null or whatever when > something else is not present, so positive tests might pass because they skip > the check, not because the check passes. > 2) Analyze table needs a test, esp analyze table after parallel insert, and > also analyze table after an invalid transaction. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[
https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532085#comment-16532085
]
Hive QA commented on HIVE-20039:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12930179/HIVE-20039.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14636 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12362/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12362/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12362/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12930179 - PreCommit-HIVE-Build
> Bucket pruning: Left Outer Join on bucketed table gives wrong result
> -
>
> Key: HIVE-20039
> URL: https://issues.apache.org/jira/browse/HIVE-20039
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20039.1.patch, HIVE-20039.2.patch,
> HIVE-20039.3.patch, HIVE-20039.4.patch
>
>
> Left outer join on bucketed table on certain cases gives wrong results.
> Depending on the order in which the table-scans are walked through, the
> FilterPruner might end up using the wrong table scan's table properties on
> the other table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19933) ALTER TABLE DROP PARTITION - Partition Not Found
[
https://issues.apache.org/jira/browse/HIVE-19933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532082#comment-16532082
]
Alice Fan commented on HIVE-19933:
--
Sounds good. Will remove the flag from master branch. Thanks all!
> ALTER TABLE DROP PARTITION - Partition Not Found
>
>
> Key: HIVE-19933
> URL: https://issues.apache.org/jira/browse/HIVE-19933
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.2
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Major
>
> {code:sql}
> ALTER TABLE web_logsz DROP PARTITION (`date`='xyz')
> -- SemanticException [Error 10001]: Table not found web_logsz
> ALTER TABLE web_logs DROP PARTITION (`date`='xyz')
> -- Success.
> {code}
> There is no 'xyz' partition for the 'date' column. To make this more
> consistent, the query should fail if the user tries to drop a partition that
> does not exist
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[
https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532080#comment-16532080
]
Hive QA commented on HIVE-20039:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 11 unchanged - 1 fixed
= 13 total (was 12) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
59s{color} | {color:red} root: The patch generated 2 new + 11 unchanged - 1
fixed = 13 total (was 12) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
59s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100
total (was 100) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 6m
40s{color} | {color:red} root generated 1 new + 370 unchanged - 1 fixed = 371
total (was 371) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12362/dev-support/hive-personality.sh
|
| git revision | master / cc8ac97 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12362/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12362/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12362/yetus/whitespace-eol.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12362/yetus/diff-javadoc-javadoc-ql.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12362/yetus/diff-javadoc-javadoc-root.txt
|
| modules | C: ql . itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12362/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Bucket pruning: Left Outer Join on bucketed table gives wrong result
> -
>
> Key: HIVE-20039
> URL: https://issues.apache.org/jira/browse/HIVE-20039
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.3.2
>
[jira] [Updated] (HIVE-20080) TxnHandler checkLock direct sql fail with ORA-01795 , if the table has more than 1000 partitions
[
https://issues.apache.org/jira/browse/HIVE-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh updated HIVE-20080:
--
Description:
with Oracle as Metastore, txnhandler checkLock fail with
"checkLockWithRetry(181398,34773) : ORA-01795: maximum number of expressions in
a list is 1000" if the write table has more than 1000 partitions.
complete stacktrace
{code}
txn.TxnHandler (TxnHandler.java:checkRetryable(2099)) - Non-retryable error in
checkLockWithRetry(181398,34773) : ORA-01795: maximum number of expressions in
a list is 1000
(SQLState=42000, ErrorCode=1795)
2018-06-25 15:09:35,999 ERROR [pool-7-thread-197]: metastore.RetryingHMSHandler
(RetryingHMSHandler.java:invokeInternal(203)) - MetaException(message:Unable to
update transaction database java.sql.SQLSyntaxErrorException: ORA-01795:
maximum number of expressions in a list is 1000
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:195)
at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:876)
at
oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1175)
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1296)
at
oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1498)
at
oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:406)
at com.jolbox.bonecp.StatementHandle.executeQuery(StatementHandle.java:464)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:2649)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLockWithRetry(TxnHandler.java:1126)
at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:895)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6123)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at com.sun.proxy.$Proxy11.lock(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:12012)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:11996)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:551)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:546)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:546)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLockWithRetry(TxnHandler.java:1131)
at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:895)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6123)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at com.sun.proxy.$Proxy11.lock(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:12012)
[jira] [Updated] (HIVE-20080) TxnHandler checkLock direct sql fail with ORA-01795 , if the table has more than 1000 partitions
[
https://issues.apache.org/jira/browse/HIVE-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh updated HIVE-20080:
--
Description:
with Oracle as Metastore, txnhandler checkLock fail with
"checkLockWithRetry(181398,34773) : ORA-01795: maximum number of expressions in
a list is 1000" if the write table has more than 1000 partitions.
complete stacktrace
{code}
txn.TxnHandler (TxnHandler.java:checkRetryable(2099)) - Non-retryable error in
checkLockWithRetry(181398,34773) : ORA-01795: maximum number of expressions in
a list is 1000
(SQLState=42000, ErrorCode=1795)
2018-06-25 15:09:35,999 ERROR [pool-7-thread-197]: metastore.RetryingHMSHandler
(RetryingHMSHandler.java:invokeInternal(203)) - MetaException(message:Unable to
update transaction database java.sql.SQLSyntaxErrorException: ORA-01795:
maximum number of expressions in a list is 1000
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:195)
at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:876)
at
oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1175)
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1296)
at
oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1498)
at
oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:406)
at com.jolbox.bonecp.StatementHandle.executeQuery(StatementHandle.java:464)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:2649)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLockWithRetry(TxnHandler.java:1126)
at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:895)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6123)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at com.sun.proxy.$Proxy11.lock(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:12012)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:11996)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:551)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:546)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:546)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLockWithRetry(TxnHandler.java:1131)
at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:895)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6123)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at com.sun.proxy.$Proxy11.lock(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:12012)
[jira] [Updated] (HIVE-20079) Populate more accurate rawDataSize for parquet format
[
https://issues.apache.org/jira/browse/HIVE-20079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-20079:
Status: Patch Available (was: Open)
patch-1: in ParquetRecordWriterWrapper, when the writer gets closed, we will
collect and update the stats from the file.
> Populate more accurate rawDataSize for parquet format
> -
>
> Key: HIVE-20079
> URL: https://issues.apache.org/jira/browse/HIVE-20079
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-20079.1.patch
>
>
> Run the following queries and you will see the raw data for the table is 4
> (that is the number of fields) incorrectly. We need to populate correct data
> size so data can be split properly.
> {noformat}
> SET hive.stats.autogather=true;
> CREATE TABLE parquet_stats (id int,str string) STORED AS PARQUET;
> INSERT INTO parquet_stats values(0, 'this is string 0'), (1, 'string 1');
> DESC FORMATTED parquet_stats;
> {noformat}
> {noformat}
> Table Parameters:
> COLUMN_STATS_ACCURATE true
> numFiles1
> numRows 2
> rawDataSize 4
> totalSize 373
> transient_lastDdlTime 1530660523
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20079) Populate more accurate rawDataSize for parquet format
[
https://issues.apache.org/jira/browse/HIVE-20079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-20079:
Attachment: HIVE-20079.1.patch
> Populate more accurate rawDataSize for parquet format
> -
>
> Key: HIVE-20079
> URL: https://issues.apache.org/jira/browse/HIVE-20079
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-20079.1.patch
>
>
> Run the following queries and you will see the raw data for the table is 4
> (that is the number of fields) incorrectly. We need to populate correct data
> size so data can be split properly.
> {noformat}
> SET hive.stats.autogather=true;
> CREATE TABLE parquet_stats (id int,str string) STORED AS PARQUET;
> INSERT INTO parquet_stats values(0, 'this is string 0'), (1, 'string 1');
> DESC FORMATTED parquet_stats;
> {noformat}
> {noformat}
> Table Parameters:
> COLUMN_STATS_ACCURATE true
> numFiles1
> numRows 2
> rawDataSize 4
> totalSize 373
> transient_lastDdlTime 1530660523
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20079) Populate more accurate rawDataSize for parquet format
[
https://issues.apache.org/jira/browse/HIVE-20079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-20079:
---
> Populate more accurate rawDataSize for parquet format
> -
>
> Key: HIVE-20079
> URL: https://issues.apache.org/jira/browse/HIVE-20079
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
>
> Run the following queries and you will see the raw data for the table is 4
> (that is the number of fields) incorrectly. We need to populate correct data
> size so data can be split properly.
> {noformat}
> SET hive.stats.autogather=true;
> CREATE TABLE parquet_stats (id int,str string) STORED AS PARQUET;
> INSERT INTO parquet_stats values(0, 'this is string 0'), (1, 'string 1');
> DESC FORMATTED parquet_stats;
> {noformat}
> {noformat}
> Table Parameters:
> COLUMN_STATS_ACCURATE true
> numFiles1
> numRows 2
> rawDataSize 4
> totalSize 373
> transient_lastDdlTime 1530660523
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20012) Implement SQL Standard Date and Timestamp Functions
[
https://issues.apache.org/jira/browse/HIVE-20012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532063#comment-16532063
]
Bharathkrishna Guruvayoor Murali commented on HIVE-20012:
-
Hi,
I can take a look at this. Do you mind me assigning this JIRA to myself?
> Implement SQL Standard Date and Timestamp Functions
> ---
>
> Key: HIVE-20012
> URL: https://issues.apache.org/jira/browse/HIVE-20012
> Project: Hive
> Issue Type: New Feature
>Reporter: Shawn Weeks
>Priority: Minor
>
> I've looked around and haven't seen an existing ticket on this. Many times
> you need to convert from arbitrary string formats to a date or a timestamp.
> The current method using the unix_timestamp function doesn't support
> milliseconds and is a bit clunky. I propose we implement a to_date and
> to_timestamp function that behave like the following. It may also be useful
> for the to_timestamp function to behave like the existing to_date function
> and convert Hive's default timestamp string into an actual timestamp.
> {code:java}
> select to_date('01-01-2000','dd-MM-');
> 2000-01-01
> select to_timestamp('01-01-2000 13:00:00.000','dd-MM- HH:mm:ss.SSS')
> 2000-01-01 13:00:00.000{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18852) Misleading error message in alter table validation
[
https://issues.apache.org/jira/browse/HIVE-18852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-18852:
-
Assignee: Andrew Sherman
> Misleading error message in alter table validation
> --
>
> Key: HIVE-18852
> URL: https://issues.apache.org/jira/browse/HIVE-18852
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.4.0
>Reporter: Dan Burkert
>Assignee: Andrew Sherman
>Priority: Major
>
> The metastore's validation error message when attempting to rename a table to
> a non-existent database is wrong. For instance, attempting to alter table
> 'db.table' to 'non_existent_database.table' results in the Thrift error:
> {{TException - service has thrown: InvalidOperationException(message=Unable
> to change partition or table. Database db does not exist Check metastore logs
> for detailed stack.non_existent_database)}}
> I believe the offending line of code is
> [here|https://github.com/apache/hive/blob/branch-2/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java?utf8=%E2%9C%93#L331-L333],
> notice that {{dbname}} is used in the message, not {{newDbName}}. I don't
> know if switching that would cause the case of a non-existing {{dbname}} case
> to regress, though.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20062) TestReplicationScenarios doesn't clean injection properly, causing bizarre interdependent failures
[ https://issues.apache.org/jira/browse/HIVE-20062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20062: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! > TestReplicationScenarios doesn't clean injection properly, causing bizarre > interdependent failures > -- > > Key: HIVE-20062 > URL: https://issues.apache.org/jira/browse/HIVE-20062 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20062.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20062) TestReplicationScenarios doesn't clean injection properly, causing bizarre interdependent failures
[
https://issues.apache.org/jira/browse/HIVE-20062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532045#comment-16532045
]
Hive QA commented on HIVE-20062:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12930057/HIVE-20062.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14635 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12360/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12360/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12360/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12930057 - PreCommit-HIVE-Build
> TestReplicationScenarios doesn't clean injection properly, causing bizarre
> interdependent failures
> --
>
> Key: HIVE-20062
> URL: https://issues.apache.org/jira/browse/HIVE-20062
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20062.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20064) Data cannot be inserted into a dynamic partition
[
https://issues.apache.org/jira/browse/HIVE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532031#comment-16532031
]
gavin_shi commented on HIVE-20064:
--
i can not understand that mr is not really supported anymore,I'm just a
beginner in hive
> Data cannot be inserted into a dynamic partition
>
>
> Key: HIVE-20064
> URL: https://issues.apache.org/jira/browse/HIVE-20064
> Project: Hive
> Issue Type: Bug
> Components: SQL
>Reporter: gavin_shi
>Priority: Blocker
>
> Data cannot be inserted into a dynamic partition,The relevant table
> information is as follows：
> The source table:
> {code:java}
> //代码占位符
> create table customer(id string,name string,gender string,state string)
> partitioned by(country string);
> {code}
> The source table data：
>
> {code:java}
> //代码占位符
> +--++--+-+---+--+
> | customer.id | customer.name | customer.gender | customer.state |
> customer.country |
> +--++--+-+---+--+
> | 1 | gavin | 1 | shanghai | china |
> +--++--+-+---+--+
> {code}
>
> The target table:
>
> {code:java}
> //代码占位符
> create table customer_bak(id string,name string,gender string,state string)
> partitioned by(country string);
> {code}
>
> Now I use dynamic techniques to insert data into the customer_bak table：
> {code:java}
> //代码占位符
> set hive.exec.dynamic.partition=true;
> set hive.exec.dynamic.partition.mode=nonstrict;
> insert into table customer_bak partition(country) select
> id,name,gender,state,country from customer;
> {code}
> The program is always executing and neither success nor failure is shown. The
> information in the hive log is as follows:
> {code:java}
> //代码占位符
> 2018-06-28T16:01:22,900 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Executing
> command(queryId=hadoop_20180628160122_f80c5f8e-98a0-4286-bc55-c25efd7a60bc):
> insert into table customer_bak partition(country) select
> id,name,gender,state,country from customer
> 2018-06-28T16:01:22,901 WARN [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. tez,
> spark) or using Hive 1.X releases.
> 2018-06-28T16:01:22,902 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: WARNING: Hive-on-MR is deprecated in Hive 2 and may not be
> available in the future versions. Consider using a different execution engine
> (i.e. tez, spark) or using Hive 1.X releases.
> 2018-06-28T16:01:22,902 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Query ID =
> hadoop_20180628160122_f80c5f8e-98a0-4286-bc55-c25efd7a60bc
> 2018-06-28T16:01:22,903 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Total jobs = 3
> 2018-06-28T16:01:22,904 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Launching Job 1 out of 3
> 2018-06-28T16:01:22,907 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Starting task [Stage-1:MAPRED] in serial mode
> 2018-06-28T16:01:22,908 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: Number of reduce tasks is set to 0 since there's no reduce operator
> 2018-06-28T16:01:22,929 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Context: New scratch dir is
> hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3
> 2018-06-28T16:01:22,942 INFO [HiveServer2-Background-Pool: Thread-97]
> mr.ExecDriver: Using org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
> 2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Processing alias customer
> 2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Adding 2 inputs; the first input is
> hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
> 2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Content Summary not cached for
> hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
> 2018-06-28T16:01:22,945 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Content Summary not cached for
> hdfs://hadoop-cluster/user/hive/warehouse/customer/country=usa
> 2018-06-28T16:01:22,978 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Changed input file
> hdfs://hadoop-cluster/user/hive/warehouse/customer/country=usa to empty file
> hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036
> (false)
[jira] [Commented] (HIVE-20062) TestReplicationScenarios doesn't clean injection properly, causing bizarre interdependent failures
[
https://issues.apache.org/jira/browse/HIVE-20062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532005#comment-16532005
]
Hive QA commented on HIVE-20062:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 6 new + 516
unchanged - 6 fixed = 522 total (was 522) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12360/dev-support/hive-personality.sh
|
| git revision | master / cc8ac97 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12360/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: itests/hive-unit U: itests/hive-unit |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12360/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestReplicationScenarios doesn't clean injection properly, causing bizarre
> interdependent failures
> --
>
> Key: HIVE-20062
> URL: https://issues.apache.org/jira/browse/HIVE-20062
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20062.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19937) Intern JobConf objects in Spark tasks
[
https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531998#comment-16531998
]
Sahil Takiar commented on HIVE-19937:
-
[~mi...@cloudera.com] I've attached a jxray report I created by running TPC-DS
query3 against HoS with {{spark.executor.cores = 40}} (which means 40 threads
per JVM and thus 40 copies of {{JobConf}}). Most of the memory goes towards
byte buffers for Netty and Parquet, but 13.5% of memory is wasted due to string
duplication.
Will look more into how to implement the changes for
{{CopyOnFirstWriteProperties}}.
> Intern JobConf objects in Spark tasks
> -
>
> Key: HIVE-19937
> URL: https://issues.apache.org/jira/browse/HIVE-19937
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19937.1.patch, report.html
>
>
> When fixing HIVE-16395, we decided that each new Spark task should clone the
> {{JobConf}} object to prevent any {{ConcurrentModificationException}} from
> being thrown. However, setting this variable comes at a cost of storing a
> duplicate {{JobConf}} object for each Spark task. These objects can take up a
> significant amount of memory, we should intern them so that Spark tasks
> running in the same JVM don't store duplicate copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[ https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531996#comment-16531996 ] Alexander Kolbasov commented on HIVE-17751: --- The new changes broke this pretty seriously - I need to spend some time to remerge. [~alangates] [~vihangk1] I am running into a serious issue - there are a lot of upcoming changes to metastore that almost always break me. This creates several problems: * By the time my patch reaches ptest there is a conflicting change, so it can never merge correctly * I have to spend huge amount of time merging with upcoming changes * It creates a high risk of mismerges - originally I just moved a lot of files and pieces of files around, now I need to make sure that more recent commits are handled properly which is error prone. Alan - how did you handle similar issues for standalone metastore project? Would it be possible to quiet down the change rate for the metastore? > Separate HMS Client and HMS server into separate sub-modules > > > Key: HIVE-17751 > URL: https://issues.apache.org/jira/browse/HIVE-17751 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-17751.01.patch, HIVE-17751.02.patch, > HIVE-17751.03.patch, HIVE-17751.04.patch, > HIVE-17751.06-standalone-metastore.patch, HIVE-17751.07.patch, > HIVE-17751.08.patch > > > external applications which are interfacing with HMS should ideally only > include HMSClient library instead of one big library containing server as > well. We should ideally have a thin client library so that cross version > support for external applications is easier. We should sub-divide the > standalone module into possibly 3 modules (one for common classes, one for > client classes and one for server) or 2 sub-modules (one for client and one > for server) so that we can generate separate jars for HMS client and server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19937) Intern JobConf objects in Spark tasks
[
https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-19937:
Attachment: report.html
> Intern JobConf objects in Spark tasks
> -
>
> Key: HIVE-19937
> URL: https://issues.apache.org/jira/browse/HIVE-19937
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19937.1.patch, report.html
>
>
> When fixing HIVE-16395, we decided that each new Spark task should clone the
> {{JobConf}} object to prevent any {{ConcurrentModificationException}} from
> being thrown. However, setting this variable comes at a cost of storing a
> duplicate {{JobConf}} object for each Spark task. These objects can take up a
> significant amount of memory, we should intern them so that Spark tasks
> running in the same JVM don't store duplicate copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19940) Push predicates with deterministic UDFs with RBO
[
https://issues.apache.org/jira/browse/HIVE-19940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531988#comment-16531988
]
Naveen Gangam commented on HIVE-19940:
--
[~janulatha] Could you please post the new query plan with the proposed fix as
well? Thank you
> Push predicates with deterministic UDFs with RBO
>
>
> Key: HIVE-19940
> URL: https://issues.apache.org/jira/browse/HIVE-19940
> Project: Hive
> Issue Type: Improvement
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-19940.1.patch, HIVE-19940.2.patch
>
>
> With RBO, predicates with any UDF doesn't get pushed down. It makes sense to
> not pushdown the predicates with non-deterministic function as the meaning of
> the query changes after the predicate is resolved to use the function. But
> pushing a deterministic function is beneficial.
> Test Case:
> {code}
> set hive.cbo.enable=false;
> CREATE TABLE `testb`(
>`cola` string COMMENT '',
>`colb` string COMMENT '',
>`colc` string COMMENT '')
> PARTITIONED BY (
>`part1` string,
>`part2` string,
>`part3` string)
> STORED AS AVRO;
> CREATE TABLE `testa`(
>`col1` string COMMENT '',
>`col2` string COMMENT '',
>`col3` string COMMENT '',
>`col4` string COMMENT '',
>`col5` string COMMENT '')
> PARTITIONED BY (
>`part1` string,
>`part2` string,
>`part3` string)
> STORED AS AVRO;
> insert into testA partition (part1='US', part2='ABC', part3='123')
> values ('12.34', '100', '200', '300', 'abc'),
> ('12.341', '1001', '2001', '3001', 'abcd');
> insert into testA partition (part1='UK', part2='DEF', part3='123')
> values ('12.34', '100', '200', '300', 'abc'),
> ('12.341', '1001', '2001', '3001', 'abcd');
> insert into testA partition (part1='US', part2='DEF', part3='200')
> values ('12.34', '100', '200', '300', 'abc'),
> ('12.341', '1001', '2001', '3001', 'abcd');
> insert into testA partition (part1='CA', part2='ABC', part3='300')
> values ('12.34', '100', '200', '300', 'abc'),
> ('12.341', '1001', '2001', '3001', 'abcd');
> insert into testB partition (part1='CA', part2='ABC', part3='300')
> values ('600', '700', 'abc'), ('601', '701', 'abcd');
> insert into testB partition (part1='CA', part2='ABC', part3='400')
> values ( '600', '700', 'abc'), ( '601', '701', 'abcd');
> insert into testB partition (part1='UK', part2='PQR', part3='500')
> values ('600', '700', 'abc'), ('601', '701', 'abcd');
> insert into testB partition (part1='US', part2='DEF', part3='200')
> values ( '600', '700', 'abc'), ('601', '701', 'abcd');
> insert into testB partition (part1='US', part2='PQR', part3='123')
> values ( '600', '700', 'abc'), ('601', '701', 'abcd');
> -- views with deterministic functions
> create view viewDeterministicUDFA partitioned on (vpart1, vpart2, vpart3) as
> select
> cast(col1 as decimal(38,18)) as vcol1,
> cast(col2 as decimal(38,18)) as vcol2,
> cast(col3 as decimal(38,18)) as vcol3,
> cast(col4 as decimal(38,18)) as vcol4,
> cast(col5 as char(10)) as vcol5,
> cast(part1 as char(2)) as vpart1,
> cast(part2 as char(3)) as vpart2,
> cast(part3 as char(3)) as vpart3
> from testa
> where part1 in ('US', 'CA');
> create view viewDeterministicUDFB partitioned on (vpart1, vpart2, vpart3) as
> select
> cast(cola as decimal(38,18)) as vcolA,
> cast(colb as decimal(38,18)) as vcolB,
> cast(colc as char(10)) as vcolC,
> cast(part1 as char(2)) as vpart1,
> cast(part2 as char(3)) as vpart2,
> cast(part3 as char(3)) as vpart3
> from testb
> where part1 in ('US', 'CA');
> explain
> select vcol1, vcol2, vcol3, vcola, vcolb
> from viewDeterministicUDFA a inner join viewDeterministicUDFB b
> on a.vpart1 = b.vpart1
> and a.vpart2 = b.vpart2
> and a.vpart3 = b.vpart3
> and a.vpart1 = 'US'
> and a.vpart2 = 'DEF'
> and a.vpart3 = '200';
> {code}
> Plan where the CAST is not pushed down.
> {code}
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: testa
> filterExpr: (part1) IN ('US', 'CA') (type: boolean)
> Statistics: Num rows: 6 Data size: 13740 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: CAST( col1 AS decimal(38,18)) (type:
> decimal(38,18)), CAST( col2 AS decimal(38,18)) (type: decimal(38,18)), CAST(
> col3 AS decimal(38,18)) (type: decimal(38,18)), CAST( part1 AS CHAR(2))
> (type: char(2)), CAST( part2 AS CHAR(3)) (type: char(3)), CAST( part3 AS
> CHAR(3)) (type: char(3))
> outputColumnNames: _col0, _col1, _col2, _col5, _col6, _col7
> Statistics: Num rows: 6 Data size: 13740 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: ((_col5 = 'US') and
[jira] [Commented] (HIVE-19267) Replicate ACID/MM tables write operations.
[
https://issues.apache.org/jira/browse/HIVE-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531981#comment-16531981
]
Hive QA commented on HIVE-19267:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929970/HIVE-19267.22.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12359/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12359/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12359/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12929970/HIVE-19267.22.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929970 - PreCommit-HIVE-Build
> Replicate ACID/MM tables write operations.
> --
>
> Key: HIVE-19267
> URL: https://issues.apache.org/jira/browse/HIVE-19267
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: ACID, DR, pull-request-available, replication
> Fix For: 4.0.0
>
> Attachments: HIVE-19267.01-branch-3.patch, HIVE-19267.01.patch,
> HIVE-19267.02.patch, HIVE-19267.03.patch, HIVE-19267.04.patch,
> HIVE-19267.05.patch, HIVE-19267.06.patch, HIVE-19267.07.patch,
> HIVE-19267.08.patch, HIVE-19267.09.patch, HIVE-19267.10.patch,
> HIVE-19267.11.patch, HIVE-19267.12.patch, HIVE-19267.13.patch,
> HIVE-19267.14.patch, HIVE-19267.15.patch, HIVE-19267.16.patch,
> HIVE-19267.17.patch, HIVE-19267.18.patch, HIVE-19267.19.patch,
> HIVE-19267.20.patch, HIVE-19267.21.patch, HIVE-19267.22.patch
>
>
>
> h1. Replicate ACID write Events
> * Create new EVENT_WRITE event with related message format to log the write
> operations with in a txn along with data associated.
> * Log this event when perform any writes (insert into, insert overwrite,
> load table, delete, update, merge, truncate) on table/partition.
> * If a single MERGE/UPDATE/INSERT/DELETE statement operates on multiple
> partitions, then need to log one event per partition.
> * DbNotificationListener should log this type of event to special metastore
> table named "MTxnWriteNotificationLog".
> * This table should maintain a map of txn ID against list of
> tables/partitions written by given txn.
> * The entry for a given txn should be removed by the cleaner thread that
> removes the expired events from EventNotificationTable.
> h1. Replicate Commit Txn operation (with writes)
> Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions
> modified within the txn.
> *Source warehouse:*
> * This event should read the EVENT_WRITEs from "MTxnWriteNotificationLog"
> metastore table to consolidate the list of tables/partitions modified within
> this txn scope.
> * Based on the list of tables/partitions modified and table Write ID, need
> to compute the list of delta files added by this txn.
> * Repl dump should read this message and dump the metadata and delta files
> list.
> *Target warehouse:*
> * Ensure snapshot isolation at target for on-going read txns which shouldn't
> view the data replicated from committed txn. (Ensured with open and allocate
> write ID events).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[
https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531971#comment-16531971
]
Hive QA commented on HIVE-17751:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12930043/HIVE-17751.08.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12358/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12358/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12358/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-07-03 21:45:26.471
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12358/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-07-03 21:45:26.474
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at cc8ac97 HIVE-19860 : HiveServer2 ObjectInspectorFactory memory
leak with cachedUnionStructObjectInspector (Rajkumar Singh)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at cc8ac97 HIVE-19860 : HiveServer2 ObjectInspectorFactory memory
leak with cachedUnionStructObjectInspector (Rajkumar Singh)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-07-03 21:45:27.819
+ rm -rf ../yetus_PreCommit-HIVE-Build-12358
+ mkdir ../yetus_PreCommit-HIVE-Build-12358
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12358
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12358/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: git apply: bad git-diff - inconsistent old filename on line 2741
error: patch failed:
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java:80
Falling back to three-way merge...
Applied patch to
'hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java'
with conflicts.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:37
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java' with
conflicts.
error: patch failed:
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java:86
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java'
cleanly.
error: patch failed:
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java:85
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java'
cleanly.
Going to apply patch with: git apply -p1
error: patch failed:
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java:80
Falling back to three-way merge...
Applied patch to
'hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java'
with conflicts.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:37
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java' with
conflicts.
/data/hiveptest/working/scratch/build.patch:6847: new blank line at EOF.
+
error: patch failed:
standalone-metastore/src/test/java/org/apache/hadoop/hive/metas
[jira] [Commented] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[
https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531967#comment-16531967
]
Hive QA commented on HIVE-17593:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12930050/HIVE-17593.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14635 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=172)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12357/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12357/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12357/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12930050 - PreCommit-HIVE-Build
> DataWritableWriter strip spaces for CHAR type before writing, but predicate
> generator doesn't do same thing.
>
>
> Key: HIVE-17593
> URL: https://issues.apache.org/jira/browse/HIVE-17593
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.0, 3.0.0
>Reporter: Junjie Chen
>Assignee: Junjie Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-17593.2.patch, HIVE-17593.3.patch, HIVE-17593.patch
>
>
> DataWritableWriter strip spaces for CHAR type before writing. While when
> generating predicate, it does NOT do same striping which should cause data
> missing!
> In current version, it doesn't cause data missing since predicate is not well
> push down to parquet due to HIVE-17261.
> Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as
> same which will build a predicate with tail spaces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20060) Refactor HiveSchemaTool and MetastoreSchemaTool
[ https://issues.apache.org/jira/browse/HIVE-20060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531961#comment-16531961 ] Alan Gates commented on HIVE-20060: --- cc [~mgergely] > Refactor HiveSchemaTool and MetastoreSchemaTool > --- > > Key: HIVE-20060 > URL: https://issues.apache.org/jira/browse/HIVE-20060 > Project: Hive > Issue Type: Task > Components: Beeline, Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HIVE-20060.patch > > > These two classes are 95% the same. Now that HIVE-19711 has split > HiveSchemaTool into multiple components it will be much easier to refactor > these so that there is only one version of the code that each shares. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20060) Refactor HiveSchemaTool and MetastoreSchemaTool
[ https://issues.apache.org/jira/browse/HIVE-20060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-20060: -- Status: Patch Available (was: Open) > Refactor HiveSchemaTool and MetastoreSchemaTool > --- > > Key: HIVE-20060 > URL: https://issues.apache.org/jira/browse/HIVE-20060 > Project: Hive > Issue Type: Task > Components: Beeline, Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HIVE-20060.patch > > > These two classes are 95% the same. Now that HIVE-19711 has split > HiveSchemaTool into multiple components it will be much easier to refactor > these so that there is only one version of the code that each shares. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20060) Refactor HiveSchemaTool and MetastoreSchemaTool
[ https://issues.apache.org/jira/browse/HIVE-20060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-20060: -- Attachment: HIVE-20060.patch > Refactor HiveSchemaTool and MetastoreSchemaTool > --- > > Key: HIVE-20060 > URL: https://issues.apache.org/jira/browse/HIVE-20060 > Project: Hive > Issue Type: Task > Components: Beeline, Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HIVE-20060.patch > > > These two classes are 95% the same. Now that HIVE-19711 has split > HiveSchemaTool into multiple components it will be much easier to refactor > these so that there is only one version of the code that each shares. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20069) Fix reoptimization in case of DPP
[ https://issues.apache.org/jira/browse/HIVE-20069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531947#comment-16531947 ] Ashutosh Chauhan commented on HIVE-20069: - +1 We need similar fix for semijoin reduction as well. > Fix reoptimization in case of DPP > - > > Key: HIVE-20069 > URL: https://issues.apache.org/jira/browse/HIVE-20069 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20069.01.patch > > > reported by [~t3rmin4t0r] > In case dynamic partition pruning; the operator statistics became partial; to > only reflect the actually scanned partitions; but they are being used as an > information about the "full" table...which leads to the exchange of the 2 > tables being joined...which is really unfortunate... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19806) Several tests do not properly sort their output
[
https://issues.apache.org/jira/browse/HIVE-19806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531934#comment-16531934
]
Alan Gates commented on HIVE-19806:
---
{quote}Bunch of tests are removed from testconfiguration. Are these disabled
tests?
{quote}
They are qfiles that aren't there anymore. People just forgot to remove them
from testconfiguration.
{quote}bq.Why did TestMetaStoreMetrics have to sleep?
{quote}
Since it requires a call from the client to the metastore in a different thread
the we have to make sure the current thread yields and the metastore thread
gets a chance to process the close request.
> Several tests do not properly sort their output
> ---
>
> Key: HIVE-19806
> URL: https://issues.apache.org/jira/browse/HIVE-19806
> Project: Hive
> Issue Type: Bug
> Components: Test
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HIVE-19806.2.patch, HIVE-19806.3.patch, HIVE-19806.patch
>
>
> A number of the tests produce unsorted output that happens to come out the
> same on people's laptops and the ptest infrastructure. But when run on a
> separate linux box the sort differences show up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20077) hcat command should follow same pattern as hive cli for getting HBase jars
[
https://issues.apache.org/jira/browse/HIVE-20077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531922#comment-16531922
]
Aihua Xu commented on HIVE-20077:
-
LGTM. +1.
> hcat command should follow same pattern as hive cli for getting HBase jars
> --
>
> Key: HIVE-20077
> URL: https://issues.apache.org/jira/browse/HIVE-20077
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.14.0, 2.3.2
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HIVE-20077.0.patch
>
>
> Currently the {{hcat}} command adds HBase jars to the classpath by using find
> to walk the directories under {{$HBASE_HOME/lib}}.
> {code}
> # Look for HBase in a BigTop-compatible way. Avoid thrift version
> # conflict with modern versions of HBase.
> HBASE_HOME=${HBASE_HOME:-"/usr/lib/hbase"}
> HBASE_CONF_DIR=${HBASE_CONF_DIR:-"${HBASE_HOME}/conf"}
> if [ -d ${HBASE_HOME} ] ; then
>for jar in $(find $HBASE_HOME -name '*.jar' -not -name '*thrift*'); do
> HBASE_CLASSPATH=$HBASE_CLASSPATH:${jar}
>done
>export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CLASSPATH}"
> fi
> if [ -d $HBASE_CONF_DIR ] ; then
> HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CONF_DIR}"
> fi
> {code}
> This is incorrect as that path contains jars for a mixture of purposes; hbase
> client jars, hbase server jars, and hbase shell specific jars. The inclusion
> of unneeded jars is mostly innocuous until the upcoming HBase 2.1.0 release.
> That release will have HBASE-20615 and HBASE-19735, which will mean most
> client facing installations will have a number of shaded client artifacts
> present.
> With those changes in place, the current implementation will include in the
> hcat runtime a mix of shaded and non-shaded hbase artifacts that include some
> Hadoop classes rewritten to use a shaded version of protobuf. When these mix
> with other Hadoop classes in the classpath that have not been rewritten hcat
> will fail with errors that look like this:
> {code}
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto
> cannot be cast to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:225)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy28.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:875)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy29.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1495)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1492)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1507)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:686)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:625)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:557)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:524)
> at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.re
[jira] [Updated] (HIVE-20078) Remove ATSHook
[ https://issues.apache.org/jira/browse/HIVE-20078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-20078: Status: Patch Available (was: Open) > Remove ATSHook > -- > > Key: HIVE-20078 > URL: https://issues.apache.org/jira/browse/HIVE-20078 > Project: Hive > Issue Type: Task > Components: Hooks >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[
https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531919#comment-16531919
]
Hive QA commented on HIVE-17593:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12357/dev-support/hive-personality.sh
|
| git revision | master / cc8ac97 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12357/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> DataWritableWriter strip spaces for CHAR type before writing, but predicate
> generator doesn't do same thing.
>
>
> Key: HIVE-17593
> URL: https://issues.apache.org/jira/browse/HIVE-17593
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.0, 3.0.0
>Reporter: Junjie Chen
>Assignee: Junjie Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-17593.2.patch, HIVE-17593.3.patch, HIVE-17593.patch
>
>
> DataWritableWriter strip spaces for CHAR type before writing. While when
> generating predicate, it does NOT do same striping which should cause data
> missing!
> In current version, it doesn't cause data missing since predicate is not well
> push down to parquet due to HIVE-17261.
> Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as
> same which will build a predicate with tail spaces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20078) Remove ATSHook
[ https://issues.apache.org/jira/browse/HIVE-20078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reassigned HIVE-20078: --- > Remove ATSHook > -- > > Key: HIVE-20078 > URL: https://issues.apache.org/jira/browse/HIVE-20078 > Project: Hive > Issue Type: Task > Components: Hooks >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20077) hcat command should follow same pattern as hive cli for getting HBase jars
[
https://issues.apache.org/jira/browse/HIVE-20077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HIVE-20077:
---
Status: Patch Available (was: In Progress)
-v0
- copy the hbase detection and jar finding from {{bin/hive}}
- tested manually on a cluster with relevant HBase changes in place, via Pig
running against hcat.
> hcat command should follow same pattern as hive cli for getting HBase jars
> --
>
> Key: HIVE-20077
> URL: https://issues.apache.org/jira/browse/HIVE-20077
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.3.2, 0.14.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HIVE-20077.0.patch
>
>
> Currently the {{hcat}} command adds HBase jars to the classpath by using find
> to walk the directories under {{$HBASE_HOME/lib}}.
> {code}
> # Look for HBase in a BigTop-compatible way. Avoid thrift version
> # conflict with modern versions of HBase.
> HBASE_HOME=${HBASE_HOME:-"/usr/lib/hbase"}
> HBASE_CONF_DIR=${HBASE_CONF_DIR:-"${HBASE_HOME}/conf"}
> if [ -d ${HBASE_HOME} ] ; then
>for jar in $(find $HBASE_HOME -name '*.jar' -not -name '*thrift*'); do
> HBASE_CLASSPATH=$HBASE_CLASSPATH:${jar}
>done
>export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CLASSPATH}"
> fi
> if [ -d $HBASE_CONF_DIR ] ; then
> HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CONF_DIR}"
> fi
> {code}
> This is incorrect as that path contains jars for a mixture of purposes; hbase
> client jars, hbase server jars, and hbase shell specific jars. The inclusion
> of unneeded jars is mostly innocuous until the upcoming HBase 2.1.0 release.
> That release will have HBASE-20615 and HBASE-19735, which will mean most
> client facing installations will have a number of shaded client artifacts
> present.
> With those changes in place, the current implementation will include in the
> hcat runtime a mix of shaded and non-shaded hbase artifacts that include some
> Hadoop classes rewritten to use a shaded version of protobuf. When these mix
> with other Hadoop classes in the classpath that have not been rewritten hcat
> will fail with errors that look like this:
> {code}
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto
> cannot be cast to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:225)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy28.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:875)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy29.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1495)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1492)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1507)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:686)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:625)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:557)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:524)
> at
[jira] [Updated] (HIVE-20077) hcat command should follow same pattern as hive cli for getting HBase jars
[
https://issues.apache.org/jira/browse/HIVE-20077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HIVE-20077:
---
Attachment: HIVE-20077.0.patch
> hcat command should follow same pattern as hive cli for getting HBase jars
> --
>
> Key: HIVE-20077
> URL: https://issues.apache.org/jira/browse/HIVE-20077
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.14.0, 2.3.2
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HIVE-20077.0.patch
>
>
> Currently the {{hcat}} command adds HBase jars to the classpath by using find
> to walk the directories under {{$HBASE_HOME/lib}}.
> {code}
> # Look for HBase in a BigTop-compatible way. Avoid thrift version
> # conflict with modern versions of HBase.
> HBASE_HOME=${HBASE_HOME:-"/usr/lib/hbase"}
> HBASE_CONF_DIR=${HBASE_CONF_DIR:-"${HBASE_HOME}/conf"}
> if [ -d ${HBASE_HOME} ] ; then
>for jar in $(find $HBASE_HOME -name '*.jar' -not -name '*thrift*'); do
> HBASE_CLASSPATH=$HBASE_CLASSPATH:${jar}
>done
>export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CLASSPATH}"
> fi
> if [ -d $HBASE_CONF_DIR ] ; then
> HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CONF_DIR}"
> fi
> {code}
> This is incorrect as that path contains jars for a mixture of purposes; hbase
> client jars, hbase server jars, and hbase shell specific jars. The inclusion
> of unneeded jars is mostly innocuous until the upcoming HBase 2.1.0 release.
> That release will have HBASE-20615 and HBASE-19735, which will mean most
> client facing installations will have a number of shaded client artifacts
> present.
> With those changes in place, the current implementation will include in the
> hcat runtime a mix of shaded and non-shaded hbase artifacts that include some
> Hadoop classes rewritten to use a shaded version of protobuf. When these mix
> with other Hadoop classes in the classpath that have not been rewritten hcat
> will fail with errors that look like this:
> {code}
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto
> cannot be cast to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:225)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy28.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:875)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy29.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1495)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1492)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1507)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:686)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:625)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:557)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:524)
> at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorIm
[jira] [Updated] (HIVE-20077) hcat command should follow same pattern as hive cli for getting HBase jars
[
https://issues.apache.org/jira/browse/HIVE-20077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HIVE-20077:
---
Affects Version/s: 2.3.2
> hcat command should follow same pattern as hive cli for getting HBase jars
> --
>
> Key: HIVE-20077
> URL: https://issues.apache.org/jira/browse/HIVE-20077
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.14.0, 2.3.2
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
> Currently the {{hcat}} command adds HBase jars to the classpath by using find
> to walk the directories under {{$HBASE_HOME/lib}}.
> {code}
> # Look for HBase in a BigTop-compatible way. Avoid thrift version
> # conflict with modern versions of HBase.
> HBASE_HOME=${HBASE_HOME:-"/usr/lib/hbase"}
> HBASE_CONF_DIR=${HBASE_CONF_DIR:-"${HBASE_HOME}/conf"}
> if [ -d ${HBASE_HOME} ] ; then
>for jar in $(find $HBASE_HOME -name '*.jar' -not -name '*thrift*'); do
> HBASE_CLASSPATH=$HBASE_CLASSPATH:${jar}
>done
>export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CLASSPATH}"
> fi
> if [ -d $HBASE_CONF_DIR ] ; then
> HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CONF_DIR}"
> fi
> {code}
> This is incorrect as that path contains jars for a mixture of purposes; hbase
> client jars, hbase server jars, and hbase shell specific jars. The inclusion
> of unneeded jars is mostly innocuous until the upcoming HBase 2.1.0 release.
> That release will have HBASE-20615 and HBASE-19735, which will mean most
> client facing installations will have a number of shaded client artifacts
> present.
> With those changes in place, the current implementation will include in the
> hcat runtime a mix of shaded and non-shaded hbase artifacts that include some
> Hadoop classes rewritten to use a shaded version of protobuf. When these mix
> with other Hadoop classes in the classpath that have not been rewritten hcat
> will fail with errors that look like this:
> {code}
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto
> cannot be cast to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:225)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy28.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:875)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy29.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1495)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1492)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1507)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:686)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:625)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:557)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:524)
> at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
[jira] [Updated] (HIVE-20077) hcat command should follow same pattern as hive cli for getting HBase jars
[
https://issues.apache.org/jira/browse/HIVE-20077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HIVE-20077:
---
Affects Version/s: 0.14.0
> hcat command should follow same pattern as hive cli for getting HBase jars
> --
>
> Key: HIVE-20077
> URL: https://issues.apache.org/jira/browse/HIVE-20077
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.14.0, 2.3.2
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
> Currently the {{hcat}} command adds HBase jars to the classpath by using find
> to walk the directories under {{$HBASE_HOME/lib}}.
> {code}
> # Look for HBase in a BigTop-compatible way. Avoid thrift version
> # conflict with modern versions of HBase.
> HBASE_HOME=${HBASE_HOME:-"/usr/lib/hbase"}
> HBASE_CONF_DIR=${HBASE_CONF_DIR:-"${HBASE_HOME}/conf"}
> if [ -d ${HBASE_HOME} ] ; then
>for jar in $(find $HBASE_HOME -name '*.jar' -not -name '*thrift*'); do
> HBASE_CLASSPATH=$HBASE_CLASSPATH:${jar}
>done
>export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CLASSPATH}"
> fi
> if [ -d $HBASE_CONF_DIR ] ; then
> HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CONF_DIR}"
> fi
> {code}
> This is incorrect as that path contains jars for a mixture of purposes; hbase
> client jars, hbase server jars, and hbase shell specific jars. The inclusion
> of unneeded jars is mostly innocuous until the upcoming HBase 2.1.0 release.
> That release will have HBASE-20615 and HBASE-19735, which will mean most
> client facing installations will have a number of shaded client artifacts
> present.
> With those changes in place, the current implementation will include in the
> hcat runtime a mix of shaded and non-shaded hbase artifacts that include some
> Hadoop classes rewritten to use a shaded version of protobuf. When these mix
> with other Hadoop classes in the classpath that have not been rewritten hcat
> will fail with errors that look like this:
> {code}
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto
> cannot be cast to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:225)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy28.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:875)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy29.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1495)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1492)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1507)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:686)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:625)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:557)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:524)
> at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
[jira] [Assigned] (HIVE-20077) hcat command should follow same pattern as hive cli for getting HBase jars
[
https://issues.apache.org/jira/browse/HIVE-20077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reassigned HIVE-20077:
--
> hcat command should follow same pattern as hive cli for getting HBase jars
> --
>
> Key: HIVE-20077
> URL: https://issues.apache.org/jira/browse/HIVE-20077
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
> Currently the {{hcat}} command adds HBase jars to the classpath by using find
> to walk the directories under {{$HBASE_HOME/lib}}.
> {code}
> # Look for HBase in a BigTop-compatible way. Avoid thrift version
> # conflict with modern versions of HBase.
> HBASE_HOME=${HBASE_HOME:-"/usr/lib/hbase"}
> HBASE_CONF_DIR=${HBASE_CONF_DIR:-"${HBASE_HOME}/conf"}
> if [ -d ${HBASE_HOME} ] ; then
>for jar in $(find $HBASE_HOME -name '*.jar' -not -name '*thrift*'); do
> HBASE_CLASSPATH=$HBASE_CLASSPATH:${jar}
>done
>export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CLASSPATH}"
> fi
> if [ -d $HBASE_CONF_DIR ] ; then
> HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CONF_DIR}"
> fi
> {code}
> This is incorrect as that path contains jars for a mixture of purposes; hbase
> client jars, hbase server jars, and hbase shell specific jars. The inclusion
> of unneeded jars is mostly innocuous until the upcoming HBase 2.1.0 release.
> That release will have HBASE-20615 and HBASE-19735, which will mean most
> client facing installations will have a number of shaded client artifacts
> present.
> With those changes in place, the current implementation will include in the
> hcat runtime a mix of shaded and non-shaded hbase artifacts that include some
> Hadoop classes rewritten to use a shaded version of protobuf. When these mix
> with other Hadoop classes in the classpath that have not been rewritten hcat
> will fail with errors that look like this:
> {code}
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto
> cannot be cast to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:225)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy28.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:875)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy29.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1495)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1492)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1507)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:686)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:625)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:557)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:524)
> at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(D
[jira] [Work started] (HIVE-20077) hcat command should follow same pattern as hive cli for getting HBase jars
[
https://issues.apache.org/jira/browse/HIVE-20077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-20077 started by Sean Busbey.
--
> hcat command should follow same pattern as hive cli for getting HBase jars
> --
>
> Key: HIVE-20077
> URL: https://issues.apache.org/jira/browse/HIVE-20077
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
> Currently the {{hcat}} command adds HBase jars to the classpath by using find
> to walk the directories under {{$HBASE_HOME/lib}}.
> {code}
> # Look for HBase in a BigTop-compatible way. Avoid thrift version
> # conflict with modern versions of HBase.
> HBASE_HOME=${HBASE_HOME:-"/usr/lib/hbase"}
> HBASE_CONF_DIR=${HBASE_CONF_DIR:-"${HBASE_HOME}/conf"}
> if [ -d ${HBASE_HOME} ] ; then
>for jar in $(find $HBASE_HOME -name '*.jar' -not -name '*thrift*'); do
> HBASE_CLASSPATH=$HBASE_CLASSPATH:${jar}
>done
>export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CLASSPATH}"
> fi
> if [ -d $HBASE_CONF_DIR ] ; then
> HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HBASE_CONF_DIR}"
> fi
> {code}
> This is incorrect as that path contains jars for a mixture of purposes; hbase
> client jars, hbase server jars, and hbase shell specific jars. The inclusion
> of unneeded jars is mostly innocuous until the upcoming HBase 2.1.0 release.
> That release will have HBASE-20615 and HBASE-19735, which will mean most
> client facing installations will have a number of shaded client artifacts
> present.
> With those changes in place, the current implementation will include in the
> hcat runtime a mix of shaded and non-shaded hbase artifacts that include some
> Hadoop classes rewritten to use a shaded version of protobuf. When these mix
> with other Hadoop classes in the classpath that have not been rewritten hcat
> will fail with errors that look like this:
> {code}
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto
> cannot be cast to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:225)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy28.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:875)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy29.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1495)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1492)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1507)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:686)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:625)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:557)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:524)
> at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccesso
[jira] [Commented] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531899#comment-16531899 ] Gopal V commented on HIVE-20039: LGTM - +1 tests pending > Bucket pruning: Left Outer Join on bucketed table gives wrong result > - > > Key: HIVE-20039 > URL: https://issues.apache.org/jira/browse/HIVE-20039 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 2.3.2 >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20039.1.patch, HIVE-20039.2.patch, > HIVE-20039.3.patch, HIVE-20039.4.patch > > > Left outer join on bucketed table on certain cases gives wrong results. > Depending on the order in which the table-scans are walked through, the > FilterPruner might end up using the wrong table scan's table properties on > the other table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19933) ALTER TABLE DROP PARTITION - Partition Not Found
[
https://issues.apache.org/jira/browse/HIVE-19933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531902#comment-16531902
]
Naveen Gangam commented on HIVE-19933:
--
[~afan] I think it would have been better if this property was never introduced
at all. Now that it has been, we have to be a bit careful removing it in a
minor release. I think we can remove it from the master branch and perhaps flip
it from true to false in other release branches so users can expect a different
behavior by default while having the ability to switch back to the current
behavior.
> ALTER TABLE DROP PARTITION - Partition Not Found
>
>
> Key: HIVE-19933
> URL: https://issues.apache.org/jira/browse/HIVE-19933
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.2
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Major
>
> {code:sql}
> ALTER TABLE web_logsz DROP PARTITION (`date`='xyz')
> -- SemanticException [Error 10001]: Table not found web_logsz
> ALTER TABLE web_logs DROP PARTITION (`date`='xyz')
> -- Success.
> {code}
> There is no 'xyz' partition for the 'date' column. To make this more
> consistent, the query should fail if the user tries to drop a partition that
> does not exist
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19337) Partition whitelist regex doesn't work (and never did)
[
https://issues.apache.org/jira/browse/HIVE-19337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531892#comment-16531892
]
Hive QA commented on HIVE-19337:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12930033/HIVE-19337.03.branch-2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12356/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12356/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12356/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-07-03 20:29:28.865
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12356/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-07-03 20:29:28.869
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
2ca70b9..cc8ac97 master -> origin/master
a47a80f..eb261cb master-txnstats -> origin/master-txnstats
+ git reset --hard HEAD
HEAD is now at 2ca70b9 HIVE-19326 : stats auto gather: incorrect aggregation
during UNION queries (may lead to incorrect results) (Zoltan Haindrich via
Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at cc8ac97 HIVE-19860 : HiveServer2 ObjectInspectorFactory memory
leak with cachedUnionStructObjectInspector (Rajkumar Singh)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-07-03 20:29:30.351
+ rm -rf ../yetus_PreCommit-HIVE-Build-12356
+ mkdir ../yetus_PreCommit-HIVE-Build-12356
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12356
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12356/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
does not exist in index
error: metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
does not exist in index
error: src/java/org/apache/hadoop/hive/metastore/ObjectStore.java: does not
exist in index
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-12356
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12930033 - PreCommit-HIVE-Build
> Partition whitelist regex doesn't work (and never did)
> --
>
> Key: HIVE-19337
> URL: https://issues.apache.org/jira/browse/HIVE-19337
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-19337.01.branch-2.patch,
> HIVE-19337.03.branch-2.patch
>
>
> {{ObjectStore.setConf()}} has the following code:
> {code:java}
> String partitionValidationRegex =
>
> hiveConf.get(HiveConf.ConfVars.METASTORE_PARTITION_NAME_WHITELIST_PATTERN.name());
> {code}
> Note that it uses name() method which returns enum name
> (METASTORE_PARTITION_NAME_WHITELIST_PATTERN) rather then .varname
> As a result the regex will always be null.
> The code was introduced as part of
> HIVE-7223 Support generic PartitionSpecs in Metastore partition-functions
> So looks like this was broken since the original code drop. This is fixed in
> Hive3 - probably when [~alangates] rework
[jira] [Commented] (HIVE-17849) alterPartition() may fail to rollback transaction
[
https://issues.apache.org/jira/browse/HIVE-17849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531889#comment-16531889
]
Hive QA commented on HIVE-17849:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12893095/HIVE-17849.01-branch-2.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10578 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=142)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query64]
(batchId=229)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_simple_select]
(batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby1]
(batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_multi_single_reducer2]
(batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[input18]
(batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join26]
(batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join32]
(batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_reorder2]
(batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_join]
(batchId=105)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion02
(batchId=263)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdate.testNonAcidToAcidConversion02
(batchId=275)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion02
(batchId=272)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12355/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12355/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12355/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12893095 - PreCommit-HIVE-Build
> alterPartition() may fail to rollback transaction
> -
>
> Key: HIVE-17849
> URL: https://issues.apache.org/jira/browse/HIVE-17849
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.0
>Reporter: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-17849.01-branch-2.3.patch
>
>
> In HiveAlterHandle.alterPartition() there is this code:
> {code}
> try {
> msdb.openTransaction();
> msdb.alterPartition(dbname, name, new_part.getValues(), oldPart);
> if (transactionalListeners != null &&
> !transactionalListeners.isEmpty()) {
> MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
>
> EventMessage.EventType.ALTER_PARTITION,
> new
> AlterPartitionEvent(new_part, oldPart, tbl, success, handler));
> }
> revertMetaDataTransaction = msdb.commitTransaction();
> } catch (Exception ex2) {
> LOG.error("Attempt to revert partition metadata change failed.
> The revert was attempted " +
> "because associated filesystem rename operation failed with
> exception " + ex.getMessage(), ex2);
> if (!revertMetaDataTransaction) {
> msdb.rollbackTransaction();
> }
> }
> {code}
> Note that there is no {{finally}} clause, so it is possible for some
> unchecked exception to occur in which case the transaction will remain
> active. Once this happens, all subsequent transactions on this thread will
> not behave correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531880#comment-16531880 ] Deepak Jaiswal commented on HIVE-20039: --- Ah, my bad. Forgot to add the test to testconfiguration. Done in patch4. > Bucket pruning: Left Outer Join on bucketed table gives wrong result > - > > Key: HIVE-20039 > URL: https://issues.apache.org/jira/browse/HIVE-20039 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 2.3.2 >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20039.1.patch, HIVE-20039.2.patch, > HIVE-20039.3.patch, HIVE-20039.4.patch > > > Left outer join on bucketed table on certain cases gives wrong results. > Depending on the order in which the table-scans are walked through, the > FilterPruner might end up using the wrong table scan's table properties on > the other table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20039: -- Attachment: HIVE-20039.4.patch > Bucket pruning: Left Outer Join on bucketed table gives wrong result > - > > Key: HIVE-20039 > URL: https://issues.apache.org/jira/browse/HIVE-20039 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 2.3.2 >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20039.1.patch, HIVE-20039.2.patch, > HIVE-20039.3.patch, HIVE-20039.4.patch > > > Left outer join on bucketed table on certain cases gives wrong results. > Depending on the order in which the table-scans are walked through, the > FilterPruner might end up using the wrong table scan's table properties on > the other table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19933) ALTER TABLE DROP PARTITION - Partition Not Found
[
https://issues.apache.org/jira/browse/HIVE-19933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531872#comment-16531872
]
Alice Fan commented on HIVE-19933:
--
Hi [~ngangam],
Aihua and David suggest we remove the hive.exec.drop.ignorenonexistent flag
entirely. Do you agree with it? Thanks, Alice
> ALTER TABLE DROP PARTITION - Partition Not Found
>
>
> Key: HIVE-19933
> URL: https://issues.apache.org/jira/browse/HIVE-19933
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.2
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Major
>
> {code:sql}
> ALTER TABLE web_logsz DROP PARTITION (`date`='xyz')
> -- SemanticException [Error 10001]: Table not found web_logsz
> ALTER TABLE web_logs DROP PARTITION (`date`='xyz')
> -- Success.
> {code}
> There is no 'xyz' partition for the 'date' column. To make this more
> consistent, the query should fail if the user tries to drop a partition that
> does not exist
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19008) Improve Spark session id logging
[ https://issues.apache.org/jira/browse/HIVE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531857#comment-16531857 ] Sahil Takiar commented on HIVE-19008: - [~aihuaxu] could you take a look? The high level idea is that Hive on Spark has a Hive Session ID and a Spark Session ID. I've basically changing the code so that the Spark Session ID is just a counter that is reset for each Hive Session. Previously, the Spark Session ID was just some random UUID, which I don't think is very informative. Furthermore, the Spark Web UI now shows the Hive Session ID rather than the Spark Session ID, which I think is more helpful as it makes it easier to associated entries in the Spark Web UI with a Hive session. > Improve Spark session id logging > > > Key: HIVE-19008 > URL: https://issues.apache.org/jira/browse/HIVE-19008 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-19008.1.patch, HIVE-19008.2.patch > > > HoS users have two session ids, one id for the Hive session and another id > for the Spark session, both are UUIDs. > I think some improvements could be made here: > The Spark session id could just be a counter that is incremented for each new > Spark session within a Hive session. Each Spark session is still globally > identifiable by its associated Hive session id + its own counter. This may > make more sense since the Hive session - Spark session has a 1-to-many > relationship, as in a single Hive session can contain multiple Spark > sessions, and each Spark session must belong to a Hive session. > Furthermore, we should include both the Hive session id and Spark session id > in the console logs + the Spark Web UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
