[jira] [Commented] (HIVE-23414) Detail Hive Java Compatibility
[
https://issues.apache.org/jira/browse/HIVE-23414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103117#comment-17103117
]
Hive QA commented on HIVE-23414:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002413/HIVE-23414.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17240 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22217/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22217/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22217/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002413 - PreCommit-HIVE-Build
> Detail Hive Java Compatibility
> --
>
> Key: HIVE-23414
> URL: https://issues.apache.org/jira/browse/HIVE-23414
> Project: Hive
> Issue Type: Improvement
> Components: Documentation
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-23414.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23414) Detail Hive Java Compatibility
[
https://issues.apache.org/jira/browse/HIVE-23414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103107#comment-17103107
]
Hive QA commented on HIVE-23414:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
57s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22217/dev-support/hive-personality.sh
|
| git revision | master / 134f3b2 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22217/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Detail Hive Java Compatibility
> --
>
> Key: HIVE-23414
> URL: https://issues.apache.org/jira/browse/HIVE-23414
> Project: Hive
> Issue Type: Improvement
> Components: Documentation
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-23414.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23407) Prompt Beeline Users To Enable Verbose Logging on Error
[
https://issues.apache.org/jira/browse/HIVE-23407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103103#comment-17103103
]
Hive QA commented on HIVE-23407:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002411/HIVE-23407.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 17236 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=184)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_insert_partitioned]
(batchId=95)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerSlowQueryExecutionTime
(batchId=212)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerTotalLaunchedTasks
(batchId=212)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerVertexTotalTasks
(batchId=212)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime
(batchId=218)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22216/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22216/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22216/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002411 - PreCommit-HIVE-Build
> Prompt Beeline Users To Enable Verbose Logging on Error
> ---
>
> Key: HIVE-23407
> URL: https://issues.apache.org/jira/browse/HIVE-23407
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-23407.1.patch, HIVE-23407.1.patch
>
>
> When an error occurs on Beeline, encourage users to enable verbose error
> logging to get more details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22334) There is a permission problem with the add jar package on the command line
[ https://issues.apache.org/jira/browse/HIVE-22334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103099#comment-17103099 ] philipse commented on HIVE-22334: - Possible issue: if you have admin role,try to set role admin to get the admin priviledge. if not, you may disable the ADD operation if allowed in Operation2Privilege class. Another way, if the jar is permanat,you can add it in the aux jar in hive-site.xml Help it can help u. > There is a permission problem with the add jar package on the command line > -- > > Key: HIVE-22334 > URL: https://issues.apache.org/jira/browse/HIVE-22334 > Project: Hive > Issue Type: Bug > Environment: hive 3.1 >Reporter: jiazhaoquan >Priority: Blocker > Attachments: 启动失败文档.jpg > > > The environment is ve3.0;Now add a jar package has the problem of removing > permissions, as follows; > >add jar > >/export/servers/hive-connector-0.9.0-SNAPSHOT-jar-with-dependencies.jar; > >Error: Error while processing statement: Permission denied: Principal > >[name=root, type=USER] does not have following privileges for operation ADD > >[ADMIN] (state=,code=1)` > > I have used Storage Based Authorization in the Metastore Server according to > the configuration;But the add jar package still has permissions issues;Put my > jar package into the lib file, and my hive started until it finally appeared > (closed on the command line).The problem log is in the attachment. > > 环境是hive3.0；现在add一个jar包出现去权限问题，如下； > >add jar > >/export/servers/hive-connector-0.9.0-SNAPSHOT-jar-with-dependencies.jar; > >Error: Error while processing statement: Permission denied: Principal > >[name=root, type=USER] does not have following privileges for operation ADD > >[ADMIN] (state=,code=1)` > 我已经根据配置使用了Storage Based Authorization in the Metastore Server；但是add > jar包还是有权限问题；将我的jar包放入lib文件，我的hive启动到最后出现（命令行出现closed），启动不了hive；问题在附件中； -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23407) Prompt Beeline Users To Enable Verbose Logging on Error
[
https://issues.apache.org/jira/browse/HIVE-23407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103094#comment-17103094
]
Hive QA commented on HIVE-23407:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22216/dev-support/hive-personality.sh
|
| git revision | master / 134f3b2 |
| Default Java | 1.8.0_111 |
| modules | C: beeline U: beeline |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22216/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Prompt Beeline Users To Enable Verbose Logging on Error
> ---
>
> Key: HIVE-23407
> URL: https://issues.apache.org/jira/browse/HIVE-23407
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-23407.1.patch, HIVE-23407.1.patch
>
>
> When an error occurs on Beeline, encourage users to enable verbose error
> logging to get more details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22668) ClassNotFoundException:HiveHBaseTableInputFormat when tez include reduce operation
[
https://issues.apache.org/jira/browse/HIVE-22668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103093#comment-17103093
]
philipse commented on HIVE-22668:
-
have you tried to put the jar as the third party jar in hiveserver?
> ClassNotFoundException:HiveHBaseTableInputFormat when tez include reduce
> operation
> --
>
> Key: HIVE-22668
> URL: https://issues.apache.org/jira/browse/HIVE-22668
> Project: Hive
> Issue Type: Bug
> Components: Beeline, Hive
>Affects Versions: 3.1.0
>Reporter: Michael
>Priority: Blocker
>
> When I use beeline to execute script which the operation is insert data from
> hive to hbase.
> If the operation include reduce step, this exception will appearance.
> I try to add jar in beeline like this:
> {code:java}
> ADD JAR /usr/hdp/3.1.4.0-315/hive/lib/hive-hbase-handler-3.1.0.3.1.4.0-315.jar
> ADD JAR /usr/hdp/3.1.4.0-315/hive/lib/guava-28.0-jre.jar
> ADD JAR /usr/hdp/3.1.4.0-315/hive/lib/zookeeper-3.4.6.3.1.4.0-315.jar{code}
> but this problem always exist.
> {code:java}
> Serialization trace:
> inputFileFormatClass (org.apache.hadoop.hive.ql.plan.TableDesc)
> tableInfo (org.apache.hadoop.hive.ql.plan.FileSinkDesc)
> conf (org.apache.hadoop.hive.ql.exec.FileSinkOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator)
> reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:185)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:326)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:314)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObjectOrNull(SerializationUtilities.java:203)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:218)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:218)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:180)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:218)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:180)
> at
>
[jira] [Commented] (HIVE-23099) Improve Logger for Operation Child Classes
[
https://issues.apache.org/jira/browse/HIVE-23099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103084#comment-17103084
]
Hive QA commented on HIVE-23099:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002410/HIVE-23099.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17240 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testReplTimeouts
(batchId=244)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22215/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22215/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22215/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002410 - PreCommit-HIVE-Build
> Improve Logger for Operation Child Classes
> --
>
> Key: HIVE-23099
> URL: https://issues.apache.org/jira/browse/HIVE-23099
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23099.1.patch, HIVE-23099.2.patch,
> HIVE-23099.3.patch, HIVE-23099.4.patch, HIVE-23099.4.patch
>
>
> The {{Operation}} class declares its logger this way:
> {code:java|title=Operation.java}
> public abstract class Operation {
> public static final Logger LOG =
> LoggerFactory.getLogger(Operation.class.getName());
> ...
> }
> {code}
> Notice that this is an {{abstract}} class, but the {{Logger}} is tied to the
> {{Operation.class.getName()}}. This means that logging cannot be controlled
> for each subclass of {{Operation}} independently since they all use the same
> static {{Logger}} instance.
> Make the LOG a {{protected}} instance variable that inherits the name of the
> child class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23099) Improve Logger for Operation Child Classes
[
https://issues.apache.org/jira/browse/HIVE-23099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103067#comment-17103067
]
Hive QA commented on HIVE-23099:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} service in master has 51 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} service: The patch generated 2 new + 57 unchanged - 2
fixed = 59 total (was 59) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22215/dev-support/hive-personality.sh
|
| git revision | master / 134f3b2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22215/yetus/diff-checkstyle-service.txt
|
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22215/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve Logger for Operation Child Classes
> --
>
> Key: HIVE-23099
> URL: https://issues.apache.org/jira/browse/HIVE-23099
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23099.1.patch, HIVE-23099.2.patch,
> HIVE-23099.3.patch, HIVE-23099.4.patch, HIVE-23099.4.patch
>
>
> The {{Operation}} class declares its logger this way:
> {code:java|title=Operation.java}
> public abstract class Operation {
> public static final Logger LOG =
> LoggerFactory.getLogger(Operation.class.getName());
> ...
> }
> {code}
> Notice that this is an {{abstract}} class, but the {{Logger}} is tied to the
> {{Operation.class.getName()}}. This means that logging cannot be controlled
> for each subclass of {{Operation}} independently since they all use the same
> static {{Logger}} instance.
> Make the LOG a {{protected}} instance variable that inherits the name of the
> child class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21820) drop database command flushing other data outside db folder
[
https://issues.apache.org/jira/browse/HIVE-21820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103061#comment-17103061
]
philipse commented on HIVE-21820:
-
for create database operation:
if no locations specific,then the “.db” is generated as the path suffix,the
rootpath will be the default warehouse path.
if locations are specific,then the location will be the whole path for the
database
{code:java}
// private void create_database_core(RawStore ms, final Database db)
throws AlreadyExistsException, InvalidObjectException, MetaException {
if (!validateName(db.getName())) {
throw new InvalidObjectException(db.getName() + " is not a valid database
name");
}
if (null == db.getLocationUri()) {
db.setLocationUri(wh.getDefaultDatabasePath(db.getName()).toString());
} else {
db.setLocationUri(wh.getDnsPath(new Path(db.getLocationUri())).toString());
}
{code}
for drop database operation:
it will delete the database location after cleaning the
meta(views,functions,tables),if other dbs share the same location as the
deleted db, meta will not feel that, the relation between the location and meta
info are not like the foreign key, maybe related with the original design of
metastore db,you can monitor the the event and do the filter if different
databases or tables share the same location.
Help it will help you.
> drop database command flushing other data outside db folder
> ---
>
> Key: HIVE-21820
> URL: https://issues.apache.org/jira/browse/HIVE-21820
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Neeraj Maheshwari
>Priority: Blocker
>
> On running the below command to create new database:
> *create database abc location /users/hive/warehouse/project_name/;*
> here expected is, this command should create a new db folder named abc.db
> under project_name directory, but with hive version 1.2.1000.2.6.3.0-235 its
> not creating abc.db folder.
>
> And on running command *drop database abc;* the entire project_name directory
> got deleted from HDFS. So here drop command rather than dropping just one db
> it deleted whole directory and just retained meta informations in hive
> metastore for other dbs present in project_name directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23387) Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse
[
https://issues.apache.org/jira/browse/HIVE-23387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103052#comment-17103052
]
Hive QA commented on HIVE-23387:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002407/HIVE-23387.patch
{color:green}SUCCESS:{color} +1 due to 9 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 47 failed/errored test(s), 17142 tests
executed
*Failed tests:*
{noformat}
TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=218)
TestHS2AuthzContext - did not produce a TEST-*.xml file (likely timed out)
(batchId=219)
TestHS2AuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=219)
TestHiveServer2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=219)
TestHiveServer2SessionTimeout - did not produce a TEST-*.xml file (likely timed
out) (batchId=219)
TestHs2ConnectionMetricsBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=219)
TestHs2ConnectionMetricsHttp - did not produce a TEST-*.xml file (likely timed
out) (batchId=219)
TestJdbcMetadataApiAuth - did not produce a TEST-*.xml file (likely timed out)
(batchId=219)
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file (likely timed
out) (batchId=218)
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=218)
TestJdbcWithMiniLlapVectorArrow - did not produce a TEST-*.xml file (likely
timed out) (batchId=218)
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file (likely
timed out) (batchId=218)
TestJdbcWithSQLAuthorization - did not produce a TEST-*.xml file (likely timed
out) (batchId=219)
TestMultiSessionsHS2WithLocalClusterSpark - did not produce a TEST-*.xml file
(likely timed out) (batchId=218)
TestNewGetSplitsFormatReturnPath - did not produce a TEST-*.xml file (likely
timed out) (batchId=218)
TestNoSaslAuth - did not produce a TEST-*.xml file (likely timed out)
(batchId=218)
TestServiceDiscoveryWithMiniHS2 - did not produce a TEST-*.xml file (likely
timed out) (batchId=219)
TestTriggersTezSessionPoolManager - did not produce a TEST-*.xml file (likely
timed out) (batchId=218)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=123)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ctas_uses_database_location]
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_part_2]
(batchId=53)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDatabaseLocation
(batchId=155)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDatabaseLocationWithPermissionProblems
(batchId=155)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testDatabaseLocation
(batchId=157)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testDatabaseLocationWithPermissionProblems
(batchId=157)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreCustomAuth.testDatabaseLocation
(batchId=163)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreCustomAuth.testDatabaseLocationWithPermissionProblems
(batchId=163)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabaseLocation
(batchId=156)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabaseLocationWithPermissionProblems
(batchId=156)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocation
(batchId=163)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocationWithPermissionProblems
(batchId=163)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDatabaseLocation
(batchId=153)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDatabaseLocationWithPermissionProblems
(batchId=153)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testDatabaseLocation
(batchId=151)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testDatabaseLocationWithPermissionProblems
(batchId=151)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testDatabaseLocation
(batchId=162)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testDatabaseLocationWithPermissionProblems
(batchId=162)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testRecycleFileNonReplDatabase
(batchId=183)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRecycleFileNonReplDatabase
(batchId=192)
org.apache.hadoop.hive.ql.util.TestHiveStrictManagedMigration.testExternalMove
(batchId=244)
org.apache.hive.jdbc.TestJdbcWithMiniLlapVectorArrowBatch.testInvalidReferenceCountScenario
(batchId=215)
org.apache.hive.minikdc.TestRemoteHiveMetaStoreDualAuthCustom.testDatabaseLocation
(batchId=235)
org.apache.hive.minikdc.TestRemoteHiveMetaStoreDualAuthCustom.testDatabaseLocationWithPermissionProblems
(batchId=235)
[jira] [Commented] (HIVE-23279) Unable to move source hdfs://xxx:8020/warehouse/.../.hive-staging_hive_... | 无法移动 hivestage 文件
[
https://issues.apache.org/jira/browse/HIVE-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103046#comment-17103046
]
philipse commented on HIVE-23279:
-
is it reproduced? Have u checked the src and destination priviledge,
> Unable to move source hdfs://xxx:8020/warehouse/.../.hive-staging_hive_... |
> 无法移动 hivestage 文件
> --
>
> Key: HIVE-23279
> URL: https://issues.apache.org/jira/browse/HIVE-23279
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.1.1
> Environment: Centos 7
>
> Cloudera Manager Agent 6.2.1
> Cloudera Manager Management Daemon 6.2.1
> Hadoop 3.0.0 cdh6.2.1
> HDFS 3.0.0 cdh6.2.1
> HttpFS 3.0.0 cdh6.2.1
> hadoop-kms 3.0.0 cdh6.2.1
> MapReduce 2 3.0.0 cdh6.2.1
> YARN 3.0.0 cdh6.2.1
> Hive 2.1.1 cdh6.2.1
> HCatalog 2.1.1 cdh6.2.1
> Hue 4.3.0 cdh6.2.1
> Impala 3.2.0 cdh6.2.1
> Java 8 java version "1.8.0_181" Java (TM) SE Runtime Environment (build
> 1.8.0_181-b13) Java HotSpot (TM) 64-Bit Server VM (build 25.181-b13, mixed
> mode)
> Parquet 1.9.0 cdh6.2.1
> spark 2.4.0 cdh6.2.1
>Reporter: geekyouth
>Assignee: geekyouth
>Priority: Blocker
> Labels: HiveServer2, cdh6
> Attachments: chrome_k3C4YK0O5V.png, chrome_qy2E1vTR3j.png
>
>
> {color:#de350b}*error log from Cloudera Manager 6.2.1:;) | cdh 日志：*{color}
>
> [HiveServer2-Background-Pool: Thread-58]: Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask. {color:#de350b}Unable to move
> source{color}
> hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/.hive-staging_hive_2020-04-23_14-29-31_758_5380087337087305328-1/-ext-1
> to destination
> hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:329)
> ~[hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:258)
> ~[hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:92)
> ~[hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
> [hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_181]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
> [hadoop-common-3.0.0-cdh6.2.1.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:357)
> [hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> {color:#de350b}Unable to move source{color}
> hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/.hive-staging_hive_2020-04-23_14-29-31_758_5380087337087305328-1/-ext-1
> to destination
> hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01
> at org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:3445)
> ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
> at org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:3401)
> ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
> at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:3396)
> ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
> at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3693)
> ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:1614)
> ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1525)
> ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1489)
> ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
> at
[jira] [Commented] (HIVE-23387) Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse
[
https://issues.apache.org/jira/browse/HIVE-23387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103037#comment-17103037
]
Hive QA commented on HIVE-23387:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
31s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 11 new + 752 unchanged - 0 fixed = 763 total (was 752) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} itests/hive-unit: The patch generated 4 new + 610
unchanged - 0 fixed = 614 total (was 610) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22214/dev-support/hive-personality.sh
|
| git revision | master / 134f3b2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22214/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22214/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server itests/hive-unit itests/hive-unit-hadoop2
U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22214/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse
> --
>
> Key: HIVE-23387
> URL: https://issues.apache.org/jira/browse/HIVE-23387
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects
[jira] [Commented] (HIVE-23362) Repl dump returns dump location and repl id but doesn't write the ack in s3a FS
[
https://issues.apache.org/jira/browse/HIVE-23362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103031#comment-17103031
]
Hive QA commented on HIVE-23362:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002406/HIVE-23362.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17241 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22213/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22213/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22213/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002406 - PreCommit-HIVE-Build
> Repl dump returns dump location and repl id but doesn't write the ack in s3a
> FS
> ---
>
> Key: HIVE-23362
> URL: https://issues.apache.org/jira/browse/HIVE-23362
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23362.01.patch, HIVE-23362.02.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23362) Repl dump returns dump location and repl id but doesn't write the ack in s3a FS
[
https://issues.apache.org/jira/browse/HIVE-23362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103014#comment-17103014
]
Hive QA commented on HIVE-23362:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 2 new + 13 unchanged - 0 fixed
= 15 total (was 13) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22213/dev-support/hive-personality.sh
|
| git revision | master / 134f3b2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22213/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22213/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Repl dump returns dump location and repl id but doesn't write the ack in s3a
> FS
> ---
>
> Key: HIVE-23362
> URL: https://issues.apache.org/jira/browse/HIVE-23362
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23362.01.patch, HIVE-23362.02.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23337) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 2
[ https://issues.apache.org/jira/browse/HIVE-23337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23337: -- Attachment: HIVE-23337.04.patch > Move q tests to TestMiniLlapLocal from TestCliDriver where the output is > different, batch 2 > --- > > Key: HIVE-23337 > URL: https://issues.apache.org/jira/browse/HIVE-23337 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23337.01.patch, HIVE-23337.02.patch, > HIVE-23337.03.patch, HIVE-23337.04.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23337) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 2
[ https://issues.apache.org/jira/browse/HIVE-23337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23337: -- Attachment: (was: HIVE-23337.04.patch) > Move q tests to TestMiniLlapLocal from TestCliDriver where the output is > different, batch 2 > --- > > Key: HIVE-23337 > URL: https://issues.apache.org/jira/browse/HIVE-23337 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23337.01.patch, HIVE-23337.02.patch, > HIVE-23337.03.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23351) Ranger Replication Scheduling
[
https://issues.apache.org/jira/browse/HIVE-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103010#comment-17103010
]
Hive QA commented on HIVE-23351:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002442/HIVE-23351.08.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 17248 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=184)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testReplTimeouts
(batchId=244)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.retryIncBootstrapAcidFromDifferentDumpWithoutCleanTablesConfig
(batchId=182)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.retryIncBootstrapAcidFromDifferentDumpWithoutCleanTablesConfig
(batchId=179)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22212/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22212/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22212/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002442 - PreCommit-HIVE-Build
> Ranger Replication Scheduling
> -
>
> Key: HIVE-23351
> URL: https://issues.apache.org/jira/browse/HIVE-23351
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch,
> HIVE-23351.03.patch, HIVE-23351.04.patch, HIVE-23351.05.patch,
> HIVE-23351.06.patch, HIVE-23351.07.patch, HIVE-23351.08.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23423) Hash aggregation is always disabled in vectorized execution of grouping set queries
[ https://issues.apache.org/jira/browse/HIVE-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Vijayaraghavan updated HIVE-23423: Attachment: HIVE-23423.WIP.patch > Hash aggregation is always disabled in vectorized execution of grouping set > queries > --- > > Key: HIVE-23423 > URL: https://issues.apache.org/jira/browse/HIVE-23423 > Project: Hive > Issue Type: Bug > Components: Hive, Operators, Query Processor >Affects Versions: 4.0.0 >Reporter: Nita Dembla >Assignee: Gopal Vijayaraghavan >Priority: Major > Labels: Performance > Attachments: HIVE-23423.WIP.patch > > > https://issues.apache.org/jira/browse/HIVE-23356 fixed the issue with > disabling hash aggregation on grouping set queries. Need a fix for > VectorGroupbyOperator operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23423) Hash aggregation is always disabled in vectorized execution of grouping set queries
[ https://issues.apache.org/jira/browse/HIVE-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Vijayaraghavan updated HIVE-23423: Status: Patch Available (was: Open) > Hash aggregation is always disabled in vectorized execution of grouping set > queries > --- > > Key: HIVE-23423 > URL: https://issues.apache.org/jira/browse/HIVE-23423 > Project: Hive > Issue Type: Bug > Components: Hive, Operators, Query Processor >Affects Versions: 4.0.0 >Reporter: Nita Dembla >Assignee: Gopal Vijayaraghavan >Priority: Major > Labels: Performance > Attachments: HIVE-23423.WIP.patch > > > https://issues.apache.org/jira/browse/HIVE-23356 fixed the issue with > disabling hash aggregation on grouping set queries. Need a fix for > VectorGroupbyOperator operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23351) Ranger Replication Scheduling
[
https://issues.apache.org/jira/browse/HIVE-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103003#comment-17103003
]
Hive QA commented on HIVE-23351:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 112 new + 49 unchanged - 3
fixed = 161 total (was 52) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
53s{color} | {color:red} ql generated 2 new + 1527 unchanged - 0 fixed = 1529
total (was 1527) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Redundant nullcheck of uri, which is known to be non-null in
org.apache.hadoop.hive.ql.exec.repl.ranger.RangerRestClientImpl.exportRangerPolicies(String,
String, String) Redundant null check at RangerRestClientImpl.java:is known to
be non-null in
org.apache.hadoop.hive.ql.exec.repl.ranger.RangerRestClientImpl.exportRangerPolicies(String,
String, String) Redundant null check at RangerRestClientImpl.java:[line 83] |
| | Exception is caught when Exception is not thrown in
org.apache.hadoop.hive.ql.exec.repl.ranger.RangerRestClientImpl.checkConnection(String)
At RangerRestClientImpl.java:is not thrown in
org.apache.hadoop.hive.ql.exec.repl.ranger.RangerRestClientImpl.checkConnection(String)
At RangerRestClientImpl.java:[line 351] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22212/dev-support/hive-personality.sh
|
| git revision | master / 134f3b2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22212/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22212/yetus/new-findbugs-ql.html
|
| modules | C: common ql itests/hive-unit U: . |
[jira] [Commented] (HIVE-23234) Optimize TxnHandler::allocateTableWriteIds
[
https://issues.apache.org/jira/browse/HIVE-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102978#comment-17102978
]
Hive QA commented on HIVE-23234:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002397/HIVE-23234.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17237 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=184)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22211/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22211/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22211/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002397 - PreCommit-HIVE-Build
> Optimize TxnHandler::allocateTableWriteIds
> --
>
> Key: HIVE-23234
> URL: https://issues.apache.org/jira/browse/HIVE-23234
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23234.1.patch, HIVE-23234.2.patch,
> HIVE-23234.3.patch, HIVE-23234.4.patch, HIVE-23234.5.patch,
> HIVE-23234.5.patch, HIVE-23234.5.patch
>
>
> Table write id allocation should be examined and optimized. One low hanging
> fruit is batching all the PreparedStatement inserts, but there might be other
> opportunities as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23085) LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache
[ https://issues.apache.org/jira/browse/HIVE-23085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102975#comment-17102975 ] Gopal Vijayaraghavan edited comment on HIVE-23085 at 5/8/20, 10:15 PM: --- This isn't about configuration - it is about failure rates with multiple mount points and mmap(). Every extra mount point used, multiplies the failure probability. was (Author: gopalv): This isn't about configuration - it is about failure rates with multiple mount points and mmap(). Every extra mount point used, doubles the failure probability. > LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache > > > Key: HIVE-23085 > URL: https://issues.apache.org/jira/browse/HIVE-23085 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23085.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently we can configure only one SSD location while using SSD cache in > LLAP. This highly undermines the capacity of some machines to use its disk > capacity to the fullest. For example *AWS* provides *r5d.4x large* series > which comes with *2 * 300 GB NVme SSD disk* with the current design only one > of the mounted *NVme SSD* disk can be used for caching. Hence adding support > for caching data at multiple ssd mounted locations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23085) LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache
[ https://issues.apache.org/jira/browse/HIVE-23085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102975#comment-17102975 ] Gopal Vijayaraghavan commented on HIVE-23085: - This isn't about configuration - it is about failure rates with multiple mount points and mmap(). Every extra mount point used, doubles the failure probability. > LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache > > > Key: HIVE-23085 > URL: https://issues.apache.org/jira/browse/HIVE-23085 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23085.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently we can configure only one SSD location while using SSD cache in > LLAP. This highly undermines the capacity of some machines to use its disk > capacity to the fullest. For example *AWS* provides *r5d.4x large* series > which comes with *2 * 300 GB NVme SSD disk* with the current design only one > of the mounted *NVme SSD* disk can be used for caching. Hence adding support > for caching data at multiple ssd mounted locations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23234) Optimize TxnHandler::allocateTableWriteIds
[
https://issues.apache.org/jira/browse/HIVE-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102961#comment-17102961
]
Hive QA commented on HIVE-23234:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 538 unchanged - 3 fixed = 539 total (was 541) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
20s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 187 unchanged - 0 fixed = 188 total (was 187) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Dead store to updateCount in
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.setHadoopJobId(String,
long) At
CompactionTxnHandler.java:org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.setHadoopJobId(String,
long) At CompactionTxnHandler.java:[line 1122] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22211/dev-support/hive-personality.sh
|
| git revision | master / 134f3b2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22211/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22211/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22211/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimize TxnHandler::allocateTableWriteIds
> --
>
> Key: HIVE-23234
> URL: https://issues.apache.org/jira/browse/HIVE-23234
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments:
[jira] [Updated] (HIVE-23389) FilterMergeRule can lead to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-23389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23389:
---
Attachment: HIVE-23389.01.patch
> FilterMergeRule can lead to AssertionError
> --
>
> Key: HIVE-23389
> URL: https://issues.apache.org/jira/browse/HIVE-23389
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23389.01.patch, HIVE-23389.patch
>
>
> I have not been able to reproduce the issue in latest master. However, this
> could potentially happen since Filter creation has a check on whether the
> expression is flat
> ([here|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Filter.java#L74])
> and Filter merge does not flatten an expression when it is created.
> {noformat}
> java.lang.AssertionError: AND(=($3, 100), OR(OR(null, IS NOT
> NULL(CAST(100):INTEGER)), =(CAST(100):INTEGER, CAST(200):INTEGER)))
> at org.apache.calcite.rel.core.Filter.(Filter.java:74)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.(HiveFilter.java:39)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(HiveRelFactories.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder.filter(HiveRelBuilder.java:99)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1055)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:81)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23389) FilterMergeRule can lead to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-23389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23389:
---
Attachment: (was: HIVE-23389.01.patch)
> FilterMergeRule can lead to AssertionError
> --
>
> Key: HIVE-23389
> URL: https://issues.apache.org/jira/browse/HIVE-23389
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23389.01.patch, HIVE-23389.patch
>
>
> I have not been able to reproduce the issue in latest master. However, this
> could potentially happen since Filter creation has a check on whether the
> expression is flat
> ([here|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Filter.java#L74])
> and Filter merge does not flatten an expression when it is created.
> {noformat}
> java.lang.AssertionError: AND(=($3, 100), OR(OR(null, IS NOT
> NULL(CAST(100):INTEGER)), =(CAST(100):INTEGER, CAST(200):INTEGER)))
> at org.apache.calcite.rel.core.Filter.(Filter.java:74)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.(HiveFilter.java:39)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(HiveRelFactories.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder.filter(HiveRelBuilder.java:99)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1055)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:81)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23389) FilterMergeRule can lead to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-23389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23389:
---
Attachment: HIVE-23389.01.patch
> FilterMergeRule can lead to AssertionError
> --
>
> Key: HIVE-23389
> URL: https://issues.apache.org/jira/browse/HIVE-23389
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23389.01.patch, HIVE-23389.patch
>
>
> I have not been able to reproduce the issue in latest master. However, this
> could potentially happen since Filter creation has a check on whether the
> expression is flat
> ([here|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Filter.java#L74])
> and Filter merge does not flatten an expression when it is created.
> {noformat}
> java.lang.AssertionError: AND(=($3, 100), OR(OR(null, IS NOT
> NULL(CAST(100):INTEGER)), =(CAST(100):INTEGER, CAST(200):INTEGER)))
> at org.apache.calcite.rel.core.Filter.(Filter.java:74)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.(HiveFilter.java:39)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(HiveRelFactories.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder.filter(HiveRelBuilder.java:99)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1055)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:81)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23389) FilterMergeRule can lead to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-23389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23389:
---
Attachment: (was: HIVE-23389.01.patch)
> FilterMergeRule can lead to AssertionError
> --
>
> Key: HIVE-23389
> URL: https://issues.apache.org/jira/browse/HIVE-23389
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23389.01.patch, HIVE-23389.patch
>
>
> I have not been able to reproduce the issue in latest master. However, this
> could potentially happen since Filter creation has a check on whether the
> expression is flat
> ([here|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Filter.java#L74])
> and Filter merge does not flatten an expression when it is created.
> {noformat}
> java.lang.AssertionError: AND(=($3, 100), OR(OR(null, IS NOT
> NULL(CAST(100):INTEGER)), =(CAST(100):INTEGER, CAST(200):INTEGER)))
> at org.apache.calcite.rel.core.Filter.(Filter.java:74)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.(HiveFilter.java:39)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(HiveRelFactories.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder.filter(HiveRelBuilder.java:99)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1055)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:81)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23085) LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache
[ https://issues.apache.org/jira/browse/HIVE-23085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102957#comment-17102957 ] David Mollitor commented on HIVE-23085: --- Instead of a "comma separated list," for usability, can it just be a single directory in the configuration and all sub-directories in there hold the mount points? > LLAP: Support Multiple NVMe-SSD disk Locations While Using SSD Cache > > > Key: HIVE-23085 > URL: https://issues.apache.org/jira/browse/HIVE-23085 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23085.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently we can configure only one SSD location while using SSD cache in > LLAP. This highly undermines the capacity of some machines to use its disk > capacity to the fullest. For example *AWS* provides *r5d.4x large* series > which comes with *2 * 300 GB NVme SSD disk* with the current design only one > of the mounted *NVme SSD* disk can be used for caching. Hence adding support > for caching data at multiple ssd mounted locations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23389) FilterMergeRule can lead to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-23389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23389:
---
Attachment: HIVE-23389.01.patch
> FilterMergeRule can lead to AssertionError
> --
>
> Key: HIVE-23389
> URL: https://issues.apache.org/jira/browse/HIVE-23389
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23389.01.patch, HIVE-23389.patch
>
>
> I have not been able to reproduce the issue in latest master. However, this
> could potentially happen since Filter creation has a check on whether the
> expression is flat
> ([here|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/Filter.java#L74])
> and Filter merge does not flatten an expression when it is created.
> {noformat}
> java.lang.AssertionError: AND(=($3, 100), OR(OR(null, IS NOT
> NULL(CAST(100):INTEGER)), =(CAST(100):INTEGER, CAST(200):INTEGER)))
> at org.apache.calcite.rel.core.Filter.(Filter.java:74)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.(HiveFilter.java:39)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(HiveRelFactories.java:126)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelBuilder.filter(HiveRelBuilder.java:99)
> at org.apache.calcite.tools.RelBuilder.filter(RelBuilder.java:1055)
> at
> org.apache.calcite.rel.rules.FilterMergeRule.onMatch(FilterMergeRule.java:81)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23425) HS2 disconnect should not affect DDL operation status
[
https://issues.apache.org/jira/browse/HIVE-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam An updated HIVE-23425:
--
Description:
I observe in a cluster that HS2 connection to HMS was timed out.
{code:java}
// code placeholder
{code}
2020-05-07 19:28:46,718 WARN
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: MetaStoreClient lost connection.
Attempting to reconnect (1 of 1) after 1s. dropTable
org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
~[hive-exec-3.1.3000.7.1.0.0-714.jar:3.1.3000.7.1.0.0-714] at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
~[hive-exec-3.1.3000.7.1.0.0-714.jar:3.1.3000.7.1.0.0-714] at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374)
~[hive-exec-3.1.3000.7.1.0.0-714.jar:3.1.3000.7.1.0.0-714] at
org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451)
~[hive-exec-3.1.3000.7.1.0.0-714.jar:3.1.3000.7.1.0.0-714] at
org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433)
~[hive-exec-3.1.3000.7.1.0.0-714.jar:3.1.3000.7.1.0.0-714] at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
~[hive-exec-3.1.3000.7.1.0.0-714.jar:3.1.3000.7.1.0.0-714]
2020-05-07 19:28:47,718 INFO
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: RetryingMetaStoreClient trying
reconnect as
hive/ascend-px-ha2.ascend-p.a8du-8mgv.cloudera.s...@ascend-p.a8du-8mgv.cloudera.site
(auth:KERBEROS)
2020-05-07 19:28:47,719 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: Closed a connection to metastore,
current connections: 29
2020-05-07 19:28:47,719 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: Trying to connect to metastore
with URI thrift://ascend-px-master4.ascend-p.a8du-8mgv.cloudera.site:9083
2020-05-07 19:28:47,719 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: HMSC::open(): Could not find
delegation token. Creating KERBEROS-based thrift connection.
2020-05-07 19:28:47,724 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: Opened a connection to metastore,
current connections: 30
2020-05-07 19:28:47,725 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: Connected to metastore.
2020-05-07 19:28:47,730 INFO
org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook:
[HiveServer2-Background-Pool: Thread-21790]: Received post-hook notification
for: hive_20200507192346_a1a0bad2-ca4f-4cd0-a0e7-22415c7050f0
2020-05-07 19:28:47,731 INFO
org.apache.hive.service.cli.operation.OperationManager:
[d900945d-9eb5-4b63-8ce9-c1166bc2474a HiveServer2-HttpHandler-Pool:
Thread-19285]: Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=db40b2c8-d291-4f34-9809-24b6300ee45c]
2020-05-07 19:28:47,731 INFO org.apache.hadoop.hive.common.LogUtils:
[d900945d-9eb5-4b63-8ce9-c1166bc2474a HiveServer2-HttpHandler-Pool:
Thread-19285]: Thread context registration is done.
2020-05-07 19:28:47,731 INFO org.apache.hive.service.cli.operation.Operation:
[d900945d-9eb5-4b63-8ce9-c1166bc2474a HiveServer2-HttpHandler-Pool:
Thread-19285]: [opType=EXECUTE_STATEMENT,
queryId=hive_20200507192847_08dbb746-b38f-405c-a92c-f6d2dd02b6f8,
startTime=1588879727730, sessionId=d900945d-9eb5-4b63-8ce9-c1166bc2474a,
createTime=1588879461427, userName=hive, ipAddress=10.28.140.56]
2020-05-07 19:28:47,732 INFO org.apache.hadoop.hive.ql.Driver:
[d900945d-9eb5-4b63-8ce9-c1166bc2474a HiveServer2-HttpHandler-Pool:
Thread-19285]: Compiling
command(queryId=hive_20200507192847_08dbb746-b38f-405c-a92c-f6d2dd02b6f8): repl
dump `*` from 1501982 with ('hive.repl.dump.metadata.only'='true',
'hive.repl.dump.include.acid.tables'='true')
2020-05-07 19:28:47,732 INFO org.apache.hadoop.hive.ql.Driver:
[HiveServer2-Background-Pool: Thread-21790]: Completed executing
command(queryId=hive_20200507192346_a1a0bad2-ca4f-4cd0-a0e7-22415c7050f0); Time
taken: 301.177 seconds
2020-05-07 19:28:47,732 INFO org.apache.hadoop.hive.ql.Driver:
[HiveServer2-Background-Pool: Thread-21790]: OK
2020-05-07 19:28:47,732 INFO org.apache.hadoop.hive.ql.lockmgr.DbTxnManager:
[HiveServer2-Background-Pool: Thread-21790]: Stopped heartbeat for query:
hive_20200507192346_a1a0bad2-ca4f-4cd0-a0e7-22415c7050f0
2020-05-07 19:28:47,738 INFO org.apache.hadoop.hive.common.LogUtils:
[HiveServer2-Background-Pool: Thread-21790]: Unregistered logging context.
However, on HMS, the drop table operation has not been completed till 5 minutes
later.
[jira] [Assigned] (HIVE-23425) HS2 disconnect should not affect DDL operation status
[ https://issues.apache.org/jira/browse/HIVE-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-23425: - > HS2 disconnect should not affect DDL operation status > - > > Key: HIVE-23425 > URL: https://issues.apache.org/jira/browse/HIVE-23425 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Minor > > I observe in a cluster that HS2 connection to HMS was timed out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23337) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 2
[
https://issues.apache.org/jira/browse/HIVE-23337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102935#comment-17102935
]
Hive QA commented on HIVE-23337:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002398/HIVE-23337.04.patch
{color:green}SUCCESS:{color} +1 due to 18 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17240 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConcurrentLineage (batchId=218)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22210/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22210/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22210/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002398 - PreCommit-HIVE-Build
> Move q tests to TestMiniLlapLocal from TestCliDriver where the output is
> different, batch 2
> ---
>
> Key: HIVE-23337
> URL: https://issues.apache.org/jira/browse/HIVE-23337
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23337.01.patch, HIVE-23337.02.patch,
> HIVE-23337.03.patch, HIVE-23337.04.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23316) Add tests to cover database managed location related DDL and fix minor issues
[ https://issues.apache.org/jira/browse/HIVE-23316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102924#comment-17102924 ] Naveen Gangam commented on HIVE-23316: -- [~mgergely] In my initial implementation I had separate *Operation for the ManagedLocation but I changed it as I thought there was redundant code and it was more efficient to reuse the existing classes for the Location. So there was some thought put into it. Was it causing an issue that we had to add new classes? If there is no issue, I think it might be better if we reused the same set of classes. The check to ensure both of them were not the same was certainly an item I had in the doc, but couldnt get to in the initial implementation. Thanks > Add tests to cover database managed location related DDL and fix minor issues > - > > Key: HIVE-23316 > URL: https://issues.apache.org/jira/browse/HIVE-23316 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23316.01.patch > > > Database managed location was recently introduced, but no tests were added to > cover it. also the following issues were fixed: > * ALTER DATABASE ... SET MANAGEDLOCATION ... commands were not handled in a > separate path as it should, as in DDL each command type have their own > Analyzer, Desc, and Operation class > * in case of setting the LOCATION or the MANAGEDLOCATION the location was > not getting qualified as in the CREATE DATABASE command > * in case of setting the LOCATION or the MANAGEDLOCATION it was not checked > if this modification makes the two the same > * some minor checkstyle issues were fixed as well > Also the DDL documentation was not modified. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23424) Remove Dependency on Log4J from hive-shims-common
[ https://issues.apache.org/jira/browse/HIVE-23424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23424: -- Status: Patch Available (was: Open) > Remove Dependency on Log4J from hive-shims-common > - > > Key: HIVE-23424 > URL: https://issues.apache.org/jira/browse/HIVE-23424 > Project: Hive > Issue Type: Improvement > Components: Shims >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23424.1.patch > > > The project uses SLF4J but not the log4j specific libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23424) Remove Dependency on Log4J from hive-shims-common
[ https://issues.apache.org/jira/browse/HIVE-23424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23424: -- Component/s: Shims > Remove Dependency on Log4J from hive-shims-common > - > > Key: HIVE-23424 > URL: https://issues.apache.org/jira/browse/HIVE-23424 > Project: Hive > Issue Type: Improvement > Components: Shims >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23424.1.patch > > > The project uses SLF4J but not the log4j specific libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23424) Remove Dependency on Log4J
[ https://issues.apache.org/jira/browse/HIVE-23424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23424: -- Attachment: HIVE-23424.1.patch > Remove Dependency on Log4J > -- > > Key: HIVE-23424 > URL: https://issues.apache.org/jira/browse/HIVE-23424 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23424.1.patch > > > The project uses SLF4J but not the log4j specific libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23424) Remove Dependency on Log4J from hive-shims-common
[ https://issues.apache.org/jira/browse/HIVE-23424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23424: -- Summary: Remove Dependency on Log4J from hive-shims-common (was: Remove Dependency on Log4J) > Remove Dependency on Log4J from hive-shims-common > - > > Key: HIVE-23424 > URL: https://issues.apache.org/jira/browse/HIVE-23424 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23424.1.patch > > > The project uses SLF4J but not the log4j specific libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23424) Remove Dependency on Log4J
[ https://issues.apache.org/jira/browse/HIVE-23424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-23424: - > Remove Dependency on Log4J > -- > > Key: HIVE-23424 > URL: https://issues.apache.org/jira/browse/HIVE-23424 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23424.1.patch > > > The project uses SLF4J but not the log4j specific libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23337) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 2
[
https://issues.apache.org/jira/browse/HIVE-23337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102903#comment-17102903
]
Hive QA commented on HIVE-23337:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
2s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 3m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22210/dev-support/hive-personality.sh
|
| git revision | master / 134f3b2 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22210/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move q tests to TestMiniLlapLocal from TestCliDriver where the output is
> different, batch 2
> ---
>
> Key: HIVE-23337
> URL: https://issues.apache.org/jira/browse/HIVE-23337
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23337.01.patch, HIVE-23337.02.patch,
> HIVE-23337.03.patch, HIVE-23337.04.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23410) ACID: Improve the delete and update operations to avoid the move step
[
https://issues.apache.org/jira/browse/HIVE-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102881#comment-17102881
]
Hive QA commented on HIVE-23410:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002394/HIVE-23410.1.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 58 failed/errored test(s), 17246 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=8)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=104)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_stats2]
(batchId=88)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_subquery]
(batchId=82)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=65)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[create_transactional_full_acid]
(batchId=110)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_whole_partition]
(batchId=59)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_3]
(batchId=109)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=86)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=65)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_into_default_keyword]
(batchId=54)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[masking_acid_no_masking]
(batchId=69)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=64)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin_hint]
(batchId=74)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sort_acid]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sqlmerge]
(batchId=114)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sqlmerge_stats]
(batchId=111)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=44)
org.apache.hadoop.hive.ql.TestTxnCommands2.testBucketCodec (batchId=271)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion2
(batchId=271)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3
(batchId=271)
org.apache.hadoop.hive.ql.TestTxnCommands2.writeBetweenWorkerAndCleaner
(batchId=271)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testBucketCodec
(batchId=294)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion2
(batchId=294)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3
(batchId=294)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.writeBetweenWorkerAndCleaner
(batchId=294)
org.apache.hadoop.hive.ql.TestTxnCommands3.testRenameTable (batchId=293)
org.apache.hadoop.hive.ql.TestTxnCommands3.testSdpoBucketed (batchId=293)
org.apache.hadoop.hive.ql.TestTxnConcatenate.testConcatenate (batchId=255)
org.apache.hadoop.hive.ql.TestTxnConcatenate.testConcatenatePart (batchId=255)
org.apache.hadoop.hive.ql.TestTxnExIm.testImport (batchId=275)
org.apache.hadoop.hive.ql.TestTxnExIm.testImportNoTarget (batchId=275)
org.apache.hadoop.hive.ql.TestTxnExIm.testImportVectorized (batchId=275)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdate (batchId=245)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdateVectorized (batchId=245)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNoBuckets (batchId=272)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversion02 (batchId=272)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversionMultiBucket
(batchId=272)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testNoBuckets (batchId=273)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testToAcidConversion02
(batchId=273)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testToAcidConversionMultiBucket
(batchId=273)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesBootstrap
(batchId=199)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidTablesBootstrap
(batchId=201)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosIncrementalLoadAcidTables.testAcidTableIncrementalReplication
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosACID.testForParallelBootstrapLoad
(batchId=184)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosACID.testMetadataOnlyDump
(batchId=184)
[jira] [Updated] (HIVE-23351) Ranger Replication Scheduling
[ https://issues.apache.org/jira/browse/HIVE-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23351: --- Attachment: HIVE-23351.08.patch Status: Patch Available (was: In Progress) > Ranger Replication Scheduling > - > > Key: HIVE-23351 > URL: https://issues.apache.org/jira/browse/HIVE-23351 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch, > HIVE-23351.03.patch, HIVE-23351.04.patch, HIVE-23351.05.patch, > HIVE-23351.06.patch, HIVE-23351.07.patch, HIVE-23351.08.patch > > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23351) Ranger Replication Scheduling
[ https://issues.apache.org/jira/browse/HIVE-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23351: --- Status: In Progress (was: Patch Available) > Ranger Replication Scheduling > - > > Key: HIVE-23351 > URL: https://issues.apache.org/jira/browse/HIVE-23351 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch, > HIVE-23351.03.patch, HIVE-23351.04.patch, HIVE-23351.05.patch, > HIVE-23351.06.patch, HIVE-23351.07.patch, HIVE-23351.08.patch > > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23423) Hash aggregation is always disabled in vectorized execution of grouping set queries
[ https://issues.apache.org/jira/browse/HIVE-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Vijayaraghavan reassigned HIVE-23423: --- Assignee: Gopal Vijayaraghavan > Hash aggregation is always disabled in vectorized execution of grouping set > queries > --- > > Key: HIVE-23423 > URL: https://issues.apache.org/jira/browse/HIVE-23423 > Project: Hive > Issue Type: Bug > Components: Hive, Operators, Query Processor >Affects Versions: 4.0.0 >Reporter: Nita Dembla >Assignee: Gopal Vijayaraghavan >Priority: Major > Labels: Performance > > https://issues.apache.org/jira/browse/HIVE-23356 fixed the issue with > disabling hash aggregation on grouping set queries. Need a fix for > VectorGroupbyOperator operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23410) ACID: Improve the delete and update operations to avoid the move step
[
https://issues.apache.org/jira/browse/HIVE-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102858#comment-17102858
]
Hive QA commented on HIVE-23410:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
51s{color} | {color:red} ql: The patch generated 12 new + 1374 unchanged - 6
fixed = 1386 total (was 1380) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
56s{color} | {color:red} ql generated 3 new + 1526 unchanged - 1 fixed = 1529
total (was 1527) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Possible null pointer dereference of
AcidInputFormat$DeltaMetaData.statementAndBucketToAttemptId in
org.apache.hadoop.hive.ql.io.AcidInputFormat$DeltaMetaData.write(DataOutput)
Dereferenced at
AcidInputFormat.java:AcidInputFormat$DeltaMetaData.statementAndBucketToAttemptId
in
org.apache.hadoop.hive.ql.io.AcidInputFormat$DeltaMetaData.write(DataOutput)
Dereferenced at AcidInputFormat.java:[line 184] |
| | Possible null pointer dereference of bucketToAttemptId in
org.apache.hadoop.hive.ql.io.AcidInputFormat$DeltaMetaData.write(DataOutput)
Dereferenced at AcidInputFormat.java:bucketToAttemptId in
org.apache.hadoop.hive.ql.io.AcidInputFormat$DeltaMetaData.write(DataOutput)
Dereferenced at AcidInputFormat.java:[line 188] |
| | Nullcheck of table_desc at line 8067 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createFileSinkDesc(String,
TableDesc, Partition, Path, int, boolean, boolean, boolean, Path,
SemanticAnalyzer$SortBucketRSCtx, DynamicPartitionCtx, ListBucketingCtx,
RowSchema, boolean, Table, Long, boolean, Integer, QB, boolean,
AcidUtils$Operation) At SemanticAnalyzer.java:8067 of value previously
dereferenced in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createFileSinkDesc(String,
TableDesc, Partition, Path, int, boolean, boolean, boolean, Path,
SemanticAnalyzer$SortBucketRSCtx, DynamicPartitionCtx, ListBucketingCtx,
RowSchema, boolean, Table, Long, boolean, Integer, QB, boolean,
AcidUtils$Operation) At SemanticAnalyzer.java:[line 8060] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22209/dev-support/hive-personality.sh
|
| git revision | master / c419a7b |
[jira] [Updated] (HIVE-23351) Ranger Replication Scheduling
[ https://issues.apache.org/jira/browse/HIVE-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23351: --- Attachment: HIVE-23351.07.patch Status: Patch Available (was: In Progress) > Ranger Replication Scheduling > - > > Key: HIVE-23351 > URL: https://issues.apache.org/jira/browse/HIVE-23351 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch, > HIVE-23351.03.patch, HIVE-23351.04.patch, HIVE-23351.05.patch, > HIVE-23351.06.patch, HIVE-23351.07.patch > > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23351) Ranger Replication Scheduling
[ https://issues.apache.org/jira/browse/HIVE-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23351: --- Status: In Progress (was: Patch Available) > Ranger Replication Scheduling > - > > Key: HIVE-23351 > URL: https://issues.apache.org/jira/browse/HIVE-23351 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch, > HIVE-23351.03.patch, HIVE-23351.04.patch, HIVE-23351.05.patch, > HIVE-23351.06.patch, HIVE-23351.07.patch > > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23338) Bump jackson-databind version up to 2.9.10.4
[
https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102842#comment-17102842
]
Hive QA commented on HIVE-23338:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
8s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 12m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
23s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22208/dev-support/hive-personality.sh
|
| git revision | master / c419a7b |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22208/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore common ql hcatalog/core
hcatalog/server-extensions hcatalog/webhcat/svr . itests/qtest-druid
testutils/ptest2 U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22208/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Bump jackson-databind version up to 2.9.10.4
>
>
> Key: HIVE-23338
> URL: https://issues.apache.org/jira/browse/HIVE-23338
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23338.01.patch, HIVE-23338.01.patch,
> HIVE-23338.01.patch, HIVE-23338.01.patch, HIVE-23338.02.patch,
> HIVE-23338.03.patch, HIVE-23338.03.patch
>
>
> And exclude a transitive dependency on
> com.fasterxml.jackson.core:jackson-databind:2.6.5
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23338) Bump jackson-databind version up to 2.9.10.4
[
https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102834#comment-17102834
]
Hive QA commented on HIVE-23338:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002382/HIVE-23338.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17240 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[6]
(batchId=142)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22208/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22208/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22208/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002382 - PreCommit-HIVE-Build
> Bump jackson-databind version up to 2.9.10.4
>
>
> Key: HIVE-23338
> URL: https://issues.apache.org/jira/browse/HIVE-23338
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23338.01.patch, HIVE-23338.01.patch,
> HIVE-23338.01.patch, HIVE-23338.01.patch, HIVE-23338.02.patch,
> HIVE-23338.03.patch, HIVE-23338.03.patch
>
>
> And exclude a transitive dependency on
> com.fasterxml.jackson.core:jackson-databind:2.6.5
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23351) Ranger Replication Scheduling
[
https://issues.apache.org/jira/browse/HIVE-23351?focusedWorklogId=432207=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-432207
]
ASF GitHub Bot logged work on HIVE-23351:
-
Author: ASF GitHub Bot
Created on: 08/May/20 17:58
Start Date: 08/May/20 17:58
Worklog Time Spent: 10m
Work Description: aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422285774
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##
@@ -103,13 +105,43 @@ public int execute() {
}
work.setRootTask(this);
this.parentTasks = null;
+if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data");
+ try {
+initiateAuthorizationLoadTask(work.dumpDirectory);
+ } catch (Exception e) {
+LOG.error("failed", e);
+setException(e);
+return ErrorMsg.getErrorMsg(e.getMessage()).getErrorCode();
+ }
+}
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
}
+ private boolean shouldLoadAuthorizationMetadata() {
+return
conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_AUTHORIZATION_METADATA);
+ }
+
+ private void initiateAuthorizationLoadTask(String hiveDumpDirectory) throws
SemanticException {
+if
(RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE)))
{
+ Path rangerLoadRoot = new Path(new Path(hiveDumpDirectory).getParent(),
ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Importing Authorization Metadata from {} ", rangerLoadRoot);
+ RangerLoadWork rangerLoadWork = new RangerLoadWork(rangerLoadRoot,
work.getSourceDbName(), work.dbNameToLoadIn);
+ Task rangerLoadTask = TaskFactory.get(rangerLoadWork,
conf);
+ if (childTasks == null) {
+childTasks = new ArrayList<>();
+ }
+ childTasks.add(rangerLoadTask);
Review comment:
Separate tasks are created.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 432207)
Time Spent: 2h 40m (was: 2.5h)
> Ranger Replication Scheduling
> -
>
> Key: HIVE-23351
> URL: https://issues.apache.org/jira/browse/HIVE-23351
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch,
> HIVE-23351.03.patch, HIVE-23351.04.patch, HIVE-23351.05.patch,
> HIVE-23351.06.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102793#comment-17102793
]
David Mollitor edited comment on HIVE-23182 at 5/8/20, 5:57 PM:
[~jfs] Thanks for pointing that out for me and for fixing it.
It's probably safe to assume that parsing always requires a context now. With
that in mind, I think you should try removing this method:
[https://github.com/apache/hive/blob/c419a7b99a69d1c1bede9394fc361591e57f4de6/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L67]
And enforcing a non-null {{Objects.requireNonNull}} and see what happens in the
qtests:
https://github.com/apache/hive/blob/c419a7b99a69d1c1bede9394fc361591e57f4de6/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L77
was (Author: belugabehr):
[~jfs] Thanks for pointing that out for me and for fixing it.
It's probably safe to assume that parsing always requires a context now. With
that in mind, I think you should try removing this method:
[https://github.com/apache/hive/blob/c419a7b99a69d1c1bede9394fc361591e57f4de6/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L67]
And enforcing a non-null {{Objects.requireNonNull}} and see what happens in the
qtests
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102793#comment-17102793
]
David Mollitor commented on HIVE-23182:
---
[~jfs] Thanks for pointing that out for me.
It's probably safe to assume that parsing always requires a context now. With
that in mind, I think you should try removing this method:
[https://github.com/apache/hive/blob/c419a7b99a69d1c1bede9394fc361591e57f4de6/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L67]
And enforcing a non-null {{Objects.requireNonNull}} and see what happens in the
qtests
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102793#comment-17102793
]
David Mollitor edited comment on HIVE-23182 at 5/8/20, 5:56 PM:
[~jfs] Thanks for pointing that out for me and for fixing it.
It's probably safe to assume that parsing always requires a context now. With
that in mind, I think you should try removing this method:
[https://github.com/apache/hive/blob/c419a7b99a69d1c1bede9394fc361591e57f4de6/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L67]
And enforcing a non-null {{Objects.requireNonNull}} and see what happens in the
qtests
was (Author: belugabehr):
[~jfs] Thanks for pointing that out for me.
It's probably safe to assume that parsing always requires a context now. With
that in mind, I think you should try removing this method:
[https://github.com/apache/hive/blob/c419a7b99a69d1c1bede9394fc361591e57f4de6/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L67]
And enforcing a non-null {{Objects.requireNonNull}} and see what happens in the
qtests
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23351) Ranger Replication Scheduling
[
https://issues.apache.org/jira/browse/HIVE-23351?focusedWorklogId=432205=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-432205
]
ASF GitHub Bot logged work on HIVE-23351:
-
Author: ASF GitHub Bot
Created on: 08/May/20 17:55
Start Date: 08/May/20 17:55
Worklog Time Spent: 10m
Work Description: aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422283834
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ranger/RangerRestClientImpl.java
##
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.ranger;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.sun.jersey.api.client.Client;
+import com.sun.jersey.api.client.ClientResponse;
+import com.sun.jersey.api.client.WebResource;
+import com.sun.jersey.api.client.config.ClientConfig;
+import com.sun.jersey.api.client.config.DefaultClientConfig;
+import com.sun.jersey.multipart.FormDataMultiPart;
+import com.sun.jersey.multipart.MultiPart;
+import com.sun.jersey.multipart.file.StreamDataBodyPart;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.eclipse.jetty.util.MultiPartWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.core.MediaType;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.File;
+import java.io.Reader;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.ArrayList;
+
+/**
+ * RangerRestClientImpl to connect to Ranger and export policies.
+ */
+public class RangerRestClientImpl implements RangerRestClient {
+ private static final Logger LOG =
LoggerFactory.getLogger(RangerRestClientImpl.class);
+ private static final String RANGER_REST_URL_EXPORTJSONFILE =
"/service/plugins/policies/exportJson";
+ private static final String RANGER_REST_URL_IMPORTJSONFILE =
+ "/service/plugins/policies/importPoliciesFromFile?updateIfExists=true";
+
+ public RangerExportPolicyList exportRangerPolicies(String
sourceRangerEndpoint,
+ String dbName, String
rangerHiveServiceName) throws Exception {
+LOG.info("Ranger endpoint for cluster " + sourceRangerEndpoint);
+ClientResponse clientResp = null;
+String uri = null;
+if (!StringUtils.isEmpty(rangerHiveServiceName)) {
+ uri = RANGER_REST_URL_EXPORTJSONFILE + "?serviceName=" +
rangerHiveServiceName + "="
+ + dbName + ":database=" + dbName
+ +
"=hive=self_or_ancestor=full";
+}
+if (sourceRangerEndpoint.endsWith("/")) {
+ sourceRangerEndpoint = StringUtils.removePattern(sourceRangerEndpoint,
"/+$");
+}
+String url = sourceRangerEndpoint + (uri != null ? (uri.startsWith("/") ?
uri : ("/" + uri)) : "");
+LOG.debug("URL to export policies from source Ranger: {}", url);
+RangerExportPolicyList rangerExportPolicyList = new
RangerExportPolicyList();
+WebResource.Builder builder = getRangerResourceBuilder(url);
+clientResp = builder.get(ClientResponse.class);
+
+String response = null;
+if (clientResp != null) {
+ if (clientResp.getStatus() == HttpServletResponse.SC_OK) {
+Gson gson = new GsonBuilder().create();
+response = clientResp.getEntity(String.class);
+LOG.debug("Response received for ranger export {} ", response);
+if (StringUtils.isNotEmpty(response)) {
+ rangerExportPolicyList = gson.fromJson(response,
RangerExportPolicyList.class);
+ return rangerExportPolicyList;
+}
+ }
[jira] [Assigned] (HIVE-22415) Upgrade to Java 11
[ https://issues.apache.org/jira/browse/HIVE-22415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-22415: - Assignee: David Mollitor > Upgrade to Java 11 > -- > > Key: HIVE-22415 > URL: https://issues.apache.org/jira/browse/HIVE-22415 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Critical > > Upgrade Hive to Java JDK 11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102779#comment-17102779
]
John Sherman commented on HIVE-23182:
-
[~belugabehr] I think that example is the one my patch fixed.
If we make changes around this area we might also want to fix or inform Apache
Impala as we do so.
[https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java#L143]
They invoke the HiveLexer but do not set HiveConf. So if we make it a
Precondition, usages like this will fail.
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23344) Bump org.scala-lang:scala-compiler version to 2.12.4
[
https://issues.apache.org/jira/browse/HIVE-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102773#comment-17102773
]
Hive QA commented on HIVE-23344:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002384/HIVE-23344.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17240 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.spark.TestHiveSparkClient.testSetJobGroupAndDescription
(batchId=290)
org.apache.hadoop.hive.ql.exec.spark.TestSparkInvalidFileFormat.readTextFileAsParquet
(batchId=290)
org.apache.hadoop.hive.ql.exec.spark.TestSparkPlan.testSetRDDCallSite
(batchId=290)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22207/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22207/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22207/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002384 - PreCommit-HIVE-Build
> Bump org.scala-lang:scala-compiler version to 2.12.4
>
>
> Key: HIVE-23344
> URL: https://issues.apache.org/jira/browse/HIVE-23344
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23344.01.patch, HIVE-23344.01.patch,
> HIVE-23344.02.patch, HIVE-23344.03.patch
>
>
> And bump up spark version, as 2.3.3 is not compatible with scala 2.12
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102771#comment-17102771
]
David Mollitor commented on HIVE-23182:
---
I shared one example here:
https://issues.apache.org/jira/browse/HIVE-23182?focusedCommentId=17081629=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17081629
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102769#comment-17102769
]
David Mollitor edited comment on HIVE-23182 at 5/8/20, 5:30 PM:
I tried to do this, I had started on some work to always pass the state.
Unfortunately, it was not very easy unfortunate (that's why I tried using a
different default value), but it needs to be done.
was (Author: belugabehr):
I tried to do this, to always pass the state. It was not very easy
unfortunate, but it needs to be done.
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102769#comment-17102769
]
David Mollitor edited comment on HIVE-23182 at 5/8/20, 5:30 PM:
I tried to do this, I had started on some work to always pass the state.
Unfortunately, it was not very easy (that's why I tried using a different
default value), but it needs to be done.
was (Author: belugabehr):
I tried to do this, I had started on some work to always pass the state.
Unfortunately, it was not very easy unfortunate (that's why I tried using a
different default value), but it needs to be done.
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102769#comment-17102769
]
David Mollitor commented on HIVE-23182:
---
I tried to do this, to always pass the state. It was not very easy
unfortunate, but it needs to be done.
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102766#comment-17102766
]
Jesus Camacho Rodriguez commented on HIVE-23182:
{quote}
Possibly the easiest thing is to use Predicate.checkState and make sure the
configuration is not 'null'
{quote}
It makes sense, we thought about the same.
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102767#comment-17102767
]
David Mollitor commented on HIVE-23182:
---
[~jcamachorodriguez] *{{Preconditions.checkState}} (my mistake)
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102763#comment-17102763
]
David Mollitor edited comment on HIVE-23182 at 5/8/20, 5:28 PM:
[~jfs],
[~jcamachorodriguez] set me straight on this. It's actually not possible to do
a default value here. My [incorrect] understanding was that "column" mode was a
super-set of "none" mode. Actually, they are a bit different and not really
compatible in this way. The context always needs to be present and the correct
mode needs to be applied.
Possibly the easiest thing is to use {{Preconditions.checkState}} and make sure
the configuration is not 'null'. If it is, an error will be generated during
qtests and you'll get a stack trace showing you where the issues are.
And now there are three modes (none, column, standard), so one can't just have
a single default return value here.
was (Author: belugabehr):
[~jfs],
[~jcamachorodriguez] set me straight on this. It's actually not possible to do
a default value here. My [incorrect] understanding was that "column" mode was a
super-set of "none" mode. Actually, they are a bit different and not really
compatible in this way. The context always needs to be present and the correct
mode needs to be applied.
Possibly the easiest thing is to use {{Predicate.checkState}} and make sure the
configuration is not 'null'. If it is, an error will be generated during
qtests and you'll get a stack trace showing you where the issues are.
And now there are three modes (none, column, standard), so one can't just have
a single default return value here.
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102763#comment-17102763
]
David Mollitor edited comment on HIVE-23182 at 5/8/20, 5:27 PM:
[~jfs],
[~jcamachorodriguez] set me straight on this. It's actually not possible to do
a default value here. My [incorrect] understanding was that "column" mode was a
super-set of "none" mode. Actually, they are a bit different and not really
compatible in this way. The context always needs to be present and the correct
mode needs to be applied.
Possibly the easiest thing is to use {{Predicate.checkState}} and make sure the
configuration is not 'null'. If it is, an error will be generated during
qtests and you'll get a stack trace showing you where the issues are.
And now there are three modes (none, column, standard), so one can't just have
a single default return value here.
was (Author: belugabehr):
[~jfs],
[~jcamachorodriguez] set me straight on this. It's actually not possible to do
a default value here. My [incorrect] understanding was that "column" mode was a
super-set of "none" mode. Actually, they are a bit different and not really
compatible in this way. The context always needs to be present and the correct
mode needs to be applied.
Possibly the easiest thing is to use {{Predicate.checkState}} and make sure the
configuration is not 'null'. If it is, an error will be generated during
qtests and you'll get a stack trace showing you where the issues are.
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102765#comment-17102765
]
Jesus Camacho Rodriguez commented on HIVE-23182:
I would rather remove the check on null unless it is not possible for some
reason?
{code}
if(hiveConf == null){
// This line here: No context provided? Feature is disable.
return false;
}
{code}
My main concern is that changing the default value may fix this particular
issue, but how do we make sure that other feature / code that relies on the
correct value from HiveConf does not hit similar issue? Thoughts?
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102763#comment-17102763
]
David Mollitor commented on HIVE-23182:
---
[~jfs],
[~jcamachorodriguez] set me straight on this. It's actually not possible to do
a default value here. My [incorrect] understanding was that "column" mode was a
super-set of "none" mode. Actually, they are a bit different and not really
compatible in this way. The context always needs to be present and the correct
mode needs to be applied.
Possibly the easiest thing is to use {{Predicate.checkState}} and make sure the
configuration is not 'null'. If it is, an error will be generated during
qtests and you'll get a stack trace showing you where the issues are.
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23344) Bump org.scala-lang:scala-compiler version to 2.12.4
[
https://issues.apache.org/jira/browse/HIVE-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102760#comment-17102760
]
Hive QA commented on HIVE-23344:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22207/dev-support/hive-personality.sh
|
| git revision | master / c419a7b |
| Default Java | 1.8.0_111 |
| modules | C: kafka-handler . itests/qtest-druid U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22207/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Bump org.scala-lang:scala-compiler version to 2.12.4
>
>
> Key: HIVE-23344
> URL: https://issues.apache.org/jira/browse/HIVE-23344
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23344.01.patch, HIVE-23344.01.patch,
> HIVE-23344.02.patch, HIVE-23344.03.patch
>
>
> And bump up spark version, as 2.3.3 is not compatible with scala 2.12
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102759#comment-17102759
]
John Sherman commented on HIVE-23182:
-
[~jcamachorodriguez] I think it would be solved by my JIRA. There may be other
areas where Parse gets called without HiveConf that I did not have a chance to
closely analyze.
It may be useful to have a new JIRA for [~belugabehr]'s idea of making it
default to be true in the lexer since this is an easy mistake to make.
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102754#comment-17102754
]
Jesus Camacho Rodriguez commented on HIVE-23182:
[~belugabehr], has this been solved by HIVE-23355 ?
Cc [~jfs]
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23376) Avoid repeated SHA computation in GenericUDTFGetSplits for hive-exec jar
[ https://issues.apache.org/jira/browse/HIVE-23376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23376: Status: Open (was: Patch Available) > Avoid repeated SHA computation in GenericUDTFGetSplits for hive-exec jar > > > Key: HIVE-23376 > URL: https://issues.apache.org/jira/browse/HIVE-23376 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23376.2.patch, HIVE-23376.3.patch, > image-2020-05-06-16-37-48-615.png > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java#L706] > > > !image-2020-05-06-16-37-48-615.png|width=946,height=639! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23376) Avoid repeated SHA computation in GenericUDTFGetSplits for hive-exec jar
[ https://issues.apache.org/jira/browse/HIVE-23376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23376: Attachment: HIVE-23376.3.patch Status: Patch Available (was: Open) > Avoid repeated SHA computation in GenericUDTFGetSplits for hive-exec jar > > > Key: HIVE-23376 > URL: https://issues.apache.org/jira/browse/HIVE-23376 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23376.2.patch, HIVE-23376.3.patch, > image-2020-05-06-16-37-48-615.png > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java#L706] > > > !image-2020-05-06-16-37-48-615.png|width=946,height=639! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23416) Investigate why predicate modified at quotedid_skew.q, skewjoinopt10.q, skewjoinopt1.q, skewjoin_mapjoin7.q, skewjoin_mapjoin1.q, skewjoin_mapjoin10.q
[ https://issues.apache.org/jira/browse/HIVE-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23416: -- Description: Check [https://reviews.apache.org/r/72485/#comment309242] and [https://reviews.apache.org/r/72485/] for details. (was: Check [https://reviews.apache.org/r/72485/#comment309242] for details.) > Investigate why predicate modified at quotedid_skew.q, skewjoinopt10.q, > skewjoinopt1.q, skewjoin_mapjoin7.q, skewjoin_mapjoin1.q, skewjoin_mapjoin10.q > -- > > Key: HIVE-23416 > URL: https://issues.apache.org/jira/browse/HIVE-23416 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Priority: Major > > Check [https://reviews.apache.org/r/72485/#comment309242] and > [https://reviews.apache.org/r/72485/] for details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23416) Investigate why predicate modified at quotedid_skew.q, skewjoinopt10.q, skewjoinopt1.q, skewjoin_mapjoin7.q, skewjoin_mapjoin1.q, skewjoin_mapjoin10.q
[ https://issues.apache.org/jira/browse/HIVE-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23416: -- Summary: Investigate why predicate modified at quotedid_skew.q, skewjoinopt10.q, skewjoinopt1.q, skewjoin_mapjoin7.q, skewjoin_mapjoin1.q, skewjoin_mapjoin10.q (was: Investigate why predicate modified at quotedid_skew.q) > Investigate why predicate modified at quotedid_skew.q, skewjoinopt10.q, > skewjoinopt1.q, skewjoin_mapjoin7.q, skewjoin_mapjoin1.q, skewjoin_mapjoin10.q > -- > > Key: HIVE-23416 > URL: https://issues.apache.org/jira/browse/HIVE-23416 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Priority: Major > > Check [https://reviews.apache.org/r/72485/#comment309242] for details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23418) Investigate why msck command found different partitions at repair.q, msck_repair*, partition_discovery.q
[ https://issues.apache.org/jira/browse/HIVE-23418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23418: -- Summary: Investigate why msck command found different partitions at repair.q, msck_repair*, partition_discovery.q (was: Investigate why msck command found different partitions at repair.q, msck_repair*) > Investigate why msck command found different partitions at repair.q, > msck_repair*, partition_discovery.q > > > Key: HIVE-23418 > URL: https://issues.apache.org/jira/browse/HIVE-23418 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23376) Avoid repeated SHA computation in GenericUDTFGetSplits for hive-exec jar
[
https://issues.apache.org/jira/browse/HIVE-23376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102713#comment-17102713
]
Hive QA commented on HIVE-23376:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002359/HIVE-23376.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17240 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22206/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22206/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22206/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002359 - PreCommit-HIVE-Build
> Avoid repeated SHA computation in GenericUDTFGetSplits for hive-exec jar
>
>
> Key: HIVE-23376
> URL: https://issues.apache.org/jira/browse/HIVE-23376
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23376.2.patch, image-2020-05-06-16-37-48-615.png
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java#L706]
>
>
> !image-2020-05-06-16-37-48-615.png|width=946,height=639!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22967) Support hive.reloadable.aux.jars.path for Hive on Tez
[
https://issues.apache.org/jira/browse/HIVE-22967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102709#comment-17102709
]
Jason Dere commented on HIVE-22967:
---
I think this makes sense. +1
> Support hive.reloadable.aux.jars.path for Hive on Tez
> -
>
> Key: HIVE-22967
> URL: https://issues.apache.org/jira/browse/HIVE-22967
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Toshihiko Uchida
>Assignee: Toshihiko Uchida
>Priority: Minor
> Attachments: HIVE-22967.1.patch, HIVE-22967.2.patch
>
>
> The jars in hive.reloadable.aux.jars.path are not localized in Tez containers.
> As a result, any query utilizing those reloadable jars fails for Hive on Tez
> due to ClassNotFoundException.
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed,
> vertexName=Map 1, vertexId=vertex_1578856704640_0087_1_00, diagnostics=[Task
> failed, taskId=task_1578856704640_0087_1_00_01, diagnostics=[TaskAttempt
> 0 failed, info=[Error: Error while running task ( failure) :
> attempt_1578856704640_0087_1_00_01_0:java.lang.RuntimeException:
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:354)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.example.hive.udf.Lower
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.initializeOp(FilterOperator.java:71)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.initializeOp(VectorFilterOperator.java:83)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:573)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:525)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:386)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.initializeMapOperator(VectorMapOperator.java:591)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:317)
> ... 17 more
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.example.hive.udf.Lower
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:134)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.isStateful(FunctionRegistry.java:1492)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.(ExprNodeGenericFuncEvaluator.java:111)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:58)
> at
>
[jira] [Commented] (HIVE-23412) Filter applied in the "LEFT JOIN" condition is pulling a null batch on last reducer (map join vectorizer operator) and throwing exception as "Hive Runtime Error while p
[
https://issues.apache.org/jira/browse/HIVE-23412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102688#comment-17102688
]
kumar ravi commented on HIVE-23412:
---
select count(t.abc),
DATE(t.aaa) as ashdjahsdj,
t.bbb,
t.ccc,
sum(b.eee / 100) as amount,
(case
when b.eee / 100 >= 0 and b.eee / 100 <= 1000 then '0-1000'
when b.eee / 100 > 1000 and b.eee / 100 <= 2000 then '1001-2000'
when b.eee / 100 > 2000 then '2000 above' end) as slab,
b.eee,
b.p_eee,
g.eee,
g.lol,
h.how,
h.pg_acquirer
from www.ashdjahsdj t
JOIN www.shjhs b ON b.abc = t.abc
JOIN www.sdjsd g ON g.abc = t.abc
LEFT JOIN www.sdsd h ON t.abc = h.pp_txn_id and h.year = '2020'
where t.state = "COMPLETED"
and b.eee in ('asdasd', 'feffe', 'vfvfv', 'fdvfv','xdfdf')
and (t.ccc == 0
or ((t.ccc & shiftleft(1, 0) == 0)
and (t.ccc & shiftleft(1, 1) == 0))
)
and t.aaa >= '2020-04-01'
and t.aaa < '2020-05-01'
and t.year = '2020' and t.month = '04'
and b.year = '2020' and b.month = '04'
and g.year = '2020' and g.month = '04'
group by g.lol, b.eee, b.p_eee, g.eee, t.bbb, t.ccc, h.how, h.pg_acquirer,
DATE(t.aaa), case
when b.eee / 100 >= 0 and b.eee / 100 <= 1000 then '0-1000'
when b.eee / 100 > 1000 and b.eee / 100 <= 2000 then '1001-2000'
when b.eee / 100 > 2000 then '2000 above' end;
> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last
> reducer (map join vectorizer operator) and throwing exception as "Hive
> Runtime Error while processing vector batch" at the map join during the query
> execution.
> ---
>
> Key: HIVE-23412
> URL: https://issues.apache.org/jira/browse/HIVE-23412
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 3.1.0
>Reporter: kumar ravi
>Priority: Major
> Attachments: Query.rtf
>
>
> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last
> reducer (map join vectorizer operator) and throwing exception as "Hive
> Runtime Error while processing vector batch" at the map join during the query
> execution.
> {{ERROR : FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map
> 2, vertexId=vertex_1588769098286_8402_1_00Vertex re-running, vertexName=Map
> 1, vertexId=vertex_1588769098286_8402_1_02Vertex re-running, vertexName=Map
> 4, vertexId=vertex_1588769098286_8402_1_04Vertex failed, vertexName=Reducer
> 6, vertexId=vertex_1588769098286_8402_1_05, diagnostics=[Task failed,
> taskId=task_1588769098286_8402_1_05_000604, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1588769098286_8402_1_05_000604_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)}}{{
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)}}{{
> at java.security.AccessController.doPrivileged(Native Method)}}{{ at
> javax.security.auth.Subject.doAs(Subject.java:422)}}{{ at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)}}{{
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)}}{{
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)}}{{
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)}}{{
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)}}{{
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}{{
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}{{
> at java.lang.Thread.run(Thread.java:748)}}{{Caused by:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector
[jira] [Updated] (HIVE-23412) Filter applied in the "LEFT JOIN" condition is pulling a null batch on last reducer (map join vectorizer operator) and throwing exception as "Hive Runtime Error while pro
[
https://issues.apache.org/jira/browse/HIVE-23412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kumar ravi updated HIVE-23412:
--
Attachment: Query.rtf
> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last
> reducer (map join vectorizer operator) and throwing exception as "Hive
> Runtime Error while processing vector batch" at the map join during the query
> execution.
> ---
>
> Key: HIVE-23412
> URL: https://issues.apache.org/jira/browse/HIVE-23412
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 3.1.0
>Reporter: kumar ravi
>Priority: Major
> Attachments: Query.rtf
>
>
> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last
> reducer (map join vectorizer operator) and throwing exception as "Hive
> Runtime Error while processing vector batch" at the map join during the query
> execution.
> {{ERROR : FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map
> 2, vertexId=vertex_1588769098286_8402_1_00Vertex re-running, vertexName=Map
> 1, vertexId=vertex_1588769098286_8402_1_02Vertex re-running, vertexName=Map
> 4, vertexId=vertex_1588769098286_8402_1_04Vertex failed, vertexName=Reducer
> 6, vertexId=vertex_1588769098286_8402_1_05, diagnostics=[Task failed,
> taskId=task_1588769098286_8402_1_05_000604, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1588769098286_8402_1_05_000604_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)}}{{
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)}}{{
> at java.security.AccessController.doPrivileged(Native Method)}}{{ at
> javax.security.auth.Subject.doAs(Subject.java:422)}}{{ at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)}}{{
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)}}{{
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)}}{{
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)}}{{
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)}}{{
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}{{
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}{{
> at java.lang.Thread.run(Thread.java:748)}}{{Caused by:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:401)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:318)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)}}{{
> ... 16 more}}{{Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:503)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:392)}}{{
> ... 19 more}}{{Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> Unexpected exception from VectorMapJoinOperator : null}}{{ at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:539)}}{{
> at
>
[jira] [Comment Edited] (HIVE-23412) Filter applied in the "LEFT JOIN" condition is pulling a null batch on last reducer (map join vectorizer operator) and throwing exception as "Hive Runtime Error wh
[
https://issues.apache.org/jira/browse/HIVE-23412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102646#comment-17102646
]
kumar ravi edited comment on HIVE-23412 at 5/8/20, 3:51 PM:
[~pgaref]: Issue is with Hive 3.1.0. Attached masked query with the JIRA.
was (Author: kumar.ravi):
[~pgaref]: Issue is with Hive 3.1.0. We cant share the query, Sorry for that.
> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last
> reducer (map join vectorizer operator) and throwing exception as "Hive
> Runtime Error while processing vector batch" at the map join during the query
> execution.
> ---
>
> Key: HIVE-23412
> URL: https://issues.apache.org/jira/browse/HIVE-23412
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 3.1.0
>Reporter: kumar ravi
>Priority: Major
>
> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last
> reducer (map join vectorizer operator) and throwing exception as "Hive
> Runtime Error while processing vector batch" at the map join during the query
> execution.
> {{ERROR : FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map
> 2, vertexId=vertex_1588769098286_8402_1_00Vertex re-running, vertexName=Map
> 1, vertexId=vertex_1588769098286_8402_1_02Vertex re-running, vertexName=Map
> 4, vertexId=vertex_1588769098286_8402_1_04Vertex failed, vertexName=Reducer
> 6, vertexId=vertex_1588769098286_8402_1_05, diagnostics=[Task failed,
> taskId=task_1588769098286_8402_1_05_000604, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1588769098286_8402_1_05_000604_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)}}{{
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)}}{{
> at java.security.AccessController.doPrivileged(Native Method)}}{{ at
> javax.security.auth.Subject.doAs(Subject.java:422)}}{{ at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)}}{{
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)}}{{
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)}}{{
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)}}{{
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)}}{{
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}{{
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}{{
> at java.lang.Thread.run(Thread.java:748)}}{{Caused by:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:401)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:318)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)}}{{
> ... 16 more}}{{Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:503)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:392)}}{{
> ... 19 more}}{{Caused by:
[jira] [Commented] (HIVE-23376) Avoid repeated SHA computation in GenericUDTFGetSplits for hive-exec jar
[
https://issues.apache.org/jira/browse/HIVE-23376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102684#comment-17102684
]
Hive QA commented on HIVE-23376:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22206/dev-support/hive-personality.sh
|
| git revision | master / c419a7b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22206/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Avoid repeated SHA computation in GenericUDTFGetSplits for hive-exec jar
>
>
> Key: HIVE-23376
> URL: https://issues.apache.org/jira/browse/HIVE-23376
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23376.2.patch, image-2020-05-06-16-37-48-615.png
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java#L706]
>
>
> !image-2020-05-06-16-37-48-615.png|width=946,height=639!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23412) Filter applied in the "LEFT JOIN" condition is pulling a null batch on last reducer (map join vectorizer operator) and throwing exception as "Hive Runtime Error while pro
[
https://issues.apache.org/jira/browse/HIVE-23412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis updated HIVE-23412:
--
Component/s: Vectorization
> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last
> reducer (map join vectorizer operator) and throwing exception as "Hive
> Runtime Error while processing vector batch" at the map join during the query
> execution.
> ---
>
> Key: HIVE-23412
> URL: https://issues.apache.org/jira/browse/HIVE-23412
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 3.1.0
>Reporter: kumar ravi
>Priority: Major
>
> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last
> reducer (map join vectorizer operator) and throwing exception as "Hive
> Runtime Error while processing vector batch" at the map join during the query
> execution.
> {{ERROR : FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map
> 2, vertexId=vertex_1588769098286_8402_1_00Vertex re-running, vertexName=Map
> 1, vertexId=vertex_1588769098286_8402_1_02Vertex re-running, vertexName=Map
> 4, vertexId=vertex_1588769098286_8402_1_04Vertex failed, vertexName=Reducer
> 6, vertexId=vertex_1588769098286_8402_1_05, diagnostics=[Task failed,
> taskId=task_1588769098286_8402_1_05_000604, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1588769098286_8402_1_05_000604_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)}}{{
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)}}{{
> at java.security.AccessController.doPrivileged(Native Method)}}{{ at
> javax.security.auth.Subject.doAs(Subject.java:422)}}{{ at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)}}{{
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)}}{{
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)}}{{
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)}}{{
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)}}{{
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}{{
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}{{
> at java.lang.Thread.run(Thread.java:748)}}{{Caused by:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:401)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:318)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)}}{{
> ... 16 more}}{{Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:503)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:392)}}{{
> ... 19 more}}{{Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> Unexpected exception from VectorMapJoinOperator : null}}{{ at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:539)}}{{
> at
>
[jira] [Updated] (HIVE-23412) Filter applied in the "LEFT JOIN" condition is pulling a null batch on last reducer (map join vectorizer operator) and throwing exception as "Hive Runtime Error while pro
[
https://issues.apache.org/jira/browse/HIVE-23412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis updated HIVE-23412:
--
Affects Version/s: 3.1.0
> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last
> reducer (map join vectorizer operator) and throwing exception as "Hive
> Runtime Error while processing vector batch" at the map join during the query
> execution.
> ---
>
> Key: HIVE-23412
> URL: https://issues.apache.org/jira/browse/HIVE-23412
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: kumar ravi
>Priority: Major
>
> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last
> reducer (map join vectorizer operator) and throwing exception as "Hive
> Runtime Error while processing vector batch" at the map join during the query
> execution.
> {{ERROR : FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map
> 2, vertexId=vertex_1588769098286_8402_1_00Vertex re-running, vertexName=Map
> 1, vertexId=vertex_1588769098286_8402_1_02Vertex re-running, vertexName=Map
> 4, vertexId=vertex_1588769098286_8402_1_04Vertex failed, vertexName=Reducer
> 6, vertexId=vertex_1588769098286_8402_1_05, diagnostics=[Task failed,
> taskId=task_1588769098286_8402_1_05_000604, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1588769098286_8402_1_05_000604_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)}}{{
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)}}{{
> at java.security.AccessController.doPrivileged(Native Method)}}{{ at
> javax.security.auth.Subject.doAs(Subject.java:422)}}{{ at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)}}{{
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)}}{{
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)}}{{
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)}}{{
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)}}{{
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}{{
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}{{
> at java.lang.Thread.run(Thread.java:748)}}{{Caused by:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:401)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:318)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)}}{{
> ... 16 more}}{{Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:503)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:392)}}{{
> ... 19 more}}{{Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> Unexpected exception from VectorMapJoinOperator : null}}{{ at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:539)}}{{
> at
>
[jira] [Commented] (HIVE-23349) ACID: Concurrent MERGE INSERT operations produce duplicates
[
https://issues.apache.org/jira/browse/HIVE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102658#comment-17102658
]
Hive QA commented on HIVE-23349:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002360/HIVE-23349.6.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17246 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22205/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22205/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22205/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002360 - PreCommit-HIVE-Build
> ACID: Concurrent MERGE INSERT operations produce duplicates
> ---
>
> Key: HIVE-23349
> URL: https://issues.apache.org/jira/browse/HIVE-23349
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23349.1.patch, HIVE-23349.2.patch,
> HIVE-23349.3.patch, HIVE-23349.4.patch, HIVE-23349.5.patch, HIVE-23349.6.patch
>
>
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or
> EXCL_WRITE if hive.txn.write.xlock=false;
> {code}
> create table target (a int, b int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> {code}
> execute in parallel:
> {code}
> insert into source values (5,6), (7,8)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23351) Ranger Replication Scheduling
[
https://issues.apache.org/jira/browse/HIVE-23351?focusedWorklogId=432153=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-432153
]
ASF GitHub Bot logged work on HIVE-23351:
-
Author: ASF GitHub Bot
Created on: 08/May/20 15:09
Start Date: 08/May/20 15:09
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422175531
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##
@@ -103,13 +105,43 @@ public int execute() {
}
work.setRootTask(this);
this.parentTasks = null;
+if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data");
+ try {
+initiateAuthorizationLoadTask(work.dumpDirectory);
+ } catch (Exception e) {
+LOG.error("failed", e);
+setException(e);
+return ErrorMsg.getErrorMsg(e.getMessage()).getErrorCode();
+ }
+}
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
}
+ private boolean shouldLoadAuthorizationMetadata() {
+return
conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_AUTHORIZATION_METADATA);
+ }
+
+ private void initiateAuthorizationLoadTask(String hiveDumpDirectory) throws
SemanticException {
+if
(RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE)))
{
+ Path rangerLoadRoot = new Path(new Path(hiveDumpDirectory).getParent(),
ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Importing Authorization Metadata from {} ", rangerLoadRoot);
+ RangerLoadWork rangerLoadWork = new RangerLoadWork(rangerLoadRoot,
work.getSourceDbName(), work.dbNameToLoadIn);
+ Task rangerLoadTask = TaskFactory.get(rangerLoadWork,
conf);
+ if (childTasks == null) {
+childTasks = new ArrayList<>();
+ }
+ childTasks.add(rangerLoadTask);
Review comment:
Metadata will run as a part of current task.
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
in either bootstrap or in incremental, this will run it as part of current
ReplLoadTask
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 432153)
Time Spent: 2h 20m (was: 2h 10m)
> Ranger Replication Scheduling
> -
>
> Key: HIVE-23351
> URL: https://issues.apache.org/jira/browse/HIVE-23351
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch,
> HIVE-23351.03.patch, HIVE-23351.04.patch, HIVE-23351.05.patch,
> HIVE-23351.06.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23384) Investigate why filterExpression wasn't simplified at merge_join_1.q, subquery_multiinsert.q
[ https://issues.apache.org/jira/browse/HIVE-23384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23384: -- Summary: Investigate why filterExpression wasn't simplified at merge_join_1.q, subquery_multiinsert.q (was: Investigate why filterExpression wasn't simplified at merge_join_1.q) > Investigate why filterExpression wasn't simplified at merge_join_1.q, > subquery_multiinsert.q > > > Key: HIVE-23384 > URL: https://issues.apache.org/jira/browse/HIVE-23384 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Priority: Major > > Check [https://reviews.apache.org/r/72466/#comment309162] for details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23412) Filter applied in the "LEFT JOIN" condition is pulling a null batch on last reducer (map join vectorizer operator) and throwing exception as "Hive Runtime Error while p
[
https://issues.apache.org/jira/browse/HIVE-23412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102646#comment-17102646
]
kumar ravi commented on HIVE-23412:
---
[~pgaref]: Issue is with Hive 3.1.0. We cant share the query, Sorry for that.
> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last
> reducer (map join vectorizer operator) and throwing exception as "Hive
> Runtime Error while processing vector batch" at the map join during the query
> execution.
> ---
>
> Key: HIVE-23412
> URL: https://issues.apache.org/jira/browse/HIVE-23412
> Project: Hive
> Issue Type: Bug
>Reporter: kumar ravi
>Priority: Major
>
> Filter applied in the "LEFT JOIN" condition is pulling a null batch on last
> reducer (map join vectorizer operator) and throwing exception as "Hive
> Runtime Error while processing vector batch" at the map join during the query
> execution.
> {{ERROR : FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map
> 2, vertexId=vertex_1588769098286_8402_1_00Vertex re-running, vertexName=Map
> 1, vertexId=vertex_1588769098286_8402_1_02Vertex re-running, vertexName=Map
> 4, vertexId=vertex_1588769098286_8402_1_04Vertex failed, vertexName=Reducer
> 6, vertexId=vertex_1588769098286_8402_1_05, diagnostics=[Task failed,
> taskId=task_1588769098286_8402_1_05_000604, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1588769098286_8402_1_05_000604_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)}}{{
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)}}{{
> at java.security.AccessController.doPrivileged(Native Method)}}{{ at
> javax.security.auth.Subject.doAs(Subject.java:422)}}{{ at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)}}{{
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)}}{{
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)}}{{
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)}}{{
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)}}{{
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)}}{{
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}{{
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}{{
> at java.lang.Thread.run(Thread.java:748)}}{{Caused by:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:401)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:318)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)}}{{
> ... 16 more}}{{Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum
> 5)}}{{ at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:503)}}{{
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:392)}}{{
> ... 19 more}}{{Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> Unexpected exception from VectorMapJoinOperator : null}}{{ at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:539)}}{{
> at
>
[jira] [Updated] (HIVE-15269) Dynamic Min-Max/BloomFilter runtime-filtering for Tez
[
https://issues.apache.org/jira/browse/HIVE-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adrian Wang updated HIVE-15269:
---
Description:
If a dimension table and fact table are joined:
{noformat}
select *
from store join store_sales on (store.id = store_sales.store_id)
where store.s_store_name = 'My Store'
{noformat}
One optimization that can be done is to get the min/max store id values that
come out of the scan/filter of the store table, and send this min/max value
(via Tez edge) to the task which is scanning the store_sales table.
We can add a BETWEEN(min, max) predicate to the store_sales TableScan, where
this predicate can be pushed down to the storage handler (for example for ORC
formats). Pushing a min/max predicate to the ORC reader would allow us to avoid
having to entire whole row groups during the table scan.
was:
If a dimension table and facthttps://issues.apache.org/jira/browse/HIVE-15269#
table are joined:
{noformat}
select *
from store join store_sales on (store.id = store_sales.store_id)
where store.s_store_name = 'My Store'
{noformat}
One optimization that can be done is to get the min/max store id values that
come out of the scan/filter of the store table, and send this min/max value
(via Tez edge) to the task which is scanning the store_sales table.
We can add a BETWEEN(min, max) predicate to the store_sales TableScan, where
this predicate can be pushed down to the storage handler (for example for ORC
formats). Pushing a min/max predicate to the ORC reader would allow us to avoid
having to entire whole row groups during the table scan.
> Dynamic Min-Max/BloomFilter runtime-filtering for Tez
> -
>
> Key: HIVE-15269
> URL: https://issues.apache.org/jira/browse/HIVE-15269
> Project: Hive
> Issue Type: New Feature
> Components: Tez
>Reporter: Jason Dere
>Assignee: Deepak Jaiswal
>Priority: Major
> Labels: TODOC2.2.0
> Fix For: 2.2.0
>
> Attachments: HIVE-15269.1.patch, HIVE-15269.10.patch,
> HIVE-15269.11.patch, HIVE-15269.12.patch, HIVE-15269.13.patch,
> HIVE-15269.14.patch, HIVE-15269.15.patch, HIVE-15269.16.patch,
> HIVE-15269.17.patch, HIVE-15269.18.patch, HIVE-15269.19.patch,
> HIVE-15269.2.patch, HIVE-15269.3.patch, HIVE-15269.4.patch,
> HIVE-15269.5.patch, HIVE-15269.6.patch, HIVE-15269.7.patch,
> HIVE-15269.8.patch, HIVE-15269.9.patch
>
>
> If a dimension table and fact table are joined:
> {noformat}
> select *
> from store join store_sales on (store.id = store_sales.store_id)
> where store.s_store_name = 'My Store'
> {noformat}
> One optimization that can be done is to get the min/max store id values that
> come out of the scan/filter of the store table, and send this min/max value
> (via Tez edge) to the task which is scanning the store_sales table.
> We can add a BETWEEN(min, max) predicate to the store_sales TableScan, where
> this predicate can be pushed down to the storage handler (for example for ORC
> formats). Pushing a min/max predicate to the ORC reader would allow us to
> avoid having to entire whole row groups during the table scan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-15269) Dynamic Min-Max/BloomFilter runtime-filtering for Tez
[
https://issues.apache.org/jira/browse/HIVE-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adrian Wang updated HIVE-15269:
---
Description:
If a dimension table and facthttps://issues.apache.org/jira/browse/HIVE-15269#
table are joined:
{noformat}
select *
from store join store_sales on (store.id = store_sales.store_id)
where store.s_store_name = 'My Store'
{noformat}
One optimization that can be done is to get the min/max store id values that
come out of the scan/filter of the store table, and send this min/max value
(via Tez edge) to the task which is scanning the store_sales table.
We can add a BETWEEN(min, max) predicate to the store_sales TableScan, where
this predicate can be pushed down to the storage handler (for example for ORC
formats). Pushing a min/max predicate to the ORC reader would allow us to avoid
having to entire whole row groups during the table scan.
was:
If a dimension table and fact table are joined:
{noformat}
select *
from store join store_sales on (store.id = store_sales.store_id)
where store.s_store_name = 'My Store'
{noformat}
One optimization that can be done is to get the min/max store id values that
come out of the scan/filter of the store table, and send this min/max value
(via Tez edge) to the task which is scanning the store_sales table.
We can add a BETWEEN(min, max) predicate to the store_sales TableScan, where
this predicate can be pushed down to the storage handler (for example for ORC
formats). Pushing a min/max predicate to the ORC reader would allow us to avoid
having to entire whole row groups during the table scan.
> Dynamic Min-Max/BloomFilter runtime-filtering for Tez
> -
>
> Key: HIVE-15269
> URL: https://issues.apache.org/jira/browse/HIVE-15269
> Project: Hive
> Issue Type: New Feature
> Components: Tez
>Reporter: Jason Dere
>Assignee: Deepak Jaiswal
>Priority: Major
> Labels: TODOC2.2.0
> Fix For: 2.2.0
>
> Attachments: HIVE-15269.1.patch, HIVE-15269.10.patch,
> HIVE-15269.11.patch, HIVE-15269.12.patch, HIVE-15269.13.patch,
> HIVE-15269.14.patch, HIVE-15269.15.patch, HIVE-15269.16.patch,
> HIVE-15269.17.patch, HIVE-15269.18.patch, HIVE-15269.19.patch,
> HIVE-15269.2.patch, HIVE-15269.3.patch, HIVE-15269.4.patch,
> HIVE-15269.5.patch, HIVE-15269.6.patch, HIVE-15269.7.patch,
> HIVE-15269.8.patch, HIVE-15269.9.patch
>
>
> If a dimension table and
> facthttps://issues.apache.org/jira/browse/HIVE-15269# table are joined:
> {noformat}
> select *
> from store join store_sales on (store.id = store_sales.store_id)
> where store.s_store_name = 'My Store'
> {noformat}
> One optimization that can be done is to get the min/max store id values that
> come out of the scan/filter of the store table, and send this min/max value
> (via Tez edge) to the task which is scanning the store_sales table.
> We can add a BETWEEN(min, max) predicate to the store_sales TableScan, where
> this predicate can be pushed down to the storage handler (for example for ORC
> formats). Pushing a min/max predicate to the ORC reader would allow us to
> avoid having to entire whole row groups during the table scan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23418) Investigate why msck command found different partitions at repair.q, msck_repair*
[ https://issues.apache.org/jira/browse/HIVE-23418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23418: -- Summary: Investigate why msck command found different partitions at repair.q, msck_repair* (was: Investigate why msck command found different partitions at repair.q) > Investigate why msck command found different partitions at repair.q, > msck_repair* > - > > Key: HIVE-23418 > URL: https://issues.apache.org/jira/browse/HIVE-23418 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Attachment: (was: HIVE-23340.4.patch) > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch, HIVE-23340.4.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Status: Patch Available (was: In Progress) > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch, HIVE-23340.4.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Attachment: HIVE-23340.4.patch > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch, HIVE-23340.4.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Status: In Progress (was: Patch Available) > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch, HIVE-23340.4.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23349) ACID: Concurrent MERGE INSERT operations produce duplicates
[
https://issues.apache.org/jira/browse/HIVE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102633#comment-17102633
]
Hive QA commented on HIVE-23349:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22205/dev-support/hive-personality.sh
|
| git revision | master / c419a7b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22205/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: Concurrent MERGE INSERT operations produce duplicates
> ---
>
> Key: HIVE-23349
> URL: https://issues.apache.org/jira/browse/HIVE-23349
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23349.1.patch, HIVE-23349.2.patch,
> HIVE-23349.3.patch, HIVE-23349.4.patch, HIVE-23349.5.patch, HIVE-23349.6.patch
>
>
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or
> EXCL_WRITE if hive.txn.write.xlock=false;
> {code}
> create table target (a int, b int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> {code}
> execute in parallel:
> {code}
> insert into source values (5,6), (7,8)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23351) Ranger Replication Scheduling
[
https://issues.apache.org/jira/browse/HIVE-23351?focusedWorklogId=432140=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-432140
]
ASF GitHub Bot logged work on HIVE-23351:
-
Author: ASF GitHub Bot
Created on: 08/May/20 14:28
Start Date: 08/May/20 14:28
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422175531
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##
@@ -103,13 +105,43 @@ public int execute() {
}
work.setRootTask(this);
this.parentTasks = null;
+if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data");
+ try {
+initiateAuthorizationLoadTask(work.dumpDirectory);
+ } catch (Exception e) {
+LOG.error("failed", e);
+setException(e);
+return ErrorMsg.getErrorMsg(e.getMessage()).getErrorCode();
+ }
+}
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
}
+ private boolean shouldLoadAuthorizationMetadata() {
+return
conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_AUTHORIZATION_METADATA);
+ }
+
+ private void initiateAuthorizationLoadTask(String hiveDumpDirectory) throws
SemanticException {
+if
(RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE)))
{
+ Path rangerLoadRoot = new Path(new Path(hiveDumpDirectory).getParent(),
ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Importing Authorization Metadata from {} ", rangerLoadRoot);
+ RangerLoadWork rangerLoadWork = new RangerLoadWork(rangerLoadRoot,
work.getSourceDbName(), work.dbNameToLoadIn);
+ Task rangerLoadTask = TaskFactory.get(rangerLoadWork,
conf);
+ if (childTasks == null) {
+childTasks = new ArrayList<>();
+ }
+ childTasks.add(rangerLoadTask);
Review comment:
Metadata will run as a part of current task.
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
in either bootstrap or in incremental, this will run it as part of current
ReplLoadTask
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 432140)
Time Spent: 2h 10m (was: 2h)
> Ranger Replication Scheduling
> -
>
> Key: HIVE-23351
> URL: https://issues.apache.org/jira/browse/HIVE-23351
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch,
> HIVE-23351.03.patch, HIVE-23351.04.patch, HIVE-23351.05.patch,
> HIVE-23351.06.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23351) Ranger Replication Scheduling
[
https://issues.apache.org/jira/browse/HIVE-23351?focusedWorklogId=432138=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-432138
]
ASF GitHub Bot logged work on HIVE-23351:
-
Author: ASF GitHub Bot
Created on: 08/May/20 14:26
Start Date: 08/May/20 14:26
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422172542
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -198,7 +223,12 @@ private void initiateDataCopyTasks() throws
SemanticException {
finishRemainingTasks();
} else {
DAGTraversal.traverse(childTasks, new
AddDependencyToLeaves(TaskFactory.get(work, conf)));
- this.childTasks = childTasks;
Review comment:
Let's say, during an incremental run, you don't have any new table to
bootstrap and you don't have even any external table in DB.
List> childTasks = new ArrayList<>();
childTasks.addAll(work.externalTableCopyTasks(taskTracker, conf));
childTasks.addAll(work.managedTableCopyTasks(taskTracker, conf));
if (childTasks.isEmpty()) {
//All table data copy work finished.
finishRemainingTasks();
}
This will go to finishRemainingTasks() immediately even though you have
rangerDumpTask as childTask of ReplDumpTask
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -170,6 +175,26 @@ public int execute() {
return 0;
}
+ private void initiateAuthorizationDumpTask(Path currentDumpPath) throws
SemanticException {
+if
(RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE)))
{
+ Path rangerDumpRoot = new Path(currentDumpPath,
ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Exporting Authorization Metadata at {} ", rangerDumpRoot);
+ RangerDumpWork rangerDumpWork = new RangerDumpWork(rangerDumpRoot,
work.dbNameOrPattern);
+ Task rangerDumpTask = TaskFactory.get(rangerDumpWork,
conf);
+ if (childTasks == null) {
Review comment:
If it is considering Atlas task then it may be fine.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 432138)
Time Spent: 2h (was: 1h 50m)
> Ranger Replication Scheduling
> -
>
> Key: HIVE-23351
> URL: https://issues.apache.org/jira/browse/HIVE-23351
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch,
> HIVE-23351.03.patch, HIVE-23351.04.patch, HIVE-23351.05.patch,
> HIVE-23351.06.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23351) Ranger Replication Scheduling
[
https://issues.apache.org/jira/browse/HIVE-23351?focusedWorklogId=432134=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-432134
]
ASF GitHub Bot logged work on HIVE-23351:
-
Author: ASF GitHub Bot
Created on: 08/May/20 14:19
Start Date: 08/May/20 14:19
Worklog Time Spent: 10m
Work Description: aasha commented on a change in pull request #1004:
URL: https://github.com/apache/hive/pull/1004#discussion_r422170394
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##
@@ -103,13 +105,43 @@ public int execute() {
}
work.setRootTask(this);
this.parentTasks = null;
+if (shouldLoadAuthorizationMetadata()) {
+ LOG.info("Loading authorization data");
+ try {
+initiateAuthorizationLoadTask(work.dumpDirectory);
+ } catch (Exception e) {
+LOG.error("failed", e);
+setException(e);
+return ErrorMsg.getErrorMsg(e.getMessage()).getErrorCode();
+ }
+}
if (work.isIncrementalLoad()) {
return executeIncrementalLoad();
} else {
return executeBootStrapLoad();
}
}
+ private boolean shouldLoadAuthorizationMetadata() {
+return
conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_AUTHORIZATION_METADATA);
+ }
+
+ private void initiateAuthorizationLoadTask(String hiveDumpDirectory) throws
SemanticException {
+if
(RANGER_AUTHORIZER.equalsIgnoreCase(conf.getVar(HiveConf.ConfVars.REPL_AUTHORIZATION_PROVIDER_SERVICE)))
{
+ Path rangerLoadRoot = new Path(new Path(hiveDumpDirectory).getParent(),
ReplUtils.REPL_RANGER_BASE_DIR);
+ LOG.info("Importing Authorization Metadata from {} ", rangerLoadRoot);
+ RangerLoadWork rangerLoadWork = new RangerLoadWork(rangerLoadRoot,
work.getSourceDbName(), work.dbNameToLoadIn);
+ Task rangerLoadTask = TaskFactory.get(rangerLoadWork,
conf);
+ if (childTasks == null) {
+childTasks = new ArrayList<>();
+ }
+ childTasks.add(rangerLoadTask);
Review comment:
No, load tasks are added after this ranger task. Even metadata is loaded
through tasks in Repl Load. No?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 432134)
Time Spent: 1h 50m (was: 1h 40m)
> Ranger Replication Scheduling
> -
>
> Key: HIVE-23351
> URL: https://issues.apache.org/jira/browse/HIVE-23351
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23351.01.patch, HIVE-23351.02.patch,
> HIVE-23351.03.patch, HIVE-23351.04.patch, HIVE-23351.05.patch,
> HIVE-23351.06.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23414) Detail Hive Java Compatibility
[ https://issues.apache.org/jira/browse/HIVE-23414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102621#comment-17102621 ] Zoltan Chovan commented on HIVE-23414: -- Seems right to me, to be honest I was under the impression that the current upstream master already supported Java 11, but then I saw https://issues.apache.org/jira/browse/HIVE-22415 > Detail Hive Java Compatibility > -- > > Key: HIVE-23414 > URL: https://issues.apache.org/jira/browse/HIVE-23414 > Project: Hive > Issue Type: Improvement > Components: Documentation >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Trivial > Attachments: HIVE-23414.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23409) If TezSession application reopen fails for Timeline service down, default TezSession from SessionPool is closed after a retry
[
https://issues.apache.org/jira/browse/HIVE-23409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102614#comment-17102614
]
Hive QA commented on HIVE-23409:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002361/HIVE-23409.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17240 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22204/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22204/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22204/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002361 - PreCommit-HIVE-Build
> If TezSession application reopen fails for Timeline service down, default
> TezSession from SessionPool is closed after a retry
> -
>
> Key: HIVE-23409
> URL: https://issues.apache.org/jira/browse/HIVE-23409
> Project: Hive
> Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Naresh P R
>Priority: Major
> Attachments: HIVE-23409.patch
>
>
> we are closing a default session from TezSessionPool at here.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java#L589]
> If all the sessions in a pool are destroyed, queries wait indefinitely at
> TezSessionPool.getSession until HS2 restarts after other service recoveries.
> [HiveServer2-Background-Pool: Thread-12345]: tez.TezSessionPoolManager (:())
> - We are closing a default session because of retry failure.
> It's better if we allow retry & fail than hung.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
