[jira] [Commented] (HIVE-20692) Enable folding of NOT x IS (NOT) [TRUE|FALSE] expressions
[
https://issues.apache.org/jira/browse/HIVE-20692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640571#comment-16640571
]
Hive QA commented on HIVE-20692:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 4 new + 41 unchanged - 2 fixed
= 45 total (was 43) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14266/dev-support/hive-personality.sh
|
| git revision | master / a4b087b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14266/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14266/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14266/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enable folding of NOT x IS (NOT) [TRUE|FALSE] expressions
> -
>
> Key: HIVE-20692
> URL: https://issues.apache.org/jira/browse/HIVE-20692
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20692.01.patch
>
>
> Expressions like: {{not ((a>0) is not true)}} could be rewritten to {{(a>0)
> is true}}.
> However currently this doesn't happen because some of these functions are not
> translated for Calcite.
> {code}
> create table t (a integer);
> explain select not ((a>0) is not true) from t group by a;
> [...]
> expressions: (not (_col0 > 0) is not true) (type: boolean) |
> [...]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20705) Vectorization: Native Vector MapJoin doesn't support Complex Big Table values
[ https://issues.apache.org/jira/browse/HIVE-20705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-20705: --- > Vectorization: Native Vector MapJoin doesn't support Complex Big Table values > - > > Key: HIVE-20705 > URL: https://issues.apache.org/jira/browse/HIVE-20705 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640565#comment-16640565
]
Hive QA commented on HIVE-20679:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942534/HIVE-20679.2.patch
{color:green}SUCCESS:{color} +1 due to 10 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 15066 tests
executed
*Failed tests:*
{noformat}
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=243)
TestReplAcidTablesWithJsonMessage - did not produce a TEST-*.xml file (likely
timed out) (batchId=243)
TestReplIncrementalLoadAcidTablesWithJsonMessage - did not produce a TEST-*.xml
file (likely timed out) (batchId=243)
TestReplicationScenariosAcidTables - did not produce a TEST-*.xml file (likely
timed out) (batchId=243)
TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file
(likely timed out) (batchId=243)
TestReplicationScenariosIncrementalLoadAcidTables - did not produce a
TEST-*.xml file (likely timed out) (batchId=243)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_related_col]
(batchId=43)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[external_jdbc_table2]
(batchId=172)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testIncrementalDumpEmptyDumpDirectory
(batchId=243)
org.apache.hadoop.hive.ql.parse.repl.dump.events.TestEventHandlerFactory.shouldProvideDefaultHandlerWhenNothingRegisteredForThatEvent
(batchId=293)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14265/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14265/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14265/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942534 - PreCommit-HIVE-Build
> DDL operations on hive might create large messages for DBNotification
> -
>
> Key: HIVE-20679
> URL: https://issues.apache.org/jira/browse/HIVE-20679
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, a.sql, b.sql
>
>
> Certain type of ddl operations might create large messages as part of
> DBNoitification, this might lead to the rdbms throwing an error when storing
> the message since its size is to large. It will also increase the footprint
> of the rdbms space usage.
> We should try store compressed messages to allow handling these situations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640563#comment-16640563
]
Hive QA commented on HIVE-20679:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
26s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/server-extensions in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
183 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
37s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
30s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
39s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s{color}
| {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 39s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hcatalog/server-extensions: The patch generated 1 new
+ 23 unchanged - 2 fixed = 24 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} itests/hive-unit: The patch generated 39 new + 802
unchanged - 35 fixed = 841 total (was 837) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 5 new + 34 unchanged - 5 fixed
= 39 total (was 39) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
39s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red} standalone-metastore/metastore-server generated 5 new
+ 181 unchanged - 2 fixed = 186 total (was 183) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} |
[jira] [Commented] (HIVE-20635) VectorizedOrcAcidRowBatchReader doesn't filter delete events for original files
[
https://issues.apache.org/jira/browse/HIVE-20635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640545#comment-16640545
]
Hive QA commented on HIVE-20635:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942528/HIVE-20635.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15028 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14264/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14264/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14264/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942528 - PreCommit-HIVE-Build
> VectorizedOrcAcidRowBatchReader doesn't filter delete events for original
> files
> ---
>
> Key: HIVE-20635
> URL: https://issues.apache.org/jira/browse/HIVE-20635
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Major
> Attachments: HIVE-20635.2.patch, HIVE-20635.3.patch, HIVE-20635.patch
>
>
> this is a followup to HIVE-16812 which adds support for delete event
> filtering for splits from native acid files
> need to add the same for {{OrcSplit.isOriginal()}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640498#comment-16640498 ] Alice Fan edited comment on HIVE-20678 at 10/6/18 3:57 AM: --- Submit patch 3 to remove getRecordWriter() implementation for OutputFormat interface and it will throw NotImplementedException when the method is used. As the HiveHBaseTableOutputFormat should be an object of HiveOutputFormat instead of OutputFormat's. Tested it on a cluster with spark shell didn't notice any exception or error after the change. was (Author: afan): Submit patch 3 to remove getRecordWriter() implementation for OutputFormat interface and it will throw NotImplementedException when the method is used. As the HiveHBaseTableOutputFormat should be an object of HiveOutputFormat instead of OutputFormat. > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch, > HIVE-20678.3.patch > > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20635) VectorizedOrcAcidRowBatchReader doesn't filter delete events for original files
[
https://issues.apache.org/jira/browse/HIVE-20635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640536#comment-16640536
]
Hive QA commented on HIVE-20635:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14264/dev-support/hive-personality.sh
|
| git revision | master / a4b087b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14264/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14264/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> VectorizedOrcAcidRowBatchReader doesn't filter delete events for original
> files
> ---
>
> Key: HIVE-20635
> URL: https://issues.apache.org/jira/browse/HIVE-20635
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Major
> Attachments: HIVE-20635.2.patch, HIVE-20635.3.patch, HIVE-20635.patch
>
>
> this is a followup to HIVE-16812 which adds support for delete event
> filtering for splits from native acid files
> need to add the same for {{OrcSplit.isOriginal()}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14431) Recognize COALESCE as CASE
[
https://issues.apache.org/jira/browse/HIVE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640529#comment-16640529
]
Hive QA commented on HIVE-14431:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942570/HIVE-14431.06.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15026 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_histogram_numeric]
(batchId=14)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14263/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14263/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14263/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942570 - PreCommit-HIVE-Build
> Recognize COALESCE as CASE
> --
>
> Key: HIVE-14431
> URL: https://issues.apache.org/jira/browse/HIVE-14431
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-14431.01.patch, HIVE-14431.03.patch,
> HIVE-14431.04.patch, HIVE-14431.05.patch, HIVE-14431.05.patch,
> HIVE-14431.06.patch, HIVE-14431.06.patch, HIVE-14431.2.patch, HIVE-14431.patch
>
>
> Transform:
> {code}
> (COALESCE(a, '') = '') OR
>(a = 'A' AND b = c) OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> into:
> {code}
> (a='') OR
>(a is null) OR
>(a = 'A' AND b = c) OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> With complex queries, this will lead us to factor more predicates that could
> be pushed to the TS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14431) Recognize COALESCE as CASE
[
https://issues.apache.org/jira/browse/HIVE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640526#comment-16640526
]
Hive QA commented on HIVE-14431:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 3 new + 80 unchanged - 0 fixed
= 83 total (was 80) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14263/dev-support/hive-personality.sh
|
| git revision | master / a4b087b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14263/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14263/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14263/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Recognize COALESCE as CASE
> --
>
> Key: HIVE-14431
> URL: https://issues.apache.org/jira/browse/HIVE-14431
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-14431.01.patch, HIVE-14431.03.patch,
> HIVE-14431.04.patch, HIVE-14431.05.patch, HIVE-14431.05.patch,
> HIVE-14431.06.patch, HIVE-14431.06.patch, HIVE-14431.2.patch, HIVE-14431.patch
>
>
> Transform:
> {code}
> (COALESCE(a, '') = '') OR
>(a = 'A' AND b = c) OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> into:
> {code}
> (a='') OR
>(a is null) OR
>(a = 'A' AND b = c) OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> With complex queries, this will lead us to factor more predicates that could
> be pushed to the TS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[
https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640519#comment-16640519
]
Hive QA commented on HIVE-20531:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942524/HIVE-20531.09.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15022 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=194)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSparkSessionTimeout
(batchId=246)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14262/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14262/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14262/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942524 - PreCommit-HIVE-Build
> Repl load on cloud storage file system can skip redundant move or add
> partition tasks.
> --
>
> Key: HIVE-20531
> URL: https://issues.apache.org/jira/browse/HIVE-20531
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20531.01.patch, HIVE-20531.02.patch,
> HIVE-20531.03.patch, HIVE-20531.04.patch, HIVE-20531.05.patch,
> HIVE-20531.06.patch, HIVE-20531.07.patch, HIVE-20531.08.patch,
> HIVE-20531.09.patch
>
>
> In replication load, both add partition and insert operations are handled
> through import. Import creates 3 major tasks. Copy, add partition and move.
> Copy does the copy of data from source location to staging directory. Then
> add partition (which runs in parallel to copy) creates the partition in meta
> store. Its a no op in case of insert and by the time this ddl task is
> executed for insert partition would be already present. The third operation
> is move. Which actually moves the file from staging directory to actual
> location. And then in case of insert it adds the insert event to notification
> table. It does this for add partition operation which is redundant as the
> event for add partition would have been written already by ddl task. With the
> optimization to copy directly to actual table location in S3, move task can
> be avoided for add partition operation replay and replay of insert need not
> create the add partition (ddl) task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20704) Extend HivePreFilteringRule to support other functions
[ https://issues.apache.org/jira/browse/HIVE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20704: --- Attachment: HIVE-20704.patch > Extend HivePreFilteringRule to support other functions > -- > > Key: HIVE-20704 > URL: https://issues.apache.org/jira/browse/HIVE-20704 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20704.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20704) Extend HivePreFilteringRule to support other functions
[ https://issues.apache.org/jira/browse/HIVE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20704: --- Status: Patch Available (was: In Progress) > Extend HivePreFilteringRule to support other functions > -- > > Key: HIVE-20704 > URL: https://issues.apache.org/jira/browse/HIVE-20704 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20704.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-20704) Extend HivePreFilteringRule to support other functions
[ https://issues.apache.org/jira/browse/HIVE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20704 started by Jesus Camacho Rodriguez. -- > Extend HivePreFilteringRule to support other functions > -- > > Key: HIVE-20704 > URL: https://issues.apache.org/jira/browse/HIVE-20704 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20704) Extend HivePreFilteringRule to support other functions
[ https://issues.apache.org/jira/browse/HIVE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20704: -- > Extend HivePreFilteringRule to support other functions > -- > > Key: HIVE-20704 > URL: https://issues.apache.org/jira/browse/HIVE-20704 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[
https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640504#comment-16640504
]
Hive QA commented on HIVE-20531:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} itests/hive-unit: The patch generated 3 new + 680
unchanged - 0 fixed = 683 total (was 680) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 1 new + 144 unchanged - 1
fixed = 145 total (was 145) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14262/dev-support/hive-personality.sh
|
| git revision | master / a4b087b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14262/yetus/diff-checkstyle-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14262/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14262/yetus/patch-asflicense-problems.txt
|
| modules | C: common itests/hive-unit ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14262/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Repl load on cloud storage file system can skip redundant move or add
> partition tasks.
> --
>
> Key: HIVE-20531
> URL: https://issues.apache.org/jira/browse/HIVE-20531
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>
[jira] [Updated] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan updated HIVE-20678: - Attachment: HIVE-20678.3.patch Status: Patch Available (was: Open) > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch, > HIVE-20678.3.patch > > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20639) Add ability to Write Data from Hive Table/Query to Kafka Topic
[ https://issues.apache.org/jira/browse/HIVE-20639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-20639: -- Attachment: HIVE-20639.4.patch > Add ability to Write Data from Hive Table/Query to Kafka Topic > -- > > Key: HIVE-20639 > URL: https://issues.apache.org/jira/browse/HIVE-20639 > Project: Hive > Issue Type: New Feature > Components: kafka integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20639.2.patch, HIVE-20639.3.patch, > HIVE-20639.4.patch, HIVE-20639.patch > > > This patch adds multiple record writers to allow Hive user writing data > directly to a Kafka Topic. > The end goal is have a writer that provides multiple write semantics modes. > * A BEST_EFFORT where all the records will be delivered with no guarantee or > reties eg best effort mode. (Done) > * B At_least_once, each record will be delivered with retries from the Kafka > Producer and Hive Write Task. (Done) > * C Exactly_once , Writer will be using Kafka Transaction API to ensure that > each record is delivered once. (not so, still need more discussions) > In addition to the new feature i have refactored the existing code to make it > more readable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan updated HIVE-20678: - Attachment: (was: HIVE-20678.1.patch) > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch > > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan updated HIVE-20678: - Status: Open (was: Patch Available) > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch > > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan updated HIVE-20678: - Attachment: HIVE-20678.1.patch Status: Patch Available (was: Open) Submit patch 3 to remove getRecordWriter() implementation for OutputFormat interface and it will throw NotImplementedException when the method is used. As the HiveHBaseTableOutputFormat should be an object of HiveOutputFormat instead of OutputFormat. > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Attachments: HIVE-20678.1.patch, HIVE-20678.1.patch, > HIVE-20678.2.patch > > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan updated HIVE-20678: - Status: Open (was: Patch Available) > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch > > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20639) Add ability to Write Data from Hive Table/Query to Kafka Topic
[ https://issues.apache.org/jira/browse/HIVE-20639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-20639: -- Attachment: HIVE-20639.3.patch > Add ability to Write Data from Hive Table/Query to Kafka Topic > -- > > Key: HIVE-20639 > URL: https://issues.apache.org/jira/browse/HIVE-20639 > Project: Hive > Issue Type: New Feature > Components: kafka integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20639.2.patch, HIVE-20639.3.patch, HIVE-20639.patch > > > This patch adds multiple record writers to allow Hive user writing data > directly to a Kafka Topic. > The end goal is have a writer that provides multiple write semantics modes. > * A At_most_once where all the records will be delivered with no guarantee or > reties eg best effort mode. (Done) > * B At_least_once, each record will be delivered with retries from the Kafka > Producer and Hive Write Task. (Done) > * C Exactly_once , Writer will be using Kafka Transaction API to ensure that > each record is delivered once. (not so, still need more discussions) > In addition to the new feature i have refactored the existing code to make it > more readable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20639) Add ability to Write Data from Hive Table/Query to Kafka Topic
[ https://issues.apache.org/jira/browse/HIVE-20639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-20639: -- Description: This patch adds multiple record writers to allow Hive user writing data directly to a Kafka Topic. The end goal is have a writer that provides multiple write semantics modes. * A BEST_EFFORT where all the records will be delivered with no guarantee or reties eg best effort mode. (Done) * B At_least_once, each record will be delivered with retries from the Kafka Producer and Hive Write Task. (Done) * C Exactly_once , Writer will be using Kafka Transaction API to ensure that each record is delivered once. (not so, still need more discussions) In addition to the new feature i have refactored the existing code to make it more readable. was: This patch adds multiple record writers to allow Hive user writing data directly to a Kafka Topic. The end goal is have a writer that provides multiple write semantics modes. * A At_most_once where all the records will be delivered with no guarantee or reties eg best effort mode. (Done) * B At_least_once, each record will be delivered with retries from the Kafka Producer and Hive Write Task. (Done) * C Exactly_once , Writer will be using Kafka Transaction API to ensure that each record is delivered once. (not so, still need more discussions) In addition to the new feature i have refactored the existing code to make it more readable. > Add ability to Write Data from Hive Table/Query to Kafka Topic > -- > > Key: HIVE-20639 > URL: https://issues.apache.org/jira/browse/HIVE-20639 > Project: Hive > Issue Type: New Feature > Components: kafka integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20639.2.patch, HIVE-20639.3.patch, HIVE-20639.patch > > > This patch adds multiple record writers to allow Hive user writing data > directly to a Kafka Topic. > The end goal is have a writer that provides multiple write semantics modes. > * A BEST_EFFORT where all the records will be delivered with no guarantee or > reties eg best effort mode. (Done) > * B At_least_once, each record will be delivered with retries from the Kafka > Producer and Hive Write Task. (Done) > * C Exactly_once , Writer will be using Kafka Transaction API to ensure that > each record is delivered once. (not so, still need more discussions) > In addition to the new feature i have refactored the existing code to make it > more readable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20703) Put dynamic sort partition optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20703: --- Status: Patch Available (was: Open) > Put dynamic sort partition optimization under cost based decision > - > > Key: HIVE-20703 > URL: https://issues.apache.org/jira/browse/HIVE-20703 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20703.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20703) Put dynamic sort partition optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20703: --- Attachment: HIVE-20703.1.patch > Put dynamic sort partition optimization under cost based decision > - > > Key: HIVE-20703 > URL: https://issues.apache.org/jira/browse/HIVE-20703 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20703.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20703) Put dynamic sort partition optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-20703: -- > Put dynamic sort partition optimization under cost based decision > - > > Key: HIVE-20703 > URL: https://issues.apache.org/jira/browse/HIVE-20703 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20703.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20648) LLAP: Vector group by operator should use memory per executor
[
https://issues.apache.org/jira/browse/HIVE-20648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640487#comment-16640487
]
Hive QA commented on HIVE-20648:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942521/HIVE-20648.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15025 tests
executed
*Failed tests:*
{noformat}
TestAutoPurgeTables - did not produce a TEST-*.xml file (likely timed out)
(batchId=244)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=244)
TestSemanticAnalyzerHookLoading - did not produce a TEST-*.xml file (likely
timed out) (batchId=244)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_related_col]
(batchId=43)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14261/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14261/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14261/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942521 - PreCommit-HIVE-Build
> LLAP: Vector group by operator should use memory per executor
> -
>
> Key: HIVE-20648
> URL: https://issues.apache.org/jira/browse/HIVE-20648
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20648.1.patch, HIVE-20648.2.patch,
> HIVE-20648.3.patch, HIVE-20648.4.patch
>
>
> HIVE-15503 treatment has to be applied for vector group by operator as well.
> Vector group by currently uses MemoryMX bean to get heap usage and heap max
> memory which will not work for LLAP. Instead it should use memory per
> executor as upper bound to make flush decision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20623) Shared work: Extend sharing of map-join cache entries in LLAP
[
https://issues.apache.org/jira/browse/HIVE-20623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20623:
---
Attachment: HIVE-20623.04.patch
> Shared work: Extend sharing of map-join cache entries in LLAP
> -
>
> Key: HIVE-20623
> URL: https://issues.apache.org/jira/browse/HIVE-20623
> Project: Hive
> Issue Type: Improvement
> Components: llap, Logical Optimizer
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20623.01.patch, HIVE-20623.02.patch,
> HIVE-20623.02.patch, HIVE-20623.02.patch, HIVE-20623.03.patch,
> HIVE-20623.03.patch, HIVE-20623.03.patch, HIVE-20623.04.patch,
> HIVE-20623.04.patch, HIVE-20623.04.patch, HIVE-20623.04.patch,
> HIVE-20623.patch, hash-shared-work.json.txt, hash-shared-work.svg
>
>
> For a query like this
> {code}
> with all_sales as (
> select ss_customer_sk as customer_sk, ss_ext_list_price-ss_ext_discount_amt
> as ext_price from store_sales
> UNION ALL
> select ws_bill_customer_sk as customer_sk,
> ws_ext_list_price-ws_ext_discount_amt as ext_price from web_sales
> UNION ALL
> select cs_bill_customer_sk as customer_sk, cs_ext_sales_price -
> cs_ext_discount_amt as ext_price from catalog_sales)
> select sum(ext_price) total_price, c_customer_id from all_sales, customer
> where customer_sk = c_customer_sk
> group by c_customer_id
> order by total_price desc
> limit 100;
> {code}
> The hashtable used for all 3 joins are identical, which is loaded 3x times in
> the same LLAP instance because they are named.
> {code}
> cacheKey = "HASH_MAP_" + this.getOperatorId() + "_container";
> {code}
> in the cache.
> If those are identical in nature (i.e vectorization, hashtable type etc),
> then the duplication is just wasted CPU, memory and network - using the cache
> name for hashtables which will be identical in layout would be extremely
> useful.
> In cases where the join is pushed through a UNION, those are identical.
> This optimization can only be done without concern for accidental delays when
> the same upstream task is generating all of these hashtables, which is what
> is achieved by the shared scan optimizer already.
> In case the shared work is not present, this has potential downsides - in
> case two customer broadcasts were sourced from "Map 1" and "Map 2", the Map 1
> builder will block the other task from reading from Map 2, even though Map 2
> might have started after, but finished ahead of Map 1.
> So this specific optimization can always be considered for cases where the
> shared work unifies the operator tree and the parents of all the RS entries
> involved are same (& the RS layout is the same).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19023) Druid storage Handler still using old select query when the CBO fails
[
https://issues.apache.org/jira/browse/HIVE-19023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan resolved HIVE-19023.
-
Resolution: Fixed
Fix Version/s: 3.1.0
4.0.0
> Druid storage Handler still using old select query when the CBO fails
> -
>
> Key: HIVE-19023
> URL: https://issues.apache.org/jira/browse/HIVE-19023
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 4.0.0, 3.1.0
>
>
> See usage of function {code}
> org.apache.hadoop.hive.druid.io.DruidQueryBasedInputFormat#createSelectStarQuery{code}
> this can be replaced by scan query that is more efficent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20698) Better error instead of NPE when timestamp is null for any row when ingesting to druid
[
https://issues.apache.org/jira/browse/HIVE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640473#comment-16640473
]
Ashutosh Chauhan commented on HIVE-20698:
-
+1 pending tests
> Better error instead of NPE when timestamp is null for any row when ingesting
> to druid
> --
>
> Key: HIVE-20698
> URL: https://issues.apache.org/jira/browse/HIVE-20698
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20698.patch
>
>
> Currently when ingesting data to druid we get a wierd NPE when timestamp is
> null for any row.
> We should provide an error with a better message which helps user to know
> what is actually wrong.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.druid.serde.DruidSerDe.serialize(DruidSerDe.java:364)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:957)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:111)
> at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:158)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:480)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20648) LLAP: Vector group by operator should use memory per executor
[
https://issues.apache.org/jira/browse/HIVE-20648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640471#comment-16640471
]
Hive QA commented on HIVE-20648:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 15 new + 565 unchanged - 0
fixed = 580 total (was 565) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14261/dev-support/hive-personality.sh
|
| git revision | master / a4b087b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14261/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14261/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14261/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Vector group by operator should use memory per executor
> -
>
> Key: HIVE-20648
> URL: https://issues.apache.org/jira/browse/HIVE-20648
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20648.1.patch, HIVE-20648.2.patch,
> HIVE-20648.3.patch, HIVE-20648.4.patch
>
>
> HIVE-15503 treatment has to be applied for vector group by operator as well.
> Vector group by currently uses MemoryMX bean to get heap usage and heap max
> memory which will not work for LLAP. Instead it should use memory per
> executor as upper bound to make flush decision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-13157) MetaStoreEventListener.onAlter triggered for INSERT and SELECT
[
https://issues.apache.org/jira/browse/HIVE-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640468#comment-16640468
]
Bharathkrishna Guruvayoor Murali edited comment on HIVE-13157 at 10/5/18 11:26
PM:
---
I have observed that 2 alter notifications are created for insert, while only
one is actually needed (still present in master 4.0).
Basically my understanding is that the loadTable in Hive.java :
[Hive.java#L2630|https://github.com/apache/hive/blob/a4b087b18bd5b0b4023bced68c85cf1e16301fed/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2630]
calls alterTable even when there is a stats change, and only change happening
is transient_lastDdl because actual alter of stats happens in the next
alter_table event that follows this.
I wanted to change the code in the following way :
{code:java}
if (hasFollowingStatsTask) {
environmentContext = new EnvironmentContext();
environmentContext.putToProperties(StatsSetupConst.DO_NOT_UPDATE_STATS,
StatsSetupConst.TRUE);
}
else {
alterTable(tbl, false, environmentContext, true);
}
{code}
Change above is adding alterTable in else part, ie. when there is no stat task
to follow. But I do not know if loadTable is used in a different code path
where this alterTable is useful irrespective of stat change.
+ [~sershe] [~vihangk1] [~akolb] [~pvary] What do you think about this, can you
think of any case where the alterTable should not be in the else part.
was (Author: bharos92):
This is still present in master. I was looking into the same issue, basically
my understanding is that the loadTable in Hive.java :
[Hive.java#L2630|https://github.com/apache/hive/blob/a4b087b18bd5b0b4023bced68c85cf1e16301fed/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2630]
calls alterTable even when there is a stats change, and only change happening
is transient_lastDdl because actual alter of stats happens in the next
alter_table event that follows this.
I wanted to change the code in the following way :
{code:java}
if (hasFollowingStatsTask) {
environmentContext = new EnvironmentContext();
environmentContext.putToProperties(StatsSetupConst.DO_NOT_UPDATE_STATS,
StatsSetupConst.TRUE);
}
else {
alterTable(tbl, false, environmentContext, true);
}
{code}
Change above is adding alterTable in else part, ie. when there is no stat task
to follow. But I do not know if loadTable is used in a different code path
where this alterTable is useful irrespective of stat change.
+ [~sershe] [~vihangk1] [~akolb] [~pvary] What do you think about this, can you
think of any case where the alterTable should not be in the else part.
> MetaStoreEventListener.onAlter triggered for INSERT and SELECT
> --
>
> Key: HIVE-13157
> URL: https://issues.apache.org/jira/browse/HIVE-13157
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1, 4.0.0
>Reporter: Eugen Stoianovici
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Critical
>
> The event onAlter from
> org.apache.hadoop.hive.metastore.MetaStoreEventListener is triggered when
> INSERT or SELECT statements are executed on the target table.
> Furthermore, the value of transient_lastDdl is updated in table properties
> for INSERT statements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-13157) MetaStoreEventListener.onAlter triggered for INSERT and SELECT
[ https://issues.apache.org/jira/browse/HIVE-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-13157: Affects Version/s: 4.0.0 > MetaStoreEventListener.onAlter triggered for INSERT and SELECT > -- > > Key: HIVE-13157 > URL: https://issues.apache.org/jira/browse/HIVE-13157 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 4.0.0 >Reporter: Eugen Stoianovici >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Critical > > The event onAlter from > org.apache.hadoop.hive.metastore.MetaStoreEventListener is triggered when > INSERT or SELECT statements are executed on the target table. > Furthermore, the value of transient_lastDdl is updated in table properties > for INSERT statements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20700) Add config to disable rollup for druid
[ https://issues.apache.org/jira/browse/HIVE-20700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640469#comment-16640469 ] Ashutosh Chauhan commented on HIVE-20700: - +1 pending tests > Add config to disable rollup for druid > -- > > Key: HIVE-20700 > URL: https://issues.apache.org/jira/browse/HIVE-20700 > Project: Hive > Issue Type: New Feature > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20700.patch > > > Add a table property - 'druid.rollup' to allow disabling rollup for druid > tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-13157) MetaStoreEventListener.onAlter triggered for INSERT and SELECT
[ https://issues.apache.org/jira/browse/HIVE-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali reassigned HIVE-13157: --- Assignee: Bharathkrishna Guruvayoor Murali > MetaStoreEventListener.onAlter triggered for INSERT and SELECT > -- > > Key: HIVE-13157 > URL: https://issues.apache.org/jira/browse/HIVE-13157 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 4.0.0 >Reporter: Eugen Stoianovici >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Critical > > The event onAlter from > org.apache.hadoop.hive.metastore.MetaStoreEventListener is triggered when > INSERT or SELECT statements are executed on the target table. > Furthermore, the value of transient_lastDdl is updated in table properties > for INSERT statements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-13157) MetaStoreEventListener.onAlter triggered for INSERT and SELECT
[
https://issues.apache.org/jira/browse/HIVE-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640468#comment-16640468
]
Bharathkrishna Guruvayoor Murali commented on HIVE-13157:
-
This is still present in master. I was looking into the same issue, basically
my understanding is that the loadTable in Hive.java :
[Hive.java#L2630|https://github.com/apache/hive/blob/a4b087b18bd5b0b4023bced68c85cf1e16301fed/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2630]
calls alterTable even when there is a stats change, and only change happening
is transient_lastDdl because actual alter of stats happens in the next
alter_table event that follows this.
I wanted to change the code in the following way :
{code:java}
if (hasFollowingStatsTask) {
environmentContext = new EnvironmentContext();
environmentContext.putToProperties(StatsSetupConst.DO_NOT_UPDATE_STATS,
StatsSetupConst.TRUE);
}
else {
alterTable(tbl, false, environmentContext, true);
}
{code}
Change above is adding alterTable in else part, ie. when there is no stat task
to follow. But I do not know if loadTable is used in a different code path
where this alterTable is useful irrespective of stat change.
+ [~sershe] [~vihangk1] [~akolb] [~pvary] What do you think about this, can you
think of any case where the alterTable should not be in the else part.
> MetaStoreEventListener.onAlter triggered for INSERT and SELECT
> --
>
> Key: HIVE-13157
> URL: https://issues.apache.org/jira/browse/HIVE-13157
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1
>Reporter: Eugen Stoianovici
>Priority: Critical
>
> The event onAlter from
> org.apache.hadoop.hive.metastore.MetaStoreEventListener is triggered when
> INSERT or SELECT statements are executed on the target table.
> Furthermore, the value of transient_lastDdl is updated in table properties
> for INSERT statements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20385) Date: date + int fails to add days
[
https://issues.apache.org/jira/browse/HIVE-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640435#comment-16640435
]
Hive QA commented on HIVE-20385:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942520/HIVE-20385.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15026 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_histogram_numeric]
(batchId=14)
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSparkSessionTimeout
(batchId=246)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=252)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14260/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14260/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14260/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942520 - PreCommit-HIVE-Build
> Date: date + int fails to add days
> --
>
> Key: HIVE-20385
> URL: https://issues.apache.org/jira/browse/HIVE-20385
> Project: Hive
> Issue Type: Bug
> Components: Types
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Bruno Pusztahazi
>Priority: Minor
> Attachments: HIVE-20385.1.patch, HIVE-20385.2.patch,
> HIVE-20385.3.patch, HIVE-20385.4.patch
>
>
> {code}
> select current_date + 5;
> FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments '5': No
> matching method for class
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPDTIPlus with (date, int)
> {code}
> This works in Postgres 9.6 - http://sqlfiddle.com/#!17/9eecb/19253/0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20651) JdbcStorageHandler password should be encrypted
[
https://issues.apache.org/jira/browse/HIVE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20651:
--
Attachment: HIVE-20651.6.patch
> JdbcStorageHandler password should be encrypted
> ---
>
> Key: HIVE-20651
> URL: https://issues.apache.org/jira/browse/HIVE-20651
> Project: Hive
> Issue Type: Improvement
> Components: StorageHandler
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20651.1.patch, HIVE-20651.2.patch,
> HIVE-20651.3.patch, HIVE-20651.4.patch, HIVE-20651.5.patch, HIVE-20651.6.patch
>
>
> Currently, external jdbc table with JdbcStorageHandler store password as
> "hive.sql.dbcp.password" table property in clear text. We should put it in a
> keystore file. Here is the proposed change:
> {code:java}
> ….
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.dbcp.password.keystore" =
> "hdfs:///user/hive/credential/postgres.jceks",
> "hive.sql.dbcp.password.key" = "mydb.password"
> );
> {code}
>
> The jceks file is created with:
> {code}
> hadoop credential create mydb.password -provider
> hdfs:///user/hive/credential/postgres.jceks -v secretpassword
> {code}
> User can choose to put all db password in one jceks, or a separate jceks for
> each db.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20651) JdbcStorageHandler password should be encrypted
[
https://issues.apache.org/jira/browse/HIVE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640426#comment-16640426
]
Daniel Dai commented on HIVE-20651:
---
Rebase after HIVE-20652.
> JdbcStorageHandler password should be encrypted
> ---
>
> Key: HIVE-20651
> URL: https://issues.apache.org/jira/browse/HIVE-20651
> Project: Hive
> Issue Type: Improvement
> Components: StorageHandler
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20651.1.patch, HIVE-20651.2.patch,
> HIVE-20651.3.patch, HIVE-20651.4.patch, HIVE-20651.5.patch, HIVE-20651.6.patch
>
>
> Currently, external jdbc table with JdbcStorageHandler store password as
> "hive.sql.dbcp.password" table property in clear text. We should put it in a
> keystore file. Here is the proposed change:
> {code:java}
> ….
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.dbcp.password.keystore" =
> "hdfs:///user/hive/credential/postgres.jceks",
> "hive.sql.dbcp.password.key" = "mydb.password"
> );
> {code}
>
> The jceks file is created with:
> {code}
> hadoop credential create mydb.password -provider
> hdfs:///user/hive/credential/postgres.jceks -v secretpassword
> {code}
> User can choose to put all db password in one jceks, or a separate jceks for
> each db.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20652:
---
Fix Version/s: 3.2.0
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20652.01.patch, HIVE-20652.02.patch,
> HIVE-20652.patch, external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-20677) JDBC storage handler ordering problem - single split flag
[ https://issues.apache.org/jira/browse/HIVE-20677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20677: --- Fix Version/s: 3.2.0 > JDBC storage handler ordering problem - single split flag > - > > Key: HIVE-20677 > URL: https://issues.apache.org/jira/browse/HIVE-20677 > Project: Hive > Issue Type: Bug > Components: StorageHandler >Reporter: Gunther Hagleitner >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 4.0.0, 3.2.0 > > > When Calcite is pushing queries into the JDBC handler splitting the query via > offset/limit can cause issues (RDBMs is not guaranteed to return the data in > the same order every time.) > For these cases we want to: > a) Add a "do not split" flag to the jdbc handler. In that mode jdbc handler > will skip the count and offset/limit processing but just run the query in a > single node. Flag default will be false. > b) Have calcite automatically set this flag. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20669) JdbcStorageHandler push union of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20669:
---
Fix Version/s: 3.2.0
> JdbcStorageHandler push union of two different datasource to jdbc driver
>
>
> Key: HIVE-20669
> URL: https://issues.apache.org/jira/browse/HIVE-20669
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 UNION ALL SELECT * FROM ext_auth2
> {code}
> Calcite complain about unknown operation:
> {code}
> java.lang.AssertionError: Relational expression HepRelVertex#387 has
> calling-convention JDBC.DERBY but does not implement the required interface
> 'interface org.apache.calcite.adapter.jdbc.JdbcRel' of that convention
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1475)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:859)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:879)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:544)
> at org.apache.calcite.plan.RelOptRule.convert(RelOptRule.java:572)
> at
> org.apache.calcite.plan.RelOptRule.lambda$convertList$2(RelOptRule.java:607)
> at
> com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:640)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
> at
> com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:239)
> at org.apache.calcite.rel.core.SetOp.(SetOp.java:61)
> at org.apache.calcite.rel.core.Union.(Union.java:43)
> at
> org.apache.calcite.adapter.jdbc.JdbcRules$JdbcUnion.(JdbcRules.java:708)
> at
> org.apache.calcite.adapter.jdbc.JdbcRules$JdbcUnionRule.convert(JdbcRules.java:697)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.jdbc.JDBCUnionPushDownRule.onMatch(JDBCUnionPushDownRule.java:80)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:315)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:415)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:280)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:211)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:198)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:2348)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1917)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1670)
> at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1043)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1429)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:476)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12319)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:356)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:669)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1872)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1819)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1814)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640406#comment-16640406
]
Alexander Kolbasov commented on HIVE-20671:
---
LockRequestBuilder should be moved as well.
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.03.patch,
> HIVE-20671.04.patch, HIVE-20671.05.patch, HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-20671:
--
Attachment: HIVE-20671.05.patch
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.03.patch,
> HIVE-20671.04.patch, HIVE-20671.05.patch, HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20385) Date: date + int fails to add days
[
https://issues.apache.org/jira/browse/HIVE-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640405#comment-16640405
]
Hive QA commented on HIVE-20385:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 13 new + 5 unchanged - 96
fixed = 18 total (was 101) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14260/dev-support/hive-personality.sh
|
| git revision | master / 159305a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14260/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14260/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Date: date + int fails to add days
> --
>
> Key: HIVE-20385
> URL: https://issues.apache.org/jira/browse/HIVE-20385
> Project: Hive
> Issue Type: Bug
> Components: Types
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Bruno Pusztahazi
>Priority: Minor
> Attachments: HIVE-20385.1.patch, HIVE-20385.2.patch,
> HIVE-20385.3.patch, HIVE-20385.4.patch
>
>
> {code}
> select current_date + 5;
> FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments '5': No
> matching method for class
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPDTIPlus with (date, int)
> {code}
> This works in Postgres 9.6 - http://sqlfiddle.com/#!17/9eecb/19253/0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19023) Druid storage Handler still using old select query when the CBO fails
[
https://issues.apache.org/jira/browse/HIVE-19023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640404#comment-16640404
]
Nishant Bangarwa commented on HIVE-19023:
-
this has been fixed as part of https://issues.apache.org/jira/browse/HIVE-19680
> Druid storage Handler still using old select query when the CBO fails
> -
>
> Key: HIVE-19023
> URL: https://issues.apache.org/jira/browse/HIVE-19023
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
>
> See usage of function {code}
> org.apache.hadoop.hive.druid.io.DruidQueryBasedInputFormat#createSelectStarQuery{code}
> this can be replaced by scan query that is more efficent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20677) JDBC storage handler ordering problem - single split flag
[ https://issues.apache.org/jira/browse/HIVE-20677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20677: --- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Fixed as part of HIVE-20652. > JDBC storage handler ordering problem - single split flag > - > > Key: HIVE-20677 > URL: https://issues.apache.org/jira/browse/HIVE-20677 > Project: Hive > Issue Type: Bug > Components: StorageHandler >Reporter: Gunther Hagleitner >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 4.0.0 > > > When Calcite is pushing queries into the JDBC handler splitting the query via > offset/limit can cause issues (RDBMs is not guaranteed to return the data in > the same order every time.) > For these cases we want to: > a) Add a "do not split" flag to the jdbc handler. In that mode jdbc handler > will skip the count and offset/limit processing but just run the query in a > single node. Flag default will be false. > b) Have calcite automatically set this flag. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20669) JdbcStorageHandler push union of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20669:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Fixed as part of HIVE-20652.
> JdbcStorageHandler push union of two different datasource to jdbc driver
>
>
> Key: HIVE-20669
> URL: https://issues.apache.org/jira/browse/HIVE-20669
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 UNION ALL SELECT * FROM ext_auth2
> {code}
> Calcite complain about unknown operation:
> {code}
> java.lang.AssertionError: Relational expression HepRelVertex#387 has
> calling-convention JDBC.DERBY but does not implement the required interface
> 'interface org.apache.calcite.adapter.jdbc.JdbcRel' of that convention
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1475)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:859)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:879)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:544)
> at org.apache.calcite.plan.RelOptRule.convert(RelOptRule.java:572)
> at
> org.apache.calcite.plan.RelOptRule.lambda$convertList$2(RelOptRule.java:607)
> at
> com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:640)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
> at
> com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:239)
> at org.apache.calcite.rel.core.SetOp.(SetOp.java:61)
> at org.apache.calcite.rel.core.Union.(Union.java:43)
> at
> org.apache.calcite.adapter.jdbc.JdbcRules$JdbcUnion.(JdbcRules.java:708)
> at
> org.apache.calcite.adapter.jdbc.JdbcRules$JdbcUnionRule.convert(JdbcRules.java:697)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.jdbc.JDBCUnionPushDownRule.onMatch(JDBCUnionPushDownRule.java:80)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:315)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:415)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:280)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:211)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:198)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:2348)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1917)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1670)
> at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1043)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1429)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:476)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12319)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:356)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:669)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1872)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1819)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1814)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
>
[jira] [Updated] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20652:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~ashutoshc]!
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20652.01.patch, HIVE-20652.02.patch,
> HIVE-20652.patch, external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
>
[jira] [Commented] (HIVE-20698) Better error instead of NPE when timestamp is null for any row when ingesting to druid
[
https://issues.apache.org/jira/browse/HIVE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640393#comment-16640393
]
Bharathkrishna Guruvayoor Murali commented on HIVE-20698:
-
LGTM.
Just nit:
{code:java}
Preconditions.checkNotNull(timestamp,"Timestamp column cannot have null
value");{code}
Needs space after ,
> Better error instead of NPE when timestamp is null for any row when ingesting
> to druid
> --
>
> Key: HIVE-20698
> URL: https://issues.apache.org/jira/browse/HIVE-20698
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20698.patch
>
>
> Currently when ingesting data to druid we get a wierd NPE when timestamp is
> null for any row.
> We should provide an error with a better message which helps user to know
> what is actually wrong.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.druid.serde.DruidSerDe.serialize(DruidSerDe.java:364)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:957)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:111)
> at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:158)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:480)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19976) Fix TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flakiness
[ https://issues.apache.org/jira/browse/HIVE-19976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa resolved HIVE-19976. - Resolution: Fixed > Fix TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flakiness > -- > > Key: HIVE-19976 > URL: https://issues.apache.org/jira/browse/HIVE-19976 > Project: Hive > Issue Type: Test > Components: Druid integration >Affects Versions: 4.0.0 >Reporter: Peter Vary >Assignee: Nishant Bangarwa >Priority: Major > > HIVE-19922, HIVE-19509 disabled > TestMiniDruidKafkaCliDriver[druidkafkamini_basic] because of flakiness. This > is a follow-up jira, to fix it and enable it back -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19201) Hive doesn't read Druid data correctly
[ https://issues.apache.org/jira/browse/HIVE-19201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640396#comment-16640396 ] Nishant Bangarwa commented on HIVE-19201: - Here is the relevant issue - https://issues.apache.org/jira/browse/HIVE-19672 > Hive doesn't read Druid data correctly > -- > > Key: HIVE-19201 > URL: https://issues.apache.org/jira/browse/HIVE-19201 > Project: Hive > Issue Type: Bug > Components: Druid integration, Hive >Affects Versions: 2.3.0 > Environment: Ubuntu 16.4 TLS Desktop > Druid 0.12.0 (StandAlone - Quickstart conf) > Hive 2.3.0 (StandAlone - Quickstart conf) > Also have Hadoop and Zookeeper running >Reporter: Tournadre >Priority: Blocker > Attachments: DruidIntegration.tar > > > I created an external table on hive pointing at my datasource : wikiticker on > druid (I already defined the broker adress). However few colums appear as > NULL without knowing why. > I have also many time *.lck file in the metadatadb on Hive and Druid to > delete (otherwise, queries don't work) > So the describe statement lists the column and their type correctly, however > I only get "NULL" when I query some column (string type). I checked in my > Druid, the data are still here. > > Help ! :( > PS : Sorry first issue and JIRA, didn't see my issue resolved yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19976) Fix TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flakiness
[ https://issues.apache.org/jira/browse/HIVE-19976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640394#comment-16640394 ] Nishant Bangarwa commented on HIVE-19976: - I re-enabled the test as part of https://issues.apache.org/jira/browse/HIVE-19170 > Fix TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flakiness > -- > > Key: HIVE-19976 > URL: https://issues.apache.org/jira/browse/HIVE-19976 > Project: Hive > Issue Type: Test > Components: Druid integration >Affects Versions: 4.0.0 >Reporter: Peter Vary >Assignee: Nishant Bangarwa >Priority: Major > > HIVE-19922, HIVE-19509 disabled > TestMiniDruidKafkaCliDriver[druidkafkamini_basic] because of flakiness. This > is a follow-up jira, to fix it and enable it back -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640390#comment-16640390
]
Ashutosh Chauhan commented on HIVE-20652:
-
+1
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.01.patch, HIVE-20652.02.patch,
> HIVE-20652.patch, external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:525)
>
[jira] [Assigned] (HIVE-19976) Fix TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flakiness
[ https://issues.apache.org/jira/browse/HIVE-19976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa reassigned HIVE-19976: --- Assignee: Nishant Bangarwa > Fix TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flakiness > -- > > Key: HIVE-19976 > URL: https://issues.apache.org/jira/browse/HIVE-19976 > Project: Hive > Issue Type: Test > Components: Druid integration >Affects Versions: 4.0.0 >Reporter: Peter Vary >Assignee: Nishant Bangarwa >Priority: Major > > HIVE-19922, HIVE-19509 disabled > TestMiniDruidKafkaCliDriver[druidkafkamini_basic] because of flakiness. This > is a follow-up jira, to fix it and enable it back -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20698) Better error instead of NPE when timestamp is null for any row when ingesting to druid
[
https://issues.apache.org/jira/browse/HIVE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640388#comment-16640388
]
Nishant Bangarwa commented on HIVE-20698:
-
+cc [~ashutoshc]
> Better error instead of NPE when timestamp is null for any row when ingesting
> to druid
> --
>
> Key: HIVE-20698
> URL: https://issues.apache.org/jira/browse/HIVE-20698
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20698.patch
>
>
> Currently when ingesting data to druid we get a wierd NPE when timestamp is
> null for any row.
> We should provide an error with a better message which helps user to know
> what is actually wrong.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.druid.serde.DruidSerDe.serialize(DruidSerDe.java:364)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:957)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:111)
> at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:158)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:480)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20700) Add config to disable rollup for druid
[ https://issues.apache.org/jira/browse/HIVE-20700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640387#comment-16640387 ] Nishant Bangarwa commented on HIVE-20700: - +cc [~ashutoshc] > Add config to disable rollup for druid > -- > > Key: HIVE-20700 > URL: https://issues.apache.org/jira/browse/HIVE-20700 > Project: Hive > Issue Type: New Feature > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20700.patch > > > Add a table property - 'druid.rollup' to allow disabling rollup for druid > tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20538) Allow to store a key value together with a transaction.
[ https://issues.apache.org/jira/browse/HIVE-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20538: --- Attachment: HIVE-20538.6.patch Status: Patch Available (was: Open) > Allow to store a key value together with a transaction. > --- > > Key: HIVE-20538 > URL: https://issues.apache.org/jira/browse/HIVE-20538 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore, Transactions >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20538.1.patch, HIVE-20538.1.patch, > HIVE-20538.2.patch, HIVE-20538.3.patch, HIVE-20538.4.patch, > HIVE-20538.5.patch, HIVE-20538.6.patch > > > This can be useful for example to know if a transaction has already happened. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20538) Allow to store a key value together with a transaction.
[ https://issues.apache.org/jira/browse/HIVE-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20538: --- Status: Open (was: Patch Available) > Allow to store a key value together with a transaction. > --- > > Key: HIVE-20538 > URL: https://issues.apache.org/jira/browse/HIVE-20538 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore, Transactions >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20538.1.patch, HIVE-20538.1.patch, > HIVE-20538.2.patch, HIVE-20538.3.patch, HIVE-20538.4.patch, > HIVE-20538.5.patch, HIVE-20538.6.patch > > > This can be useful for example to know if a transaction has already happened. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19201) Hive doesn't read Druid data correctly
[ https://issues.apache.org/jira/browse/HIVE-19201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640386#comment-16640386 ] Nishant Bangarwa commented on HIVE-19201: - [~Korge] Make sure the column names in druid are all in lowercase. If you have a column in druid which has upper case characters and hive sends the query with lowercase column names, it can lead to getting null in results. > Hive doesn't read Druid data correctly > -- > > Key: HIVE-19201 > URL: https://issues.apache.org/jira/browse/HIVE-19201 > Project: Hive > Issue Type: Bug > Components: Druid integration, Hive >Affects Versions: 2.3.0 > Environment: Ubuntu 16.4 TLS Desktop > Druid 0.12.0 (StandAlone - Quickstart conf) > Hive 2.3.0 (StandAlone - Quickstart conf) > Also have Hadoop and Zookeeper running >Reporter: Tournadre >Priority: Blocker > Attachments: DruidIntegration.tar > > > I created an external table on hive pointing at my datasource : wikiticker on > druid (I already defined the broker adress). However few colums appear as > NULL without knowing why. > I have also many time *.lck file in the metadatadb on Hive and Druid to > delete (otherwise, queries don't work) > So the describe statement lists the column and their type correctly, however > I only get "NULL" when I query some column (string type). I checked in my > Druid, the data are still here. > > Help ! :( > PS : Sorry first issue and JIRA, didn't see my issue resolved yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20702) Account for overhead from datastructure aware estimations during mapjoin selection
[ https://issues.apache.org/jira/browse/HIVE-20702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20702: --- Attachment: HIVE-20702.patch > Account for overhead from datastructure aware estimations during mapjoin > selection > -- > > Key: HIVE-20702 > URL: https://issues.apache.org/jira/browse/HIVE-20702 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20702.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20702) Account for overhead from datastructure aware estimations during mapjoin selection
[ https://issues.apache.org/jira/browse/HIVE-20702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20702: --- Status: Patch Available (was: In Progress) > Account for overhead from datastructure aware estimations during mapjoin > selection > -- > > Key: HIVE-20702 > URL: https://issues.apache.org/jira/browse/HIVE-20702 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20702) Account for overhead from datastructure aware estimations during mapjoin selection
[ https://issues.apache.org/jira/browse/HIVE-20702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20702: -- > Account for overhead from datastructure aware estimations during mapjoin > selection > -- > > Key: HIVE-20702 > URL: https://issues.apache.org/jira/browse/HIVE-20702 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-20702) Account for overhead from datastructure aware estimations during mapjoin selection
[ https://issues.apache.org/jira/browse/HIVE-20702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20702 started by Jesus Camacho Rodriguez. -- > Account for overhead from datastructure aware estimations during mapjoin > selection > -- > > Key: HIVE-20702 > URL: https://issues.apache.org/jira/browse/HIVE-20702 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20701) Allow HiveStreaming to receive a key value to commit atomically together with the transaction
[ https://issues.apache.org/jira/browse/HIVE-20701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20701: --- Description: Following up with HIVE-20538 it'd be nice to be able to use this feature with hive streaming (was: Following up with https://issues.apache.org/jira/browse/HIVE-20538 it'd be nice to be able to use this feature with hive streaming) > Allow HiveStreaming to receive a key value to commit atomically together with > the transaction > - > > Key: HIVE-20701 > URL: https://issues.apache.org/jira/browse/HIVE-20701 > Project: Hive > Issue Type: Improvement >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > > Following up with HIVE-20538 it'd be nice to be able to use this feature with > hive streaming -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20701) Allow HiveStreaming to receive a key value to commit atomically together with the transaction
[ https://issues.apache.org/jira/browse/HIVE-20701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M reassigned HIVE-20701: -- Assignee: Jaume M > Allow HiveStreaming to receive a key value to commit atomically together with > the transaction > - > > Key: HIVE-20701 > URL: https://issues.apache.org/jira/browse/HIVE-20701 > Project: Hive > Issue Type: Improvement >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > > Following up with https://issues.apache.org/jira/browse/HIVE-20538 it'd be > nice to be able to use this feature with hive streaming -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[
https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640374#comment-16640374
]
Hive QA commented on HIVE-20678:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942514/HIVE-20678.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15026 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14259/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14259/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14259/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942514 - PreCommit-HIVE-Build
> HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure
> compatibility
>
>
> Key: HIVE-20678
> URL: https://issues.apache.org/jira/browse/HIVE-20678
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler, Hive, Spark
>Affects Versions: 4.0.0
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Major
> Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch
>
>
> This issue was caused by same reason described in
> https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems
> to be abandoned, so create this one to resolve the issue.
> HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will
> break the compatibility in spark's API when it is using hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20700) Add config to disable rollup for druid
[ https://issues.apache.org/jira/browse/HIVE-20700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20700: Status: Patch Available (was: Open) > Add config to disable rollup for druid > -- > > Key: HIVE-20700 > URL: https://issues.apache.org/jira/browse/HIVE-20700 > Project: Hive > Issue Type: New Feature > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20700.patch > > > Add a table property - 'druid.rollup' to allow disabling rollup for druid > tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20700) Add config to disable rollup for druid
[ https://issues.apache.org/jira/browse/HIVE-20700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20700: Attachment: HIVE-20700.patch > Add config to disable rollup for druid > -- > > Key: HIVE-20700 > URL: https://issues.apache.org/jira/browse/HIVE-20700 > Project: Hive > Issue Type: New Feature > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20700.patch > > > Add a table property - 'druid.rollup' to allow disabling rollup for druid > tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20651) JdbcStorageHandler password should be encrypted
[
https://issues.apache.org/jira/browse/HIVE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640354#comment-16640354
]
Daniel Dai commented on HIVE-20651:
---
Retest.
> JdbcStorageHandler password should be encrypted
> ---
>
> Key: HIVE-20651
> URL: https://issues.apache.org/jira/browse/HIVE-20651
> Project: Hive
> Issue Type: Improvement
> Components: StorageHandler
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20651.1.patch, HIVE-20651.2.patch,
> HIVE-20651.3.patch, HIVE-20651.4.patch, HIVE-20651.5.patch
>
>
> Currently, external jdbc table with JdbcStorageHandler store password as
> "hive.sql.dbcp.password" table property in clear text. We should put it in a
> keystore file. Here is the proposed change:
> {code:java}
> ….
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.dbcp.password.keystore" =
> "hdfs:///user/hive/credential/postgres.jceks",
> "hive.sql.dbcp.password.key" = "mydb.password"
> );
> {code}
>
> The jceks file is created with:
> {code}
> hadoop credential create mydb.password -provider
> hdfs:///user/hive/credential/postgres.jceks -v secretpassword
> {code}
> User can choose to put all db password in one jceks, or a separate jceks for
> each db.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20651) JdbcStorageHandler password should be encrypted
[
https://issues.apache.org/jira/browse/HIVE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20651:
--
Attachment: HIVE-20651.5.patch
> JdbcStorageHandler password should be encrypted
> ---
>
> Key: HIVE-20651
> URL: https://issues.apache.org/jira/browse/HIVE-20651
> Project: Hive
> Issue Type: Improvement
> Components: StorageHandler
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20651.1.patch, HIVE-20651.2.patch,
> HIVE-20651.3.patch, HIVE-20651.4.patch, HIVE-20651.5.patch
>
>
> Currently, external jdbc table with JdbcStorageHandler store password as
> "hive.sql.dbcp.password" table property in clear text. We should put it in a
> keystore file. Here is the proposed change:
> {code:java}
> ….
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.dbcp.password.keystore" =
> "hdfs:///user/hive/credential/postgres.jceks",
> "hive.sql.dbcp.password.key" = "mydb.password"
> );
> {code}
>
> The jceks file is created with:
> {code}
> hadoop credential create mydb.password -provider
> hdfs:///user/hive/credential/postgres.jceks -v secretpassword
> {code}
> User can choose to put all db password in one jceks, or a separate jceks for
> each db.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20700) Add config to disable rollup for druid
[ https://issues.apache.org/jira/browse/HIVE-20700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa reassigned HIVE-20700: --- > Add config to disable rollup for druid > -- > > Key: HIVE-20700 > URL: https://issues.apache.org/jira/browse/HIVE-20700 > Project: Hive > Issue Type: New Feature > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > > Add a table property - 'druid.rollup' to allow disabling rollup for druid > tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640333#comment-16640333 ] Alice Fan commented on HIVE-20678: -- Thanks [~ychena] very much for reviewing the patch. Because HiveOutputFormat is an interface extends interface OutputFormat. As HiveHBaseTableOutputFormat is going to implement HiveOutputFormat, so we will have to implements methods for both interfaces, which are getHiveRecordWriter() and getRecordWriter(). Let me know if you have any other feedback to help to improve the patch :) Thanks! > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch > > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20696) msck_*.q tests are broken
[ https://issues.apache.org/jira/browse/HIVE-20696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640329#comment-16640329 ] Jesus Camacho Rodriguez commented on HIVE-20696: [~prasanth_j], could you review it? Thanks > msck_*.q tests are broken > - > > Key: HIVE-20696 > URL: https://issues.apache.org/jira/browse/HIVE-20696 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20696.patch > > > Broke by HIVE-19617. Replaced table names but did not replaced folders paths > in q files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20696) msck_*.q tests are broken
[ https://issues.apache.org/jira/browse/HIVE-20696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20696: --- Status: Patch Available (was: In Progress) > msck_*.q tests are broken > - > > Key: HIVE-20696 > URL: https://issues.apache.org/jira/browse/HIVE-20696 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20696.patch > > > Broke by HIVE-19617. Replaced table names but did not replaced folders paths > in q files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20696) msck_*.q tests are broken
[ https://issues.apache.org/jira/browse/HIVE-20696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20696: --- Attachment: HIVE-20696.patch > msck_*.q tests are broken > - > > Key: HIVE-20696 > URL: https://issues.apache.org/jira/browse/HIVE-20696 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20696.patch > > > Broke by HIVE-19617. Replaced table names but did not replaced folders paths > in q files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-20696) msck_*.q tests are broken
[ https://issues.apache.org/jira/browse/HIVE-20696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20696 started by Jesus Camacho Rodriguez. -- > msck_*.q tests are broken > - > > Key: HIVE-20696 > URL: https://issues.apache.org/jira/browse/HIVE-20696 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20696.patch > > > Broke by HIVE-19617. Replaced table names but did not replaced folders paths > in q files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20538) Allow to store a key value together with a transaction.
[ https://issues.apache.org/jira/browse/HIVE-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640320#comment-16640320 ] Jaume M commented on HIVE-20538: @Can you review again [~ekoifman]? > Allow to store a key value together with a transaction. > --- > > Key: HIVE-20538 > URL: https://issues.apache.org/jira/browse/HIVE-20538 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore, Transactions >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20538.1.patch, HIVE-20538.1.patch, > HIVE-20538.2.patch, HIVE-20538.3.patch, HIVE-20538.4.patch, HIVE-20538.5.patch > > > This can be useful for example to know if a transaction has already happened. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20698) Better error instead of NPE when timestamp is null for any row when ingesting to druid
[
https://issues.apache.org/jira/browse/HIVE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-20698:
Status: Patch Available (was: Open)
> Better error instead of NPE when timestamp is null for any row when ingesting
> to druid
> --
>
> Key: HIVE-20698
> URL: https://issues.apache.org/jira/browse/HIVE-20698
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20698.patch
>
>
> Currently when ingesting data to druid we get a wierd NPE when timestamp is
> null for any row.
> We should provide an error with a better message which helps user to know
> what is actually wrong.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.druid.serde.DruidSerDe.serialize(DruidSerDe.java:364)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:957)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:111)
> at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:158)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:480)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20698) Better error instead of NPE when timestamp is null for any row when ingesting to druid
[
https://issues.apache.org/jira/browse/HIVE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-20698:
Attachment: HIVE-20698.patch
> Better error instead of NPE when timestamp is null for any row when ingesting
> to druid
> --
>
> Key: HIVE-20698
> URL: https://issues.apache.org/jira/browse/HIVE-20698
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20698.patch
>
>
> Currently when ingesting data to druid we get a wierd NPE when timestamp is
> null for any row.
> We should provide an error with a better message which helps user to know
> what is actually wrong.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.druid.serde.DruidSerDe.serialize(DruidSerDe.java:364)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:957)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:111)
> at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:158)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:480)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[
https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640319#comment-16640319
]
Hive QA commented on HIVE-20678:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hbase-handler: The patch generated 1 new + 6 unchanged
- 1 fixed = 7 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} hbase-handler generated 0 new + 7 unchanged - 6
fixed = 7 total (was 13) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14259/dev-support/hive-personality.sh
|
| git revision | master / 159305a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14259/yetus/diff-checkstyle-hbase-handler.txt
|
| modules | C: hbase-handler U: hbase-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14259/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure
> compatibility
>
>
> Key: HIVE-20678
> URL: https://issues.apache.org/jira/browse/HIVE-20678
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler, Hive, Spark
>Affects Versions: 4.0.0
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Major
> Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch
>
>
> This issue was caused by same reason described in
> https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems
> to be abandoned, so create this one to resolve the issue.
> HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will
> break the compatibility in spark's API when it is using hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables
[ https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640317#comment-16640317 ] Andrew Sherman commented on HIVE-20699: --- Hi [~ekoifman] can you add a description please? > Query based compactor for full CRUD Acid tables > --- > > Key: HIVE-20699 > URL: https://issues.apache.org/jira/browse/HIVE-20699 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640308#comment-16640308
]
Vihang Karajgaonkar commented on HIVE-20671:
I was testing this patch before merging it in and I saw that it fails with the
below error when you run {{mvn -DskipTests -Pitests clean test-compile
-DskipTests=true}} as described in the JIRA
{noformat}
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 03:16 min
[INFO] Finished at: 2018-10-05T13:15:54-07:00
[INFO] Final Memory: 380M/1676M
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile)
on project hive-streaming: Compilation failure
[ERROR]
hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
cannot find symbol
[ERROR] symbol: class LockRequestBuilder
[ERROR] location: package org.apache.hadoop.hive.metastore
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-streaming
{noformat}
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.03.patch,
> HIVE-20671.04.patch, HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20651) JdbcStorageHandler password should be encrypted
[
https://issues.apache.org/jira/browse/HIVE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640303#comment-16640303
]
Hive QA commented on HIVE-20651:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942513/HIVE-20651.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15020 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=194)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_histogram_numeric]
(batchId=14)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14258/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14258/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14258/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942513 - PreCommit-HIVE-Build
> JdbcStorageHandler password should be encrypted
> ---
>
> Key: HIVE-20651
> URL: https://issues.apache.org/jira/browse/HIVE-20651
> Project: Hive
> Issue Type: Improvement
> Components: StorageHandler
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20651.1.patch, HIVE-20651.2.patch,
> HIVE-20651.3.patch, HIVE-20651.4.patch
>
>
> Currently, external jdbc table with JdbcStorageHandler store password as
> "hive.sql.dbcp.password" table property in clear text. We should put it in a
> keystore file. Here is the proposed change:
> {code:java}
> ….
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.dbcp.password.keystore" =
> "hdfs:///user/hive/credential/postgres.jceks",
> "hive.sql.dbcp.password.key" = "mydb.password"
> );
> {code}
>
> The jceks file is created with:
> {code}
> hadoop credential create mydb.password -provider
> hdfs:///user/hive/credential/postgres.jceks -v secretpassword
> {code}
> User can choose to put all db password in one jceks, or a separate jceks for
> each db.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20651) JdbcStorageHandler password should be encrypted
[
https://issues.apache.org/jira/browse/HIVE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640291#comment-16640291
]
Hive QA commented on HIVE-20651:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
40s{color} | {color:red} root in master failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} jdbc-handler in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
40s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 40s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} jdbc-handler: The patch generated 1 new + 26 unchanged
- 3 fixed = 27 total (was 29) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 1 new + 374 unchanged - 0
fixed = 375 total (was 374) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} jdbc-handler generated 0 new + 10 unchanged - 1
fixed = 10 total (was 11) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
8s{color} | {color:red} ql generated 2 new + 2318 unchanged - 2 fixed = 2320
total (was 2320) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Found reliance on default encoding in
org.apache.hadoop.hive.ql.exec.Utilities.copyJobSecretToTableProperties(TableDesc):in
org.apache.hadoop.hive.ql.exec.Utilities.copyJobSecretToTableProperties(TableDesc):
new String(byte[]) At Utilities.java:[line 2314] |
| | Found reliance on default encoding in
org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobConf(TableDesc,
JobConf):in
org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobConf(TableDesc, JobConf):
String.getBytes() At PlanUtils.java:[line 989] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14258/dev-support/hive-personality.sh
|
| git revision | master / 9593e46 |
| Default Java | 1.8.0_111 |
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14258/yetus/branch-compile-root.txt
|
| findbugs | v3.0.0 |
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14258/yetus/patch-compile-root.txt
|
| javac |
[jira] [Updated] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20610:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch,
> HIVE-20610.3.patch, HIVE-20610.4.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
> 2018-09-19T06:42:05,072 ERROR [main] utils.MetaStoreUtils: Got exception:
> org.apache.hadoop.hive.metastore.api.MetaException Unable to delete
> directory: file:/tmp
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to delete
> directory: file:/tmp
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl.deleteDir(HiveMetaStoreFsImpl.java:45)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:365)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:353)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.deleteTableData(HiveMetaStore.java:2562)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2523)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:2685)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.drop_table_with_environment_context(Unknown
> Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.drop_table_with_environment_context(HiveMetaStoreClient.java:3204)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1492)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1432)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable(TestDbNotificationListener.java:522)
> [test-classes/:?]
> at
[jira] [Commented] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640243#comment-16640243
]
Andrew Sherman commented on HIVE-20610:
---
Pushed to master, thanks [~bharos92]
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch,
> HIVE-20610.3.patch, HIVE-20610.4.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
> 2018-09-19T06:42:05,072 ERROR [main] utils.MetaStoreUtils: Got exception:
> org.apache.hadoop.hive.metastore.api.MetaException Unable to delete
> directory: file:/tmp
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to delete
> directory: file:/tmp
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl.deleteDir(HiveMetaStoreFsImpl.java:45)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:365)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:353)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.deleteTableData(HiveMetaStore.java:2562)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2523)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:2685)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.drop_table_with_environment_context(Unknown
> Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.drop_table_with_environment_context(HiveMetaStoreClient.java:3204)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1492)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1432)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable(TestDbNotificationListener.java:522)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]{code}
>
>
--
[jira] [Commented] (HIVE-20684) Analyze table compute stats fails for tables containing timestamp with local time zone column
[
https://issues.apache.org/jira/browse/HIVE-20684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640230#comment-16640230
]
Nishant Bangarwa commented on HIVE-20684:
-
[~ashutoshc] there were plan changes in 2 druid tests, looks related to removal
of virtual columns from the table desc for non-native tables.
Updated patch.
> Analyze table compute stats fails for tables containing timestamp with local
> time zone column
> -
>
> Key: HIVE-20684
> URL: https://issues.apache.org/jira/browse/HIVE-20684
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20684.1.patch, HIVE-20684.patch
>
>
> Analyze table druid_table compute statistics for columns;
> Reference Exception -
> {code}
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: Only
> integer/long/timestamp/date/float/double/string/binary/boolean/decimal
> type argument is accepted but timestamp with local time zone is passed.
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats.getEvaluator(GenericUDAFComputeStats.java:105)
> at
> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:48)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1043)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:4817)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:5482)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:6496)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10617)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11557)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12229)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12319)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11802)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20684) Analyze table compute stats fails for tables containing timestamp with local time zone column
[
https://issues.apache.org/jira/browse/HIVE-20684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-20684:
Attachment: HIVE-20684.1.patch
> Analyze table compute stats fails for tables containing timestamp with local
> time zone column
> -
>
> Key: HIVE-20684
> URL: https://issues.apache.org/jira/browse/HIVE-20684
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20684.1.patch, HIVE-20684.patch
>
>
> Analyze table druid_table compute statistics for columns;
> Reference Exception -
> {code}
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: Only
> integer/long/timestamp/date/float/double/string/binary/boolean/decimal
> type argument is accepted but timestamp with local time zone is passed.
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats.getEvaluator(GenericUDAFComputeStats.java:105)
> at
> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:48)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1043)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:4817)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:5482)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:6496)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10617)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11557)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12229)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12319)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11802)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[
https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640222#comment-16640222
]
Hive QA commented on HIVE-20644:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942510/HIVE-20644.05
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15025 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_histogram_numeric]
(batchId=14)
org.apache.hive.jdbc.TestJdbcDriver2.testYarnATSGuid (batchId=253)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=252)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14257/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14257/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14257/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942510 - PreCommit-HIVE-Build
> Avoid exposing sensitive infomation through a Hive Runtime exception
>
>
> Key: HIVE-20644
> URL: https://issues.apache.org/jira/browse/HIVE-20644
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03,
> HIVE-20644.04, HIVE-20644.05
>
>
> The HiveException raised from the following methods is exposing the datarow
> the caused the run time exception.
> # ReduceRecordSource::GroupIterator::next() - around line 372
> # MapOperator::process() - around line 567
> # ExecReducer::reduce() - around line 243
> In all the cases, a string representation of the row is constructed on the
> fly and is included in
> the error message.
> VectorMapOperator::process() - around line 973 raises the same exception but
> it's not exposing the row since the row contents are not included in the
> error message.
> While trying to reproduce above error, I also found that the arguments to a
> UDF get exposed in log messages from FunctionRegistry::invoke() around line
> 1114. This too can cause sensitive information to be leaked through error
> message.
> This way some sensitive information is leaked to a user through exception
> message. That information may not be available to the user otherwise. Hence
> it's a kind of security breach or violation of access control.
> The contents of the row or the arguments to a function may be useful for
> debugging and hence it's worth to add those to logs. Hence proposal here to
> log a separate message with log level DEBUG or INFO containing the string
> representation of the row. Users can configure their logging so that
> DEBUG/INFO messages do not go to the client but at the same time are
> available in the hive server logs for debugging. The actual exception message
> will not contain any sensitive data like row data or argument data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20699) Query based compactor for full CRUD Acid tables
[ https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-20699: - > Query based compactor for full CRUD Acid tables > --- > > Key: HIVE-20699 > URL: https://issues.apache.org/jira/browse/HIVE-20699 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640213#comment-16640213
]
Bharathkrishna Guruvayoor Murali commented on HIVE-20610:
-
Test failure looks unrelated.
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch,
> HIVE-20610.3.patch, HIVE-20610.4.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
> 2018-09-19T06:42:05,072 ERROR [main] utils.MetaStoreUtils: Got exception:
> org.apache.hadoop.hive.metastore.api.MetaException Unable to delete
> directory: file:/tmp
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to delete
> directory: file:/tmp
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl.deleteDir(HiveMetaStoreFsImpl.java:45)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:365)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:353)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.deleteTableData(HiveMetaStore.java:2562)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2523)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:2685)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.drop_table_with_environment_context(Unknown
> Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.drop_table_with_environment_context(HiveMetaStoreClient.java:3204)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1492)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1432)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable(TestDbNotificationListener.java:522)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
[jira] [Assigned] (HIVE-19550) Enable TestAcidOnTez#testNonStandardConversion01
[ https://issues.apache.org/jira/browse/HIVE-19550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-19550: - Assignee: Eugene Koifman > Enable TestAcidOnTez#testNonStandardConversion01 > > > Key: HIVE-19550 > URL: https://issues.apache.org/jira/browse/HIVE-19550 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Eugene Koifman >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19549) Enable TestAcidOnTez#testCtasTezUnion
[ https://issues.apache.org/jira/browse/HIVE-19549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-19549: - Assignee: Eugene Koifman > Enable TestAcidOnTez#testCtasTezUnion > - > > Key: HIVE-19549 > URL: https://issues.apache.org/jira/browse/HIVE-19549 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Eugene Koifman >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
[ https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640211#comment-16640211 ] Bharathkrishna Guruvayoor Murali edited comment on HIVE-20545 at 10/5/18 6:51 PM: -- Hi [~anishek] , An example would be when Impala writes stats information to Partition objects and accesses it, as shown : [Impala stats|http://github.mtv.cloudera.com/CDH/Impala/blob/6f2d928734a33ace15ec6abd5659651173b9e69e/fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java#L45] But this information is not relevant as a Partition metadata for any other purpose. Also, these parameters have considerably large-size. On running some preliminary tests, we observe that there is a performance hit on increasing message size of HMS Notifications, so we can use this configuration to filter parameters like above mentioned. was (Author: bharos92): Hi [~anishek] , An example would be when Impala writes stats information to Partition objects and accesses it, as shown : [Impala reading stats |http://github.mtv.cloudera.com/CDH/Impala/blob/6f2d928734a33ace15ec6abd5659651173b9e69e/fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java#L45] But this information is not relevant as a Partition metadata for any other purpose. Also, these parameters have considerably large-size. On running some preliminary tests, we observe that there is a performance hit on increasing message size of HMS Notifications, so we can use this configuration to filter parameters like above mentioned. > Ability to exclude potentially large parameters in HMS Notifications > > > Key: HIVE-20545 > URL: https://issues.apache.org/jira/browse/HIVE-20545 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch, > HIVE-20545.3.branch-3.patch, HIVE-20545.3.patch, HIVE-20545.4.patch, > HIVE-20545.6.patch, HIVE-20545.7.patch > > > Clients can add large-sized parameters in Table/Partition objects. So we need > to enable adding regex patterns through HiveConf to match parameters to be > filtered from table and partition objects before serialization in HMS > notifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
[ https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640211#comment-16640211 ] Bharathkrishna Guruvayoor Murali commented on HIVE-20545: - Hi [~anishek] , An example would be when Impala writes stats information to Partition objects and accesses it, as shown : [Impala reading stats |http://github.mtv.cloudera.com/CDH/Impala/blob/6f2d928734a33ace15ec6abd5659651173b9e69e/fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java#L45] But this information is not relevant as a Partition metadata for any other purpose. Also, these parameters have considerably large-size. On running some preliminary tests, we observe that there is a performance hit on increasing message size of HMS Notifications, so we can use this configuration to filter parameters like above mentioned. > Ability to exclude potentially large parameters in HMS Notifications > > > Key: HIVE-20545 > URL: https://issues.apache.org/jira/browse/HIVE-20545 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch, > HIVE-20545.3.branch-3.patch, HIVE-20545.3.patch, HIVE-20545.4.patch, > HIVE-20545.6.patch, HIVE-20545.7.patch > > > Clients can add large-sized parameters in Table/Partition objects. So we need > to enable adding regex patterns through HiveConf to match parameters to be > filtered from table and partition objects before serialization in HMS > notifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[ https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640184#comment-16640184 ] Vihang Karajgaonkar commented on HIVE-20697: cc [~sankarh] [~anishek] [~thejas] Can you please take a look since you are more familiar with this code? Can we disable the tests until this is fixed? > Some replication tests are super slow and cause batch timeouts > -- > > Key: HIVE-20697 > URL: https://issues.apache.org/jira/browse/HIVE-20697 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Priority: Major > > Some of these tests are taking a long time and can cause test batch timeouts > given that we only give 40 min for a batch to complete. We should speed these > tests up. > TestReplicationScenarios 20 min > TestReplicationScenariosAcidTables11 min > TestReplicationScenariosAcrossInstances 5 min 14 sec > TestReplicationScenariosIncrementalLoadAcidTables 20 min -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20698) Better error instead of NPE when timestamp is null for any row when ingesting to druid
[
https://issues.apache.org/jira/browse/HIVE-20698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa reassigned HIVE-20698:
---
> Better error instead of NPE when timestamp is null for any row when ingesting
> to druid
> --
>
> Key: HIVE-20698
> URL: https://issues.apache.org/jira/browse/HIVE-20698
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
>
> Currently when ingesting data to druid we get a wierd NPE when timestamp is
> null for any row.
> We should provide an error with a better message which helps user to know
> what is actually wrong.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.druid.serde.DruidSerDe.serialize(DruidSerDe.java:364)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:957)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:111)
> at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:158)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:480)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20686) Sync QueryIDs across hive and druid
[ https://issues.apache.org/jira/browse/HIVE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20686: Attachment: HIVE-20686.2.patch > Sync QueryIDs across hive and druid > --- > > Key: HIVE-20686 > URL: https://issues.apache.org/jira/browse/HIVE-20686 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20686.1.patch, HIVE-20686.2.patch, HIVE-20686.patch > > > For the queries that hive passes to druid, pass on additional queryID as > query context. > It will be useful in tracing query level metrics across druid and hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[
https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640179#comment-16640179
]
Hive QA commented on HIVE-20644:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 1 new + 161 unchanged - 0
fixed = 162 total (was 161) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 3 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14257/dev-support/hive-personality.sh
|
| git revision | master / 9593e46 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14257/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14257/yetus/whitespace-tabs.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14257/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Avoid exposing sensitive infomation through a Hive Runtime exception
>
>
> Key: HIVE-20644
> URL: https://issues.apache.org/jira/browse/HIVE-20644
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03,
> HIVE-20644.04, HIVE-20644.05
>
>
> The HiveException raised from the following methods is exposing the datarow
> the caused the run time exception.
> # ReduceRecordSource::GroupIterator::next() - around line 372
> # MapOperator::process() - around line 567
> # ExecReducer::reduce() - around line 243
> In all the cases, a string representation of the row is constructed on the
> fly and is included in
> the error message.
> VectorMapOperator::process() - around line 973 raises the same exception but
> it's not exposing the row since the row contents are not included in the
> error message.
> While trying to reproduce above error, I also found that the arguments to a
> UDF get exposed in log messages from FunctionRegistry::invoke() around line
> 1114. This too can cause sensitive information to be leaked through error
> message.
>
