[jira] [Commented] (HIVE-23506) Move getAcidVersionFrom...File utility methods to TestTxnCommands
[ https://issues.apache.org/jira/browse/HIVE-23506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1754#comment-1754 ] Peter Vary commented on HIVE-23506: --- +1 pending test > Move getAcidVersionFrom...File utility methods to TestTxnCommands > - > > Key: HIVE-23506 > URL: https://issues.apache.org/jira/browse/HIVE-23506 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23506.01.patch > > > They're only used in test, and since they contain expensive file accesses, > it's best to remove the temptation to use them -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22971) Eliminate file rename in insert-only compactor
[
https://issues.apache.org/jira/browse/HIVE-22971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1748#comment-1748
]
Hive QA commented on HIVE-22971:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003345/HIVE-22971.06.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17274 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_materialized_view_rewrite_ssb]
(batchId=131)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22457/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22457/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22457/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003345 - PreCommit-HIVE-Build
> Eliminate file rename in insert-only compactor
> --
>
> Key: HIVE-22971
> URL: https://issues.apache.org/jira/browse/HIVE-22971
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: ACID, compaction
> Attachments: HIVE-22971.01.patch, HIVE-22971.02.patch,
> HIVE-22971.03.patch, HIVE-22971.04.patch, HIVE-22971.05.patch,
> HIVE-22971.06.patch, HIVE-22971.06.patch
>
>
> File rename is expensive for object stores, so MM (insert-only) compaction
> should skip that step when committing and write directly to base_x_cZ or
> delta_x_y_cZ.
> This also fixes the issue that for MM QB compaction the temp tables were
> stored under the table directory, and these temp dirs were never cleaned up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23504) Propagate query cancellations to druid when a hive query is cancelled
[
https://issues.apache.org/jira/browse/HIVE-23504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23504:
--
Attachment: HIVE-23504.01.patch
> Propagate query cancellations to druid when a hive query is cancelled
> -
>
> Key: HIVE-23504
> URL: https://issues.apache.org/jira/browse/HIVE-23504
> Project: Hive
> Issue Type: Bug
> Components: Druid integration, Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23504.01.patch
>
>
> See Query cancellation here -
> [https://druid.apache.org/docs/latest/querying/querying.html]
> Queries can be cancelled explicitly using their unique identifier. If the
> query identifier is set at the time of query, or is otherwise known, the
> following endpoint can be used on the Broker or Router to cancel the query.
>
> {{DELETE /druid/v2/{queryId}}}
>
> Copy
> For example, if the query ID is {{abc123}}, the query can be cancelled as
> follows:
>
> {{curl -X DELETE "http://host:port/druid/v2/abc123"}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23504) Propagate query cancellations to druid when a hive query is cancelled
[
https://issues.apache.org/jira/browse/HIVE-23504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23504:
--
Status: Patch Available (was: Open)
> Propagate query cancellations to druid when a hive query is cancelled
> -
>
> Key: HIVE-23504
> URL: https://issues.apache.org/jira/browse/HIVE-23504
> Project: Hive
> Issue Type: Bug
> Components: Druid integration, Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23504.01.patch
>
>
> See Query cancellation here -
> [https://druid.apache.org/docs/latest/querying/querying.html]
> Queries can be cancelled explicitly using their unique identifier. If the
> query identifier is set at the time of query, or is otherwise known, the
> following endpoint can be used on the Broker or Router to cancel the query.
>
> {{DELETE /druid/v2/{queryId}}}
>
> Copy
> For example, if the query ID is {{abc123}}, the query can be cancelled as
> follows:
>
> {{curl -X DELETE "http://host:port/druid/v2/abc123"}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23499) REPL: repl load should honor "hive.repl.dump.metadata.only=true"
[
https://issues.apache.org/jira/browse/HIVE-23499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-23499:
Attachment: HIVE-23499.1.patch
> REPL: repl load should honor "hive.repl.dump.metadata.only=true"
>
>
> Key: HIVE-23499
> URL: https://issues.apache.org/jira/browse/HIVE-23499
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23499.1.patch
>
>
> "{{hive.repl.dump.metadata.only=true"}} is not currently honored during
> "{{repl load"}}. Currently, it ends up copying all files even if this option
> is specified in "repl load". E.g
> {noformat}
> repl load airline_ontime_orc into another_airline_ontime_orc with
> ('hive.repl.rootdir'='s3a://blah/', 'hive.repl.dump.metadata.only'='true');
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23499) REPL: repl load should honor "hive.repl.dump.metadata.only=true"
[
https://issues.apache.org/jira/browse/HIVE-23499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-23499:
Assignee: Rajesh Balamohan
Status: Patch Available (was: Open)
> REPL: repl load should honor "hive.repl.dump.metadata.only=true"
>
>
> Key: HIVE-23499
> URL: https://issues.apache.org/jira/browse/HIVE-23499
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23499.1.patch
>
>
> "{{hive.repl.dump.metadata.only=true"}} is not currently honored during
> "{{repl load"}}. Currently, it ends up copying all files even if this option
> is specified in "repl load". E.g
> {noformat}
> repl load airline_ontime_orc into another_airline_ontime_orc with
> ('hive.repl.rootdir'='s3a://blah/', 'hive.repl.dump.metadata.only'='true');
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23506) Move getAcidVersionFrom...File utility methods to TestTxnCommands
[ https://issues.apache.org/jira/browse/HIVE-23506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23506: - Attachment: HIVE-23506.01.patch Status: Patch Available (was: Open) > Move getAcidVersionFrom...File utility methods to TestTxnCommands > - > > Key: HIVE-23506 > URL: https://issues.apache.org/jira/browse/HIVE-23506 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23506.01.patch > > > They're only used in test, and since they contain expensive file accesses, > it's best to remove the temptation to use them -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22971) Eliminate file rename in insert-only compactor
[
https://issues.apache.org/jira/browse/HIVE-22971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1737#comment-1737
]
Hive QA commented on HIVE-22971:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
25s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 134
unchanged - 5 fixed = 134 total (was 139) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22457/dev-support/hive-personality.sh
|
| git revision | master / 2f7a607 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: common ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22457/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Eliminate file rename in insert-only compactor
> --
>
> Key: HIVE-22971
> URL: https://issues.apache.org/jira/browse/HIVE-22971
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: ACID, compaction
> Attachments: HIVE-22971.01.patch, HIVE-22971.02.patch,
> HIVE-22971.03.patch, HIVE-22971.04.patch, HIVE-22971.05.patch,
> HIVE-22971.06.patch, HIVE-22971.06.patch
>
>
> File rename is expensive for object stores, so MM (insert-only) compaction
> should skip that step when
[jira] [Assigned] (HIVE-23506) Move getAcidVersionFrom...File utility methods to TestTxnCommands
[ https://issues.apache.org/jira/browse/HIVE-23506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-23506: > Move getAcidVersionFrom...File utility methods to TestTxnCommands > - > > Key: HIVE-23506 > URL: https://issues.apache.org/jira/browse/HIVE-23506 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > > They're only used in test, and since they contain expensive file accesses, > it's best to remove the temptation to use them -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23504) Propagate query cancellations to druid when a hive query is cancelled
[
https://issues.apache.org/jira/browse/HIVE-23504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely reassigned HIVE-23504:
-
> Propagate query cancellations to druid when a hive query is cancelled
> -
>
> Key: HIVE-23504
> URL: https://issues.apache.org/jira/browse/HIVE-23504
> Project: Hive
> Issue Type: Bug
> Components: Druid integration, Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
>
> See Query cancellation here -
> [https://druid.apache.org/docs/latest/querying/querying.html]
> Queries can be cancelled explicitly using their unique identifier. If the
> query identifier is set at the time of query, or is otherwise known, the
> following endpoint can be used on the Broker or Router to cancel the query.
>
> {{DELETE /druid/v2/{queryId}}}
>
> Copy
> For example, if the query ID is {{abc123}}, the query can be cancelled as
> follows:
>
> {{curl -X DELETE "http://host:port/druid/v2/abc123"}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23491) Move ParseDriver to parser module
[
https://issues.apache.org/jira/browse/HIVE-23491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1703#comment-1703
]
Hive QA commented on HIVE-23491:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003342/HIVE-23491.3.patch
{color:green}SUCCESS:{color} +1 due to 13 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17269 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22456/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22456/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22456/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003342 - PreCommit-HIVE-Build
> Move ParseDriver to parser module
> -
>
> Key: HIVE-23491
> URL: https://issues.apache.org/jira/browse/HIVE-23491
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23491.2.patch, HIVE-23491.3.patch
>
>
> Move *ParseDriver* class and syntax Parsing related unit tests to the parse
> module
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23491) Move ParseDriver to parser module
[
https://issues.apache.org/jira/browse/HIVE-23491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111095#comment-17111095
]
Hive QA commented on HIVE-23491:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
19s{color} | {color:blue} parser in master has 3 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
21s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
52s{color} | {color:blue} itests/util in master has 54 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} parser: The patch generated 194 new + 0 unchanged - 0
fixed = 194 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
2s{color} | {color:red} ql: The patch generated 1 new + 650 unchanged - 195
fixed = 651 total (was 845) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/util: The patch generated 1 new + 18 unchanged
- 0 fixed = 19 total (was 18) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22456/dev-support/hive-personality.sh
|
| git revision | master / 47d202b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22456/yetus/diff-checkstyle-parser.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22456/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22456/yetus/diff-checkstyle-itests_util.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22456/yetus/whitespace-eol.txt
|
| modules | C: parser ql itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22456/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically
[jira] [Work logged] (HIVE-23353) Atlas metadata replication scheduling
[
https://issues.apache.org/jira/browse/HIVE-23353?focusedWorklogId=434909=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434909
]
ASF GitHub Bot logged work on HIVE-23353:
-
Author: ASF GitHub Bot
Created on: 19/May/20 11:28
Start Date: 19/May/20 11:28
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #1021:
URL: https://github.com/apache/hive/pull/1021#discussion_r427228411
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/atlas/AtlasExportProcess.java
##
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.atlas;
+
+import org.apache.atlas.model.impexp.AtlasExportRequest;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.repl.util.ReplUtils;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+/**
+ * Performs Atlas metadata export.
+ */
+public class AtlasExportProcess extends AtlasProcess {
+ private FileSystem fileSystem = null;
+ protected static final Logger LOG =
LoggerFactory.getLogger(AtlasExportProcess.class);
+ private static final int DEF_BUF_SIZE = 8 * 1024;
+
+ public void run(AtlasReplInfo atlasReplInfo) throws SemanticException {
+LOG.info("HiveAtlasPlugin: Starting export from:{}",
atlasReplInfo.getStagingDir());
+try {
+ AtlasExportRequest exportRequest =
atlasRequestBuilder.createExportRequest(atlasReplInfo,
+ getAtlasClusterName(atlasReplInfo.getSrcCluster()));
+ InputStream inputStream = exportData(atlasReplInfo.getAtlasEndpoint(),
exportRequest, atlasReplInfo.getConf());
+ FileSystem fs = getFileSystem(atlasReplInfo);
+ Path exportFilePath = new Path(atlasReplInfo.getStagingDir(),
ReplUtils.REPL_ATLAS_EXPORT_FILE_NAME);
+ writeDataToFile(fs, exportFilePath, inputStream);
+} catch (SemanticException ex) {
+ throw ex;
+} catch (Exception ex) {
+ throw new SemanticException(ex);
+}
+ }
+
+ FileSystem getFileSystem(AtlasReplInfo atlasReplInfo) throws IOException {
+if (fileSystem != null) {
+ return fileSystem;
+}
+return FileSystem.get(atlasReplInfo.getStagingDir().toUri(),
atlasReplInfo.getConf());
+ }
+
+ protected InputStream exportData(String atlasEndpoint, AtlasExportRequest
request, HiveConf conf) throws Exception {
+return getClient(atlasEndpoint, conf).exportData(request);
+ }
+
+ private void writeDataToFile(FileSystem fs, Path exportFilePath, InputStream
is) throws IOException {
+long numBytesWritten = writeFile(fs, exportFilePath, is);
+LOG.info("HiveAtlasPlugin: writing to {} ({} bytes)", exportFilePath,
numBytesWritten);
+ }
+
+ private long writeFile(FileSystem fs, Path exportFilePath, InputStream is)
throws IOException {
Review comment:
done
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/atlas/DummyAtlasRESTClient.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.atlas;
+
+import
[jira] [Work logged] (HIVE-23353) Atlas metadata replication scheduling
[
https://issues.apache.org/jira/browse/HIVE-23353?focusedWorklogId=434908=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434908
]
ASF GitHub Bot logged work on HIVE-23353:
-
Author: ASF GitHub Bot
Created on: 19/May/20 11:27
Start Date: 19/May/20 11:27
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #1021:
URL: https://github.com/apache/hive/pull/1021#discussion_r427227921
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##
@@ -82,6 +82,16 @@
// Root base directory name for ranger.
public static final String REPL_RANGER_BASE_DIR = "ranger";
+ // Root base directory name for atlas.
+ public static final String REPL_ATLAS_BASE_DIR = "atlas";
+
+ // Atlas meta data export file.
+ public static final String REPL_ATLAS_EXPORT_FILE_NAME = "atlas_export.zip";
+
+ // Config for hadoop default file system.
+ public static final String DEFAULT_FS_CONFIG = "fs.defaultFS";
Review comment:
yes
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/atlas/RESTClientBuilder.java
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.atlas;
+
+import org.apache.atlas.ApplicationProperties;
+import org.apache.atlas.AtlasClientV2;
+import org.apache.atlas.AtlasException;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.configuration.ConfigurationConverter;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Properties;
+
+/**
+ * RestClientBuilder for AtlasRESTClient.
+ */
+public class RESTClientBuilder {
+ private static final Logger LOG =
LoggerFactory.getLogger(RESTClientBuilder.class);
+ private static final String ATLAS_PROPERTY_CLIENT_HA_RETRIES_KEY =
"atlas.client.ha.retries";
+ private static final String ATLAS_PROPERTY_CLIENT_HA_SLEEP_INTERVAL_MS_KEY =
"atlas.client.ha.sleep.interval.ms";
+ private static final String ATLAS_PROPERTY_REST_ADDRESS =
"atlas.rest.address";
+ private static final String ATLAS_PROPERTY_AUTH_KERBEROS =
"atlas.authentication.method.kerberos";
+ private static final String URL_SEPERATOR = ",";
+
+ private AuthStrategy authStrategy;
+ private UserGroupInformation userGroupInformation;
+ protected String incomingUrl;
+ protected String[] baseUrls;
+
+ enum AuthStrategy {
+KERBEROS
+ }
+
+ public RESTClientBuilder() {
+ }
+
+ public RESTClientBuilder baseUrl(String urls) {
+this.incomingUrl = urls;
+if (urls.contains(URL_SEPERATOR)) {
+ this.baseUrls = urls.split(URL_SEPERATOR);
+} else {
+ this.baseUrls = new String[]{urls};
+}
+
+return this;
+ }
+
+ public RESTClientBuilder setAuthStrategy() throws SemanticException {
+return inferKerberosAuthStrategy();
+ }
+
+ private RESTClientBuilder inferKerberosAuthStrategy() throws
SemanticException {
+try {
+ authStrategy = AuthStrategy.KERBEROS;
+ this.userGroupInformation = UserGroupInformation.getLoginUser();
+ LOG.info("HiveAtlasPlugin: authStrategy: {} : urls: {}:
userGroupInformation: {}",
+ authStrategy, baseUrls, userGroupInformation);
+} catch (Exception e) {
+ throw new SemanticException("Error: setAuthStrategy:
UserGroupInformation.getLoginUser: failed!", e);
+}
+return this;
+ }
+
+ public AtlasRESTClient create() throws SemanticException {
+if (baseUrls == null || baseUrls.length == 0) {
+ throw new SemanticException("baseUrls is not set.");
+}
+setAuthStrategy();
+initializeAtlasApplicationProperties();
+AtlasClientV2 clientV2;
+LOG.info("HiveAtlasPlugin: authStrategyUsed: {} : {}", authStrategy,
baseUrls);
+switch (authStrategy) {
+ case KERBEROS:
Review comment:
Done
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/atlas/RESTClientBuilder.java
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed
[jira] [Updated] (HIVE-23234) Optimize TxnHandler::allocateTableWriteIds
[ https://issues.apache.org/jira/browse/HIVE-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23234: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Optimize TxnHandler::allocateTableWriteIds > -- > > Key: HIVE-23234 > URL: https://issues.apache.org/jira/browse/HIVE-23234 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23234.1.patch, HIVE-23234.2.patch, > HIVE-23234.3.patch, HIVE-23234.4.patch, HIVE-23234.5.patch, > HIVE-23234.5.patch, HIVE-23234.5.patch, HIVE-23234.6.patch, > HIVE-23234.6.patch, HIVE-23234.6.patch, HIVE-23234.6.patch > > > Table write id allocation should be examined and optimized. One low hanging > fruit is batching all the PreparedStatement inserts, but there might be other > opportunities as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23234) Optimize TxnHandler::allocateTableWriteIds
[ https://issues.apache.org/jira/browse/HIVE-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111074#comment-17111074 ] Denys Kuzmenko commented on HIVE-23234: --- Pushed to master. [~Marton Bod], thank you for the patch and [~pvary] for review! > Optimize TxnHandler::allocateTableWriteIds > -- > > Key: HIVE-23234 > URL: https://issues.apache.org/jira/browse/HIVE-23234 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23234.1.patch, HIVE-23234.2.patch, > HIVE-23234.3.patch, HIVE-23234.4.patch, HIVE-23234.5.patch, > HIVE-23234.5.patch, HIVE-23234.5.patch, HIVE-23234.6.patch, > HIVE-23234.6.patch, HIVE-23234.6.patch, HIVE-23234.6.patch > > > Table write id allocation should be examined and optimized. One low hanging > fruit is batching all the PreparedStatement inserts, but there might be other > opportunities as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23485) Bound GroupByOperator stats using largest NDV among columns
[
https://issues.apache.org/jira/browse/HIVE-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-23485:
---
Status: Patch Available (was: Open)
Add a lower bound for the cardinality of the GroupByOperator by using the
largest known NDV among the columns present in the aggregation when there are
no grouping sets.
More precise stats could also be derived when grouping sets are present but we
shall do it as part of another jira.
Update LLAP plans with the estimation changes. More plans to be fixed in
follow-up commits.
> Bound GroupByOperator stats using largest NDV among columns
> ---
>
> Key: HIVE-23485
> URL: https://issues.apache.org/jira/browse/HIVE-23485
> Project: Hive
> Issue Type: Improvement
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: HIVE-23485.01.patch
>
>
> Consider the following SQL query:
> {code:sql}
> select id, name from person group by id, name;
> {code}
> and assume that the person table contains the following tuples:
> {code:sql}
> insert into person values (0, 'A') ;
> insert into person values (1, 'A') ;
> insert into person values (2, 'B') ;
> insert into person values (3, 'B') ;
> insert into person values (4, 'B') ;
> insert into person values (5, 'C') ;
> {code}
> If we know the number of distinct values (NDV) for all columns in the group
> by clause then we can infer a lower bound for the total number of rows by
> taking the maximun NDV of the involved columns.
> Currently the query in the scenario above has the following plan:
> {noformat}
> Vertex dependency in root stage
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 2 vectorized
> File Output Operator [FS_11]
> Group By Operator [GBY_10] (rows=3 width=92)
> Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
> <-Map 1 [SIMPLE_EDGE] vectorized
> SHUFFLE [RS_9]
> PartitionCols:_col0, _col1
> Group By Operator [GBY_8] (rows=3 width=92)
> Output:["_col0","_col1"],keys:id, name
> Select Operator [SEL_7] (rows=6 width=92)
> Output:["id","name"]
> TableScan [TS_0] (rows=6 width=92)
>
> default@person,person,Tbl:COMPLETE,Col:COMPLETE,Output:["id","name"]{noformat}
> Observe that the stats for group by report 3 rows but given that the ID
> attribute is part of the aggregation the rows cannot be less than 6.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23485) Bound GroupByOperator stats using largest NDV among columns
[
https://issues.apache.org/jira/browse/HIVE-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-23485:
---
Attachment: HIVE-23485.01.patch
> Bound GroupByOperator stats using largest NDV among columns
> ---
>
> Key: HIVE-23485
> URL: https://issues.apache.org/jira/browse/HIVE-23485
> Project: Hive
> Issue Type: Improvement
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: HIVE-23485.01.patch
>
>
> Consider the following SQL query:
> {code:sql}
> select id, name from person group by id, name;
> {code}
> and assume that the person table contains the following tuples:
> {code:sql}
> insert into person values (0, 'A') ;
> insert into person values (1, 'A') ;
> insert into person values (2, 'B') ;
> insert into person values (3, 'B') ;
> insert into person values (4, 'B') ;
> insert into person values (5, 'C') ;
> {code}
> If we know the number of distinct values (NDV) for all columns in the group
> by clause then we can infer a lower bound for the total number of rows by
> taking the maximun NDV of the involved columns.
> Currently the query in the scenario above has the following plan:
> {noformat}
> Vertex dependency in root stage
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 2 vectorized
> File Output Operator [FS_11]
> Group By Operator [GBY_10] (rows=3 width=92)
> Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
> <-Map 1 [SIMPLE_EDGE] vectorized
> SHUFFLE [RS_9]
> PartitionCols:_col0, _col1
> Group By Operator [GBY_8] (rows=3 width=92)
> Output:["_col0","_col1"],keys:id, name
> Select Operator [SEL_7] (rows=6 width=92)
> Output:["id","name"]
> TableScan [TS_0] (rows=6 width=92)
>
> default@person,person,Tbl:COMPLETE,Col:COMPLETE,Output:["id","name"]{noformat}
> Observe that the stats for group by report 3 rows but given that the ID
> attribute is part of the aggregation the rows cannot be less than 6.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23353) Atlas metadata replication scheduling
[
https://issues.apache.org/jira/browse/HIVE-23353?focusedWorklogId=434891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434891
]
ASF GitHub Bot logged work on HIVE-23353:
-
Author: ASF GitHub Bot
Created on: 19/May/20 11:00
Start Date: 19/May/20 11:00
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #1021:
URL: https://github.com/apache/hive/pull/1021#discussion_r427213821
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/atlas/RESTClientBuilder.java
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.atlas;
+
+import org.apache.atlas.ApplicationProperties;
+import org.apache.atlas.AtlasClientV2;
+import org.apache.atlas.AtlasException;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.configuration.ConfigurationConverter;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Properties;
+
+/**
+ * RestClientBuilder for AtlasRESTClient.
+ */
+public class RESTClientBuilder {
+ private static final Logger LOG =
LoggerFactory.getLogger(RESTClientBuilder.class);
+ private static final String ATLAS_PROPERTY_CLIENT_HA_RETRIES_KEY =
"atlas.client.ha.retries";
+ private static final String ATLAS_PROPERTY_CLIENT_HA_SLEEP_INTERVAL_MS_KEY =
"atlas.client.ha.sleep.interval.ms";
+ private static final String ATLAS_PROPERTY_REST_ADDRESS =
"atlas.rest.address";
+ private static final String ATLAS_PROPERTY_AUTH_KERBEROS =
"atlas.authentication.method.kerberos";
+ private static final String URL_SEPERATOR = ",";
+
+ private AuthStrategy authStrategy;
+ private UserGroupInformation userGroupInformation;
+ protected String incomingUrl;
+ protected String[] baseUrls;
+
+ enum AuthStrategy {
+KERBEROS
+ }
+
+ public RESTClientBuilder() {
+ }
+
+ public RESTClientBuilder baseUrl(String urls) {
+this.incomingUrl = urls;
+if (urls.contains(URL_SEPERATOR)) {
+ this.baseUrls = urls.split(URL_SEPERATOR);
+} else {
+ this.baseUrls = new String[]{urls};
+}
+
+return this;
+ }
+
+ public RESTClientBuilder setAuthStrategy() throws SemanticException {
+return inferKerberosAuthStrategy();
+ }
+
+ private RESTClientBuilder inferKerberosAuthStrategy() throws
SemanticException {
+try {
+ authStrategy = AuthStrategy.KERBEROS;
+ this.userGroupInformation = UserGroupInformation.getLoginUser();
+ LOG.info("HiveAtlasPlugin: authStrategy: {} : urls: {}:
userGroupInformation: {}",
+ authStrategy, baseUrls, userGroupInformation);
+} catch (Exception e) {
+ throw new SemanticException("Error: setAuthStrategy:
UserGroupInformation.getLoginUser: failed!", e);
+}
+return this;
+ }
+
+ public AtlasRESTClient create() throws SemanticException {
+if (baseUrls == null || baseUrls.length == 0) {
+ throw new SemanticException("baseUrls is not set.");
+}
+setAuthStrategy();
+initializeAtlasApplicationProperties();
+AtlasClientV2 clientV2;
+LOG.info("HiveAtlasPlugin: authStrategyUsed: {} : {}", authStrategy,
baseUrls);
+switch (authStrategy) {
+ case KERBEROS:
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434891)
Time Spent: 3h (was: 2h 50m)
> Atlas metadata replication scheduling
> -
>
> Key: HIVE-23353
> URL: https://issues.apache.org/jira/browse/HIVE-23353
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>
[jira] [Work logged] (HIVE-23353) Atlas metadata replication scheduling
[
https://issues.apache.org/jira/browse/HIVE-23353?focusedWorklogId=434890=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434890
]
ASF GitHub Bot logged work on HIVE-23353:
-
Author: ASF GitHub Bot
Created on: 19/May/20 11:00
Start Date: 19/May/20 11:00
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #1021:
URL: https://github.com/apache/hive/pull/1021#discussion_r427213470
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/atlas/AtlasImportProcess.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.atlas;
+
+import org.apache.atlas.model.impexp.AtlasImportRequest;
+import org.apache.atlas.model.impexp.AtlasImportResult;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Performs Atlas' Import.
+ */
+public class AtlasImportProcess extends AtlasProcess {
+ protected static final Logger LOG =
LoggerFactory.getLogger(AtlasImportProcess.class);
Review comment:
Done
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/atlas/AtlasRESTClientImpl.java
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl.atlas;
+
+import org.apache.atlas.AtlasClientV2;
+import org.apache.atlas.AtlasServiceException;
+import org.apache.atlas.model.impexp.AtlasExportRequest;
+import org.apache.atlas.model.impexp.AtlasImportRequest;
+import org.apache.atlas.model.impexp.AtlasImportResult;
+import org.apache.atlas.model.impexp.AtlasServer;
+import org.apache.atlas.model.instance.AtlasEntity;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.exec.repl.util.ReplUtils;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.InputStream;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import static com.sun.jersey.api.client.ClientResponse.Status.NOT_FOUND;
+
+/**
+ * Implementation of RESTClient, encapsulates Atlas' REST APIs.
+ */
+public class AtlasRESTClientImpl extends RetryingClient implements
AtlasRESTClient{
Review comment:
Fixed
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/atlas/DummyAtlasRESTClient.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed
[jira] [Work logged] (HIVE-23353) Atlas metadata replication scheduling
[
https://issues.apache.org/jira/browse/HIVE-23353?focusedWorklogId=434889=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434889
]
ASF GitHub Bot logged work on HIVE-23353:
-
Author: ASF GitHub Bot
Created on: 19/May/20 10:57
Start Date: 19/May/20 10:57
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #1021:
URL: https://github.com/apache/hive/pull/1021#discussion_r427212447
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcrossInstances.java
##
@@ -1561,4 +1562,94 @@ public void
testFailureUnsupportedAuthorizerReplication() throws Throwable {
assertEquals("Authorizer sentry not supported for replication ",
e.getMessage());
}
}
+
+ //Testing just the configs and no impact on existing replication
+ @Test
+ public void testAtlasReplication() throws Throwable {
+Map confMap = defaultAtlasConfMap();
+primary.run("use " + primaryDbName)
+.run("create table acid_table (key int, value int) partitioned by
(load_date date) " +
+"clustered by(key) into 2 buckets stored as orc
tblproperties ('transactional'='true')")
+.run("create table table1 (i String)")
+.run("insert into table1 values (1)")
+.run("insert into table1 values (2)")
+.dump(primaryDbName, getAtlasClause(defaultAtlasConfMap()));
+
+confMap.remove("hive.repl.atlas.replicatedto");
Review comment:
Nope, need not be, but I was testing that we are fine without this
config.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 434889)
Time Spent: 2h 40m (was: 2.5h)
> Atlas metadata replication scheduling
> -
>
> Key: HIVE-23353
> URL: https://issues.apache.org/jira/browse/HIVE-23353
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23353.01.patch, HIVE-23353.02.patch,
> HIVE-23353.03.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23433) Add Deny Policy on Target Database After Ranger Replication to avoid writes
[ https://issues.apache.org/jira/browse/HIVE-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23433: --- Status: In Progress (was: Patch Available) > Add Deny Policy on Target Database After Ranger Replication to avoid writes > --- > > Key: HIVE-23433 > URL: https://issues.apache.org/jira/browse/HIVE-23433 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23433.01.patch, HIVE-23433.02.patch, > HIVE-23433.03.patch, HIVE-23433.04.patch, HIVE-23433.05.patch, > HIVE-23433.06.patch, HIVE-23433.07.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23433) Add Deny Policy on Target Database After Ranger Replication to avoid writes
[ https://issues.apache.org/jira/browse/HIVE-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23433: --- Attachment: HIVE-23433.07.patch Status: Patch Available (was: In Progress) > Add Deny Policy on Target Database After Ranger Replication to avoid writes > --- > > Key: HIVE-23433 > URL: https://issues.apache.org/jira/browse/HIVE-23433 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23433.01.patch, HIVE-23433.02.patch, > HIVE-23433.03.patch, HIVE-23433.04.patch, HIVE-23433.05.patch, > HIVE-23433.06.patch, HIVE-23433.07.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23353) Atlas metadata replication scheduling
[
https://issues.apache.org/jira/browse/HIVE-23353?focusedWorklogId=434887=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-434887
]
ASF GitHub Bot logged work on HIVE-23353:
-
Author: ASF GitHub Bot
Created on: 19/May/20 10:55
Start Date: 19/May/20 10:55
Worklog Time Spent: 10m
Work Description: aasha commented on a change in pull request #1021:
URL: https://github.com/apache/hive/pull/1021#discussion_r427194647
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/AtlasDumpTask.java
##
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl;
+
+import org.apache.atlas.model.impexp.AtlasExportRequest;
+import org.apache.atlas.model.impexp.AtlasServer;
+import org.apache.atlas.model.instance.AtlasObjectId;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.exec.Task;
+import org.apache.hadoop.hive.ql.exec.repl.atlas.AtlasReplInfo;
+import org.apache.hadoop.hive.ql.exec.repl.atlas.AtlasRequestBuilder;
+import org.apache.hadoop.hive.ql.exec.repl.atlas.AtlasRestClient;
+import org.apache.hadoop.hive.ql.exec.repl.atlas.AtlasRestClientBuilder;
+import org.apache.hadoop.hive.ql.exec.repl.util.ReplUtils;
+import org.apache.hadoop.hive.ql.parse.EximUtil;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.apache.hadoop.hive.ql.parse.repl.dump.Utils;
+import org.apache.hadoop.hive.ql.plan.api.StageType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.InputStream;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Atlas Metadata Replication Dump Task.
+ **/
+public class AtlasDumpTask extends Task implements Serializable
{
+
+ protected static transient Logger LOG =
LoggerFactory.getLogger(AtlasDumpTask.class);
+ private static final long serialVersionUID = 1L;
+ private transient AtlasRestClientBuilder clientBuilder = new
AtlasRestClientBuilder();
+ protected transient AtlasRequestBuilder atlasRequestBuilder = new
AtlasRequestBuilder();
+ private transient AtlasRestClient atlasRestClient = null;
+
+ @Override
+ public int execute() {
+try {
+ AtlasReplInfo atlasReplInfo = work.getAtlasReplInfo();
+ LOG.info("Dumping Atlas metadata of srcDb: {}, for TgtDb: {} to staging
location:",
+ atlasReplInfo.getSrcDB(), atlasReplInfo.getTgtDB(),
atlasReplInfo.getStagingDir());
+ atlasRestClient =
clientBuilder.getClient(atlasReplInfo.getAtlasEndpoint(),
atlasReplInfo.getConf());
+ String entityGuid = checkHiveEntityGuid(atlasReplInfo.getSrcCluster(),
atlasReplInfo.getSrcDB());
+ long currentModifiedTime = getCurrentTimestamp(atlasReplInfo,
entityGuid);
+ dumpAtlasMetaData(atlasReplInfo);
+ createDumpMetadata(atlasReplInfo, currentModifiedTime);
+ return 0;
+} catch (Exception e) {
+ LOG.error("Exception while dumping atlas metadata", e);
+ setException(e);
+ return ErrorMsg.getErrorMsg(e.getMessage()).getErrorCode();
+}
+ }
+
+ public long getCurrentTimestamp(AtlasReplInfo atlasReplInfo, String
entityGuid) throws SemanticException {
Review comment:
Do you need this to be public?
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcrossInstances.java
##
@@ -1561,4 +1568,123 @@ public void
testFailureUnsupportedAuthorizerReplication() throws Throwable {
assertEquals("Authorizer sentry not supported for replication ",
e.getMessage());
}
}
+
+ //Testing just the configs and no impact on existing replication
+ @Test
+ public void testAtlasReplication() throws Throwable {
+Map confMap = defaultAtlasConfMap();
+primary.run("use " + primaryDbName)
+.run("create table acid_table (key int, value int) partitioned by
(load_date date) " +
+"clustered by(key) into 2 buckets
[jira] [Updated] (HIVE-23503) ValidTxnManager doesn't consider txns opened and committed between snapshot generation and locking when evaluating ValidTxnListState
[ https://issues.apache.org/jira/browse/HIVE-23503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23503: -- Description: ValidTxnManager doesn't consider txns opened and committed between snapshot generation and locking when evaluating ValidTxnListState. This cause issues like duplicate insert in case of concurrent merge insert & insert. > ValidTxnManager doesn't consider txns opened and committed between snapshot > generation and locking when evaluating ValidTxnListState > > > Key: HIVE-23503 > URL: https://issues.apache.org/jira/browse/HIVE-23503 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23503.1.patch > > > ValidTxnManager doesn't consider txns opened and committed between snapshot > generation and locking when evaluating ValidTxnListState. This cause issues > like duplicate insert in case of concurrent merge insert & insert. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23503) ValidTxnManager doesn't consider txns opened and committed between snapshot generation and locking when evaluating ValidTxnListState
[ https://issues.apache.org/jira/browse/HIVE-23503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23503: -- Status: Patch Available (was: Open) > ValidTxnManager doesn't consider txns opened and committed between snapshot > generation and locking when evaluating ValidTxnListState > > > Key: HIVE-23503 > URL: https://issues.apache.org/jira/browse/HIVE-23503 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23503.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23353) Atlas metadata replication scheduling
[ https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-23353: -- Attachment: HIVE-23353.03.patch > Atlas metadata replication scheduling > - > > Key: HIVE-23353 > URL: https://issues.apache.org/jira/browse/HIVE-23353 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23353.01.patch, HIVE-23353.02.patch, > HIVE-23353.03.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23503) ValidTxnManager doesn't consider txns opened and committed between snapshot generation and locking when evaluating ValidTxnListState
[ https://issues.apache.org/jira/browse/HIVE-23503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23503: -- Attachment: HIVE-23503.1.patch > ValidTxnManager doesn't consider txns opened and committed between snapshot > generation and locking when evaluating ValidTxnListState > > > Key: HIVE-23503 > URL: https://issues.apache.org/jira/browse/HIVE-23503 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23503.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23503) ValidTxnManager doesn't consider txns opened and committed between snapshot generation and locking when evaluating ValidTxnListState
[ https://issues.apache.org/jira/browse/HIVE-23503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko reassigned HIVE-23503: - Assignee: Denys Kuzmenko > ValidTxnManager doesn't consider txns opened and committed between snapshot > generation and locking when evaluating ValidTxnListState > > > Key: HIVE-23503 > URL: https://issues.apache.org/jira/browse/HIVE-23503 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23433) Add Deny Policy on Target Database After Ranger Replication to avoid writes
[
https://issues.apache.org/jira/browse/HIVE-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111064#comment-17111064
]
Hive QA commented on HIVE-23433:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003340/HIVE-23433.06.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17271 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_materialized_view_rewrite_ssb]
(batchId=131)
org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testAcidTablesReplLoadBootstrapIncr
(batchId=208)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22455/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22455/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22455/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003340 - PreCommit-HIVE-Build
> Add Deny Policy on Target Database After Ranger Replication to avoid writes
> ---
>
> Key: HIVE-23433
> URL: https://issues.apache.org/jira/browse/HIVE-23433
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23433.01.patch, HIVE-23433.02.patch,
> HIVE-23433.03.patch, HIVE-23433.04.patch, HIVE-23433.05.patch,
> HIVE-23433.06.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23492) Remove unnecessary FileSystem#exists calls from ql module
[ https://issues.apache.org/jira/browse/HIVE-23492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23492: - Attachment: HIVE-23492.04.patch > Remove unnecessary FileSystem#exists calls from ql module > - > > Key: HIVE-23492 > URL: https://issues.apache.org/jira/browse/HIVE-23492 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23492.01.patch, HIVE-23492.02.patch, > HIVE-23492.03.patch, HIVE-23492.04.patch > > > Wherever there is an exists() call before open() or delete(), remove it and > infer from the FileNotFoundException raised in open/delete that the file does > not exist. Exists() just checks for a FileNotFoundException so it's a waste > of time, especially on clunkier FSes -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23433) Add Deny Policy on Target Database After Ranger Replication to avoid writes
[
https://issues.apache.org/jira/browse/HIVE-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111048#comment-17111048
]
Hive QA commented on HIVE-23433:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
19s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22455/dev-support/hive-personality.sh
|
| git revision | master / 47d202b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22455/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add Deny Policy on Target Database After Ranger Replication to avoid writes
> ---
>
> Key: HIVE-23433
> URL: https://issues.apache.org/jira/browse/HIVE-23433
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23433.01.patch, HIVE-23433.02.patch,
> HIVE-23433.03.patch, HIVE-23433.04.patch, HIVE-23433.05.patch,
> HIVE-23433.06.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23353) Atlas metadata replication scheduling
[
https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111026#comment-17111026
]
Hive QA commented on HIVE-23353:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
4s{color} | {color:red} ql: The patch generated 6 new + 25 unchanged - 0 fixed
= 31 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m
12s{color} | {color:red} root: The patch generated 9 new + 516 unchanged - 0
fixed = 525 total (was 516) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} itests/hive-unit: The patch generated 3 new + 114
unchanged - 0 fixed = 117 total (was 114) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
44s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 7m
54s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red} patch/itests/hive-unit cannot run setBugDatabaseInfo
from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 15m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22453/dev-support/hive-personality.sh
|
| git revision | master / 47d202b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22453/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22453/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22453/yetus/diff-checkstyle-itests_hive-unit.txt
|
| findbugs |
[jira] [Commented] (HIVE-23498) Disable HTTP Trace method on ThriftHttpCliService
[
https://issues.apache.org/jira/browse/HIVE-23498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111011#comment-17111011
]
Hive QA commented on HIVE-23498:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003334/HIVE-23498.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22454/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22454/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22454/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-19 09:19:10.109
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22454/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-19 09:19:10.112
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 47d202b HIVE-23392: Metastore upgrade script TXN_LOCK_TBL rename
inconsistency (Peter Varga via Peter Vary)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 47d202b HIVE-23392: Metastore upgrade script TXN_LOCK_TBL rename
inconsistency (Peter Varga via Peter Vary)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-19 09:19:11.401
+ rm -rf ../yetus_PreCommit-HIVE-Build-22454
+ mkdir ../yetus_PreCommit-HIVE-Build-22454
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22454
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22454/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error:
a/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java:
does not exist in index
Trying to apply the patch with -p1
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc8180676865762747326.exe, --version]
libprotoc 2.6.1
protoc-jar: executing: [/tmp/protoc8180676865762747326.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc7900543739213010745.exe, --version]
libprotoc 2.6.1
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/parser/target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveLexer.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/parser/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
org/apache/hadoop/hive/ql/parse/HiveLexer.g
Output file
[jira] [Commented] (HIVE-23501) AOOB in VectorDeserializeRow when complex types are converted to primitive types
[ https://issues.apache.org/jira/browse/HIVE-23501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111000#comment-17111000 ] Panagiotis Garefalakis commented on HIVE-23501: --- Hey [~rameshkumar] seems that the patch uses your static user path for the input data -- should probably be replaced with a relative data path. > AOOB in VectorDeserializeRow when complex types are converted to primitive > types > > > Key: HIVE-23501 > URL: https://issues.apache.org/jira/browse/HIVE-23501 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23501.1.patch > > > AOOB in VectorDeserializeRow when complex types are converted to primitive > types -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23353) Atlas metadata replication scheduling
[
https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110997#comment-17110997
]
Hive QA commented on HIVE-23353:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003347/HIVE-23353.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17273 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22453/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22453/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22453/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003347 - PreCommit-HIVE-Build
> Atlas metadata replication scheduling
> -
>
> Key: HIVE-23353
> URL: https://issues.apache.org/jira/browse/HIVE-23353
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23353.01.patch, HIVE-23353.02.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23501) AOOB in VectorDeserializeRow when complex types are converted to primitive types
[ https://issues.apache.org/jira/browse/HIVE-23501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23501: Attachment: HIVE-23501.1.patch Status: Patch Available (was: Open) > AOOB in VectorDeserializeRow when complex types are converted to primitive > types > > > Key: HIVE-23501 > URL: https://issues.apache.org/jira/browse/HIVE-23501 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23501.1.patch > > > AOOB in VectorDeserializeRow when complex types are converted to primitive > types -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23501) AOOB in VectorDeserializeRow when complex types are converted to primitive types
[ https://issues.apache.org/jira/browse/HIVE-23501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan reassigned HIVE-23501: --- > AOOB in VectorDeserializeRow when complex types are converted to primitive > types > > > Key: HIVE-23501 > URL: https://issues.apache.org/jira/browse/HIVE-23501 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > > AOOB in VectorDeserializeRow when complex types are converted to primitive > types -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Status: Open (was: Patch Available) > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, > HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, > HIVE-22869.7.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Status: Patch Available (was: Open) > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, > HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, > HIVE-22869.7.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Attachment: HIVE-22869.7.patch > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, > HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.6.patch, > HIVE-22869.7.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23391: - Status: Open (was: Patch Available) > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.3.patch, > HIVE-23391.4.patch, HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23391: - Attachment: HIVE-23391.4.patch > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.3.patch, > HIVE-23391.4.patch, HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23391: - Status: Patch Available (was: Open) > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.3.patch, > HIVE-23391.4.patch, HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23500) [Kubernetes] Use Extend NodeId for LLAP registration
[ https://issues.apache.org/jira/browse/HIVE-23500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110957#comment-17110957 ] Attila Magyar commented on HIVE-23500: -- Yeah, just noticed. > [Kubernetes] Use Extend NodeId for LLAP registration > > > Key: HIVE-23500 > URL: https://issues.apache.org/jira/browse/HIVE-23500 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > > In kubernetes environment where pods can have same host name and port, there > can be situations where node trackers could be retaining old instance of the > pod in its cache. In case of Hive LLAP, where the llap tez task scheduler > maintains the membership of nodes based on zookeeper registry events there > can be cases where NODE_ADDED followed by NODE_REMOVED event could end up > removing the node/host from node trackers because of stable hostname and > service port. The NODE_REMOVED event in this case is old stale event of the > already dead pod but ZK will send only after session timeout (in case of > non-graceful shutdown). If this sequence of events happen, a node/host is > completely lost form the schedulers perspective. > To support this scenario, tez can extend yarn's NodeId to include > uniqueIdentifier. Llap task scheduler can construct the container object with > this new NodeId that includes uniqueIdentifier as well so that stale events > like above will only remove the host/node that matches the old > uniqueIdentifier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23353) Atlas metadata replication scheduling
[ https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-23353: -- Attachment: HIVE-23353.02.patch > Atlas metadata replication scheduling > - > > Key: HIVE-23353 > URL: https://issues.apache.org/jira/browse/HIVE-23353 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23353.01.patch, HIVE-23353.02.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23353) Atlas metadata replication scheduling
[ https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-23353: -- Attachment: (was: HIVE-23353.02.patch) > Atlas metadata replication scheduling > - > > Key: HIVE-23353 > URL: https://issues.apache.org/jira/browse/HIVE-23353 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23353.01.patch, HIVE-23353.02.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-23500) [Kubernetes] Use Extend NodeId for LLAP registration
[ https://issues.apache.org/jira/browse/HIVE-23500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar resolved HIVE-23500. -- Resolution: Duplicate > [Kubernetes] Use Extend NodeId for LLAP registration > > > Key: HIVE-23500 > URL: https://issues.apache.org/jira/browse/HIVE-23500 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > > In kubernetes environment where pods can have same host name and port, there > can be situations where node trackers could be retaining old instance of the > pod in its cache. In case of Hive LLAP, where the llap tez task scheduler > maintains the membership of nodes based on zookeeper registry events there > can be cases where NODE_ADDED followed by NODE_REMOVED event could end up > removing the node/host from node trackers because of stable hostname and > service port. The NODE_REMOVED event in this case is old stale event of the > already dead pod but ZK will send only after session timeout (in case of > non-graceful shutdown). If this sequence of events happen, a node/host is > completely lost form the schedulers perspective. > To support this scenario, tez can extend yarn's NodeId to include > uniqueIdentifier. Llap task scheduler can construct the container object with > this new NodeId that includes uniqueIdentifier as well so that stale events > like above will only remove the host/node that matches the old > uniqueIdentifier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23500) [Kubernetes] Use Extend NodeId for LLAP registration
[ https://issues.apache.org/jira/browse/HIVE-23500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110955#comment-17110955 ] Prasanth Jayachandran commented on HIVE-23500: -- HIVE-23466 is the same? > [Kubernetes] Use Extend NodeId for LLAP registration > > > Key: HIVE-23500 > URL: https://issues.apache.org/jira/browse/HIVE-23500 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > > In kubernetes environment where pods can have same host name and port, there > can be situations where node trackers could be retaining old instance of the > pod in its cache. In case of Hive LLAP, where the llap tez task scheduler > maintains the membership of nodes based on zookeeper registry events there > can be cases where NODE_ADDED followed by NODE_REMOVED event could end up > removing the node/host from node trackers because of stable hostname and > service port. The NODE_REMOVED event in this case is old stale event of the > already dead pod but ZK will send only after session timeout (in case of > non-graceful shutdown). If this sequence of events happen, a node/host is > completely lost form the schedulers perspective. > To support this scenario, tez can extend yarn's NodeId to include > uniqueIdentifier. Llap task scheduler can construct the container object with > this new NodeId that includes uniqueIdentifier as well so that stale events > like above will only remove the host/node that matches the old > uniqueIdentifier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23214) Remove skipCorrupt from OrcEncodedDataConsumer
[
https://issues.apache.org/jira/browse/HIVE-23214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110954#comment-17110954
]
Hive QA commented on HIVE-23214:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003324/HIVE-23214.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17269 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22452/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22452/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22452/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003324 - PreCommit-HIVE-Build
> Remove skipCorrupt from OrcEncodedDataConsumer
> --
>
> Key: HIVE-23214
> URL: https://issues.apache.org/jira/browse/HIVE-23214
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23214.01.patch, HIVE-23214.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> SkipCorrupt is always the default (false) so there is no reason to pass it
> around.
> [https://github.com/apache/hive/blob/3e4f6122c32b1ffa22e1458806ae8ee30e51a41f/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java#L86]
> If we want to change the default behaviour we could set
> "orc.skip.corrupt.data" as part of the configuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23500) [Kubernetes] Use Extend NodeId for LLAP registration
[ https://issues.apache.org/jira/browse/HIVE-23500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar reassigned HIVE-23500: > [Kubernetes] Use Extend NodeId for LLAP registration > > > Key: HIVE-23500 > URL: https://issues.apache.org/jira/browse/HIVE-23500 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > > In kubernetes environment where pods can have same host name and port, there > can be situations where node trackers could be retaining old instance of the > pod in its cache. In case of Hive LLAP, where the llap tez task scheduler > maintains the membership of nodes based on zookeeper registry events there > can be cases where NODE_ADDED followed by NODE_REMOVED event could end up > removing the node/host from node trackers because of stable hostname and > service port. The NODE_REMOVED event in this case is old stale event of the > already dead pod but ZK will send only after session timeout (in case of > non-graceful shutdown). If this sequence of events happen, a node/host is > completely lost form the schedulers perspective. > To support this scenario, tez can extend yarn's NodeId to include > uniqueIdentifier. Llap task scheduler can construct the container object with > this new NodeId that includes uniqueIdentifier as well so that stale events > like above will only remove the host/node that matches the old > uniqueIdentifier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23353) Atlas metadata replication scheduling
[ https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-23353: -- Attachment: HIVE-23353.02.patch > Atlas metadata replication scheduling > - > > Key: HIVE-23353 > URL: https://issues.apache.org/jira/browse/HIVE-23353 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23353.01.patch, HIVE-23353.02.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23353) Atlas metadata replication scheduling
[ https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-23353: -- Attachment: (was: HIVE-23353.02.patch) > Atlas metadata replication scheduling > - > > Key: HIVE-23353 > URL: https://issues.apache.org/jira/browse/HIVE-23353 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23353.01.patch, HIVE-23353.02.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23392) Metastore upgrade script TXN_LOCK_TBL rename inconsistency
[ https://issues.apache.org/jira/browse/HIVE-23392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23392: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the patch [~pvargacl]! > Metastore upgrade script TXN_LOCK_TBL rename inconsistency > -- > > Key: HIVE-23392 > URL: https://issues.apache.org/jira/browse/HIVE-23392 > Project: Hive > Issue Type: Bug >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23392.1.patch > > > HIVE-23048 introduced a bug in the metastore upgrade scripts, by not renaming > correctly the columns in TXN_LOCK_TBL -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23214) Remove skipCorrupt from OrcEncodedDataConsumer
[
https://issues.apache.org/jira/browse/HIVE-23214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110920#comment-17110920
]
Hive QA commented on HIVE-23214:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
51s{color} | {color:blue} llap-server in master has 88 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} llap-server: The patch generated 1 new + 158 unchanged
- 6 fixed = 159 total (was 164) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22452/dev-support/hive-personality.sh
|
| git revision | master / 61c9b2e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22452/yetus/diff-checkstyle-llap-server.txt
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22452/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove skipCorrupt from OrcEncodedDataConsumer
> --
>
> Key: HIVE-23214
> URL: https://issues.apache.org/jira/browse/HIVE-23214
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23214.01.patch, HIVE-23214.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> SkipCorrupt is always the default (false) so there is no reason to pass it
> around.
> [https://github.com/apache/hive/blob/3e4f6122c32b1ffa22e1458806ae8ee30e51a41f/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java#L86]
> If we want to change the default behaviour we could set
> "orc.skip.corrupt.data" as part of the configuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23495) AcidUtils.getAcidState cleanup
[
https://issues.apache.org/jira/browse/HIVE-23495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110905#comment-17110905
]
Hive QA commented on HIVE-23495:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003318/HIVE-23495.1.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 17268 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=27)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mm_exim]
(batchId=124)
org.apache.hadoop.hive.ql.TestTxnCommands.testMmExim (batchId=301)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1
(batchId=274)
org.apache.hadoop.hive.ql.TestTxnCommands2.updateDeletePartitioned (batchId=274)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1
(batchId=297)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.updateDeletePartitioned
(batchId=297)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteForPartitionedMmTable
(batchId=247)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithUnionAll
(batchId=247)
org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteForPartitionedMmTable
(batchId=271)
org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteWithUnionAll
(batchId=271)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMmExim
(batchId=277)
org.apache.hadoop.hive.ql.TestTxnExIm.testMM (batchId=278)
org.apache.hadoop.hive.ql.TestTxnExIm.testMMCreate (batchId=278)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithPartitionDirMissing
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithTableDirMissing
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testMultiDBTxn
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithPartitionDirMissing
(batchId=204)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithTableDirMissing
(batchId=204)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testMultiDBTxn
(batchId=204)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosIncrementalLoadAcidTables.testAcidTableIncrementalReplication
(batchId=205)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22451/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22451/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22451/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003318 - PreCommit-HIVE-Build
> AcidUtils.getAcidState cleanup
> --
>
> Key: HIVE-23495
> URL: https://issues.apache.org/jira/browse/HIVE-23495
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23495.1.patch
>
>
> since HIVE-21225 there are two redundant implementation of the
> AcidUtils.getAcidState.
> The previous implementation (without the recursive listing) can be removed.
> Also the performance can be improved, by removing unnecessary fileStatus
> calls.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23495) AcidUtils.getAcidState cleanup
[
https://issues.apache.org/jira/browse/HIVE-23495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110897#comment-17110897
]
Hive QA commented on HIVE-23495:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
25s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} ql: The patch generated 0 new + 863 unchanged - 5
fixed = 863 total (was 868) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} hcatalog/streaming: The patch generated 0 new + 228
unchanged - 1 fixed = 228 total (was 229) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} streaming: The patch generated 0 new + 453 unchanged
- 3 fixed = 453 total (was 456) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
9s{color} | {color:red} ql generated 2 new + 98 unchanged - 2 fixed = 100 total
(was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22451/dev-support/hive-personality.sh
|
| git revision | master / 61c9b2e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22451/yetus/whitespace-eol.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22451/yetus/diff-javadoc-javadoc-ql.txt
|
| modules | C: ql hcatalog/streaming streaming U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22451/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> AcidUtils.getAcidState cleanup
> --
>
> Key: HIVE-23495
> URL: https://issues.apache.org/jira/browse/HIVE-23495
> Project: Hive
> Issue Type: Improvement
>Reporter:
