[jira] [Commented] (HIVE-21857) Sort conditions in a filter predicate to accelerate query processing
[
https://issues.apache.org/jira/browse/HIVE-21857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872078#comment-16872078
]
Hive QA commented on HIVE-21857:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} common: The patch generated 1 new + 437 unchanged - 0
fixed = 438 total (was 437) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 48 new + 268 unchanged - 4
fixed = 316 total (was 272) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17720/dev-support/hive-personality.sh
|
| git revision | master / 84b5ba7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17720/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17720/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql accumulo-handler hbase-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17720/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Sort conditions in a filter predicate to accelerate query processing
>
>
> Key: HIVE-21857
> URL: https://issues.apache.org/jira/browse/HIVE-21857
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Cam
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Attachment: HIVE-21914.03.patch > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch, > HIVE-21914.03.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Attachment: (was: HIVE-21914.03.patch) > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch, > HIVE-21914.03.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Status: Open (was: Patch Available) > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch, > HIVE-21914.03.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Status: Patch Available (was: Open) > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch, > HIVE-21914.03.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Attachment: (was: HIVE-21914.03.patch) > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch, > HIVE-21914.03.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Attachment: HIVE-21914.03.patch > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch, > HIVE-21914.03.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21918) Handle each Alter Database types in a separate desc / operation
[
https://issues.apache.org/jira/browse/HIVE-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872049#comment-16872049
]
Hive QA commented on HIVE-21918:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972751/HIVE-21918.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16339 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17719/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17719/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17719/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972751 - PreCommit-HIVE-Build
> Handle each Alter Database types in a separate desc / operation
> ---
>
> Key: HIVE-21918
> URL: https://issues.apache.org/jira/browse/HIVE-21918
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21918.01.patch
>
>
> AlterDatabaseDesc / AlterDatabaseOperation handles all kind of alter database
> commands. Following the logic of the DDL handling, they should be in separate
> classes, with a mini framework handled by abstract ancestors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21918) Handle each Alter Database types in a separate desc / operation
[
https://issues.apache.org/jira/browse/HIVE-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872012#comment-16872012
]
Hive QA commented on HIVE-21918:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 1 new + 229 unchanged - 0
fixed = 230 total (was 229) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} ql generated 0 new + 2253 unchanged - 1 fixed = 2253
total (was 2254) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17719/dev-support/hive-personality.sh
|
| git revision | master / 84b5ba7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17719/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17719/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Handle each Alter Database types in a separate desc / operation
> ---
>
> Key: HIVE-21918
> URL: https://issues.apache.org/jira/browse/HIVE-21918
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21918.01.patch
>
>
> AlterDatabaseDesc / AlterDatabaseOperation handles all kind of alter database
> commands. Following the logic of the DDL handling, they should be in separate
> classes, with a mini framework handled by abstract ancestors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[
https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871994#comment-16871994
]
Hive QA commented on HIVE-21914:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972752/HIVE-21914.03.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16339 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence
(batchId=242)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking5
(batchId=340)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17718/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17718/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17718/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972752 - PreCommit-HIVE-Build
> Move Function and Macro related DDL operations into the DDL framework
> -
>
> Key: HIVE-21914
> URL: https://issues.apache.org/jira/browse/HIVE-21914
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch,
> HIVE-21914.03.patch
>
>
> Some Function and Macro related operations are handled by FunctionTask, and
> FunctionWork while they belong to the DDL framework.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[
https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871972#comment-16871972
]
Hive QA commented on HIVE-21914:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
19s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} llap-server in master has 82 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 1 new + 326 unchanged - 17
fixed = 327 total (was 343) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
26s{color} | {color:red} ql generated 1 new + 2253 unchanged - 1 fixed = 2254
total (was 2254) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA238 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
48391-48404] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17718/dev-support/hive-personality.sh
|
| git revision | master / 84b5ba7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17718/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17718/yetus/new-findbugs-ql.html
|
| modules | C: ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17718/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move Function and Macro related DDL operations into the DDL framework
> -
>
> Key: HIVE-21914
> URL: https://issues.apache.org/jira/browse/HIVE-21914
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch,
> HIVE-21914.03.patch
>
>
> Some Function and Macro related operations are handled by FunctionTask, and
> Fu
[jira] [Commented] (HIVE-21391) LLAP: Pool of column vector buffers can cause memory pressure
[
https://issues.apache.org/jira/browse/HIVE-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871950#comment-16871950
]
Hive QA commented on HIVE-21391:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972750/HIVE-21391.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16340 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17717/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17717/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17717/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972750 - PreCommit-HIVE-Build
> LLAP: Pool of column vector buffers can cause memory pressure
> -
>
> Key: HIVE-21391
> URL: https://issues.apache.org/jira/browse/HIVE-21391
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21391.1.patch, HIVE-21391.2.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Where there are too many columns (in the order of 100s), with decimal, string
> types the column vector pool of buffers created here
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java#L59]
> can cause memory pressure.
> Example:
> 128 (poolSize) * 300 (numCols) * 1024 (batchSize) * 80 (decimalSize) ~= 3GB
> The pool size keeps increasing when there is slow consumer but fast llap io
> (SSDs) leading to GC pressure when all LLAP io threads read splits from same
> table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21391) LLAP: Pool of column vector buffers can cause memory pressure
[
https://issues.apache.org/jira/browse/HIVE-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871932#comment-16871932
]
Hive QA commented on HIVE-21391:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} llap-common in master has 84 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} llap-server in master has 82 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
21s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
23s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 23s{color}
| {color:red} llap-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} storage-api: The patch generated 0 new + 0 unchanged
- 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch llap-common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} llap-server: The patch generated 8 new + 30 unchanged
- 6 fixed = 38 total (was 36) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
21s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17717/dev-support/hive-personality.sh
|
| git revision | master / 84b5ba7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17717/yetus/patch-mvninstall-llap-server.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17717/yetus/patch-compile-llap-server.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17717/yetus/patch-compile-llap-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17717/yetus/diff-checkstyle-llap-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17717/yetus/patch-findbugs
[jira] [Commented] (HIVE-18735) Create table like loses transactional attribute
[
https://issues.apache.org/jira/browse/HIVE-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871913#comment-16871913
]
Hive QA commented on HIVE-18735:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972721/HIVE-18735.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 47 failed/errored test(s), 16339 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_rename_table]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_stats_status]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2]
(batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_3]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_8]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_alter_list_bucketing_table1]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_like2]
(batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_like] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_like_tbl_props]
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_table_like_stats]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[describe_table]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_ddl] (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part8]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[merge3] (batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_default] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_discovery]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats12] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats13] (batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats14] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats15] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats2] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats4] (batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats6] (batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats7] (batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats8] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats9] (batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_noscan_1]
(batchId=63)
org.apache.hadoop.hive.cli.TestErasureCodingHDFSCliDriver.testCliDriver[erasure_simple]
(batchId=196)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_queries]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[whroot_external1]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[autoColumnStats_1]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[autoColumnStats_2]
(batchId=184)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_noscan_1]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part8]
(batchId=142)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats12]
(batchId=140)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats13]
(batchId=149)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats14]
(batchId=141)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats15]
(batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats2]
(batchId=140)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats6]
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats7]
(batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats8]
(batchId=145)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats9]
(batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats_noscan_1]
(batchId=138)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17716/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17716/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17716/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.exec
[jira] [Commented] (HIVE-18735) Create table like loses transactional attribute
[
https://issues.apache.org/jira/browse/HIVE-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871900#comment-16871900
]
Hive QA commented on HIVE-18735:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
19s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
32s{color} | {color:red} ql generated 1 new + 2254 unchanged - 0 fixed = 2255
total (was 2254) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Possible null pointer dereference of likeTable in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(ASTNode,
QB, SemanticAnalyzer$PlannerContext) Dereferenced at
SemanticAnalyzer.java:likeTable in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(ASTNode,
QB, SemanticAnalyzer$PlannerContext) Dereferenced at
SemanticAnalyzer.java:[line 13593] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17716/dev-support/hive-personality.sh
|
| git revision | master / 84b5ba7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17716/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17716/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create table like loses transactional attribute
> ---
>
> Key: HIVE-18735
> URL: https://issues.apache.org/jira/browse/HIVE-18735
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Eugene Koifman
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-18735.01.patch
>
>
> {noformat}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('transactional'='true')";
> create table T like T1;
> show create table T ;
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t'
> TBLPROP
[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21921: --- Status: Patch Available (was: Open) > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21921.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-21921: -- > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21921.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21921: --- Attachment: HIVE-21921.1.patch > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21921.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21874) Implement add partitions related methods on temporary table
[
https://issues.apache.org/jira/browse/HIVE-21874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871890#comment-16871890
]
Hive QA commented on HIVE-21874:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972734/HIVE-21874.02.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16597 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17715/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17715/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17715/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972734 - PreCommit-HIVE-Build
> Implement add partitions related methods on temporary table
> ---
>
> Key: HIVE-21874
> URL: https://issues.apache.org/jira/browse/HIVE-21874
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21874.01.patch, HIVE-21874.02.patch
>
>
> IMetaStoreClient exposes the following add partition related methods:
> {code:java}
> Partition add_partition(Partition partition);
> int add_partitions(List partitions);
> int add_partitions_pspec(PartitionSpecProxy partitionSpec);
> List add_partitions(List partitions, boolean
> ifNotExists, boolean needResults);
> {code}
> These methods should be implemented in order to handle addition of partitions
> to temporary tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21874) Implement add partitions related methods on temporary table
[
https://issues.apache.org/jira/browse/HIVE-21874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871881#comment-16871881
]
Hive QA commented on HIVE-21874:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
23s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 7 new + 15 unchanged - 0 fixed
= 22 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17715/dev-support/hive-personality.sh
|
| git revision | master / 84b5ba7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17715/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17715/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement add partitions related methods on temporary table
> ---
>
> Key: HIVE-21874
> URL: https://issues.apache.org/jira/browse/HIVE-21874
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21874.01.patch, HIVE-21874.02.patch
>
>
> IMetaStoreClient exposes the following add partition related methods:
> {code:java}
> Partition add_partition(Partition partition);
> int add_partitions(List partitions);
> int add_partitions_pspec(PartitionSpecProxy partitionSpec);
> List add_partitions(List partitions, boolean
> ifNotExists, boolean needResults);
> {code}
> These methods should be implemented in order to handle addition of partitions
> to temporary tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21868) Vectorize CAST...FORMAT
[
https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871863#comment-16871863
]
Hive QA commented on HIVE-21868:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972727/HIVE-21868.03.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16346 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17714/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17714/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17714/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972727 - PreCommit-HIVE-Build
> Vectorize CAST...FORMAT
> ---
>
> Key: HIVE-21868
> URL: https://issues.apache.org/jira/browse/HIVE-21868
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21868.01.patch, HIVE-21868.01.patch,
> HIVE-21868.02.patch, HIVE-21868.03.patch
>
>
> Vectorize UDFs for CAST ( AS STRING/CHAR/VARCHAR FORMAT
> ) and CAST ( AS TIMESTAMP/DATE FORMAT ).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21848) Table property name definition between ORC and Parquet encrytion
[ https://issues.apache.org/jira/browse/HIVE-21848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871852#comment-16871852 ] Owen O'Malley commented on HIVE-21848: -- You might also want to look at the ORC KeyProvider API since that is how ORC gets the key metadata and local keys. It currently is implemented by the Hadoop/Ranger KMS or an in memory implementation. It will also be implemented for the Amazon and Azure KMS. https://github.com/apache/orc/blob/master/java/shims/src/java/org/apache/orc/impl/HadoopShims.java#L150 > Table property name definition between ORC and Parquet encrytion > > > Key: HIVE-21848 > URL: https://issues.apache.org/jira/browse/HIVE-21848 > Project: Hive > Issue Type: Task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Xinli Shang >Assignee: Xinli Shang >Priority: Major > Fix For: 3.0.0 > > > The goal of this Jira is to define a superset of unified table property names > that can be used for both Parquet and ORC column encryption. There is no code > change needed for this Jira. > *Background:* > ORC-14 and Parquet-1178 introduced column encryption to ORC and Parquet. To > configure the encryption, e.g. which column is sensitive, what master key to > be used, algorithm, etc, table properties can be used. It is important that > both Parquet and ORC can use unified names. > According to the slide > [https://www.slideshare.net/oom65/fine-grain-access-control-for-big-data-orc-column-encryption-137308692], > ORC use table properties like orc.encrypt.pii, orc.encrypt.credit. While in > the Parquet community, it is still discussing to provide several ways and > using table properties is one of the options, while there is no detailed > design of the table property names yet. > So it is a good time to discuss within two communities to have unified table > names as a superset. > *Proposal:* > There are several encryption properties that need to be specified for a > table. Here is the list. This is the superset of Parquet and ORC. Some of > them might not apply to both. > # PII columns including nest columns > # Column key metadata, master key metadata > # Encryption algorithm, for example, Parquet support AES_GCM and AES_CTR. > ORC might support AES_CTR. > # Encryption footer - Parquet allow footer to be encrypted or plaintext > # Footer key metadata > Here is the table properties proposal. > |*Table Property Name*|*Value*|*Notes*| > |encrypt_algorithm|aes_ctr, aes_gcm|The algorithm to be used for encryption.| > |encrypt_footer_plaintext|true, false|Parquet support plaintext and encrypted > footer. By default, it is encrypted.| > |encrypt_footer_key_metadata|base64 string of footer key metadata|It is up to > the KMS to define what key metadata is. The metadata should have enough > information to figure out the corresponding key by the KMS. | > |encrypt_col_xxx|base64 string of column key metadata|‘xxx’ is the column > name for example, ‘address.zipcode’. > > It is up to the KMS to define what key metadata is. The metadata should have > enough information to figure out the corresponding key by the KMS.| > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21913) GenericUDTFGetSplits should handle usernames in the same way as LLAP
[ https://issues.apache.org/jira/browse/HIVE-21913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21913: - Resolution: Fixed Fix Version/s: 3.2.0 4.0.0 Status: Resolved (was: Patch Available) Committed to master and branch-3. Thanks for the review! > GenericUDTFGetSplits should handle usernames in the same way as LLAP > > > Key: HIVE-21913 > URL: https://issues.apache.org/jira/browse/HIVE-21913 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-21913.1.patch > > > LLAP ZK registry namespacing includes current user name which is typically > hive. But in some deployments, usernames are created with '_' like hive_dev. > RegistryUtils.currentUser() (that LLAP uses) replaces '_' with '-' for DNS > reasons. But GenericUDTFSplits uses UGI login user which does not do the > underscore replacement. As a result, LlapBaseInputFormat is finding any LLAP > daemons even though they are running. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21756) Backport HIVE-21404 to branch-3: MSSQL upgrade script alters the wrong column
[ https://issues.apache.org/jira/browse/HIVE-21756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-21756: -- Resolution: Fixed Status: Resolved (was: Patch Available) Patch committed to branch-3. > Backport HIVE-21404 to branch-3: MSSQL upgrade script alters the wrong column > - > > Key: HIVE-21756 > URL: https://issues.apache.org/jira/browse/HIVE-21756 > Project: Hive > Issue Type: Bug > Components: Metastore, Standalone Metastore >Affects Versions: 3.1.1 >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0, 3.1.2 > > Attachments: HIVE-21756.01.branch-3.patch, > HIVE-21756.branch-3.01.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Backport of HIVE-21404to branch-3. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-21225: Attachment: (was: HIVE-21225.4.patch) > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-21225: Attachment: HIVE-21225.4.patch > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-21225: Attachment: HIVE-21225.4.patch > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21868) Vectorize CAST...FORMAT
[
https://issues.apache.org/jira/browse/HIVE-21868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871833#comment-16871833
]
Hive QA commented on HIVE-21868:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} ql: The patch generated 0 new + 412 unchanged - 1
fixed = 412 total (was 413) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
46s{color} | {color:red} common generated 1 new + 62 unchanged - 0 fixed = 63
total (was 62) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
18s{color} | {color:red} ql generated 2 new + 2254 unchanged - 0 fixed = 2256
total (was 2254) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | Class
org.apache.hadoop.hive.common.format.datetime.HiveSqlDateTimeFormatter defines
non-transient non-serializable instance field tokens In
HiveSqlDateTimeFormatter.java:instance field tokens In
HiveSqlDateTimeFormatter.java |
| FindBugs | module:ql |
| | Found reliance on default encoding in
org.apache.hadoop.hive.ql.exec.vector.expressions.CastDateToString.sqlFormat(BytesColumnVector,
long[], int, HiveSqlDateTimeFormatter):in
org.apache.hadoop.hive.ql.exec.vector.expressions.CastDateToString.sqlFormat(BytesColumnVector,
long[], int, HiveSqlDateTimeFormatter): String.getBytes() At
CastDateToString.java:[line 70] |
| | Found reliance on default encoding in
org.apache.hadoop.hive.ql.exec.vector.expressions.CastTimestampToString.sqlFormat(BytesColumnVector,
TimestampColumnVector, int, HiveSqlDateTimeFormatter):in
org.apache.hadoop.hive.ql.exec.vector.expressions.CastTimestampToString.sqlFormat(BytesColumnVector,
TimestampColumnVector, int, HiveSqlDateTimeFormatter): String.getBytes() At
CastTimestampToString.java:[line 79] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17714/dev-support/hive-personality.sh
|
[jira] [Updated] (HIVE-21786) Update repo URLs in poms
[ https://issues.apache.org/jira/browse/HIVE-21786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-21786: -- Fix Version/s: 2.4.0 > Update repo URLs in poms > > > Key: HIVE-21786 > URL: https://issues.apache.org/jira/browse/HIVE-21786 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 4.0.0, 3.1.1, 2.3.5 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Fix For: 2.4.0, 4.0.0 > > Attachments: HIVE-21786.patch > > > Need to update repo URLs in the poms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21786) Update repo URLs in poms
[ https://issues.apache.org/jira/browse/HIVE-21786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871819#comment-16871819 ] Alan Gates commented on HIVE-21786: --- Patch committed to branch-2 > Update repo URLs in poms > > > Key: HIVE-21786 > URL: https://issues.apache.org/jira/browse/HIVE-21786 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 4.0.0, 3.1.1, 2.3.5 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Fix For: 2.4.0, 4.0.0 > > Attachments: HIVE-21786.patch > > > Need to update repo URLs in the poms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[
https://issues.apache.org/jira/browse/HIVE-21907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871802#comment-16871802
]
Hive QA commented on HIVE-21907:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972733/HIVE-21907.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 16345 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity
(batchId=350)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.org.apache.hadoop.hive.ql.TestWarehouseExternalDir
(batchId=255)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testExternalDefaultPaths
(batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17713/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17713/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17713/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972733 - PreCommit-HIVE-Build
> Add a new LlapDaemon Management API method to set the daemon capacity
> -
>
> Key: HIVE-21907
> URL: https://issues.apache.org/jira/browse/HIVE-21907
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21907.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Add a new method to LlapManagementProtocol API which can disable an Llap node.
> It would be even better, if we can dynamically set the number of executors
> and the size of the wait queue. This way we can disable the node setting them
> to 0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21680) Backport HIVE-17644 to branch-2 and branch-2.3
[
https://issues.apache.org/jira/browse/HIVE-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-21680:
--
Fix Version/s: 2.4.0
> Backport HIVE-17644 to branch-2 and branch-2.3
> --
>
> Key: HIVE-21680
> URL: https://issues.apache.org/jira/browse/HIVE-21680
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Fix For: 2.4.0, 2.3.5
>
> Attachments: HIVE-21680.branch-2.3.patch, HIVE-21680.branch-2.patch
>
>
> Backport HIVE-17644 to fix the warning in {{get statistics when not analyzed
> in Hive or Spark}}:
> {code:scala}
> test("get statistics when not analyzed in Hive or Spark") {
> val tabName = "tab1"
> withTable(tabName) {
> createNonPartitionedTable(tabName, analyzedByHive = false,
> analyzedBySpark = false)
> checkTableStats(tabName, hasSizeInBytes = true, expectedRowCounts =
> None)
> // ALTER TABLE SET TBLPROPERTIES invalidates some contents of Hive
> specific statistics
> // This is triggered by the Hive alterTable API
> val describeResult = hiveClient.runSqlHive(s"DESCRIBE FORMATTED
> $tabName")
> val rawDataSize = extractStatsPropValues(describeResult, "rawDataSize")
> val numRows = extractStatsPropValues(describeResult, "numRows")
> val totalSize = extractStatsPropValues(describeResult, "totalSize")
> assert(rawDataSize.isEmpty, "rawDataSize should not be shown without
> table analysis")
> assert(numRows.isEmpty, "numRows should not be shown without table
> analysis")
> assert(totalSize.isDefined && totalSize.get > 0, "totalSize is lost")
> }
> }
> //
> https://github.com/apache/spark/blob/43dcb91a4cb25aa7e1cc5967194f098029a0361e/sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala#L789-L806
> {code}
> {noformat}
> 06:23:46.103 WARN org.apache.hadoop.hive.metastore.MetaStoreDirectSql: Failed
> to execute [SELECT "DBS"."NAME", "TBLS"."TBL_NAME",
> "COLUMNS_V2"."COLUMN_NAME","KEY_CONSTRAINTS"."POSITION",
> "KEY_CONSTRAINTS"."CONSTRAINT_NAME", "KEY_CONSTRAINTS"."ENABLE_VALIDATE_RELY"
> FROM "TBLS" INNER JOIN "KEY_CONSTRAINTS" ON "TBLS"."TBL_ID" =
> "KEY_CONSTRAINTS"."PARENT_TBL_ID" INNER JOIN "DBS" ON "TBLS"."DB_ID" =
> "DBS"."DB_ID" INNER JOIN "COLUMNS_V2" ON "COLUMNS_V2"."CD_ID" =
> "KEY_CONSTRAINTS"."PARENT_CD_ID" AND "COLUMNS_V2"."INTEGER_IDX" =
> "KEY_CONSTRAINTS"."PARENT_INTEGER_IDX" WHERE
> "KEY_CONSTRAINTS"."CONSTRAINT_TYPE" = 0 AND "DBS"."NAME" = ? AND
> "TBLS"."TBL_NAME" = ?] with parameters [default, tab1]
> javax.jdo.JDODataStoreException: Error executing SQL query "SELECT
> "DBS"."NAME", "TBLS"."TBL_NAME",
> "COLUMNS_V2"."COLUMN_NAME","KEY_CONSTRAINTS"."POSITION",
> "KEY_CONSTRAINTS"."CONSTRAINT_NAME", "KEY_CONSTRAINTS"."ENABLE_VALIDATE_RELY"
> FROM "TBLS" INNER JOIN "KEY_CONSTRAINTS" ON "TBLS"."TBL_ID" =
> "KEY_CONSTRAINTS"."PARENT_TBL_ID" INNER JOIN "DBS" ON "TBLS"."DB_ID" =
> "DBS"."DB_ID" INNER JOIN "COLUMNS_V2" ON "COLUMNS_V2"."CD_ID" =
> "KEY_CONSTRAINTS"."PARENT_CD_ID" AND "COLUMNS_V2"."INTEGER_IDX" =
> "KEY_CONSTRAINTS"."PARENT_INTEGER_IDX" WHERE
> "KEY_CONSTRAINTS"."CONSTRAINT_TYPE" = 0 AND "DBS"."NAME" = ? AND
> "TBLS"."TBL_NAME" = ?".
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1750)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPrimaryKeys(MetaStoreDirectSql.java:1939)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:8213)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:8209)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2719)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPrimaryKeysInternal(ObjectStore.java:8221)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPrimaryKeys(ObjectStore.java:8199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
> at com.sun.proxy.$Proxy24.getPrimaryKeys(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.
[jira] [Commented] (HIVE-21680) Backport HIVE-17644 to branch-2 and branch-2.3
[
https://issues.apache.org/jira/browse/HIVE-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871795#comment-16871795
]
Alan Gates commented on HIVE-21680:
---
Committed patch to branch-2, which I should have done initially.
> Backport HIVE-17644 to branch-2 and branch-2.3
> --
>
> Key: HIVE-21680
> URL: https://issues.apache.org/jira/browse/HIVE-21680
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Fix For: 2.4.0, 2.3.5
>
> Attachments: HIVE-21680.branch-2.3.patch, HIVE-21680.branch-2.patch
>
>
> Backport HIVE-17644 to fix the warning in {{get statistics when not analyzed
> in Hive or Spark}}:
> {code:scala}
> test("get statistics when not analyzed in Hive or Spark") {
> val tabName = "tab1"
> withTable(tabName) {
> createNonPartitionedTable(tabName, analyzedByHive = false,
> analyzedBySpark = false)
> checkTableStats(tabName, hasSizeInBytes = true, expectedRowCounts =
> None)
> // ALTER TABLE SET TBLPROPERTIES invalidates some contents of Hive
> specific statistics
> // This is triggered by the Hive alterTable API
> val describeResult = hiveClient.runSqlHive(s"DESCRIBE FORMATTED
> $tabName")
> val rawDataSize = extractStatsPropValues(describeResult, "rawDataSize")
> val numRows = extractStatsPropValues(describeResult, "numRows")
> val totalSize = extractStatsPropValues(describeResult, "totalSize")
> assert(rawDataSize.isEmpty, "rawDataSize should not be shown without
> table analysis")
> assert(numRows.isEmpty, "numRows should not be shown without table
> analysis")
> assert(totalSize.isDefined && totalSize.get > 0, "totalSize is lost")
> }
> }
> //
> https://github.com/apache/spark/blob/43dcb91a4cb25aa7e1cc5967194f098029a0361e/sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala#L789-L806
> {code}
> {noformat}
> 06:23:46.103 WARN org.apache.hadoop.hive.metastore.MetaStoreDirectSql: Failed
> to execute [SELECT "DBS"."NAME", "TBLS"."TBL_NAME",
> "COLUMNS_V2"."COLUMN_NAME","KEY_CONSTRAINTS"."POSITION",
> "KEY_CONSTRAINTS"."CONSTRAINT_NAME", "KEY_CONSTRAINTS"."ENABLE_VALIDATE_RELY"
> FROM "TBLS" INNER JOIN "KEY_CONSTRAINTS" ON "TBLS"."TBL_ID" =
> "KEY_CONSTRAINTS"."PARENT_TBL_ID" INNER JOIN "DBS" ON "TBLS"."DB_ID" =
> "DBS"."DB_ID" INNER JOIN "COLUMNS_V2" ON "COLUMNS_V2"."CD_ID" =
> "KEY_CONSTRAINTS"."PARENT_CD_ID" AND "COLUMNS_V2"."INTEGER_IDX" =
> "KEY_CONSTRAINTS"."PARENT_INTEGER_IDX" WHERE
> "KEY_CONSTRAINTS"."CONSTRAINT_TYPE" = 0 AND "DBS"."NAME" = ? AND
> "TBLS"."TBL_NAME" = ?] with parameters [default, tab1]
> javax.jdo.JDODataStoreException: Error executing SQL query "SELECT
> "DBS"."NAME", "TBLS"."TBL_NAME",
> "COLUMNS_V2"."COLUMN_NAME","KEY_CONSTRAINTS"."POSITION",
> "KEY_CONSTRAINTS"."CONSTRAINT_NAME", "KEY_CONSTRAINTS"."ENABLE_VALIDATE_RELY"
> FROM "TBLS" INNER JOIN "KEY_CONSTRAINTS" ON "TBLS"."TBL_ID" =
> "KEY_CONSTRAINTS"."PARENT_TBL_ID" INNER JOIN "DBS" ON "TBLS"."DB_ID" =
> "DBS"."DB_ID" INNER JOIN "COLUMNS_V2" ON "COLUMNS_V2"."CD_ID" =
> "KEY_CONSTRAINTS"."PARENT_CD_ID" AND "COLUMNS_V2"."INTEGER_IDX" =
> "KEY_CONSTRAINTS"."PARENT_INTEGER_IDX" WHERE
> "KEY_CONSTRAINTS"."CONSTRAINT_TYPE" = 0 AND "DBS"."NAME" = ? AND
> "TBLS"."TBL_NAME" = ?".
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1750)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPrimaryKeys(MetaStoreDirectSql.java:1939)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:8213)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:8209)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2719)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPrimaryKeysInternal(ObjectStore.java:8221)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPrimaryKeys(ObjectStore.java:8199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
> at com.sun.proxy.$Proxy24.getPr
[jira] [Commented] (HIVE-21639) Spark test failed since HIVE-10632
[
https://issues.apache.org/jira/browse/HIVE-21639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871781#comment-16871781
]
Alan Gates commented on HIVE-21639:
---
Patch committed to branch-2.
> Spark test failed since HIVE-10632
> --
>
> Key: HIVE-21639
> URL: https://issues.apache.org/jira/browse/HIVE-21639
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Fix For: 2.4.0, 2.3.5
>
> Attachments: HIVE-21639.branch-2.3.patch, HIVE-21639.branch-2.patch
>
>
> We hint the following exception when [upgrading Spark‘s built-in Hive to
> 2.3.4|https://issues.apache.org/jira/browse/SPARK-23710]:
> {noformat}
> ..
> [info] Cause: java.sql.SQLException: Failed to start database 'metastore_db'
> with class loader
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@2439ab23, see
> the next exception for details.
> [info] at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> [info] at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> [info] at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
> [info] at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown
> Source)
> [info] at org.apache.derby.impl.jdbc.EmbedConnection.(Unknown Source)
> [info] at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
> ...
> {noformat}
> This issue Introduced by HIVE-10632 and fixed by HIVE-17561, I'd like to
> backport part of HIVE-17561 to fix this issue for branch-2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21639) Spark test failed since HIVE-10632
[
https://issues.apache.org/jira/browse/HIVE-21639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-21639:
--
Fix Version/s: 2.4.0
> Spark test failed since HIVE-10632
> --
>
> Key: HIVE-21639
> URL: https://issues.apache.org/jira/browse/HIVE-21639
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Fix For: 2.4.0, 2.3.5
>
> Attachments: HIVE-21639.branch-2.3.patch, HIVE-21639.branch-2.patch
>
>
> We hint the following exception when [upgrading Spark‘s built-in Hive to
> 2.3.4|https://issues.apache.org/jira/browse/SPARK-23710]:
> {noformat}
> ..
> [info] Cause: java.sql.SQLException: Failed to start database 'metastore_db'
> with class loader
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@2439ab23, see
> the next exception for details.
> [info] at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> [info] at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> [info] at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
> [info] at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown
> Source)
> [info] at org.apache.derby.impl.jdbc.EmbedConnection.(Unknown Source)
> [info] at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
> ...
> {noformat}
> This issue Introduced by HIVE-10632 and fixed by HIVE-17561, I'd like to
> backport part of HIVE-17561 to fix this issue for branch-2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21920) Extract command authorisation from the Driver
[ https://issues.apache.org/jira/browse/HIVE-21920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21920: -- Attachment: HIVE-21920.01.patch > Extract command authorisation from the Driver > - > > Key: HIVE-21920 > URL: https://issues.apache.org/jira/browse/HIVE-21920 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21920.01.patch > > > There are ~400 lines of command authorisation in the Driver class, which are > also used by ExplainTask. Extract them into a separate package under > org.apache.hadoop.hive.ql.security.authorization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21920) Extract command authorisation from the Driver
[ https://issues.apache.org/jira/browse/HIVE-21920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21920: -- Status: Patch Available (was: Open) > Extract command authorisation from the Driver > - > > Key: HIVE-21920 > URL: https://issues.apache.org/jira/browse/HIVE-21920 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21920.01.patch > > > There are ~400 lines of command authorisation in the Driver class, which are > also used by ExplainTask. Extract them into a separate package under > org.apache.hadoop.hive.ql.security.authorization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-15177: - Attachment: HIVE-15177.1.patch > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-15177.1.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21920) Extract command authorisation from the Driver
[ https://issues.apache.org/jira/browse/HIVE-21920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21920: -- Summary: Extract command authorisation from the Driver (was: Extract authorisation from the Driver) > Extract command authorisation from the Driver > - > > Key: HIVE-21920 > URL: https://issues.apache.org/jira/browse/HIVE-21920 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > There are ~400 lines of command authorisation in the Driver class, which are > also used by ExplainTask. Extract them into a separate package under > org.apache.hadoop.hive.ql.security.authorization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-15177: - Attachment: (was: HIVE-15177.patch) > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-15177.1.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21920) Extract authorisation from the Driver
[ https://issues.apache.org/jira/browse/HIVE-21920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-21920: - > Extract authorisation from the Driver > - > > Key: HIVE-21920 > URL: https://issues.apache.org/jira/browse/HIVE-21920 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > There are ~400 lines of command authorisation in the Driver class, which are > also used by ExplainTask. Extract them into a separate package under > org.apache.hadoop.hive.ql.security.authorization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21919) Refactor Driver
[ https://issues.apache.org/jira/browse/HIVE-21919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21919: -- Issue Type: Improvement (was: Bug) > Refactor Driver > --- > > Key: HIVE-21919 > URL: https://issues.apache.org/jira/browse/HIVE-21919 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > The Driver class is 3000+ lines long. It does a lot of things, it's structure > is hard to follow. Need to put it into a cleaner form, and thus make it more > readable. It should be cut into many pieces for having separate classes for > different subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21919) Refactor Driver
[ https://issues.apache.org/jira/browse/HIVE-21919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-21919: - > Refactor Driver > --- > > Key: HIVE-21919 > URL: https://issues.apache.org/jira/browse/HIVE-21919 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > The Driver class is 3000+ lines long. It does a lot of things, it's structure > is hard to follow. Need to put it into a cleaner form, and thus make it more > readable. It should be cut into many pieces for having separate classes for > different subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21918) Handle each Alter Database types in a separate desc / operation
[ https://issues.apache.org/jira/browse/HIVE-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21918: -- Issue Type: Sub-task (was: Improvement) Parent: HIVE-21346 > Handle each Alter Database types in a separate desc / operation > --- > > Key: HIVE-21918 > URL: https://issues.apache.org/jira/browse/HIVE-21918 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21918.01.patch > > > AlterDatabaseDesc / AlterDatabaseOperation handles all kind of alter database > commands. Following the logic of the DDL handling, they should be in separate > classes, with a mini framework handled by abstract ancestors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871777#comment-16871777 ] Prasanth Jayachandran commented on HIVE-15177: -- lgtm, +1 > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-15177.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21918) Handle each Alter Database types in a separate desc / operation
[ https://issues.apache.org/jira/browse/HIVE-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21918: -- Issue Type: Improvement (was: Bug) > Handle each Alter Database types in a separate desc / operation > --- > > Key: HIVE-21918 > URL: https://issues.apache.org/jira/browse/HIVE-21918 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21918.01.patch > > > AlterDatabaseDesc / AlterDatabaseOperation handles all kind of alter database > commands. Following the logic of the DDL handling, they should be in separate > classes, with a mini framework handled by abstract ancestors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871769#comment-16871769 ] Oliver Draese commented on HIVE-15177: -- [https://github.com/apache/hive/pull/686] > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-15177.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-15177: - Fix Version/s: 3.1.1 Status: Patch Available (was: Open) Added resolving of _HOST for the fromSubject kerberos path as well. > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-15177.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-15177: Assignee: Oliver Draese > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-15177.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-15177: - Attachment: HIVE-15177.patch > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-15177.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[
https://issues.apache.org/jira/browse/HIVE-21907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871764#comment-16871764
]
Hive QA commented on HIVE-21907:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} llap-common in master has 84 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
47s{color} | {color:blue} llap-server in master has 82 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
19s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 19s{color}
| {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} llap-server: The patch generated 13 new + 168
unchanged - 0 fixed = 181 total (was 168) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 10 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red} llap-common generated 6 new + 84 unchanged - 0 fixed =
90 total (was 84) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
20s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:llap-common |
| |
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$SetCapacityRequestProto.PARSER
isn't final but should be At LlapDaemonProtocolProtos.java:be At
LlapDaemonProtocolProtos.java:[line 21807] |
| | Class
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$SetCapacityRequestProto
defines non-transient non-serializable instance field unknownFields In
LlapDaemonProtocolProtos.java:instance field unknownFields In
LlapDaemonProtocolProtos.java |
| | Useless control flow in
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$SetCapacityRequestProto$Builder.maybeForceBuilderInitialization()
At LlapDaemonProtocolProtos.java: At LlapDaemonProtocolProtos.java:[line
22048] |
| |
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$SetCapacityResponseProto.PARSER
isn't final but should be At LlapDaemonProtocolProtos.java:be At
LlapDaemonProtocolProtos.java:[line 22300] |
| | Class
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$SetCapacityResponseProto
defines non-transient non-serializable instance field unknownFields In
LlapDaemonProtocolProtos.java:instance field unknownFields In
LlapDaemonProtocolProtos.java |
| | Useless control flow in
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$SetCapacityResponseProto$Builder.maybeForceBuilderInitialization()
At LlapDaemonProtocolProtos.java: At LlapDaemonProtocolProtos.java:[line
22474] |
\\
\\
|| Subsystem || Report/N
[jira] [Commented] (HIVE-21905) Generics improvement around the FetchOperator class
[
https://issues.apache.org/jira/browse/HIVE-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871753#comment-16871753
]
Hive QA commented on HIVE-21905:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972725/HIVE-21905.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16339 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17712/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17712/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17712/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972725 - PreCommit-HIVE-Build
> Generics improvement around the FetchOperator class
> ---
>
> Key: HIVE-21905
> URL: https://issues.apache.org/jira/browse/HIVE-21905
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Minor
> Attachments: HIVE-21905.1.patch, HIVE-21905.1.patch,
> HIVE-21905.2.patch
>
>
> In and around the org.apache.hadoop.hive.ql.exec.FetchOperator class the
> generics are handled poorly. Lot's of declarations are missing generics,
> which makes lots of noise in the IDE and makes it hard to be sure of the
> correctness of the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21905) Generics improvement around the FetchOperator class
[
https://issues.apache.org/jira/browse/HIVE-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871729#comment-16871729
]
Hive QA commented on HIVE-21905:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} ql: The patch generated 0 new + 152 unchanged - 1
fixed = 152 total (was 153) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} ql generated 0 new + 2253 unchanged - 1 fixed = 2253
total (was 2254) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17712/dev-support/hive-personality.sh
|
| git revision | master / c0a9ebd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17712/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Generics improvement around the FetchOperator class
> ---
>
> Key: HIVE-21905
> URL: https://issues.apache.org/jira/browse/HIVE-21905
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Minor
> Attachments: HIVE-21905.1.patch, HIVE-21905.1.patch,
> HIVE-21905.2.patch
>
>
> In and around the org.apache.hadoop.hive.ql.exec.FetchOperator class the
> generics are handled poorly. Lot's of declarations are missing generics,
> which makes lots of noise in the IDE and makes it hard to be sure of the
> correctness of the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17429) Hive JDBC doesn't return rows when querying Impala
[ https://issues.apache.org/jira/browse/HIVE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871728#comment-16871728 ] Alan Gates commented on HIVE-17429: --- Patch committed to branch-2 > Hive JDBC doesn't return rows when querying Impala > -- > > Key: HIVE-17429 > URL: https://issues.apache.org/jira/browse/HIVE-17429 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 2.1.0, 2.2.0, 2.3.0, 2.3.1, 2.3.2 >Reporter: Zach Amsden >Assignee: Zach Amsden >Priority: Major > Fix For: 2.1.0, 2.1.1, 3.0.0, 2.4.0, 2.2.1, 2.3.4 > > Attachments: HIVE-17429.1.patch, HIVE-17429.2.patch > > > The Hive JDBC driver used to return a result set when querying Impala. Now, > instead, it gets data back but interprets the data as query logs instead of a > resultSet. This causes many issues (we see complaints about beeline as well > as test failures). > This appears to be a regression introduced with asynchronous operation > against Hive. > Ideally, we could make both behaviors work. I have a simple patch that > should fix the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17429) Hive JDBC doesn't return rows when querying Impala
[ https://issues.apache.org/jira/browse/HIVE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-17429: -- Fix Version/s: 2.3.4 2.4.0 > Hive JDBC doesn't return rows when querying Impala > -- > > Key: HIVE-17429 > URL: https://issues.apache.org/jira/browse/HIVE-17429 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 2.1.0, 2.2.0, 2.3.0, 2.3.1, 2.3.2 >Reporter: Zach Amsden >Assignee: Zach Amsden >Priority: Major > Fix For: 2.1.0, 2.1.1, 3.0.0, 2.4.0, 2.2.1, 2.3.4 > > Attachments: HIVE-17429.1.patch, HIVE-17429.2.patch > > > The Hive JDBC driver used to return a result set when querying Impala. Now, > instead, it gets data back but interprets the data as query logs instead of a > resultSet. This causes many issues (we see complaints about beeline as well > as test failures). > This appears to be a regression introduced with asynchronous operation > against Hive. > Ideally, we could make both behaviors work. I have a simple patch that > should fix the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[
https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-21746:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Thanks [~jcamachorodriguez], I've committed to master.
> ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with
> CBO disabled
> --
>
> Key: HIVE-21746
> URL: https://issues.apache.org/jira/browse/HIVE-21746
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21746.hive2.patch, HIVE-21746.patch,
> HIVE-21746.patch, HIVE-21746.patch
>
>
> ArrayIndexOutOfBounds exception during query execution with dynamically
> partitioned hash join.
> Found on Hive 2.x. Seems to occur with CBO disabled/failed.
> Disabling constant propagation seems to allow the query to succeed.
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: 203
> at
> org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.unpack(MapJoinBytesTableContainer.java:744)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:730)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:605)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:70)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:34)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:819)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:924)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:456)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:359)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:290)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:377)
> ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_112]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> ~
[jira] [Updated] (HIVE-21825) Improve client error msg when Active/Passive HA is enabled
[
https://issues.apache.org/jira/browse/HIVE-21825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21825:
-
Resolution: Fixed
Fix Version/s: 3.2.0
4.0.0
Status: Resolved (was: Patch Available)
Committed to master and branch-3.2. Thanks [~rzhappy] for the contribution!
> Improve client error msg when Active/Passive HA is enabled
> --
>
> Key: HIVE-21825
> URL: https://issues.apache.org/jira/browse/HIVE-21825
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Richard Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 3.2.0
>
> Attachments: Hive-21825.1.patch, Hive-21825.2.patch,
> Hive-21825.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When Active/Passive HA is enabled and when client tries to connect to Passive
> HA or when HS2 is still starting up, clients will receive the following the
> error msg
> {code:java}
> 'Cannot open sessions on an inactive HS2 instance; use service discovery to
> connect'{code}
> This error msg can be improved to say that HS2 is still starting up (or more
> user-friendly error msg).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19568) Active/Passive HS2 HA: Disallow direct connection to passive HS2 instance
[ https://issues.apache.org/jira/browse/HIVE-19568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19568: - Fix Version/s: 3.2.0 > Active/Passive HS2 HA: Disallow direct connection to passive HS2 instance > - > > Key: HIVE-19568 > URL: https://issues.apache.org/jira/browse/HIVE-19568 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19568.01.patch, HIVE-19568.02.patch, > HIVE-19568.03.patch, HIVE-19568.04.patch, HIVE-19568.patch > > > The recommended usage for clients when connecting to HS2 with Active/Passive > HA configuration is via ZK service discovery URL. But some applications do > not support ZK service discovery in which case they use direct URL to connect > to HS2 instance. If direct connection is to passive HS2 instance, the > connection should be dropped with proper error message. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21915) Hive with TEZ UNION ALL and UDTF results in data loss
[
https://issues.apache.org/jira/browse/HIVE-21915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871688#comment-16871688
]
Hive QA commented on HIVE-21915:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972724/HIVE-21915.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17711/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17711/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17711/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12972724/HIVE-21915.01.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972724 - PreCommit-HIVE-Build
> Hive with TEZ UNION ALL and UDTF results in data loss
> -
>
> Key: HIVE-21915
> URL: https://issues.apache.org/jira/browse/HIVE-21915
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1
>Reporter: Wei Zhang
>Assignee: Wei Zhang
>Priority: Major
> Attachments: HIVE-21915.01.patch
>
>
> The HQL syntax is like this:
> CREATE TEMPORARY TABLE tez_union_all_loss_data AS
> SELECT xxx, yyy, zzz,1 as tag
> FROM ods_1
> UNION ALL
> SELECT xxx, yyy, zzz, tag
> FROM
> (
> SELECT xxx
> ,get_json_object(get_json_object(tb,'$.a'),'$.b') AS yyy
> ,zzz
> ,2 as tag
> FROM ods_2
> LATERAL VIEW EXPLODE(some_udf(uuu)) team_number AS tb
> ) tbl
> ;
>
> With above HQL, we are expecting that rows with both tag = 2 and tag = 1
> appear. In our case however, all the rows with tag = 1 are lost.
> Dig deeper we can find that the generated two maps have identical task tmp
> paths. And that results from when UDTF is present, the FileSinkOperator will
> be processed twice generating the tmp path in
> GenTezUtils.removeUnionOperators();
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21915) Hive with TEZ UNION ALL and UDTF results in data loss
[
https://issues.apache.org/jira/browse/HIVE-21915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871686#comment-16871686
]
Hive QA commented on HIVE-21915:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972724/HIVE-21915.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16339 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.service.TestDFSErrorHandling.testAccessDenied (batchId=272)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17710/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17710/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17710/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972724 - PreCommit-HIVE-Build
> Hive with TEZ UNION ALL and UDTF results in data loss
> -
>
> Key: HIVE-21915
> URL: https://issues.apache.org/jira/browse/HIVE-21915
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1
>Reporter: Wei Zhang
>Assignee: Wei Zhang
>Priority: Major
> Attachments: HIVE-21915.01.patch
>
>
> The HQL syntax is like this:
> CREATE TEMPORARY TABLE tez_union_all_loss_data AS
> SELECT xxx, yyy, zzz,1 as tag
> FROM ods_1
> UNION ALL
> SELECT xxx, yyy, zzz, tag
> FROM
> (
> SELECT xxx
> ,get_json_object(get_json_object(tb,'$.a'),'$.b') AS yyy
> ,zzz
> ,2 as tag
> FROM ods_2
> LATERAL VIEW EXPLODE(some_udf(uuu)) team_number AS tb
> ) tbl
> ;
>
> With above HQL, we are expecting that rows with both tag = 2 and tag = 1
> appear. In our case however, all the rows with tag = 1 are lost.
> Dig deeper we can find that the generated two maps have identical task tmp
> paths. And that results from when UDTF is present, the FileSinkOperator will
> be processed twice generating the tmp path in
> GenTezUtils.removeUnionOperators();
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16312) Flaky test: TestHCatClient.testTransportFailure
[
https://issues.apache.org/jira/browse/HIVE-16312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-16312:
--
Fix Version/s: 2.4.0
> Flaky test: TestHCatClient.testTransportFailure
> ---
>
> Key: HIVE-16312
> URL: https://issues.apache.org/jira/browse/HIVE-16312
> Project: Hive
> Issue Type: Sub-task
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
>Priority: Major
> Fix For: 3.0.0, 2.4.0, 2.3.2
>
> Attachments: HIVE-16312.01-branch-2.3.patch, HIVE-16312.01.patch
>
>
> The test has been failing consistently for 10+ builds.
> Error message:
> {code}
> Error Message
> The expected exception was never thrown.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16312) Flaky test: TestHCatClient.testTransportFailure
[
https://issues.apache.org/jira/browse/HIVE-16312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871670#comment-16871670
]
Alan Gates commented on HIVE-16312:
---
Committed to branch-2.
> Flaky test: TestHCatClient.testTransportFailure
> ---
>
> Key: HIVE-16312
> URL: https://issues.apache.org/jira/browse/HIVE-16312
> Project: Hive
> Issue Type: Sub-task
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
>Priority: Major
> Fix For: 3.0.0, 2.4.0, 2.3.2
>
> Attachments: HIVE-16312.01-branch-2.3.patch, HIVE-16312.01.patch
>
>
> The test has been failing consistently for 10+ builds.
> Error message:
> {code}
> Error Message
> The expected exception was never thrown.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[
https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871659#comment-16871659
]
Jesus Camacho Rodriguez commented on HIVE-21746:
+1
> ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with
> CBO disabled
> --
>
> Key: HIVE-21746
> URL: https://issues.apache.org/jira/browse/HIVE-21746
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-21746.hive2.patch, HIVE-21746.patch,
> HIVE-21746.patch, HIVE-21746.patch
>
>
> ArrayIndexOutOfBounds exception during query execution with dynamically
> partitioned hash join.
> Found on Hive 2.x. Seems to occur with CBO disabled/failed.
> Disabling constant propagation seems to allow the query to succeed.
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: 203
> at
> org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.unpack(MapJoinBytesTableContainer.java:744)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:730)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:605)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:70)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:34)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:819)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:924)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:456)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:359)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:290)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:377)
> ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_112]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> ~[hadoop-common-2.7.3.2.6.4.119-3.jar:?]
> at
> org.apache.tez.runtime.task.TaskRunner2Callabl
[jira] [Commented] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[
https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871654#comment-16871654
]
Jason Dere commented on HIVE-21746:
---
[~jcamachorodriguez] I was unable to get a qfile test working for this one, but
would you still be able to +1 this patch as this does fix error?
> ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with
> CBO disabled
> --
>
> Key: HIVE-21746
> URL: https://issues.apache.org/jira/browse/HIVE-21746
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-21746.hive2.patch, HIVE-21746.patch,
> HIVE-21746.patch, HIVE-21746.patch
>
>
> ArrayIndexOutOfBounds exception during query execution with dynamically
> partitioned hash join.
> Found on Hive 2.x. Seems to occur with CBO disabled/failed.
> Disabling constant propagation seems to allow the query to succeed.
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: 203
> at
> org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.unpack(MapJoinBytesTableContainer.java:744)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:730)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:605)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:70)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:34)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:819)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:924)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:456)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:359)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:290)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3]
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:377)
> ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_112]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
[jira] [Commented] (HIVE-17096) Fix test failures in 2.3 branch
[ https://issues.apache.org/jira/browse/HIVE-17096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871647#comment-16871647 ] Alan Gates commented on HIVE-17096: --- Committed patch to branch-2. > Fix test failures in 2.3 branch > --- > > Key: HIVE-17096 > URL: https://issues.apache.org/jira/browse/HIVE-17096 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Major > Fix For: 2.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15144) JSON.org license is now CatX
[
https://issues.apache.org/jira/browse/HIVE-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871638#comment-16871638
]
Alan Gates commented on HIVE-15144:
---
Committed to branch-2
> JSON.org license is now CatX
>
>
> Key: HIVE-15144
> URL: https://issues.apache.org/jira/browse/HIVE-15144
> Project: Hive
> Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Owen O'Malley
>Priority: Blocker
> Fix For: 2.2.0, 2.3.0, 3.0.0
>
> Attachments: HIVE-15144.patch, HIVE-15144.patch, HIVE-15144.patch,
> HIVE-15144.patch
>
>
> per [update resolved legal|http://www.apache.org/legal/resolved.html#json]:
> {quote}
> CAN APACHE PRODUCTS INCLUDE WORKS LICENSED UNDER THE JSON LICENSE?
> No. As of 2016-11-03 this has been moved to the 'Category X' license list.
> Prior to this, use of the JSON Java library was allowed. See Debian's page
> for a list of alternatives.
> {quote}
> I'm not sure when this dependency was first introduced, but it looks like
> it's currently used in a few places:
> https://github.com/apache/hive/search?p=1&q=%22org.json%22&utf8=%E2%9C%93
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21915) Hive with TEZ UNION ALL and UDTF results in data loss
[
https://issues.apache.org/jira/browse/HIVE-21915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871562#comment-16871562
]
Hive QA commented on HIVE-21915:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17710/dev-support/hive-personality.sh
|
| git revision | master / 11f7856 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17710/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive with TEZ UNION ALL and UDTF results in data loss
> -
>
> Key: HIVE-21915
> URL: https://issues.apache.org/jira/browse/HIVE-21915
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1
>Reporter: Wei Zhang
>Assignee: Wei Zhang
>Priority: Major
> Attachments: HIVE-21915.01.patch
>
>
> The HQL syntax is like this:
> CREATE TEMPORARY TABLE tez_union_all_loss_data AS
> SELECT xxx, yyy, zzz,1 as tag
> FROM ods_1
> UNION ALL
> SELECT xxx, yyy, zzz, tag
> FROM
> (
> SELECT xxx
> ,get_json_object(get_json_object(tb,'$.a'),'$.b') AS yyy
> ,zzz
> ,2 as tag
> FROM ods_2
> LATERAL VIEW EXPLODE(some_udf(uuu)) team_number AS tb
> ) tbl
> ;
>
> With above HQL, we are expecting that rows with both tag = 2 and tag = 1
> appear. In our case however, all the rows with tag = 1 are lost.
> Dig deeper we can find that the generated two maps have identical task tmp
> paths. And that results from when UDTF is present, the FileSinkOperator will
> be processed twice generating the tmp path in
> GenTezUtils.removeUnionOperators();
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16640) The ASF Headers have some errors in some class
[ https://issues.apache.org/jira/browse/HIVE-16640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871550#comment-16871550 ] Alan Gates commented on HIVE-16640: --- Commit to branch-2, which should have been done initially when I committed it to branch-2.3 > The ASF Headers have some errors in some class > -- > > Key: HIVE-16640 > URL: https://issues.apache.org/jira/browse/HIVE-16640 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Minor > Fix For: 2.3.0, 3.0.0 > > Attachments: HIVE-16640.1.patch > > > I found some class license hive placed in an incorrect location, some classes > missing license -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21880) Enable flaky test TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
[
https://issues.apache.org/jira/browse/HIVE-21880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871535#comment-16871535
]
Hive QA commented on HIVE-21880:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972710/HIVE-21880.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 155 failed/errored test(s), 16100 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=281)
TestHCatClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=206)
TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml
file (likely timed out) (batchId=255)
TestReplWithJsonMessageFormat - did not produce a TEST-*.xml file (likely timed
out) (batchId=249)
TestReplicationScenarios - did not produce a TEST-*.xml file (likely timed out)
(batchId=256)
TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file
(likely timed out) (batchId=257)
TestReplicationScenariosExternalTables - did not produce a TEST-*.xml file
(likely timed out) (batchId=255)
TestTableLevelReplicationScenarios - did not produce a TEST-*.xml file (likely
timed out) (batchId=255)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=170)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testNotificationOps
(batchId=232)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testAggrStat
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testAggrStatAbortTxn
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testAggrStatTxnTable
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testDatabaseOpsForUpdateUsingEvents
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testPartitionOpsForUpdateUsingEvents
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableColumnStatistics
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableColumnStatisticsTxnTable
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableColumnStatisticsTxnTableMulti
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableColumnStatisticsTxnTableMultiAbort
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableColumnStatisticsTxnTableOpenTxn
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableOpsForUpdateUsingEvents
(batchId=242)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=265)
org.apache.hadoop.hive.ql.parse.TestExportImport.dataImportAfterMetadataOnlyImport
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestExportImport.databaseTheTableIsImportedIntoShouldBeParsedFromCommandLine
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestExportImport.shouldExportImportATemporaryTable
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestExportImport.testExportExternalTableSetFalse
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestExportImport.testExportExternalTableSetTrue
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestExportImport.testExportNonNativeTable
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.retryIncBootstrapAcidFromDifferentDumpWithoutCleanTablesConfig
(batchId=248)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testAcidTablesBootstrapDuringIncremental
(batchId=248)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testAcidTablesBootstrapDuringIncrementalWithOpenTxnsTimeout
(batchId=248)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
(batchId=248)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testRetryAcidTablesBootstrapFromDifferentDump
(batchId=248)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAbortTxnEvent
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidBootstrapReplLoadRetryAfterFailure
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesBootstrap
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesBootstrapWithConcurrentDropTable
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesBootstrapWithConcurrentWrites
(batchId=260)
org.ap
[jira] [Commented] (HIVE-21848) Table property name definition between ORC and Parquet encrytion
[
https://issues.apache.org/jira/browse/HIVE-21848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871532#comment-16871532
]
Owen O'Malley commented on HIVE-21848:
--
For ORC, given a structure like:
{code}
table Customers (
name: struct,
address: struct
credit_cards:list>
)
{code}
If you encrypt *address*, all of the subfields of *address* will be encrypted.
The column finding code in ORC also deals with subfields and so
*address.street* will return the *street* subfield. The other complex types are
also addressable as:
* list child = _elem
* map children = _key and _value
* union children =
So to encrypt just the card numbers, it would be "encrypt.columns" =
"credit:credit_cards._elem.card_number".
That said, I assumed that Parquet didn't have that kind of column handling, but
that it wouldn't be hard to build. However, the vast majority of the column
encryption is likely to be at the top level and thus if Parquet can start with
that it would be great.
> Table property name definition between ORC and Parquet encrytion
>
>
> Key: HIVE-21848
> URL: https://issues.apache.org/jira/browse/HIVE-21848
> Project: Hive
> Issue Type: Task
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Xinli Shang
>Assignee: Xinli Shang
>Priority: Major
> Fix For: 3.0.0
>
>
> The goal of this Jira is to define a superset of unified table property names
> that can be used for both Parquet and ORC column encryption. There is no code
> change needed for this Jira.
> *Background:*
> ORC-14 and Parquet-1178 introduced column encryption to ORC and Parquet. To
> configure the encryption, e.g. which column is sensitive, what master key to
> be used, algorithm, etc, table properties can be used. It is important that
> both Parquet and ORC can use unified names.
> According to the slide
> [https://www.slideshare.net/oom65/fine-grain-access-control-for-big-data-orc-column-encryption-137308692],
> ORC use table properties like orc.encrypt.pii, orc.encrypt.credit. While in
> the Parquet community, it is still discussing to provide several ways and
> using table properties is one of the options, while there is no detailed
> design of the table property names yet.
> So it is a good time to discuss within two communities to have unified table
> names as a superset.
> *Proposal:*
> There are several encryption properties that need to be specified for a
> table. Here is the list. This is the superset of Parquet and ORC. Some of
> them might not apply to both.
> # PII columns including nest columns
> # Column key metadata, master key metadata
> # Encryption algorithm, for example, Parquet support AES_GCM and AES_CTR.
> ORC might support AES_CTR.
> # Encryption footer - Parquet allow footer to be encrypted or plaintext
> # Footer key metadata
> Here is the table properties proposal.
> |*Table Property Name*|*Value*|*Notes*|
> |encrypt_algorithm|aes_ctr, aes_gcm|The algorithm to be used for encryption.|
> |encrypt_footer_plaintext|true, false|Parquet support plaintext and encrypted
> footer. By default, it is encrypted.|
> |encrypt_footer_key_metadata|base64 string of footer key metadata|It is up to
> the KMS to define what key metadata is. The metadata should have enough
> information to figure out the corresponding key by the KMS. |
> |encrypt_col_xxx|base64 string of column key metadata|‘xxx’ is the column
> name for example, ‘address.zipcode’.
>
> It is up to the KMS to define what key metadata is. The metadata should have
> enough information to figure out the corresponding key by the KMS.|
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21857) Sort conditions in a filter predicate to accelerate query processing
[
https://issues.apache.org/jira/browse/HIVE-21857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871529#comment-16871529
]
Vineet Garg commented on HIVE-21857:
+1 pending tests.
> Sort conditions in a filter predicate to accelerate query processing
>
>
> Key: HIVE-21857
> URL: https://issues.apache.org/jira/browse/HIVE-21857
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21857.01.patch, HIVE-21857.02.patch,
> HIVE-21857.03.patch, HIVE-21857.04.patch, HIVE-21857.05.patch,
> HIVE-21857.06.patch, HIVE-21857.07.patch, HIVE-21857.08.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Following approach similar to
> http://db.cs.berkeley.edu/jmh/miscpapers/sigmod93.pdf .
> To reorder predicates in AND conditions, we could rank each of elements in
> the clauses in increasing order based on following formula:
> {code}
> rank = (selectivity - 1) / cost per tuple
> {code}
> Similarly, for OR conditions:
> {code}
> rank = (-selectivity) / cost per tuple
> {code}
> Selectivity can be computed with FilterSelectivityEstimator. For cost per
> tuple, we will need to come up with some heuristic based on how expensive is
> the evaluation of the functions contained in that predicate. Custom UDFs
> could be annotated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21880) Enable flaky test TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
[
https://issues.apache.org/jira/browse/HIVE-21880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871514#comment-16871514
]
Hive QA commented on HIVE-21880:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
35s{color} | {color:blue} standalone-metastore/metastore-common in master has
31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} itests/hcatalog-unit: The patch generated 6 new + 0
unchanged - 0 fixed = 6 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 32
unchanged - 0 fixed = 33 total (was 32) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17709/dev-support/hive-personality.sh
|
| git revision | master / 11f7856 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17709/yetus/diff-checkstyle-itests_hcatalog-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17709/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql hcatalog/server-extensions
itests/hcatalog-unit itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17709/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enable flaky test
> TestRepli
[jira] [Commented] (HIVE-21848) Table property name definition between ORC and Parquet encrytion
[ https://issues.apache.org/jira/browse/HIVE-21848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871510#comment-16871510 ] Gidon Gershinsky commented on HIVE-21848: - I'm also fine with this variant. Since key ids can be long and ugly, its good to keep them on the right side, and to avoid their repetition. > Table property name definition between ORC and Parquet encrytion > > > Key: HIVE-21848 > URL: https://issues.apache.org/jira/browse/HIVE-21848 > Project: Hive > Issue Type: Task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Xinli Shang >Assignee: Xinli Shang >Priority: Major > Fix For: 3.0.0 > > > The goal of this Jira is to define a superset of unified table property names > that can be used for both Parquet and ORC column encryption. There is no code > change needed for this Jira. > *Background:* > ORC-14 and Parquet-1178 introduced column encryption to ORC and Parquet. To > configure the encryption, e.g. which column is sensitive, what master key to > be used, algorithm, etc, table properties can be used. It is important that > both Parquet and ORC can use unified names. > According to the slide > [https://www.slideshare.net/oom65/fine-grain-access-control-for-big-data-orc-column-encryption-137308692], > ORC use table properties like orc.encrypt.pii, orc.encrypt.credit. While in > the Parquet community, it is still discussing to provide several ways and > using table properties is one of the options, while there is no detailed > design of the table property names yet. > So it is a good time to discuss within two communities to have unified table > names as a superset. > *Proposal:* > There are several encryption properties that need to be specified for a > table. Here is the list. This is the superset of Parquet and ORC. Some of > them might not apply to both. > # PII columns including nest columns > # Column key metadata, master key metadata > # Encryption algorithm, for example, Parquet support AES_GCM and AES_CTR. > ORC might support AES_CTR. > # Encryption footer - Parquet allow footer to be encrypted or plaintext > # Footer key metadata > Here is the table properties proposal. > |*Table Property Name*|*Value*|*Notes*| > |encrypt_algorithm|aes_ctr, aes_gcm|The algorithm to be used for encryption.| > |encrypt_footer_plaintext|true, false|Parquet support plaintext and encrypted > footer. By default, it is encrypted.| > |encrypt_footer_key_metadata|base64 string of footer key metadata|It is up to > the KMS to define what key metadata is. The metadata should have enough > information to figure out the corresponding key by the KMS. | > |encrypt_col_xxx|base64 string of column key metadata|‘xxx’ is the column > name for example, ‘address.zipcode’. > > It is up to the KMS to define what key metadata is. The metadata should have > enough information to figure out the corresponding key by the KMS.| > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21915) Hive with TEZ UNION ALL and UDTF results in data loss
[ https://issues.apache.org/jira/browse/HIVE-21915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871507#comment-16871507 ] Vineet Garg commented on HIVE-21915: [~zhangweilst] Thanks for providing the patch. Can you please add a test? > Hive with TEZ UNION ALL and UDTF results in data loss > - > > Key: HIVE-21915 > URL: https://issues.apache.org/jira/browse/HIVE-21915 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1 >Reporter: Wei Zhang >Assignee: Wei Zhang >Priority: Major > Attachments: HIVE-21915.01.patch > > > The HQL syntax is like this: > CREATE TEMPORARY TABLE tez_union_all_loss_data AS > SELECT xxx, yyy, zzz,1 as tag > FROM ods_1 > UNION ALL > SELECT xxx, yyy, zzz, tag > FROM > ( > SELECT xxx > ,get_json_object(get_json_object(tb,'$.a'),'$.b') AS yyy > ,zzz > ,2 as tag > FROM ods_2 > LATERAL VIEW EXPLODE(some_udf(uuu)) team_number AS tb > ) tbl > ; > > With above HQL, we are expecting that rows with both tag = 2 and tag = 1 > appear. In our case however, all the rows with tag = 1 are lost. > Dig deeper we can find that the generated two maps have identical task tmp > paths. And that results from when UDTF is present, the FileSinkOperator will > be processed twice generating the tmp path in > GenTezUtils.removeUnionOperators(); > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21846) Create a thread in TezAM which periodically fetches LlapDaemon metrics
[
https://issues.apache.org/jira/browse/HIVE-21846?focusedWorklogId=265788&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265788
]
ASF GitHub Bot logged work on HIVE-21846:
-
Author: ASF GitHub Bot
Created on: 24/Jun/19 16:17
Start Date: 24/Jun/19 16:17
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #683: HIVE-21846:
Create a thread in TezAM which periodically fetches LlapDaemon metrics
URL: https://github.com/apache/hive/pull/683#discussion_r296803805
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -790,6 +794,8 @@ public void run() {
}
}, 0, 1L, TimeUnit.MILLISECONDS);
+ llapMetricsCollector.start();
Review comment:
Should we maybe move the start below the code, adding the collector to the
registry? Otherwise, the collector doesn't have any LLAP daemons and the first
scheduler execution is a NOOP
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 265788)
Time Spent: 40m (was: 0.5h)
> Create a thread in TezAM which periodically fetches LlapDaemon metrics
> --
>
> Key: HIVE-21846
> URL: https://issues.apache.org/jira/browse/HIVE-21846
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21846.01.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> LlapTaskSchedulerService should start a thread which periodically fetches the
> LlapDaemon metrics and stores them in the NodeInfo object.
> This should be just the first implementation - later we should find a way
> where we do not need NxM requests between N TezAM and M LlapDaemon
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21846) Create a thread in TezAM which periodically fetches LlapDaemon metrics
[
https://issues.apache.org/jira/browse/HIVE-21846?focusedWorklogId=265790&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265790
]
ASF GitHub Bot logged work on HIVE-21846:
-
Author: ASF GitHub Bot
Created on: 24/Jun/19 16:17
Start Date: 24/Jun/19 16:17
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #683: HIVE-21846:
Create a thread in TezAM which periodically fetches LlapDaemon metrics
URL: https://github.com/apache/hive/pull/683#discussion_r296801109
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapMetricsCollector.java
##
@@ -0,0 +1,194 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.tezplugins.metrics;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import com.google.protobuf.ServiceException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.registry.ServiceInstanceStateChangeListener;
+import org.apache.hadoop.io.retry.RetryPolicies;
+import org.apache.hadoop.io.retry.RetryPolicy;
+import org.apache.hadoop.net.NetUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.SocketFactory;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Collect metrics from the llap daemons in every given milliseconds.
+ */
+public class LlapMetricsCollector implements
ServiceInstanceStateChangeListener {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(LlapMetricsCollector.class);
+ private static final String THREAD_NAME =
"LlapTaskSchedulerMetricsCollectorThread";
+ private static final long INITIAL_DELAY_MSEC = 6L;
+
+ private final Configuration conf;
+ private final ScheduledExecutorService scheduledMetricsExecutor;
+ private final LlapManagementProtocolClientImplFactory clientFactory;
+ private final Map llapClients;
+ private final Map instanceStatisticsMap;
+ private final long metricsCollectionMs;
+
+
+ public LlapMetricsCollector(Configuration conf) {
+this(
+conf,
+Executors.newSingleThreadScheduledExecutor(
+new
ThreadFactoryBuilder().setDaemon(true).setNameFormat(THREAD_NAME)
+.build()),
+LlapManagementProtocolClientImplFactory.basicInstance(conf));
+ }
+
+ @VisibleForTesting
+ LlapMetricsCollector(Configuration conf, ScheduledExecutorService
scheduledMetricsExecutor,
+ LlapManagementProtocolClientImplFactory clientFactory) {
+this.conf = conf;
+this.scheduledMetricsExecutor = scheduledMetricsExecutor;
+this.clientFactory = clientFactory;
+this.llapClients = new HashMap<>();
+this.instanceStatisticsMap = new ConcurrentHashMap<>();
+this.metricsCollectionMs = HiveConf.getTimeVar(conf,
+
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_AM_COLLECT_DAEMON_METRICS_MS,
TimeUnit.MILLISECONDS);
+ }
+
+ public void start() {
+if (enabled()) {
+ scheduledMetricsExecutor.scheduleAtFixedRate(() -> {
+collectMetrics();
+ }, INITIAL_DELAY_MSEC, metricsCollectionMs, TimeUnit.MILLISECONDS);
+}
+ }
+
+ private boolean enabled() {
+return metricsCollectionMs > 0;
+ }
+
+ public void shutdown() {
+scheduledMetricsExecutor.shutdownNow();
+ }
+
+ @VisibleForTesting
+ void collectMetrics() {
+if (enabled()) {
+ for (Map.Entry entry :
llapClients.entrySet()) {
+String identity = entry.getKey();
+LlapManagementProtocolClientImpl client = entry.getValue();
+try {
+ LlapDaemonProtocolProtos.GetDaemonMetricsResponseProto metrics =
+ client.getDaemonMetrics(null,
+
LlapDaemonProtocolProtos.GetDaemonMetricsRequestProto.newBuilder().build());
+ instance
[jira] [Updated] (HIVE-21857) Sort conditions in a filter predicate to accelerate query processing
[
https://issues.apache.org/jira/browse/HIVE-21857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21857:
---
Attachment: HIVE-21857.08.patch
> Sort conditions in a filter predicate to accelerate query processing
>
>
> Key: HIVE-21857
> URL: https://issues.apache.org/jira/browse/HIVE-21857
> Project: Hive
> Issue Type: New Feature
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21857.01.patch, HIVE-21857.02.patch,
> HIVE-21857.03.patch, HIVE-21857.04.patch, HIVE-21857.05.patch,
> HIVE-21857.06.patch, HIVE-21857.07.patch, HIVE-21857.08.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Following approach similar to
> http://db.cs.berkeley.edu/jmh/miscpapers/sigmod93.pdf .
> To reorder predicates in AND conditions, we could rank each of elements in
> the clauses in increasing order based on following formula:
> {code}
> rank = (selectivity - 1) / cost per tuple
> {code}
> Similarly, for OR conditions:
> {code}
> rank = (-selectivity) / cost per tuple
> {code}
> Selectivity can be computed with FilterSelectivityEstimator. For cost per
> tuple, we will need to come up with some heuristic based on how expensive is
> the evaluation of the functions contained in that predicate. Custom UDFs
> could be annotated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21846) Create a thread in TezAM which periodically fetches LlapDaemon metrics
[
https://issues.apache.org/jira/browse/HIVE-21846?focusedWorklogId=265789&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265789
]
ASF GitHub Bot logged work on HIVE-21846:
-
Author: ASF GitHub Bot
Created on: 24/Jun/19 16:17
Start Date: 24/Jun/19 16:17
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #683: HIVE-21846:
Create a thread in TezAM which periodically fetches LlapDaemon metrics
URL: https://github.com/apache/hive/pull/683#discussion_r296798202
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapMetricsCollector.java
##
@@ -0,0 +1,194 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.tezplugins.metrics;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import com.google.protobuf.ServiceException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.registry.ServiceInstanceStateChangeListener;
+import org.apache.hadoop.io.retry.RetryPolicies;
+import org.apache.hadoop.io.retry.RetryPolicy;
+import org.apache.hadoop.net.NetUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.SocketFactory;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Collect metrics from the llap daemons in every given milliseconds.
+ */
+public class LlapMetricsCollector implements
ServiceInstanceStateChangeListener {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(LlapMetricsCollector.class);
+ private static final String THREAD_NAME =
"LlapTaskSchedulerMetricsCollectorThread";
+ private static final long INITIAL_DELAY_MSEC = 6L;
+
+ private final Configuration conf;
Review comment:
This conf is currently used nowhere.
I would suggest to either make the collector a Configurable (and have a
getConf()) or to remove the variable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 265789)
Time Spent: 50m (was: 40m)
> Create a thread in TezAM which periodically fetches LlapDaemon metrics
> --
>
> Key: HIVE-21846
> URL: https://issues.apache.org/jira/browse/HIVE-21846
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21846.01.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> LlapTaskSchedulerService should start a thread which periodically fetches the
> LlapDaemon metrics and stores them in the NodeInfo object.
> This should be just the first implementation - later we should find a way
> where we do not need NxM requests between N TezAM and M LlapDaemon
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21846) Create a thread in TezAM which periodically fetches LlapDaemon metrics
[
https://issues.apache.org/jira/browse/HIVE-21846?focusedWorklogId=265786&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265786
]
ASF GitHub Bot logged work on HIVE-21846:
-
Author: ASF GitHub Bot
Created on: 24/Jun/19 16:17
Start Date: 24/Jun/19 16:17
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #683: HIVE-21846:
Create a thread in TezAM which periodically fetches LlapDaemon metrics
URL: https://github.com/apache/hive/pull/683#discussion_r296797083
##
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##
@@ -4342,6 +4342,9 @@ private static void
populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal
"Number of threads to use in LLAP task plugin client."),
LLAP_DAEMON_DOWNLOAD_PERMANENT_FNS("hive.llap.daemon.download.permanent.fns",
false,
"Whether LLAP daemon should localize the resources for permanent
UDFs."),
+
LLAP_TASK_SCHEDULER_AM_COLLECT_DAEMON_METRICS_MS("hive.llap.task.scheduler.am.collect.daemon.metrics",
"0ms",
+ new TimeValidator(TimeUnit.MILLISECONDS), "Collect llap daemon metrics
in the AM every given milliseconds." +
Review comment:
Some hint, why these metrics are collected every x milliseconds (reason)
would be good.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 265786)
Time Spent: 20m (was: 10m)
> Create a thread in TezAM which periodically fetches LlapDaemon metrics
> --
>
> Key: HIVE-21846
> URL: https://issues.apache.org/jira/browse/HIVE-21846
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21846.01.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> LlapTaskSchedulerService should start a thread which periodically fetches the
> LlapDaemon metrics and stores them in the NodeInfo object.
> This should be just the first implementation - later we should find a way
> where we do not need NxM requests between N TezAM and M LlapDaemon
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21846) Create a thread in TezAM which periodically fetches LlapDaemon metrics
[
https://issues.apache.org/jira/browse/HIVE-21846?focusedWorklogId=265787&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265787
]
ASF GitHub Bot logged work on HIVE-21846:
-
Author: ASF GitHub Bot
Created on: 24/Jun/19 16:17
Start Date: 24/Jun/19 16:17
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #683: HIVE-21846:
Create a thread in TezAM which periodically fetches LlapDaemon metrics
URL: https://github.com/apache/hive/pull/683#discussion_r296799724
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/metrics/LlapMetricsCollector.java
##
@@ -0,0 +1,194 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap.tezplugins.metrics;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import com.google.protobuf.ServiceException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.registry.ServiceInstanceStateChangeListener;
+import org.apache.hadoop.io.retry.RetryPolicies;
+import org.apache.hadoop.io.retry.RetryPolicy;
+import org.apache.hadoop.net.NetUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.SocketFactory;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Collect metrics from the llap daemons in every given milliseconds.
+ */
+public class LlapMetricsCollector implements
ServiceInstanceStateChangeListener {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(LlapMetricsCollector.class);
+ private static final String THREAD_NAME =
"LlapTaskSchedulerMetricsCollectorThread";
+ private static final long INITIAL_DELAY_MSEC = 6L;
+
+ private final Configuration conf;
+ private final ScheduledExecutorService scheduledMetricsExecutor;
+ private final LlapManagementProtocolClientImplFactory clientFactory;
+ private final Map llapClients;
+ private final Map instanceStatisticsMap;
+ private final long metricsCollectionMs;
+
+
+ public LlapMetricsCollector(Configuration conf) {
+this(
+conf,
+Executors.newSingleThreadScheduledExecutor(
+new
ThreadFactoryBuilder().setDaemon(true).setNameFormat(THREAD_NAME)
+.build()),
+LlapManagementProtocolClientImplFactory.basicInstance(conf));
+ }
+
+ @VisibleForTesting
+ LlapMetricsCollector(Configuration conf, ScheduledExecutorService
scheduledMetricsExecutor,
+ LlapManagementProtocolClientImplFactory clientFactory) {
+this.conf = conf;
+this.scheduledMetricsExecutor = scheduledMetricsExecutor;
+this.clientFactory = clientFactory;
+this.llapClients = new HashMap<>();
+this.instanceStatisticsMap = new ConcurrentHashMap<>();
+this.metricsCollectionMs = HiveConf.getTimeVar(conf,
+
HiveConf.ConfVars.LLAP_TASK_SCHEDULER_AM_COLLECT_DAEMON_METRICS_MS,
TimeUnit.MILLISECONDS);
+ }
+
+ public void start() {
+if (enabled()) {
+ scheduledMetricsExecutor.scheduleAtFixedRate(() -> {
+collectMetrics();
+ }, INITIAL_DELAY_MSEC, metricsCollectionMs, TimeUnit.MILLISECONDS);
+}
+ }
+
+ private boolean enabled() {
+return metricsCollectionMs > 0;
+ }
+
+ public void shutdown() {
+scheduledMetricsExecutor.shutdownNow();
+ }
+
+ @VisibleForTesting
+ void collectMetrics() {
+if (enabled()) {
Review comment:
Unnecessary check: The thread, calling collectMetrics is never created if ms
<= 0.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue
[jira] [Commented] (HIVE-21848) Table property name definition between ORC and Parquet encrytion
[ https://issues.apache.org/jira/browse/HIVE-21848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871480#comment-16871480 ] Xinli Shang commented on HIVE-21848: I am OK with this variant of #2 overall. The only thing(non-critical) is when we implement HIVE-21849 like CTAS to carry over column encryption properties, we need to search columns. In that case, the column explicitly as the key would be easier. But this is not a big problem as we have to parse the string anyway and it won't make much difference. One more question of our definition here(not specific to this variant proposal) is what if a key is specified for the parent column in the nested column scenario? Let's say column 'parent' has two children columns 'child1' and 'child2'. If a key is specified for 'parent', is it considered as valid? If yes, what does it mean? I prefer to consider it as INVALID, but like to hear your suggestions. > Table property name definition between ORC and Parquet encrytion > > > Key: HIVE-21848 > URL: https://issues.apache.org/jira/browse/HIVE-21848 > Project: Hive > Issue Type: Task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Xinli Shang >Assignee: Xinli Shang >Priority: Major > Fix For: 3.0.0 > > > The goal of this Jira is to define a superset of unified table property names > that can be used for both Parquet and ORC column encryption. There is no code > change needed for this Jira. > *Background:* > ORC-14 and Parquet-1178 introduced column encryption to ORC and Parquet. To > configure the encryption, e.g. which column is sensitive, what master key to > be used, algorithm, etc, table properties can be used. It is important that > both Parquet and ORC can use unified names. > According to the slide > [https://www.slideshare.net/oom65/fine-grain-access-control-for-big-data-orc-column-encryption-137308692], > ORC use table properties like orc.encrypt.pii, orc.encrypt.credit. While in > the Parquet community, it is still discussing to provide several ways and > using table properties is one of the options, while there is no detailed > design of the table property names yet. > So it is a good time to discuss within two communities to have unified table > names as a superset. > *Proposal:* > There are several encryption properties that need to be specified for a > table. Here is the list. This is the superset of Parquet and ORC. Some of > them might not apply to both. > # PII columns including nest columns > # Column key metadata, master key metadata > # Encryption algorithm, for example, Parquet support AES_GCM and AES_CTR. > ORC might support AES_CTR. > # Encryption footer - Parquet allow footer to be encrypted or plaintext > # Footer key metadata > Here is the table properties proposal. > |*Table Property Name*|*Value*|*Notes*| > |encrypt_algorithm|aes_ctr, aes_gcm|The algorithm to be used for encryption.| > |encrypt_footer_plaintext|true, false|Parquet support plaintext and encrypted > footer. By default, it is encrypted.| > |encrypt_footer_key_metadata|base64 string of footer key metadata|It is up to > the KMS to define what key metadata is. The metadata should have enough > information to figure out the corresponding key by the KMS. | > |encrypt_col_xxx|base64 string of column key metadata|‘xxx’ is the column > name for example, ‘address.zipcode’. > > It is up to the KMS to define what key metadata is. The metadata should have > enough information to figure out the corresponding key by the KMS.| > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Attachment: (was: HIVE-21914.03.patch) > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch, > HIVE-21914.03.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[ https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21914: -- Attachment: HIVE-21914.03.patch > Move Function and Macro related DDL operations into the DDL framework > - > > Key: HIVE-21914 > URL: https://issues.apache.org/jira/browse/HIVE-21914 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch, > HIVE-21914.03.patch > > > Some Function and Macro related operations are handled by FunctionTask, and > FunctionWork while they belong to the DDL framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21918) Handle each Alter Database types in a separate desc / operation
[ https://issues.apache.org/jira/browse/HIVE-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21918: -- Summary: Handle each Alter Database types in a separate desc / operation (was: handle each Alter Database types in a separate desc / operation) > Handle each Alter Database types in a separate desc / operation > --- > > Key: HIVE-21918 > URL: https://issues.apache.org/jira/browse/HIVE-21918 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21918.01.patch > > > AlterDatabaseDesc / AlterDatabaseOperation handles all kind of alter database > commands. Following the logic of the DDL handling, they should be in separate > classes, with a mini framework handled by abstract ancestors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21918) Handle each Alter Database types in a separate desc / operation
[ https://issues.apache.org/jira/browse/HIVE-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21918: -- Status: Patch Available (was: Open) > Handle each Alter Database types in a separate desc / operation > --- > > Key: HIVE-21918 > URL: https://issues.apache.org/jira/browse/HIVE-21918 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21918.01.patch > > > AlterDatabaseDesc / AlterDatabaseOperation handles all kind of alter database > commands. Following the logic of the DDL handling, they should be in separate > classes, with a mini framework handled by abstract ancestors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21918) Handle each Alter Database types in a separate desc / operation
[ https://issues.apache.org/jira/browse/HIVE-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21918: -- Attachment: HIVE-21918.01.patch > Handle each Alter Database types in a separate desc / operation > --- > > Key: HIVE-21918 > URL: https://issues.apache.org/jira/browse/HIVE-21918 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21918.01.patch > > > AlterDatabaseDesc / AlterDatabaseOperation handles all kind of alter database > commands. Following the logic of the DDL handling, they should be in separate > classes, with a mini framework handled by abstract ancestors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[ https://issues.apache.org/jira/browse/HIVE-21907?focusedWorklogId=265774&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265774 ] ASF GitHub Bot logged work on HIVE-21907: - Author: ASF GitHub Bot Created on: 24/Jun/19 15:56 Start Date: 24/Jun/19 15:56 Worklog Time Spent: 10m Work Description: odraese commented on pull request #685: HIVE-21907: Add a new LlapDaemon Management API method to set the daemon capacity URL: https://github.com/apache/hive/pull/685#discussion_r296790415 ## File path: llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java ## @@ -40,6 +40,7 @@ import org.apache.hadoop.hive.llap.daemon.QueryFailedHandler; import org.apache.hadoop.hive.llap.daemon.SchedulerFragmentCompletingListener; import org.apache.hadoop.hive.llap.daemon.impl.LlapTokenChecker.LlapTokenInfo; +import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos; Review comment: This seems to be an unused import. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 265774) Time Spent: 40m (was: 0.5h) > Add a new LlapDaemon Management API method to set the daemon capacity > - > > Key: HIVE-21907 > URL: https://issues.apache.org/jira/browse/HIVE-21907 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21907.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Add a new method to LlapManagementProtocol API which can disable an Llap node. > It would be even better, if we can dynamically set the number of executors > and the size of the wait queue. This way we can disable the node setting them > to 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21918) handle each Alter Database types in a separate desc / operation
[ https://issues.apache.org/jira/browse/HIVE-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-21918: - > handle each Alter Database types in a separate desc / operation > --- > > Key: HIVE-21918 > URL: https://issues.apache.org/jira/browse/HIVE-21918 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > AlterDatabaseDesc / AlterDatabaseOperation handles all kind of alter database > commands. Following the logic of the DDL handling, they should be in separate > classes, with a mini framework handled by abstract ancestors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[ https://issues.apache.org/jira/browse/HIVE-21907?focusedWorklogId=265772&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265772 ] ASF GitHub Bot logged work on HIVE-21907: - Author: ASF GitHub Bot Created on: 24/Jun/19 15:56 Start Date: 24/Jun/19 15:56 Worklog Time Spent: 10m Work Description: odraese commented on pull request #685: HIVE-21907: Add a new LlapDaemon Management API method to set the daemon capacity URL: https://github.com/apache/hive/pull/685#discussion_r296793593 ## File path: llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java ## @@ -176,6 +181,31 @@ public TaskExecutorService(int numExecutors, int waitQueueSize, Futures.addCallback(future, new WaitQueueWorkerCallback()); } + /** + * Sets the TaskExecutorService capacity to the new values. Both the number of executors and the + * queue size should be smaller than that original values, so we do not mess up with the other Review comment: What other settings? Maybe the comment could be more specific? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 265772) Time Spent: 20m (was: 10m) > Add a new LlapDaemon Management API method to set the daemon capacity > - > > Key: HIVE-21907 > URL: https://issues.apache.org/jira/browse/HIVE-21907 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21907.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Add a new method to LlapManagementProtocol API which can disable an Llap node. > It would be even better, if we can dynamically set the number of executors > and the size of the wait queue. This way we can disable the node setting them > to 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[
https://issues.apache.org/jira/browse/HIVE-21907?focusedWorklogId=265773&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265773
]
ASF GitHub Bot logged work on HIVE-21907:
-
Author: ASF GitHub Bot
Created on: 24/Jun/19 15:56
Start Date: 24/Jun/19 15:56
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #685: HIVE-21907: Add
a new LlapDaemon Management API method to set the daemon capacity
URL: https://github.com/apache/hive/pull/685#discussion_r296793765
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
##
@@ -176,6 +181,31 @@ public TaskExecutorService(int numExecutors, int
waitQueueSize,
Futures.addCallback(future, new WaitQueueWorkerCallback());
}
+ /**
+ * Sets the TaskExecutorService capacity to the new values. Both the number
of executors and the
+ * queue size should be smaller than that original values, so we do not mess
up with the other
+ * settings. Setting smaller capacity will not cancel or reject already
executing or queued tasks
+ * in itself.
+ * @param newNumExecutors The new number of executors
+ * @param newWaitQueueSize The new number of wait queue size
+ */
+ @Override
+ public synchronized void setCapacity(int newNumExecutors, int
newWaitQueueSize) {
+if (newNumExecutors > configuredMaxExecutors) {
Review comment:
Should we also verify that the new values are >= zero?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 265773)
Time Spent: 0.5h (was: 20m)
> Add a new LlapDaemon Management API method to set the daemon capacity
> -
>
> Key: HIVE-21907
> URL: https://issues.apache.org/jira/browse/HIVE-21907
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21907.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Add a new method to LlapManagementProtocol API which can disable an Llap node.
> It would be even better, if we can dynamically set the number of executors
> and the size of the wait queue. This way we can disable the node setting them
> to 0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[
https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871437#comment-16871437
]
Hive QA commented on HIVE-21914:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972707/HIVE-21914.03.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16339 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=350)
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence
(batchId=242)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17708/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17708/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17708/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972707 - PreCommit-HIVE-Build
> Move Function and Macro related DDL operations into the DDL framework
> -
>
> Key: HIVE-21914
> URL: https://issues.apache.org/jira/browse/HIVE-21914
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch,
> HIVE-21914.03.patch
>
>
> Some Function and Macro related operations are handled by FunctionTask, and
> FunctionWork while they belong to the DDL framework.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21848) Table property name definition between ORC and Parquet encrytion
[ https://issues.apache.org/jira/browse/HIVE-21848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871405#comment-16871405 ] Owen O'Malley commented on HIVE-21848: -- How about a variant of #2 that looks like: "encrypt.columns" = "pii:col1,col2;credit:col3" So: root = | ";" key-columns = ":" column-list = | "," > Table property name definition between ORC and Parquet encrytion > > > Key: HIVE-21848 > URL: https://issues.apache.org/jira/browse/HIVE-21848 > Project: Hive > Issue Type: Task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Xinli Shang >Assignee: Xinli Shang >Priority: Major > Fix For: 3.0.0 > > > The goal of this Jira is to define a superset of unified table property names > that can be used for both Parquet and ORC column encryption. There is no code > change needed for this Jira. > *Background:* > ORC-14 and Parquet-1178 introduced column encryption to ORC and Parquet. To > configure the encryption, e.g. which column is sensitive, what master key to > be used, algorithm, etc, table properties can be used. It is important that > both Parquet and ORC can use unified names. > According to the slide > [https://www.slideshare.net/oom65/fine-grain-access-control-for-big-data-orc-column-encryption-137308692], > ORC use table properties like orc.encrypt.pii, orc.encrypt.credit. While in > the Parquet community, it is still discussing to provide several ways and > using table properties is one of the options, while there is no detailed > design of the table property names yet. > So it is a good time to discuss within two communities to have unified table > names as a superset. > *Proposal:* > There are several encryption properties that need to be specified for a > table. Here is the list. This is the superset of Parquet and ORC. Some of > them might not apply to both. > # PII columns including nest columns > # Column key metadata, master key metadata > # Encryption algorithm, for example, Parquet support AES_GCM and AES_CTR. > ORC might support AES_CTR. > # Encryption footer - Parquet allow footer to be encrypted or plaintext > # Footer key metadata > Here is the table properties proposal. > |*Table Property Name*|*Value*|*Notes*| > |encrypt_algorithm|aes_ctr, aes_gcm|The algorithm to be used for encryption.| > |encrypt_footer_plaintext|true, false|Parquet support plaintext and encrypted > footer. By default, it is encrypted.| > |encrypt_footer_key_metadata|base64 string of footer key metadata|It is up to > the KMS to define what key metadata is. The metadata should have enough > information to figure out the corresponding key by the KMS. | > |encrypt_col_xxx|base64 string of column key metadata|‘xxx’ is the column > name for example, ‘address.zipcode’. > > It is up to the KMS to define what key metadata is. The metadata should have > enough information to figure out the corresponding key by the KMS.| > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21391) LLAP: Pool of column vector buffers can cause memory pressure
[ https://issues.apache.org/jira/browse/HIVE-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-21391: -- Attachment: HIVE-21391.2.patch > LLAP: Pool of column vector buffers can cause memory pressure > - > > Key: HIVE-21391 > URL: https://issues.apache.org/jira/browse/HIVE-21391 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: slim bouguerra >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21391.1.patch, HIVE-21391.2.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Where there are too many columns (in the order of 100s), with decimal, string > types the column vector pool of buffers created here > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java#L59] > can cause memory pressure. > Example: > 128 (poolSize) * 300 (numCols) * 1024 (batchSize) * 80 (decimalSize) ~= 3GB > The pool size keeps increasing when there is slow consumer but fast llap io > (SSDs) leading to GC pressure when all LLAP io threads read splits from same > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21914) Move Function and Macro related DDL operations into the DDL framework
[
https://issues.apache.org/jira/browse/HIVE-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871394#comment-16871394
]
Hive QA commented on HIVE-21914:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} llap-server in master has 82 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 1 new + 326 unchanged - 17
fixed = 327 total (was 343) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
13s{color} | {color:red} ql generated 1 new + 2253 unchanged - 1 fixed = 2254
total (was 2254) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA238 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
48391-48404] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17708/dev-support/hive-personality.sh
|
| git revision | master / 11f7856 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17708/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17708/yetus/new-findbugs-ql.html
|
| modules | C: ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17708/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move Function and Macro related DDL operations into the DDL framework
> -
>
> Key: HIVE-21914
> URL: https://issues.apache.org/jira/browse/HIVE-21914
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21914.01.patch, HIVE-21914.02.patch,
> HIVE-21914.03.patch
>
>
> Some Function and Macro related operations are handled by FunctionTask, and
> Fu
[jira] [Updated] (HIVE-21917) COMPLETED_TXN_COMPONENTS table is never cleaned up unless Compactor runs
[ https://issues.apache.org/jira/browse/HIVE-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Condit updated HIVE-21917: Summary: COMPLETED_TXN_COMPONENTS table is never cleaned up unless Compactor runs (was: COMPLETED_TXN_COMPONENTS table is never cleaned up unless Compator runs) > COMPLETED_TXN_COMPONENTS table is never cleaned up unless Compactor runs > > > Key: HIVE-21917 > URL: https://issues.apache.org/jira/browse/HIVE-21917 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0, 3.1.1 >Reporter: Craig Condit >Priority: Major > > The Initiator thread in the metastore repeatedly loops over entries in the > COMPLETED_TXN_COMPONENTS table to determine which partitions / tables might > need to be compacted. However, entries are never removed from this table > except by a completed Compactor run. > In a cluster where most tables / partitions are write-once read-many, this > results in stale entries in this table never being cleaned up. In a small > test cluster, we have observed approximately 45k entries in this table > (virtually equal to the number of partitions in the cluster) while < 100 of > these tables have delta files at all. Since most of the tables will never get > enough writes to trigger a compaction (and in fact have only ever been > written to once), the initiator thread keeps trying to evaluate them on every > loop. > On this test cluster, it takes approximately 10 minutes to loop through all > the entries and results in severe performance degradation on metastore > operations. With the default run timing of 5 minutes, the initiator basically > never stops running. > On a production cluster with 2M partitions, this would be a non-starter. > The initiator thread should proactively remove entries from > COMPLETED_TXN_COMPONENTS when it determines that a compaction is not needed, > so that they are not evaluated again on the next loop. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21846) Create a thread in TezAM which periodically fetches LlapDaemon metrics
[
https://issues.apache.org/jira/browse/HIVE-21846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871235#comment-16871235
]
Hive QA commented on HIVE-21846:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972702/HIVE-21846.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16346 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17707/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17707/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17707/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972702 - PreCommit-HIVE-Build
> Create a thread in TezAM which periodically fetches LlapDaemon metrics
> --
>
> Key: HIVE-21846
> URL: https://issues.apache.org/jira/browse/HIVE-21846
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21846.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> LlapTaskSchedulerService should start a thread which periodically fetches the
> LlapDaemon metrics and stores them in the NodeInfo object.
> This should be just the first implementation - later we should find a way
> where we do not need NxM requests between N TezAM and M LlapDaemon
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18735) Create table like loses transactional attribute
[
https://issues.apache.org/jira/browse/HIVE-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-18735:
-
Status: Patch Available (was: Open)
> Create table like loses transactional attribute
> ---
>
> Key: HIVE-18735
> URL: https://issues.apache.org/jira/browse/HIVE-18735
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Eugene Koifman
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-18735.01.patch
>
>
> {noformat}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('transactional'='true')";
> create table T like T1;
> show create table T ;
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t'
> TBLPROPERTIES (
> 'transient_lastDdlTime'='1518813564')
> {noformat}
> Specifying props explicitly does work
> {noformat}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('transactional'='true')";
> create table T like T1 TBLPROPERTIES ('transactional'='true');
> show create table T ;
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518814098564/warehouse/t'
> TBLPROPERTIES (
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1518814111')
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21874) Implement add partitions related methods on temporary table
[
https://issues.apache.org/jira/browse/HIVE-21874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-21874:
-
Attachment: HIVE-21874.02.patch
> Implement add partitions related methods on temporary table
> ---
>
> Key: HIVE-21874
> URL: https://issues.apache.org/jira/browse/HIVE-21874
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21874.01.patch, HIVE-21874.02.patch
>
>
> IMetaStoreClient exposes the following add partition related methods:
> {code:java}
> Partition add_partition(Partition partition);
> int add_partitions(List partitions);
> int add_partitions_pspec(PartitionSpecProxy partitionSpec);
> List add_partitions(List partitions, boolean
> ifNotExists, boolean needResults);
> {code}
> These methods should be implemented in order to handle addition of partitions
> to temporary tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[ https://issues.apache.org/jira/browse/HIVE-21907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-21907: -- Attachment: HIVE-21907.patch > Add a new LlapDaemon Management API method to set the daemon capacity > - > > Key: HIVE-21907 > URL: https://issues.apache.org/jira/browse/HIVE-21907 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21907.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Add a new method to LlapManagementProtocol API which can disable an Llap node. > It would be even better, if we can dynamically set the number of executors > and the size of the wait queue. This way we can disable the node setting them > to 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[ https://issues.apache.org/jira/browse/HIVE-21907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-21907: -- Attachment: (was: HIVE-21907.patch) > Add a new LlapDaemon Management API method to set the daemon capacity > - > > Key: HIVE-21907 > URL: https://issues.apache.org/jira/browse/HIVE-21907 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21907.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Add a new method to LlapManagementProtocol API which can disable an Llap node. > It would be even better, if we can dynamically set the number of executors > and the size of the wait queue. This way we can disable the node setting them > to 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
