[jira] [Commented] (HIVE-21722) REPL:: logs are missing in hiveStatement.getQueryLog output during parallel execution mode.
[
https://issues.apache.org/jira/browse/HIVE-21722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839168#comment-16839168
]
Hive QA commented on HIVE-21722:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968632/HIVE-21722.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16008 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17203/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17203/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17203/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12968632 - PreCommit-HIVE-Build
> REPL:: logs are missing in hiveStatement.getQueryLog output during parallel
> execution mode.
> ---
>
> Key: HIVE-21722
> URL: https://issues.apache.org/jira/browse/HIVE-21722
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21722.01.patch, HIVE-21722.02.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> getQueryLog only reads logs from Background thread scope. If parallel
> execution is set to true, a new thread is created for execution and all the
> logs added by the new thread are not added to the parent Background thread
> scope. In replication scope, replStateLogTasks are started in parallel mode
> causing the logs to be skipped from getQueryLog scope.
> There is one more issue, with the conf is not passed while creating
> replStateLogTask during bootstrap load end. The same issue is there with
> event load during incremental load. The incremental load end log task is
> created with the proper config.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21686) Brute Force eviction can lead to a random uncontrolled eviction pattern.
[
https://issues.apache.org/jira/browse/HIVE-21686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839180#comment-16839180
]
Hive QA commented on HIVE-21686:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} llap-server in master has 81 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
23s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
23s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 23s{color}
| {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} llap-server: The patch generated 12 new + 85 unchanged
- 15 fixed = 97 total (was 100) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17204/dev-support/hive-personality.sh
|
| git revision | master / af97e84 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17204/yetus/patch-mvninstall-llap-server.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17204/yetus/patch-compile-llap-server.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17204/yetus/patch-compile-llap-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17204/yetus/diff-checkstyle-llap-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17204/yetus/patch-findbugs-llap-server.txt
|
| modules | C: common llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17204/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Brute Force eviction can lead to a random uncontrolled eviction pattern.
>
>
> Key: HIVE-21686
> URL: https://issues.apache.org/jira/browse/HIVE-21686
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: Cache_hitrate_improvement.csv, HIVE-21686.2.patch,
> HIVE-21686.3.patch, HIVE-21686.4.patch, HIVE-216
[jira] [Commented] (HIVE-21686) Brute Force eviction can lead to a random uncontrolled eviction pattern.
[
https://issues.apache.org/jira/browse/HIVE-21686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839214#comment-16839214
]
Hive QA commented on HIVE-21686:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968634/HIVE-21686.4.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16011 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testParallelInsertAnalyzeStats
(batchId=321)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17204/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17204/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17204/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12968634 - PreCommit-HIVE-Build
> Brute Force eviction can lead to a random uncontrolled eviction pattern.
>
>
> Key: HIVE-21686
> URL: https://issues.apache.org/jira/browse/HIVE-21686
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: Cache_hitrate_improvement.csv, HIVE-21686.2.patch,
> HIVE-21686.3.patch, HIVE-21686.4.patch, HIVE-21686.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Current logic used by brute force eviction can lead to a perpetual random
> eviction pattern.
> For instance if the cache build a small pocket of free memory where the total
> size is greater than incoming allocation request, the allocator will randomly
> evict block that fits a particular size.
> This can happen over and over therefore all the eviction will be random.
> In Addition this random eviction will lead a leak in the linked list
> maintained by the policy since it does not know anymore about what is evicted
> and what not.
> The improvement of this patch is very substantial to TPC-DS benchmark. I
> have tested it with 10TB scale 9 llap nodes and 32GB cache size per node.
> The patch has showed very noticeable difference in the Hit rate for raw
> number [^Cache_hitrate_improvement.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21714) Insert overwrite on an acid/mm table is ineffective if the input is empty
[ https://issues.apache.org/jira/browse/HIVE-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-21714: --- Attachment: HIVE-21714.3.patch > Insert overwrite on an acid/mm table is ineffective if the input is empty > - > > Key: HIVE-21714 > URL: https://issues.apache.org/jira/browse/HIVE-21714 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21714.1.patch, HIVE-21714.1.patch, > HIVE-21714.2.patch, HIVE-21714.3.patch > > > The issue of HIVE-18702 is present for ACID tables as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21714) Insert overwrite on an acid/mm table is ineffective if the input is empty
[
https://issues.apache.org/jira/browse/HIVE-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839312#comment-16839312
]
Hive QA commented on HIVE-21714:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
37s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17205/dev-support/hive-personality.sh
|
| git revision | master / de69a0f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests/hcatalog-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17205/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Insert overwrite on an acid/mm table is ineffective if the input is empty
> -
>
> Key: HIVE-21714
> URL: https://issues.apache.org/jira/browse/HIVE-21714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Major
> Attachments: HIVE-21714.1.patch, HIVE-21714.1.patch,
> HIVE-21714.2.patch, HIVE-21714.3.patch
>
>
> The issue of HIVE-18702 is present for ACID tables as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21656) Vectorize UDF mask
[ https://issues.apache.org/jira/browse/HIVE-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839324#comment-16839324 ] Matt McCline commented on HIVE-21656: - Great caution is needed here. Not clear setting the output batch’s noNulls field to true is safe. > Vectorize UDF mask > -- > > Key: HIVE-21656 > URL: https://issues.apache.org/jira/browse/HIVE-21656 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21656.1.patch, HIVE-21656.2.patch, > HIVE-21656.3.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21714) Insert overwrite on an acid/mm table is ineffective if the input is empty
[
https://issues.apache.org/jira/browse/HIVE-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839334#comment-16839334
]
Hive QA commented on HIVE-21714:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968657/HIVE-21714.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16008 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_overwrite]
(batchId=165)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17205/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17205/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17205/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12968657 - PreCommit-HIVE-Build
> Insert overwrite on an acid/mm table is ineffective if the input is empty
> -
>
> Key: HIVE-21714
> URL: https://issues.apache.org/jira/browse/HIVE-21714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Major
> Attachments: HIVE-21714.1.patch, HIVE-21714.1.patch,
> HIVE-21714.2.patch, HIVE-21714.3.patch
>
>
> The issue of HIVE-18702 is present for ACID tables as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-12971) Hive Support for Kudu
[ https://issues.apache.org/jira/browse/HIVE-12971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839346#comment-16839346 ] Jean-Marc Spaggiari commented on HIVE-12971: Is there any activity here? Or this is stalled? :-/ > Hive Support for Kudu > - > > Key: HIVE-12971 > URL: https://issues.apache.org/jira/browse/HIVE-12971 > Project: Hive > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Lenni Kuff >Assignee: Clemens Valiente >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > JIRA for tracking work related to Hive/Kudu integration. > It would be useful to allow Kudu data to be accessible via Hive. This would > involve creating a Kudu SerDe/StorageHandler and implementing support for > QUERY and DML commands like SELECT, INSERT, UPDATE, and DELETE. Kudu > Input/OutputFormats classes already exist. The work can be staged to support > this functionality incrementally. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21714) Insert overwrite on an acid/mm table is ineffective if the input is empty
[ https://issues.apache.org/jira/browse/HIVE-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-21714: --- Attachment: HIVE-21714.4.patch > Insert overwrite on an acid/mm table is ineffective if the input is empty > - > > Key: HIVE-21714 > URL: https://issues.apache.org/jira/browse/HIVE-21714 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21714.1.patch, HIVE-21714.1.patch, > HIVE-21714.2.patch, HIVE-21714.3.patch, HIVE-21714.4.patch > > > The issue of HIVE-18702 is present for ACID tables as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21722) REPL:: logs are missing in hiveStatement.getQueryLog output during parallel execution mode.
[ https://issues.apache.org/jira/browse/HIVE-21722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21722: --- Resolution: Fixed Status: Resolved (was: Patch Available) [^HIVE-21722.02.patch] committed to master. Thanks [~sankarh] for review. > REPL:: logs are missing in hiveStatement.getQueryLog output during parallel > execution mode. > --- > > Key: HIVE-21722 > URL: https://issues.apache.org/jira/browse/HIVE-21722 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21722.01.patch, HIVE-21722.02.patch > > Time Spent: 2h 40m > Remaining Estimate: 0h > > getQueryLog only reads logs from Background thread scope. If parallel > execution is set to true, a new thread is created for execution and all the > logs added by the new thread are not added to the parent Background thread > scope. In replication scope, replStateLogTasks are started in parallel mode > causing the logs to be skipped from getQueryLog scope. > There is one more issue, with the conf is not passed while creating > replStateLogTask during bootstrap load end. The same issue is there with > event load during incremental load. The incremental load end log task is > created with the proper config. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21717) Rename is failing for directory in move task
[ https://issues.apache.org/jira/browse/HIVE-21717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21717: --- Resolution: Fixed Status: Resolved (was: Patch Available) [^HIVE-21717.01.patch] committed to master. Thanks [~sankarh] for review. > Rename is failing for directory in move task > - > > Key: HIVE-21717 > URL: https://issues.apache.org/jira/browse/HIVE-21717 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21717.01.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Rename fails with destination directory not empty in case a directory is move > directly to the table location from staging directory as rename cannot > overwrite non empty destination directory. > > In replication scenarios, if user does some concurrent write during bootstrap > dump, it may happen that some data which are already replicated through > bootstrap, will be tried during next incremental load also. This is handled > by making the operations reentrant during repl load. But here move task is > not able to delete the directory created by bootstrap load even though > replace flag is set to true. This is causing the incremental load to fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21714) Insert overwrite on an acid/mm table is ineffective if the input is empty
[
https://issues.apache.org/jira/browse/HIVE-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839431#comment-16839431
]
Hive QA commented on HIVE-21714:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
30s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17206/dev-support/hive-personality.sh
|
| git revision | master / de69a0f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests/hcatalog-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17206/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Insert overwrite on an acid/mm table is ineffective if the input is empty
> -
>
> Key: HIVE-21714
> URL: https://issues.apache.org/jira/browse/HIVE-21714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Major
> Attachments: HIVE-21714.1.patch, HIVE-21714.1.patch,
> HIVE-21714.2.patch, HIVE-21714.3.patch, HIVE-21714.4.patch
>
>
> The issue of HIVE-18702 is present for ACID tables as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21686) Brute Force eviction can lead to a random uncontrolled eviction pattern.
[ https://issues.apache.org/jira/browse/HIVE-21686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-21686: -- Attachment: HIVE-21686.5.patch > Brute Force eviction can lead to a random uncontrolled eviction pattern. > > > Key: HIVE-21686 > URL: https://issues.apache.org/jira/browse/HIVE-21686 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Labels: pull-request-available > Attachments: Cache_hitrate_improvement.csv, HIVE-21686.2.patch, > HIVE-21686.3.patch, HIVE-21686.4.patch, HIVE-21686.5.patch, HIVE-21686.patch > > Time Spent: 3.5h > Remaining Estimate: 0h > > Current logic used by brute force eviction can lead to a perpetual random > eviction pattern. > For instance if the cache build a small pocket of free memory where the total > size is greater than incoming allocation request, the allocator will randomly > evict block that fits a particular size. > This can happen over and over therefore all the eviction will be random. > In Addition this random eviction will lead a leak in the linked list > maintained by the policy since it does not know anymore about what is evicted > and what not. > The improvement of this patch is very substantial to TPC-DS benchmark. I > have tested it with 10TB scale 9 llap nodes and 32GB cache size per node. > The patch has showed very noticeable difference in the Hit rate for raw > number [^Cache_hitrate_improvement.csv] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21686) Brute Force eviction can lead to a random uncontrolled eviction pattern.
[
https://issues.apache.org/jira/browse/HIVE-21686?focusedWorklogId=241731&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241731
]
ASF GitHub Bot logged work on HIVE-21686:
-
Author: ASF GitHub Bot
Created on: 14/May/19 14:14
Start Date: 14/May/19 14:14
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #626: [HIVE-21686]
ensure that memory allocator does not evict using brute …
URL: https://github.com/apache/hive/pull/626#discussion_r283820576
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java
##
@@ -294,25 +385,18 @@ public void allocateMultiple(MemoryBuffer[] dest, int
size, BufferObjectFactory
// allocator thread (or threads per arena).
// The 2nd one is probably much simpler and will allow us to get rid of a
lot of sync code.
// But for now we will just retry. We will evict more each time.
-int attempt = 0;
+int discardsAttempt = 0;
boolean isFailed = false;
int memoryForceReleased = 0;
try {
int discardFailed = 0;
while (true) {
// Try to split bigger blocks.
-int startArenaIx = (int)((threadId + attempt) % arenaCount);
+startArenaIx = (int)((threadId + discardsAttempt) % arenaCount);
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 241731)
Time Spent: 3h 40m (was: 3.5h)
> Brute Force eviction can lead to a random uncontrolled eviction pattern.
>
>
> Key: HIVE-21686
> URL: https://issues.apache.org/jira/browse/HIVE-21686
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: Cache_hitrate_improvement.csv, HIVE-21686.2.patch,
> HIVE-21686.3.patch, HIVE-21686.4.patch, HIVE-21686.5.patch, HIVE-21686.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Current logic used by brute force eviction can lead to a perpetual random
> eviction pattern.
> For instance if the cache build a small pocket of free memory where the total
> size is greater than incoming allocation request, the allocator will randomly
> evict block that fits a particular size.
> This can happen over and over therefore all the eviction will be random.
> In Addition this random eviction will lead a leak in the linked list
> maintained by the policy since it does not know anymore about what is evicted
> and what not.
> The improvement of this patch is very substantial to TPC-DS benchmark. I
> have tested it with 10TB scale 9 llap nodes and 32GB cache size per node.
> The patch has showed very noticeable difference in the Hit rate for raw
> number [^Cache_hitrate_improvement.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21686) Brute Force eviction can lead to a random uncontrolled eviction pattern.
[ https://issues.apache.org/jira/browse/HIVE-21686?focusedWorklogId=241734&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241734 ] ASF GitHub Bot logged work on HIVE-21686: - Author: ASF GitHub Bot Created on: 14/May/19 14:19 Start Date: 14/May/19 14:19 Worklog Time Spent: 10m Work Description: b-slim commented on issue #626: [HIVE-21686] ensure that memory allocator does not evict using brute … URL: https://github.com/apache/hive/pull/626#issuecomment-492257694 @odraese Thanks for the review, i did address most of you comment can you please take a second look when you have time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 241734) Time Spent: 3h 50m (was: 3h 40m) > Brute Force eviction can lead to a random uncontrolled eviction pattern. > > > Key: HIVE-21686 > URL: https://issues.apache.org/jira/browse/HIVE-21686 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Labels: pull-request-available > Attachments: Cache_hitrate_improvement.csv, HIVE-21686.2.patch, > HIVE-21686.3.patch, HIVE-21686.4.patch, HIVE-21686.5.patch, HIVE-21686.patch > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Current logic used by brute force eviction can lead to a perpetual random > eviction pattern. > For instance if the cache build a small pocket of free memory where the total > size is greater than incoming allocation request, the allocator will randomly > evict block that fits a particular size. > This can happen over and over therefore all the eviction will be random. > In Addition this random eviction will lead a leak in the linked list > maintained by the policy since it does not know anymore about what is evicted > and what not. > The improvement of this patch is very substantial to TPC-DS benchmark. I > have tested it with 10TB scale 9 llap nodes and 32GB cache size per node. > The patch has showed very noticeable difference in the Hit rate for raw > number [^Cache_hitrate_improvement.csv] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21686) Brute Force eviction can lead to a random uncontrolled eviction pattern.
[ https://issues.apache.org/jira/browse/HIVE-21686?focusedWorklogId=241735&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241735 ] ASF GitHub Bot logged work on HIVE-21686: - Author: ASF GitHub Bot Created on: 14/May/19 14:19 Start Date: 14/May/19 14:19 Worklog Time Spent: 10m Work Description: b-slim commented on issue #626: [HIVE-21686] ensure that memory allocator does not evict using brute … URL: https://github.com/apache/hive/pull/626#issuecomment-492257694 @odraese Thanks for the review, i did fix most of your comments can you please take a second look when you have time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 241735) Time Spent: 4h (was: 3h 50m) > Brute Force eviction can lead to a random uncontrolled eviction pattern. > > > Key: HIVE-21686 > URL: https://issues.apache.org/jira/browse/HIVE-21686 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Labels: pull-request-available > Attachments: Cache_hitrate_improvement.csv, HIVE-21686.2.patch, > HIVE-21686.3.patch, HIVE-21686.4.patch, HIVE-21686.5.patch, HIVE-21686.patch > > Time Spent: 4h > Remaining Estimate: 0h > > Current logic used by brute force eviction can lead to a perpetual random > eviction pattern. > For instance if the cache build a small pocket of free memory where the total > size is greater than incoming allocation request, the allocator will randomly > evict block that fits a particular size. > This can happen over and over therefore all the eviction will be random. > In Addition this random eviction will lead a leak in the linked list > maintained by the policy since it does not know anymore about what is evicted > and what not. > The improvement of this patch is very substantial to TPC-DS benchmark. I > have tested it with 10TB scale 9 llap nodes and 32GB cache size per node. > The patch has showed very noticeable difference in the Hit rate for raw > number [^Cache_hitrate_improvement.csv] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21714) Insert overwrite on an acid/mm table is ineffective if the input is empty
[
https://issues.apache.org/jira/browse/HIVE-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839461#comment-16839461
]
Hive QA commented on HIVE-21714:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968679/HIVE-21714.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16008 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
(batchId=248)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17206/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17206/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17206/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12968679 - PreCommit-HIVE-Build
> Insert overwrite on an acid/mm table is ineffective if the input is empty
> -
>
> Key: HIVE-21714
> URL: https://issues.apache.org/jira/browse/HIVE-21714
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Major
> Attachments: HIVE-21714.1.patch, HIVE-21714.1.patch,
> HIVE-21714.2.patch, HIVE-21714.3.patch, HIVE-21714.4.patch
>
>
> The issue of HIVE-18702 is present for ACID tables as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21724) Nested ARRAY and STRUCT inside MAP don't work with LazySimpleDeserializeRead
[
https://issues.apache.org/jira/browse/HIVE-21724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839482#comment-16839482
]
Daniel Voros commented on HIVE-21724:
-
Thanks [~vgarg] for pointing this out, I missed this before. Based on
[https://cwiki.apache.org/confluence/display/Hive/Vectorized+Query+Execution]
it seems it does not, but we should fall back to "row-at-a-time execution". I
guess this is what's happening here.
> Nested ARRAY and STRUCT inside MAP don't work with LazySimpleDeserializeRead
>
>
> Key: HIVE-21724
> URL: https://issues.apache.org/jira/browse/HIVE-21724
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-21724.1.patch
>
>
> The logic during vectorized execution that keeps track of how deep we are in
> the nested structure doesn't work for ARRAYs and STRUCTs embedded inside maps.
> Repro steps (with hive.vectorized.execution.enabled=true):
> {code}
> CREATE TABLE srctable(a map>) STORED AS TEXTFILE;
> create table desttable(c1 map>);
> insert into srctable values (map(1, array(1, 2, 3)));
> insert into desttable select a from srctable;
> select * from desttable;
> {code}
> Will produce:
> {code}
> {1:[null]}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21686) Brute Force eviction can lead to a random uncontrolled eviction pattern.
[
https://issues.apache.org/jira/browse/HIVE-21686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839528#comment-16839528
]
Hive QA commented on HIVE-21686:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
47s{color} | {color:blue} llap-server in master has 81 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
21s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
24s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 24s{color}
| {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} llap-server: The patch generated 5 new + 83 unchanged
- 17 fixed = 88 total (was 100) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17207/dev-support/hive-personality.sh
|
| git revision | master / de69a0f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17207/yetus/patch-mvninstall-llap-server.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17207/yetus/patch-compile-llap-server.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17207/yetus/patch-compile-llap-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17207/yetus/diff-checkstyle-llap-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17207/yetus/patch-findbugs-llap-server.txt
|
| modules | C: common llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17207/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Brute Force eviction can lead to a random uncontrolled eviction pattern.
>
>
> Key: HIVE-21686
> URL: https://issues.apache.org/jira/browse/HIVE-21686
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: Cache_hitrate_improvement.csv, HIVE-21686.2.patch,
> HIVE-21686.3.patch, HIVE-21686.4.patch, HIVE-2168
[jira] [Commented] (HIVE-21727) Allow For Ordinal Substitution
[
https://issues.apache.org/jira/browse/HIVE-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839530#comment-16839530
]
David Mollitor commented on HIVE-21727:
---
{code:sql}
SELECT int_col / 2, sum(x)
FROM t
GROUP BY 1;
{code}
> Allow For Ordinal Substitution
> ---
>
> Key: HIVE-21727
> URL: https://issues.apache.org/jira/browse/HIVE-21727
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Priority: Major
>
> Impala allows for ordinal substitution. Add a compatible feature to Hive to
> allow Hive to be more compatible with Impala. Allows for more of a drop-in
> replacement.
> [IMPALA-8548]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21724) Nested ARRAY and STRUCT inside MAP don't work with LazySimpleDeserializeRead
[
https://issues.apache.org/jira/browse/HIVE-21724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Voros updated HIVE-21724:
Attachment: HIVE-21724.2.patch
> Nested ARRAY and STRUCT inside MAP don't work with LazySimpleDeserializeRead
>
>
> Key: HIVE-21724
> URL: https://issues.apache.org/jira/browse/HIVE-21724
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-21724.1.patch, HIVE-21724.2.patch
>
>
> The logic during vectorized execution that keeps track of how deep we are in
> the nested structure doesn't work for ARRAYs and STRUCTs embedded inside maps.
> Repro steps (with hive.vectorized.execution.enabled=true):
> {code}
> CREATE TABLE srctable(a map>) STORED AS TEXTFILE;
> create table desttable(c1 map>);
> insert into srctable values (map(1, array(1, 2, 3)));
> insert into desttable select a from srctable;
> select * from desttable;
> {code}
> Will produce:
> {code}
> {1:[null]}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21724) Nested ARRAY and STRUCT inside MAP don't work with LazySimpleDeserializeRead
[
https://issues.apache.org/jira/browse/HIVE-21724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839536#comment-16839536
]
Daniel Voros commented on HIVE-21724:
-
Attached patch #2 that adds the LLAP q.out file that I've missed in the first
patch.
> Nested ARRAY and STRUCT inside MAP don't work with LazySimpleDeserializeRead
>
>
> Key: HIVE-21724
> URL: https://issues.apache.org/jira/browse/HIVE-21724
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-21724.1.patch, HIVE-21724.2.patch
>
>
> The logic during vectorized execution that keeps track of how deep we are in
> the nested structure doesn't work for ARRAYs and STRUCTs embedded inside maps.
> Repro steps (with hive.vectorized.execution.enabled=true):
> {code}
> CREATE TABLE srctable(a map>) STORED AS TEXTFILE;
> create table desttable(c1 map>);
> insert into srctable values (map(1, array(1, 2, 3)));
> insert into desttable select a from srctable;
> select * from desttable;
> {code}
> Will produce:
> {code}
> {1:[null]}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21718) Improvement performance of UpdateInputAccessTimeHook
[ https://issues.apache.org/jira/browse/HIVE-21718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21718: - Attachment: HIVE-21718.patch > Improvement performance of UpdateInputAccessTimeHook > > > Key: HIVE-21718 > URL: https://issues.apache.org/jira/browse/HIVE-21718 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 2.1.1 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21718.patch > > > Currently, Hive does not update the lastAccessTime property for any entities > when a query accesses them. Thus it has not possible to know when a table was > last accessed. > Hive does provide a configurable hook to HS2 that is execcuted as a pre-query > hook prior to the query being executed. However, this hook is inefficient > because for each table or partition it is attempting to update time for, it > executes an "alter table ... " command internally. This is bad > 1) For a query touching 1000's of partitions, this hook takes forever to > update them. > 2) Meanwhile, it is holding up the original query from executing. > So even though we do not recommend using the hook, because the reward is too > little (having lastAccessTime updated), we realize there is no other means to > achieve this. > Also, we can improve the performance of the hook significantly by adding a > new thrift API on HMS to update the lastAccessTime on the database rows > directly instead of going to HMS front end for 1 entity at time (leading to > 1000's of HMS calls that lead to multiple 1000's of calls to the database). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21718) Improvement performance of UpdateInputAccessTimeHook
[ https://issues.apache.org/jira/browse/HIVE-21718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21718: - Status: Patch Available (was: Open) This patch adds a new thrift API for HMS that uses the "update table ... " sql query to update lastAccessTime for a batch of rows within a single transaction. This API is then being used by the pre-hook that currently updates one row at a time by using the "alter table" API for HMS. > Improvement performance of UpdateInputAccessTimeHook > > > Key: HIVE-21718 > URL: https://issues.apache.org/jira/browse/HIVE-21718 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 2.1.1 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21718.patch > > > Currently, Hive does not update the lastAccessTime property for any entities > when a query accesses them. Thus it has not possible to know when a table was > last accessed. > Hive does provide a configurable hook to HS2 that is execcuted as a pre-query > hook prior to the query being executed. However, this hook is inefficient > because for each table or partition it is attempting to update time for, it > executes an "alter table ... " command internally. This is bad > 1) For a query touching 1000's of partitions, this hook takes forever to > update them. > 2) Meanwhile, it is holding up the original query from executing. > So even though we do not recommend using the hook, because the reward is too > little (having lastAccessTime updated), we realize there is no other means to > achieve this. > Also, we can improve the performance of the hook significantly by adding a > new thrift API on HMS to update the lastAccessTime on the database rows > directly instead of going to HMS front end for 1 entity at time (leading to > 1000's of HMS calls that lead to multiple 1000's of calls to the database). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21727) Allow For Ordinal Substitution
[ https://issues.apache.org/jira/browse/HIVE-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839553#comment-16839553 ] Gopal V commented on HIVE-21727: hive.groupby.position.alias ? > Allow For Ordinal Substitution > --- > > Key: HIVE-21727 > URL: https://issues.apache.org/jira/browse/HIVE-21727 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Affects Versions: 4.0.0, 3.2.0 >Reporter: David Mollitor >Priority: Major > > Impala allows for ordinal substitution. Add a compatible feature to Hive to > allow Hive to be more compatible with Impala. Allows for more of a drop-in > replacement. > [IMPALA-8548] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21709) Count with expression does not work in Parquet
[
https://issues.apache.org/jira/browse/HIVE-21709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839578#comment-16839578
]
David Lavati commented on HIVE-21709:
-
Hi [~mainakghosh], thank you for your investigation, if you're interested in
contributing, feel free to assign this to yourself - you can find more info on
the process here:
[https://cwiki.apache.org/confluence/display/Hive/HowToContribute]
As for the review, either the reviewboard or a github PR @
[https://github.com/apache/hive] is welcome.
Cheers,
David
> Count with expression does not work in Parquet
> --
>
> Key: HIVE-21709
> URL: https://issues.apache.org/jira/browse/HIVE-21709
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.2
>Reporter: Mainak Ghosh
>Priority: Major
>
> For parquet file with nested schema, count with expression as column name
> does not work when you are filtering on another column in the same struct.
> Here are the steps to reproduce:
> {code:java}
> CREATE TABLE `test_table`( `rtb_win` struct<`impression_id`:string,
> `pub_id`:string>) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS
> INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
> OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat';
> INSERT INTO TABLE test_table SELECT named_struct('impression_id', 'cat',
> 'pub_id', '2');
> select count(rtb_win.impression_id) from test_table where rtb_win.pub_id ='2';
> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. spark,
> tez) or using Hive 1.X releases.
> +--+
> | _c0 |
> +--+
> | 0 |
> +--+
> select count(*) from test_parquet_count_mghosh where rtb_win.pub_id ='2';
> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. spark,
> tez) or using Hive 1.X releases.
> +--+
> | _c0 |
> +--+
> | 1 |
> +--+{code}
> As you can see the first query returns the wrong result while the second one
> returns the correct result.
> The issue is an column order mismatch between the actual parquet file
> (impression_id first and pub_id second) and the Hive prunedCols datastructure
> (reverse). As a result in the filter we compare with the wrong value and the
> count returns 0. I have been able to identify the cause of this mismatch.
> I would love to get the code reviewed and merged. Some of the code changes
> are changes to commits from Ferdinand Xu and Chao Sun.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21727) Allow For Ordinal Substitution
[
https://issues.apache.org/jira/browse/HIVE-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839576#comment-16839576
]
David Mollitor commented on HIVE-21727:
---
h5. hive.groupby.position.alias
* Default Value: {{false}}
* Added In: Hive 2.2.0 with HIVE-15797
Whether to enable using Column Position Alias in [GROUP
BY|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+GroupBy].
h5. hive.orderby.position.alias
* Default Value: {{true}}
* Added In: Hive 2.2.0 with HIVE-15797
Whether to enable using Column Position Alias in [ORDER
BY|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+SortBy#LanguageManualSortBy-SyntaxofOrderBy].
Do you know why 'orderby' is 'false' and 'groupby' is 'true'?
> Allow For Ordinal Substitution
> ---
>
> Key: HIVE-21727
> URL: https://issues.apache.org/jira/browse/HIVE-21727
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Priority: Major
>
> Impala allows for ordinal substitution. Add a compatible feature to Hive to
> allow Hive to be more compatible with Impala. Allows for more of a drop-in
> replacement.
> [IMPALA-8548]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21727) Allow For Ordinal Substitution
[
https://issues.apache.org/jira/browse/HIVE-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839575#comment-16839575
]
David Mollitor commented on HIVE-21727:
---
h5. hive.groupby.orderby.position.alias
* Default Value: {{false}}
* Added In: Hive 0.11.0 with HIVE-581
* Deprecated In: Hive 2.2.0 with HIVE-15797
Whether to enable using Column Position Alias in [GROUP
BY|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+GroupBy] and
[ORDER
BY|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+SortBy#LanguageManualSortBy-SyntaxofOrderBy]
clauses of queries (deprecated as of Hive 2.2.0; use
[hive.groupby.position.alias|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.groupby.position.alias]
and
[hive.orderby.position.alias|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.orderby.position.alias]
instead).
> Allow For Ordinal Substitution
> ---
>
> Key: HIVE-21727
> URL: https://issues.apache.org/jira/browse/HIVE-21727
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Priority: Major
>
> Impala allows for ordinal substitution. Add a compatible feature to Hive to
> allow Hive to be more compatible with Impala. Allows for more of a drop-in
> replacement.
> [IMPALA-8548]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21686) Brute Force eviction can lead to a random uncontrolled eviction pattern.
[
https://issues.apache.org/jira/browse/HIVE-21686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839574#comment-16839574
]
Hive QA commented on HIVE-21686:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968681/HIVE-21686.5.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16012 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17207/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17207/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17207/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12968681 - PreCommit-HIVE-Build
> Brute Force eviction can lead to a random uncontrolled eviction pattern.
>
>
> Key: HIVE-21686
> URL: https://issues.apache.org/jira/browse/HIVE-21686
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: Cache_hitrate_improvement.csv, HIVE-21686.2.patch,
> HIVE-21686.3.patch, HIVE-21686.4.patch, HIVE-21686.5.patch, HIVE-21686.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Current logic used by brute force eviction can lead to a perpetual random
> eviction pattern.
> For instance if the cache build a small pocket of free memory where the total
> size is greater than incoming allocation request, the allocator will randomly
> evict block that fits a particular size.
> This can happen over and over therefore all the eviction will be random.
> In Addition this random eviction will lead a leak in the linked list
> maintained by the policy since it does not know anymore about what is evicted
> and what not.
> The improvement of this patch is very substantial to TPC-DS benchmark. I
> have tested it with 10TB scale 9 llap nodes and 32GB cache size per node.
> The patch has showed very noticeable difference in the Hit rate for raw
> number [^Cache_hitrate_improvement.csv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21724) Nested ARRAY and STRUCT inside MAP don't work with LazySimpleDeserializeRead
[
https://issues.apache.org/jira/browse/HIVE-21724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839607#comment-16839607
]
Hive QA commented on HIVE-21724:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} serde in master has 193 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
27s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} serde: The patch generated 24 new + 421 unchanged - 5
fixed = 445 total (was 426) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17208/dev-support/hive-personality.sh
|
| git revision | master / de69a0f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17208/yetus/diff-checkstyle-serde.txt
|
| modules | C: serde ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17208/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Nested ARRAY and STRUCT inside MAP don't work with LazySimpleDeserializeRead
>
>
> Key: HIVE-21724
> URL: https://issues.apache.org/jira/browse/HIVE-21724
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-21724.1.patch, HIVE-21724.2.patch
>
>
> The logic during vectorized execution that keeps track of how deep we are in
> the nested structure doesn't work for ARRAYs and STRUCTs embedded inside maps.
> Repro steps (with hive.vectorized.execution.enabled=true):
> {code}
> CREATE TABLE srctable(a map>) STORED AS TEXTFILE;
> create table desttable(c1 map>);
> insert into srctable values (map(1, array(1, 2, 3)));
> insert into desttable select a from srctable;
> select * from desttable;
> {code}
> Will produce:
> {code}
> {1:[null]}
> {code}
--
Thi
[jira] [Commented] (HIVE-21727) Allow For Ordinal Substitution
[ https://issues.apache.org/jira/browse/HIVE-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839621#comment-16839621 ] Greg Rahn commented on HIVE-21727: -- [~jcamachorodriguez] - Might you know? > Allow For Ordinal Substitution > --- > > Key: HIVE-21727 > URL: https://issues.apache.org/jira/browse/HIVE-21727 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Affects Versions: 4.0.0, 3.2.0 >Reporter: David Mollitor >Priority: Major > > Impala allows for ordinal substitution. Add a compatible feature to Hive to > allow Hive to be more compatible with Impala. Allows for more of a drop-in > replacement. > [IMPALA-8548] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21724) Nested ARRAY and STRUCT inside MAP don't work with LazySimpleDeserializeRead
[
https://issues.apache.org/jira/browse/HIVE-21724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839625#comment-16839625
]
Hive QA commented on HIVE-21724:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968684/HIVE-21724.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16009 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_part_1]
(batchId=164)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17208/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17208/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17208/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12968684 - PreCommit-HIVE-Build
> Nested ARRAY and STRUCT inside MAP don't work with LazySimpleDeserializeRead
>
>
> Key: HIVE-21724
> URL: https://issues.apache.org/jira/browse/HIVE-21724
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-21724.1.patch, HIVE-21724.2.patch
>
>
> The logic during vectorized execution that keeps track of how deep we are in
> the nested structure doesn't work for ARRAYs and STRUCTs embedded inside maps.
> Repro steps (with hive.vectorized.execution.enabled=true):
> {code}
> CREATE TABLE srctable(a map>) STORED AS TEXTFILE;
> create table desttable(c1 map>);
> insert into srctable values (map(1, array(1, 2, 3)));
> insert into desttable select a from srctable;
> select * from desttable;
> {code}
> Will produce:
> {code}
> {1:[null]}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21718) Improvement performance of UpdateInputAccessTimeHook
[
https://issues.apache.org/jira/browse/HIVE-21718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839634#comment-16839634
]
Hive QA commented on HIVE-21718:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968685/HIVE-21718.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17209/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17209/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17209/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/slf4j/jul-to-slf4j/1.7.10/jul-to-slf4j-1.7.10.jar(org/slf4j/bridge/SLF4JBridgeHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.25.v20180904/jetty-runner-9.3.25.v20180904.jar(javax/servlet/DispatcherType.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.25.v20180904/jetty-runner-9.3.25.v20180904.jar(javax/servlet/Filter.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.25.v20180904/jetty-runner-9.3.25.v20180904.jar(javax/servlet/FilterChain.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.25.v20180904/jetty-runner-9.3.25.v20180904.jar(javax/servlet/FilterConfig.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.25.v20180904/jetty-runner-9.3.25.v20180904.jar(javax/servlet/ServletException.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.25.v20180904/jetty-runner-9.3.25.v20180904.jar(javax/servlet/ServletRequest.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.25.v20180904/jetty-runner-9.3.25.v20180904.jar(javax/servlet/ServletResponse.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.25.v20180904/jetty-runner-9.3.25.v20180904.jar(javax/servlet/annotation/WebFilter.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.25.v20180904/jetty-runner-9.3.25.v20180904.jar(javax/servlet/http/HttpServletRequest.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-runner/9.3.25.v20180904/jetty-runner-9.3.25.v20180904.jar(javax/servlet/http/HttpServletResponse.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/target/hive-classification-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceAudience$LimitedPrivate.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/target/hive-classification-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability$Unstable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/ByteArrayOutputStream.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/OutputStream.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Closeable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/AutoCloseable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Flushable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(javax/xml/bind/annotation/XmlRootElement.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar(org/apache/commons/exec/ExecuteException.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/security/PrivilegedExceptionAction.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ExecutionException.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/TimeoutException.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/fs/FileSystem.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShimsSecure.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/h
[jira] [Updated] (HIVE-21663) Hive Metastore Translation Layer
[ https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21663: - Status: Open (was: Patch Available) > Hive Metastore Translation Layer > > > Key: HIVE-21663 > URL: https://issues.apache.org/jira/browse/HIVE-21663 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21663.3.patch, HIVE-21663.4.patch, HMS Translation > Layer_v1.0.pdf > > > This task is for the implementation of the default provider for translation, > that is extensible if needed for a custom translator. Please refer the spec > for additional details on the translation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21663) Hive Metastore Translation Layer
[ https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21663: - Attachment: HIVE-21663.5.patch > Hive Metastore Translation Layer > > > Key: HIVE-21663 > URL: https://issues.apache.org/jira/browse/HIVE-21663 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21663.3.patch, HIVE-21663.4.patch, > HIVE-21663.5.patch, HMS Translation Layer_v1.0.pdf > > > This task is for the implementation of the default provider for translation, > that is extensible if needed for a custom translator. Please refer the spec > for additional details on the translation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21663) Hive Metastore Translation Layer
[ https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21663: - Status: Patch Available (was: Open) > Hive Metastore Translation Layer > > > Key: HIVE-21663 > URL: https://issues.apache.org/jira/browse/HIVE-21663 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21663.3.patch, HIVE-21663.4.patch, > HIVE-21663.5.patch, HMS Translation Layer_v1.0.pdf > > > This task is for the implementation of the default provider for translation, > that is extensible if needed for a custom translator. Please refer the spec > for additional details on the translation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21727) Allow For Ordinal Substitution
[ https://issues.apache.org/jira/browse/HIVE-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839682#comment-16839682 ] Jesus Camacho Rodriguez commented on HIVE-21727: [~grahn], I am not sure why those default values for the config variables were chosen. [~ashutoshc] may have more context. > Allow For Ordinal Substitution > --- > > Key: HIVE-21727 > URL: https://issues.apache.org/jira/browse/HIVE-21727 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Affects Versions: 4.0.0, 3.2.0 >Reporter: David Mollitor >Priority: Major > > Impala allows for ordinal substitution. Add a compatible feature to Hive to > allow Hive to be more compatible with Impala. Allows for more of a drop-in > replacement. > [IMPALA-8548] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21663) Hive Metastore Translation Layer
[
https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839696#comment-16839696
]
Hive QA commented on HIVE-21663:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968695/HIVE-21663.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17210/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17210/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17210/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-05-14 18:31:00.926
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-17210/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-05-14 18:31:00.930
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at de69a0f HIVE-21722 : REPL:: logs are missing in
hiveStatement.getQueryLog output during parallel execution mode. (Mahesh Kumar
Behera reviewed by Sankar Hariappan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at de69a0f HIVE-21722 : REPL:: logs are missing in
hiveStatement.getQueryLog output during parallel execution mode. (Mahesh Kumar
Behera reviewed by Sankar Hariappan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-05-14 18:31:02.752
+ rm -rf ../yetus_PreCommit-HIVE-Build-17210
+ mkdir ../yetus_PreCommit-HIVE-Build-17210
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-17210
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-17210/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:908: trailing whitespace.
tmpMap.put(_Fields.TBL_NAME, new
org.apache.thrift.meta_data.FieldMetaData("tblName",
org.apache.thrift.TFieldRequirementType.REQUIRED,
/data/hiveptest/working/scratch/build.patch:910: trailing whitespace.
tmpMap.put(_Fields.ACCESS_TYPE, new
org.apache.thrift.meta_data.FieldMetaData("accessType",
org.apache.thrift.TFieldRequirementType.OPTIONAL,
/data/hiveptest/working/scratch/build.patch:912: trailing whitespace.
tmpMap.put(_Fields.PROCESSOR_CAPABILITIES, new
org.apache.thrift.meta_data.FieldMetaData("processorCapabilities",
org.apache.thrift.TFieldRequirementType.OPTIONAL,
/data/hiveptest/working/scratch/build.patch:913: trailing whitespace.
new
org.apache.thrift.meta_data.ListMetaData(org.apache.thrift.protocol.TType.LIST,
/data/hiveptest/working/scratch/build.patch:1291: trailing whitespace.
if (schemeField.type == org.apache.thrift.protocol.TType.STOP) {
warning: squelched 73 whitespace errors
warning: 78 lines add whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc4062071187926459605.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc4062071187926459605.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/o
[jira] [Updated] (HIVE-21718) Improvement performance of UpdateInputAccessTimeHook
[ https://issues.apache.org/jira/browse/HIVE-21718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21718: - Status: Open (was: Patch Available) > Improvement performance of UpdateInputAccessTimeHook > > > Key: HIVE-21718 > URL: https://issues.apache.org/jira/browse/HIVE-21718 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 2.1.1 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21718.patch > > > Currently, Hive does not update the lastAccessTime property for any entities > when a query accesses them. Thus it has not possible to know when a table was > last accessed. > Hive does provide a configurable hook to HS2 that is execcuted as a pre-query > hook prior to the query being executed. However, this hook is inefficient > because for each table or partition it is attempting to update time for, it > executes an "alter table ... " command internally. This is bad > 1) For a query touching 1000's of partitions, this hook takes forever to > update them. > 2) Meanwhile, it is holding up the original query from executing. > So even though we do not recommend using the hook, because the reward is too > little (having lastAccessTime updated), we realize there is no other means to > achieve this. > Also, we can improve the performance of the hook significantly by adding a > new thrift API on HMS to update the lastAccessTime on the database rows > directly instead of going to HMS front end for 1 entity at time (leading to > 1000's of HMS calls that lead to multiple 1000's of calls to the database). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21718) Improvement performance of UpdateInputAccessTimeHook
[ https://issues.apache.org/jira/browse/HIVE-21718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21718: - Attachment: HIVE-21718.2.patch > Improvement performance of UpdateInputAccessTimeHook > > > Key: HIVE-21718 > URL: https://issues.apache.org/jira/browse/HIVE-21718 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 2.1.1 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21718.2.patch, HIVE-21718.patch > > > Currently, Hive does not update the lastAccessTime property for any entities > when a query accesses them. Thus it has not possible to know when a table was > last accessed. > Hive does provide a configurable hook to HS2 that is execcuted as a pre-query > hook prior to the query being executed. However, this hook is inefficient > because for each table or partition it is attempting to update time for, it > executes an "alter table ... " command internally. This is bad > 1) For a query touching 1000's of partitions, this hook takes forever to > update them. > 2) Meanwhile, it is holding up the original query from executing. > So even though we do not recommend using the hook, because the reward is too > little (having lastAccessTime updated), we realize there is no other means to > achieve this. > Also, we can improve the performance of the hook significantly by adding a > new thrift API on HMS to update the lastAccessTime on the database rows > directly instead of going to HMS front end for 1 entity at time (leading to > 1000's of HMS calls that lead to multiple 1000's of calls to the database). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21718) Improvement performance of UpdateInputAccessTimeHook
[ https://issues.apache.org/jira/browse/HIVE-21718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21718: - Status: Patch Available (was: Open) > Improvement performance of UpdateInputAccessTimeHook > > > Key: HIVE-21718 > URL: https://issues.apache.org/jira/browse/HIVE-21718 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 2.1.1 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21718.2.patch, HIVE-21718.patch > > > Currently, Hive does not update the lastAccessTime property for any entities > when a query accesses them. Thus it has not possible to know when a table was > last accessed. > Hive does provide a configurable hook to HS2 that is execcuted as a pre-query > hook prior to the query being executed. However, this hook is inefficient > because for each table or partition it is attempting to update time for, it > executes an "alter table ... " command internally. This is bad > 1) For a query touching 1000's of partitions, this hook takes forever to > update them. > 2) Meanwhile, it is holding up the original query from executing. > So even though we do not recommend using the hook, because the reward is too > little (having lastAccessTime updated), we realize there is no other means to > achieve this. > Also, we can improve the performance of the hook significantly by adding a > new thrift API on HMS to update the lastAccessTime on the database rows > directly instead of going to HMS front end for 1 entity at time (leading to > 1000's of HMS calls that lead to multiple 1000's of calls to the database). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21663) Hive Metastore Translation Layer
[ https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21663: - Status: Open (was: Patch Available) > Hive Metastore Translation Layer > > > Key: HIVE-21663 > URL: https://issues.apache.org/jira/browse/HIVE-21663 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21663.3.patch, HIVE-21663.4.patch, HMS Translation > Layer_v1.0.pdf > > > This task is for the implementation of the default provider for translation, > that is extensible if needed for a custom translator. Please refer the spec > for additional details on the translation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21663) Hive Metastore Translation Layer
[ https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21663: - Attachment: (was: HIVE-21663.5.patch) > Hive Metastore Translation Layer > > > Key: HIVE-21663 > URL: https://issues.apache.org/jira/browse/HIVE-21663 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21663.3.patch, HIVE-21663.4.patch, HMS Translation > Layer_v1.0.pdf > > > This task is for the implementation of the default provider for translation, > that is extensible if needed for a custom translator. Please refer the spec > for additional details on the translation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21663) Hive Metastore Translation Layer
[ https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21663: - Attachment: HIVE-21663.5.patch > Hive Metastore Translation Layer > > > Key: HIVE-21663 > URL: https://issues.apache.org/jira/browse/HIVE-21663 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21663.3.patch, HIVE-21663.4.patch, > HIVE-21663.5.patch, HMS Translation Layer_v1.0.pdf > > > This task is for the implementation of the default provider for translation, > that is extensible if needed for a custom translator. Please refer the spec > for additional details on the translation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21663) Hive Metastore Translation Layer
[ https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21663: - Status: Patch Available (was: Open) > Hive Metastore Translation Layer > > > Key: HIVE-21663 > URL: https://issues.apache.org/jira/browse/HIVE-21663 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21663.3.patch, HIVE-21663.4.patch, > HIVE-21663.5.patch, HMS Translation Layer_v1.0.pdf > > > This task is for the implementation of the default provider for translation, > that is extensible if needed for a custom translator. Please refer the spec > for additional details on the translation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21663) Hive Metastore Translation Layer
[ https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839740#comment-16839740 ] Daniel Dai commented on HIVE-21663: --- +1ed on RB. > Hive Metastore Translation Layer > > > Key: HIVE-21663 > URL: https://issues.apache.org/jira/browse/HIVE-21663 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21663.3.patch, HIVE-21663.4.patch, > HIVE-21663.5.patch, HMS Translation Layer_v1.0.pdf > > > This task is for the implementation of the default provider for translation, > that is extensible if needed for a custom translator. Please refer the spec > for additional details on the translation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21663) Hive Metastore Translation Layer
[ https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839740#comment-16839740 ] Daniel Dai edited comment on HIVE-21663 at 5/14/19 7:56 PM: +1ed on RB, pending tests. was (Author: daijy): +1ed on RB. > Hive Metastore Translation Layer > > > Key: HIVE-21663 > URL: https://issues.apache.org/jira/browse/HIVE-21663 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21663.3.patch, HIVE-21663.4.patch, > HIVE-21663.5.patch, HMS Translation Layer_v1.0.pdf > > > This task is for the implementation of the default provider for translation, > that is extensible if needed for a custom translator. Please refer the spec > for additional details on the translation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21718) Improvement performance of UpdateInputAccessTimeHook
[
https://issues.apache.org/jira/browse/HIVE-21718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839781#comment-16839781
]
Hive QA commented on HIVE-21718:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
56s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
30s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 1 new + 388 unchanged - 0 fixed = 389 total (was 388) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 5 new + 1638 unchanged - 0 fixed = 1643 total (was 1638) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 1 new + 208 unchanged - 0
fixed = 209 total (was 208) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
26s{color} | {color:red} standalone-metastore/metastore-server generated 3 new
+ 181 unchanged - 0 fixed = 184 total (was 181) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
5s{color} | {color:red} standalone-metastore_metastore-common generated 2 new +
45 unchanged - 0 fixed = 47 total (was 45) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| |
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.updateLastAccessTime(Map,
int) concatenates strings using + in a loop At MetaStoreDirectSql.java:+ in a
loop At MetaStoreDirectSql.java:[line 545] |
| |
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.updateLastAccessTime(Map,
int) passes a nonconstant String to an execute or addBatch method on an SQL
statement At MetaStoreDirectSql.java:String to an execute or addBatch method
on an SQL statement At MetaStoreDirectSql.java:[line 558] |
| |
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.updateLastAccessTime(Map,
int) makes inefficient use of keySet iterator instead of entrySet iterator At
MetaStoreDirectSql.java:of keySet iterator instead of entrySet iterator At
MetaStoreDirectSql.java:[line 536] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-
[jira] [Commented] (HIVE-21718) Improvement performance of UpdateInputAccessTimeHook
[
https://issues.apache.org/jira/browse/HIVE-21718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839803#comment-16839803
]
Hive QA commented on HIVE-21718:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968701/HIVE-21718.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16008 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17211/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17211/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17211/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12968701 - PreCommit-HIVE-Build
> Improvement performance of UpdateInputAccessTimeHook
>
>
> Key: HIVE-21718
> URL: https://issues.apache.org/jira/browse/HIVE-21718
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 2.1.1
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-21718.2.patch, HIVE-21718.patch
>
>
> Currently, Hive does not update the lastAccessTime property for any entities
> when a query accesses them. Thus it has not possible to know when a table was
> last accessed.
> Hive does provide a configurable hook to HS2 that is execcuted as a pre-query
> hook prior to the query being executed. However, this hook is inefficient
> because for each table or partition it is attempting to update time for, it
> executes an "alter table ... " command internally. This is bad
> 1) For a query touching 1000's of partitions, this hook takes forever to
> update them.
> 2) Meanwhile, it is holding up the original query from executing.
> So even though we do not recommend using the hook, because the reward is too
> little (having lastAccessTime updated), we realize there is no other means to
> achieve this.
> Also, we can improve the performance of the hook significantly by adding a
> new thrift API on HMS to update the lastAccessTime on the database rows
> directly instead of going to HMS front end for 1 entity at time (leading to
> 1000's of HMS calls that lead to multiple 1000's of calls to the database).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21727) Allow For Ordinal Substitution
[ https://issues.apache.org/jira/browse/HIVE-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839811#comment-16839811 ] David Mollitor commented on HIVE-21727: --- I'm also curious to know why this is optional at all. Is there some sort of overhead on this? > Allow For Ordinal Substitution > --- > > Key: HIVE-21727 > URL: https://issues.apache.org/jira/browse/HIVE-21727 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Affects Versions: 4.0.0, 3.2.0 >Reporter: David Mollitor >Priority: Major > > Impala allows for ordinal substitution. Add a compatible feature to Hive to > allow Hive to be more compatible with Impala. Allows for more of a drop-in > replacement. > [IMPALA-8548] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21718) Improvement performance of UpdateInputAccessTimeHook
[ https://issues.apache.org/jira/browse/HIVE-21718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839819#comment-16839819 ] Naveen Gangam commented on HIVE-21718: -- [~aihuaxu] [~ychena] [~daijy] Could you please review this ? Thanks > Improvement performance of UpdateInputAccessTimeHook > > > Key: HIVE-21718 > URL: https://issues.apache.org/jira/browse/HIVE-21718 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 2.1.1 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21718.2.patch, HIVE-21718.patch > > > Currently, Hive does not update the lastAccessTime property for any entities > when a query accesses them. Thus it has not possible to know when a table was > last accessed. > Hive does provide a configurable hook to HS2 that is execcuted as a pre-query > hook prior to the query being executed. However, this hook is inefficient > because for each table or partition it is attempting to update time for, it > executes an "alter table ... " command internally. This is bad > 1) For a query touching 1000's of partitions, this hook takes forever to > update them. > 2) Meanwhile, it is holding up the original query from executing. > So even though we do not recommend using the hook, because the reward is too > little (having lastAccessTime updated), we realize there is no other means to > achieve this. > Also, we can improve the performance of the hook significantly by adding a > new thrift API on HMS to update the lastAccessTime on the database rows > directly instead of going to HMS front end for 1 entity at time (leading to > 1000's of HMS calls that lead to multiple 1000's of calls to the database). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21663) Hive Metastore Translation Layer
[
https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839820#comment-16839820
]
Hive QA commented on HIVE-21663:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
3s{color} | {color:blue} standalone-metastore/metastore-common in master has 29
extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
20s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 29 new + 471 unchanged - 0 fixed = 500 total (was 471) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 408 new + 797 unchanged - 2 fixed = 1205 total (was 799) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
14s{color} | {color:red} standalone-metastore/metastore-common generated 2 new
+ 29 unchanged - 0 fixed = 31 total (was 29) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
29s{color} | {color:red} standalone-metastore/metastore-server generated 3 new
+ 181 unchanged - 0 fixed = 184 total (was 181) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
4s{color} | {color:red} standalone-metastore_metastore-common generated 2 new +
45 unchanged - 0 fixed = 47 total (was 45) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
25s{color} | {color:red} standalone-metastore_metastore-server generated 1 new
+ 25 unchanged - 0 fixed = 26 total (was 25) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-common |
| |
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.setProcessorCapabilities(String[])
may expose internal static state by storing a mutable object into a static
field
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.processorCapabilities At
HiveMetaStoreClient.java:state by storing a mutable object into a static field
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.processorCapabilities At
HiveMetaStoreClient.java:[line 725] |
| | new
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$TableCapabilityBuilder()
invokes inefficient new String() constructor At
HiveMetaStoreClient.java:String() constructor At
HiveMetaStoreClient.java:
[jira] [Commented] (HIVE-21718) Improvement performance of UpdateInputAccessTimeHook
[ https://issues.apache.org/jira/browse/HIVE-21718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839822#comment-16839822 ] Naveen Gangam commented on HIVE-21718: -- Review posted to RB at https://reviews.apache.org/r/70645/ > Improvement performance of UpdateInputAccessTimeHook > > > Key: HIVE-21718 > URL: https://issues.apache.org/jira/browse/HIVE-21718 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 2.1.1 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21718.2.patch, HIVE-21718.patch > > > Currently, Hive does not update the lastAccessTime property for any entities > when a query accesses them. Thus it has not possible to know when a table was > last accessed. > Hive does provide a configurable hook to HS2 that is execcuted as a pre-query > hook prior to the query being executed. However, this hook is inefficient > because for each table or partition it is attempting to update time for, it > executes an "alter table ... " command internally. This is bad > 1) For a query touching 1000's of partitions, this hook takes forever to > update them. > 2) Meanwhile, it is holding up the original query from executing. > So even though we do not recommend using the hook, because the reward is too > little (having lastAccessTime updated), we realize there is no other means to > achieve this. > Also, we can improve the performance of the hook significantly by adding a > new thrift API on HMS to update the lastAccessTime on the database rows > directly instead of going to HMS front end for 1 entity at time (leading to > 1000's of HMS calls that lead to multiple 1000's of calls to the database). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[ https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21725: -- Status: Open (was: Patch Available) > Break up DDLTask - extract Column and Constraint related operations > --- > > Key: HIVE-21725 > URL: https://issues.apache.org/jira/browse/HIVE-21725 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #9: extract all the column and constraint related operations from the > old DDLTask, and move them under the new package each. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[ https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21725: -- Attachment: HIVE-21725.03.patch > Break up DDLTask - extract Column and Constraint related operations > --- > > Key: HIVE-21725 > URL: https://issues.apache.org/jira/browse/HIVE-21725 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch, > HIVE-21725.03.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #9: extract all the column and constraint related operations from the > old DDLTask, and move them under the new package each. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[ https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21725: -- Status: Patch Available (was: Open) > Break up DDLTask - extract Column and Constraint related operations > --- > > Key: HIVE-21725 > URL: https://issues.apache.org/jira/browse/HIVE-21725 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch, > HIVE-21725.03.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #9: extract all the column and constraint related operations from the > old DDLTask, and move them under the new package each. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21663) Hive Metastore Translation Layer
[
https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839845#comment-16839845
]
Hive QA commented on HIVE-21663:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968702/HIVE-21663.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 16050 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testGetPartitionsByNames
(batchId=226)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetPartitionsByNames
(batchId=228)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetPartitionsByNames
(batchId=232)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetPartitionsByNames
(batchId=223)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testGetPartitionsByNames
(batchId=221)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testGetPartitionsByNames
(batchId=231)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17212/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17212/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17212/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12968702 - PreCommit-HIVE-Build
> Hive Metastore Translation Layer
>
>
> Key: HIVE-21663
> URL: https://issues.apache.org/jira/browse/HIVE-21663
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-21663.3.patch, HIVE-21663.4.patch,
> HIVE-21663.5.patch, HMS Translation Layer_v1.0.pdf
>
>
> This task is for the implementation of the default provider for translation,
> that is extensible if needed for a custom translator. Please refer the spec
> for additional details on the translation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21697) Remove periodical full refresh in HMS cache
[ https://issues.apache.org/jira/browse/HIVE-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21697: -- Attachment: HIVE-21697.3.patch > Remove periodical full refresh in HMS cache > --- > > Key: HIVE-21697 > URL: https://issues.apache.org/jira/browse/HIVE-21697 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21697.1.patch, HIVE-21697.2.patch, > HIVE-21697.3.patch > > > In HIVE-18661, we added periodical notification based refresh in HMS cache. > We shall remove periodical full refresh to simplify the code as it will no > longer be used. In the mean time, we introduced mechanism to provide > monotonic reads through the CachedStore.commitTransaction. This will no > longer be needed after HIVE-21637. So I will remove related code as well. > This will provide some performance benefits include: > 1. We don't have to slow down write to catch up notification logs. Write can > be done immediately and tag the cache with writeids > 2. We can read from cache even if updateUsingNotificationEvents is running. > Read will compare the writeids of the cache so monotonic reads will be > guaranteed > I'd like to put a patch separately with HIVE-21637 so it can be tested > independently. HMW will use periodical notification based refresh to update > cache. And it will temporarily lift the monotonic reads guarantee until > HIVE-21637 checkin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21728) WorkloadManager logging fix
[ https://issues.apache.org/jira/browse/HIVE-21728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh reassigned HIVE-21728: - > WorkloadManager logging fix > > > Key: HIVE-21728 > URL: https://issues.apache.org/jira/browse/HIVE-21728 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.2.0 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > > logger skip the following message if HS2 is running in INFO level. > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java#L705 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21728) WorkloadManager logging fix
[ https://issues.apache.org/jira/browse/HIVE-21728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21728 started by Rajkumar Singh. - > WorkloadManager logging fix > > > Key: HIVE-21728 > URL: https://issues.apache.org/jira/browse/HIVE-21728 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.2.0 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > > logger skip the following message if HS2 is running in INFO level. > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java#L705 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21728) WorkloadManager logging fix
[ https://issues.apache.org/jira/browse/HIVE-21728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-21728: -- Attachment: HIVE-21728.patch Status: Patch Available (was: In Progress) > WorkloadManager logging fix > > > Key: HIVE-21728 > URL: https://issues.apache.org/jira/browse/HIVE-21728 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.2.0 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21728.patch > > > logger skip the following message if HS2 is running in INFO level. > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java#L705 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[
https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839861#comment-16839861
]
Hive QA commented on HIVE-21725:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
26s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 14 new + 285 unchanged - 22
fixed = 299 total (was 307) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
41s{color} | {color:red} ql generated 2 new + 2257 unchanged - 1 fixed = 2259
total (was 2258) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Unwritten field:AlterTableWithWriteIdDesc.java:[line 68] |
| | Class
org.apache.hadoop.hive.ql.ddl.table.constaint.AlterTableDropConstraintDesc
defines non-transient non-serializable instance field replicationSpec In
AlterTableDropConstraintDesc.java:instance field replicationSpec In
AlterTableDropConstraintDesc.java |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17213/dev-support/hive-personality.sh
|
| git revision | master / de69a0f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17213/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17213/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17213/yetus/new-findbugs-ql.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17213/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17213/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Break up DDLTask - extract Column and Constraint related operations
> ---
>
> Key: HIVE-21725
> URL: https://issues.apache.org/jira/browse/HIVE-21725
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch,
> HIVE-21725.03.patch
>
>
> DDLTask is a huge class, more than 5000 lines long. The related DDLWork is
> also a huge class, which has
[jira] [Commented] (HIVE-21728) WorkloadManager logging fix
[ https://issues.apache.org/jira/browse/HIVE-21728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839870#comment-16839870 ] Gopal V commented on HIVE-21728: Would it be more useful to have a new Logger category for WM-Audit (similar to RM-audit logs)? > WorkloadManager logging fix > > > Key: HIVE-21728 > URL: https://issues.apache.org/jira/browse/HIVE-21728 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.2.0 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21728.patch > > > logger skip the following message if HS2 is running in INFO level. > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java#L705 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[
https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839882#comment-16839882
]
Hive QA commented on HIVE-21725:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968716/HIVE-21725.03.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15979 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=185)
[vector_udf_adaptor_1.q,schema_evol_text_vec_part_llap_io.q,join_is_not_distinct_from.q,tez_schema_evolution.q,vector_multi_insert.q,input16_cc.q,materialized_view_rewrite_no_join_opt.q,temp_table.q,cross_product_check_2.q,partition_pruning.q,vectorization_17.q,materialized_view_create_rewrite_2.q,bucket_groupby.q,vector_nullsafe_join.q,vector_decimal_udf.q,materialized_view_rewrite_3.q,schema_evol_text_nonvec_part_all_primitive_llap_io.q,bucketpruning1.q,vector_groupby6.q,vector_char_varchar_1.q,smb_mapjoin_5.q,vector_char_4.q,auto_sortmerge_join_5.q,auto_join0.q,external_jdbc_table3.q,cbo_limit.q,vector_udf_inline.q,vector_windowing.q,schema_evol_orc_nonvec_part_all_primitive.q,auto_sortmerge_join_11.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17213/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17213/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17213/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12968716 - PreCommit-HIVE-Build
> Break up DDLTask - extract Column and Constraint related operations
> ---
>
> Key: HIVE-21725
> URL: https://issues.apache.org/jira/browse/HIVE-21725
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch,
> HIVE-21725.03.patch
>
>
> DDLTask is a huge class, more than 5000 lines long. The related DDLWork is
> also a huge class, which has a field for each DDL operation it supports. The
> goal is to refactor these in order to have everything cut into more
> handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each operation
> * have a package for each operation group (database ddl, table ddl, etc), so
> the amount of classes under a package is more manageable
> * make all the requests (DDLDesc subclasses) immutable
> * DDLTask should be agnostic to the actual operations
> * right now let's ignore the issue of having some operations handled by
> DDLTask which are not actual DDL operations (lock, unlock, desc...)
> In the interim time when there are two DDLTask and DDLWork classes in the
> code base the new ones in the new package are called DDLTask2 and DDLWork2
> thus avoiding the usage of fully qualified class names where both the old and
> the new classes are in use.
> Step #9: extract all the column and constraint related operations from the
> old DDLTask, and move them under the new package each.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[ https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21725: -- Attachment: HIVE-21725.04.patch > Break up DDLTask - extract Column and Constraint related operations > --- > > Key: HIVE-21725 > URL: https://issues.apache.org/jira/browse/HIVE-21725 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch, > HIVE-21725.03.patch, HIVE-21725.04.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #9: extract all the column and constraint related operations from the > old DDLTask, and move them under the new package each. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[ https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21725: -- Status: Open (was: Patch Available) > Break up DDLTask - extract Column and Constraint related operations > --- > > Key: HIVE-21725 > URL: https://issues.apache.org/jira/browse/HIVE-21725 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch, > HIVE-21725.03.patch, HIVE-21725.04.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #9: extract all the column and constraint related operations from the > old DDLTask, and move them under the new package each. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[ https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21725: -- Status: Patch Available (was: Open) > Break up DDLTask - extract Column and Constraint related operations > --- > > Key: HIVE-21725 > URL: https://issues.apache.org/jira/browse/HIVE-21725 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch, > HIVE-21725.03.patch, HIVE-21725.04.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #9: extract all the column and constraint related operations from the > old DDLTask, and move them under the new package each. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21697) Remove periodical full refresh in HMS cache
[
https://issues.apache.org/jira/browse/HIVE-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839896#comment-16839896
]
Hive QA commented on HIVE-21697:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
58s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
18s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 3 new + 571 unchanged - 8 fixed = 574 total (was 579) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
23s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 181 unchanged - 0 fixed = 182 total (was 181) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Load of known null value in
org.apache.hadoop.hive.metastore.cache.CachedStore.get_aggr_stats_for(String,
String, String, List, List, String) At CachedStore.java:in
org.apache.hadoop.hive.metastore.cache.CachedStore.get_aggr_stats_for(String,
String, String, List, List, String) At CachedStore.java:[line 2009] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17214/dev-support/hive-personality.sh
|
| git revision | master / de69a0f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17214/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17214/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17214/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove periodical full refresh in HMS cache
> -
[jira] [Updated] (HIVE-21663) Hive Metastore Translation Layer
[ https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21663: - Status: Patch Available (was: Open) > Hive Metastore Translation Layer > > > Key: HIVE-21663 > URL: https://issues.apache.org/jira/browse/HIVE-21663 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21663.3.patch, HIVE-21663.4.patch, > HIVE-21663.5.patch, HIVE-21663.6.patch, HMS Translation Layer_v1.0.pdf > > > This task is for the implementation of the default provider for translation, > that is extensible if needed for a custom translator. Please refer the spec > for additional details on the translation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21663) Hive Metastore Translation Layer
[ https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21663: - Attachment: HIVE-21663.6.patch > Hive Metastore Translation Layer > > > Key: HIVE-21663 > URL: https://issues.apache.org/jira/browse/HIVE-21663 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21663.3.patch, HIVE-21663.4.patch, > HIVE-21663.5.patch, HIVE-21663.6.patch, HMS Translation Layer_v1.0.pdf > > > This task is for the implementation of the default provider for translation, > that is extensible if needed for a custom translator. Please refer the spec > for additional details on the translation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21663) Hive Metastore Translation Layer
[ https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21663: - Status: Open (was: Patch Available) > Hive Metastore Translation Layer > > > Key: HIVE-21663 > URL: https://issues.apache.org/jira/browse/HIVE-21663 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21663.3.patch, HIVE-21663.4.patch, > HIVE-21663.5.patch, HIVE-21663.6.patch, HMS Translation Layer_v1.0.pdf > > > This task is for the implementation of the default provider for translation, > that is extensible if needed for a custom translator. Please refer the spec > for additional details on the translation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21697) Remove periodical full refresh in HMS cache
[
https://issues.apache.org/jira/browse/HIVE-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839903#comment-16839903
]
Hive QA commented on HIVE-21697:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968719/HIVE-21697.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 16005 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testAlterDatabase
(batchId=232)
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testAlterTable
(batchId=232)
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testCacheUpdate
(batchId=232)
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testCreateAndGetTable
(batchId=232)
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testDropDatabase
(batchId=232)
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testDropTable
(batchId=232)
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testPrewarmBlackList
(batchId=232)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testAggrStat
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testAggrStatAbortTxn
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testAggrStatTxnTable
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableColumnStatisticsTxnTableMultiAbort
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableColumnStatisticsTxnTableOpenTxn
(batchId=242)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableOpsForUpdateUsingEvents
(batchId=242)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=274)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=274)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=274)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17214/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17214/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17214/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12968719 - PreCommit-HIVE-Build
> Remove periodical full refresh in HMS cache
> ---
>
> Key: HIVE-21697
> URL: https://issues.apache.org/jira/browse/HIVE-21697
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21697.1.patch, HIVE-21697.2.patch,
> HIVE-21697.3.patch
>
>
> In HIVE-18661, we added periodical notification based refresh in HMS cache.
> We shall remove periodical full refresh to simplify the code as it will no
> longer be used. In the mean time, we introduced mechanism to provide
> monotonic reads through the CachedStore.commitTransaction. This will no
> longer be needed after HIVE-21637. So I will remove related code as well.
> This will provide some performance benefits include:
> 1. We don't have to slow down write to catch up notification logs. Write can
> be done immediately and tag the cache with writeids
> 2. We can read from cache even if updateUsingNotificationEvents is running.
> Read will compare the writeids of the cache so monotonic reads will be
> guaranteed
> I'd like to put a patch separately with HIVE-21637 so it can be tested
> independently. HMW will use periodical notification based refresh to update
> cache. And it will temporarily lift the monotonic reads guarantee until
> HIVE-21637 checkin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21728) WorkloadManager logging fix
[
https://issues.apache.org/jira/browse/HIVE-21728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839922#comment-16839922
]
Hive QA commented on HIVE-21728:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
36s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17215/dev-support/hive-personality.sh
|
| git revision | master / de69a0f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17215/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> WorkloadManager logging fix
>
>
> Key: HIVE-21728
> URL: https://issues.apache.org/jira/browse/HIVE-21728
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.2.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21728.patch
>
>
> logger skip the following message if HS2 is running in INFO level.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java#L705
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21728) WorkloadManager logging fix
[
https://issues.apache.org/jira/browse/HIVE-21728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839949#comment-16839949
]
Hive QA commented on HIVE-21728:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968720/HIVE-21728.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16008 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17215/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17215/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17215/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12968720 - PreCommit-HIVE-Build
> WorkloadManager logging fix
>
>
> Key: HIVE-21728
> URL: https://issues.apache.org/jira/browse/HIVE-21728
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.2.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21728.patch
>
>
> logger skip the following message if HS2 is running in INFO level.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java#L705
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[
https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839977#comment-16839977
]
Hive QA commented on HIVE-21725:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
26s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 1 new + 285 unchanged - 22
fixed = 286 total (was 307) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
37s{color} | {color:red} ql generated 2 new + 2257 unchanged - 1 fixed = 2259
total (was 2258) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Unwritten field:AlterTableWithWriteIdDesc.java:[line 89] |
| | Class
org.apache.hadoop.hive.ql.ddl.table.constaint.AlterTableDropConstraintDesc
defines non-transient non-serializable instance field replicationSpec In
AlterTableDropConstraintDesc.java:instance field replicationSpec In
AlterTableDropConstraintDesc.java |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17216/dev-support/hive-personality.sh
|
| git revision | master / de69a0f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17216/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17216/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17216/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Break up DDLTask - extract Column and Constraint related operations
> ---
>
> Key: HIVE-21725
> URL: https://issues.apache.org/jira/browse/HIVE-21725
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch,
> HIVE-21725.03.patch, HIVE-21725.04.patch
>
>
> DDLTask is a huge class, more than 5000 lines long. The related DDLWork is
> also a huge class, which has a field for each DDL operation it supports. The
> goal is to refactor these in order to have everything cut into more
> handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each operation
> * have a pack
[jira] [Commented] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[
https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840004#comment-16840004
]
Hive QA commented on HIVE-21725:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968724/HIVE-21725.04.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 16008 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_ctas]
(batchId=188)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.checkExpectedLocks
(batchId=338)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMetadataOperationLocks
(batchId=338)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
(batchId=246)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17216/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17216/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17216/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12968724 - PreCommit-HIVE-Build
> Break up DDLTask - extract Column and Constraint related operations
> ---
>
> Key: HIVE-21725
> URL: https://issues.apache.org/jira/browse/HIVE-21725
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch,
> HIVE-21725.03.patch, HIVE-21725.04.patch
>
>
> DDLTask is a huge class, more than 5000 lines long. The related DDLWork is
> also a huge class, which has a field for each DDL operation it supports. The
> goal is to refactor these in order to have everything cut into more
> handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each operation
> * have a package for each operation group (database ddl, table ddl, etc), so
> the amount of classes under a package is more manageable
> * make all the requests (DDLDesc subclasses) immutable
> * DDLTask should be agnostic to the actual operations
> * right now let's ignore the issue of having some operations handled by
> DDLTask which are not actual DDL operations (lock, unlock, desc...)
> In the interim time when there are two DDLTask and DDLWork classes in the
> code base the new ones in the new package are called DDLTask2 and DDLWork2
> thus avoiding the usage of fully qualified class names where both the old and
> the new classes are in use.
> Step #9: extract all the column and constraint related operations from the
> old DDLTask, and move them under the new package each.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21663) Hive Metastore Translation Layer
[
https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840013#comment-16840013
]
Hive QA commented on HIVE-21663:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
46s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 29 new + 471 unchanged - 0 fixed = 500 total (was 471) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 407 new + 797 unchanged - 2 fixed = 1204 total (was 799) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
58s{color} | {color:red} standalone-metastore/metastore-common generated 2 new
+ 29 unchanged - 0 fixed = 31 total (was 29) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
27s{color} | {color:red} standalone-metastore/metastore-server generated 3 new
+ 181 unchanged - 0 fixed = 184 total (was 181) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
58s{color} | {color:red} standalone-metastore_metastore-common generated 2 new
+ 45 unchanged - 0 fixed = 47 total (was 45) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore_metastore-server generated 1 new
+ 25 unchanged - 0 fixed = 26 total (was 25) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-common |
| |
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.setProcessorCapabilities(String[])
may expose internal static state by storing a mutable object into a static
field
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.processorCapabilities At
HiveMetaStoreClient.java:state by storing a mutable object into a static field
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.processorCapabilities At
HiveMetaStoreClient.java:[line 725] |
| | new
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$TableCapabilityBuilder()
invokes inefficient new String() constructor At
HiveMetaStoreClient.java:String() constructor At
HiveMetaStoreClient.java:
[jira] [Updated] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[ https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21725: -- Attachment: HIVE-21725.05.patch > Break up DDLTask - extract Column and Constraint related operations > --- > > Key: HIVE-21725 > URL: https://issues.apache.org/jira/browse/HIVE-21725 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch, > HIVE-21725.03.patch, HIVE-21725.04.patch, HIVE-21725.05.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #9: extract all the column and constraint related operations from the > old DDLTask, and move them under the new package each. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[ https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21725: -- Status: Open (was: Patch Available) > Break up DDLTask - extract Column and Constraint related operations > --- > > Key: HIVE-21725 > URL: https://issues.apache.org/jira/browse/HIVE-21725 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch, > HIVE-21725.03.patch, HIVE-21725.04.patch, HIVE-21725.05.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #9: extract all the column and constraint related operations from the > old DDLTask, and move them under the new package each. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[ https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21725: -- Status: Patch Available (was: Open) > Break up DDLTask - extract Column and Constraint related operations > --- > > Key: HIVE-21725 > URL: https://issues.apache.org/jira/browse/HIVE-21725 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch, > HIVE-21725.03.patch, HIVE-21725.04.patch, HIVE-21725.05.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #9: extract all the column and constraint related operations from the > old DDLTask, and move them under the new package each. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21729) Arrow serializer sometimes shifts timestamp by one second
[
https://issues.apache.org/jira/browse/HIVE-21729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shubham Chaurasia reassigned HIVE-21729:
> Arrow serializer sometimes shifts timestamp by one second
> -
>
> Key: HIVE-21729
> URL: https://issues.apache.org/jira/browse/HIVE-21729
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>
> This happens due to
> [secondInMicros|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Serializer.java#L445]
> are calculated like
> {code}
> final long secondInMicros = (secondInMillis - secondInMillis %
> MILLIS_PER_SECOND) * MICROS_PER_MILLIS;
> {code}
> Instead this should be calculated like(by taking nanos from
> timestampColumnVector itself)
> {code}
> final long nanos = timestampColumnVector.getNanos(j);
> final long secondInMicros = (secondInMillis - nanos / NS_PER_MILLIS) *
> MICROS_PER_MILLIS;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21729) Arrow serializer sometimes shifts timestamp by one second
[
https://issues.apache.org/jira/browse/HIVE-21729?focusedWorklogId=242248&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242248
]
ASF GitHub Bot logged work on HIVE-21729:
-
Author: ASF GitHub Bot
Created on: 15/May/19 05:28
Start Date: 15/May/19 05:28
Worklog Time Spent: 10m
Work Description: ShubhamChaurasia commented on pull request #627:
HIVE-21729: Arrow serializer sometimes shifts timestamp by one second
URL: https://github.com/apache/hive/pull/627
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242248)
Time Spent: 10m
Remaining Estimate: 0h
> Arrow serializer sometimes shifts timestamp by one second
> -
>
> Key: HIVE-21729
> URL: https://issues.apache.org/jira/browse/HIVE-21729
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This happens due to
> [secondInMicros|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Serializer.java#L445]
> are calculated like
> {code}
> final long secondInMicros = (secondInMillis - secondInMillis %
> MILLIS_PER_SECOND) * MICROS_PER_MILLIS;
> {code}
> Instead this should be calculated like(by taking nanos from
> timestampColumnVector itself)
> {code}
> final long nanos = timestampColumnVector.getNanos(j);
> final long secondInMicros = (secondInMillis - nanos / NS_PER_MILLIS) *
> MICROS_PER_MILLIS;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21729) Arrow serializer sometimes shifts timestamp by one second
[
https://issues.apache.org/jira/browse/HIVE-21729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-21729:
--
Labels: pull-request-available (was: )
> Arrow serializer sometimes shifts timestamp by one second
> -
>
> Key: HIVE-21729
> URL: https://issues.apache.org/jira/browse/HIVE-21729
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
>
> This happens due to
> [secondInMicros|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Serializer.java#L445]
> are calculated like
> {code}
> final long secondInMicros = (secondInMillis - secondInMillis %
> MILLIS_PER_SECOND) * MICROS_PER_MILLIS;
> {code}
> Instead this should be calculated like(by taking nanos from
> timestampColumnVector itself)
> {code}
> final long nanos = timestampColumnVector.getNanos(j);
> final long secondInMicros = (secondInMillis - nanos / NS_PER_MILLIS) *
> MICROS_PER_MILLIS;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21729) Arrow serializer sometimes shifts timestamp by one second
[
https://issues.apache.org/jira/browse/HIVE-21729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shubham Chaurasia updated HIVE-21729:
-
Attachment: HIVE-21729.1.patch
> Arrow serializer sometimes shifts timestamp by one second
> -
>
> Key: HIVE-21729
> URL: https://issues.apache.org/jira/browse/HIVE-21729
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21729.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This happens due to
> [secondInMicros|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Serializer.java#L445]
> are calculated like
> {code}
> final long secondInMicros = (secondInMillis - secondInMillis %
> MILLIS_PER_SECOND) * MICROS_PER_MILLIS;
> {code}
> Instead this should be calculated like(by taking nanos from
> timestampColumnVector itself)
> {code}
> final long nanos = timestampColumnVector.getNanos(j);
> final long secondInMicros = (secondInMillis - nanos / NS_PER_MILLIS) *
> MICROS_PER_MILLIS;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21730) HiveStatement.getQueryId throws TProtocolException when response is null.
[
https://issues.apache.org/jira/browse/HIVE-21730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan reassigned HIVE-21730:
---
> HiveStatement.getQueryId throws TProtocolException when response is null.
> -
>
> Key: HIVE-21730
> URL: https://issues.apache.org/jira/browse/HIVE-21730
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>
> HiveStatement.getQueryId is failing with below exception.
> {code}
> 24256 2019-05-14T02:09:01,355 INFO [HiveServer2-Background-Pool:
> Thread-1829] ql.Driver: Executing
> command(queryId=hive_20190514020858_530a33d9-0b19-4f72-ae08-b631fb4749cb):
> create table household_demographics
> 24257 stored as orc as
> 24258 select * from household_demographics_txt
> 24259 2019-05-14T02:09:01,356 INFO [HiveServer2-Background-Pool:
> Thread-1829] hooks.HiveProtoLoggingHook: Received pre-hook notification for:
> hive_20190514020858_530a33d9-0b19-4f72-ae08-b631fb4749cb
> 24260 2019-05-14T02:09:01,356 ERROR [HiveServer2-Handler-Pool: Thread-131]
> server.TThreadPoolServer: Thrift error occurred during processing of message.
> 24261 org.apache.thrift.protocol.TProtocolException: Required field
> 'queryId' is unset! Struct:TGetQueryIdResp(queryId:null)
>
>
> 24216,1
> 10%
> 24260 2019-05-14T02:09:01,356 ERROR [HiveServer2-Handler-Pool: Thread-131]
> server.TThreadPoolServer: Thrift error occurred during processing of message.
> 24261 org.apache.thrift.protocol.TProtocolException: Required field
> 'queryId' is unset! Struct:TGetQueryIdResp(queryId:null)
> 24262 at
> org.apache.hive.service.rpc.thrift.TGetQueryIdResp.validate(TGetQueryIdResp.java:294)
> ~[hive-exec-2.1.0.2.6.5.1150-19.jar:2.1.0.2.6.5.1150-19]
> 24263 at
> org.apache.hive.service.rpc.thrift.TCLIService$GetQueryId_result.validate(TCLIService.java:18890)
> ~[hive-exec-2.1.0.2.6.5.1150-19.jar:2.1.0.2.6.5.1150-19]
> 24264 at
> org.apache.hive.service.rpc.thrift.TCLIService$GetQueryId_result$GetQueryId_resultStandardScheme.write(TCLIService.java:18947)
> ~[hive-exec-2.1.0.2.6.5.1150-19.jar:2.1.0.2.6.5.1150-19]
> 24265 at
> org.apache.hive.service.rpc.thrift.TCLIService$GetQueryId_result$GetQueryId_resultStandardScheme.write(TCLIService.java:18916)
> ~[hive-exec-2.1.0.2.6.5.1150-19.jar:2.1.0.2.6.5.1150-19]
> 24266 at
> org.apache.hive.service.rpc.thrift.TCLIService$GetQueryId_result.write(TCLIService.java:18867)
> ~[hive-exec-2.1.0.2.6.5.1150-19.jar:2.1.0.2.6.5.1150-19]
> 24267 at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:53)
> ~[hive-exec-2.1.0.2.6.5.1150-19.jar:2.1.0.2.6.5.1150-19]
> 24268 at
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[hive-exec-2.1.0.2.6.5.1150-19.jar:2.1.0.2.6.5.1150-19]
> 24269 at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> ~[hive-service-2.1.0.2.6.5.1150-19.jar:2.1.0.2.6.5.1150-19]
> 24270 at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.0.2.6.5.1150-19.jar:2.1.0.2.6.5.1150-19]
> 24271 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_161]
> 24272 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_161]
> 24273 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21663) Hive Metastore Translation Layer
[
https://issues.apache.org/jira/browse/HIVE-21663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840050#comment-16840050
]
Hive QA commented on HIVE-21663:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12968730/HIVE-21663.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 60 failed/errored test(s), 16050 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testComplexTypeApi
(batchId=226)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testConcurrentMetastores
(batchId=226)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testCreateAndGetTableWithDriver
(batchId=226)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testFunctionWithResources
(batchId=226)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testGetPartitionsByNames
(batchId=226)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testGetTableObjects
(batchId=226)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testPartitionFilter
(batchId=226)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testRenamePartition
(batchId=226)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testSimpleFunction
(batchId=226)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testTableDatabase
(batchId=226)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testComplexTypeApi
(batchId=228)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testConcurrentMetastores
(batchId=228)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testCreateAndGetTableWithDriver
(batchId=228)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFunctionWithResources
(batchId=228)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetPartitionsByNames
(batchId=228)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetTableObjects
(batchId=228)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testPartitionFilter
(batchId=228)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testRenamePartition
(batchId=228)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleFunction
(batchId=228)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testTableDatabase
(batchId=228)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTypeApi
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testConcurrentMetastores
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateAndGetTableWithDriver
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFunctionWithResources
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetPartitionsByNames
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetTableObjects
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartitionFilter
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRenamePartition
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleFunction
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testTableDatabase
(batchId=232)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testComplexTypeApi
(batchId=223)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testConcurrentMetastores
(batchId=223)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testCreateAndGetTableWithDriver
(batchId=223)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testFunctionWithResources
(batchId=223)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetPartitionsByNames
(batchId=223)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetTableObjects
(batchId=223)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartitionFilter
(batchId=223)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testRenamePartition
(batchId=223)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSimpleFunction
(batchId=223)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testTableDatabase
(batchId=223)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testComplexTypeApi
(batchId=221)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testConcurrentMetastores
(batchId=221)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testCreateAndGetTableWithDriver
(batchId=221)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testFunctionWithResources
(batchId=221)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testGetPartitionsByNames
(batchId=221)
org.apa
[jira] [Commented] (HIVE-21725) Break up DDLTask - extract Column and Constraint related operations
[
https://issues.apache.org/jira/browse/HIVE-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840085#comment-16840085
]
Hive QA commented on HIVE-21725:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
29s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} ql: The patch generated 0 new + 285 unchanged - 22
fixed = 285 total (was 307) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
39s{color} | {color:red} ql generated 2 new + 2257 unchanged - 1 fixed = 2259
total (was 2258) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Unwritten field:AlterTableWithWriteIdDesc.java:[line 89] |
| | Class
org.apache.hadoop.hive.ql.ddl.table.constaint.AlterTableDropConstraintDesc
defines non-transient non-serializable instance field replicationSpec In
AlterTableDropConstraintDesc.java:instance field replicationSpec In
AlterTableDropConstraintDesc.java |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17218/dev-support/hive-personality.sh
|
| git revision | master / de69a0f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17218/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17218/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Break up DDLTask - extract Column and Constraint related operations
> ---
>
> Key: HIVE-21725
> URL: https://issues.apache.org/jira/browse/HIVE-21725
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21725.01.patch, HIVE-21725.02.patch,
> HIVE-21725.03.patch, HIVE-21725.04.patch, HIVE-21725.05.patch
>
>
> DDLTask is a huge class, more than 5000 lines long. The related DDLWork is
> also a huge class, which has a field for each DDL operation it supports. The
> goal is to refactor these in order to have everything cut into more
> handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each operation
> * have a package for each operation group (database ddl, table ddl, etc), so
> the amou
[jira] [Commented] (HIVE-21729) Arrow serializer sometimes shifts timestamp by one second
[
https://issues.apache.org/jira/browse/HIVE-21729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840087#comment-16840087
]
Sankar Hariappan commented on HIVE-21729:
-
+1, pending tests
> Arrow serializer sometimes shifts timestamp by one second
> -
>
> Key: HIVE-21729
> URL: https://issues.apache.org/jira/browse/HIVE-21729
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21729.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This happens due to
> [secondInMicros|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Serializer.java#L445]
> are calculated like
> {code}
> final long secondInMicros = (secondInMillis - secondInMillis %
> MILLIS_PER_SECOND) * MICROS_PER_MILLIS;
> {code}
> Instead this should be calculated like(by taking nanos from
> timestampColumnVector itself)
> {code}
> final long nanos = timestampColumnVector.getNanos(j);
> final long secondInMicros = (secondInMillis - nanos / NS_PER_MILLIS) *
> MICROS_PER_MILLIS;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
