[jira] [Commented] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065334#comment-17065334
]
Hive QA commented on HIVE-22997:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 76 unchanged - 5
fixed = 76 total (was 81) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 649
unchanged - 1 fixed = 649 total (was 650) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} ql generated 0 new + 1530 unchanged - 1 fixed = 1530
total (was 1531) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21244/dev-support/hive-personality.sh
|
| git revision | master / dac2bcd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21244/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch,
> HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch,
> HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.17.patch,
> HIVE-22997.18.patch, HIVE-22997.19.patch, HIVE-22997.2.patch,
>
[jira] [Commented] (HIVE-22928) Allow hive.exec.stagingdir to be a fully qualified directory name
[
https://issues.apache.org/jira/browse/HIVE-22928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065315#comment-17065315
]
Hive QA commented on HIVE-22928:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997487/HIVE-22928.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18126 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.TestContext.testStagingDirOnSameFSDespiteConfiguration
(batchId=345)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21243/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21243/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21243/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997487 - PreCommit-HIVE-Build
> Allow hive.exec.stagingdir to be a fully qualified directory name
> -
>
> Key: HIVE-22928
> URL: https://issues.apache.org/jira/browse/HIVE-22928
> Project: Hive
> Issue Type: Improvement
> Components: Configuration, Hive
>Affects Versions: 3.1.2
>Reporter: Thomas Poepping
>Assignee: Thomas Poepping
>Priority: Minor
> Attachments: HIVE-22928.2.patch, HIVE-22928.patch
>
>
> Currently, {{hive.exec.stagingdir}} can only be set as a relative directory
> name that, for operations like {{insert}} or {{insert overwrite}}, will be
> placed either under the table directory or the partition directory.
> For cases where an HDFS cluster is small but the data being inserted is very
> large (greater than the capacity of the HDFS cluster, as mentioned in a
> comment by [~ashutoshc] on [HIVE-14270]), the client may want to set their
> staging directory to be an explicit blobstore path (or any filesystem path),
> rather than relying on Hive to intelligently build the blobstore path based
> on an interpretation of the job. We may lose locality guarantees, but because
> renames are just as expensive on blobstores no matter what the prefix is,
> this isn't considered a terribly large loss (assuming only blobstore
> customers use this functionality).
> Note that {{hive.blobstore.use.blobstore.as.scratchdir}} doesn't actually
> suffice in this case, as the stagingdir is not the same.
> This commit enables Hive customers to set an absolute location for all
> staging directories. For instances where the configured stagingdir scheme is
> not the same as the scheme for the table location, the default stagingdir
> configuration is used. This avoids a cross-filesystem rename, which is
> impossible anyway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22995) Add support for location for managed tables on database
[
https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-22995:
-
Status: Patch Available (was: Open)
> Add support for location for managed tables on database
> ---
>
> Key: HIVE-22995
> URL: https://issues.apache.org/jira/browse/HIVE-22995
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22995.1.patch, HIVE-22995.2.patch,
> HIVE-22995.3.patch, HIVE-22995.4.patch, HIVE-22995.5.patch,
> HIVE-22995.6.patch, HIVE-22995.7.patch, HIVE-22995.8.patch,
> HIVE-22995.9.patch, Hive Metastore Support for Tenant-based storage
> heirarchy.pdf
>
>
> I have attached the initial spec to this jira.
> Default location for database would be the external table base directory.
> Managed location can be optionally specified.
> {code}
> CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
> [COMMENT database_comment]
> [LOCATION hdfs_path]
> [MANAGEDLOCATION hdfs_path]
> [WITH DBPROPERTIES (property_name=property_value, ...)];
> ALTER (DATABASE|SCHEMA) database_name SET
> MANAGEDLOCATION
> hdfs_path;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22995) Add support for location for managed tables on database
[
https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-22995:
-
Attachment: HIVE-22995.9.patch
> Add support for location for managed tables on database
> ---
>
> Key: HIVE-22995
> URL: https://issues.apache.org/jira/browse/HIVE-22995
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22995.1.patch, HIVE-22995.2.patch,
> HIVE-22995.3.patch, HIVE-22995.4.patch, HIVE-22995.5.patch,
> HIVE-22995.6.patch, HIVE-22995.7.patch, HIVE-22995.8.patch,
> HIVE-22995.9.patch, Hive Metastore Support for Tenant-based storage
> heirarchy.pdf
>
>
> I have attached the initial spec to this jira.
> Default location for database would be the external table base directory.
> Managed location can be optionally specified.
> {code}
> CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
> [COMMENT database_comment]
> [LOCATION hdfs_path]
> [MANAGEDLOCATION hdfs_path]
> [WITH DBPROPERTIES (property_name=property_value, ...)];
> ALTER (DATABASE|SCHEMA) database_name SET
> MANAGEDLOCATION
> hdfs_path;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22995) Add support for location for managed tables on database
[
https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-22995:
-
Status: Open (was: Patch Available)
> Add support for location for managed tables on database
> ---
>
> Key: HIVE-22995
> URL: https://issues.apache.org/jira/browse/HIVE-22995
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22995.1.patch, HIVE-22995.2.patch,
> HIVE-22995.3.patch, HIVE-22995.4.patch, HIVE-22995.5.patch,
> HIVE-22995.6.patch, HIVE-22995.7.patch, HIVE-22995.8.patch, Hive Metastore
> Support for Tenant-based storage heirarchy.pdf
>
>
> I have attached the initial spec to this jira.
> Default location for database would be the external table base directory.
> Managed location can be optionally specified.
> {code}
> CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
> [COMMENT database_comment]
> [LOCATION hdfs_path]
> [MANAGEDLOCATION hdfs_path]
> [WITH DBPROPERTIES (property_name=property_value, ...)];
> ALTER (DATABASE|SCHEMA) database_name SET
> MANAGEDLOCATION
> hdfs_path;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22928) Allow hive.exec.stagingdir to be a fully qualified directory name
[
https://issues.apache.org/jira/browse/HIVE-22928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065300#comment-17065300
]
Hive QA commented on HIVE-22928:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 24 new + 83 unchanged - 0
fixed = 107 total (was 83) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21243/dev-support/hive-personality.sh
|
| git revision | master / dac2bcd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21243/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21243/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Allow hive.exec.stagingdir to be a fully qualified directory name
> -
>
> Key: HIVE-22928
> URL: https://issues.apache.org/jira/browse/HIVE-22928
> Project: Hive
> Issue Type: Improvement
> Components: Configuration, Hive
>Affects Versions: 3.1.2
>Reporter: Thomas Poepping
>Assignee: Thomas Poepping
>Priority: Minor
> Attachments: HIVE-22928.2.patch, HIVE-22928.patch
>
>
> Currently, {{hive.exec.stagingdir}} can only be set as a relative directory
> name that, for operations like {{insert}} or {{insert overwrite}}, will be
> placed either under the table directory or the partition directory.
> For cases where an HDFS cluster is small but the data being inserted is very
> large (greater than the capacity of the HDFS cluster, as mentioned in a
> comment by [~ashutoshc] on [HIVE-14270]), the client may want to set their
> staging directory to be an explicit blobstore path (or any filesystem path),
> rather than relying on Hive to intelligently build the blobstore path based
> on an interpretation of the job. We may lose locality guarantees, but because
> renames are just as expensive on blobstores no matter what the prefix is,
> this isn't considered a terribly large loss (assuming only blobstore
> customers use this functionality).
> Note that {{hive.blobstore.use.blobstore.as.scratchdir}} doesn't actually
> suffice in this case, as the stagingdir is not the
[jira] [Commented] (HIVE-23030) Enable sketch union-s to be rolled up
[
https://issues.apache.org/jira/browse/HIVE-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065286#comment-17065286
]
Hive QA commented on HIVE-23030:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997484/HIVE-23030.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18126 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_reducers_power_two]
(batchId=15)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21242/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21242/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21242/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997484 - PreCommit-HIVE-Build
> Enable sketch union-s to be rolled up
> -
>
> Key: HIVE-23030
> URL: https://issues.apache.org/jira/browse/HIVE-23030
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23030.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Enabling rolling up sketch aggregates could enable the matching of
> materialized views created for higher dimensions to be applied for lower
> dimension cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-22997: -- Attachment: (was: HIVE-22997.24.patch) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch, > HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch, > HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.17.patch, > HIVE-22997.18.patch, HIVE-22997.19.patch, HIVE-22997.2.patch, > HIVE-22997.20.patch, HIVE-22997.21.patch, HIVE-22997.22.patch, > HIVE-22997.23.patch, HIVE-22997.24.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, > HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 9h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-22997: -- Attachment: HIVE-22997.24.patch > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch, > HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch, > HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.17.patch, > HIVE-22997.18.patch, HIVE-22997.19.patch, HIVE-22997.2.patch, > HIVE-22997.20.patch, HIVE-22997.21.patch, HIVE-22997.22.patch, > HIVE-22997.23.patch, HIVE-22997.24.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, > HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 9h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23030) Enable sketch union-s to be rolled up
[
https://issues.apache.org/jira/browse/HIVE-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065260#comment-17065260
]
Hive QA commented on HIVE-23030:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 147 unchanged - 0
fixed = 149 total (was 147) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21242/dev-support/hive-personality.sh
|
| git revision | master / dac2bcd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21242/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21242/yetus/whitespace-eol.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21242/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enable sketch union-s to be rolled up
> -
>
> Key: HIVE-23030
> URL: https://issues.apache.org/jira/browse/HIVE-23030
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23030.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Enabling rolling up sketch aggregates could enable the matching of
> materialized views created for higher dimensions to be applied for lower
> dimension cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22262) Aggregate pushdown through join may generate additional rewriting opportunities
[ https://issues.apache.org/jira/browse/HIVE-22262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065259#comment-17065259 ] Vineet Garg commented on HIVE-22262: Of course! I don't know why I was thinking that count(*) could be computed over already grouped set of data > Aggregate pushdown through join may generate additional rewriting > opportunities > --- > > Key: HIVE-22262 > URL: https://issues.apache.org/jira/browse/HIVE-22262 > Project: Hive > Issue Type: Sub-task > Components: CBO, Materialized views >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Attachments: eager-v2.sql > > > In this case, there is a function used in the query and materialized view, > but the aggregate is not being pushed down. Script is attached. > Example query and materialized view: > create materialized view av1 stored as orc as select fk1, fk2, fk3, > to_date(fk4), sum(1) from fact group by 1, 2, 3, 4; > explain cbo select pk1, dim2.fk4, sum(1), count(c1) > from fact, dim2 > where to_date(fact.fk4) = dim2.fk4 > group by 1, 2 > order by 1, 2; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22262) Aggregate pushdown through join may generate additional rewriting opportunities
[ https://issues.apache.org/jira/browse/HIVE-22262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065257#comment-17065257 ] Jesus Camacho Rodriguez commented on HIVE-22262: The count in the query is computing the number of rows in each group; without storing the count per group in the MV, we will not be able to compute the former (in fact, the rollup in the rewriting would be done with a sum). > Aggregate pushdown through join may generate additional rewriting > opportunities > --- > > Key: HIVE-22262 > URL: https://issues.apache.org/jira/browse/HIVE-22262 > Project: Hive > Issue Type: Sub-task > Components: CBO, Materialized views >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Attachments: eager-v2.sql > > > In this case, there is a function used in the query and materialized view, > but the aggregate is not being pushed down. Script is attached. > Example query and materialized view: > create materialized view av1 stored as orc as select fk1, fk2, fk3, > to_date(fk4), sum(1) from fact group by 1, 2, 3, 4; > explain cbo select pk1, dim2.fk4, sum(1), count(c1) > from fact, dim2 > where to_date(fact.fk4) = dim2.fk4 > group by 1, 2 > order by 1, 2; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22262) Aggregate pushdown through join may generate additional rewriting opportunities
[ https://issues.apache.org/jira/browse/HIVE-22262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065249#comment-17065249 ] Vineet Garg commented on HIVE-22262: [~jcamachorodriguez] My thinking was that while introducing aggregate on top of view we keep this count beside adding necessary aggregations like sum(fk3). Since this count doesn' have any aggregate column it should be logically equivalent. What do you think? > Aggregate pushdown through join may generate additional rewriting > opportunities > --- > > Key: HIVE-22262 > URL: https://issues.apache.org/jira/browse/HIVE-22262 > Project: Hive > Issue Type: Sub-task > Components: CBO, Materialized views >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Attachments: eager-v2.sql > > > In this case, there is a function used in the query and materialized view, > but the aggregate is not being pushed down. Script is attached. > Example query and materialized view: > create materialized view av1 stored as orc as select fk1, fk2, fk3, > to_date(fk4), sum(1) from fact group by 1, 2, 3, 4; > explain cbo select pk1, dim2.fk4, sum(1), count(c1) > from fact, dim2 > where to_date(fact.fk4) = dim2.fk4 > group by 1, 2 > order by 1, 2; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065243#comment-17065243
]
Hive QA commented on HIVE-22997:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997476/HIVE-22997.23.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 18132 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.retryIncBootstrapAcidFromDifferentDumpWithoutCleanTablesConfig
(batchId=259)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testAcidTablesBootstrapDuringIncremental
(batchId=259)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testAcidTablesBootstrapDuringIncrementalWithOpenTxnsTimeout
(batchId=259)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
(batchId=259)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testRetryAcidTablesBootstrapFromDifferentDump
(batchId=259)
org.apache.hadoop.hive.ql.parse.TestReplTableMigrationWithJsonFormat.testMigrationWithUpgrade
(batchId=275)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.retryIncBootstrapAcidFromDifferentDumpWithoutCleanTablesConfig
(batchId=257)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testAcidTablesBootstrapDuringIncremental
(batchId=257)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testAcidTablesBootstrapDuringIncrementalWithOpenTxnsTimeout
(batchId=257)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
(batchId=257)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testRetryAcidTablesBootstrapFromDifferentDump
(batchId=257)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testMigrationWithUpgrade
(batchId=263)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testBasicReplaceReplPolicy
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testBootstrapAcidTablesIncrementalPhaseWithIncludeAndExcludeList
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testCaseInSensitiveNatureOfReplPolicy
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testRenameTableScenariosAcidTable
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testRenameTableScenariosBasic
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testRenameTableScenariosExternalTable
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testRenameTableScenariosUpgrade
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testRenameTableScenariosWithDmlOperations
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testRenameTableScenariosWithReplacePolicyDMLOperattion
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testReplacePolicyOnBootstrapAcidTablesIncrementalPhase
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testReplacePolicyOnBootstrapExternalTablesIncrementalPhase
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testReplacePolicyWhenAcidTablesDisabledForRepl
(batchId=264)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21241/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21241/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21241/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997476 - PreCommit-HIVE-Build
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch,
>
[jira] [Comment Edited] (HIVE-22098) Data loss occurs when multiple tables are join with different bucket_version
[
https://issues.apache.org/jira/browse/HIVE-22098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065237#comment-17065237
]
David Mollitor edited comment on HIVE-22098 at 3/24/20, 1:51 AM:
-
[~jithendhir92] Sorry for any confusion.
I am not saying that this issue cannot be reproduced by using the method you
have proposed. I am saying that this issue is all the more important because
there are MULTIPLE ways to get the table into this state.
was (Author: belugabehr):
[~jithendhir92] Sorry for any confusion.
I am not saying that this issue cannot be reproduced by using the method you
have propose. I am saying that this issue is all the more important because
there are MULTIPLE ways to get the table into this state.
> Data loss occurs when multiple tables are join with different bucket_version
>
>
> Key: HIVE-22098
> URL: https://issues.apache.org/jira/browse/HIVE-22098
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Affects Versions: 3.1.0, 3.1.2
>Reporter: LuGuangMing
>Assignee: LuGuangMing
>Priority: Blocker
> Labels: data-loss, wrongresults
> Attachments: HIVE-22098.1.patch, image-2019-08-12-18-45-15-771.png,
> join_test.sql, table_a_data.orc, table_b_data.orc, table_c_data.orc
>
>
> When different bucketVersion of tables do join and no of reducers is greater
> than 2, the result is incorrect (*data loss*).
> *Scenario 1*: Three tables join. The temporary result data of table_a in the
> first table and table_b in the second table joins result is recorded as
> tmp_a_b, When it joins with the third table, the bucket_version=2 of the
> table created by default after hive-3.0.0, temporary data tmp_a_b initialized
> the bucketVerison=-1, and then ReduceSinkOperator Verketison=-1 is joined. In
> the init method, the hash algorithm of selecting join column is selected
> according to bucketVersion. If bucketVersion = 2 and is not an acid
> operation, it will acquired the new algorithm of hash. Otherwise, the old
> algorithm of hash is acquired. Because of the inconsistency of the algorithm
> of hash, the partition of data allocation caused are different. At stage of
> Reducer, Data with the same key can not be paired resulting in data loss.
> *Scenario 2*: create two test tables, create table
> table_bucketversion_1(col_1 string, col_2 string) TBLPROPERTIES
> ('bucketing_version'='1'); table_bucketversion_2(col_1 string, col_2 string)
> TBLPROPERTIES ('bucketing_version'='2');
> when use table_bucketversion_1 to join table_bucketversion_2, partial result
> data will be loss due to bucketVerison is different.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22098) Data loss occurs when multiple tables are join with different bucket_version
[
https://issues.apache.org/jira/browse/HIVE-22098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065237#comment-17065237
]
David Mollitor commented on HIVE-22098:
---
[~jithendhir92] Sorry for any confusion.
I am not saying that this issue cannot be reproduced by using the method you
have propose. I am saying that this issue is all the more important because
there are MULTIPLE ways to get the table into this state.
> Data loss occurs when multiple tables are join with different bucket_version
>
>
> Key: HIVE-22098
> URL: https://issues.apache.org/jira/browse/HIVE-22098
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Affects Versions: 3.1.0, 3.1.2
>Reporter: LuGuangMing
>Assignee: LuGuangMing
>Priority: Blocker
> Labels: data-loss, wrongresults
> Attachments: HIVE-22098.1.patch, image-2019-08-12-18-45-15-771.png,
> join_test.sql, table_a_data.orc, table_b_data.orc, table_c_data.orc
>
>
> When different bucketVersion of tables do join and no of reducers is greater
> than 2, the result is incorrect (*data loss*).
> *Scenario 1*: Three tables join. The temporary result data of table_a in the
> first table and table_b in the second table joins result is recorded as
> tmp_a_b, When it joins with the third table, the bucket_version=2 of the
> table created by default after hive-3.0.0, temporary data tmp_a_b initialized
> the bucketVerison=-1, and then ReduceSinkOperator Verketison=-1 is joined. In
> the init method, the hash algorithm of selecting join column is selected
> according to bucketVersion. If bucketVersion = 2 and is not an acid
> operation, it will acquired the new algorithm of hash. Otherwise, the old
> algorithm of hash is acquired. Because of the inconsistency of the algorithm
> of hash, the partition of data allocation caused are different. At stage of
> Reducer, Data with the same key can not be paired resulting in data loss.
> *Scenario 2*: create two test tables, create table
> table_bucketversion_1(col_1 string, col_2 string) TBLPROPERTIES
> ('bucketing_version'='1'); table_bucketversion_2(col_1 string, col_2 string)
> TBLPROPERTIES ('bucketing_version'='2');
> when use table_bucketversion_1 to join table_bucketversion_2, partial result
> data will be loss due to bucketVerison is different.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-22997: -- Attachment: HIVE-22997.24.patch > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch, > HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch, > HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.17.patch, > HIVE-22997.18.patch, HIVE-22997.19.patch, HIVE-22997.2.patch, > HIVE-22997.20.patch, HIVE-22997.21.patch, HIVE-22997.22.patch, > HIVE-22997.23.patch, HIVE-22997.24.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch, > HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 9h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22262) Aggregate pushdown through join may generate additional rewriting opportunities
[
https://issues.apache.org/jira/browse/HIVE-22262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065232#comment-17065232
]
Jesus Camacho Rodriguez commented on HIVE-22262:
{quote}
More specific repro
{quote}
Can you compute the {{count}} in the query if you do not store it in the MV?
> Aggregate pushdown through join may generate additional rewriting
> opportunities
> ---
>
> Key: HIVE-22262
> URL: https://issues.apache.org/jira/browse/HIVE-22262
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: eager-v2.sql
>
>
> In this case, there is a function used in the query and materialized view,
> but the aggregate is not being pushed down. Script is attached.
> Example query and materialized view:
> create materialized view av1 stored as orc as select fk1, fk2, fk3,
> to_date(fk4), sum(1) from fact group by 1, 2, 3, 4;
> explain cbo select pk1, dim2.fk4, sum(1), count(c1)
> from fact, dim2
> where to_date(fact.fk4) = dim2.fk4
> group by 1, 2
> order by 1, 2;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23068) Error when submitting fragment to LLAP via external client: IllegalStateException: Only a single registration allowed per entity
[
https://issues.apache.org/jira/browse/HIVE-23068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-23068:
--
Attachment: HIVE-23068.1.patch
> Error when submitting fragment to LLAP via external client:
> IllegalStateException: Only a single registration allowed per entity
>
>
> Key: HIVE-23068
> URL: https://issues.apache.org/jira/browse/HIVE-23068
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-23068.1.patch
>
>
> LLAP external client (via hive-warehouse-connector) somehow seems to be
> sending duplicate submissions for the same fragment/attempt. When the 2nd
> request is sent this results in the following error:
> {noformat}
> 2020-03-17T06:49:11,239 WARN [IPC Server handler 2 on 15001 ()]
> org.apache.hadoop.ipc.Server: IPC Server handler 2 on 15001, call Call#75
> Retry#0
> org.apache.hadoop.hive.llap.protocol.LlapProtocolBlockingPB.submitWork from
> 19.40.252.114:33906
> java.lang.IllegalStateException: Only a single registration allowed per
> entity. Duplicate for
> TaskWrapper{task=attempt_1854104024183112753_6052_0_00_000128_1,
> inWaitQueue=true, inPreemptionQueue=false, registeredForNotifications=true,
> canFinish=true, canFinish(in queue)=true, isGuaranteed=false,
> firstAttemptStartTime=1584442003327, dagStartTime=1584442003327,
> withinDagPriority=0, vertexParallelism= 2132, selfAndUpstreamParallelism=
> 2132, selfAndUpstreamComplete= 0}
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo$FinishableStateTracker.registerForUpdates(QueryInfo.java:233)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo.registerForFinishableStateUpdates(QueryInfo.java:205)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryFragmentInfo.registerForFinishableStateUpdates(QueryFragmentInfo.java:160)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$TaskWrapper.maybeRegisterForFinishedStateNotifications(TaskExecutorService.java:1167)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.schedule(TaskExecutorService.java:564)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.schedule(TaskExecutorService.java:93)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl.submitWork(ContainerRunnerImpl.java:292)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.submitWork(LlapDaemon.java:610)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapProtocolServerImpl.submitWork(LlapProtocolServerImpl.java:122)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$LlapDaemonProtocol$2.callBlockingMethod(LlapDaemonProtocolProtos.java:22695)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_191]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_191]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> {noformat}
> I think the issue here is that this error occurred too late - based on the
> stack trace, LLAP has already accepted/registered the fragment. The
> subsequent cleanup of this fragment/attempt also affects the first request.
> Which results in the LLAP crash described in HIVE-23061:
> {noformat}
> 2020-03-17T06:49:11,304 ERROR [ExecutionCompletionThread #0 ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread
> Thread[ExecutionCompletionThread
[jira] [Commented] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065223#comment-17065223
]
Hive QA commented on HIVE-22997:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 1 new + 76 unchanged - 5 fixed
= 77 total (was 81) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
26s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 649
unchanged - 1 fixed = 651 total (was 650) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
2s{color} | {color:red} ql generated 1 new + 1530 unchanged - 1 fixed = 1531
total (was 1531) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to extTableCopyWorks in
org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(Path,
DumpMetaData, Path, Hive) At
ReplDumpTask.java:org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(Path,
DumpMetaData, Path, Hive) At ReplDumpTask.java:[line 457] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21241/dev-support/hive-personality.sh
|
| git revision | master / dac2bcd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21241/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21241/yetus/diff-checkstyle-itests_hive-unit.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21241/yetus/new-findbugs-ql.html
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21241/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type:
[jira] [Commented] (HIVE-22760) Add Clock caching eviction based strategy
[
https://issues.apache.org/jira/browse/HIVE-22760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065213#comment-17065213
]
Hive QA commented on HIVE-22760:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997474/HIVE-22760.3.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18125 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query3]
(batchId=306)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21240/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21240/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21240/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997474 - PreCommit-HIVE-Build
> Add Clock caching eviction based strategy
> -
>
> Key: HIVE-22760
> URL: https://issues.apache.org/jira/browse/HIVE-22760
> Project: Hive
> Issue Type: New Feature
> Components: llap
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22760.2.patch, HIVE-22760.3.patch,
> HIVE-22760.patch, HIVE-22760.patch
>
>
> LRFU is the current default right now.
> The main issue with such Strategy is that it has a very high memory overhead,
> in addition to that, most of the accounting has to happen under locks thus
> can be source of contentions.
> Add Simpler policy like clock, can help with both issues.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22262) Aggregate pushdown through join may generate additional rewriting opportunities
[ https://issues.apache.org/jira/browse/HIVE-22262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065207#comment-17065207 ] Vineet Garg commented on HIVE-22262: Once we have improvement for the above, AggregateJoinTransposeRule should help us trigger more rewritings > Aggregate pushdown through join may generate additional rewriting > opportunities > --- > > Key: HIVE-22262 > URL: https://issues.apache.org/jira/browse/HIVE-22262 > Project: Hive > Issue Type: Sub-task > Components: CBO, Materialized views >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Attachments: eager-v2.sql > > > In this case, there is a function used in the query and materialized view, > but the aggregate is not being pushed down. Script is attached. > Example query and materialized view: > create materialized view av1 stored as orc as select fk1, fk2, fk3, > to_date(fk4), sum(1) from fact group by 1, 2, 3, 4; > explain cbo select pk1, dim2.fk4, sum(1), count(c1) > from fact, dim2 > where to_date(fact.fk4) = dim2.fk4 > group by 1, 2 > order by 1, 2; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22262) Aggregate pushdown through join may generate additional rewriting opportunities
[
https://issues.apache.org/jira/browse/HIVE-22262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065206#comment-17065206
]
Vineet Garg commented on HIVE-22262:
Unfortunately rewriting still isn't triggered for above query.
AggregateJoinTransposeRule does group by pushdown (along with eager count) and
adds an extra count() on both side generating the following plan
{noformat}
HiveAggregate(group=[{3, 4}], agg#0=[sum($8)], agg#1=[$SUM0($9)])
HiveProject(TO_DATE=[$0], $f1=[$1], count=[$2], pk1=[$3], fk4=[$4],
CAST=[$5], count0=[$6], $f4=[$7], $f8=[*($1, $6)], $f9=[*($2, $7)])
HiveJoin(condition=[=($0, $5)], joinType=[inner], algorithm=[none],
cost=[not available])
HiveAggregate(group=[{1}], agg#0=[sum($0)], count=[count()])
HiveProject(subset=[rel#500:Subset#2.HIVE.[]], fk3=[$2],
TO_DATE=[TO_DATE($3)])
HiveFilter(subset=[rel#498:Subset#1.HIVE.[]], condition=[IS NOT
NULL(TO_DATE($3))])
HiveTableScan(subset=[rel#496:Subset#0.HIVE.[]], table=[[default,
fact]], table:alias=[fact])
HiveAggregate(group=[{0, 1, 3}], count=[count()], agg#1=[count($2)])
HiveProject(subset=[rel#505:Subset#5.HIVE.[]], pk1=[$0], fk4=[$1],
c1=[$2], CAST=[CAST($1):DATE])
HiveFilter(subset=[rel#503:Subset#4.HIVE.[]], condition=[IS NOT
NULL(CAST($1):DATE)])
HiveTableScan(subset=[rel#501:Subset#3.HIVE.[]], table=[[default,
dim2]], table:alias=[dim2]) {noformat}
Currently MV rewriting isn't capable of rewriting if there is an aggregate on
query which isn't in view
More specific repro
{code:sql}
create materialized view av3 stored as orc as select fk1, fk2, fk3, fk4,
sum(fk3) from fact group by fk1,fk2,fk3,fk4;
explain cbo select sum(fk3), count(*) from fact group by fk4;
{code}
> Aggregate pushdown through join may generate additional rewriting
> opportunities
> ---
>
> Key: HIVE-22262
> URL: https://issues.apache.org/jira/browse/HIVE-22262
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: eager-v2.sql
>
>
> In this case, there is a function used in the query and materialized view,
> but the aggregate is not being pushed down. Script is attached.
> Example query and materialized view:
> create materialized view av1 stored as orc as select fk1, fk2, fk3,
> to_date(fk4), sum(1) from fact group by 1, 2, 3, 4;
> explain cbo select pk1, dim2.fk4, sum(1), count(c1)
> from fact, dim2
> where to_date(fact.fk4) = dim2.fk4
> group by 1, 2
> order by 1, 2;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22262) Aggregate pushdown through join may generate additional rewriting opportunities
[
https://issues.apache.org/jira/browse/HIVE-22262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065205#comment-17065205
]
Vineet Garg edited comment on HIVE-22262 at 3/24/20, 12:10 AM:
---
The reason {{HiveAggregateProjectMergeRule}} isn't able to merge is because
Project is introducing a constant expression in this case.
Following query better encapsulate the case where
{{AggregateJoinTransposeRule}} should help trigger rewriting
{code:sql}
explain cbo select pk1, dim2.fk4, sum(fk3), count(c1)
from fact, dim2
where to_date(fact.fk4) = dim2.fk4
group by pk1,dim2.fk4
order by pk1,dim2.fk4;
{code}
was (Author: vgarg):
The reason {{HiveAggregateProjectMergeRule}} isn't able to merge is because
Project is introducing a constant expression in this case.
Following query better encapsulate case where {{AggregateJoinTransposeRule}}
should help trigger rewriting
{code:sql}
explain cbo select pk1, dim2.fk4, sum(fk3), count(c1)
from fact, dim2
where to_date(fact.fk4) = dim2.fk4
group by pk1,dim2.fk4
order by pk1,dim2.fk4;
{code}
> Aggregate pushdown through join may generate additional rewriting
> opportunities
> ---
>
> Key: HIVE-22262
> URL: https://issues.apache.org/jira/browse/HIVE-22262
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: eager-v2.sql
>
>
> In this case, there is a function used in the query and materialized view,
> but the aggregate is not being pushed down. Script is attached.
> Example query and materialized view:
> create materialized view av1 stored as orc as select fk1, fk2, fk3,
> to_date(fk4), sum(1) from fact group by 1, 2, 3, 4;
> explain cbo select pk1, dim2.fk4, sum(1), count(c1)
> from fact, dim2
> where to_date(fact.fk4) = dim2.fk4
> group by 1, 2
> order by 1, 2;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22262) Aggregate pushdown through join may generate additional rewriting opportunities
[
https://issues.apache.org/jira/browse/HIVE-22262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065205#comment-17065205
]
Vineet Garg commented on HIVE-22262:
The reason {{HiveAggregateProjectMergeRule}} isn't able to merge is because
Project is introducing a constant expression in this case.
Following query better encapsulate case where {{AggregateJoinTransposeRule}}
should help trigger rewriting
{code:sql}
explain cbo select pk1, dim2.fk4, sum(fk3), count(c1)
from fact, dim2
where to_date(fact.fk4) = dim2.fk4
group by pk1,dim2.fk4
order by pk1,dim2.fk4;
{code}
> Aggregate pushdown through join may generate additional rewriting
> opportunities
> ---
>
> Key: HIVE-22262
> URL: https://issues.apache.org/jira/browse/HIVE-22262
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: eager-v2.sql
>
>
> In this case, there is a function used in the query and materialized view,
> but the aggregate is not being pushed down. Script is attached.
> Example query and materialized view:
> create materialized view av1 stored as orc as select fk1, fk2, fk3,
> to_date(fk4), sum(1) from fact group by 1, 2, 3, 4;
> explain cbo select pk1, dim2.fk4, sum(1), count(c1)
> from fact, dim2
> where to_date(fact.fk4) = dim2.fk4
> group by 1, 2
> order by 1, 2;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22760) Add Clock caching eviction based strategy
[
https://issues.apache.org/jira/browse/HIVE-22760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065189#comment-17065189
]
Hive QA commented on HIVE-22760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} storage-api in master has 51 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} llap-client in master has 27 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} storage-api: The patch generated 1 new + 1 unchanged -
2 fixed = 2 total (was 3) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} common: The patch generated 1 new + 374 unchanged - 0
fixed = 375 total (was 374) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} llap-client: The patch generated 2 new + 2 unchanged -
3 fixed = 4 total (was 5) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} llap-server: The patch generated 36 new + 133
unchanged - 24 fixed = 169 total (was 157) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} storage-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} llap-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} llap-server generated 0 new + 87 unchanged - 3 fixed
= 87 total (was 90) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} llap-client generated 1 new + 6 unchanged - 0 fixed =
7 total (was 6) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
|
[jira] [Commented] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065176#comment-17065176
]
Hive QA commented on HIVE-23052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997465/HIVE-23052.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 186 failed/errored test(s), 18125 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_insert_overwrite]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats2] (batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_subquery]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_directories_test]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_transactional_full_acid]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_transactional_insert_only]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_partitioned]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_partitioned]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_whole_partition]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_locks]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_acid_dynamic_partition]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_dynamic_partitioned]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_buckets] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_insert_values_dynamic_partitioned]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_partitioned]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_where_partitioned]
(batchId=71)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_dynamic]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_dp]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[tez_acid_union_dynamic_partition]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[tez_acid_union_dynamic_partition_2]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=187)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ctas]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_all_partitioned]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_where_partitioned]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_whole_partition]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_bucketing]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_overwrite]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_dynamic_partitioned]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_data_using_job]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create]
(batchId=187)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mm_exim]
(batchId=194)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[murmur_hash_migration]
(batchId=184)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sample10_mm]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part_update]
(batchId=187)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part_update_llap_io]
[jira] [Commented] (HIVE-23068) Error when submitting fragment to LLAP via external client: IllegalStateException: Only a single registration allowed per entity
[
https://issues.apache.org/jira/browse/HIVE-23068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065170#comment-17065170
]
Jason Dere commented on HIVE-23068:
---
Ideally the duplicate fragment/attempt request should fail at a point where it
does not have to invoke fragmentComplete()/unregisterFragment(), which may have
bad effects on the first request that was already submitted.
> Error when submitting fragment to LLAP via external client:
> IllegalStateException: Only a single registration allowed per entity
>
>
> Key: HIVE-23068
> URL: https://issues.apache.org/jira/browse/HIVE-23068
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
>
> LLAP external client (via hive-warehouse-connector) somehow seems to be
> sending duplicate submissions for the same fragment/attempt. When the 2nd
> request is sent this results in the following error:
> {noformat}
> 2020-03-17T06:49:11,239 WARN [IPC Server handler 2 on 15001 ()]
> org.apache.hadoop.ipc.Server: IPC Server handler 2 on 15001, call Call#75
> Retry#0
> org.apache.hadoop.hive.llap.protocol.LlapProtocolBlockingPB.submitWork from
> 19.40.252.114:33906
> java.lang.IllegalStateException: Only a single registration allowed per
> entity. Duplicate for
> TaskWrapper{task=attempt_1854104024183112753_6052_0_00_000128_1,
> inWaitQueue=true, inPreemptionQueue=false, registeredForNotifications=true,
> canFinish=true, canFinish(in queue)=true, isGuaranteed=false,
> firstAttemptStartTime=1584442003327, dagStartTime=1584442003327,
> withinDagPriority=0, vertexParallelism= 2132, selfAndUpstreamParallelism=
> 2132, selfAndUpstreamComplete= 0}
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo$FinishableStateTracker.registerForUpdates(QueryInfo.java:233)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo.registerForFinishableStateUpdates(QueryInfo.java:205)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryFragmentInfo.registerForFinishableStateUpdates(QueryFragmentInfo.java:160)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$TaskWrapper.maybeRegisterForFinishedStateNotifications(TaskExecutorService.java:1167)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.schedule(TaskExecutorService.java:564)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.schedule(TaskExecutorService.java:93)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl.submitWork(ContainerRunnerImpl.java:292)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.submitWork(LlapDaemon.java:610)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapProtocolServerImpl.submitWork(LlapProtocolServerImpl.java:122)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$LlapDaemonProtocol$2.callBlockingMethod(LlapDaemonProtocolProtos.java:22695)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_191]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_191]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> {noformat}
> I think the issue here is that this error occurred too late - based on the
> stack trace, LLAP has already accepted/registered the fragment. The
> subsequent cleanup of this fragment/attempt also affects the first request.
> Which results in the LLAP crash described in
[jira] [Assigned] (HIVE-23068) Error when submitting fragment to LLAP via external client: IllegalStateException: Only a single registration allowed per entity
[
https://issues.apache.org/jira/browse/HIVE-23068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere reassigned HIVE-23068:
-
> Error when submitting fragment to LLAP via external client:
> IllegalStateException: Only a single registration allowed per entity
>
>
> Key: HIVE-23068
> URL: https://issues.apache.org/jira/browse/HIVE-23068
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
>
> LLAP external client (via hive-warehouse-connector) somehow seems to be
> sending duplicate submissions for the same fragment/attempt. When the 2nd
> request is sent this results in the following error:
> {noformat}
> 2020-03-17T06:49:11,239 WARN [IPC Server handler 2 on 15001 ()]
> org.apache.hadoop.ipc.Server: IPC Server handler 2 on 15001, call Call#75
> Retry#0
> org.apache.hadoop.hive.llap.protocol.LlapProtocolBlockingPB.submitWork from
> 19.40.252.114:33906
> java.lang.IllegalStateException: Only a single registration allowed per
> entity. Duplicate for
> TaskWrapper{task=attempt_1854104024183112753_6052_0_00_000128_1,
> inWaitQueue=true, inPreemptionQueue=false, registeredForNotifications=true,
> canFinish=true, canFinish(in queue)=true, isGuaranteed=false,
> firstAttemptStartTime=1584442003327, dagStartTime=1584442003327,
> withinDagPriority=0, vertexParallelism= 2132, selfAndUpstreamParallelism=
> 2132, selfAndUpstreamComplete= 0}
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo$FinishableStateTracker.registerForUpdates(QueryInfo.java:233)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo.registerForFinishableStateUpdates(QueryInfo.java:205)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryFragmentInfo.registerForFinishableStateUpdates(QueryFragmentInfo.java:160)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$TaskWrapper.maybeRegisterForFinishedStateNotifications(TaskExecutorService.java:1167)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.schedule(TaskExecutorService.java:564)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.schedule(TaskExecutorService.java:93)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl.submitWork(ContainerRunnerImpl.java:292)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.submitWork(LlapDaemon.java:610)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapProtocolServerImpl.submitWork(LlapProtocolServerImpl.java:122)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$LlapDaemonProtocol$2.callBlockingMethod(LlapDaemonProtocolProtos.java:22695)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_191]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_191]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> ~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
> {noformat}
> I think the issue here is that this error occurred too late - based on the
> stack trace, LLAP has already accepted/registered the fragment. The
> subsequent cleanup of this fragment/attempt also affects the first request.
> Which results in the LLAP crash described in HIVE-23061:
> {noformat}
> 2020-03-17T06:49:11,304 ERROR [ExecutionCompletionThread #0 ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread
> Thread[ExecutionCompletionThread #0,5,main] threw an Exception. Shutting down
> now...
>
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[
https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=408360=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408360
]
ASF GitHub Bot logged work on HIVE-22821:
-
Author: ASF GitHub Bot
Created on: 23/Mar/20 22:23
Start Date: 23/Mar/20 22:23
Worklog Time Spent: 10m
Work Description: szlta commented on pull request #909: HIVE-22821
URL: https://github.com/apache/hive/pull/909#discussion_r396792555
##
File path: ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java
##
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import javax.net.SocketFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.io.retry.RetryPolicies;
+import org.apache.hadoop.io.retry.RetryPolicy;
+import org.apache.hadoop.net.NetUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Through this class the caller (typically HS2) can request eviction of
buffers from LLAP cache by specifying a DB,
+ * table or partition name/(value). Request sending is implemented here.
+ */
+public final class ProactiveEviction {
+
+ private ProactiveEviction() {
+// Not to be used;
+ }
+
+ /**
+ * Trigger LLAP cache eviction of buffers related to entities residing in
request parameter.
+ * @param conf
+ * @param request
+ */
+ public static void evict(Configuration conf, Request request) {
+if (!HiveConf.getBoolVar(conf,
HiveConf.ConfVars.LLAP_IO_PROACTIVE_EVICTION_ENABLED)) {
+ return;
+}
+
+try {
+ LlapRegistryService llapRegistryService =
LlapRegistryService.getClient(conf);
+ Collection instances =
llapRegistryService.getInstances().getAll();
+ if (instances.size() == 0) {
+// Not in LLAP mode.
+return;
+ }
+ ExecutorService executorService = Executors.newCachedThreadPool();
Review comment:
Makes sense. I guess we can do something like this then:
https://github.com/apache/hive/pull/909/commits/4b285ef48e95f0a7b83029ab42de7a535d93d4d5
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 408360)
Time Spent: 4h 50m (was: 4h 40m)
> Add necessary endpoints for proactive cache eviction
>
>
> Key: HIVE-22821
> URL: https://issues.apache.org/jira/browse/HIVE-22821
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch,
> HIVE-22821.2.patch
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Implement the parts required for iHS2 -> LLAP daemons communication:
> * protobuf message schema and endpoints
> * Hive configuration
> * for use cases:
> ** dropping db
> ** dropping table
> ** dropping partition
[jira] [Commented] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065134#comment-17065134
]
Hive QA commented on HIVE-23052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
19s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
24s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 33 new + 533 unchanged - 37 fixed = 566 total (was 570) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21239/dev-support/hive-personality.sh
|
| git revision | master / dac2bcd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21239/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21239/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimize lock enqueueing in TxnHandler
> --
>
> Key: HIVE-23052
> URL: https://issues.apache.org/jira/browse/HIVE-23052
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23052.1.patch, HIVE-23052.2.patch,
> HIVE-23052.3.patch, HIVE-23052.4.patch, HIVE-23052.5.patch
>
>
> * Reduce scope of next_lock_id select-for-update by moving the txn_component
> inserts before the S4U + inserting the hive_locks entries before the S4U
> (first with a temp ID, which will be replaced later in a single update). This
> helps decrease the overall time that the next_lock_id table is locked,
> thereby increasing concurrency
> * Insert txn_components in a batch instead of one-by-one (also in
> TxnHandler::addDynamicPartition)
> * Increment next_lock_id and update hive_locks table in a single batch
> statement
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22875) Refactor query creation in QueryCompactor implementations
[
https://issues.apache.org/jira/browse/HIVE-22875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065123#comment-17065123
]
Hive QA commented on HIVE-22875:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997466/HIVE-22875.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 18125 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez.testMinorCompactionWhileStreaming
(batchId=270)
org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez.testMinorCompactionWhileStreamingAfterAbort
(batchId=270)
org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez.testMinorCompactionWhileStreamingWithAbort
(batchId=270)
org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez.testMinorCompactionWhileStreamingWithAbortInMiddle
(batchId=270)
org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez.testMinorCompactionWhileStreamingWithSplitUpdate
(batchId=270)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21238/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21238/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21238/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997466 - PreCommit-HIVE-Build
> Refactor query creation in QueryCompactor implementations
> -
>
> Key: HIVE-22875
> URL: https://issues.apache.org/jira/browse/HIVE-22875
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22875.01.patch, HIVE-22875.02.patch,
> HIVE-22875.02.patch, HIVE-22875.03.patch, HIVE-22875.04.patch
>
>
> There is a lot of repetition where creation/compaction/drop queries are
> created in MajorQueryCompactor, MinorQueryCompactor, MmMajorQueryCompactor
> and MmMinorQueryCompactor.
> Initial idea is to create a CompactionQueryBuilder that all 4 implementations
> would use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22928) Allow hive.exec.stagingdir to be a fully qualified directory name
[
https://issues.apache.org/jira/browse/HIVE-22928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065115#comment-17065115
]
Thomas Poepping commented on HIVE-22928:
Thanks [~dkuzmenko], I'll give that a shot!
> Allow hive.exec.stagingdir to be a fully qualified directory name
> -
>
> Key: HIVE-22928
> URL: https://issues.apache.org/jira/browse/HIVE-22928
> Project: Hive
> Issue Type: Improvement
> Components: Configuration, Hive
>Affects Versions: 3.1.2
>Reporter: Thomas Poepping
>Assignee: Thomas Poepping
>Priority: Minor
> Attachments: HIVE-22928.2.patch, HIVE-22928.patch
>
>
> Currently, {{hive.exec.stagingdir}} can only be set as a relative directory
> name that, for operations like {{insert}} or {{insert overwrite}}, will be
> placed either under the table directory or the partition directory.
> For cases where an HDFS cluster is small but the data being inserted is very
> large (greater than the capacity of the HDFS cluster, as mentioned in a
> comment by [~ashutoshc] on [HIVE-14270]), the client may want to set their
> staging directory to be an explicit blobstore path (or any filesystem path),
> rather than relying on Hive to intelligently build the blobstore path based
> on an interpretation of the job. We may lose locality guarantees, but because
> renames are just as expensive on blobstores no matter what the prefix is,
> this isn't considered a terribly large loss (assuming only blobstore
> customers use this functionality).
> Note that {{hive.blobstore.use.blobstore.as.scratchdir}} doesn't actually
> suffice in this case, as the stagingdir is not the same.
> This commit enables Hive customers to set an absolute location for all
> staging directories. For instances where the configured stagingdir scheme is
> not the same as the scheme for the table location, the default stagingdir
> configuration is used. This avoids a cross-filesystem rename, which is
> impossible anyway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22928) Allow hive.exec.stagingdir to be a fully qualified directory name
[
https://issues.apache.org/jira/browse/HIVE-22928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Poepping updated HIVE-22928:
---
Attachment: HIVE-22928.2.patch
> Allow hive.exec.stagingdir to be a fully qualified directory name
> -
>
> Key: HIVE-22928
> URL: https://issues.apache.org/jira/browse/HIVE-22928
> Project: Hive
> Issue Type: Improvement
> Components: Configuration, Hive
>Affects Versions: 3.1.2
>Reporter: Thomas Poepping
>Assignee: Thomas Poepping
>Priority: Minor
> Attachments: HIVE-22928.2.patch, HIVE-22928.patch
>
>
> Currently, {{hive.exec.stagingdir}} can only be set as a relative directory
> name that, for operations like {{insert}} or {{insert overwrite}}, will be
> placed either under the table directory or the partition directory.
> For cases where an HDFS cluster is small but the data being inserted is very
> large (greater than the capacity of the HDFS cluster, as mentioned in a
> comment by [~ashutoshc] on [HIVE-14270]), the client may want to set their
> staging directory to be an explicit blobstore path (or any filesystem path),
> rather than relying on Hive to intelligently build the blobstore path based
> on an interpretation of the job. We may lose locality guarantees, but because
> renames are just as expensive on blobstores no matter what the prefix is,
> this isn't considered a terribly large loss (assuming only blobstore
> customers use this functionality).
> Note that {{hive.blobstore.use.blobstore.as.scratchdir}} doesn't actually
> suffice in this case, as the stagingdir is not the same.
> This commit enables Hive customers to set an absolute location for all
> staging directories. For instances where the configured stagingdir scheme is
> not the same as the scheme for the table location, the default stagingdir
> configuration is used. This avoids a cross-filesystem rename, which is
> impossible anyway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22928) Allow hive.exec.stagingdir to be a fully qualified directory name
[
https://issues.apache.org/jira/browse/HIVE-22928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065109#comment-17065109
]
Denys Kuzmenko commented on HIVE-22928:
---
[~poeppt], just upload patch file with incremented version #, like
HIVE-22928.2.patch
> Allow hive.exec.stagingdir to be a fully qualified directory name
> -
>
> Key: HIVE-22928
> URL: https://issues.apache.org/jira/browse/HIVE-22928
> Project: Hive
> Issue Type: Improvement
> Components: Configuration, Hive
>Affects Versions: 3.1.2
>Reporter: Thomas Poepping
>Assignee: Thomas Poepping
>Priority: Minor
> Attachments: HIVE-22928.patch
>
>
> Currently, {{hive.exec.stagingdir}} can only be set as a relative directory
> name that, for operations like {{insert}} or {{insert overwrite}}, will be
> placed either under the table directory or the partition directory.
> For cases where an HDFS cluster is small but the data being inserted is very
> large (greater than the capacity of the HDFS cluster, as mentioned in a
> comment by [~ashutoshc] on [HIVE-14270]), the client may want to set their
> staging directory to be an explicit blobstore path (or any filesystem path),
> rather than relying on Hive to intelligently build the blobstore path based
> on an interpretation of the job. We may lose locality guarantees, but because
> renames are just as expensive on blobstores no matter what the prefix is,
> this isn't considered a terribly large loss (assuming only blobstore
> customers use this functionality).
> Note that {{hive.blobstore.use.blobstore.as.scratchdir}} doesn't actually
> suffice in this case, as the stagingdir is not the same.
> This commit enables Hive customers to set an absolute location for all
> staging directories. For instances where the configured stagingdir scheme is
> not the same as the scheme for the table location, the default stagingdir
> configuration is used. This avoids a cross-filesystem rename, which is
> impossible anyway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22262) Aggregate pushdown through join may generate additional rewriting opportunities
[
https://issues.apache.org/jira/browse/HIVE-22262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065099#comment-17065099
]
Vineet Garg commented on HIVE-22262:
[~jcamachorodriguez] This is due to {{HiveAggregateJoinTransposeRule}} not
matching because there is a Project b/w Aggregate and Join. I am trying
{{HiveAggregateProjectMergeRule}} but it still doesn't match.
> Aggregate pushdown through join may generate additional rewriting
> opportunities
> ---
>
> Key: HIVE-22262
> URL: https://issues.apache.org/jira/browse/HIVE-22262
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: eager-v2.sql
>
>
> In this case, there is a function used in the query and materialized view,
> but the aggregate is not being pushed down. Script is attached.
> Example query and materialized view:
> create materialized view av1 stored as orc as select fk1, fk2, fk3,
> to_date(fk4), sum(1) from fact group by 1, 2, 3, 4;
> explain cbo select pk1, dim2.fk4, sum(1), count(c1)
> from fact, dim2
> where to_date(fact.fk4) = dim2.fk4
> group by 1, 2
> order by 1, 2;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22262) Aggregate pushdown through join may generate additional rewriting opportunities
[
https://issues.apache.org/jira/browse/HIVE-22262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065096#comment-17065096
]
Jesus Camacho Rodriguez commented on HIVE-22262:
[~vgarg], iirc {{HiveAggregateJoinTransposeRule}} has a cost check before
triggering the transformation; you may want to check whether using the original
{{AggregateJoinTransposeRule}} makes a difference.
> Aggregate pushdown through join may generate additional rewriting
> opportunities
> ---
>
> Key: HIVE-22262
> URL: https://issues.apache.org/jira/browse/HIVE-22262
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: eager-v2.sql
>
>
> In this case, there is a function used in the query and materialized view,
> but the aggregate is not being pushed down. Script is attached.
> Example query and materialized view:
> create materialized view av1 stored as orc as select fk1, fk2, fk3,
> to_date(fk4), sum(1) from fact group by 1, 2, 3, 4;
> explain cbo select pk1, dim2.fk4, sum(1), count(c1)
> from fact, dim2
> where to_date(fact.fk4) = dim2.fk4
> group by 1, 2
> order by 1, 2;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[
https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=408310=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408310
]
ASF GitHub Bot logged work on HIVE-22821:
-
Author: ASF GitHub Bot
Created on: 23/Mar/20 21:01
Start Date: 23/Mar/20 21:01
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #909: HIVE-22821
URL: https://github.com/apache/hive/pull/909#discussion_r396744224
##
File path: ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java
##
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import javax.net.SocketFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.io.retry.RetryPolicies;
+import org.apache.hadoop.io.retry.RetryPolicy;
+import org.apache.hadoop.net.NetUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Through this class the caller (typically HS2) can request eviction of
buffers from LLAP cache by specifying a DB,
+ * table or partition name/(value). Request sending is implemented here.
+ */
+public final class ProactiveEviction {
+
+ private ProactiveEviction() {
+// Not to be used;
+ }
+
+ /**
+ * Trigger LLAP cache eviction of buffers related to entities residing in
request parameter.
+ * @param conf
+ * @param request
+ */
+ public static void evict(Configuration conf, Request request) {
+if (!HiveConf.getBoolVar(conf,
HiveConf.ConfVars.LLAP_IO_PROACTIVE_EVICTION_ENABLED)) {
+ return;
+}
+
+try {
+ LlapRegistryService llapRegistryService =
LlapRegistryService.getClient(conf);
+ Collection instances =
llapRegistryService.getInstances().getAll();
+ if (instances.size() == 0) {
+// Not in LLAP mode.
+return;
+ }
+ ExecutorService executorService = Executors.newCachedThreadPool();
Review comment:
You can check this link about[ Double checking
locking](https://wiki.sei.cmu.edu/confluence/display/java/LCK10-J.+Use+a+correct+form+of+the+double-checked+locking+idiom)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 408310)
Time Spent: 4h 40m (was: 4.5h)
> Add necessary endpoints for proactive cache eviction
>
>
> Key: HIVE-22821
> URL: https://issues.apache.org/jira/browse/HIVE-22821
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch,
> HIVE-22821.2.patch
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Implement the parts required for iHS2 -> LLAP daemons communication:
> * protobuf message schema and endpoints
> * Hive configuration
> * for use cases:
> ** dropping db
> ** dropping table
>
[jira] [Commented] (HIVE-22875) Refactor query creation in QueryCompactor implementations
[
https://issues.apache.org/jira/browse/HIVE-22875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065081#comment-17065081
]
Hive QA commented on HIVE-22875:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21238/dev-support/hive-personality.sh
|
| git revision | master / dac2bcd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21238/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Refactor query creation in QueryCompactor implementations
> -
>
> Key: HIVE-22875
> URL: https://issues.apache.org/jira/browse/HIVE-22875
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22875.01.patch, HIVE-22875.02.patch,
> HIVE-22875.02.patch, HIVE-22875.03.patch, HIVE-22875.04.patch
>
>
> There is a lot of repetition where creation/compaction/drop queries are
> created in MajorQueryCompactor, MinorQueryCompactor, MmMajorQueryCompactor
> and MmMinorQueryCompactor.
> Initial idea is to create a CompactionQueryBuilder that all 4 implementations
> would use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[
https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=408289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408289
]
ASF GitHub Bot logged work on HIVE-22821:
-
Author: ASF GitHub Bot
Created on: 23/Mar/20 20:43
Start Date: 23/Mar/20 20:43
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #909: HIVE-22821
URL: https://github.com/apache/hive/pull/909#discussion_r396744224
##
File path: ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java
##
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import javax.net.SocketFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.io.retry.RetryPolicies;
+import org.apache.hadoop.io.retry.RetryPolicy;
+import org.apache.hadoop.net.NetUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Through this class the caller (typically HS2) can request eviction of
buffers from LLAP cache by specifying a DB,
+ * table or partition name/(value). Request sending is implemented here.
+ */
+public final class ProactiveEviction {
+
+ private ProactiveEviction() {
+// Not to be used;
+ }
+
+ /**
+ * Trigger LLAP cache eviction of buffers related to entities residing in
request parameter.
+ * @param conf
+ * @param request
+ */
+ public static void evict(Configuration conf, Request request) {
+if (!HiveConf.getBoolVar(conf,
HiveConf.ConfVars.LLAP_IO_PROACTIVE_EVICTION_ENABLED)) {
+ return;
+}
+
+try {
+ LlapRegistryService llapRegistryService =
LlapRegistryService.getClient(conf);
+ Collection instances =
llapRegistryService.getInstances().getAll();
+ if (instances.size() == 0) {
+// Not in LLAP mode.
+return;
+ }
+ ExecutorService executorService = Executors.newCachedThreadPool();
Review comment:
You can check this link about Double checking locking
https://en.wikipedia.org/wiki/Double-checked_locking
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 408289)
Time Spent: 4.5h (was: 4h 20m)
> Add necessary endpoints for proactive cache eviction
>
>
> Key: HIVE-22821
> URL: https://issues.apache.org/jira/browse/HIVE-22821
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch,
> HIVE-22821.2.patch
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Implement the parts required for iHS2 -> LLAP daemons communication:
> * protobuf message schema and endpoints
> * Hive configuration
> * for use cases:
> ** dropping db
> ** dropping table
> ** dropping partition from a table
--
This message was sent
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[
https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=408287=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408287
]
ASF GitHub Bot logged work on HIVE-22821:
-
Author: ASF GitHub Bot
Created on: 23/Mar/20 20:39
Start Date: 23/Mar/20 20:39
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #909: HIVE-22821
URL: https://github.com/apache/hive/pull/909#discussion_r396742513
##
File path: ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java
##
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import javax.net.SocketFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.io.retry.RetryPolicies;
+import org.apache.hadoop.io.retry.RetryPolicy;
+import org.apache.hadoop.net.NetUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Through this class the caller (typically HS2) can request eviction of
buffers from LLAP cache by specifying a DB,
+ * table or partition name/(value). Request sending is implemented here.
+ */
+public final class ProactiveEviction {
+
+ private ProactiveEviction() {
+// Not to be used;
+ }
+
+ /**
+ * Trigger LLAP cache eviction of buffers related to entities residing in
request parameter.
+ * @param conf
+ * @param request
+ */
+ public static void evict(Configuration conf, Request request) {
+if (!HiveConf.getBoolVar(conf,
HiveConf.ConfVars.LLAP_IO_PROACTIVE_EVICTION_ENABLED)) {
+ return;
+}
+
+try {
+ LlapRegistryService llapRegistryService =
LlapRegistryService.getClient(conf);
+ Collection instances =
llapRegistryService.getInstances().getAll();
+ if (instances.size() == 0) {
+// Not in LLAP mode.
+return;
+ }
+ ExecutorService executorService = Executors.newCachedThreadPool();
Review comment:
@szlta the new commit introduced new problem, The problem is that this does
not work when using multiple threads each thread will see a null executor and
each one will create a new one and same with the shutdown method.
One way to fix this is to make the executor static and final, therefore will
be constructed at the time class loading. Also shutdown hook might be needed
here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 408287)
Time Spent: 4h 20m (was: 4h 10m)
> Add necessary endpoints for proactive cache eviction
>
>
> Key: HIVE-22821
> URL: https://issues.apache.org/jira/browse/HIVE-22821
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch,
> HIVE-22821.2.patch
>
> Time Spent: 4h 20m
>
[jira] [Commented] (HIVE-23030) Enable sketch union-s to be rolled up
[ https://issues.apache.org/jira/browse/HIVE-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065068#comment-17065068 ] Zoltan Haindrich commented on HIVE-23030: - [~jcamachorodriguez] Could you please take a look? I tried to make the SqlFunctionConverter "cleaner" - by not polluting it with the DS related mergability logic; and I went to register the DS functions into the converter; the result is much better than my earlier version - but to do this I had to almost completely replace the datasketches function registering logic - which added a lot of bulk changes... > Enable sketch union-s to be rolled up > - > > Key: HIVE-23030 > URL: https://issues.apache.org/jira/browse/HIVE-23030 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23030.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Enabling rolling up sketch aggregates could enable the matching of > materialized views created for higher dimensions to be applied for lower > dimension cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23030) Enable sketch union-s to be rolled up
[ https://issues.apache.org/jira/browse/HIVE-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23030: -- Labels: pull-request-available (was: ) > Enable sketch union-s to be rolled up > - > > Key: HIVE-23030 > URL: https://issues.apache.org/jira/browse/HIVE-23030 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23030.01.patch > > > Enabling rolling up sketch aggregates could enable the matching of > materialized views created for higher dimensions to be applied for lower > dimension cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23030) Enable sketch union-s to be rolled up
[ https://issues.apache.org/jira/browse/HIVE-23030?focusedWorklogId=408249=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408249 ] ASF GitHub Bot logged work on HIVE-23030: - Author: ASF GitHub Bot Created on: 23/Mar/20 20:23 Start Date: 23/Mar/20 20:23 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on pull request #960: HIVE-23030 ds rollup union URL: https://github.com/apache/hive/pull/960 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 408249) Remaining Estimate: 0h Time Spent: 10m > Enable sketch union-s to be rolled up > - > > Key: HIVE-23030 > URL: https://issues.apache.org/jira/browse/HIVE-23030 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23030.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Enabling rolling up sketch aggregates could enable the matching of > materialized views created for higher dimensions to be applied for lower > dimension cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23030) Enable sketch union-s to be rolled up
[ https://issues.apache.org/jira/browse/HIVE-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23030: Status: Patch Available (was: Open) > Enable sketch union-s to be rolled up > - > > Key: HIVE-23030 > URL: https://issues.apache.org/jira/browse/HIVE-23030 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23030.01.patch > > > Enabling rolling up sketch aggregates could enable the matching of > materialized views created for higher dimensions to be applied for lower > dimension cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23030) Enable sketch union-s to be rolled up
[ https://issues.apache.org/jira/browse/HIVE-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23030: Attachment: HIVE-23030.01.patch > Enable sketch union-s to be rolled up > - > > Key: HIVE-23030 > URL: https://issues.apache.org/jira/browse/HIVE-23030 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23030.01.patch > > > Enabling rolling up sketch aggregates could enable the matching of > materialized views created for higher dimensions to be applied for lower > dimension cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23045) Zookeeper SSL/TLS support
[
https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065061#comment-17065061
]
Hive QA commented on HIVE-23045:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997459/HIVE-23045.10.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18130 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21237/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21237/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21237/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997459 - PreCommit-HIVE-Build
> Zookeeper SSL/TLS support
> -
>
> Key: HIVE-23045
> URL: https://issues.apache.org/jira/browse/HIVE-23045
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC, Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-23045.1.patch, HIVE-23045.10.patch,
> HIVE-23045.2.patch, HIVE-23045.3.patch, HIVE-23045.4.patch,
> HIVE-23045.5.patch, HIVE-23045.6.patch, HIVE-23045.7.patch,
> HIVE-23045.8.patch, HIVE-23045.9.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its
> clients.
> [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide]
> The SSL communication should be possible in the different part of HIVE, where
> it communicates with Zookeeper servers. The Zookeeper clients are used in the
> following places:
> * HiveServer2 PrivilegeSynchronizer
> * HiveServer2 register/remove server from Zookeeper
> * HS2ActivePassiveHARegistryClient
> * ZooKeeperHiveLockManager
> * LLapZookeeperRegistryImpl
> * TezAmRegistryImpl
> * WebHCat ZooKeeperStorage
> * JDBC Driver server lookup
> * Metastore - ZookeeperTokenStore
> * Metastore register/remove server from Zookeeper
> The flag to enable SSL communication and the required parameters should be
> provided by different configuration parameters, corresponding the different
> use cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23045) Zookeeper SSL/TLS support
[
https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065050#comment-17065050
]
Hive QA commented on HIVE-23045:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
45s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} llap-client in master has 27 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} service in master has 50 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} hcatalog/webhcat/svr in master has 96 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} llap-client: The patch generated 0 new + 38
unchanged - 5 fixed = 38 total (was 43) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 36 unchanged - 1 fixed = 36 total (was 37) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 7 unchanged - 3
fixed = 7 total (was 10) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} service: The patch generated 0 new + 37 unchanged -
1 fixed = 37 total (was 38) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch jdbc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hcatalog/webhcat/svr: The patch generated 0 new + 19
unchanged - 2 fixed = 19 total (was 21) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} |
[jira] [Commented] (HIVE-22262) Aggregate pushdown through join may generate additional rewriting opportunities
[
https://issues.apache.org/jira/browse/HIVE-22262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065043#comment-17065043
]
Vineet Garg commented on HIVE-22262:
{code:sql}
explain cbo select pk1, dim2.fk4, sum(1), count(c1)
from fact, dim2
where to_date(fact.fk4) = dim2.fk4
group by pk1,dim2.fk4
order by pk1,dim2.fk4;
{code}
This doesn't fail anymore but we aren't able to rewrite it. I tried adding
{{HiveAggregateJoinTransposeRule}} before rewriting but it still doesn't kick
in rewriting.
Plan after the introduction of rule
{code}
HiveSortLimit(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC])
HiveProject($f0=[$0], $f1=[$1], $f2=[$2], $f3=[$3])
HiveAggregate(group=[{0, 1}], agg#0=[sum($2)], agg#1=[count($3)])
HiveProject($f0=[$1], $f1=[$2], $f2=[1], $f3=[$3])
HiveJoin(condition=[=($0, $4)], joinType=[inner], algorithm=[none],
cost=[not available])
HiveProject(TO_DATE=[TO_DATE($3)])
HiveFilter(condition=[IS NOT NULL(TO_DATE($3))])
HiveTableScan(table=[[default, fact]], table:alias=[fact])
HiveProject(pk1=[$0], fk4=[$1], c1=[$2], CAST=[CAST($1):DATE])
HiveFilter(condition=[IS NOT NULL(CAST($1):DATE)])
HiveTableScan(table=[[default, dim2]], table:alias=[dim2])
{code}
> Aggregate pushdown through join may generate additional rewriting
> opportunities
> ---
>
> Key: HIVE-22262
> URL: https://issues.apache.org/jira/browse/HIVE-22262
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Materialized views
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Priority: Major
> Attachments: eager-v2.sql
>
>
> In this case, there is a function used in the query and materialized view,
> but the aggregate is not being pushed down. Script is attached.
> Example query and materialized view:
> create materialized view av1 stored as orc as select fk1, fk2, fk3,
> to_date(fk4), sum(1) from fact group by 1, 2, 3, 4;
> explain cbo select pk1, dim2.fk4, sum(1), count(c1)
> from fact, dim2
> where to_date(fact.fk4) = dim2.fk4
> group by 1, 2
> order by 1, 2;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22262) Aggregate pushdown through join may generate additional rewriting opportunities
[ https://issues.apache.org/jira/browse/HIVE-22262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-22262: -- Assignee: Vineet Garg > Aggregate pushdown through join may generate additional rewriting > opportunities > --- > > Key: HIVE-22262 > URL: https://issues.apache.org/jira/browse/HIVE-22262 > Project: Hive > Issue Type: Sub-task > Components: CBO, Materialized views >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Attachments: eager-v2.sql > > > In this case, there is a function used in the query and materialized view, > but the aggregate is not being pushed down. Script is attached. > Example query and materialized view: > create materialized view av1 stored as orc as select fk1, fk2, fk3, > to_date(fk4), sum(1) from fact group by 1, 2, 3, 4; > explain cbo select pk1, dim2.fk4, sum(1), count(c1) > from fact, dim2 > where to_date(fact.fk4) = dim2.fk4 > group by 1, 2 > order by 1, 2; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23067) Use batch inserts in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065038#comment-17065038 ] Marton Bod commented on HIVE-23067: --- Yes, sure :) thanks for the heads-up > Use batch inserts in TxnHandler > --- > > Key: HIVE-23067 > URL: https://issues.apache.org/jira/browse/HIVE-23067 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > > To reduce the number of database calls and network roundtrips, we could use > more batching in TxnHandler, where currently in many places we call insert > commands in loops sequentially. > Some examples: > * openTxns (TXNS, REPL_TXN_MAP) > * commitTxn (COMPLETED_TXN_COMPONENTS) > * replTableWriteIdState (TXN_TO_WRITE_ID) > * allocateTableWriteIds (TXN_TO_WRITE_ID) > * > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-22997: -- Attachment: HIVE-22997.23.patch > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch, > HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch, > HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.17.patch, > HIVE-22997.18.patch, HIVE-22997.19.patch, HIVE-22997.2.patch, > HIVE-22997.20.patch, HIVE-22997.21.patch, HIVE-22997.22.patch, > HIVE-22997.23.patch, HIVE-22997.4.patch, HIVE-22997.5.patch, > HIVE-22997.6.patch, HIVE-22997.7.patch, HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 9h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23067) Use batch inserts in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065034#comment-17065034 ] Peter Vary commented on HIVE-23067: --- Currently working on the openTxns bit, so I will take care of that code path if you don't mind :) > Use batch inserts in TxnHandler > --- > > Key: HIVE-23067 > URL: https://issues.apache.org/jira/browse/HIVE-23067 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > > To reduce the number of database calls and network roundtrips, we could use > more batching in TxnHandler, where currently in many places we call insert > commands in loops sequentially. > Some examples: > * openTxns (TXNS, REPL_TXN_MAP) > * commitTxn (COMPLETED_TXN_COMPONENTS) > * replTableWriteIdState (TXN_TO_WRITE_ID) > * allocateTableWriteIds (TXN_TO_WRITE_ID) > * > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22760) Add Clock caching eviction based strategy
[ https://issues.apache.org/jira/browse/HIVE-22760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22760: -- Attachment: HIVE-22760.3.patch > Add Clock caching eviction based strategy > - > > Key: HIVE-22760 > URL: https://issues.apache.org/jira/browse/HIVE-22760 > Project: Hive > Issue Type: New Feature > Components: llap >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22760.2.patch, HIVE-22760.3.patch, > HIVE-22760.patch, HIVE-22760.patch > > > LRFU is the current default right now. > The main issue with such Strategy is that it has a very high memory overhead, > in addition to that, most of the accounting has to happen under locks thus > can be source of contentions. > Add Simpler policy like clock, can help with both issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23018) Provide a bulk API to fire multiple insert events
[ https://issues.apache.org/jira/browse/HIVE-23018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065021#comment-17065021 ] Vihang Karajgaonkar commented on HIVE-23018: precommit didn't pick up the patch. Reattaching. > Provide a bulk API to fire multiple insert events > - > > Key: HIVE-23018 > URL: https://issues.apache.org/jira/browse/HIVE-23018 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-23018.01.nothriftpatch, HIVE-23018.01.patch, > HIVE-23018.02.patch > > > Metastore provides a API to fire a listener event (currently only supports > INSERT event). The problem with that API is that it only takes in one > partition at a time. A typical query may insert data into multiple partitions > at a time. In such a case query engines like HS2 or Impala will have to issue > multiple RPCs to metastore sequentially to fire these events. This can show > up as a slowdown to the user if the query engines do not return the prompt to > the user until all the events are fired (In case of HS2 and Impala). It would > be great if we have bulk API which takes in multiple partitions for a table > so that metastore can generate many such events in one RPC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23018) Provide a bulk API to fire multiple insert events
[ https://issues.apache.org/jira/browse/HIVE-23018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-23018: --- Attachment: HIVE-23018.02.patch > Provide a bulk API to fire multiple insert events > - > > Key: HIVE-23018 > URL: https://issues.apache.org/jira/browse/HIVE-23018 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-23018.01.nothriftpatch, HIVE-23018.01.patch, > HIVE-23018.02.patch > > > Metastore provides a API to fire a listener event (currently only supports > INSERT event). The problem with that API is that it only takes in one > partition at a time. A typical query may insert data into multiple partitions > at a time. In such a case query engines like HS2 or Impala will have to issue > multiple RPCs to metastore sequentially to fire these events. This can show > up as a slowdown to the user if the query engines do not return the prompt to > the user until all the events are fired (In case of HS2 and Impala). It would > be great if we have bulk API which takes in multiple partitions for a table > so that metastore can generate many such events in one RPC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23060) Query failing with error "Grouping sets expression is not in GROUP BY key. Error encountered near token"
[
https://issues.apache.org/jira/browse/HIVE-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065013#comment-17065013
]
Hive QA commented on HIVE-23060:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997453/HIVE-23060.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18127 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[groupby_grouping_sets_view]
(batchId=107)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21236/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21236/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21236/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997453 - PreCommit-HIVE-Build
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
>
>
> Key: HIVE-23060
> URL: https://issues.apache.org/jira/browse/HIVE-23060
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Luis E Martinez-Poblete
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-23060.01.patch, HIVE-23060.02.patch
>
>
> Synopsis:
> =
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
> Problem:
>
> A Hive query in a view which fails with the following error:
> Error while compiling statement: FAILED: SemanticException 35:21 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'l0_equities_region_id'
> Reproduction case:
> {noformat}
> create database test;
> create table test.case665558 (c1 string, c2 string);
> -- Working query
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> create view test.viewcase665558
> as
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> Select * from test.viewcase665558 ;
> Error: Error while compiling statement: FAILED: SemanticException 17:1 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'c1' (state=42000,code=4)
> {noformat}
> The issue is because when the view is created, it adds the name of the table
> to the columns. This seems to be confusing Hive:
> {noformat}
> +-+--+
> | createtab_stmt |
> +-+--+
> | CREATE VIEW `test.viewcase665558` AS select |
> | case|
> | when GROUPING__ID = 255 then `case665558`.`c1` |
> | end as `col_1`, |
> | case|
> | when GROUPING__ID = 255 then 3 |
> | end as `col_2`, |
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | from|
> | `test`.`case665558` |
> | group by|
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | GROUPING SETS |
> | ( |
> | (c1), |
> | (c1, c2)|
> | ) |
> +-+--+
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23060) Query failing with error "Grouping sets expression is not in GROUP BY key. Error encountered near token"
[
https://issues.apache.org/jira/browse/HIVE-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064975#comment-17064975
]
Hive QA commented on HIVE-23060:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21236/dev-support/hive-personality.sh
|
| git revision | master / dac2bcd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21236/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
>
>
> Key: HIVE-23060
> URL: https://issues.apache.org/jira/browse/HIVE-23060
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Luis E Martinez-Poblete
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-23060.01.patch, HIVE-23060.02.patch
>
>
> Synopsis:
> =
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
> Problem:
>
> A Hive query in a view which fails with the following error:
> Error while compiling statement: FAILED: SemanticException 35:21 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'l0_equities_region_id'
> Reproduction case:
> {noformat}
> create database test;
> create table test.case665558 (c1 string, c2 string);
> -- Working query
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> create view test.viewcase665558
> as
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
[jira] [Updated] (HIVE-23067) Use batch inserts in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23067: -- Description: To reduce the number of database calls and network roundtrips, we could use more batching in TxnHandler, where currently in many places we call insert commands in loops sequentially. Some examples: * openTxns (TXNS, REPL_TXN_MAP) * commitTxn (COMPLETED_TXN_COMPONENTS) * replTableWriteIdState (TXN_TO_WRITE_ID) * allocateTableWriteIds (TXN_TO_WRITE_ID) * was: To reduce the number of database calls and network roundtrips, we could use more batching in TxnHandler, where currently in many places we call insert commands in loops sequentially. Some examples: * openTxns (TXNS, REPL_TXN_MAP * commitTxn (COMPLETED_TXN_COMPONENTS) * replTableWriteIdState (TXN_TO_WRITE_ID) * allocateTableWriteIds (TXN_TO_WRITE_ID) * > Use batch inserts in TxnHandler > --- > > Key: HIVE-23067 > URL: https://issues.apache.org/jira/browse/HIVE-23067 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > > To reduce the number of database calls and network roundtrips, we could use > more batching in TxnHandler, where currently in many places we call insert > commands in loops sequentially. > Some examples: > * openTxns (TXNS, REPL_TXN_MAP) > * commitTxn (COMPLETED_TXN_COMPONENTS) > * replTableWriteIdState (TXN_TO_WRITE_ID) > * allocateTableWriteIds (TXN_TO_WRITE_ID) > * > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23067) Use batch inserts in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod reassigned HIVE-23067: - > Use batch inserts in TxnHandler > --- > > Key: HIVE-23067 > URL: https://issues.apache.org/jira/browse/HIVE-23067 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > > To reduce the number of database calls and network roundtrips, we could use > more batching in TxnHandler, where currently in many places we call insert > commands in loops sequentially. > Some examples: > * openTxns (TXNS, REPL_TXN_MAP > * commitTxn (COMPLETED_TXN_COMPONENTS) > * replTableWriteIdState (TXN_TO_WRITE_ID) > * allocateTableWriteIds (TXN_TO_WRITE_ID) > * > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22875) Refactor query creation in QueryCompactor implementations
[ https://issues.apache.org/jira/browse/HIVE-22875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-22875: - Attachment: HIVE-22875.04.patch > Refactor query creation in QueryCompactor implementations > - > > Key: HIVE-22875 > URL: https://issues.apache.org/jira/browse/HIVE-22875 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22875.01.patch, HIVE-22875.02.patch, > HIVE-22875.02.patch, HIVE-22875.03.patch, HIVE-22875.04.patch > > > There is a lot of repetition where creation/compaction/drop queries are > created in MajorQueryCompactor, MinorQueryCompactor, MmMajorQueryCompactor > and MmMinorQueryCompactor. > Initial idea is to create a CompactionQueryBuilder that all 4 implementations > would use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23052: -- Attachment: HIVE-23052.5.patch > Optimize lock enqueueing in TxnHandler > -- > > Key: HIVE-23052 > URL: https://issues.apache.org/jira/browse/HIVE-23052 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23052.1.patch, HIVE-23052.2.patch, > HIVE-23052.3.patch, HIVE-23052.4.patch, HIVE-23052.5.patch > > > * Reduce scope of next_lock_id select-for-update by moving the txn_component > inserts before the S4U + inserting the hive_locks entries before the S4U > (first with a temp ID, which will be replaced later in a single update). This > helps decrease the overall time that the next_lock_id table is locked, > thereby increasing concurrency > * Insert txn_components in a batch instead of one-by-one (also in > TxnHandler::addDynamicPartition) > * Increment next_lock_id and update hive_locks table in a single batch > statement > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064960#comment-17064960
]
Hive QA commented on HIVE-22785:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997451/HIVE-22785.9.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 18127 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_cbo_rewrite]
(batchId=106)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_cbo_rewrite_2]
(batchId=107)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[update_notnull_constraint]
(batchId=106)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21235/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21235/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21235/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997451 - PreCommit-HIVE-Build
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch,
> HIVE-22785.5.patch, HIVE-22785.6.patch, HIVE-22785.7.patch,
> HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[ https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-23062: - Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Fix has been pushed to master. Thanks [~samuelan] for the patch. > Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue > -- > > Key: HIVE-23062 > URL: https://issues.apache.org/jira/browse/HIVE-23062 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23062.1.patch, HIVE-23062.2.patch, > HIVE-23062.3.patch > > > Currently if custom Tez queue is used, Hive will only check the Http port, so > it is not handling TLS and Yarn HA mode URL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-8123) Support parquet ACID
[ https://issues.apache.org/jira/browse/HIVE-8123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063607#comment-17063607 ] Vipin Vishvkarma edited comment on HIVE-8123 at 3/23/20, 4:50 PM: -- We have picked up the work and progressed a bit on this. Here is a [design doc|https://tinyurl.com/wgo2hx3], feel free to comment on the same. Will upload a WIP patch in the next few days. cc: [~gates] [~gopalv] [~pvary] was (Author: vpnvishv): We have picked up the work and progressed a bit on this. Here is a [design doc|*https://tinyurl.com/wgo2hx3*], feel free to comment on the same. Will upload a WIP patch in the next few days. cc: [~gates] [~gopalv] [~pvary] > Support parquet ACID > > > Key: HIVE-8123 > URL: https://issues.apache.org/jira/browse/HIVE-8123 > Project: Hive > Issue Type: Task >Reporter: Brock Noland >Assignee: Ferdinand Xu >Priority: Major > > Hive "ACID" work currently only works with ORC. It should work with Parquet > as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23045) Zookeeper SSL/TLS support
[ https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23045: --- Status: Patch Available (was: In Progress) > Zookeeper SSL/TLS support > - > > Key: HIVE-23045 > URL: https://issues.apache.org/jira/browse/HIVE-23045 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23045.1.patch, HIVE-23045.10.patch, > HIVE-23045.2.patch, HIVE-23045.3.patch, HIVE-23045.4.patch, > HIVE-23045.5.patch, HIVE-23045.6.patch, HIVE-23045.7.patch, > HIVE-23045.8.patch, HIVE-23045.9.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its > clients. > [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide] > The SSL communication should be possible in the different part of HIVE, where > it communicates with Zookeeper servers. The Zookeeper clients are used in the > following places: > * HiveServer2 PrivilegeSynchronizer > * HiveServer2 register/remove server from Zookeeper > * HS2ActivePassiveHARegistryClient > * ZooKeeperHiveLockManager > * LLapZookeeperRegistryImpl > * TezAmRegistryImpl > * WebHCat ZooKeeperStorage > * JDBC Driver server lookup > * Metastore - ZookeeperTokenStore > * Metastore register/remove server from Zookeeper > The flag to enable SSL communication and the required parameters should be > provided by different configuration parameters, corresponding the different > use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-8123) Support parquet ACID
[ https://issues.apache.org/jira/browse/HIVE-8123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063607#comment-17063607 ] Vipin Vishvkarma edited comment on HIVE-8123 at 3/23/20, 4:49 PM: -- We have picked up the work and progressed a bit on this. Here is a [design doc|*https://tinyurl.com/wgo2hx3*], feel free to comment on the same. Will upload a WIP patch in the next few days. cc: [~gates] [~gopalv] [~pvary] was (Author: vpnvishv): We have picked up the work and progressed a bit on this. Here is a design doc [[https://docs.google.com/document/d/19XQ3W-jyXP2M_94ltAeIc3Gdgum2oxI-CikRsuItcr0/edit#]], feel free to comment on the same. Will upload a WIP patch in the next few days. cc: [~gates] [~gopalv] [~pvary] > Support parquet ACID > > > Key: HIVE-8123 > URL: https://issues.apache.org/jira/browse/HIVE-8123 > Project: Hive > Issue Type: Task >Reporter: Brock Noland >Assignee: Ferdinand Xu >Priority: Major > > Hive "ACID" work currently only works with ORC. It should work with Parquet > as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23045) Zookeeper SSL/TLS support
[ https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23045: --- Attachment: HIVE-23045.10.patch > Zookeeper SSL/TLS support > - > > Key: HIVE-23045 > URL: https://issues.apache.org/jira/browse/HIVE-23045 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23045.1.patch, HIVE-23045.10.patch, > HIVE-23045.2.patch, HIVE-23045.3.patch, HIVE-23045.4.patch, > HIVE-23045.5.patch, HIVE-23045.6.patch, HIVE-23045.7.patch, > HIVE-23045.8.patch, HIVE-23045.9.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its > clients. > [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide] > The SSL communication should be possible in the different part of HIVE, where > it communicates with Zookeeper servers. The Zookeeper clients are used in the > following places: > * HiveServer2 PrivilegeSynchronizer > * HiveServer2 register/remove server from Zookeeper > * HS2ActivePassiveHARegistryClient > * ZooKeeperHiveLockManager > * LLapZookeeperRegistryImpl > * TezAmRegistryImpl > * WebHCat ZooKeeperStorage > * JDBC Driver server lookup > * Metastore - ZookeeperTokenStore > * Metastore register/remove server from Zookeeper > The flag to enable SSL communication and the required parameters should be > provided by different configuration parameters, corresponding the different > use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23045) Zookeeper SSL/TLS support
[ https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23045: --- Status: In Progress (was: Patch Available) > Zookeeper SSL/TLS support > - > > Key: HIVE-23045 > URL: https://issues.apache.org/jira/browse/HIVE-23045 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23045.1.patch, HIVE-23045.2.patch, > HIVE-23045.3.patch, HIVE-23045.4.patch, HIVE-23045.5.patch, > HIVE-23045.6.patch, HIVE-23045.7.patch, HIVE-23045.8.patch, HIVE-23045.9.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its > clients. > [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide] > The SSL communication should be possible in the different part of HIVE, where > it communicates with Zookeeper servers. The Zookeeper clients are used in the > following places: > * HiveServer2 PrivilegeSynchronizer > * HiveServer2 register/remove server from Zookeeper > * HS2ActivePassiveHARegistryClient > * ZooKeeperHiveLockManager > * LLapZookeeperRegistryImpl > * TezAmRegistryImpl > * WebHCat ZooKeeperStorage > * JDBC Driver server lookup > * Metastore - ZookeeperTokenStore > * Metastore register/remove server from Zookeeper > The flag to enable SSL communication and the required parameters should be > provided by different configuration parameters, corresponding the different > use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064930#comment-17064930 ] Zoltan Chovan commented on HIVE-23044: -- Out of the 141 test, 10 fail the rest timeouts. The failing tests fail without the patch applied as well. > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23044.1.branch-3.1.patch, > HIVE-23044.2.branch-3.1.patch, HIVE-23044.3.branch-3.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064915#comment-17064915
]
Hive QA commented on HIVE-22785:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 2 new + 529 unchanged - 8
fixed = 531 total (was 537) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
2s{color} | {color:red} ql generated 2 new + 1529 unchanged - 2 fixed = 1531
total (was 1531) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveSortExchange.getKeyExpressions()
may expose internal representation by returning
HiveSortExchange.keyExpressions At HiveSortExchange.java:by returning
HiveSortExchange.keyExpressions At HiveSortExchange.java:[line 82] |
| |
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveSortExchange.setKeyExpressions(ExprNodeDesc[])
may expose internal representation by storing an externally mutable object
into HiveSortExchange.keyExpressions At HiveSortExchange.java:by storing an
externally mutable object into HiveSortExchange.keyExpressions At
HiveSortExchange.java:[line 86] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21235/dev-support/hive-personality.sh
|
| git revision | master / 1cc00b1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21235/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21235/yetus/new-findbugs-ql.html
|
| modules | C: ql itests itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21235/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee:
[jira] [Updated] (HIVE-23060) Query failing with error "Grouping sets expression is not in GROUP BY key. Error encountered near token"
[
https://issues.apache.org/jira/browse/HIVE-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-23060:
---
Attachment: HIVE-23060.02.patch
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
>
>
> Key: HIVE-23060
> URL: https://issues.apache.org/jira/browse/HIVE-23060
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Luis E Martinez-Poblete
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-23060.01.patch, HIVE-23060.02.patch
>
>
> Synopsis:
> =
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
> Problem:
>
> A Hive query in a view which fails with the following error:
> Error while compiling statement: FAILED: SemanticException 35:21 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'l0_equities_region_id'
> Reproduction case:
> {noformat}
> create database test;
> create table test.case665558 (c1 string, c2 string);
> -- Working query
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> create view test.viewcase665558
> as
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> Select * from test.viewcase665558 ;
> Error: Error while compiling statement: FAILED: SemanticException 17:1 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'c1' (state=42000,code=4)
> {noformat}
> The issue is because when the view is created, it adds the name of the table
> to the columns. This seems to be confusing Hive:
> {noformat}
> +-+--+
> | createtab_stmt |
> +-+--+
> | CREATE VIEW `test.viewcase665558` AS select |
> | case|
> | when GROUPING__ID = 255 then `case665558`.`c1` |
> | end as `col_1`, |
> | case|
> | when GROUPING__ID = 255 then 3 |
> | end as `col_2`, |
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | from|
> | `test`.`case665558` |
> | group by|
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | GROUPING SETS |
> | ( |
> | (c1), |
> | (c1, c2)|
> | ) |
> +-+--+
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23060) Query failing with error "Grouping sets expression is not in GROUP BY key. Error encountered near token"
[
https://issues.apache.org/jira/browse/HIVE-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-23060:
---
Status: Patch Available (was: Open)
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
>
>
> Key: HIVE-23060
> URL: https://issues.apache.org/jira/browse/HIVE-23060
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Luis E Martinez-Poblete
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-23060.01.patch, HIVE-23060.02.patch
>
>
> Synopsis:
> =
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
> Problem:
>
> A Hive query in a view which fails with the following error:
> Error while compiling statement: FAILED: SemanticException 35:21 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'l0_equities_region_id'
> Reproduction case:
> {noformat}
> create database test;
> create table test.case665558 (c1 string, c2 string);
> -- Working query
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> create view test.viewcase665558
> as
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> Select * from test.viewcase665558 ;
> Error: Error while compiling statement: FAILED: SemanticException 17:1 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'c1' (state=42000,code=4)
> {noformat}
> The issue is because when the view is created, it adds the name of the table
> to the columns. This seems to be confusing Hive:
> {noformat}
> +-+--+
> | createtab_stmt |
> +-+--+
> | CREATE VIEW `test.viewcase665558` AS select |
> | case|
> | when GROUPING__ID = 255 then `case665558`.`c1` |
> | end as `col_1`, |
> | case|
> | when GROUPING__ID = 255 then 3 |
> | end as `col_2`, |
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | from|
> | `test`.`case665558` |
> | group by|
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | GROUPING SETS |
> | ( |
> | (c1), |
> | (c1, c2)|
> | ) |
> +-+--+
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23060) Query failing with error "Grouping sets expression is not in GROUP BY key. Error encountered near token"
[
https://issues.apache.org/jira/browse/HIVE-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-23060:
---
Status: Open (was: Patch Available)
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
>
>
> Key: HIVE-23060
> URL: https://issues.apache.org/jira/browse/HIVE-23060
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Luis E Martinez-Poblete
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-23060.01.patch
>
>
> Synopsis:
> =
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
> Problem:
>
> A Hive query in a view which fails with the following error:
> Error while compiling statement: FAILED: SemanticException 35:21 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'l0_equities_region_id'
> Reproduction case:
> {noformat}
> create database test;
> create table test.case665558 (c1 string, c2 string);
> -- Working query
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> create view test.viewcase665558
> as
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> Select * from test.viewcase665558 ;
> Error: Error while compiling statement: FAILED: SemanticException 17:1 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'c1' (state=42000,code=4)
> {noformat}
> The issue is because when the view is created, it adds the name of the table
> to the columns. This seems to be confusing Hive:
> {noformat}
> +-+--+
> | createtab_stmt |
> +-+--+
> | CREATE VIEW `test.viewcase665558` AS select |
> | case|
> | when GROUPING__ID = 255 then `case665558`.`c1` |
> | end as `col_1`, |
> | case|
> | when GROUPING__ID = 255 then 3 |
> | end as `col_2`, |
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | from|
> | `test`.`case665558` |
> | group by|
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | GROUPING SETS |
> | ( |
> | (c1), |
> | (c1, c2)|
> | ) |
> +-+--+
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[
https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=408014=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408014
]
ASF GitHub Bot logged work on HIVE-22821:
-
Author: ASF GitHub Bot
Created on: 23/Mar/20 16:07
Start Date: 23/Mar/20 16:07
Worklog Time Spent: 10m
Work Description: szlta commented on pull request #909: HIVE-22821
URL: https://github.com/apache/hive/pull/909#discussion_r396568344
##
File path: ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java
##
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import javax.net.SocketFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.io.retry.RetryPolicies;
+import org.apache.hadoop.io.retry.RetryPolicy;
+import org.apache.hadoop.net.NetUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Through this class the caller (typically HS2) can request eviction of
buffers from LLAP cache by specifying a DB,
+ * table or partition name/(value). Request sending is implemented here.
+ */
+public final class ProactiveEviction {
+
+ private ProactiveEviction() {
+// Not to be used;
+ }
+
+ /**
+ * Trigger LLAP cache eviction of buffers related to entities residing in
request parameter.
+ * @param conf
+ * @param request
+ */
+ public static void evict(Configuration conf, Request request) {
+if (!HiveConf.getBoolVar(conf,
HiveConf.ConfVars.LLAP_IO_PROACTIVE_EVICTION_ENABLED)) {
+ return;
+}
+
+try {
+ LlapRegistryService llapRegistryService =
LlapRegistryService.getClient(conf);
+ Collection instances =
llapRegistryService.getInstances().getAll();
+ if (instances.size() == 0) {
+// Not in LLAP mode.
+return;
+ }
+ ExecutorService executorService = Executors.newCachedThreadPool();
Review comment:
Thanks for the detailed comments! I agree with all the points above, I did
miss the fact that I was creating a new pool at each invocation - my bad.
I've amended the change with
https://github.com/szlta/hive/commit/74364b022f14edfa597a7d95b0774def8f3ddefd
to fix this and address the other points as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 408014)
Time Spent: 4h 10m (was: 4h)
> Add necessary endpoints for proactive cache eviction
>
>
> Key: HIVE-22821
> URL: https://issues.apache.org/jira/browse/HIVE-22821
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch,
> HIVE-22821.2.patch
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Implement the parts required for iHS2 -> LLAP daemons
[jira] [Commented] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064886#comment-17064886
]
Hive QA commented on HIVE-23052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997449/HIVE-23052.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21234/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21234/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21234/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-03-23 15:34:34.414
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-21234/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-03-23 15:34:34.417
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
95176bb..1cc00b1 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 95176bb HIVE-22888: Rewrite checkLock inner select with JOIN
operator (Denys Kuzmenko, reviewed by Peter Vary)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 1cc00b1 HIVE-23047: Calculate the epoch on DB side (Peter Vary
reviewed by Denys Kuzmenko)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-03-23 15:34:35.686
+ rm -rf ../yetus_PreCommit-HIVE-Build-21234
+ mkdir ../yetus_PreCommit-HIVE-Build-21234
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-21234
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-21234/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:2388
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java'
with conflicts.
Going to apply patch with: git apply -p0
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:2388
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java'
with conflicts.
U
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-21234
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997449 - PreCommit-HIVE-Build
> Optimize lock enqueueing in TxnHandler
> --
>
> Key: HIVE-23052
> URL: https://issues.apache.org/jira/browse/HIVE-23052
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23052.1.patch, HIVE-23052.2.patch,
> HIVE-23052.3.patch, HIVE-23052.4.patch
>
>
> * Reduce scope of next_lock_id select-for-update by moving the txn_component
> inserts before the S4U + inserting the hive_locks entries before the S4U
> (first with a temp ID, which will be replaced later in a single update). This
> helps decrease the overall time that the next_lock_id table is locked,
> thereby increasing concurrency
> *
[jira] [Commented] (HIVE-23060) Query failing with error "Grouping sets expression is not in GROUP BY key. Error encountered near token"
[
https://issues.apache.org/jira/browse/HIVE-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064884#comment-17064884
]
Hive QA commented on HIVE-23060:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997440/HIVE-23060.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18127 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[groupby_grouping_sets_view]
(batchId=107)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21233/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21233/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21233/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997440 - PreCommit-HIVE-Build
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
>
>
> Key: HIVE-23060
> URL: https://issues.apache.org/jira/browse/HIVE-23060
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Luis E Martinez-Poblete
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-23060.01.patch
>
>
> Synopsis:
> =
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
> Problem:
>
> A Hive query in a view which fails with the following error:
> Error while compiling statement: FAILED: SemanticException 35:21 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'l0_equities_region_id'
> Reproduction case:
> {noformat}
> create database test;
> create table test.case665558 (c1 string, c2 string);
> -- Working query
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> create view test.viewcase665558
> as
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> Select * from test.viewcase665558 ;
> Error: Error while compiling statement: FAILED: SemanticException 17:1 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'c1' (state=42000,code=4)
> {noformat}
> The issue is because when the view is created, it adds the name of the table
> to the columns. This seems to be confusing Hive:
> {noformat}
> +-+--+
> | createtab_stmt |
> +-+--+
> | CREATE VIEW `test.viewcase665558` AS select |
> | case|
> | when GROUPING__ID = 255 then `case665558`.`c1` |
> | end as `col_1`, |
> | case|
> | when GROUPING__ID = 255 then 3 |
> | end as `col_2`, |
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | from|
> | `test`.`case665558` |
> | group by|
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | GROUPING SETS |
> | ( |
> | (c1), |
> | (c1, c2)|
> | ) |
> +-+--+
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Attachment: HIVE-22785.9.patch
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch,
> HIVE-22785.5.patch, HIVE-22785.6.patch, HIVE-22785.7.patch,
> HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Status: Patch Available (was: Open)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch,
> HIVE-22785.5.patch, HIVE-22785.6.patch, HIVE-22785.7.patch,
> HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Status: Open (was: Patch Available)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch,
> HIVE-22785.5.patch, HIVE-22785.6.patch, HIVE-22785.7.patch,
> HIVE-22785.8.patch, HIVE-22785.9.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23052: -- Description: * Reduce scope of next_lock_id select-for-update by moving the txn_component inserts before the S4U + inserting the hive_locks entries before the S4U (first with a temp ID, which will be replaced later in a single update). This helps decrease the overall time that the next_lock_id table is locked, thereby increasing concurrency * Insert txn_components in a batch instead of one-by-one (also in TxnHandler::addDynamicPartition) * Increment next_lock_id and update hive_locks table in a single batch statement was: * Reduce scope of next_lock_id select-for-update by moving the txn_component inserts before the S4U + inserting the hive_locks entries before the S4U (first with a temp ID, which will be replaced later in a single update). This helps decrease the overall time that the next_lock_id table is locked, thereby increasing concurrency * Insert txn_components in a batch instead of one-by-one * Increment next_lock_id and update hive_locks table in a single batch statement > Optimize lock enqueueing in TxnHandler > -- > > Key: HIVE-23052 > URL: https://issues.apache.org/jira/browse/HIVE-23052 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23052.1.patch, HIVE-23052.2.patch, > HIVE-23052.3.patch, HIVE-23052.4.patch > > > * Reduce scope of next_lock_id select-for-update by moving the txn_component > inserts before the S4U + inserting the hive_locks entries before the S4U > (first with a temp ID, which will be replaced later in a single update). This > helps decrease the overall time that the next_lock_id table is locked, > thereby increasing concurrency > * Insert txn_components in a batch instead of one-by-one (also in > TxnHandler::addDynamicPartition) > * Increment next_lock_id and update hive_locks table in a single batch > statement > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064866#comment-17064866 ] László Pintér edited comment on HIVE-23058 at 3/23/20, 3:14 PM: [~rtrivedi12] Could you please add some unit tests? You could put an empty file in the scratch dir of the compaction, trigger a compaction and make sure the execution succeeds. Thanks was (Author: lpinter): [~rtrivedi12] Could you please add some unit tests? Thanks > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23047) Calculate the epoch on DB side
[ https://issues.apache.org/jira/browse/HIVE-23047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23047: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the review [~dkuzmenko]! > Calculate the epoch on DB side > -- > > Key: HIVE-23047 > URL: https://issues.apache.org/jira/browse/HIVE-23047 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23047.2.patch, HIVE-23047.patch > > > We use TxnHandler.getDbTime to calculate the epoch on the DB server, and > immediately insert the value back again. We would be better of by using sql > to calculate the value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064866#comment-17064866 ] László Pintér commented on HIVE-23058: -- [~rtrivedi12] Could you please add some unit tests? Thanks > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23042) Merge queries to a single one for updating MIN_OPEN_TXNS table
[
https://issues.apache.org/jira/browse/HIVE-23042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-23042:
--
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for the review [~dkuzmenko]!
> Merge queries to a single one for updating MIN_OPEN_TXNS table
> --
>
> Key: HIVE-23042
> URL: https://issues.apache.org/jira/browse/HIVE-23042
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23042.2.patch, HIVE-23042.3.patch,
> HIVE-23042.4.patch, HIVE-23042.patch
>
>
> When opening a new transaction we issue 2 queries to update the MIN_OPEN_TXN
> table.
> {code}
>
> values(763, 763)>
> {code}
> This could be archived with a single query faster, if we do not open
> transactions in batch, like:
> {code}
>SELECT ?, MIN("TXN_ID") FROM "TXNS" WHERE "TXN_STATE" = 'o'>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23052: -- Attachment: HIVE-23052.4.patch > Optimize lock enqueueing in TxnHandler > -- > > Key: HIVE-23052 > URL: https://issues.apache.org/jira/browse/HIVE-23052 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23052.1.patch, HIVE-23052.2.patch, > HIVE-23052.3.patch, HIVE-23052.4.patch > > > * Reduce scope of next_lock_id select-for-update by moving the txn_component > inserts before the S4U + inserting the hive_locks entries before the S4U > (first with a temp ID, which will be replaced later in a single update). This > helps decrease the overall time that the next_lock_id table is locked, > thereby increasing concurrency > * Insert txn_components in a batch instead of one-by-one > * Increment next_lock_id and update hive_locks table in a single batch > statement > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23060) Query failing with error "Grouping sets expression is not in GROUP BY key. Error encountered near token"
[
https://issues.apache.org/jira/browse/HIVE-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064842#comment-17064842
]
Hive QA commented on HIVE-23060:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21233/dev-support/hive-personality.sh
|
| git revision | master / 95176bb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21233/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
>
>
> Key: HIVE-23060
> URL: https://issues.apache.org/jira/browse/HIVE-23060
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Luis E Martinez-Poblete
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-23060.01.patch
>
>
> Synopsis:
> =
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
> Problem:
>
> A Hive query in a view which fails with the following error:
> Error while compiling statement: FAILED: SemanticException 35:21 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'l0_equities_region_id'
> Reproduction case:
> {noformat}
> create database test;
> create table test.case665558 (c1 string, c2 string);
> -- Working query
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> create view test.viewcase665558
> as
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>
[jira] [Commented] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064824#comment-17064824
]
Hive QA commented on HIVE-22785:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997405/HIVE-22785.8.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 18127 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_cbo_rewrite]
(batchId=106)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_cbo_rewrite_2]
(batchId=107)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[update_notnull_constraint]
(batchId=106)
org.apache.hadoop.hive.ql.metadata.TestHiveRemote.testMetaStoreApiTiming
(batchId=342)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21232/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21232/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21232/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997405 - PreCommit-HIVE-Build
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch,
> HIVE-22785.5.patch, HIVE-22785.6.patch, HIVE-22785.7.patch, HIVE-22785.8.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23060) Query failing with error "Grouping sets expression is not in GROUP BY key. Error encountered near token"
[
https://issues.apache.org/jira/browse/HIVE-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-23060:
---
Attachment: HIVE-23060.01.patch
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
>
>
> Key: HIVE-23060
> URL: https://issues.apache.org/jira/browse/HIVE-23060
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Luis E Martinez-Poblete
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-23060.01.patch
>
>
> Synopsis:
> =
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
> Problem:
>
> A Hive query in a view which fails with the following error:
> Error while compiling statement: FAILED: SemanticException 35:21 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'l0_equities_region_id'
> Reproduction case:
> {noformat}
> create database test;
> create table test.case665558 (c1 string, c2 string);
> -- Working query
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> create view test.viewcase665558
> as
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> Select * from test.viewcase665558 ;
> Error: Error while compiling statement: FAILED: SemanticException 17:1 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'c1' (state=42000,code=4)
> {noformat}
> The issue is because when the view is created, it adds the name of the table
> to the columns. This seems to be confusing Hive:
> {noformat}
> +-+--+
> | createtab_stmt |
> +-+--+
> | CREATE VIEW `test.viewcase665558` AS select |
> | case|
> | when GROUPING__ID = 255 then `case665558`.`c1` |
> | end as `col_1`, |
> | case|
> | when GROUPING__ID = 255 then 3 |
> | end as `col_2`, |
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | from|
> | `test`.`case665558` |
> | group by|
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | GROUPING SETS |
> | ( |
> | (c1), |
> | (c1, c2)|
> | ) |
> +-+--+
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23060) Query failing with error "Grouping sets expression is not in GROUP BY key. Error encountered near token"
[
https://issues.apache.org/jira/browse/HIVE-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-23060:
---
Status: Patch Available (was: Open)
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
>
>
> Key: HIVE-23060
> URL: https://issues.apache.org/jira/browse/HIVE-23060
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Luis E Martinez-Poblete
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-23060.01.patch
>
>
> Synopsis:
> =
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
> Problem:
>
> A Hive query in a view which fails with the following error:
> Error while compiling statement: FAILED: SemanticException 35:21 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'l0_equities_region_id'
> Reproduction case:
> {noformat}
> create database test;
> create table test.case665558 (c1 string, c2 string);
> -- Working query
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> create view test.viewcase665558
> as
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> Select * from test.viewcase665558 ;
> Error: Error while compiling statement: FAILED: SemanticException 17:1 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'c1' (state=42000,code=4)
> {noformat}
> The issue is because when the view is created, it adds the name of the table
> to the columns. This seems to be confusing Hive:
> {noformat}
> +-+--+
> | createtab_stmt |
> +-+--+
> | CREATE VIEW `test.viewcase665558` AS select |
> | case|
> | when GROUPING__ID = 255 then `case665558`.`c1` |
> | end as `col_1`, |
> | case|
> | when GROUPING__ID = 255 then 3 |
> | end as `col_2`, |
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | from|
> | `test`.`case665558` |
> | group by|
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | GROUPING SETS |
> | ( |
> | (c1), |
> | (c1, c2)|
> | ) |
> +-+--+
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064795#comment-17064795
]
Hive QA commented on HIVE-22785:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 35 new + 527 unchanged - 7
fixed = 562 total (was 534) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
0s{color} | {color:red} ql generated 1 new + 1531 unchanged - 0 fixed = 1532
total (was 1531) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to sortFields in
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSortPullUpConstantsRuleBase.onMatch(RelOptRuleCall)
At
HiveSortPullUpConstantsRuleBase.java:org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSortPullUpConstantsRuleBase.onMatch(RelOptRuleCall)
At HiveSortPullUpConstantsRuleBase.java:[line 150] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21232/dev-support/hive-personality.sh
|
| git revision | master / 95176bb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21232/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21232/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21232/yetus/new-findbugs-ql.html
|
| modules | C: ql itests itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21232/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch,
[jira] [Comment Edited] (HIVE-22957) Support For Filter Expression In MSCK Command
[ https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064752#comment-17064752 ] Syed Shameerur Rahman edited comment on HIVE-22957 at 3/23/20, 1:06 PM: [~thejas] [~pvary] [~apivovarov] [~jcamachorodriguez] Could you take a look at the patch? was (Author: srahman): [~thejas] [~kgyrtkirk] [~jcamachorodriguez] Could you take a look at the patch? > Support For Filter Expression In MSCK Command > - > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[
https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064788#comment-17064788
]
Sungwoo commented on HIVE-21164:
I tested with Hive 4 on Tez, and confirm that the same phenomenon occurs. Here
is the summary of the setup:
- Hive 4 commit: ffee30e6267e85f00a22767262192abb9681cfb7 (HIVE-21164: ACID:
...), Fri Feb 21
- Tez commit: fd19ce6c93bc1f899ccca7161b0c0407f850bd77 (TEZ-4123. ...), Wed Feb
12
- hive.acid.direct.insert.enabled set to true
- The warehouse directories reside on S3 (simulated with MinIO), not on HDFS.
- minor changes to tez/pom.xml and hive/pom.xml to fix compilation issues
Result:
0: jdbc:hive2://indigo1:9842/> select * from web_sales limit 100;
No rows selected (99.906 seconds)
0: jdbc:hive2://indigo1:9842/> select count(*) from web_sales;
+--+
| _c0|
+--+
| 1438883 |
+--+
1 row selected (0.613 seconds)
If we do not create a transactional table, the result is okay. If we add the
following line, the resultant table is empty:
TBLPROPERTIES('transactional'='true', 'transactional_properties'='default');
>From the log of HiveServer2, it seems that HiveServer2 deletes the output
>directories because Utilities.handleDirectInsertTableFinalPath() is called
>twice:
20/03/23 12:38:20 INFO FileOperations: Deleting
s3a://hivemr3/warehouse/tpcds_bin_partitioned_orc_2.db/web_sales/ws_sold_date_sk=2451145/base_001/bucket_0_0
that was not committed
> ACID: explore how we can avoid a move step during inserts/compaction
>
>
> Key: HIVE-21164
> URL: https://issues.apache.org/jira/browse/HIVE-21164
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch,
> HIVE-21164.11.patch, HIVE-21164.11.patch, HIVE-21164.12.patch,
> HIVE-21164.13.patch, HIVE-21164.14.patch, HIVE-21164.14.patch,
> HIVE-21164.15.patch, HIVE-21164.16.patch, HIVE-21164.17.patch,
> HIVE-21164.18.patch, HIVE-21164.19.patch, HIVE-21164.2.patch,
> HIVE-21164.20.patch, HIVE-21164.21.patch, HIVE-21164.22.patch,
> HIVE-21164.3.patch, HIVE-21164.4.patch, HIVE-21164.5.patch,
> HIVE-21164.6.patch, HIVE-21164.7.patch, HIVE-21164.8.patch, HIVE-21164.9.patch
>
>
> Currently, we write compacted data to a temporary location and then move the
> files to a final location, which is an expensive operation on some cloud file
> systems. Since HIVE-20823 is already in, it can control the visibility of
> compacted data for the readers. Therefore, we can perhaps avoid writing data
> to a temporary location and directly write compacted data to the intended
> final path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064769#comment-17064769
]
Hive QA commented on HIVE-23052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997404/HIVE-23052.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 77 failed/errored test(s), 18125 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop_timestamp_date_cast]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_locks]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_nonpart]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_sizebug]
(batchId=94)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_3]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_into_default_keyword]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_only_empty_query]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_window]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[runtime_stats_merge]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_materialized]
(batchId=184)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin_hint]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sqlmerge]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sqlmerge_stats]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_acid4]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_mapjoin_complex_values]
(batchId=185)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=117)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testLocksWithTxn
(batchId=241)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testFindPotentialCompactions
(batchId=317)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleanedCleansTxnsAndTxnComponents
(batchId=317)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testCheckLockAcquireAfterWaiting
(batchId=317)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testCheckLockTxnAborted
(batchId=317)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockEESW (batchId=317)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockESRSW (batchId=317)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockSRSW (batchId=317)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockSWSR (batchId=317)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockSWSWSR (batchId=317)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockSWSWSW (batchId=317)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testUnlockOnAbort
(batchId=317)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testUnlockOnCommit
(batchId=317)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testUnlockWithTxn
(batchId=317)
org.apache.hadoop.hive.ql.TestTxnCommands.testMergeOnTezEdges (batchId=362)
org.apache.hadoop.hive.ql.TestTxnCommands2.testOrcPPD (batchId=343)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testOrcPPD
(batchId=357)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteForPartitionedMmTable
(batchId=317)
org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteForPartitionedMmTable
(batchId=339)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMergeOnTezEdges
(batchId=343)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNoBuckets (batchId=343)
[jira] [Commented] (HIVE-22957) Support For Filter Expression In MSCK Command
[ https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064752#comment-17064752 ] Syed Shameerur Rahman commented on HIVE-22957: -- [~thejas] [~kgyrtkirk] [~jcamachorodriguez] Could you take a look at the patch? > Support For Filter Expression In MSCK Command > - > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-17978) Shared work optimizer may leave useless operator branches in the plan
[
https://issues.apache.org/jira/browse/HIVE-17978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064726#comment-17064726
]
philipse commented on HIVE-17978:
-
[~jcamachorodriguez]
> Shared work optimizer may leave useless operator branches in the plan
> -
>
> Key: HIVE-17978
> URL: https://issues.apache.org/jira/browse/HIVE-17978
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-17978.patch
>
>
> Failed with the following Exception
> {code}
> ERROR [6c707c4e-2849-4ff2-809d-946581e6b83a HiveServer2-Handler-Pool:
> Thread-78] ql.Driver: FAILED: SemanticException
> org.apache.hadoop.hive.ql.metadata.HiveException: The column KEY._col0 is not
> in the vectorization context column map {_col0=0}.
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.metadata.HiveException: The column KEY._col0 is not
> in the vectorization context column map {_col0=0}.
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationNodeProcessor.doVectorize(Vectorizer.java:1665)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$MapWorkVectorizationNodeProcessor.process(Vectorizer.java:1725)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:43)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.vectorizeMapWork(Vectorizer.java:1245)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:671)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:616)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:1902)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:674)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:271)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11621)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:296)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:169)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:599)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1463)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1420)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:201)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:288)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:249)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:532)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:518)
> at
>
[jira] [Commented] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064725#comment-17064725
]
Hive QA commented on HIVE-23052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 35 new + 522 unchanged - 37 fixed = 557 total (was 559) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21231/dev-support/hive-personality.sh
|
| git revision | master / 95176bb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21231/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21231/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimize lock enqueueing in TxnHandler
> --
>
> Key: HIVE-23052
> URL: https://issues.apache.org/jira/browse/HIVE-23052
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23052.1.patch, HIVE-23052.2.patch,
> HIVE-23052.3.patch
>
>
> * Reduce scope of next_lock_id select-for-update by moving the txn_component
> inserts before the S4U + inserting the hive_locks entries before the S4U
> (first with a temp ID, which will be replaced later in a single update). This
> helps decrease the overall time that the next_lock_id table is locked,
> thereby increasing concurrency
> * Insert txn_components in a batch instead of one-by-one
> * Increment next_lock_id and update hive_locks table in a single batch
> statement
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-17978) Shared work optimizer may leave useless operator branches in the plan
[
https://issues.apache.org/jira/browse/HIVE-17978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064711#comment-17064711
]
philipse commented on HIVE-17978:
-
[~ashutoshc] i meet the same issue in hive2.1，but i don't know whether it has
been fixed in branch 2? SharedWorkOptimizer does not exists in hive2.1
> Shared work optimizer may leave useless operator branches in the plan
> -
>
> Key: HIVE-17978
> URL: https://issues.apache.org/jira/browse/HIVE-17978
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Nita Dembla
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-17978.patch
>
>
> Failed with the following Exception
> {code}
> ERROR [6c707c4e-2849-4ff2-809d-946581e6b83a HiveServer2-Handler-Pool:
> Thread-78] ql.Driver: FAILED: SemanticException
> org.apache.hadoop.hive.ql.metadata.HiveException: The column KEY._col0 is not
> in the vectorization context column map {_col0=0}.
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.metadata.HiveException: The column KEY._col0 is not
> in the vectorization context column map {_col0=0}.
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationNodeProcessor.doVectorize(Vectorizer.java:1665)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$MapWorkVectorizationNodeProcessor.process(Vectorizer.java:1725)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:43)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.vectorizeMapWork(Vectorizer.java:1245)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:671)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:616)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:1902)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:674)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:271)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11621)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:296)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:169)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:599)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1463)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1420)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:201)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:288)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:249)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:532)
> at
>
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Attachment: (was: HIVE-22785.8.patch)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch,
> HIVE-22785.5.patch, HIVE-22785.6.patch, HIVE-22785.7.patch, HIVE-22785.8.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Status: Patch Available (was: Open)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch,
> HIVE-22785.5.patch, HIVE-22785.6.patch, HIVE-22785.7.patch, HIVE-22785.8.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
