[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100585#comment-17100585
]
Hive QA commented on HIVE-23282:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002132/HIVE-23282.8.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17216 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22155/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22155/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22155/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002132 - PreCommit-HIVE-Build
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> -
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-23282
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch,
> HIVE-23282.5.patch, HIVE-23282.6.patch, HIVE-23282.7.patch,
> HIVE-23282.8.patch, image-2020-04-23-14-07-06-077.png
>
>
> ObjectStore::getPartitionsByExprInternal internally uses Table information
> for getting partitionKeys, table, catalog name.
>
> For this, it ends up populating entire table data from DB (including skew
> column, parameters, sort, bucket cols etc). This makes it a lot more
> expensive call. It would be good to check if MTable itself can be used
> instead of Table.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669]
>
> !image-2020-04-23-14-07-06-077.png|width=665,height=592!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100560#comment-17100560
]
Hive QA commented on HIVE-23282:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
14s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
36s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 20 new + 598 unchanged - 6 fixed = 618 total (was 604) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22155/dev-support/hive-personality.sh
|
| git revision | master / 772bfda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22155/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22155/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> -
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-23282
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch,
> HIVE-23282.5.patch, HIVE-23282.6.patch, HIVE-23282.7.patch,
> HIVE-23282.8.patch, image-2020-04-23-14-07-06-077.png
>
>
> ObjectStore::getPartitionsByExprInternal internally uses Table information
> for getting partitionKeys, table, catalog name.
>
> For this, it ends up populating entire table data from DB (including skew
> column, parameters, sort, bucket cols etc). This makes it a lot more
> expensive call. It would be good to check if MTable
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100362#comment-17100362
]
Hive QA commented on HIVE-23282:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002092/HIVE-23282.8.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17216 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testReplTimeouts
(batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22151/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22151/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22151/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002092 - PreCommit-HIVE-Build
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> -
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-23282
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch,
> HIVE-23282.5.patch, HIVE-23282.6.patch, HIVE-23282.7.patch,
> HIVE-23282.8.patch, image-2020-04-23-14-07-06-077.png
>
>
> ObjectStore::getPartitionsByExprInternal internally uses Table information
> for getting partitionKeys, table, catalog name.
>
> For this, it ends up populating entire table data from DB (including skew
> column, parameters, sort, bucket cols etc). This makes it a lot more
> expensive call. It would be good to check if MTable itself can be used
> instead of Table.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669]
>
> !image-2020-04-23-14-07-06-077.png|width=665,height=592!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100348#comment-17100348
]
Hive QA commented on HIVE-23282:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 20 new + 598 unchanged - 6 fixed = 618 total (was 604) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22151/dev-support/hive-personality.sh
|
| git revision | master / 772bfda |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22151/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22151/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> -
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-23282
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch,
> HIVE-23282.5.patch, HIVE-23282.6.patch, HIVE-23282.7.patch,
> HIVE-23282.8.patch, image-2020-04-23-14-07-06-077.png
>
>
> ObjectStore::getPartitionsByExprInternal internally uses Table information
> for getting partitionKeys, table, catalog name.
>
> For this, it ends up populating entire table data from DB (including skew
> column, parameters, sort, bucket cols etc). This makes it a lot more
> expensive call. It would be good to check if MTable
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100217#comment-17100217 ] Ashutosh Chauhan commented on HIVE-23282: - got it. LGTM +1 > Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal > - > > Key: HIVE-23282 > URL: https://issues.apache.org/jira/browse/HIVE-23282 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch, > HIVE-23282.5.patch, HIVE-23282.6.patch, HIVE-23282.7.patch, > HIVE-23282.8.patch, image-2020-04-23-14-07-06-077.png > > > ObjectStore::getPartitionsByExprInternal internally uses Table information > for getting partitionKeys, table, catalog name. > > For this, it ends up populating entire table data from DB (including skew > column, parameters, sort, bucket cols etc). This makes it a lot more > expensive call. It would be good to check if MTable itself can be used > instead of Table. > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669] > > !image-2020-04-23-14-07-06-077.png|width=665,height=592! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100208#comment-17100208 ] Attila Magyar commented on HIVE-23282: -- [~ashutoshc], that's strange, it's already updated but reviewboard doesn't properly show the difference between the last two revisions. > Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal > - > > Key: HIVE-23282 > URL: https://issues.apache.org/jira/browse/HIVE-23282 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch, > HIVE-23282.5.patch, HIVE-23282.6.patch, HIVE-23282.7.patch, > HIVE-23282.8.patch, image-2020-04-23-14-07-06-077.png > > > ObjectStore::getPartitionsByExprInternal internally uses Table information > for getting partitionKeys, table, catalog name. > > For this, it ends up populating entire table data from DB (including skew > column, parameters, sort, bucket cols etc). This makes it a lot more > expensive call. It would be good to check if MTable itself can be used > instead of Table. > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669] > > !image-2020-04-23-14-07-06-077.png|width=665,height=592! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100050#comment-17100050 ] Ashutosh Chauhan commented on HIVE-23282: - [~amagyar] looks like latest patch didnt get updated on RB correctly. Can you add latest version on RB? > Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal > - > > Key: HIVE-23282 > URL: https://issues.apache.org/jira/browse/HIVE-23282 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch, > HIVE-23282.5.patch, HIVE-23282.6.patch, HIVE-23282.7.patch, > HIVE-23282.8.patch, image-2020-04-23-14-07-06-077.png > > > ObjectStore::getPartitionsByExprInternal internally uses Table information > for getting partitionKeys, table, catalog name. > > For this, it ends up populating entire table data from DB (including skew > column, parameters, sort, bucket cols etc). This makes it a lot more > expensive call. It would be good to check if MTable itself can be used > instead of Table. > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669] > > !image-2020-04-23-14-07-06-077.png|width=665,height=592! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099860#comment-17099860
]
Hive QA commented on HIVE-23282:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002048/HIVE-23282.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17203 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22139/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22139/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22139/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002048 - PreCommit-HIVE-Build
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> -
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-23282
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch,
> HIVE-23282.5.patch, HIVE-23282.6.patch, HIVE-23282.7.patch,
> image-2020-04-23-14-07-06-077.png
>
>
> ObjectStore::getPartitionsByExprInternal internally uses Table information
> for getting partitionKeys, table, catalog name.
>
> For this, it ends up populating entire table data from DB (including skew
> column, parameters, sort, bucket cols etc). This makes it a lot more
> expensive call. It would be good to check if MTable itself can be used
> instead of Table.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669]
>
> !image-2020-04-23-14-07-06-077.png|width=665,height=592!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099841#comment-17099841
]
Hive QA commented on HIVE-23282:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
9s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 20 new + 598 unchanged - 6 fixed = 618 total (was 604) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22139/dev-support/hive-personality.sh
|
| git revision | master / dbc04ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22139/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22139/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> -
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-23282
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch,
> HIVE-23282.5.patch, HIVE-23282.6.patch, HIVE-23282.7.patch,
> image-2020-04-23-14-07-06-077.png
>
>
> ObjectStore::getPartitionsByExprInternal internally uses Table information
> for getting partitionKeys, table, catalog name.
>
> For this, it ends up populating entire table data from DB (including skew
> column, parameters, sort, bucket cols etc). This makes it a lot more
> expensive call. It would be good to check if MTable itself can be used
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099533#comment-17099533 ] Ashutosh Chauhan commented on HIVE-23282: - Left some comments on RB. In patch, there is an additional RDBMS call which can be avoided. > Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal > - > > Key: HIVE-23282 > URL: https://issues.apache.org/jira/browse/HIVE-23282 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch, > HIVE-23282.5.patch, HIVE-23282.6.patch, image-2020-04-23-14-07-06-077.png > > > ObjectStore::getPartitionsByExprInternal internally uses Table information > for getting partitionKeys, table, catalog name. > > For this, it ends up populating entire table data from DB (including skew > column, parameters, sort, bucket cols etc). This makes it a lot more > expensive call. It would be good to check if MTable itself can be used > instead of Table. > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669] > > !image-2020-04-23-14-07-06-077.png|width=665,height=592! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099221#comment-17099221
]
Hive QA commented on HIVE-23282:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001991/HIVE-23282.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17201 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22122/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22122/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22122/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001991 - PreCommit-HIVE-Build
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> -
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-23282
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch,
> HIVE-23282.5.patch, HIVE-23282.6.patch, image-2020-04-23-14-07-06-077.png
>
>
> ObjectStore::getPartitionsByExprInternal internally uses Table information
> for getting partitionKeys, table, catalog name.
>
> For this, it ends up populating entire table data from DB (including skew
> column, parameters, sort, bucket cols etc). This makes it a lot more
> expensive call. It would be good to check if MTable itself can be used
> instead of Table.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669]
>
> !image-2020-04-23-14-07-06-077.png|width=665,height=592!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099177#comment-17099177
]
Hive QA commented on HIVE-23282:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
11s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 21 new + 598 unchanged - 6 fixed = 619 total (was 604) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 187 unchanged - 0 fixed = 188 total (was 187) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Possible null pointer dereference of null in
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilterInternal(String,
String, String, String, short, boolean, boolean) Dereferenced at
ObjectStore.java:null in
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilterInternal(String,
String, String, String, short, boolean, boolean) Dereferenced at
ObjectStore.java:[line 4077] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22122/dev-support/hive-personality.sh
|
| git revision | master / c34ee9d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22122/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22122/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22122/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> -
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-23282
>
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096822#comment-17096822
]
Hive QA commented on HIVE-23282:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001631/HIVE-23282.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 17190 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartitionFilter
(batchId=155)
org.apache.hadoop.hive.metastore.TestMetastoreExpr.testPartitionExpr
(batchId=244)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testPartitionFilter
(batchId=157)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreCustomAuth.testPartitionFilter
(batchId=163)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testPartitionFilter
(batchId=156)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartitionFilter
(batchId=163)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartitionFilter
(batchId=153)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testPartitionFilter
(batchId=151)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testPartitionFilter
(batchId=162)
org.apache.hadoop.hive.ql.exec.TestMsckDropPartitionsInBatches.testBatchingWhenException
(batchId=277)
org.apache.hadoop.hive.ql.exec.TestMsckDropPartitionsInBatches.testEqualNumberOfPartitions
(batchId=277)
org.apache.hadoop.hive.ql.exec.TestMsckDropPartitionsInBatches.testNumberOfDropPartitionsCalls
(batchId=277)
org.apache.hadoop.hive.ql.exec.TestMsckDropPartitionsInBatches.testSmallNumberOfPartitions
(batchId=277)
org.apache.hadoop.hive.ql.exec.TestMsckDropPartitionsInBatches.testUnevenNumberOfCreatePartitionCalls
(batchId=277)
org.apache.hive.minikdc.TestRemoteHiveMetaStoreDualAuthCustom.testPartitionFilter
(batchId=235)
org.apache.hive.minikdc.TestRemoteHiveMetaStoreDualAuthKerb.testPartitionFilter
(batchId=235)
org.apache.hive.minikdc.TestRemoteHiveMetaStoreKerberos.testPartitionFilter
(batchId=235)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22037/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22037/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22037/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001631 - PreCommit-HIVE-Build
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> -
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-23282
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch,
> HIVE-23282.5.patch, image-2020-04-23-14-07-06-077.png
>
>
> ObjectStore::getPartitionsByExprInternal internally uses Table information
> for getting partitionKeys, table, catalog name.
>
> For this, it ends up populating entire table data from DB (including skew
> column, parameters, sort, bucket cols etc). This makes it a lot more
> expensive call. It would be good to check if MTable itself can be used
> instead of Table.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669]
>
> !image-2020-04-23-14-07-06-077.png|width=665,height=592!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096789#comment-17096789
]
Hive QA commented on HIVE-23282:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
33s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 21 new + 598 unchanged - 6 fixed = 619 total (was 604) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
23s{color} | {color:red} standalone-metastore/metastore-server generated 2 new
+ 187 unchanged - 0 fixed = 189 total (was 187) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Possible null pointer dereference of null in
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(String,
String, String, byte[], String, short, List, boolean, boolean) Dereferenced
at ObjectStore.java:null in
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(String,
String, String, byte[], String, short, List, boolean, boolean) Dereferenced
at ObjectStore.java:[line 3468] |
| | Possible null pointer dereference of null in
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilterInternal(String,
String, String, String, short, boolean, boolean) Dereferenced at
ObjectStore.java:null in
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilterInternal(String,
String, String, String, short, boolean, boolean) Dereferenced at
ObjectStore.java:[line 4061] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22037/dev-support/hive-personality.sh
|
| git revision | master / 252fcea |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22037/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22037/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093951#comment-17093951
]
Hive QA commented on HIVE-23282:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001308/HIVE-23282.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 590 failed/errored test(s), 17158 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_10]
(batchId=230)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=230)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=230)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_nonstd_partitions_loc]
(batchId=236)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_nonstd_partitions_loc]
(batchId=236)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_nonstd_partitions_loc]
(batchId=236)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats2] (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats3] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats4] (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStatsPart]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_part]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[archive_excludeHadoop20]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[archive_multi]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_8]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join14] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19_inclause]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join32] (batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join9] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_schema_evolution_native]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark1]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark3]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin10]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin11]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin12]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin13]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin8]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin9]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_1]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_3]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_4]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_5]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_8]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_const] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_input26] (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_ppd_non_deterministic]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_outer_join_ppr]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_simple_select]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_union_view]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[combine2] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop_2]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_dp]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer11]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_partition_skip_default]
(batchId=38)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093923#comment-17093923
]
Hive QA commented on HIVE-23282:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
23s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
27s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 5 new + 601 unchanged - 3 fixed = 606 total (was 604) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21975/dev-support/hive-personality.sh
|
| git revision | master / d6ad73c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21975/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21975/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> -
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-23282
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-23282.1.patch, HIVE-23282.3.patch,
> image-2020-04-23-14-07-06-077.png
>
>
> ObjectStore::getPartitionsByExprInternal internally uses Table information
> for getting partitionKeys, table, catalog name.
>
> For this, it ends up populating entire table data from DB (including skew
> column, parameters, sort, bucket cols etc). This makes it a lot more
> expensive call. It would be good to check if MTable itself can be used
> instead of Table.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669]
>
> !image-2020-04-23-14-07-06-077.png|width=665,height=592!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092784#comment-17092784
]
Hive QA commented on HIVE-23282:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001222/HIVE-23282.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 691 failed/errored test(s), 17143 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_10]
(batchId=231)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=231)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_part]
(batchId=237)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_nonstd_partitions_loc]
(batchId=237)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_format_part]
(batchId=237)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_nonstd_partitions_loc]
(batchId=237)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_format_part]
(batchId=237)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_nonstd_partitions_loc]
(batchId=237)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats2] (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats3] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats4] (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStatsPart]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_part]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[archive_excludeHadoop20]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[archive_multi]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_8]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join14] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19_inclause]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join32] (batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join9] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_schema_evolution_native]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark1]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark3]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin10]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin11]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin12]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin13]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin8]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin9]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_1]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_3]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_4]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_5]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_8]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_const] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_input26] (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_ppd_non_deterministic]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_outer_join_ppr]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_simple_select]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_union_view]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_partlvl]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_partlvl_dp]
(batchId=25)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[
https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092777#comment-17092777
]
Hive QA commented on HIVE-23282:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
22s{color} | {color:blue} standalone-metastore/metastore-server in master has
189 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
26s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 4 new + 601 unchanged - 3 fixed = 605 total (was 604) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21955/dev-support/hive-personality.sh
|
| git revision | master / abbe72e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21955/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21955/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
> -
>
> Key: HIVE-23282
> URL: https://issues.apache.org/jira/browse/HIVE-23282
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23282.1.patch, image-2020-04-23-14-07-06-077.png
>
>
> ObjectStore::getPartitionsByExprInternal internally uses Table information
> for getting partitionKeys, table, catalog name.
>
> For this, it ends up populating entire table data from DB (including skew
> column, parameters, sort, bucket cols etc). This makes it a lot more
> expensive call. It would be good to check if MTable itself can be used
> instead of Table.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669]
>
> !image-2020-04-23-14-07-06-077.png|width=665,height=592!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092649#comment-17092649 ] Rajesh Balamohan commented on HIVE-23282: - [~ashutoshc]: PartitionKeys are needed in sqlfilter. Posted the patch and the RB link. > Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal > - > > Key: HIVE-23282 > URL: https://issues.apache.org/jira/browse/HIVE-23282 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23282.1.patch, image-2020-04-23-14-07-06-077.png > > > ObjectStore::getPartitionsByExprInternal internally uses Table information > for getting partitionKeys, table, catalog name. > > For this, it ends up populating entire table data from DB (including skew > column, parameters, sort, bucket cols etc). This makes it a lot more > expensive call. It would be good to check if MTable itself can be used > instead of Table. > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669] > > !image-2020-04-23-14-07-06-077.png|width=665,height=592! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23282) Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal
[ https://issues.apache.org/jira/browse/HIVE-23282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092568#comment-17092568 ] Ashutosh Chauhan commented on HIVE-23282: - getPartitionsByExprInternal() doesn't need Table object at all. it can run(false) It only needs tbl name, db name and cat name which are already present. No need to fetch full Table object. > Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal > - > > Key: HIVE-23282 > URL: https://issues.apache.org/jira/browse/HIVE-23282 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Priority: Major > Attachments: image-2020-04-23-14-07-06-077.png > > > ObjectStore::getPartitionsByExprInternal internally uses Table information > for getting partitionKeys, table, catalog name. > > For this, it ends up populating entire table data from DB (including skew > column, parameters, sort, bucket cols etc). This makes it a lot more > expensive call. It would be good to check if MTable itself can be used > instead of Table. > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3327] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L3669] > > !image-2020-04-23-14-07-06-077.png|width=665,height=592! -- This message was sent by Atlassian Jira (v8.3.4#803005)
