[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=844996&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-844996 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 13/Feb/23 05:55 Start Date: 13/Feb/23 05:55 Worklog Time Spent: 10m Work Description: kasakrisz merged PR #4000: URL: https://github.com/apache/hive/pull/4000 Issue Time Tracking --- Worklog Id: (was: 844996) Time Spent: 5h 20m (was: 5h 10m) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 5h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=844705&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-844705 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 10/Feb/23 07:07 Start Date: 10/Feb/23 07:07 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1425286954 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4000) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 844705) Time Spent: 5h 10m (was: 5h) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 5h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=844672&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-844672 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 10/Feb/23 00:02 Start Date: 10/Feb/23 00:02 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1424997332 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4000) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 844672) Time Spent: 5h (was: 4h 50m) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=844644&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-844644 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 09/Feb/23 19:24 Start Date: 09/Feb/23 19:24 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1424699612 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4000) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 844644) Time Spent: 4h 50m (was: 4h 40m) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=844583&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-844583
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 09/Feb/23 13:56
Start Date: 09/Feb/23 13:56
Worklog Time Spent: 10m
Work Description: simhadri-g commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1101495223
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -454,6 +468,27 @@ private static Statistics collectStatistics(HiveConf conf,
PrunedPartitionList p
return stats;
}
+ private static Map getValidBasicStatsFromStorageHandler(Table
table) {
+
+Map storageHandlerBasicStatsMap =
+(table.isNonNative() &&
table.getStorageHandler().canProvideBasicStatistics()) ?
table.getStorageHandler()
+.getBasicStatistics(Partish.buildFor(table)) : null;
Review Comment:
Done.
##
iceberg/iceberg-handler/src/test/queries/positive/use_basic_stats_from_iceberg.q:
##
@@ -0,0 +1,32 @@
+
Issue Time Tracking
---
Worklog Id: (was: 844583)
Time Spent: 4h 40m (was: 4.5h)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=844580&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-844580
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 09/Feb/23 13:54
Start Date: 09/Feb/23 13:54
Worklog Time Spent: 10m
Work Description: simhadri-g commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1101492692
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -454,6 +468,27 @@ private static Statistics collectStatistics(HiveConf conf,
PrunedPartitionList p
return stats;
}
+ private static Map getValidBasicStatsFromStorageHandler(Table
table) {
+
+Map storageHandlerBasicStatsMap =
+(table.isNonNative() &&
table.getStorageHandler().canProvideBasicStatistics()) ?
table.getStorageHandler()
+.getBasicStatistics(Partish.buildFor(table)) : null;
Review Comment:
Done.
Issue Time Tracking
---
Worklog Id: (was: 844580)
Time Spent: 4.5h (was: 4h 20m)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=843926&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843926 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 06/Feb/23 19:09 Start Date: 06/Feb/23 19:09 Worklog Time Spent: 10m Work Description: soumyakanti3578 commented on code in PR #4000: URL: https://github.com/apache/hive/pull/4000#discussion_r1097809424 ## iceberg/iceberg-handler/src/test/queries/positive/use_basic_stats_from_iceberg.q: ## @@ -0,0 +1,32 @@ + Issue Time Tracking --- Worklog Id: (was: 843926) Time Spent: 4h 20m (was: 4h 10m) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=843758&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843758
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 06/Feb/23 10:05
Start Date: 06/Feb/23 10:05
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1097173244
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -454,6 +468,27 @@ private static Statistics collectStatistics(HiveConf conf,
PrunedPartitionList p
return stats;
}
+ private static Map getValidBasicStatsFromStorageHandler(Table
table) {
+
+Map storageHandlerBasicStatsMap =
+(table.isNonNative() &&
table.getStorageHandler().canProvideBasicStatistics()) ?
table.getStorageHandler()
+.getBasicStatistics(Partish.buildFor(table)) : null;
Review Comment:
How about moving this check and fetch into `BasicStats` and
`StatsUtils.collectStatistics` don't have don't have to be changed at all?
Example:
https://github.com/kasakrisz/hive/commit/e03441477a0c616f1b3012dc4ddc6dfe0f01743b
Issue Time Tracking
---
Worklog Id: (was: 843758)
Time Spent: 4h 10m (was: 4h)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=843511&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843511 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 03/Feb/23 15:18 Start Date: 03/Feb/23 15:18 Worklog Time Spent: 10m Work Description: InvisibleProgrammer commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1416015458 Nice, thank you. Issue Time Tracking --- Worklog Id: (was: 843511) Time Spent: 4h (was: 3h 50m) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=843498&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843498 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 03/Feb/23 14:53 Start Date: 03/Feb/23 14:53 Worklog Time Spent: 10m Work Description: simhadri-g commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1415983594 @InvisibleProgrammer , on time-travel the current snapshot id will change to the snapshot we are rolling back to. So currentSnapshot summary will provide the correct stats. Issue Time Tracking --- Worklog Id: (was: 843498) Time Spent: 3h 50m (was: 3h 40m) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=843496&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843496 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 03/Feb/23 14:40 Start Date: 03/Feb/23 14:40 Worklog Time Spent: 10m Work Description: InvisibleProgrammer commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1415962968 Now we use the current snapshot's statistics, by default. Is that means in case of time travel queries we use the current snapshot statistics as too? I assume we have no better solution, I think Hive has no time travel statistics as well. Anyway, LGTM. Issue Time Tracking --- Worklog Id: (was: 843496) Time Spent: 3h 40m (was: 3.5h) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 3h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=843486&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843486 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 03/Feb/23 13:06 Start Date: 03/Feb/23 13:06 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1415846127 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4000) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 843486) Time Spent: 3.5h (was: 3h 20m) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=843408&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843408 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 03/Feb/23 09:10 Start Date: 03/Feb/23 09:10 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1415421782 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4000) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 843408) Time Spent: 3h 20m (was: 3h 10m) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=843385&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843385
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 03/Feb/23 06:44
Start Date: 03/Feb/23 06:44
Worklog Time Spent: 10m
Work Description: simhadri-g commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1095407711
##
iceberg/iceberg-handler/src/test/results/positive/use_basic_stats_from_iceberg.q.out:
##
@@ -0,0 +1,162 @@
+PREHOOK: query: drop table if exists tbl_ice
+PREHOOK: type: DROPTABLE
+POSTHOOK: query: drop table if exists tbl_ice
+POSTHOOK: type: DROPTABLE
+PREHOOK: query: create external table tbl_ice(a int, b string, c int) stored
by iceberg tblproperties ('format-version'='2')
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@tbl_ice
+POSTHOOK: query: create external table tbl_ice(a int, b string, c int) stored
by iceberg tblproperties ('format-version'='2')
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@tbl_ice
+PREHOOK: query: select count(*) from tbl_ice
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl_ice
+PREHOOK: Output: hdfs://### HDFS PATH ###
+POSTHOOK: query: select count(*) from tbl_ice
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl_ice
+POSTHOOK: Output: hdfs://### HDFS PATH ###
+0
+PREHOOK: query: insert into tbl_ice values (1, 'one', 50), (2, 'two', 51),(2,
'two', 51),(2, 'two', 51), (3, 'three', 52), (4, 'four', 53), (5, 'five', 54),
(111, 'one', 55), (333, 'two', 56)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@tbl_ice
+POSTHOOK: query: insert into tbl_ice values (1, 'one', 50), (2, 'two', 51),(2,
'two', 51),(2, 'two', 51), (3, 'three', 52), (4, 'four', 53), (5, 'five', 54),
(111, 'one', 55), (333, 'two', 56)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@tbl_ice
+PREHOOK: query: explain select * from tbl_ice order by a, b, c
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl_ice
+PREHOOK: Output: hdfs://### HDFS PATH ###
+POSTHOOK: query: explain select * from tbl_ice order by a, b, c
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl_ice
+POSTHOOK: Output: hdfs://### HDFS PATH ###
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+
+Stage-0
+ Fetch Operator
+limit:-1
+Stage-1
+ Reducer 2 vectorized
+ File Output Operator [FS_8]
+Select Operator [SEL_7] (rows=9 width=95)
Review Comment:
Sure.
Issue Time Tracking
---
Worklog Id: (was: 843385)
Time Spent: 3h 10m (was: 3h)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=843384&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843384
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 03/Feb/23 06:40
Start Date: 03/Feb/23 06:40
Worklog Time Spent: 10m
Work Description: simhadri-g commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1095405415
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -454,6 +468,27 @@ private static Statistics collectStatistics(HiveConf conf,
PrunedPartitionList p
return stats;
}
+ private static Map getValidBasicStatsFromStorageHandler(Table
table) {
+
+Map storageHandlerBasicStatsMap =
+(table.isNonNative() &&
table.getStorageHandler().canProvideBasicStatistics()) ?
table.getStorageHandler()
+.getBasicStatistics(Partish.buildFor(table)) : null;
+
+try {
+ if (storageHandlerBasicStatsMap != null &&
+
Long.parseLong(storageHandlerBasicStatsMap.get(StatsSetupConst.ROW_COUNT)) > 0)
{
Review Comment:
Yes, snapshots are not created for a new table until data is inserted. As a
result, stats are not available.
The following line turns out to be null for the new table.
` if (table.currentSnapshot() != null) `
https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java#L316
Issue Time Tracking
---
Worklog Id: (was: 843384)
Time Spent: 3h (was: 2h 50m)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=843374&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843374
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 03/Feb/23 05:00
Start Date: 03/Feb/23 05:00
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1095356638
##
iceberg/iceberg-handler/src/test/results/positive/use_basic_stats_from_iceberg.q.out:
##
@@ -0,0 +1,162 @@
+PREHOOK: query: drop table if exists tbl_ice
+PREHOOK: type: DROPTABLE
+POSTHOOK: query: drop table if exists tbl_ice
+POSTHOOK: type: DROPTABLE
+PREHOOK: query: create external table tbl_ice(a int, b string, c int) stored
by iceberg tblproperties ('format-version'='2')
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@tbl_ice
+POSTHOOK: query: create external table tbl_ice(a int, b string, c int) stored
by iceberg tblproperties ('format-version'='2')
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@tbl_ice
+PREHOOK: query: select count(*) from tbl_ice
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl_ice
+PREHOOK: Output: hdfs://### HDFS PATH ###
+POSTHOOK: query: select count(*) from tbl_ice
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl_ice
+POSTHOOK: Output: hdfs://### HDFS PATH ###
+0
+PREHOOK: query: insert into tbl_ice values (1, 'one', 50), (2, 'two', 51),(2,
'two', 51),(2, 'two', 51), (3, 'three', 52), (4, 'four', 53), (5, 'five', 54),
(111, 'one', 55), (333, 'two', 56)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@tbl_ice
+POSTHOOK: query: insert into tbl_ice values (1, 'one', 50), (2, 'two', 51),(2,
'two', 51),(2, 'two', 51), (3, 'three', 52), (4, 'four', 53), (5, 'five', 54),
(111, 'one', 55), (333, 'two', 56)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@tbl_ice
+PREHOOK: query: explain select * from tbl_ice order by a, b, c
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl_ice
+PREHOOK: Output: hdfs://### HDFS PATH ###
+POSTHOOK: query: explain select * from tbl_ice order by a, b, c
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl_ice
+POSTHOOK: Output: hdfs://### HDFS PATH ###
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+
+Stage-0
+ Fetch Operator
+limit:-1
+Stage-1
+ Reducer 2 vectorized
+ File Output Operator [FS_8]
+Select Operator [SEL_7] (rows=9 width=95)
Review Comment:
Please add an other test case when joining empty tables or at least one of
the tables are empty.
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -454,6 +468,27 @@ private static Statistics collectStatistics(HiveConf conf,
PrunedPartitionList p
return stats;
}
+ private static Map getValidBasicStatsFromStorageHandler(Table
table) {
+
+Map storageHandlerBasicStatsMap =
+(table.isNonNative() &&
table.getStorageHandler().canProvideBasicStatistics()) ?
table.getStorageHandler()
+.getBasicStatistics(Partish.buildFor(table)) : null;
+
+try {
+ if (storageHandlerBasicStatsMap != null &&
+
Long.parseLong(storageHandlerBasicStatsMap.get(StatsSetupConst.ROW_COUNT)) > 0)
{
Review Comment:
Does it mean that an empty table has no stats?
Issue Time Tracking
---
Worklog Id: (was: 843374)
Time Spent: 2h 50m (was: 2h 40m)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=843294&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843294
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 02/Feb/23 17:21
Start Date: 02/Feb/23 17:21
Worklog Time Spent: 10m
Work Description: simhadri-g commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1094837888
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -454,6 +468,26 @@ private static Statistics collectStatistics(HiveConf conf,
PrunedPartitionList p
return stats;
}
+ private static Map getValidBasisStatsFromStorageHandler(Table
table) {
Review Comment:
Thanks @akshat0395 , i have updated the PR :) .
Issue Time Tracking
---
Worklog Id: (was: 843294)
Time Spent: 2h 40m (was: 2.5h)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=843275&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843275 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 02/Feb/23 15:40 Start Date: 02/Feb/23 15:40 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1413945063 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4000) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 843275) Time Spent: 2.5h (was: 2h 20m) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=843023&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-843023 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 01/Feb/23 21:10 Start Date: 01/Feb/23 21:10 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1412727326 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4000) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 843023) Time Spent: 2h 20m (was: 2h 10m) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=842992&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842992
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 01/Feb/23 18:44
Start Date: 01/Feb/23 18:44
Worklog Time Spent: 10m
Work Description: akshat0395 commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1093602583
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -454,6 +468,26 @@ private static Statistics collectStatistics(HiveConf conf,
PrunedPartitionList p
return stats;
}
+ private static Map getValidBasisStatsFromStorageHandler(Table
table) {
Review Comment:
Seems like a typo in method name, Should it be
`getValidBasicStatsFromStorageHandler`?
Issue Time Tracking
---
Worklog Id: (was: 842992)
Time Spent: 2h 10m (was: 2h)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=842901&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842901
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 01/Feb/23 14:31
Start Date: 01/Feb/23 14:31
Worklog Time Spent: 10m
Work Description: simhadri-g commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1093299476
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -277,23 +277,31 @@ private static Statistics collectStatistics(HiveConf
conf, PrunedPartitionList p
boolean metaTable = table.getMetaTable() != null;
if (!table.isPartitioned()) {
+ long ds, nr, fs;
+ if (table.isNonNative() &&
table.getStorageHandler().canProvideBasicStatistics()) {
+Map icebergBasicStatMap =
table.getStorageHandler().getBasicStatistics(Partish.buildFor(table));
+ds = Long.parseLong(icebergBasicStatMap.get("totalSize"));
+nr = Long.parseLong(icebergBasicStatMap.get("numRows"));
+fs = Long.parseLong(icebergBasicStatMap.get("numFiles"));
Review Comment:
Thanks, @kasakrisz , I have updated the PR to catch numberFormatException
and fall back to the pre-existing implementation of basic stats in case the
storage handler cannot provide the stats.
Please have a look at the updated PR.
Issue Time Tracking
---
Worklog Id: (was: 842901)
Time Spent: 2h (was: 1h 50m)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=842897&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842897
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 01/Feb/23 14:27
Start Date: 01/Feb/23 14:27
Worklog Time Spent: 10m
Work Description: simhadri-g commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1093299476
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -277,23 +277,31 @@ private static Statistics collectStatistics(HiveConf
conf, PrunedPartitionList p
boolean metaTable = table.getMetaTable() != null;
if (!table.isPartitioned()) {
+ long ds, nr, fs;
+ if (table.isNonNative() &&
table.getStorageHandler().canProvideBasicStatistics()) {
+Map icebergBasicStatMap =
table.getStorageHandler().getBasicStatistics(Partish.buildFor(table));
+ds = Long.parseLong(icebergBasicStatMap.get("totalSize"));
+nr = Long.parseLong(icebergBasicStatMap.get("numRows"));
+fs = Long.parseLong(icebergBasicStatMap.get("numFiles"));
Review Comment:
Thanks @kasakrisz , i have updated the PR to catch numberFormatException and
fall back to the pre existing implementation of basic stats.
Issue Time Tracking
---
Worklog Id: (was: 842897)
Time Spent: 1h 50m (was: 1h 40m)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=842847&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842847 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 01/Feb/23 11:48 Start Date: 01/Feb/23 11:48 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1411928657 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4000) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842847) Time Spent: 1h 40m (was: 1.5h) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=842768&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842768 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 01/Feb/23 07:00 Start Date: 01/Feb/23 07:00 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1411554829 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4000) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842768) Time Spent: 1.5h (was: 1h 20m) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=842725&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842725 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 31/Jan/23 23:26 Start Date: 31/Jan/23 23:26 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1411212708 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4000) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842725) Time Spent: 1h 20m (was: 1h 10m) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=842663&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842663 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 31/Jan/23 18:13 Start Date: 31/Jan/23 18:13 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1410856046 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4000) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842663) Time Spent: 1h 10m (was: 1h) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=842621&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842621
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 31/Jan/23 15:08
Start Date: 31/Jan/23 15:08
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1092068769
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -277,23 +277,31 @@ private static Statistics collectStatistics(HiveConf
conf, PrunedPartitionList p
boolean metaTable = table.getMetaTable() != null;
if (!table.isPartitioned()) {
+ long ds, nr, fs;
+ if (table.isNonNative() &&
table.getStorageHandler().canProvideBasicStatistics()) {
+Map icebergBasicStatMap =
table.getStorageHandler().getBasicStatistics(Partish.buildFor(table));
+ds = Long.parseLong(icebergBasicStatMap.get("totalSize"));
+nr = Long.parseLong(icebergBasicStatMap.get("numRows"));
+fs = Long.parseLong(icebergBasicStatMap.get("numFiles"));
Review Comment:
1. It seems to me that it check if the currentSnapshot exists or not but
what if summary does not contain for example
`SnapshotSummary.TOTAL_DATA_FILES_PROP`.
2. The line
```
Long.parseLong(icebergBasicStatMap.get("totalSize"))
```
can throw `NumberFormatException` because
* `icebergBasicStatMap.get("totalSize")` returns null because the map does
have an entry with `totalSize`
* or it has such entry but the value is not a valid `Long` value.
3. I think the name `icebergBasicStatMap` is too specific here because at
this point we should not expect an `IcebergStorageHandler` but any
`HiveStorageHandler` which supports providing BasicStatistics. So a name like
`basicStatMap` would be more generic.
Issue Time Tracking
---
Worklog Id: (was: 842621)
Time Spent: 1h (was: 50m)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=842592&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842592
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 31/Jan/23 14:03
Start Date: 31/Jan/23 14:03
Worklog Time Spent: 10m
Work Description: simhadri-g commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1091975847
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -277,23 +277,31 @@ private static Statistics collectStatistics(HiveConf
conf, PrunedPartitionList p
boolean metaTable = table.getMetaTable() != null;
if (!table.isPartitioned()) {
+ long ds, nr, fs;
Review Comment:
Done
Issue Time Tracking
---
Worklog Id: (was: 842592)
Time Spent: 50m (was: 40m)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=842589&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842589 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 31/Jan/23 13:53 Start Date: 31/Jan/23 13:53 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4000: URL: https://github.com/apache/hive/pull/4000#issuecomment-1410397698 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4000) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4000&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4000&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4000&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842589) Time Spent: 40m (was: 0.5h) > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=842579&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842579
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 31/Jan/23 13:24
Start Date: 31/Jan/23 13:24
Worklog Time Spent: 10m
Work Description: simhadri-g commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1091920961
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -277,23 +277,31 @@ private static Statistics collectStatistics(HiveConf
conf, PrunedPartitionList p
boolean metaTable = table.getMetaTable() != null;
if (!table.isPartitioned()) {
+ long ds, nr, fs;
+ if (table.isNonNative() &&
table.getStorageHandler().canProvideBasicStatistics()) {
+Map icebergBasicStatMap =
table.getStorageHandler().getBasicStatistics(Partish.buildFor(table));
+ds = Long.parseLong(icebergBasicStatMap.get("totalSize"));
+nr = Long.parseLong(icebergBasicStatMap.get("numRows"));
+fs = Long.parseLong(icebergBasicStatMap.get("numFiles"));
Review Comment:
Thanks for the review :) .
I thought that "null" check here would be sufficient:
https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java#L316
I will update the PR and include a null check.
Issue Time Tracking
---
Worklog Id: (was: 842579)
Time Spent: 0.5h (was: 20m)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[
https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=842566&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842566
]
ASF GitHub Bot logged work on HIVE-27007:
-
Author: ASF GitHub Bot
Created on: 31/Jan/23 12:45
Start Date: 31/Jan/23 12:45
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #4000:
URL: https://github.com/apache/hive/pull/4000#discussion_r1091850767
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -277,23 +277,31 @@ private static Statistics collectStatistics(HiveConf
conf, PrunedPartitionList p
boolean metaTable = table.getMetaTable() != null;
if (!table.isPartitioned()) {
+ long ds, nr, fs;
+ if (table.isNonNative() &&
table.getStorageHandler().canProvideBasicStatistics()) {
+Map icebergBasicStatMap =
table.getStorageHandler().getBasicStatistics(Partish.buildFor(table));
+ds = Long.parseLong(icebergBasicStatMap.get("totalSize"));
+nr = Long.parseLong(icebergBasicStatMap.get("numRows"));
+fs = Long.parseLong(icebergBasicStatMap.get("numFiles"));
Review Comment:
Should a null check be added here and/or handle NumberFormatException?
What would be the stats values if it can not be extracted via storage
handler?
In case of non-storage handler path there are so called `enhancers` to
estimate and fix the values:
```
basicStatsFactory.addEnhancer(new
BasicStats.RowNumEstimator(estimateRowSizeFromSchema(conf, schema)));
basicStatsFactory.addEnhancer(new BasicStats.SetMinRowNumber01());
```
##
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##
@@ -277,23 +277,31 @@ private static Statistics collectStatistics(HiveConf
conf, PrunedPartitionList p
boolean metaTable = table.getMetaTable() != null;
if (!table.isPartitioned()) {
+ long ds, nr, fs;
Review Comment:
Could you please declare each variable in separate lines.
Issue Time Tracking
---
Worklog Id: (was: 842566)
Time Spent: 20m (was: 10m)
> Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for
> query planning
> --
>
> Key: HIVE-27007
> URL: https://issues.apache.org/jira/browse/HIVE-27007
> Project: Hive
> Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27007) Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for query planning
[ https://issues.apache.org/jira/browse/HIVE-27007?focusedWorklogId=842500&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842500 ] ASF GitHub Bot logged work on HIVE-27007: - Author: ASF GitHub Bot Created on: 31/Jan/23 10:28 Start Date: 31/Jan/23 10:28 Worklog Time Spent: 10m Work Description: simhadri-g opened a new pull request, #4000: URL: https://github.com/apache/hive/pull/4000 …shot.summary() for query planning ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? q test and manual tests Issue Time Tracking --- Worklog Id: (was: 842500) Remaining Estimate: 0h Time Spent: 10m > Iceberg: Use BasicStats from iceberg table's currrentSnapshot.summary() for > query planning > -- > > Key: HIVE-27007 > URL: https://issues.apache.org/jira/browse/HIVE-27007 > Project: Hive > Issue Type: Improvement >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
