Copilot commented on code in PR #6576:
URL: https://github.com/apache/hive/pull/6576#discussion_r3548695969
##########
ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java:
##########
@@ -215,12 +229,43 @@ public int persistColumnStats(Hive db, Table tbl) throws
HiveException, MetaExce
long maxNumStats =
conf.getLongVar(HiveConf.ConfVars.HIVE_STATS_MAX_NUM_STATS);
while (!done) {
List<ColumnStatistics> colStats = new ArrayList<>();
+ Map<String, List<String>> failedColumnStatsByTarget = new HashMap<>();
- long start = System. currentTimeMillis();
- done = constructColumnStatsFromPackedRows(tbl, colStats, maxNumStats);
+ long start = System.currentTimeMillis();
+ done = constructColumnStatsFromPackedRows(tbl, colStats, maxNumStats,
failedColumnStatsByTarget);
long end = System.currentTimeMillis();
LOG.info("Time taken to build " + colStats.size() + " stats desc : " +
((end - start)/1000F) + " seconds.");
+ // Remove inaccurate column stats markers
+ List<Partition> partitionsToUpdate = new ArrayList<>();
+ for (Map.Entry<String, List<String>> entry :
failedColumnStatsByTarget.entrySet()) {
+ List<String> failedColumns = entry.getValue();
+ if (CollectionUtils.isEmpty(failedColumns)) {
+ continue;
+ }
+
+ if (tbl.isNonNative() &&
tbl.getStorageHandler().canSetColStatistics(tbl)) {
+ if (!(tbl.isMaterializedView() || tbl.isView() ||
tbl.isTemporary())) {
+ setOrRemoveColumnStatsAccurateProperty(db, tbl, failedColumns,
false);
+ }
+ } else {
Review Comment:
For non-native partitioned tables collecting partition-level column stats
(i.e., colStatDesc.isTblLevel()==false), this branch calls
setOrRemoveColumnStatsAccurateProperty(...), but that helper returns early when
!colStatDesc.isTblLevel(). As a result, no COLUMN_STATS_ACCURATE cleanup
happens at all for non-native partitioned tables, contradicting the stated
limitation of doing table-level cleanup.
##########
ql/src/test/queries/clientpositive/stats_col_stats_inaccurate.q:
##########
@@ -0,0 +1,97 @@
+set hive.stats.autogather=true;
+set hive.stats.column.autogather=true;
+set hive.stats.fetch.column.stats=true;
+set hive.support.concurrency=true;
+set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
+set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
+
+-- Check partitioned tables on float/double columns with Infinity/NaN on
inaccurate stats.
+
+create table stats_t1(
+ c_double double,
+ c_float float,
+ c_str string)
+partitioned by (p int)
+STORED AS ORC TBLPROPERTIES ('transactional'='true');
+
+insert into table stats_t1 partition(p=1) values
+ (cast('Infinity' as double), cast('Infinity' as float), 'row1'),
+ (cast('-Infinity' as double), cast('-Infinity' as float), 'row2'),
+ (cast('NAN' as double), cast('NaN' as float), 'row3'),
+ (cast(1234 as double), 123.456, 'row4');
+
+describe formatted stats_t1 partition(p=1) c_double;
+describe formatted stats_t1 partition(p=1) c_float;
+describe formatted stats_t1 partition(p=1);
+
+
+-- Check non-partitioned tables on flout/double columns with Infinity/NaN on
inaccurate stats.
Review Comment:
Typo in the comment: "flout" should be "float".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
