karuppayya commented on code in PR #12450:
URL: https://github.com/apache/iceberg/pull/12450#discussion_r2098882746
##########
api/src/main/java/org/apache/iceberg/actions/ComputePartitionStats.java:
##########
@@ -34,6 +34,16 @@ public interface ComputePartitionStats
*/
ComputePartitionStats snapshot(long snapshotId);
+ /**
+ * Clears existing partition statistics for all snapshots, allowing the
action to recompute them
+ * from scratch.
+ *
+ * <p>Use this option if existing statistics files are corrupted or invalid.
+ *
+ * @return this for method chaining
+ */
+ ComputePartitionStats forceRefresh();
Review Comment:
Should we say `forceCompute`
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ComputePartitionStatsSparkAction.java:
##########
@@ -80,6 +88,16 @@ public Result execute() {
}
private Result doExecute() {
+ if (forceRefresh) {
+ LOG.info(
+ "Clearing the existing partition statistics for all snapshots to
enforce force refresh");
+ UpdatePartitionStatistics clearStats = table.updatePartitionStatistics();
+ table
+ .partitionStatisticsFiles()
Review Comment:
Should do we remove the statistics file for the given snapshot and initiate
statistics collection fresh(instead of incremental) for that given snapshot?
Do we also need to clear stats of other snapshots?
##########
api/src/main/java/org/apache/iceberg/actions/ComputePartitionStats.java:
##########
@@ -34,6 +34,16 @@ public interface ComputePartitionStats
*/
ComputePartitionStats snapshot(long snapshotId);
+ /**
+ * Clears existing partition statistics for all snapshots, allowing the
action to recompute them
Review Comment:
Do we want to highlight that this will create more than one commit(one for
the removal and one for the update)
##########
api/src/main/java/org/apache/iceberg/actions/ComputePartitionStats.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import org.apache.iceberg.PartitionStatisticsFile;
+
+/**
+ * An action that computes and writes the partition statistics of an Iceberg
table. Current snapshot
+ * is used by default.
+ */
+public interface ComputePartitionStats
+ extends Action<ComputePartitionStats, ComputePartitionStats.Result> {
Review Comment:
I am not sure if completeness of functionality should determine this.
I am mainly coming from the end user perspective of having to use different
action for stats whereas it is different in most other engine/formats.
But I dont have a strong opinion on this, I am fine either ways.
Would be good to get @aokolnychyi 's thought here since it was raised by him
[earlier](https://github.com/apache/iceberg/pull/10986#issuecomment-2305512491)
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/ComputePartitionStatsSparkAction.java:
##########
@@ -80,6 +88,16 @@ public Result execute() {
}
private Result doExecute() {
+ if (forceRefresh) {
+ LOG.info(
+ "Clearing the existing partition statistics for all snapshots to
enforce force refresh");
+ UpdatePartitionStatistics clearStats = table.updatePartitionStatistics();
Review Comment:
clearStats -> updatePartitionStatistics or something similar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
