cshuo commented on code in PR #20672:
URL: https://github.com/apache/hudi/pull/20672#discussion_r4613438986
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BloomFiltersIndexSupport.scala:
##########
@@ -23,16 +23,32 @@ import org.apache.hudi.HoodieConversionUtils.toScalaOption
import org.apache.hudi.common.config.HoodieMetadataConfig
import org.apache.hudi.common.model.FileSlice
import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.config.HoodieIndexConfig
import org.apache.hudi.core.read.BaseHoodieTableFileIndex
+import org.apache.hudi.index.HoodieIndex.IndexType
import org.apache.hudi.metadata.HoodieTableMetadataUtil
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.expressions.Expression
+import scala.collection.JavaConverters._
+
class BloomFiltersIndexSupport(spark: SparkSession,
metadataConfig: HoodieMetadataConfig,
metaClient: HoodieTableMetaClient) extends
SparkBaseIndexSupport(spark, metadataConfig, metaClient) {
+ // Like bucket pruning, trust the declared reader index type to match the
writer. These Spark
+ // indexes write new keys to base files. A custom class overrides the type
and has no such guarantee.
+ // Do not infer the writer's default SIMPLE index when the reader has not
declared a type.
+ private val canPruneLogSlices = {
+ val indexType =
metadataConfig.getStringOrDefault(HoodieIndexConfig.INDEX_TYPE, "")
+ metadataConfig.getStringOrDefault(HoodieIndexConfig.INDEX_CLASS_NAME,
"").isEmpty &&
+
BloomFiltersIndexSupport.BASE_BLOOM_COVERED_INDEX_TYPES.exists(_.name().equalsIgnoreCase(indexType))
+ }
+
+ private def mustRetainSlice(fileSlice: FileSlice): Boolean =
+ fileSlice.hasLogFiles && (fileSlice.getBaseFile.isEmpty ||
!canPruneLogSlices)
Review Comment:
The allowlist assumes that the reader’s hoodie.index.type matches the index
used to write the file slice. The reader has no reliable way to determine that
from the slice itself. Spark’s simple bucket pruning makes a similar assumption
when it trusts the reader’s bucket configuration to match the on-disk layout.
As you pointed out, a reader configured with SIMPLE can still encounter logs
written earlier with BUCKET. If we want snapshot queries to be correct
regardless of that history, the conservative approach is to retain every file
slice that contains logs and use base blooms to prune base-only slices. The
trade-off is less effective pruning when the logs only update keys already
present in the base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]