puneetdixit200 opened a new pull request, #18901: URL: https://github.com/apache/hudi/pull/18901
### Describe the issue this Pull Request addresses Closes #18758. ### Summary and Changelog Collect Lance column ranges as records are written, then pass them to the metadata index through `HoodieWriteStat`. This addresses the request to avoid reading newly written files again. - Add an optional writer callback after row encoding and metadata population. - Reuse the existing range collector for Spark and Flink inserts, merges, and bulk writes; copy mutable bounds before retaining them. - Enable column stats and partition stats for Lance tables. Existing files use projected reads when an index is initialized or write statistics are unavailable. - Handle null Flink field values and select the installed engine's Lance reader for initialization. ### Impact Lance tables can use the existing column-stats configuration and, when partitioned, the partition-stats index. The writer interface gains a default callback method. No new configuration keys or file-format changes are introduced. ### Risk Level medium This enables metadata indexes that affect file and partition pruning. Tests cover populated and omitted metadata fields, nulls, reused input buffers, both index versions, and SQL inserts, updates, deletes, and compaction for partitioned and unpartitioned COW/MOR tables. ### Documentation Update Companion update to `website/docs/metadata_indexing.md` on `asf-site`: https://github.com/apache/hudi/pull/20440 ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable ### Validation 187 tests passed with no failures or skips on Java 17: common (94), Hadoop metadata utilities (19), collector/indexer (12), Spark (31), and Flink (31). Checkstyle, license checks, and `git diff --check` passed. The local test classpath needed `hadoop-client-runtime-3.3.5.jar` alongside the resolved Hadoop client API. No dependency declarations were changed. In the commands below, `HADOOP_TEST_RUNTIME` points to that jar. ```sh mvn -B -Punit-tests -pl hudi-spark-datasource/hudi-spark -am \ '-Dtest=TestColumnStatsCollector,TestHoodieTableMetadataUtil,TestMetadataPartitionType,TestColumnStatsStreamingTracker,TestColumnStatsIndexer,TestHoodieSparkLanceWriter,TestHoodieRowCreateHandle,TestLanceDataSource#testSqlCommands' \ -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false -DwildcardSuites=abc \ -Dspark3.5 -Dlance.skip.tests=false -Dmaven.test.additionalClasspath="$HADOOP_TEST_RUNTIME" test mvn -B -Punit-tests -pl hudi-flink-datasource/hudi-flink -am \ '-Dtest=TestHoodieRowDataLanceWriter,TestHoodieRowDataCreateHandle,TestHoodieRowDataLanceReader' \ -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false -DwildcardSuites=abc \ -Dflink2.2 -Dlance.skip.tests=false -Dmaven.test.additionalClasspath="$HADOOP_TEST_RUNTIME" test ``` The full project test suite was not run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
