puneetdixit200 opened a new pull request, #18901:
URL: https://github.com/apache/hudi/pull/18901

   ### Describe the issue this Pull Request addresses
   
   Closes #18758.
   
   ### Summary and Changelog
   
   Collect Lance column ranges as records are written, then pass them to the 
metadata index through `HoodieWriteStat`. This addresses the request to avoid 
reading newly written files again.
   
   - Add an optional writer callback after row encoding and metadata population.
   - Reuse the existing range collector for Spark and Flink inserts, merges, 
and bulk writes; copy mutable bounds before retaining them.
   - Enable column stats and partition stats for Lance tables. Existing files 
use projected reads when an index is initialized or write statistics are 
unavailable.
   - Handle null Flink field values and select the installed engine's Lance 
reader for initialization.
   
   ### Impact
   
   Lance tables can use the existing column-stats configuration and, when 
partitioned, the partition-stats index. The writer interface gains a default 
callback method. No new configuration keys or file-format changes are 
introduced.
   
   ### Risk Level
   
   medium
   
   This enables metadata indexes that affect file and partition pruning. Tests 
cover populated and omitted metadata fields, nulls, reused input buffers, both 
index versions, and SQL inserts, updates, deletes, and compaction for 
partitioned and unpartitioned COW/MOR tables.
   
   ### Documentation Update
   
   Companion update to `website/docs/metadata_indexing.md` on `asf-site`: 
https://github.com/apache/hudi/pull/20440
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   ### Validation
   
   187 tests passed with no failures or skips on Java 17: common (94), Hadoop 
metadata utilities (19), collector/indexer (12), Spark (31), and Flink (31). 
Checkstyle, license checks, and `git diff --check` passed.
   
   The local test classpath needed `hadoop-client-runtime-3.3.5.jar` alongside 
the resolved Hadoop client API. No dependency declarations were changed. In the 
commands below, `HADOOP_TEST_RUNTIME` points to that jar.
   
   ```sh
   mvn -B -Punit-tests -pl hudi-spark-datasource/hudi-spark -am \
     
'-Dtest=TestColumnStatsCollector,TestHoodieTableMetadataUtil,TestMetadataPartitionType,TestColumnStatsStreamingTracker,TestColumnStatsIndexer,TestHoodieSparkLanceWriter,TestHoodieRowCreateHandle,TestLanceDataSource#testSqlCommands'
 \
     -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false 
-DwildcardSuites=abc \
     -Dspark3.5 -Dlance.skip.tests=false 
-Dmaven.test.additionalClasspath="$HADOOP_TEST_RUNTIME" test
   
   mvn -B -Punit-tests -pl hudi-flink-datasource/hudi-flink -am \
     
'-Dtest=TestHoodieRowDataLanceWriter,TestHoodieRowDataCreateHandle,TestHoodieRowDataLanceReader'
 \
     -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false 
-DwildcardSuites=abc \
     -Dflink2.2 -Dlance.skip.tests=false 
-Dmaven.test.additionalClasspath="$HADOOP_TEST_RUNTIME" test
   ```
   
   The full project test suite was not run.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to