Gabriel39 opened a new pull request, #66802: URL: https://github.com/apache/doris/pull/66802
## Proposed changes This PR adds the BE-only part of Parquet row-group split refinement for Iceberg and Paimon scans. It assumes FE schedules splits from the same file to the same BE instance. - Refine an FE source split into row-group children after reading Parquet metadata. - Support generated children for both local and remote SplitSourceConnector implementations without reporting EOS while a parent is being refined. - Add a concurrent single-flight file-context registry owned by FileScanLocalState. - Share immutable Parquet footer metadata across source splits and generated children, including Iceberg delete-file readers. - Keep the scheduling envelope BE-local; this PR does not change Thrift or FE code. The generated children share the large source descriptor and only materialize their row-group bounds when claimed, keeping queued memory proportional to lightweight split metadata. Depends on #66548. ## Verification - ASAN BE unit-test binary built successfully. - 54 focused scanner, split-source, Parquet, Iceberg, Paimon, and Hudi tests passed. - 64 NewParquetReaderTest and ParquetScanMetadataSafetyTest tests passed. - clang-format 16 check passed for every affected C/C++ file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
