Gabriel39 opened a new issue, #66816: URL: https://github.com/apache/doris/issues/66816
### Search before asking - [x] I searched existing issues and pull requests and found no equivalent request. ### Description When investigating lakehouse data problems, locating anomalous records, or building row-level operations, users need to identify both the physical data file that produced a row and the row's absolute physical position in that file. File Scanner V2 native Parquet/ORC readers should expose this information for Iceberg and Paimon tables through format-compatible metadata columns: | Table format | File path column | Physical row position column | | --- | --- | --- | | Iceberg | `_file STRING NOT NULL` | `_pos BIGINT NOT NULL` | | Paimon | `__paimon_file_path STRING NOT NULL` | `__paimon_row_index BIGINT NOT NULL` | The names and semantics should follow the [Iceberg metadata column specification](https://iceberg.apache.org/spec/) and [Paimon hidden metadata column definitions](https://paimon.apache.org/docs/master/spark/sql-query/). These columns must not be included in `SELECT *` by default. They must remain explicitly addressable when `show_hidden_columns=false`; when `show_hidden_columns=true`, `DESC` and `SELECT *` should expose them. ### Required semantics #### File path - Return the original path recorded in the table-format metadata. - Iceberg `_file` must use the path stored in the Iceberg data-file metadata. - Paimon `__paimon_file_path` must use the corresponding `RawFile.path()`. - Do not return a BE cache path, temporary path, or reader-internal path. - Preserve the original URI even if Doris normalizes its scheme or endpoint for storage access. #### Physical row position - Positions are zero-based and absolute within the complete physical data file. - The position must not be relative to a scan split, Parquet row group, ORC stripe, filtered result, or output order. - `(file_path, row_position)` should uniquely locate one physical record within a table snapshot. - Filtering, lazy materialization, runtime filters, delete files, and deletion vectors must remove rows without renumbering the surviving positions. - Compaction or file rewrites may change the physical location of the same logical record. This feature exposes physical positions only. Stable logical row IDs such as Iceberg V3 `_row_id` or Paimon row-tracking `_ROW_ID` are out of scope. ### Examples ```sql SELECT id, _file, _pos FROM iceberg_catalog.db.tbl WHERE _pos >= 0 ORDER BY _file, _pos; ``` ```sql SELECT id, __paimon_file_path, __paimon_row_index FROM paimon_catalog.db.tbl ORDER BY __paimon_file_path, __paimon_row_index; ``` The metadata columns must also work in predicates, comparisons, ranges, `ORDER BY`, `GROUP BY`, aggregates, and joins. Predicate pushdown is not required initially, but final results must be correct. ### Supported scope - File Scanner V2. - Native Parquet and ORC readers. - Iceberg data tables. - Paimon tables whose scan ranges can all be converted to native `RawFile` splits. - Normal scans, predicate filtering, lazy materialization, and runtime filters. - Iceberg position deletes, equality deletes, and deletion vectors. - Paimon deletion vectors. - File sub-split parallel scans. - Snapshot and time-travel queries. ### Native-reader boundary When a query references these metadata columns, every scan range must use a File Scanner V2 native Parquet/ORC reader. The query must fail with a clear error before returning any rows when: - Paimon uses only JNI splits or mixes native and JNI splits; - `force_jni_scanner=true`; - File Scanner V2 is disabled; or - the data-file format is not Parquet or ORC. Returning `NULL`, default values, incorrect values, or results for only the native subset is not acceptable. Existing queries that do not reference these columns must keep their current JNI, mixed-split, and File Scanner V1 behavior. A suggested diagnostic is: ```text Metadata column '<column>' is only supported by FileScannerV2 native Parquet/ORC reader for <table_format>; actual reader is <reader_type>. ``` ### Acceptance criteria 1. The columns are virtual metadata columns and are never read as physical Parquet/ORC fields. 2. Paths match the original paths in Iceberg/Paimon metadata, including schemes such as `s3a://` or `oss://`. 3. Row positions start at zero and remain file-absolute across row groups, stripes, scan ranges, BEs, and parallel scheduling. 4. Sub-splits produce no duplicate, overlapping, or missing `(path, position)` pairs. 5. Ordinary predicates, reader pruning, lazy reads, runtime filters, delete files, and deletion vectors preserve row alignment and original positions. 6. Snapshot/time-travel queries return locations from the selected snapshot rather than the latest snapshot. 7. A metadata-column predicate must disable any file/table-level `COUNT` pushdown that cannot evaluate it; Doris must scan instead of returning an unfiltered count. 8. Unsupported reader combinations fail before the first output block and never return partial results. 9. Repeated queries against the same snapshot produce stable locations independent of scan-range partitioning. ### Performance and resource requirements - Queries that do not reference these columns must not materialize row positions or perform observable additional data-file I/O. - Queries that reference them must not read additional physical file columns. - Prefer a constant-column or equivalent low-memory representation for file paths instead of copying a long path for every row in intermediate blocks. - Additional row-position memory should be close to eight bytes per row. - Large files with many row groups or stripes must not cause significant memory inflation or OOM. ### Validation Tests should cover: - Iceberg Parquet and ORC files with multiple row groups/stripes; - filtered and lazy-materialized scans with known physical positions; - Iceberg position/equality deletes and deletion vectors; - Paimon native scans and deletion vectors; - single files divided into multiple parallel scan ranges; - Paimon JNI-only, mixed native/JNI, and forced-JNI rejection; - metadata predicates combined with `COUNT(*)`; - snapshot/time-travel queries before and after file rewrites; and - memory and I/O overhead with long object-storage paths and large files. ### Are you willing to submit PR? - [ ] Yes, I am willing to submit a PR. ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
