wjixiang opened a new issue, #2837:
URL: https://github.com/apache/iceberg-rust/issues/2837
### Apache Iceberg Rust version
None
### Describe the bug
## Summary
A table with a user data column named `pos` writes fine and appears
correctly in the catalog schema, but **any scan that projects it fails** with
`External(Unexpected =>
"field not found")`. Renaming the column to anything else (e.g. `position`)
makes it readable again. The same applies to `file_path`.
The bare names `pos` and `file_path` (no leading underscore) are reserved as
metadata-column names (`RESERVED_COL_NAME_DELETE_FILE_POS`,
`RESERVED_COL_NAME_DELETE_FILE_P
ATH`), and the data-table scan resolves column names with **metadata-first
precedence and no fallback to a same-named data column**, so a real data column
named `pos`/`f
ile_path` is shadowed.
## Minimal reproduction (pure `iceberg` crate, no catalog/storage)
Uses only the public `metadata_columns` API — runs in milliseconds with no
infrastructure:
```rust
use iceberg::metadata_columns::{
get_metadata_field_id, is_metadata_column_name,
RESERVED_COL_NAME_DELETE_FILE_POS, RESERVED_FIELD_ID_DELETE_FILE_POS,
};
fn main() {
// The reserved name is the bare `pos` (not the spec's `_pos`).
assert_eq!(RESERVED_COL_NAME_DELETE_FILE_POS, "pos");
// A user data column named `pos` is misclassified as a metadata column.
// Per the Iceberg spec only `_`-prefixed names are metadata columns,
// so this should be `false`.
assert!(is_metadata_column_name("pos"));
assert!(is_metadata_column_name("file_path"));
// So a data column `pos` resolves to the reserved delete-file field id
// (i32::MAX - 102) instead of its real schema field id.
assert_eq!(
get_metadata_field_id("pos").unwrap(),
RESERVED_FIELD_ID_DELETE_FILE_POS,
);
}
```
## Expected behavior
A user data column named `pos` should be readable. The [Iceberg
spec](https://iceberg.apache.org/spec/#reserved-field-ids) only reserves
**`_`-prefixed** names for metad
ata columns (`_file`, `_pos`, `_partition`, `_spec_id`, `_deleted`). The
bare name `pos` is a valid data column name — the Java reference implementation
does not treat it as a metadata column and reads it normally.
## Actual behavior (end-to-end)
With a real table whose schema is `chrom: String (id=1)`, `pos: Int (id=2)`:
```text
SELECT chrom FROM t -> OK
SELECT pos FROM t -> External error: Unexpected => "field not found"
SELECT * FROM t -> External error: Unexpected => "field not found"
(pos is in the projection)
```
The catalog schema is correct (`id=2 name=pos type=Int`), so this is purely
a read-path problem. Workaround: rename the column before writing (`pos` ->
`position`, `chro
m_pos`, etc.).
## Root cause
`crates/iceberg/src/metadata_columns.rs` defines the bare reserved names —
these describe the columns **inside** a position-delete file, not metadata
columns of a data t
able:
```rust
pub const RESERVED_COL_NAME_DELETE_FILE_POS: &str = "pos"; // no
underscore
pub const RESERVED_COL_NAME_DELETE_FILE_PATH: &str = "file_path"; // no
underscore
```
`is_metadata_column_name()` (via `get_metadata_field_id`) returns `Ok` for
both.
`crates/iceberg/src/scan/mod.rs` applies that check to **every projected
column of a data-table scan**, with metadata-first precedence and `continue`
(no fallback to the
data schema):
```rust
for column_name in column_names.iter() {
if is_metadata_column_name(column_name) {
field_ids.push(get_metadata_field_id(column_name)?); // pos ->
RESERVED_FIELD_ID_DELETE_FILE_POS
continue; // never
reaches field_id_by_name
}
let field_id = schema.field_id_by_name(column_name)?; // real data
column never consulted
...
}
```
So even though the table has `pos` as field id `2`, the name `pos` is
intercepted by the metadata branch before the data-schema lookup, and the
scanner goes looking for
`RESERVED_FIELD_ID_DELETE_FILE_POS` in the Parquet file, which has no such
column → `field not found`.
### Why metadata-first precedence is usually fine — and why it breaks here
The data-table metadata columns (`_file`, `_pos`, `_partition`, `_spec_id`,
`_deleted`) are **virtual**: they don't exist in the data schema or the Parquet
file, they ar
e injected by the scanner via a reserved field id. The "check metadata
first, then `continue`" rule is how those virtual columns get materialized. The
spec makes this sa
fe by restricting metadata names to a `_` prefix, which **cannot collide
with user data columns**. The bug is that `pos`/`file_path` (delete-file
internal schema columns
) were added to the same name set consulted by data-table scans, but
**without** the `_` prefix — so they can and do collide with legitimate data
column names, and the m
etadata-first precedence silently shadows them.
| name | spec role | `_`-prefixed? | metadata-first safe? |
|------|-----------|---------------|----------------------|
| `_file` `_pos` `_partition` | virtual metadata columns | yes | yes (can't
collide) |
| `pos` `file_path` | delete-file **internal** schema columns | no | **no**
(collides with user data) |
This is a real-world problem for bioinformatics: VCF `POS` is one of the
most common column names (oxbow emits it as `pos`), so any VCF → Iceberg
pipeline hits this.
## Suggested fix
Separate the two concepts so data-table scans only match the spec's
`_`-prefixed metadata names:
- Keep `RESERVED_COL_NAME_POS = "_pos"` (and `_file`, `_partition`,
`_spec_id`, `_deleted`) in the set consulted by the scan path's
`is_metadata_column_name` check.
- Do **not** let `RESERVED_COL_NAME_DELETE_FILE_POS = "pos"` /
`RESERVED_COL_NAME_DELETE_FILE_PATH = "file_path"` satisfy the scan-path
metadata check — they should only
be used when reading a delete file's own schema.
(Alternative: in the metadata branch, if the name also exists as a real data
column in the current table schema, prefer the data column's field id. But
restricting the s
et to `_`-prefixed names is cleaner and restores the spec's collision-safety
guarantee.)
## Environment
- iceberg-rust: git rev `713e201` (workspace `crates/iceberg`)
- observed via the `iceberg-datafusion` table provider + REST catalog, but
the bug is in core `scan` + `metadata_columns`
Happy to open a PR for the suggested fix if the approach sounds right.
### To Reproduce
_No response_
### Expected behavior
_No response_
### Willingness to contribute
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
