taoran92 opened a new issue, #10478:
URL: https://github.com/apache/paimon/issues/10478
### Search before asking
- [x] I searched the issues and found no report of the same bucket/partition
query failure. #5467 concerns file-path metadata, which has a different
restriction.
### Paimon version
Reproduced locally with Paimon `2.2-SNAPSHOT`, commit
`8588fd3306310078915bd447ddd0c666390b2fb2`.
### Compute Engine
Spark 3.5.9, Scala 2.12, Java 11.
### Minimal reproduce step
We discovered this issue in an internal production workload at our company
while querying `__paimon_bucket` and `__paimon_partition` on a primary-key
table.
Both columns are readable after the first insert. After writing another
version of the same primary key in a separate insert, queries selecting either
metadata column fail with a `ClassCastException`. A query selecting only the
data columns still returns the updated record correctly.
The following SQL reproduces the issue. `paimon` is a configured Paimon
catalog. Compaction and deletion vectors are disabled to keep the reproduction
consistent.
```sql
CREATE DATABASE IF NOT EXISTS paimon.repro;
CREATE TABLE paimon.repro.metadata_query (
id INT,
v STRING,
dt STRING
)
PARTITIONED BY (dt)
TBLPROPERTIES (
'primary-key' = 'dt,id',
'bucket' = '1',
'write-only' = 'true',
'deletion-vectors.enabled' = 'false'
);
INSERT INTO paimon.repro.metadata_query
VALUES (1, 'old', '2026-10-09');
-- Succeeds with a single data file.
SELECT id, v, __paimon_bucket, __paimon_partition
FROM paimon.repro.metadata_query;
-- Write a new version of the same key in a separate file.
INSERT INTO paimon.repro.metadata_query
VALUES (1, 'new', '2026-10-09');
-- Succeeds and returns (1, 'new').
SELECT id, v
FROM paimon.repro.metadata_query;
-- Run these separately. Each query fails.
SELECT __paimon_bucket
FROM paimon.repro.metadata_query;
SELECT __paimon_partition
FROM paimon.repro.metadata_query;
SELECT id, v, __paimon_bucket, __paimon_partition
FROM paimon.repro.metadata_query;
```
### What doesn't meet your expectations?
Bucket and partition metadata should remain readable when records are merged
across files. The combined query should return:
```text
id v __paimon_bucket __paimon_partition
1 new 0 {"dt":"2026-10-09"}
```
Instead, all three metadata queries fail with the same exception. Relevant
stack trace from the local reproduction:
```text
java.lang.ClassCastException: class
org.apache.paimon.table.source.ValueContentRowDataRecordIterator cannot be cast
to class org.apache.paimon.reader.FileRecordIterator
at
org.apache.paimon.spark.PaimonRecordReaderIterator.advanceIfNeeded(PaimonRecordReaderIterator.scala:136)
at
org.apache.paimon.spark.PaimonRecordReaderIterator.hasNext(PaimonRecordReaderIterator.scala:72)
at
org.apache.paimon.spark.PaimonPartitionReader.advanceIfNeeded(PaimonPartitionReader.scala:98)
at
org.apache.paimon.spark.PaimonPartitionReader.next(PaimonPartitionReader.scala:71)
at
org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:168)
at
org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:206)
at
org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1(DataSourceRDD.scala:104)
at
org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1$adapted(DataSourceRDD.scala:104)
at scala.Option.exists(Option.scala:376)
at
org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:104)
at
org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.advanceToNextIter(DataSourceRDD.scala:145)
at
org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:104)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
...
```
### Anything else?
`PaimonRecordReaderIterator` casts the iterator to `FileRecordIterator`
before updating any metadata column. The merge reader returns a
`ValueContentRowDataRecordIterator`, so the cast fails even when only bucket or
partition metadata is requested.
Both values are available from `DataSplit` and do not require a file-level
iterator. The existing restrictions on `__paimon_file_path` and
`__paimon_row_index` can remain unchanged.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
