wangyong9999 opened a new pull request, #302:
URL: https://github.com/apache/paimon-cpp/pull/302
### Purpose
Linked issue: close #298
`KeyValueFileStoreScan::SplitAndSetKeyValueFilter` folds EQUAL predicates on
the bucket keys into one bucket id computed with the bucket number of the table
schema, and everything downstream (`FilterManifestFileMeta`,
`FilterManifestEntry`, the snapshot live manifest entry cache,
`ManifestFile::ReadBucketEntries`) only looks at that id. A partition keeps the
bucket number it was written with, so after `ALTER TABLE ... SET ('bucket' =
...)` the partitions of one table carry different `total_buckets`, and keys of
the partitions whose number differs from the schema's are looked up in the
wrong bucket: the plan has no split and the read returns nothing. A reader that
captured the table schema before the option changed misses keys the same way
after the table is rewritten. Java evaluates the predicate per entry with the
entry's own `totalBuckets` (`KeyValueFileStoreScan#withTotalAwareBucketFilter`,
`BucketSelector.test(bucket, numBucket)`).
The scan now keeps the key instead of a bucket id and selects buckets per
bucket number:
- `BucketSelector` (in `bucket_select_converter.h`) holds the key literals
and the bucket function and computes the bucket for any bucket number.
`BucketSelectConverter::CreateSelector` builds it; `Convert` is a thin wrapper
over it and keeps its behaviour.
- `FileStoreScan` takes the selector through `SetBucketSelector` (only when
no explicit bucket filter is set, as before). Before reading a snapshot,
`PrepareBucketSelection` resolves one bucket per bucket number found in the
schema history, and `MatchesSelectedBucket` checks every manifest entry against
the bucket the key has under `entry.TotalBuckets()`. Manifest pruning, the
entry cache and lazy decoding run over the resolved buckets, one per distinct
bucket number, so a table whose schema history holds a single bucket number
stays on exactly the previous single-bucket path.
- The bucket numbers a table can carry are the `bucket` options of its
schema versions (a writer with another number cannot commit into an existing
partition). `BucketNumbersInSchemaHistory` lists the schema directory and reads
the schema files once per snapshot; with a `Cache` on the context the result is
cached under the snapshot's `timeMillis`, so a table whose schema does not
change lists its schema directory once per commit.
### Tests
- `BucketSelectConverterTest.SelectorFollowsBucketNumber` (new): the
selector agrees with `DefaultBucketFunction` and with `Convert` for several
bucket numbers, rejects a non-positive one, and is absent for a non-EQUAL
predicate.
- `KeyValueFileStoreScanTest.TestKeyBucketSelectionFollowsEntryBucketNumber`
(new): partition `p1` is written with `bucket = 2`, the schema is evolved to
`bucket = 4` and partition `p2` is written with it (the test asserts `p2` uses
buckets `p1` does not have). Every key of both partitions is then looked up
through `TableScan` and `TableRead` with `dt = ... AND k = ...`, on the latest
schema and on a scan pinned to the two-bucket schema via `SetTableSchema`, each
without the entry cache and with `scan.manifest-entry-cache.max-snapshots = 2`
plus lazy decoding (run twice for a cache hit). All 200 lookups return exactly
their row; without the fix the first `p1` key already comes back empty.
### API and Format
No change under `include/` and no storage format change.
`FileStoreScan::SetBucketFilterIfAbsent` (internal header) is replaced by
`SetBucketSelector`; the only caller was `KeyValueFileStoreScan`.
### Documentation
No.
### Generative AI tooling
Generated-by: Claude Code (Claude Fable 5.1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
