wombatu-kun opened a new pull request, #20314: URL: https://github.com/apache/hudi/pull/20314
### Describe the issue this Pull Request addresses Closes #20313. On a Flink bucket-index table with one primary key and two or more partition fields, an equality filter on the primary key returns no rows. The complex key generator stores such keys as `uuid:id1` under the default `FIELD_PREFIXED` encoding and the bucket writers hash that stored key, while `PrimaryKeyPruners.getBucketIdFunc` hashed the bare filter literal `id1` and kept another bucket's file group. ### Summary and Changelog Bucket pruning now hashes a field-prefixed single record key the way the bucket writers store and hash it. - `HoodieTableSource#getDataBucketFunc` resolves the table's record key encoding with `KeyGenUtils.resolveComplexKeyGenEncoding`, as the Flink record-level index reader does, and hashes the prefixed key only when the Flink key generator of the job prefixes too: the Flink writer decides the prefix from the job configuration, so an explicit complex key generator with one partition field writes bare keys into a table that records `FIELD_PREFIXED`. A single-field complex keygen table whose encoding cannot be resolved is not pruned by key, and neither is a streaming read of a table that does not exist yet. - `RowDataKeyGen#isFieldPrefixedSingleRecordKey` exposes the key generator's existing prefix decision, so the writer and the pruner share it. - `PrimaryKeyPruners#getBucketIdFunc` rebuilds `<field>:<value>` for a field-prefixed single record key and hashes it through `BucketIdentifier.getBucketId(recordKey, indexKeyFields, numBuckets)`, the call the bucket writers make. The two-argument overload, now only used by tests, keeps the bare-value behavior. - `KeyGenUtils.extractRecordKeysByFields` no longer loops forever when the value of a skipped trailing field holds `:`, which the pruner would otherwise hit for a literal such as `'a,b:c:d'`. The skipped-field loop gets the `keyValueSep2 > 0` guard its matching-field sibling already has; it only changes inputs that never returned, so no stored bucket id moves, and bucket writers stop hanging on keys such as `id:1,note:10:00` hashed by `id`. - Tests: `ITTestHoodieDataSource#testBucketPruningWithSingleRecordKeyAndMultiplePartitionFields` reproduces the issue with Flink SQL on COPY_ON_WRITE and MERGE_ON_READ, for the default `FIELD_PREFIXED` encoding and a version 8 `VALUE_ONLY` table. `TestHoodieTableSource` checks the pruning bucket against the writer's key generator for a recorded, a deduced and an unresolvable encoding (including keys holding both separators), for an explicit complex key generator with one partition field, for a failure to resolve the encoding, and for a streaming read before the table exists. `TestKeyGenUtils#testExtractRecordKeysWithColonsInSkippedLastValue` covers the parser. ### Impact Equality queries on the primary key of such tables return their rows again. Compound record keys, tables with any other key generator, `VALUE_ONLY` tables and tables whose recorded encoding the Flink job does not follow keep their current bucket ids. No storage format, public API or configuration change. ### Risk Level Low. The reader changes its bucket only when the table and the Flink key generator both store `<field>:<value>`, and then goes through the writers' own hashing. For a single-field complex keygen table without a recorded encoding, the encoding is deduced from the data at planning time, as the record-level index reader does; a failure there disables bucket pruning instead of failing the query. Verified: the new IT fails on master for both `FIELD_PREFIXED` cases (`Expected: is "[+I[id1]]" but: was "[]"`) and passes with the fix; each new unit-test assertion fails with its part of the fix removed: the prefix, the agreement with the job's key generator, the null meta client guard, the unresolvable-encoding fallback, the catch around the encoding resolution, the parser guard, and hashing through `BucketIdentifier` rather than a list of the prefixed string (the `id9,x` key); `TestHoodieTableSource`, `TestPrimaryKeyPruners`, `TestRowDataKeyGens`, `TestHoodieTableFactory` and the new `TestKeyGenUtils` test pass; the change merges cleanly with #20292, which edits the same method for compound keys. ### Documentation Update None. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
