taoran92 opened a new issue, #10722: URL: https://github.com/apache/paimon/issues/10722
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version The SQL reproduction below was recorded with Apache Paimon `2.2-SNAPSHOT` at master commit `8c09824c060520f3f1f4a03ce2369de371e5c7c6`. ### Compute Engine Apache Spark 3.5.9 for the SQL reproduction below. Additional scenarios were verified through Paimon Core tests using actual table reads and a mocked `Catalog.authTableQuery`. These cover fallback/primary branch reads, chain-table batch and streaming reads, and data evolution tables. ### Minimal reproduce step Configure `spark_test` as a REST catalog with column-level query authorization. Enable `query-auth.enabled` on the table. The authorization service rejects requests for columns outside the user's allowlist and interprets `select: null` as requesting all columns. No row filters or column masks are configured. 1. As an administrator, create a partitioned table and write data only to its backup branch: ```sql CREATE DATABASE spark_test.auth_repro; CREATE TABLE spark_test.auth_repro.orders ( id INT, payload STRING, dt STRING ) USING paimon PARTITIONED BY (dt) TBLPROPERTIES ( 'primary-key' = 'dt,id', 'bucket' = '1', 'query-auth.enabled' = 'true' ); CALL spark_test.sys.create_branch('auth_repro.orders', 'backup'); -- Keep the main branch empty so the query reads from the fallback branch. INSERT INTO spark_test.auth_repro.`orders$branch_backup` VALUES (1, 'secret', '20250810'); ALTER TABLE spark_test.auth_repro.orders SET TBLPROPERTIES ('scan.fallback-branch' = 'backup'); ``` 2. Configure the authorization service to allow `alice` to select only `id` and `dt` on both `orders` and `orders$branch_backup`. Do not grant access to `payload`. In a new session authenticated as `alice`, run: ```sql -- Succeeds and returns (1, '20250810'). SELECT id, dt FROM spark_test.auth_repro.`orders$branch_backup`; -- Fails with a column authorization error for payload. SELECT id, dt FROM spark_test.auth_repro.orders; ``` 3. Restrict the column allowlist to only `id` on both branches and start a new session as `alice`: ```sql -- Succeeds and returns 1. SELECT id FROM spark_test.auth_repro.`orders$branch_backup`; -- Fails, although the query references only the authorized column id. SELECT id FROM spark_test.auth_repro.orders; ``` The second query should return `1` without requiring access to either `payload` or the partition column `dt`. ### What doesn't meet your expectations? Column authorization should consistently cover all columns referenced by the user's query, including columns used only in filters. A supported query whose referenced columns are all authorized should not fail column authorization merely because Paimon uses fallback, primary, or chain-table scans instead of reading a branch directly. Paimon may additionally use fields required for internal partition selection, snapshot lookup, and record merging. Those execution requirements should not expand the set of column permissions required from the user. Internal scans must preserve the query's authorization requirements without adding execution-only fields to authorization requests or dropping user-referenced fields. In the Spark reproduction, the authorization requests differ even though both queries project the same columns: ```text SELECT id, dt | +-- Direct backup branch --> select=[id,dt] --> succeeds | +-- Fallback table -------> select=null ----> denied on payload ``` The fallback query reports: ```text Catalog$TableNoPermissionException: Table auth_repro.orders has no permission. Cause by Forbidden: Column access denied: payload; requested columns: null. ``` The same distinction applies to internal partition predicates. For example, `SELECT id` requires permission on `id`; Paimon's use of `dt` to locate fallback partitions should not add a permission requirement on `dt`. If the user explicitly references `dt` in the query, it must be authorized. With access restricted to `id`, all of the following queries must still be denied: ```sql -- Explicitly projects an unauthorized column. SELECT dt FROM spark_test.auth_repro.orders; -- References an unauthorized partition column in a user filter. SELECT id FROM spark_test.auth_repro.orders WHERE dt = '20250810'; -- References an unauthorized non-partition column in a user filter. SELECT id FROM spark_test.auth_repro.orders WHERE payload = 'secret'; ``` Projection pruning, filter pushdown, and branch selection must preserve these user-filter authorization requirements, including when a filter is evaluated after records from multiple branches are merged. ### Anything else? Core tests also reproduce the authorization failure in these scenarios: - Fallback and primary branch reads of an append table `(pt INT, a INT)`, partitioned by `pt`, when the query projects only `a` and only `a` is authorized. This includes tables with `data-evolution.enabled=true` and `row-tracking.enabled=true`. - Chain table batch and streaming reads that project only authorized columns. For a table with columns `(k, seq, v, region, dt, hour)`, partition keys `(region, dt, hour)`, and `chain-table.chain-partition-keys=dt,hour`, a query projecting `(k, dt, hour)` fails when only those columns are authorized. Here, each `region` has its own chain ordered by `(dt, hour)`; the query does not reference `region`, `seq`, or `v`. The streaming case occurs with `chain-table.streaming.merge-snapshot` both disabled and enabled. These additional cases use actual table reads with a mocked authorization service. The Spark SQL reproduction above covers fallback reads; the chain and data evolution cases were verified at the Core API level. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
