plusplusjiajia opened a new pull request, #10618:
URL: https://github.com/apache/paimon/pull/10618
### Purpose
Following #10503, the mock REST server gains the data policy endpoints,
after Java `RESTCatalogServer`, and AuthTableQuery answers with the policies of
the principals the caller acts as, unless a test stored a response in
`table_query_auth_store`:
- Policies are stored per table, need `query-auth.enabled` and existing
columns, conflict as `POLICY`, follow table renames and go away with their
table or database. Listings honor `maxResults` and `pageToken`.
- Policy JSON is parsed as Java parses it: row filter functions and literals
as `LeafPredicate.fromJson` does, transform inputs against the declared and the
actual column types as the transform constructors check, `CAST` by the
`CastExecutors` rules, and a column mask must return the logical type of its
column, as in `RESTCatalogServerUtils.parseColumnMask`. Malformed JSON is
answered with 400.
- A created policy is stored the way `RESTCatalogServer` stores it, and
AuthTableQuery reparses it against the current schema and returns it written
for that schema. Two masks on one column deny the query.
- A schema change is refused if it would disable query authorization or move
a policy column off its field.
- Tables with branches cannot take policies, tables with policies cannot get
branches, and policy or authorization requests for a missing branch are
answered with 404.
- Declared types must be written as Java writes them, floating point
literals are not compared as strings, and a FLOAT or DOUBLE literal is written
for a column of another numeric type only when its shortest decimal form is
exact, since JDK 8 to 17 write other values with different digits
(`9.9999998E10` rather than `1.0E11` for a FLOAT `1e11`), which become
different values once read as another type. The mock answers all three with 400
rather than reproduce Java's type grammar and `Float.toString`.
`rest_permission_server_test.py` gains the policy rules.
`sql_permission_test.py` also checks that policies made with the `CALL sys.*`
procedures are listed by PyPaimon and the other way round, and that PyPaimon
and the DataFusion `SQLContext` return the same rows and values under a row
filter and column masks (`NULL` on a BOOLEAN column and `CONCAT('***')` on a
STRING one). As in #10503, these run only in the Native CI job; the read tests
need pypaimon-rust 0.5 and are skipped on older versions.
### Tests
`pypaimon/tests/rest` passes (443 tests, excluding one that needs the
optional `lance` package), and on Python 3.6 the mock server tests pass. With
pypaimon-rust built from apache/paimon-rust main (`4ceef6b`) and all five
`PYPAIMON_TEST_NATIVE_*` switches set, `sql_permission_test.py` passes together
with the native query auth tests; with pypaimon-rust 0.4.0 its three read tests
are skipped, and with 0.3.0 all five are.
I also compared the mock with Java on 955 policies, built from master:
`PredicateRemapper`, the transform constructors and `JsonSerdeUtil.toFlatJson`
accept and reject the same policies and write the same JSON, apart from the
cases above that the mock rejects on purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
