plusplusjiajia commented on PR #8136:
URL: https://github.com/apache/paimon/pull/8136#issuecomment-4891581770
Thanks for working on this — pypaimon needs query auth, and the
plan()-fetch + read-time enforcement shape matches the Java client. Verified
findings, in severity order:
1. Write paths read through auth → silent data corruption.
table_update.py:248, table_delete.py:136, table_upsert_by_key.py:507,
table_update_by_row_id.py:110 and file_store_write.py:300 all read existing
data via new_read_builder().new_scan().plan(), and plan() now applies the row
filter/masking unconditionally (there is
no _skip_auth or equivalent on this branch). With a row filter active,
upsert treats hidden rows as absent (INSERT instead of UPDATE) and masked keys
never match. Java avoids this structurally: auth lives in
AbstractDataTableScan/Read (public read layer) while writes read via
FileStoreRead below it.
2. LIMIT pushdown drops authorized rows. scanner/file_scanner.py:566 trims
splits by raw merged_row_count before the auth filter is applied —
_has_non_partition_filter() only checks the user predicate. E.g.
with_limit(100) + auth filter region='cn', first 100-row split all us → 0 rows
returned. This is exactly what
authHasNonPartitionFilter guards in Java's DataTableBatchScan (#8447);
3. predicate_json_parser crashes on common policy types (fail-closed,
feature-blocking):
- LTZ: Java emits TIMESTAMP(p) WITH LOCAL TIME ZONE
(LocalZonedTimestampType.FORMAT), but _paimon_type_to_arrow:259 can't match
parens mid-string and the branch keys at :291 are spellings Paimon never emits;
the unit test asserts the never-emitted format, masking this.
- Temporal literals: on the wire, DATE/TIME/TIMESTAMP literals are
serialized from JSR-310 objects (LeafPredicate.serializeLiterals →
convertToJavaObject returns LocalDate etc., and JsonSerdeUtil registers
JavaTimeModule), but _convert_literal:233 only handles ISO strings. Suggest
fixture tests using the exact JSON
strings from Java's PredicateJsonSerdeTest to lock wire parity.
- pc.cast at :83 defaults to safe=True and aborts the whole read on any
truncating row (2.7 → INT), where Java's cast rules truncate/wrap and keep
evaluating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
