DB Tsai created SPARK-60771:
-------------------------------
Summary: Avoid generic scalar probes for exact Python datetime
input in Spark Connect Rust
Key: SPARK-60771
URL: https://issues.apache.org/jira/browse/SPARK-60771
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
Exact builtin Python datetime values currently reach timestamp recognition
after Decimal detection, integer/float/string extraction and container/Row/map
probes in the Rust Python binding. In a fresh client-only benchmark of master
617fa1481b3e6fc7b825928130f280506328c9b1, the existing 10,000 x 3 tuple
timestamp case took a median 56.817 ms, versus 49.426 ms for the pinned Python
reference. This is priority evidence from three fresh-process samples, not a
promised speedup.
Investigate a narrow early dispatch for ordinary exact builtin
datetime.datetime values that calls the existing datetime_to_micros unchanged.
Preserve dynamic class/module binding behavior, subclass numeric-protocol
precedence, timezone callbacks and exception order, pandas Timestamp/NaT
fallback, null/schema behavior and RPC counts. Do not change timestamp
arithmetic, rounding, row traversal, Arrow builders or configuration reads.
PyO3 ABI3 caches its imported datetime type. Establish correct behavior under
rebinding before and after first lookup, heap-class impostors, native aliases
and missing/replaced datetime module attributes before enabling a fast path.
Reject the optimization if a small supported-API guard cannot retain existing
semantics with worthwhile benefit.
Validation should compare separate frozen baseline/candidate environments with
unchanged client harness and independent decoding of all outbound LocalRelation
IPC cells, schema, nullability and timestamp microseconds. Cover naive/aware
values, pre-epoch and microsecond boundaries, DST folds, custom tzinfo,
subclasses and hooks, mixed/nested rows, schema/error order and dynamic
bindings. Measure warmed target throughput, tiny negative guards,
first-feature/import cost, CPU/RSS and normalized RPC counts. No JVM or live
Spark server is required for these local conversion tests.
This is distinct from SPARK-60685 / PR128 (exact date scalar dispatch) and
SPARK-60686 / PR130 (columnar pandas timestamps). Existing upstream PRs and
Apache Connect datetime/timestamp issues were checked before filing.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
