Ruifeng Zheng created SPARK-61547:
-------------------------------------

             Summary: Cache the UTC epoch for local timestamp conversion
                 Key: SPARK-61547
                 URL: https://issues.apache.org/jira/browse/SPARK-61547
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Ruifeng Zheng


infer.rs::instant_micros() imports datetime, resolves timezone.utc, and 
constructs datetime(1970, 1, 1, tzinfo=UTC) with a keyword dictionary for each 
local TimestampType value converted by createDataFrame().

Cache the immutable UTC timezone and epoch datetime together with PyOnceLock. 
Continue calling astimezone() and datetime subtraction for each input, 
preserving local-timezone/DST handling, datetime subclass hooks, and integer 
microsecond precision. The conversion also covers timestamp values inside 
arrays, maps, and structs.

Release-build benchmarks on CPython 3.13 measured 3.32x faster local 
createDataFrame construction for 1,000 UTC timestamps, 2.80x for 10,000 UTC 
timestamps, and 3.68x for 1,000 rows with arrays of four timestamps. 
Offset-aware and naive input improved by 2.41x and 1.64x. Integer and 
TimestampNTZ controls were effectively unchanged. These measurements exclude 
Spark execution and use a mock Connect server for metadata requests.

Add regression coverage for reuse across calls and nested fields, timezone 
changes after cache initialization, DST folds, pre-epoch and extreme datetime 
values, microsecond precision, and subclass hook errors.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to