[ 
https://issues.apache.org/jira/browse/SPARK-61547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61547.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 575
[https://github.com/apache/spark-connect-rust/pull/575]

> Cache the UTC epoch for local timestamp conversion
> --------------------------------------------------
>
>                 Key: SPARK-61547
>                 URL: https://issues.apache.org/jira/browse/SPARK-61547
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Ruifeng Zheng
>            Assignee: Ruifeng Zheng
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> infer.rs::instant_micros() imports datetime, resolves timezone.utc, and 
> constructs datetime(1970, 1, 1, tzinfo=UTC) with a keyword dictionary for 
> each local TimestampType value converted by createDataFrame().
> Cache the immutable UTC timezone and epoch datetime together with PyOnceLock. 
> Continue calling astimezone() and datetime subtraction for each input, 
> preserving local-timezone/DST handling, datetime subclass hooks, and integer 
> microsecond precision. The conversion also covers timestamp values inside 
> arrays, maps, and structs.
> Release-build benchmarks on CPython 3.13 measured 3.32x faster local 
> createDataFrame construction for 1,000 UTC timestamps, 2.80x for 10,000 UTC 
> timestamps, and 3.68x for 1,000 rows with arrays of four timestamps. 
> Offset-aware and naive input improved by 2.41x and 1.64x. Integer and 
> TimestampNTZ controls were effectively unchanged. These measurements exclude 
> Spark execution and use a mock Connect server for metadata requests.
> Add regression coverage for reuse across calls and nested fields, timezone 
> changes after cache initialization, DST folds, pre-epoch and extreme datetime 
> values, microsecond precision, and subclass hook errors.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to