ndemir commented on issue #7287:
URL: https://github.com/apache/arrow-rs/issues/7287#issuecomment-2726006303
Confirmed the issue with the following PySpark code:
```
import pyspark
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit, timestamp_seconds, from_unixtime, expr
# Create a Spark session
spark = SparkSession.builder \
.appName("TimestampTest") \
.getOrCreate()
# The timestamp value in microseconds
timestamp_value = 9089380393200000000
# Create a DataFrame with the timestamp value
df = spark.createDataFrame([(timestamp_value,)], ["timestamp_micros"])
# Convert microseconds to timestamp using different methods
df = df.withColumn("timestamp_from_micros",
expr(f"TIMESTAMP_MICROS({timestamp_value})"))
# Convert to a human-readable string
df = df.withColumn("timestamp_string",
df["timestamp_from_micros"].cast("string"))
# Show the results
df.show(truncate=False)
# Alternative approach - create the timestamp directly
df2 = spark.sql(f"SELECT TIMESTAMP_MICROS({timestamp_value}) as
timestamp_value")
df2.show(truncate=False)
# Try with milliseconds by dividing the value
timestamp_value_millis = timestamp_value // 1000
df3 = spark.createDataFrame([(timestamp_value_millis,)],
["timestamp_millis"])
df3 = df3.withColumn("timestamp_from_millis",
expr(f"TIMESTAMP_MILLIS({timestamp_value_millis})"))
df3 = df3.withColumn("timestamp_string",
df3["timestamp_from_millis"].cast("string"))
print("Testing with milliseconds:")
df3.show(truncate=False)
# Clean up
spark.stop()
```
```
+-------------------+----------------------+----------------------+
|timestamp_micros |timestamp_from_micros |timestamp_string |
+-------------------+----------------------+----------------------+
|9089380393200000000|+290000-12-30 23:00:00|+290000-12-30 23:00:00|
+-------------------+----------------------+----------------------+
+----------------------+
|timestamp_value |
+----------------------+
|+290000-12-30 23:00:00|
+----------------------+
Testing with milliseconds:
+----------------+----------------------+----------------------+
|timestamp_millis|timestamp_from_millis |timestamp_string |
+----------------+----------------------+----------------------+
|9089380393200000|+290000-12-30 23:00:00|+290000-12-30 23:00:00|
+----------------+----------------------+----------------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
