Spenserrrr commented on code in PR #58000:
URL: https://github.com/apache/spark/pull/58000#discussion_r3782213861
##########
python/pyspark/testing/goldenutils.py:
##########
@@ -311,12 +311,33 @@ def repr_type(t: Any) -> str:
# "halffloat" -> "float16", "float" -> "float32", "double" ->
"float64"
return _ARROW_FLOAT_ALIASES.get(s, s)
+ @staticmethod
+ def _scalar_str(scalar: Any) -> str:
+ """
+ Render one PyArrow scalar for a golden cell via PyArrow's own
``str(scalar)``.
+
+ A temporal value can be valid in Arrow yet outside Python's
``datetime`` range
+ (e.g. a date32 past year 9999), where ``str`` builds a Python datetime
and
+ raises ``OverflowError``. For those, record the raw stored count as
+ ``raw=<value>`` (its unit is in the type suffix) instead of failing. A
+ non-temporal ``OverflowError`` is unexpected, so it propagates.
+ """
+ try:
+ return str(scalar).replace("\x00", "\\0")
+ except OverflowError:
+ # No Python datetime exists for this value; record the raw stored
count as
+ # ``raw=<value>``. A non-temporal overflow is unexpected, so
re-raise.
+ if pa.types.is_temporal(scalar.type):
+ return f"raw={scalar.value}"
Review Comment:
Thanks @Yicong-Huang!
(1) format: yeah `raw=` is not clear. I updated it to
`temporal[raw](<value>)` shape, so the cell now reads
`[temporal[raw](2147483647), temporal[raw](-2147483648), None]@date32[day]`.
The unit stays in the `@date32[day]` suffix.
(2) Spark's date range: I checked the Scala side. In
`sql/api/src/main/scala/org/apache/spark/sql/types/DateType.scala`, the
[0001-01-01, 9999-12-31] is a documented "valid range" in line 23-24, but it
isn't actually enforced:
- In that file, DateType is defined as an int32 day-count in line 35, so its
real range is the same as Arrow's date32
- In
`sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala`,
`stringToDate` parses up to 7-digit years, with the comment "An integer is
able to represent a date within [+-]5 million years" in line 578
The one place year 9999 matters is in `toPandas`. When Spark passes
`date_as_object=True`, each date becomes a Python `datetime.date`, and this is
hard-capped at year 9999. With `date_as_object=False` it wouldn't raise.
So I agree that a year that large isn't a meaningful Spark date, but we need
a rendering for this cell anyway because int32:max/min is a valid Arrow date32
and Python's str() can't render it correctly. Another way to make this explicit
is that we can just drop the value and render it as <out-of-range> when the
date overflows. Please let me know if you have any other suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
