kz930 commented on code in PR #8597:
URL: https://github.com/apache/texera/pull/8597#discussion_r4254103516
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/StandaloneHelpers.scala:
##########
@@ -153,20 +235,5 @@ object StandaloneHelpers {
| return s.map(lambda x: None if pd.isna(x) else ("true" if x
else "false"))
| if pd.api.types.is_integer_dtype(s):
| return s.map(lambda x: None if pd.isna(x) else str(int(x)))
- | if pd.api.types.is_datetime64_any_dtype(s):
- | # java.sql.Timestamp.toString: trailing zeros dropped from the
- | # fraction, but never all of them. Python's own str() writes no
- | # fraction at all on a whole second and six digits otherwise.
- | def _ts(x):
- | if pd.isna(x):
- | return None
- | text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
- | return text + "0" if text.endswith(".") else text
- |
- | return s.map(_ts)
- | return s.map(
- | lambda x: None
- | if pd.isna(x)
- | else ("true" if x else "false") if pd.api.types.is_bool(x) else
str(x)
- | )""".stripMargin
+ | return s.map(lambda x: None if pd.isna(x) else
_texera_java_text(x))""".stripMargin
Review Comment:
Fixed in d9665356a: the string cast now reads empty cells from the column's
null mask, so a stored NaN reaches `_texera_java_text` and comes out as "NaN".
I added your case (NaN, null, 1.0, plus the infinities and -0.0) as a parity
test against `AttributeTypeUtils`; it fails without the fix and passes with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
