kz930 commented on code in PR #8597:
URL: https://github.com/apache/texera/pull/8597#discussion_r4254225002
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/StandaloneHelpers.scala:
##########
@@ -153,20 +235,5 @@ object StandaloneHelpers {
| return s.map(lambda x: None if pd.isna(x) else ("true" if x
else "false"))
| if pd.api.types.is_integer_dtype(s):
| return s.map(lambda x: None if pd.isna(x) else str(int(x)))
- | if pd.api.types.is_datetime64_any_dtype(s):
- | # java.sql.Timestamp.toString: trailing zeros dropped from the
- | # fraction, but never all of them. Python's own str() writes no
- | # fraction at all on a whole second and six digits otherwise.
- | def _ts(x):
- | if pd.isna(x):
- | return None
- | text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
- | return text + "0" if text.endswith(".") else text
- |
- | return s.map(_ts)
- | return s.map(
- | lambda x: None
- | if pd.isna(x)
- | else ("true" if x else "false") if pd.api.types.is_bool(x) else
str(x)
- | )""".stripMargin
+ | return s.map(lambda x: None if pd.isna(x) else
_texera_java_text(x))""".stripMargin
Review Comment:
Also fixed the same NaN handling in the integer, long and boolean casts in
1f15e1501. A stored NaN now gives 0, 0 and true as the engine does, and an
infinity saturates instead of stopping the script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
