carloea2 commented on code in PR #8340:
URL: https://github.com/apache/texera/pull/8340#discussion_r3975701862
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/typecasting/TypeCastingOpDesc.scala:
##########
@@ -72,4 +74,51 @@ class TypeCastingOpDesc extends MapOpDesc {
List(OutputPort())
)
}
+
+ override def generateStandaloneCode(): String = {
+ val units = Option(typeCastingUnits).getOrElse(List.empty)
+ if (units.isEmpty) return "out1df = in1df.copy()"
+
+ val lines = scala.collection.mutable.ArrayBuffer[String]("out1df =
in1df.copy()")
+ units.foreach { unit =>
+ val colLit = pyStringLiteral(unit.attribute)
+ // Every cast goes through the transcription of AttributeTypeUtils rather
+ // than through Python's own conversions, which answer differently: a
+ // non-empty string is always a true boolean, and a coercing numeric cast
+ // reads "6.7" as an integer the engine refuses.
+ //
+ // A timestamp is the one that stays approximate. The engine reads it
with
+ // DateParserUtils, which accepts a set of formats no single pandas call
+ // states, so this coerces what it cannot read rather than claiming a
+ // match it does not have.
+ val expr = unit.resultType match {
+ case AttributeType.STRING =>
+ // `astype(str)` gets two things wrong against `toString`: an empty
+ // cell renders as the text "nan", and a boolean capitalises. The
+ // helper handles both, and reads the point of a double off the
+ // column's type rather than off the value, which cannot tell a whole
+ // double from an integer.
+ s"""_texera_cast_string(out1df[$colLit])"""
+ case AttributeType.INTEGER | AttributeType.LONG =>
+ // A hole survives the cast, because parseField returns a null field
+ // untouched; pandas' nullable "Int64" holds one where numpy's int
+ // cannot.
+ s"""out1df[$colLit].apply(lambda x: pd.NA if pd.isna(x) else
_texera_cast_integral(x)).astype("Int64")"""
+ case AttributeType.DOUBLE =>
+ // NaN rather than pd.NA: float64 is how a double column is held
here,
+ // and it carries its hole as NaN. pd.NA would not survive the
astype.
+ s"""out1df[$colLit].apply(lambda x: float("nan") if pd.isna(x) else
_texera_cast_double(x)).astype("float64")"""
+ case AttributeType.BOOLEAN =>
+ // Nullable "boolean" for the same reason, and because
`.astype(bool)`
+ // reads NaN as True: NaN is a non-zero float.
+ s"""out1df[$colLit].apply(lambda x: pd.NA if pd.isna(x) else
_texera_cast_boolean(x)).astype("boolean")"""
+ case AttributeType.TIMESTAMP => s"""pd.to_datetime(out1df[$colLit],
errors="coerce")"""
Review Comment:
LONG timestamps use different units here. With input 1700000000000, the
native cast returns 2023-11-14 22:13:20 in UTC, but the export returns
1970-01-01 00:28:20. I reproduced this with both runtimes on UTC; zero passes
as a control. AttributeTypeUtils interprets LONG as milliseconds, while
pd.to_datetime defaults to nanoseconds. Please preserve the input type and
epoch unit.
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/typecasting/TypeCastingOpDesc.scala:
##########
@@ -72,4 +74,51 @@ class TypeCastingOpDesc extends MapOpDesc {
List(OutputPort())
)
}
+
+ override def generateStandaloneCode(): String = {
+ val units = Option(typeCastingUnits).getOrElse(List.empty)
+ if (units.isEmpty) return "out1df = in1df.copy()"
+
+ val lines = scala.collection.mutable.ArrayBuffer[String]("out1df =
in1df.copy()")
+ units.foreach { unit =>
+ val colLit = pyStringLiteral(unit.attribute)
+ // Every cast goes through the transcription of AttributeTypeUtils rather
+ // than through Python's own conversions, which answer differently: a
+ // non-empty string is always a true boolean, and a coercing numeric cast
+ // reads "6.7" as an integer the engine refuses.
+ //
+ // A timestamp is the one that stays approximate. The engine reads it
with
+ // DateParserUtils, which accepts a set of formats no single pandas call
+ // states, so this coerces what it cannot read rather than claiming a
+ // match it does not have.
+ val expr = unit.resultType match {
+ case AttributeType.STRING =>
+ // `astype(str)` gets two things wrong against `toString`: an empty
+ // cell renders as the text "nan", and a boolean capitalises. The
+ // helper handles both, and reads the point of a double off the
+ // column's type rather than off the value, which cannot tell a whole
+ // double from an integer.
+ s"""_texera_cast_string(out1df[$colLit])"""
+ case AttributeType.INTEGER | AttributeType.LONG =>
+ // A hole survives the cast, because parseField returns a null field
+ // untouched; pandas' nullable "Int64" holds one where numpy's int
+ // cannot.
+ s"""out1df[$colLit].apply(lambda x: pd.NA if pd.isna(x) else
_texera_cast_integral(x)).astype("Int64")"""
Review Comment:
INTEGER and LONG cannot share the same 64-bit result conversion. Casting
LONG 2147483648 to INTEGER returns -2147483648 in the native executor but
2147483648 in the export, outside the declared INTEGER range. The boundary
value 2147483647 passes the same test. Please handle INTEGER narrowing
separately and cover the signed boundaries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
