carloea2 commented on code in PR #8980:
URL: https://github.com/apache/texera/pull/8980#discussion_r4362885465
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDesc.scala:
##########
@@ -63,42 +64,183 @@ class SpecializedFilterOpDesc extends FilterOpDesc with
StandaloneCodeGenerator
)
}
- override def generateStandaloneCode(): String = {
+ override def generateStandaloneCode(): String =
generateStandaloneCode(Map.empty)
+
+ override def generateStandaloneCode(inputSchemas: Map[PortIdentity,
Schema]): String = {
// No predicate keeps no row: the executor's filter is `predicates.exists`,
// which answers false on an empty list. Passing the frame through would be
// the opposite answer.
if (predicates.isEmpty) return "out1df = in1df.iloc[0:0].copy()"
+ val declared: Map[String, AttributeType] = inputSchemas.values.headOption
+ .map(_.getAttributes.map(a => a.getName -> a.getType).toMap)
+ .getOrElse(Map.empty)
val conditions = predicates.map { p =>
val colLit = pyStringLiteral(p.attribute)
p.condition match {
case ComparisonType.IS_NULL => s"""(in1df[$colLit].isna())"""
case ComparisonType.IS_NOT_NULL => s"""(in1df[$colLit].notna())"""
- case other =>
- val op = other.getName // returns "=", ">=", "<", etc. (see
ComparisonType.java)
- val pyOp = if (op == "=") "==" else op
- // notna mirrors FilterPredicate, which answers false for every
condition
- // but IS_NULL / IS_NOT_NULL once the field is null. Only `!=` needs
it —
- // pandas answers True there, where every other operator answers
False —
- // but guarding all of them keeps the one rule visible in one place.
- s"""(in1df[$colLit].notna() & (in1df[$colLit] $pyOp
${coerceValue(p.value)}))"""
+ case other =>
+ // Without a schema the column's dtype stands in for its type.
+ val kind = declared.get(p.attribute).map(t =>
pyStringLiteral(t.name)).getOrElse("None")
Review Comment:
Please resolve the declared type with the same case-insensitive lookup as
the column name in #9035. With a LONG column Key, predicate KEY =
9007199254740993, and rows 9007199254740992 and 9007199254740993, the native
filter keeps only the second row. The combined export keeps both: declared.get
returns None, the fallback treats Int64 as INTEGER, and double conversion loses
the distinction. The ordinary matching-case LONG tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
