Hello guys,
I am migrating from Spark 1.6 to 2.2 and have this issue:
I am casting string to short and comparing them with equal .
Original code is:
... when(col(fieldName).equalTo(castedValueCol), castedValueCol).
otherwise(defaultErrorValueCol)
Reproduce (version 2.3.0.cloudera4):
scala> val df = Seq("3.15").toDF("tier_id")
df: org.apache.spark.sql.DataFrame = [tier_id: string]
scala> val colShort = col("tier_id").cast(ShortType)
colShort: org.apache.spark.sql.Column = CAST(tier_id AS SMALLINT)
scala> val colString = col("tier_id")
colString: org.apache.spark.sql.Column = tier_id
scala> res4.select(colString, colShort, colShort.equalTo(colString)).show
+-------+-------+-------------------------------------+
|tier_id|tier_id|(CAST(tier_id AS SMALLINT) = tier_id)|
+-------+-------+-------------------------------------+
| 3.15| 3| true|
+-------+-------+-------------------------------------+
scala>
Why is this?
--
--
Artur Sukhenko
