Joseph K. Bradley created SPARK-19039: -----------------------------------------
Summary: UDF ClosureCleaner bug when UDF, col applied in paste mode in REPL Key: SPARK-19039 URL: https://issues.apache.org/jira/browse/SPARK-19039 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.1.0, 2.0.2, 1.6.3 Reporter: Joseph K. Bradley When I try this: * Define UDF * Apply UDF to get Column * Use Column in a DataFrame I can find weird behavior in the spark-shell when using paste mode. To reproduce this, paste this into the spark-shell: {code} import org.apache.spark.sql.functions._ val df = spark.createDataFrame(Seq( ("hi", 1), ("there", 2), ("the", 3), ("end", 4) )).toDF("a", "b") val myNumbers = Set(1,2,3) val tmpUDF = udf { (n: Int) => myNumbers.contains(n) } val rowHasMyNumber = tmpUDF($"b") df.where(rowHasMyNumber).show() {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org