Issues with `when` in Column class
I’m trying to iterate through a list of Columns and create new Columns based on a condition. However, the when method keeps giving me errors that don’t quite make sense. If I do `when(col === “abc”, 1).otherwise(0)` I get the following error at compile time: [error] not found: value when However, this works in the REPL just fine after I import org.apache.spark.sql.Column. On the other hand, if I do `col.when(col === “abc”, 1).otherwise(0)`, it will compile successfully, but then at runtime, I get this error: java.lang.IllegalArgumentException: when() can only be applied on a Column previously generated by when() function This appears to be pretty circular logic. How can `when` only be applied to a Column previously generated by `when`? How would I call `when` in the first place?
Re: Issues with `when` in Column class
Hi Chris, Have you imported org.apache.spark.sql.functions._? Thanks, Yin On Fri, Jun 12, 2015 at 8:05 AM, Chris Freeman cfree...@alteryx.com wrote: I’m trying to iterate through a list of Columns and create new Columns based on a condition. However, the when method keeps giving me errors that don’t quite make sense. If I do `when(col === “abc”, 1).otherwise(0)` I get the following error at compile time: [error] not found: value when However, this works in the REPL just fine after I import org.apache.spark.sql.Column. On the other hand, if I do `col.when(col === “abc”, 1).otherwise(0)`, it will compile successfully, but then at runtime, I get this error: java.lang.IllegalArgumentException: when() can only be applied on a Column previously generated by when() function This appears to be pretty circular logic. How can `when` only be applied to a Column previously generated by `when`? How would I call `when` in the first place?
Re: Issues with `when` in Column class
That did it! Thanks! From: Yin Huai Date: Friday, June 12, 2015 at 10:31 AM To: Chris Freeman Cc: user@spark.apache.orgmailto:user@spark.apache.org Subject: Re: Issues with `when` in Column class Hi Chris, Have you imported org.apache.spark.sql.functions._? Thanks, Yin On Fri, Jun 12, 2015 at 8:05 AM, Chris Freeman cfree...@alteryx.commailto:cfree...@alteryx.com wrote: I’m trying to iterate through a list of Columns and create new Columns based on a condition. However, the when method keeps giving me errors that don’t quite make sense. If I do `when(col === “abc”, 1).otherwise(0)` I get the following error at compile time: [error] not found: value when However, this works in the REPL just fine after I import org.apache.spark.sql.Column. On the other hand, if I do `col.when(col === “abc”, 1).otherwise(0)`, it will compile successfully, but then at runtime, I get this error: java.lang.IllegalArgumentException: when() can only be applied on a Column previously generated by when() function This appears to be pretty circular logic. How can `when` only be applied to a Column previously generated by `when`? How would I call `when` in the first place?
