On Thu, Nov 10, 2016 at 11:14 AM, Perttu Ranta-aho <ranta...@iki.fi> wrote:
> Hello,
>
> I want to create an UDF which modifies one column value depending on value
> of some other column. But Python version of the code fails always in column
> value comparison. Below are simple examples, scala version works as expected
> but Python version throws an execption. Am I missing something obvious? As
> can be seen from PySpark exception I'm using Spark 2.0.1.
>
> -Perttu
>
> import org.apache.spark.sql.functions.udf
> val df = spark.createDataFrame(List(("a",1), ("b",2), ("c",
> 3))).withColumnRenamed("_1", "name").withColumnRenamed("_2", "value")
> def myUdf = udf((name: String, value: Int) => {if (name == "c") { value * 2
> } else { value }})
> df.withColumn("udf", myUdf(df("name"), df("value"))).show()
> +----+-----+---+
> |name|value|udf|
> +----+-----+---+
> | a| 1| 1|
> | b| 2| 2|
> | c| 3| 6|
> +----+-----+---+
>
>
> from pyspark.sql.types import StringType, IntegerType
> import pyspark.sql.functions as F
>
> df = sqlContext.createDataFrame((('a',1), ('b',2), ('c', 3)),
> ('name','value'))
>
> def my_udf(name, value):
> if name == 'c':
> return value * 2
> return value
> F.udf(my_udf, IntegerType())
udf = F.udf(my_udf, IntegerType())
df.withColumn("udf", udf(df.name, df.value)).show()
>
> df.withColumn("udf", my_udf(df.name, df.value)).show()
>
> ---------------------------------------------------------------------------
> ValueError Traceback (most recent call last)
> <ipython-input-6-10032e941fc4> in <module>()
> ----> 1 df.withColumn("udf", my_udf(df.name, df.value)).show()
>
> <ipython-input-5-c103a6066373> in my_udf(name, value)
> 3
> 4 def my_udf(name, value):
> ----> 5 if name == 'c':
> 6 return value * 2
> 7 return value
>
> /home/ec2-user/spark-2.0.1-bin-hadoop2.4/python/pyspark/sql/column.pyc in
> __nonzero__(self)
> 425
> 426 def __nonzero__(self):
> --> 427 raise ValueError("Cannot convert column into bool: please
> use '&' for 'and', '|' for 'or', "
> 428 "'~' for 'not' when building DataFrame
> boolean expressions.")
> 429 __bool__ = __nonzero__
>
> ValueError: Cannot convert column into bool: please use '&' for 'and', '|'
> for 'or', '~' for 'not' when building DataFrame boolean expressions.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
