[
https://issues.apache.org/jira/browse/SPARK-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15431371#comment-15431371
]
Andrew Davidson commented on SPARK-17172:
-----------------------------------------
Hi Sean
the data center was created using spark-ec2 from spark-1.6.1-bin-hadoop2.6
ec2-user@ip-172-31-22-140 root]$ cat /root/spark/RELEASE
Spark 1.6.1 built for Hadoop 2.0.0-mr1-cdh4.2.0
Build flags: -Psparkr -Phadoop-1 -Phive -Phive-thriftserver
-Dhadoop.version=2.0.0-mr1-cdh4.2.0 -DzincPort=3032
[ec2-user@ip-172-31-22-140 root]$
> pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while
> calling None.org.apache.spark.sql.hive.HiveContext.
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-17172
> URL: https://issues.apache.org/jira/browse/SPARK-17172
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.6.2
> Environment: spark version: 1.6.2
> python version: 3.4.2 (v3.4.2:ab2c023a9432, Oct 5 2014, 20:42:22)
> [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
> Reporter: Andrew Davidson
> Attachments: hiveUDFBug.html, hiveUDFBug.ipynb
>
>
> from pyspark.sql import HiveContext
> sqlContext = HiveContext(sc)
> # Define udf
> from pyspark.sql.functions import udf
> def scoreToCategory(score):
> if score >= 80: return 'A'
> elif score >= 60: return 'B'
> elif score >= 35: return 'C'
> else: return 'D'
>
> udfScoreToCategory=udf(scoreToCategory, StringType())
> throws exception
> Py4JJavaError: An error occurred while calling
> None.org.apache.spark.sql.hive.HiveContext.
> : java.lang.RuntimeException: java.lang.RuntimeException: Unable to
> instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
