[Spark Sql/ UDFs] Spark and Hive UDFs parity
Thanks Georg. But I'm not sure how mapPartitions is relevant here. Can you
elaborate?
On Thu, Jun 15, 2017 at 4:18 AM, Georg Heiler
<georg.kf.hei...@gmail.com<mailto:georg.kf.hei...@gmail.com>> wrote:
What about using map partitions
gt;>> which a lot of our users use. We have now a lot of UDFs in Hive which make
>>> use of these methods. We plan to move to UDFs to Spark UDFs but are being
>>> limited by not having similar lifecycle methods.
>>>Are there plans to address these? Or do people
ch a lot of our users use. We have now a lot of UDFs in Hive which make
>> use of these methods. We plan to move to UDFs to Spark UDFs but are being
>> limited by not having similar lifecycle methods.
>>Are there plans to address these? Or do people usually adopt some sort
ere plans to address these? Or do people usually adopt some sort
> of workaround?
>
>If we directly use the Hive UDFs in Spark we pay a performance
> penalty. I think Spark anyways does a conversion from InternalRow to Row
> back to InternalRow for native spark udfs and fo
which make
use of these methods. We plan to move to UDFs to Spark UDFs but are being
limited by not having similar lifecycle methods.
Are there plans to address these? Or do people usually adopt some sort
of workaround?
If we directly use the Hive UDFs in Spark we pay a performance
) as reversedString)
Thanks in advance!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Hive-UDFs-tp23707.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
dataframe.limit(1).selectExpr(xxx).collect()?
-Original Message-
From: chrish2312 [mailto:c...@palantir.com]
Sent: Wednesday, July 8, 2015 6:20 AM
To: user@spark.apache.org
Subject: Hive UDFs
I know the typical way to apply a hive UDF to a dataframe is basically
something like
://apache-spark-user-list.1001560.n3.nabble.com/Is-there-a-way-to-access-Hive-UDFs-in-a-HiveContext-tp21510p21527.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr
Version: Spark 1.2 on CDH 5.3.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Is-there-a-way-to-access-Hive-UDFs-in-a-HiveContext-tp21510.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
