Hello Shahab, Are you able to read tables created in Hive from Spark SQL ? If yes, how are you referring them ?
On Thu, Mar 26, 2015 at 1:11 PM, Takeshi Yamamuro <linguin....@gmail.com> wrote: > I think it is not `sqlContext` but hiveContext because `create temporary > function` is not supported in SQLContext. > > On Wed, Mar 25, 2015 at 5:58 AM, Jon Chase <jon.ch...@gmail.com> wrote: > >> Shahab - >> >> This should do the trick until Hao's changes are out: >> >> >> sqlContext.sql("create temporary function foobar as >> 'com.myco.FoobarUDAF'"); >> >> sqlContext.sql("select foobar(some_column) from some_table"); >> >> >> This works without requiring to 'deploy' a JAR with the UDAF in it - just >> make sure the UDAF is in your project's classpath. >> >> >> >> >> On Tue, Mar 10, 2015 at 8:21 PM, Cheng, Hao <hao.ch...@intel.com> wrote: >> >>> Oh, sorry, my bad, currently Spark SQL doesn’t provide the user >>> interface for UDAF, but it can work seamlessly with Hive UDAF (via >>> HiveContext). >>> >>> >>> >>> I am also working on the UDAF interface refactoring, after that we can >>> provide the custom interface for extension. >>> >>> >>> >>> https://github.com/apache/spark/pull/3247 >>> >>> >>> >>> >>> >>> *From:* shahab [mailto:shahab.mok...@gmail.com] >>> *Sent:* Wednesday, March 11, 2015 1:44 AM >>> *To:* Cheng, Hao >>> *Cc:* user@spark.apache.org >>> *Subject:* Re: Registering custom UDAFs with HiveConetxt in SparkSQL, >>> how? >>> >>> >>> >>> Thanks Hao, >>> >>> But my question concerns UDAF (user defined aggregation function ) not >>> UDTF( user defined type function ). >>> >>> I appreciate if you could point me to some starting point on UDAF >>> development in Spark. >>> >>> >>> >>> Thanks >>> >>> Shahab >>> >>> On Tuesday, March 10, 2015, Cheng, Hao <hao.ch...@intel.com> wrote: >>> >>> Currently, Spark SQL doesn’t provide interface for developing the >>> custom UDTF, but it can work seamless with Hive UDTF. >>> >>> >>> >>> I am working on the UDTF refactoring for Spark SQL, hopefully will >>> provide an Hive independent UDTF soon after that. >>> >>> >>> >>> *From:* shahab [mailto:shahab.mok...@gmail.com] >>> *Sent:* Tuesday, March 10, 2015 5:44 PM >>> *To:* user@spark.apache.org >>> *Subject:* Registering custom UDAFs with HiveConetxt in SparkSQL, how? >>> >>> >>> >>> Hi, >>> >>> >>> >>> I need o develop couple of UDAFs and use them in the SparkSQL. While >>> UDFs can be registered as a function in HiveContext, I could not find any >>> documentation of how UDAFs can be registered in the HiveContext?? so far >>> what I have found is to make a JAR file, out of developed UDAF class, and >>> then deploy the JAR file to SparkSQL . >>> >>> >>> >>> But is there any way to avoid deploying the jar file and register it >>> programmatically? >>> >>> >>> >>> >>> >>> best, >>> >>> /Shahab >>> >>> >> > > > -- > --- > Takeshi Yamamuro > -- Deepak