Re: Can't create UDF through thriftserver, no error reported

[Antonio Piccolboni]

Mmmmh, I may have been stuck with some stale lib. A complete reset of the
client (when by complete I mean beyond what reason would require) solved
the problem. I think we can consider this solved unless new evidence
appears. Thanks!

On Wed, Dec 16, 2015 at 9:16 AM Antonio Piccolboni 
wrote:

> Can repro only in custom client, not beeline. Let me dig deeper.
>
> On Wed, Dec 16, 2015 at 9:03 AM Antonio Piccolboni <
> anto...@piccolboni.info> wrote:
>
>> Hi Jeff,
>> the ticket is certainly relevant, thanks for digging it out, but as I
>> said I can repro in 1.6.0-rc2. Will try again just to make sure.
>>
>> On Tue, Dec 15, 2015 at 5:17 PM Jeff Zhang  wrote:
>>
>>> It should be resolved by this ticket
>>> https://issues.apache.org/jira/browse/SPARK-11191
>>>
>>>
>>>
>>> On Wed, Dec 16, 2015 at 3:14 AM, Antonio Piccolboni <
>>> anto...@piccolboni.info> wrote:
>>>
 Hi,
 I am trying to create a UDF using the thiftserver. I followed this
 example , which is originally
 for hive. My understanding is that the thriftserver creates a hivecontext
 and Hive UDFs should be supported. I then sent this query to the
 thriftserver (I use the RJDBC module for R but I doubt any other JDBC
 client would be any different):


 CREATE TEMPORARY FUNCTION NVL2 AS 'khanolkar.HiveUDFs.NVL2GenericUDF'

 I only changed some name wrt  the posted examples, but I think the
 class was found just right because 1)There's no errors in the log or
 console 2)I can generate a class not found error mistyping the class name,
 and I see it in the logs 3) I can use the reflect builtin to invoke a
 different function that I wrote and supplied to spark in the same way
 (--jars option to start-thriftserver)

 After this, I can't use the NVL2 function in a query and I can't even
 do a  DESCRIBE query on it,  nor does it list with SHOW FUNCTIONS. I tried
 both 1.5.1 and 1.6.0-rc2 built with thriftserver support for Hadoop 2.6

 I know the HiveContext is slightly behind the latest Hive as far as
 features, I believe one or two revs, so that may be one potential problem,
 but all these feature I believe are present in Hive 0.11 and should have
 made it into Spark. At the very least, I would like to see some message in
 the logs and console so that I can find the error of my ways, repent and
 fix my code. Any suggestions? Anything I should post to support
 troubleshooting? Is this JIRA-worthy? Thanks

 Antonio




>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>

Re: Can't create UDF through thriftserver, no error reported

[Antonio Piccolboni]

Hi Jeff,
the ticket is certainly relevant, thanks for digging it out, but as I said
I can repro in 1.6.0-rc2. Will try again just to make sure.

On Tue, Dec 15, 2015 at 5:17 PM Jeff Zhang  wrote:

> It should be resolved by this ticket
> https://issues.apache.org/jira/browse/SPARK-11191
>
>
>
> On Wed, Dec 16, 2015 at 3:14 AM, Antonio Piccolboni <
> anto...@piccolboni.info> wrote:
>
>> Hi,
>> I am trying to create a UDF using the thiftserver. I followed this
>> example , which is originally
>> for hive. My understanding is that the thriftserver creates a hivecontext
>> and Hive UDFs should be supported. I then sent this query to the
>> thriftserver (I use the RJDBC module for R but I doubt any other JDBC
>> client would be any different):
>>
>>
>> CREATE TEMPORARY FUNCTION NVL2 AS 'khanolkar.HiveUDFs.NVL2GenericUDF'
>>
>> I only changed some name wrt  the posted examples, but I think the class
>> was found just right because 1)There's no errors in the log or console 2)I
>> can generate a class not found error mistyping the class name, and I see it
>> in the logs 3) I can use the reflect builtin to invoke a different function
>> that I wrote and supplied to spark in the same way (--jars option to
>> start-thriftserver)
>>
>> After this, I can't use the NVL2 function in a query and I can't even do
>> a  DESCRIBE query on it,  nor does it list with SHOW FUNCTIONS. I tried
>> both 1.5.1 and 1.6.0-rc2 built with thriftserver support for Hadoop 2.6
>>
>> I know the HiveContext is slightly behind the latest Hive as far as
>> features, I believe one or two revs, so that may be one potential problem,
>> but all these feature I believe are present in Hive 0.11 and should have
>> made it into Spark. At the very least, I would like to see some message in
>> the logs and console so that I can find the error of my ways, repent and
>> fix my code. Any suggestions? Anything I should post to support
>> troubleshooting? Is this JIRA-worthy? Thanks
>>
>> Antonio
>>
>>
>>
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Can't create UDF through thriftserver, no error reported

[Antonio Piccolboni]

Hi,
I am trying to create a UDF using the thiftserver. I followed this example
, which is originally for hive. My
understanding is that the thriftserver creates a hivecontext and Hive UDFs
should be supported. I then sent this query to the thriftserver (I use the
RJDBC module for R but I doubt any other JDBC client would be any
different):


CREATE TEMPORARY FUNCTION NVL2 AS 'khanolkar.HiveUDFs.NVL2GenericUDF'

I only changed some name wrt  the posted examples, but I think the class
was found just right because 1)There's no errors in the log or console 2)I
can generate a class not found error mistyping the class name, and I see it
in the logs 3) I can use the reflect builtin to invoke a different function
that I wrote and supplied to spark in the same way (--jars option to
start-thriftserver)

After this, I can't use the NVL2 function in a query and I can't even do a
 DESCRIBE query on it,  nor does it list with SHOW FUNCTIONS. I tried both
1.5.1 and 1.6.0-rc2 built with thriftserver support for Hadoop 2.6

I know the HiveContext is slightly behind the latest Hive as far as
features, I believe one or two revs, so that may be one potential problem,
but all these feature I believe are present in Hive 0.11 and should have
made it into Spark. At the very least, I would like to see some message in
the logs and console so that I can find the error of my ways, repent and
fix my code. Any suggestions? Anything I should post to support
troubleshooting? Is this JIRA-worthy? Thanks

Antonio

Re: Can't create UDF through thriftserver, no error reported

[Jeff Zhang]

It should be resolved by this ticket
https://issues.apache.org/jira/browse/SPARK-11191



On Wed, Dec 16, 2015 at 3:14 AM, Antonio Piccolboni  wrote:

> Hi,
> I am trying to create a UDF using the thiftserver. I followed this example
> , which is originally for hive.
> My understanding is that the thriftserver creates a hivecontext and Hive
> UDFs should be supported. I then sent this query to the thriftserver (I use
> the RJDBC module for R but I doubt any other JDBC client would be any
> different):
>
>
> CREATE TEMPORARY FUNCTION NVL2 AS 'khanolkar.HiveUDFs.NVL2GenericUDF'
>
> I only changed some name wrt  the posted examples, but I think the class
> was found just right because 1)There's no errors in the log or console 2)I
> can generate a class not found error mistyping the class name, and I see it
> in the logs 3) I can use the reflect builtin to invoke a different function
> that I wrote and supplied to spark in the same way (--jars option to
> start-thriftserver)
>
> After this, I can't use the NVL2 function in a query and I can't even do a
>  DESCRIBE query on it,  nor does it list with SHOW FUNCTIONS. I tried both
> 1.5.1 and 1.6.0-rc2 built with thriftserver support for Hadoop 2.6
>
> I know the HiveContext is slightly behind the latest Hive as far as
> features, I believe one or two revs, so that may be one potential problem,
> but all these feature I believe are present in Hive 0.11 and should have
> made it into Spark. At the very least, I would like to see some message in
> the logs and console so that I can find the error of my ways, repent and
> fix my code. Any suggestions? Anything I should post to support
> troubleshooting? Is this JIRA-worthy? Thanks
>
> Antonio
>
>
>
>


-- 
Best Regards

Jeff Zhang