GitHub user maropu opened a pull request:
https://github.com/apache/spark/pull/7248
[SPARK-6747] [SQL] Throw an AnalysisException when unsupported Java list
types used in Hive UDF
The current implementation can't handle List<> as a return type in Hive UDF
and
throws meaningless Match Error.
We assume an UDF below;
public class UDFToListString extends UDF {
public List<String> evaluate(Object o)
{ return Arrays.asList("xxx", "yyy", "zzz"); }
}
An exception of scala.MatchError is thrown as follows when the UDF used;
scala.MatchError: interface java.util.List (of class java.lang.Class)
at
org.apache.spark.sql.hive.HiveInspectors$class.javaClassToDataType(HiveInspectors.scala:174)
at
org.apache.spark.sql.hive.HiveSimpleUdf.javaClassToDataType(hiveUdfs.scala:76)
at
org.apache.spark.sql.hive.HiveSimpleUdf.dataType$lzycompute(hiveUdfs.scala:106)
at org.apache.spark.sql.hive.HiveSimpleUdf.dataType(hiveUdfs.scala:106)
at
org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:131)
at
org.apache.spark.sql.catalyst.planning.PhysicalOperation$$anonfun$collectAliases$1.applyOrElse(patterns.scala:95)
at
org.apache.spark.sql.catalyst.planning.PhysicalOperation$$anonfun$collectAliases$1.applyOrElse(patterns.scala:94)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at
scala.collection.TraversableLike$$anonfun$collect$1.apply(TraversableLike.scala:278)
...
To make udf developers more understood, we need to throw a more suitable
exception.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/spark FixBugInHiveInspectors
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7248.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7248
----
commit e553f10c67f129e59c526d6149e60c77348f546a
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-04-07T16:41:17Z
Support List as a return type in Hive UDF
commit e21ce7e33139b96581dbd37785f99854e60379fd
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-04-08T02:18:22Z
Add a blank line at the end of UDFToListString
commit 94064169380d129a41574e7ee3ccc994532f71f0
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-04-14T16:46:44Z
Apply review comments
commit f965c34bcea8efa6b58443ef31c4d857b5feb09e
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-04-14T18:04:31Z
Fix code-style errors
commit 1c7b9d1aae14de43ca9bf6e3414a45f1104e5043
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-04-17T00:51:14Z
Remove a new type
commit a4887127834fc73063851fd57f095ff212f9b483
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-05-15T04:36:50Z
Add StringToUtf8 to comvert String into UTF8String
commit 21e8763ccba103187561c64a2fa6e38286ff13b1
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-05-15T07:18:40Z
Add TODO comments in UDFToListString of HiveUdfSuite
commit 1e823167986113cc264be13bd22c631eeb1452e4
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-05-16T02:58:46Z
Apply comments
commit ee232db94ffcb08b8cd33b94d03543bb4a663b60
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-04-07T16:41:17Z
Support List as a return type in Hive UDF
commit 93e3d4e7e1fc93e95ce09e01318e3f6fb7639d8b
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-04-08T02:18:22Z
Add a blank line at the end of UDFToListString
commit 6984bf4067ef8d98c21097cd2b2265aece3b8a87
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-04-14T16:46:44Z
Apply review comments
commit 7f812fd8f6d07f6779011e2e98b631733264f2a9
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-04-14T18:04:31Z
Fix code-style errors
commit af61f2ecc821cede51a0bb446ba67230d39b3f10
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-04-17T00:51:14Z
Remove a new type
commit fdb2ae4a01b6cea154311632078f8fb3026ce945
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-05-15T04:36:50Z
Add StringToUtf8 to comvert String into UTF8String
commit 7114a4735a4ccbd1211b10fa757f0e27012ef96d
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-05-15T07:18:40Z
Add TODO comments in UDFToListString of HiveUdfSuite
commit 2844a8e6c4cbfafafadff7e072a971c6656ae113
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-05-16T02:58:46Z
Apply comments
commit 92ed7a6710ced76823365f493f753bc6ca947b30
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-07-06T23:11:33Z
Throw an exception when java list type used
commit 56305deddff7b7db1d800d72a2a43b09cbf127cc
Author: Takeshi YAMAMURO <linguin....@gmail.com>
Date: 2015-07-06T23:43:01Z
Fix conflicts
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
