Hello, I am experiencing inconsistent behavior when trying to use UDF on 0.13.1 on Amazon's EMR (AMI 3.2.1).
I generated a uber jar and deployed a UDF like so: create temporary function someFunction as "hive.udf.localization.MyUDF" using jar "s3://waze.mapreduce.shared/scripts/Hive/MyHive.jar"; I am having these 2 (related?) problems: 1) When I simply try to use my UDF I get Error: java.lang.RuntimeException: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException Serialization trace: childRectangles (SomeClass1) statesTree (SomeClass2) states (SomeClass3) genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:360) at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:271) at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:431) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:410) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException... After digging I found this: http://mail-archives.apache.org/mod_mbox/hive-dev/201408.mbox/%3CJIRA.12733732.1407927053435.81293.1408008733570@arcas%3E *and after setting * *hive.plan.serialization.format=javaXML* *the UDF is running OK on my test data set of 50 lines.* 2) When running the UDF in a more complex 2-joins query I am getting a somewhat related error: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException Serialization trace: org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task Error: java.lang.NullPointerException Serialization trace: childRectangles (SomeClass1) statesTree (SomeClass2) states (SomeClass3) genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator) opParseCtxMap (org.apache.hadoop.hive.ql.plan.MapWork) mapWork (org.apache.hadoop.hive.ql.plan.MapredWork)... *This does not go away even after setting hive.plan.serialization.format=javaXML * Can someone please advise? Many thanks, Harel.
