Hi, If I get your code right, you are trying to implement a sort of explode function for Map<Map<String,String>> that lists "aFieldName" values in the inner map. I think you could achieve this with a simple LATERAL VIEW explode(mapmap) T as key, map and then selecting map["aFieldName"].
If you want to go on with your UDTF (to complexify it I presume), I suggest you to have a look at the implementation of the explode UDTF https://github.com/apache/hive/blob/b8250ac2f30539f6b23ce80a20a9e338d3d31458/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java and try to do everything in a similar way. For instance I noticed that your forwardObj is of type ArrayList<String>, while it should probably be a a Text[]. Regards, Furcy 2014-09-09 21:33 GMT+02:00 Echo Li <echo...@gmail.com>: > Hi Furcy, > > Thanks for sharing. I modified my code to mark the map variables > "transient" but still got same error. this is the code: > > public class fun_name extends GenericUDTF { > > private PrimitiveObjectInspector stringOI = null; > > transient Map<String, Map<String, String>> mapObject; > transient Map<String, String> eventDetails; > > @Override > public void close() throws HiveException { > > } > > @Override > public StructObjectInspector initialize(ObjectInspector[] args) throws > UDFArgumentException { > > if (args.length != 1) { > throw new UDFArgumentException("fun_name() takes exactly one > argument"); > } > > if (args[0].getCategory() != ObjectInspector.Category.PRIMITIVE > && ((PrimitiveObjectInspector) args[0]).getPrimitiveCategory() > != PrimitiveObjectInspector.PrimitiveCategory.STRING) { > throw new UDFArgumentException("fun_name() takes a string as a > parameter"); > } > > stringOI = (PrimitiveObjectInspector) args[0]; > > > ArrayList<String> fieldNames = new ArrayList<String>(1); > ArrayList<ObjectInspector> fieldOIs = new > ArrayList<ObjectInspector>(1); > > fieldNames.add("field1"); > fieldNames.add("field2"); > > > fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector); > > fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector); > > return > ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, > fieldOIs); > } > > @Override > public void process(Object[] record) throws HiveException { > > ObjectMapper mapper = new ObjectMapper(); > > try { > > final String document = (String) > stringOI.getPrimitiveJavaObject(record[0]); > > mapObject = mapper.readValue(document, > new TypeReference<Map<String, Map<String, String>>>() { > }); > > for (String someId : mapObject.keySet()) { > > > eventDetails = mapObject.get(someId); > > List<String> forwardObj = new ArrayList<String>(); > > forwardObj.add(someId); > forwardObj.add(eventDetails.get("aFieldName")); > > forward(forwardObj); > } > > } > } > } > > to compile and build the jar: > > *javac -Xlint:deprecation fun_naemjavajar cvf from_json.jar > from_json.class 'fun_name$1.class'* > > add jar and create function in hive: > > > *add jar fun_name.jar;create temporary function fun_name as 'fun_name';* > succeeded; > > when run a query, got error: > > Vertex failed, vertexName=Map 1, vertexId=vertex_1407434664593_24842_1_00, > diagnostics=[Vertex Input: impression_test initializer failed., > o*rg.apache.hive.com.esotericsoftware.kryo.KryoException: > Unable to find class: fun_name* > Serialization trace: > genericUDTF (org.apache.hadoop.hive.ql.plan.UDTFDesc) > conf (org.apache.hadoop.hive.ql.exec.UDTFOperator) > childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator) > childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) > aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)] > DAG failed due to vertex failure. failedVertices:1 killedVertices:0 > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.tez.TezTask > > > could be something else? > > On Tue, Sep 9, 2014 at 1:31 AM, Furcy Pin <furcy....@flaminem.com> wrote: > >> Hi, >> >> I think I encountered this kind of serialization problem when writing >> UDFs. >> Usually, marking every fields of the UDF as *transient* does the trick. >> >> I guess the error means that Kryo tries to serialize the UDF class and >> everything that is inside, and by marking them as transient >> you ensure that it will not and that they will be instantiated in the >> default constructor or during the call of initialize() >> >> Please keep me informed if it works or not, >> >> Regards, >> >> Furcy >> >> >> 2014-09-09 1:44 GMT+02:00 Echo Li <echo...@gmail.com>: >> >>> I wrote a UDTF in hive 0.13, the function parse a column which is json >>> string and return a table. The function compiles successfully by adding >>> hive-exec-0.13.0.2.1.2.1-471.jar to classpath, however when the jar is >>> added to hive and a function created using the jar then I try to run a >>> query using that function, I got error: >>> >>> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find >>> class: class_name >>> >>> I went through all steps in a lower version hive (0.10) everything works >>> fine, I searched around and seams that is caused by the ‘kryo” serde, so my >>> question is, is there a fix? and where to find it? >>> >>> thank you. >>> >> >> >
