As suggested, I changed the query like this: select x.f1,x,f2,x,f3,x.f4 from ( select e.f1 as f1,e.f2 as f2,e.f3 as f3,e.f4 as f4 from mytable LATERAL VIEW myfunc(p1,p2,p3,p4) e as f1,f2,f3,f4 where lang=123) x where x.f3 is not null;
And it still doesn't work. I am getting the same error. If anyone has any ideas, please let me know. Thanks. On Friday, February 21, 2014 11:27 AM, Kumar V <kumarbuyonl...@yahoo.com> wrote: Line 316 in my UDTF where is shows the error is the line where I call forward(). The whole trace is : Caused by: java.lang.RuntimeException: cannot find field key from [0:_col0, 1:_col2, 2:_col6, 3:_col7, 4:_col8, 5:_col9] at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:346) at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:143) at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57) at org.apache.hadoop.hive.ql.exec.ExprNodeFieldEvaluator.initialize(ExprNodeFieldEvaluator.java:55) at org.apache.hadoop.hive.ql.exec.ExprNodeFieldEvaluator.initialize(ExprNodeFieldEvaluator.java:55) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:128) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:128) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:128) at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:85) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800) at org.apache.hadoop.hive.ql.exec.LateralViewJoinOperator.processOp(LateralViewJoinOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800) at org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:112) at org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:44) at org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:81) at pb2CSVReadFile.FlatTableFileUDTFTx.process(FlatTableFileUDTFTx.java:316) at org.apache.hadoop.hive.ql.exec.UDTFOperator.processOp(UDTFOperator.java:98) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800) at org.apache.hadoop.hive.ql.exec.LateralViewForwardOperator.processOp(LateralViewForwardOperator.java:37) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800) at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:132) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:658) ... 9 more On Friday, February 21, 2014 11:18 AM, java8964 <java8...@hotmail.com> wrote: What is your stracktrace? Can you paste here? It is maybe a different bug. If you put e.f3 <> null at an outsider query? Does that work? Or maybe you have to enhance your UDTF to push that filter into your UDTF. It is not perfect, but maybe a solution for you as now. You can create a new Jira if it is a new bug. Yong ________________________________ Date: Fri, 21 Feb 2014 07:18:32 -0800 From: kumarbuyonl...@yahoo.com Subject: java.lang.RuntimeException: cannot find field key from [0:_col0, 1:_col2, 2:_col3] To: user@hive.apache.org Hi, I have a UDTF which works fine except when I do a query like the following : select e.* from mytable LATERAL VIEW myfunc(p1,p2,p3,p4) e as f1,f2,f3,f4 where lang=123and e.f3 <> null; The error I see is: java.lang.RuntimeException: cannot find field key from [0:_col0, 1:_col2, 2:_col3] If i remove 'and e.f3 <> null' from the WHERE clause, it works fine. Also, with e.f3 <> null in the WHERE clause, if I add the setting hive.optimize.ppd=false it works fine, but now, instead of using 600 mappers, it uses about 10,000 mappers and runs for more than 2 hours instead of a few minutes. I am using hive 0.10. I saw the jira HIVE-3226 which says that it has been fixed in hive 0.10. Is this the bug that I am hitting now ? Any other ideas of how to make it work ? I am actually on CDH 4.4 which has hive 0.10. Please let me know. Thanks, Murali.
