[ https://issues.apache.org/jira/browse/HIVE-18827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381123#comment-16381123 ]
Jason Dere commented on HIVE-18827: ----------------------------------- Is this causing the LLAP IO read to fail, or just lots of annoying warning messges in the logs? Most likely this may be occurring because the LLAP IO elevator is going ahead with the read of a table, before the dynamic values from the semijoin reduction (min/max/bloomfilter) have been delivered from the other side of the join. The current behavior in the SearchArgumentImpl is to simply propagate the NoDynamicValuesException (which causes YES_NO and selects the row group for reading), but looking at the ORC min/max evaluation it looks like it can gracefully handle null/empty values being returned in this case. In which case we can change the behavior of the SearchArgumentImpl here to not propagate the exception. > useless dynamic value exceptions strike back > -------------------------------------------- > > Key: HIVE-18827 > URL: https://issues.apache.org/jira/browse/HIVE-18827 > Project: Hive > Issue Type: Bug > Reporter: Sergey Shelukhin > Priority: Major > > Looking at ~master, I can see tons of exceptions like this in LLAP log: > {noformat} > 2018-02-27T14:07:51,989 WARN [IO-Elevator-Thread-12 > (1515669035295_0909_1_08_000117_0)] impl.RecordReaderImpl: > NoDynamicValuesException when evaluating predicate. Skipping ORC PPD. Stats: > numberOfValues: 9750 > intStatistics { > minimum: 11335 > maximum: 5999960 > sum: 27648854404 > } > hasNull: true > Predicate: (BETWEEN ss_addr_sk > DynamicValue(RS_27_customer_address_ca_address_sk_min) > DynamicValue(RS_27_customer_address_ca_address_sk_max)) > org.apache.hadoop.hive.ql.plan.DynamicValue$NoDynamicValuesException: Value > does not exist in registry: RS_27_customer_address_ca_address_sk_min > at > org.apache.hadoop.hive.ql.exec.tez.DynamicValueRegistryTez.getValue(DynamicValueRegistryTez.java:77) > ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:137) > ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getJavaValue(DynamicValue.java:97) > ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getLiteral(DynamicValue.java:93) > ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.getLiteralList(SearchArgumentImpl.java:120) > ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateMinMax(RecordReaderImpl.java:553) > ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:463) > ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateProto(RecordReaderImpl.java:423) > ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at > org.apache.orc.impl.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:848) > ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.determineRgsToRead(OrcEncodedDataReader.java:835) > ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:335) > ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:276) > ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:273) > ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at java.security.AccessController.doPrivileged(Native Method) > ~[?:1.8.0_112] > at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112] > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965) > ~[hadoop-common-3.0.0.3.0.0.0-776.jar:?] > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:273) > ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:110) > ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > ~[tez-common-0.9.2-SNAPSHOT.jar:0.9.2-SNAPSHOT] > at > org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110) > ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[?:1.8.0_112] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > ~[?:1.8.0_112] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > ~[?:1.8.0_112] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)