> On March 2, 2018, 11:19 p.m., Ashutosh Chauhan wrote: > > data/conf/hive-site.xml > > Lines 338 (patched) > > <https://reviews.apache.org/r/65422/diff/6/?file=1968339#file1968339line338> > > > > Is there a reason for this? AFAICT existing tests should not get > > affected by this, so this shouldn't impact tests, so we might as well turn > > it to true.
at some point I was not sure if this patch is broking some tests or not; but I'll submit it with and without this for testing right now...I think some negative tests will be broken - because the reexecution will cause some new log lines...I'll remove this if possible in the next patch > On March 2, 2018, 11:19 p.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/Driver.java > > Lines 652 (patched) > > <https://reviews.apache.org/r/65422/diff/6/?file=1968353#file1968353line661> > > > > We want to compute operator signatures and store stats corresponding to > > it only if we decide to reexecute it. Currently, it seems it > > unconditionally does this. Also, at this point plan hasn't been executed, > > so there are no stats for it. So, is this only computing signatures? added a conditional; yes, it's only computing the signatures - the idea is that this call will calculate the "plan independent" part of the planmapper. > On March 2, 2018, 11:19 p.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java > > Lines 1581 (patched) > > <https://reviews.apache.org/r/65422/diff/6/?file=1968362#file1968362line1588> > > > > Also add a comment that its currently used only when reoptimization is > > attempted to match operators on 2 trees. I think in HIVE-18703 I will probably move this method to somewhere else... > On March 2, 2018, 11:19 p.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java > > Lines 111 (patched) > > <https://reviews.apache.org/r/65422/diff/6/?file=1968412#file1968412line111> > > > > Can there be a semantics about whether subsequent plugin run depends on > > results on plugins before it? I am thinking of a scenario where ReOptimize > > plugin executes first but it fails to generate an alternate plan (either > > because it can't get stats for an operator it was looking for or can't > > match op trees or whatever). So, we will know before executing second time > > via Reoptimize plugin that it won't be successful. At that time we make > > Overlay plugin go with auto.convert.join=off. I think this will be useful, > > although can be a follow-up. that would be an interesting extension...I don't know how would it be the best right now...but I think this should be investigated in a followup - Zoltan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65422/#review198543 ----------------------------------------------------------- On March 1, 2018, 3:35 p.m., Zoltan Haindrich wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65422/ > ----------------------------------------------------------- > > (Updated March 1, 2018, 3:35 p.m.) > > > Review request for hive and Ashutosh Chauhan. > > > Bugs: HIVE-17626 > https://issues.apache.org/jira/browse/HIVE-17626 > > > Repository: hive-git > > > Description > ------- > > preview > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6 > data/conf/hive-site.xml b56cbd2469 > data/conf/llap/hive-site.xml c4c299c5de > data/conf/perf-reg/spark/hive-site.xml 5ca660d4e7 > data/conf/perf-reg/tez/hive-site.xml 62ecb74f3f > data/conf/rlist/hive-site.xml 630e481edc > data/conf/spark/local/hive-site.xml 8ff6256638 > data/conf/spark/standalone/hive-site.xml 84851c769c > data/conf/spark/yarn-client/hive-site.xml 6c633623ab > data/conf/tez/hive-site.xml 236adc7087 > > druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java > cbeac2c00a > itests/src/test/resources/testconfiguration.properties 2776fe95f1 > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java > 6cd7a136ae > itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java > c01d87bf51 > ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b > ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93 > ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 > ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 > ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 > ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java > 08b791ad42 > ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java > 131127e50d > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03 > ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java > 1e28ca843f > ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java > 2b345d6ec7 > ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 > ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java > 395a5f450f > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java > 085d6a7d94 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java > 0d31e6e422 > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java > 8dd7cfe58c > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java > 134fc0ff0b > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java > 1eb72ce4d9 > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java > 384bd74686 > ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java > b75850760f > ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java > b0cf3bd94e > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java > 8c1bcb3f62 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java > 783a672c47 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java > d1c53cf345 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java > 9a3f81c98f > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1 > ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java > 714cf3914b > ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java > 7d5be6ba81 > ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java > 7332693513 > ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java > 5d3fdb8b63 > ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ce61fc5a2e > ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08 > ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 86cc77d43b > ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java > 9c651ab3ab > ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af28 > ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1 > ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java > 3837a49934 > ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00 > ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cf4ab606f2 > ql/src/java/org/apache/hadoop/hive/ql/plan/OperatorDesc.java 870b4d9a26 > ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java bf24ff8f09 > ql/src/java/org/apache/hadoop/hive/ql/plan/ScriptDesc.java 858de98229 > ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java e38e7e468f > ql/src/java/org/apache/hadoop/hive/ql/plan/Signature.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/SignatureUtils1.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6 > ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 59968fa8ef > ql/src/java/org/apache/hadoop/hive/ql/plan/UDTFDesc.java cf8e6e547e > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java > 6c469bf944 > > ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java > fa3f89fda1 > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOpSigFactory.java > PRE-CREATION > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java > PRE-CREATION > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java > PRE-CREATION > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java > PRE-CREATION > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java > PRE-CREATION > ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java cc97317f55 > ql/src/test/org/apache/hive/testutils/TestHiveTestEnvSetup.java 91613667f8 > ql/src/test/org/apache/hive/testutils/TestX1.java PRE-CREATION > ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION > ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION > ql/src/test/queries/clientpositive/retry_failure_stat_changes.q > PRE-CREATION > ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b > ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1 > ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f68 > ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out eb54a81d1a > ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 590437c708 > ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out > c7c8993d0e > ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION > ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out > PRE-CREATION > ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out > PRE-CREATION > ql/src/test/results/clientpositive/llap/tez_input_counters.q.out 71d14f4049 > ql/src/test/results/clientpositive/show_functions.q.out 4e8ceaf6af > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/hooks/URIResolverHook.java > d3be5dd0a2 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/messaging/CreateDatabaseMessage.java > 3d64c7341a > > > Diff: https://reviews.apache.org/r/65422/diff/6/ > > > Testing > ------- > > > Thanks, > > Zoltan Haindrich > >
