[jira] [Commented] (HIVE-21713) Explain consistency Task/Operator
[ https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885988#comment-16885988 ] Zoltan Haindrich commented on HIVE-21713: - Makes sense to me :) Could you submit a patch with the proposed changes? note: optionally, you could write a hook to programatically check that operator/task names conform to the new naming conventions an example of a hook like that could be found here: ./ql/src/java/org/apache/hadoop/hive/ql/hooks/NoOperatorReuseCheckerHook.java and if you add it to "hive.exec.post.hooks" in "data/conf/llap/hive-site.xml" - it will be invoked during qtest runs > Explain consistency Task/Operator > - > > Key: HIVE-21713 > URL: https://issues.apache.org/jira/browse/HIVE-21713 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Manoj Narayanan >Priority: Major > > there seems to be some inconsistency what gets printed in the explain; > for example "MoveTask" is shown as "Move Operator" in the explain > Becase Tasks and Operators are basically different things - this should be > corrected as it may cause confusion -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21713) Explain consistency Task/Operator
[ https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884427#comment-16884427 ] Manoj Narayanan commented on HIVE-21713: |OperatorClass|OfType|Current Explain String|Proposed Name|NeedsChange|hasOperator|hasTask|ChangeToOperator|AddOperator| |CommonMergeJoinOperator|CommonMergeJoinDesc|Merge Join Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |SMBMapJoinOperator|SMBJoinDesc|Sorted Merge Bucket Map Join Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |ScriptOperator|ScriptDesc|Transform Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |AbstractMapOperator|MapWork|NO_EXPLAIN_STRING| |FALSE|FALSE|FALSE|FALSE|TRUE| |AbstractMapJoinOperator|MapJoinDesc|Map Join Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |GroupByOperator|GroupByDesc|Group By Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |LimitOperator|LimitDesc|Limit|Limit Operator|TRUE|FALSE|FALSE|FALSE|TRUE| |AbstractFileMergeOperator|FileMergeDesc|NO_EXPLAIN_STRING| |FALSE|FALSE|FALSE|FALSE|TRUE| |HashTableDummyOperator|HashTableDummyDesc|HashTable Dummy Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |SelectOperator|SelectDesc|Select Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |LateralViewJoinOperator|LateralViewJoinDesc|Lateral View Join Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |ReduceSinkOperator|ReduceSinkDesc|Reduce Output Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |HashTableSinkOperator|HashTableSinkDesc|HashTable Sink Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |ForwardOperator|ForwardDesc|Forward|Forward Operator|TRUE|FALSE|FALSE|FALSE|TRUE| |TableScanOperator|TableScanDesc|TableScan|TableScan Operator|TRUE|FALSE|FALSE|FALSE|TRUE| |TopNKeyOperator|TopNKeyDesc|Top N Key Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |TopNKeyOperator|TopNKeyDesc|keys|TopNKeyOperator|TRUE|FALSE|FALSE|FALSE|TRUE| |FileSinkOperator|FileSinkDesc|File Output Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |JoinOperator|JoinDesc|Join Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |UnionOperator|UnionDesc|Union|Union Operator|TRUE|FALSE|FALSE|FALSE|TRUE| |UDTFOperator|UDTFDesc|UDTF Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |CollectOperator|CollectDesc|Collect|Collect Operator|TRUE|FALSE|FALSE|FALSE|TRUE| |FilterOperator|FilterDesc|Filter Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |MapJoinOperator|MapJoinDesc|Map Join Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |DemuxOperator|DemuxDesc|Demux Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |PTFOperator|PTFDesc|PTF Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |AppMasterEventOperator|AppMasterEventDesc|Application Master Event Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |LateralViewForwardOperator|LateralViewForwardDesc|Lateral View Forward|Lateral View Forward Operator|TRUE|FALSE|FALSE|FALSE|TRUE| |CommonJoinOperator|JoinDesc|Join Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |DummyStoreOperator|DummyStoreDesc|Dummy Store|Dummy Store Operator|TRUE|FALSE|FALSE|FALSE|TRUE| |MuxOperator|MuxDesc|Mux Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |ListSinkOperator|ListSinkDesc|ListSink|List Sink Operator|TRUE|FALSE|FALSE|FALSE|TRUE| |FetchOperator|FetchWork|Fetch Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |MapOperator| |NO_EXPLAIN_STRING| | | | | | | |OrcFileMergeOperator|OrcFileMergeDesc|ORC File Merge Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |RCFileMergeOperator|RCFileMergeDesc|RCFile Merge Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |SparkHashTableSinkOperator|SparkHashTableSinkDesc|Spark HashTable Sink Operator| |FALSE|TRUE|FALSE|FALSE|FALSE| |TemporaryHashSinkOperator| |HashTable Sink Operator|Temporary HashSink Operator| | | | | | |TezDummyStoreOperator| |Dummy Store|Tez DummyStore Operator| | | | | | | | | | | | | | | | |OperatorClass|OfType|ExplainString| |NeedsChange|hasOperator|hasTask|ChangeToTask|AddTask| | | | | | | | | | | |FunctionTask|FunctionWork|NO_EXPLAIN_STRING|Function Task|TRUE|FALSE|FALSE|FALSE|TRUE| |DDLTask|DDLWork|NO_EXPLAIN_STRING|DDL Task|TRUE|FALSE|FALSE|FALSE|TRUE| |FetchTask|FetchWork|Fetch Operator|Fetch Task|TRUE|FALSE|TRUE|TRUE|FALSE| |ReplCopyTask|ReplCopyWork|Repl Copy|Replication Copy Task|TRUE|FALSE|FALSE|FALSE|TRUE| |ConditionalTask|ConditionalWork|Conditional Operator|Conditional Task|TRUE|FALSE|TRUE|TRUE|FALSE| |MaterializedViewTask|MaterializedViewDesc|Materialized View Work|MaterializedView Task|TRUE|FALSE|FALSE|FALSE|TRUE| |ColumnStatsUpdateTask|ColumnStatsUpdateWork|Column Stats Update Work|Column Stats UpdateTask|TRUE|FALSE|FALSE|FALSE|TRUE| |ReplTxnTask|ReplTxnWork|Replication Transaction|Replication Transaction Task|TRUE|FALSE|FALSE|FALSE|TRUE| |DependencyCollectionTask|DependencyCollectionWork|Dependency Collection|Dependency Collection Task|TRUE|FALSE|FALSE|FALSE|TRUE| |ExplainTask|ExplainWork|NO_EXPLAIN_STRING|Explain Task|TRUE|FALSE|FALSE|FALSE|TRUE| |ExportTask|ExportWork|Export Work|Export Task|TRUE|FALSE|FALSE|FALSE|TRUE| |CopyTask|CopyWork|Copy|Copy Task|TRUE|FALSE|FALSE|FALSE|TRUE| |MoveTask|MoveWork|Move Oper
[jira] [Commented] (HIVE-21713) Explain consistency Task/Operator
[ https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884425#comment-16884425 ] Manoj Narayanan commented on HIVE-21713: Ok. I think we should * Have Operator or Task at the end based on the type. * Operator/Task Class name where words are separated by space. (That's how currently most of Operators are named) * No changes if already above rule is met. (for e.g.:Merge Join Operator) > Explain consistency Task/Operator > - > > Key: HIVE-21713 > URL: https://issues.apache.org/jira/browse/HIVE-21713 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Manoj Narayanan >Priority: Major > > there seems to be some inconsistency what gets printed in the explain; > for example "MoveTask" is shown as "Move Operator" in the explain > Becase Tasks and Operators are basically different things - this should be > corrected as it may cause confusion -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21713) Explain consistency Task/Operator
[ https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880091#comment-16880091 ] Zoltan Haindrich commented on HIVE-21713: - wow :) what a great overview about these things! sorry for getting back to this just now...I've overlooked the notification... there are quite a few "very" interesting onse; like "keys" for TopNKeyOperator :D I think [~mnarayanan2018] you have exposed this problem very well; some of them are very exotic - I think the best would be if you could came up with a logical naming scheme and propose to rename them! > Explain consistency Task/Operator > - > > Key: HIVE-21713 > URL: https://issues.apache.org/jira/browse/HIVE-21713 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Manoj Narayanan >Priority: Major > > there seems to be some inconsistency what gets printed in the explain; > for example "MoveTask" is shown as "Move Operator" in the explain > Becase Tasks and Operators are basically different things - this should be > corrected as it may cause confusion -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21713) Explain consistency Task/Operator
[ https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873410#comment-16873410 ] Manoj Narayanan commented on HIVE-21713: In the above comment I tried to document each Operator and Task I could find and what their 'ExplainString' is. Also tried to note whether it should be changed (Column Needschange) and if so whether it should 'Add Operator' or 'Change to Operator' or 'Add Task' or 'Change to Task'. > Explain consistency Task/Operator > - > > Key: HIVE-21713 > URL: https://issues.apache.org/jira/browse/HIVE-21713 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > > there seems to be some inconsistency what gets printed in the explain; > for example "MoveTask" is shown as "Move Operator" in the explain > Becase Tasks and Operators are basically different things - this should be > corrected as it may cause confusion -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21713) Explain consistency Task/Operator
[ https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873408#comment-16873408 ] Manoj Narayanan commented on HIVE-21713: |OperatorClass|OfType|ExplainString|hasOperator|hasTask|NeedsChange|ChangeToOperator|AddOperator| |CommonMergeJoinOperator|CommonMergeJoinDesc|Merge Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |SMBMapJoinOperator|SMBJoinDesc|Sorted Merge Bucket Map Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |ScriptOperator|ScriptDesc|Transform Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |AbstractMapOperator|MapWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE| |AbstractMapJoinOperator|MapJoinDesc|Map Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |GroupByOperator|GroupByDesc|Group By Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |LimitOperator|LimitDesc|Limit|FALSE|FALSE|TRUE|FALSE|TRUE| |AbstractFileMergeOperator|FileMergeDesc|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE| |HashTableDummyOperator|HashTableDummyDesc|HashTable Dummy Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |SelectOperator|SelectDesc|Select Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |LateralViewJoinOperator|LateralViewJoinDesc|Lateral View Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |ReduceSinkOperator|ReduceSinkDesc|Reduce Output Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |HashTableSinkOperator|HashTableSinkDesc|HashTable Sink Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |ForwardOperator|ForwardDesc|Forward|FALSE|FALSE|TRUE|FALSE|TRUE| |TableScanOperator|TableScanDesc|TableScan|FALSE|FALSE|TRUE|FALSE|TRUE| |TopNKeyOperator|TopNKeyDesc|Top N Key Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |TopNKeyOperator|TopNKeyDesc|keys|FALSE|FALSE|TRUE|FALSE|TRUE| |FileSinkOperator|FileSinkDesc|File Output Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |JoinOperator|JoinDesc|Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |UnionOperator|UnionDesc|Union|FALSE|FALSE|TRUE|FALSE|TRUE| |UDTFOperator|UDTFDesc|UDTF Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |CollectOperator|CollectDesc|Collect|FALSE|FALSE|TRUE|FALSE|TRUE| |FilterOperator|FilterDesc|Filter Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |MapJoinOperator|MapJoinDesc|Map Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |DemuxOperator|DemuxDesc|Demux Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |PTFOperator|PTFDesc|PTF Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |AppMasterEventOperator|AppMasterEventDesc|Application Master Event Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |LateralViewForwardOperator|LateralViewForwardDesc|Lateral View Forward|FALSE|FALSE|TRUE|FALSE|TRUE| |CommonJoinOperator|JoinDesc|Join Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |DummyStoreOperator|DummyStoreDesc|Dummy Store|FALSE|FALSE|TRUE|FALSE|TRUE| |MuxOperator|MuxDesc|Mux Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |ListSinkOperator|ListSinkDesc|ListSink|FALSE|FALSE|TRUE|FALSE|TRUE| |FetchOperator|FetchWork|Fetch Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |MapOperator| | | | | | | | |OrcFileMergeOperator|OrcFileMergeDesc|ORC File Merge Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |RCFileMergeOperator|RCFileMergeDesc|RCFile Merge Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |SparkHashTableSinkOperator|SparkHashTableSinkDesc|Spark HashTable Sink Operator|TRUE|FALSE|FALSE|FALSE|FALSE| |TemporaryHashSinkOperator| |HashTable Sink Operator| | | | | | |TezDummyStoreOperator| |Dummy Store| | | | | | | | | | | | | | | |OperatorClass|OfType|ExplainString|hasOperator|hasTask|NeedsChange|ChangeToTask|AddTask| | | | | | | | | | |FunctionTask|FunctionWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE| | | | | | | | | | |DDLTask|DDLWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE| |FetchTask|FetchWork|Fetch Operator|FALSE|TRUE|TRUE|TRUE|FALSE| |ReplCopyTask|ReplCopyWork|Repl Copy|FALSE|FALSE|TRUE|FALSE|TRUE| |ConditionalTask|ConditionalWork|Conditional Operator|FALSE|TRUE|TRUE|TRUE|FALSE| |MaterializedViewTask|MaterializedViewDesc|Materialized View Work|FALSE|FALSE|TRUE|FALSE|TRUE| |ColumnStatsUpdateTask|ColumnStatsUpdateWork|Column Stats Update Work|FALSE|FALSE|TRUE|FALSE|TRUE| |ReplTxnTask|ReplTxnWork|Replication Transaction|FALSE|FALSE|TRUE|FALSE|TRUE| |DependencyCollectionTask|DependencyCollectionWork|Dependency Collection|FALSE|FALSE|TRUE|FALSE|TRUE| |ExplainTask|ExplainWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE| |ExportTask|ExportWork|Export Work|FALSE|FALSE|TRUE|FALSE|TRUE| |CopyTask|CopyWork|Copy|FALSE|FALSE|TRUE|FALSE|TRUE| |MoveTask|MoveWork|Move Operator|FALSE|TRUE|TRUE|TRUE|FALSE| |ExplainSQRewriteTask|ExplainSQRewriteWork|NO_EXPLAIN_STRING|FALSE|FALSE|TRUE|FALSE|TRUE| |StatsTask|StatsWork|Stats Work|FALSE|FALSE|TRUE|FALSE|TRUE| | | | | | | | > Explain consistency Task/Operator > - > > Key: HIVE-21713 > URL: https://issues.apache.org/jira/browse/HIVE-21713 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > > there seems to
[jira] [Commented] (HIVE-21713) Explain consistency Task/Operator
[ https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16848915#comment-16848915 ] Zoltan Haindrich commented on HIVE-21713: - [~mnarayanan2018]: yes; I was thinking about that line. There are 2 kind of things: * there are "tasks" which belong to the type tree of: Task * and there are "operators" which belong to Operator Calling "operator" the first one might be misleading in explain outputs > Explain consistency Task/Operator > - > > Key: HIVE-21713 > URL: https://issues.apache.org/jira/browse/HIVE-21713 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > > there seems to be some inconsistency what gets printed in the explain; > for example "MoveTask" is shown as "Move Operator" in the explain > Becase Tasks and Operators are basically different things - this should be > corrected as it may cause confusion -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21713) Explain consistency Task/Operator
[ https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16848691#comment-16848691 ] Manoj Narayanan commented on HIVE-21713: [~kgyrtkirk] I assume you are referring to changing the name given here, [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/plan/MoveWork.java#L35]. I assume it must be a straight forward change. But I had a few questions, it seems at least some of Explain classes are named as Operator. I did a code search and 66 classes had named as Operator out of total 600 odd ones. Only 1 of them had Task in the name. If that is the case, is it a convention to have Operator in the name ? So is it OK to continue to name it as 'Move Operator' ? Just wanted to confirm before proceeding with a patch. find . -name \*.java | grep -v Test | xargs grep @Explain | grep display | grep -i operator | wc -l 66 find . -name \*.java | grep -v Test | xargs grep @Explain | grep display | wc -l 641 find . -name \*.java | grep -v Test | xargs grep @Explain | grep display | grep -i task |wc -l 2 > Explain consistency Task/Operator > - > > Key: HIVE-21713 > URL: https://issues.apache.org/jira/browse/HIVE-21713 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Priority: Major > > there seems to be some inconsistency what gets printed in the explain; > for example "MoveTask" is shown as "Move Operator" in the explain > Becase Tasks and Operators are basically different things - this should be > corrected as it may cause confusion -- This message was sent by Atlassian JIRA (v7.6.3#76005)
