[
https://issues.apache.org/jira/browse/HIVE-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13791563#comment-13791563
]
jeff little commented on HIVE-5245:
-----------------------------------
hive (test)> set hive.auto.convert.join.noconditionaltask=true;
hive (test)> explain
> create table test_10 as
> select a.* from test_01 a
> join test_02 b
> on (a.id=b.id);
OK
ABSTRACT SYNTAX TREE:
(TOK_CREATETABLE (TOK_TABNAME test_10) TOK_LIKETABLE (TOK_QUERY (TOK_FROM
(TOK_JOIN (TOK_TABREF (TOK_TABNAME test_01) a) (TOK_TABREF (TOK_TABNAME
test_02) b) (= (. (TOK_TABLE_OR_COL a) id) (. (TOK_TABLE_OR_COL b) id))))
(TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR
(TOK_ALLCOLREF (TOK_TABNAME a)))))))
STAGE DEPENDENCIES:
Stage-8 is a root stage
Stage-6 depends on stages: Stage-8
Stage-5 depends on stages: Stage-6 , consists of Stage-7
Stage-7
Stage-4 depends on stages: Stage-7
Stage-0 depends on stages: Stage-1, Stage-4
Stage-9 depends on stages: Stage-0
Stage-2 depends on stages: Stage-9
STAGE PLANS:
Stage: Stage-8
Map Reduce Local Work
Alias -> Map Local Tables:
a
Fetch Operator
limit: -1
Alias -> Map Local Operator Tree:
a
TableScan
alias: a
HashTable Sink Operator
condition expressions:
0 {id} {name} {sex} {record_day}
1
handleSkewJoin: false
keys:
0 [Column[id]]
1 [Column[id]]
Position of Big Table: 1
Stage: Stage-6
Map Reduce
Alias -> Map Operator Tree:
b
TableScan
alias: b
Map Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {id} {name} {sex} {record_day}
1
handleSkewJoin: false
keys:
0 [Column[id]]
1 [Column[id]]
outputColumnNames: _col0, _col1, _col2, _col3
Position of Big Table: 1
Select Operator
expressions:
expr: _col0
type: int
expr: _col1
type: string
expr: _col2
type: string
expr: _col3
type: string
outputColumnNames: _col0, _col1, _col2, _col3
File Output Operator
compressed: false
GlobalTableId: 1
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
name: test.test_10
Local Work:
Map Reduce Local Work
Stage: Stage-5
Conditional Operator
Stage: Stage-7
Map Reduce Local Work
Alias -> Map Local Tables:
1
Fetch Operator
limit: -1
Alias -> Map Local Operator Tree:
1
HashTable Sink Operator
condition expressions:
0 {0_VALUE_0} {0_VALUE_1} {0_VALUE_2} {0_VALUE_3}
1
handleSkewJoin: false
keys:
0 [Column[joinkey0]]
1 [Column[joinkey0]]
Position of Big Table: 0
Stage: Stage-4
Map Reduce
Alias -> Map Operator Tree:
0
Map Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {0_VALUE_0} {0_VALUE_1} {0_VALUE_2} {0_VALUE_3}
1
handleSkewJoin: false
keys:
0 [Column[joinkey0]]
1 [Column[joinkey0]]
outputColumnNames: _col0, _col1, _col2, _col3
Position of Big Table: 0
Select Operator
expressions:
expr: _col0
type: int
expr: _col1
type: string
expr: _col2
type: string
expr: _col3
type: string
outputColumnNames: _col0, _col1, _col2, _col3
File Output Operator
compressed: false
GlobalTableId: 1
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
name: test.test_10
Local Work:
Map Reduce Local Work
Stage: Stage-0
Move Operator
files:
hdfs directory: true
destination: hdfs://namenode:9000/user/hive/warehouse/test.db/test_10
Stage: Stage-9
Create Table Operator:
Create Table
columns: id int, name string, sex string, record_day string
if not exists: false
input format: org.apache.hadoop.mapred.TextInputFormat
# buckets: -1
output format: org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
name: test_10
isExternal: false
Stage: Stage-2
Stats-Aggr Operator
Time taken: 0.293 seconds, Fetched: 152 row(s)
hive (test)>set hive.auto.convert.join.noconditionaltask=false;
hive (test)> explain
> create table test_10 as
> select a.* from test_01 a
> join test_02 b
> on (a.id=b.id);
ABSTRACT SYNTAX TREE:
(TOK_CREATETABLE (TOK_TABNAME test_10) TOK_LIKETABLE (TOK_QUERY (TOK_FROM
(TOK_JOIN (TOK_TABREF (TOK_TABNAME test_01) a) (TOK_TABREF (TOK_TABNAME
test_02) b) (= (. (TOK_TABL
E_OR_COL a) id) (. (TOK_TABLE_OR_COL b) id)))) (TOK_INSERT (TOK_DESTINATION
(TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_ALLCOLREF (TOK_TABNAME
a)))))))
STAGE DEPENDENCIES:
Stage-8 is a root stage
Stage-6 depends on stages: Stage-8
Stage-5 depends on stages: Stage-6 , consists of Stage-7
Stage-7
Stage-4 depends on stages: Stage-7
Stage-0 depends on stages: Stage-1, Stage-4
Stage-9 depends on stages: Stage-0
Stage-2 depends on stages: Stage-9
STAGE PLANS:
Stage: Stage-8
Map Reduce Local Work
Alias -> Map Local Tables:
a
Fetch Operator
limit: -1
Alias -> Map Local Operator Tree:
a
TableScan
alias: a
HashTable Sink Operator
condition expressions:
0 {id} {name} {sex} {record_day}
1
handleSkewJoin: false
keys:
0 [Column[id]]
1 [Column[id]]
Position of Big Table: 1
Stage: Stage-6
Map Reduce
Alias -> Map Operator Tree:
b
TableScan
alias: b
Map Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {id} {name} {sex} {record_day}
1
handleSkewJoin: false
keys:
0 [Column[id]]
1 [Column[id]]
outputColumnNames: _col0, _col1, _col2, _col3
Position of Big Table: 1
Select Operator
expressions:
expr: _col0
type: int
expr: _col1
type: string
expr: _col2
type: string
expr: _col3
type: string
outputColumnNames: _col0, _col1, _col2, _col3
File Output Operator
compressed: false
GlobalTableId: 1
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
name: test.test_10
Local Work:
Map Reduce Local Work
Stage: Stage-5
Conditional Operator
Stage: Stage-7
Map Reduce Local Work
Alias -> Map Local Tables:
1
Fetch Operator
limit: -1
Alias -> Map Local Operator Tree:
1
HashTable Sink Operator
condition expressions:
0 {0_VALUE_0} {0_VALUE_1} {0_VALUE_2} {0_VALUE_3}
1
handleSkewJoin: false
keys:
0 [Column[joinkey0]]
1 [Column[joinkey0]]
Position of Big Table: 0
Stage: Stage-4
Map Reduce
Alias -> Map Operator Tree:
0
Map Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {0_VALUE_0} {0_VALUE_1} {0_VALUE_2} {0_VALUE_3}
1
handleSkewJoin: false
keys:
0 [Column[joinkey0]]
1 [Column[joinkey0]]
outputColumnNames: _col0, _col1, _col2, _col3
Position of Big Table: 0
Select Operator
expressions:
expr: _col0
type: int
expr: _col1
type: string
expr: _col2
type: string
expr: _col3
type: string
outputColumnNames: _col0, _col1, _col2, _col3
File Output Operator
compressed: false
GlobalTableId: 1
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
name: test.test_10
Local Work:
Map Reduce Local Work
Stage: Stage-0
Move Operator
files:
hdfs directory: true
destination: hdfs://namenode:9000/user/hive/warehouse/test.db/test_10
Stage: Stage-9
Create Table Operator:
Create Table
columns: id int, name string, sex string, record_day string
if not exists: false
input format: org.apache.hadoop.mapred.TextInputFormat
# buckets: -1
output format: org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
name: test_10
isExternal: false
Stage: Stage-2
Stats-Aggr Operator
Hi, Yin Huai. I had tried to set hive.auto.convert.join.noconditionaltask=true
or false, but it was still noneffective. The explain content above. Plz help
me! Thanks!
> hive create table as select(CTAS) can not work(not support) with join on
> operator
> ---------------------------------------------------------------------------------
>
> Key: HIVE-5245
> URL: https://issues.apache.org/jira/browse/HIVE-5245
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 0.11.0
> Reporter: jeff little
> Labels: CTAS, hive
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> hello everyone, recently i came across one hive problem as below:
> hive (test)> create table test_09 as
> > select a.* from test_01 a
> > join test_02 b
> > on (a.id=b.id);
> Automatically selecting local only mode for query
> Total MapReduce jobs = 2
> setting HADOOP_USER_NAME hadoop
> 13/09/09 17:22:36 WARN conf.Configuration:
> file:/tmp/hadoop/hive_2013-09-09_17-22-34_848_1629553341892012305/-local-10008/jobconf.xml:a
> attempt to override final parameter: mapred.system.dir; Ignoring.
> 13/09/09 17:22:36 WARN conf.Configuration:
> file:/tmp/hadoop/hive_2013-09-09_17-22-34_848_1629553341892012305/-local-10008/jobconf.xml:a
> attempt to override final parameter: mapred.local.dir; Ignoring.
> Execution log at: /tmp/hadoop/.log
> 2013-09-09 05:22:36 Starting to launch local task to process map join;
> maximum memory = 932118528
> 2013-09-09 05:22:37 Processing rows: 4 Hashtable size: 4
> Memory usage: 113068056 rate: 0.121
> 2013-09-09 05:22:37 Dump the hashtable into file:
> file:/tmp/hadoop/hive_2013-09-09_17-22-34_848_1629553341892012305/-local-10005/HashTable-Stage-6/MapJoin-mapfile90--.hashtable
> 2013-09-09 05:22:37 Upload 1 File to:
> file:/tmp/hadoop/hive_2013-09-09_17-22-34_848_1629553341892012305/-local-10005/HashTable-Stage-6/MapJoin-mapfile90--.hashtable
> File size: 788
> 2013-09-09 05:22:37 End of local task; Time Taken: 0.444 sec.
> Execution completed successfully
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Launching Job 1 out of 2
> Number of reduce tasks is set to 0 since there's no reduce operator
> 13/09/09 17:22:38 WARN conf.Configuration:
> file:/tmp/hadoop/hive_2013-09-09_17-22-34_848_1629553341892012305/-local-10009/jobconf.xml:a
> attempt to override final parameter: mapred.system.dir; Ignoring.
> 13/09/09 17:22:38 WARN conf.Configuration:
> file:/tmp/hadoop/hive_2013-09-09_17-22-34_848_1629553341892012305/-local-10009/jobconf.xml:a
> attempt to override final parameter: mapred.local.dir; Ignoring.
> Execution log at: /tmp/hadoop/.log
> Job running in-process (local Hadoop)
> Hadoop job information for null: number of mappers: 0; number of reducers: 0
> 2013-09-09 17:22:41,807 null map = 0%, reduce = 0%
> 2013-09-09 17:22:44,814 null map = 100%, reduce = 0%
> Ended Job = job_local_0001
> Execution completed successfully
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Stage-7 is filtered out by condition resolver.
> OK
> Time taken: 13.138 seconds
> hive (test)> select * from test_09;
> FAILED: SemanticException [Error 10001]: Line 1:14 Table not found 'test_09'
> hive (test)>
> Problem:
> I can't get the created table, namely this CTAS is nonavailable, and this
> table is not created by this hql sentence at all.who can explain for
> me.Thanks.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
