Fail to create temporary directory when execute bucket map join
---------------------------------------------------------------
Key: HIVE-2880
URL: https://issues.apache.org/jira/browse/HIVE-2880
Project: Hive
Issue Type: Bug
Components: SQL
Affects Versions: 0.8.1
Environment: Cluster with 4 PC: CPU Intel dual core 2.2Ghz, 2Gb ram,
80G hdd, cent OS 5.0, NIC 100Mbps
1 of them: Namenode + 2nd namenode + job tracker + hive server (also Datanode
and Task tracker)
3 of them: only datanode + task tracker
All of them could ssh password-less to each other and used the same user
account: hduser
Reporter: makiet
I have 2 tables
CREATE TABLE data(calling STRING COMMENT 'Calling number',
volumn_download BIGINT COMMENT 'Volume download',
volumn_upload BIGINT COMMENT 'Volume upload')
PARTITIONED BY(ds STRING)
CLUSTERED BY (calling) INTO 100 BUCKETS;
CREATE TABLE sub(isdn STRING, sub_id STRING)
CLUSTERED BY (isdn) INTO 100 BUCKETS;
The DATA table has 15m records while SUB table only has 600k records.
The following SQL script were executed successfully:
select /*+ MAPJOIN(b) */ a.calling, b.sub_id from ggsn_bucket a join sub_bucket
b on a.calling=b.isdn;
But when I used Bucket map join by setting: set hive.optimize.bucketmapjoin =
true
the above SQL script failed
select /*+ MAPJOIN(b) */ a.calling, b.sub_id from ggsn_bucket a join sub_bucket
b on a.calling=b.isdn;
hive> set hive.optimize.bucketmapjoin = true;
hive> select /*+ MAPJOIN(b) */ a.calling, b.sub_id from ggsn_bucket a join
sub_bucket b on a.calling=b.isdn;
Total MapReduce jobs = 1
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Execution log at:
/tmp/hduser/hduser_20120320080909_8e6a3419-4d2c-4148-a0c9-166d051c8274.log
2012-03-20 08:09:34 Starting to launch local task to process map join;
maximum memory = 932118528
2012-03-20 08:09:34 End of local task; Time Taken: 0.072 sec.
Execution completed successfully
Mapred Local Task Succeeded . Convert the Join into MapJoin
Mapred Local Task Succeeded . Convert the Join into MapJoin
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
org.apache.hadoop.util.Shell$ExitCodeException: bash: line 0: cd:
/u01/app/hduser/hadoop-0.20.203.0/tempdir/hduser/hive_2012-03-20_08-09-27_810_1393729636696443501/-local-10002/HashTable-Stage-1:
No such file or directory
tar: Cowardly refusing to create an empty archive
Try `tar --help' or `tar --usage' for more information.
at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:
375)
at org.apache.hadoop.hive.common.FileUtils.tar(FileUtils.java:260)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:407
)
at
org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136
)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.ja
va:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:2
55)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Job Submission failed with exception
'org.apache.hadoop.util.Shell$ExitCodeExcep tion(bash:
line 0: cd: /u01/app/hduser/hadoop-0.20.203.0/tempdir/hduser/hive_201
2-03-20_08-09-27_810_1393729636696443501/-local-10002/HashTable-Stage-1: No
such file or directory
tar: Cowardly refusing to create an empty archive
Try `tar --help' or `tar --usage' for more information.
)'
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at
org.apache.hadoop.hive.ql.exec.Utilities.getHiveJobID(Utilities.java:
379)
at
org.apache.hadoop.hive.ql.exec.Utilities.clearMapRedWork(Utilities.ja
va:192)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:476
)
at
org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136
)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.ja
va:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:2
55)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MapRedTask
in hadoop-env.sh, I set:
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true
-Djava.io.tmpdir=/u01/app/hduser/hadoop-0.20.203.0/tempdir"
It looked like hive could not create temporary directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
