Hi Ted Thanks for your advice, i found that there is something wrong with
"hadoop fs -get" command, 'cause I believe the localization of
hdfs://spark-study:9000/HiBench/Aggregation/temp/user_agents to
/tmp/hadoop-root/mapred/local/1437016615898/user_agents is a behaviour like
"hadoop fs -get" I am forwarding to solve below issue
now,:[root@spark-study HiBench-master]# hadoop fs -get
/HiBench/Aggregation/temp/user_agents
15/07/16 12:31:55 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/07/16 12:31:58 WARN hdfs.DFSClient: DFSInputStream has been closed already
Hi Mike: I am new to Hibench, so I just setup a test enviroment of 1 node
spark/hadoop cluster to test, no data actually. Because Hibench will
autogenerate test data itself.
--------------------------------
Thanks&Best regards!
San.Luo
----- 原始邮件 -----
发件人：Ted Yu <yuzhih...@gmail.com>
收件人：罗辉 <luohui20...@sina.com>
抄送人："jie.huang" <jie.hu...@intel.com>, user <user@spark.apache.org>
主题：Re: HiBench test for hadoop/hive/spark cluster
日期：2015年07月16日 12点17分
From log file:
15/07/16 11:16:56 INFO mapred.LocalDistributedCacheManager: Creating symlink:
/tmp/hadoop-root/mapred/local/1437016615898/user_agents <-
/opt/HiBench-master/user_agents15/07/16 11:16:56 INFO
mapred.LocalDistributedCacheManager: Localized
hdfs://spark-study:9000/HiBench/Aggregation/temp/user_agents as
file:/tmp/hadoop-root/mapred/local/
1437016615898/user_agents...java.io.FileNotFoundException:
file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or
directory) at java.io.FileInputStream.open(Native Method)
However, FileNotFoundException didn't happen to other localized files, such as
country_codes
FYI
On Wed, Jul 15, 2015 at 8:53 PM, <luohui20...@sina.com> wrote:
Hi all
when I am running my HiBench in my spark/Hadoop/Hive cluster. I found
there is always a failure in my aggregation test. I doubt this problem maybe
some issue relative with my hive settings? attaches are my config file and log
file . Any idea to solve this issue?
my Spark cluster is a one node standalone cluster
:hadoop:2.7spark:1.3hive:1.2.1
here is the log:Prepare aggregation ...
Exec script: /opt/HiBench-master/workloads/aggregation/prepare/prepare.sh
Parsing conf: /opt/HiBench-master/conf/00-default-properties.conf
Parsing conf: /opt/HiBench-master/conf/10-data-scale-profile.conf
Parsing conf: /opt/HiBench-master/conf/99-user_defined_properties.conf
Parsing conf:
/opt/HiBench-master/workloads/aggregation/conf/00-aggregation-default.conf
Parsing conf:
/opt/HiBench-master/workloads/aggregation/conf/10-aggregation-userdefine.conf
start HadoopPrepareAggregation bench
hdfs rm -r: /usr/lib/hadoop/bin/hadoop --config /usr/lib/hadoop/etc/hadoop fs
-rm -r -skipTrash hdfs://spark-study:9000/HiBench/Aggregation/Input
15/07/16 11:16:26 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Deleted hdfs://spark-study:9000/HiBench/Aggregation/Input
Pages:120000, USERVISITS:1000000
Submit MapReduce Job: /usr/lib/hadoop/bin/hadoop --config
/usr/lib/hadoop/etc/hadoop jar
/opt/HiBench-master/src/autogen/target/autogen-4.0-SNAPSHOT-jar-with-dependencies.jar
HiBench.DataGen -t hive -b hdfs://spark-study:9000/HiBench/Aggregation -n
Input -m 12 -r 6 -p 120000 -v 1000000 -o sequence
15/07/16 11:16:34 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
sfubmzperrrbupqoq
15/07/16 11:16:38 WARN mapreduce.JobResourceUploader: Hadoop command-line
option parsing not performed. Implement the Tool interface and execute your
application with ToolRunner to remedy this.
15/07/16 11:16:40 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:42 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:43 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:43 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:44 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:45 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:45 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:46 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:46 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:48 INFO reduce.EventFetcher: EventFetcher is interrupted..
Returning
15/07/16 11:16:50 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 0, 0
erros, 0 missed
15/07/16 11:16:50 INFO reduce.EventFetcher: EventFetcher is interrupted..
Returning
15/07/16 11:16:51 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 1, 0
erros, 0 missed
15/07/16 11:16:51 INFO reduce.EventFetcher: EventFetcher is interrupted..
Returning
15/07/16 11:16:51 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 2, 0
erros, 0 missed
15/07/16 11:16:52 INFO reduce.EventFetcher: EventFetcher is interrupted..
Returning
15/07/16 11:16:52 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 3, 0
erros, 0 missed
15/07/16 11:16:52 INFO reduce.EventFetcher: EventFetcher is interrupted..
Returning
15/07/16 11:16:53 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 4, 0
erros, 0 missed
15/07/16 11:16:53 INFO reduce.EventFetcher: EventFetcher is interrupted..
Returning
15/07/16 11:16:54 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 5, 0
erros, 0 missed
15/07/16 11:16:55 WARN mapreduce.JobResourceUploader: Hadoop command-line
option parsing not performed. Implement the Tool interface and execute your
application with ToolRunner to remedy this.
java.io.FileNotFoundException:
file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or
directory)
java.io.FileNotFoundException:
file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or
directory)
java.io.FileNotFoundException:
file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or
directory)
java.io.FileNotFoundException:
file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or
directory)
15/07/16 11:17:01 WARN mapred.LocalJobRunner: job_local563760878_0002
java.lang.Exception: java.lang.NullPointerException
Caused by: java.lang.NullPointerException
Exception in thread "main" java.io.IOException: Job failed!
15/07/16 11:17:02 INFO mapred.LocalJobRunner:
hdfs://spark-study:9000/HiBench/Aggregation/temp/dummy:7+2 > map
ERROR: Hadoop job
/opt/HiBench-master/src/autogen/target/autogen-4.0-SNAPSHOT-jar-with-dependencies.jar
HiBench.DataGen failed to run successfully.
Hint: You can goto
/opt/HiBench-master/report/aggregation/prepare/conf/../bench.log to check for
detailed log.
Opening log tail for you:
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at HiBench.DataGen.main(DataGen.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
15/07/16 11:17:02 INFO mapred.LocalJobRunner:
hdfs://spark-study:9000/HiBench/Aggregation/temp/dummy:7+2 > map
ERROR: aggregation prepare failed!
--------------------------------
Thanks&Best regards!
San.Luo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
