>From log file: 15/07/16 11:16:56 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-root/mapred/local/1437016615898/user_agents <- /opt/HiBench-master/user_agents 15/07/16 11:16:56 INFO mapred.LocalDistributedCacheManager: Localized hdfs://spark-study:9000/HiBench/Aggregation/temp/user_agents as file:/tmp/hadoop-root/mapred/local/ 1437016615898/user_agents ... java.io.FileNotFoundException: file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or directory) at java.io.FileInputStream.open(Native Method)
However, FileNotFoundException didn't happen to other localized files, such as country_codes FYI On Wed, Jul 15, 2015 at 8:53 PM, <luohui20...@sina.com> wrote: > Hi all > > when I am running my HiBench in my spark/Hadoop/Hive cluster. I > found there is always a failure in my aggregation test. I doubt this > problem maybe some issue relative with my hive settings? attaches are my > config file and log file . > > Any idea to solve this issue? > > my Spark cluster is a one node standalone cluster : > > hadoop:2.7 > > spark:1.3 > > hive:1.2.1 > > here is the log: > > Prepare aggregation ... > Exec script: /opt/HiBench-master/workloads/aggregation/prepare/prepare.sh > Parsing conf: /opt/HiBench-master/conf/00-default-properties.conf > Parsing conf: /opt/HiBench-master/conf/10-data-scale-profile.conf > Parsing conf: /opt/HiBench-master/conf/99-user_defined_properties.conf > Parsing conf: > /opt/HiBench-master/workloads/aggregation/conf/00-aggregation-default.conf > Parsing conf: > /opt/HiBench-master/workloads/aggregation/conf/10-aggregation-userdefine.conf > start HadoopPrepareAggregation bench > hdfs rm -r: /usr/lib/hadoop/bin/hadoop --config /usr/lib/hadoop/etc/hadoop > fs -rm -r -skipTrash hdfs://spark-study:9000/HiBench/Aggregation/Input > 15/07/16 11:16:26 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Deleted hdfs://spark-study:9000/HiBench/Aggregation/Input > Pages:120000, USERVISITS:1000000 > Submit MapReduce Job: /usr/lib/hadoop/bin/hadoop --config > /usr/lib/hadoop/etc/hadoop jar > /opt/HiBench-master/src/autogen/target/autogen-4.0-SNAPSHOT-jar-with-dependencies.jar > HiBench.DataGen -t hive -b hdfs://spark-study:9000/HiBench/Aggregation -n > Input -m 12 -r 6 -p 120000 -v 1000000 -o sequence > 15/07/16 11:16:34 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > sfubmzperrrbupqoq > 15/07/16 11:16:38 WARN mapreduce.JobResourceUploader: Hadoop command-line > option parsing not performed. Implement the Tool interface and execute your > application with ToolRunner to remedy this. > 15/07/16 11:16:40 INFO HiBench.HtmlCore: WARNING: dict empty!!! > 15/07/16 11:16:42 INFO HiBench.HtmlCore: WARNING: dict empty!!! > 15/07/16 11:16:43 INFO HiBench.HtmlCore: WARNING: dict empty!!! > 15/07/16 11:16:43 INFO HiBench.HtmlCore: WARNING: dict empty!!! > 15/07/16 11:16:44 INFO HiBench.HtmlCore: WARNING: dict empty!!! > 15/07/16 11:16:45 INFO HiBench.HtmlCore: WARNING: dict empty!!! > 15/07/16 11:16:45 INFO HiBench.HtmlCore: WARNING: dict empty!!! > 15/07/16 11:16:46 INFO HiBench.HtmlCore: WARNING: dict empty!!! > 15/07/16 11:16:46 INFO HiBench.HtmlCore: WARNING: dict empty!!! > 15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!! > 15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!! > 15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!! > 15/07/16 11:16:48 INFO reduce.EventFetcher: EventFetcher is interrupted.. > Returning > 15/07/16 11:16:50 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 0, 0 > erros, 0 missed > 15/07/16 11:16:50 INFO reduce.EventFetcher: EventFetcher is interrupted.. > Returning > 15/07/16 11:16:51 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 1, 0 > erros, 0 missed > 15/07/16 11:16:51 INFO reduce.EventFetcher: EventFetcher is interrupted.. > Returning > 15/07/16 11:16:51 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 2, 0 > erros, 0 missed > 15/07/16 11:16:52 INFO reduce.EventFetcher: EventFetcher is interrupted.. > Returning > 15/07/16 11:16:52 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 3, 0 > erros, 0 missed > 15/07/16 11:16:52 INFO reduce.EventFetcher: EventFetcher is interrupted.. > Returning > 15/07/16 11:16:53 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 4, 0 > erros, 0 missed > 15/07/16 11:16:53 INFO reduce.EventFetcher: EventFetcher is interrupted.. > Returning > 15/07/16 11:16:54 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 5, 0 > erros, 0 missed > 15/07/16 11:16:55 WARN mapreduce.JobResourceUploader: Hadoop command-line > option parsing not performed. Implement the Tool interface and execute your > application with ToolRunner to remedy this. > java.io.FileNotFoundException: > file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file > or directory) > java.io.FileNotFoundException: > file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file > or directory) > java.io.FileNotFoundException: > file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file > or directory) > java.io.FileNotFoundException: > file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file > or directory) > 15/07/16 11:17:01 WARN mapred.LocalJobRunner: job_local563760878_0002 > java.lang.Exception: java.lang.NullPointerException > Caused by: java.lang.NullPointerException > Exception in thread "main" java.io.IOException: Job failed! > 15/07/16 11:17:02 INFO mapred.LocalJobRunner: > hdfs://spark-study:9000/HiBench/Aggregation/temp/dummy:7+2 > map > ERROR: Hadoop job > /opt/HiBench-master/src/autogen/target/autogen-4.0-SNAPSHOT-jar-with-dependencies.jar > HiBench.DataGen failed to run successfully. > Hint: You can goto > /opt/HiBench-master/report/aggregation/prepare/conf/../bench.log to check > for detailed log. > Opening log tail for you: > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at HiBench.DataGen.main(DataGen.java:45) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > 15/07/16 11:17:02 INFO mapred.LocalJobRunner: > hdfs://spark-study:9000/HiBench/Aggregation/temp/dummy:7+2 > map > > ERROR: aggregation prepare failed! > > -------------------------------- > > Thanks&Best regards! > San.Luo > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >