I am a complete newbie to hadoop. I'm running a job on 19 Amazon Elastic MapReduce servers and am trying to understand two separate issues.
1) The job is ending with an error "ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 6015: During execution, encountered a Hadoop error." I do not have the hadoop log files and will have to rerun the job with different settings to obtain them. Once I have them, I'll add them to the posting. 2) The job seems to complete %19 within 1 minute and then takes 15 minutes to complete another %20. It then takes over 40 minutes to complete the last %60. Why the sudden slow down? Am I misunderstanding the messages? The following is the ouput from the job: 10/08/05 16:39:30 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively 2010-08-05 16:39:30,688 [main] INFO org.apache.pig.Main - Logging error messages to: /mnt/var/lib/hadoop/steps/2/pig_1281026370670.log 2010-08-05 16:39:31,663 [main] INFO org.apache.hadoop.fs.s3native.NativeS3FileSystem - Opening '/scripts/calculateTrackingErrors.pig' for reading 2010-08-05 16:39:32,071 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://domU-12-31-39-09-F1-D2.compute-1.internal:9000 2010-08-05 16:39:32,662 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: domU-12-31-39-09-F1-D2.compute-1.internal:9001 2010-08-05 16:39:33,360 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned for covMatrix 2010-08-05 16:39:33,360 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for covMatrix 2010-08-05 16:39:33,362 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned for fundExposures2 2010-08-05 16:39:33,362 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for fundExposures2 2010-08-05 16:39:33,363 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned for fundExposures1 2010-08-05 16:39:33,363 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for fundExposures1 2010-08-05 16:39:33,469 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 1 time(s). 2010-08-05 16:39:33,484 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage->POForEach to POJoinPackage 2010-08-05 16:39:33,485 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage->POForEach to POJoinPackage 2010-08-05 16:39:33,486 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage->POForEach to POJoinPackage 2010-08-05 16:39:33,495 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 5 2010-08-05 16:39:33,496 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - Merged 0 out of total 3 MR operators. 2010-08-05 16:39:33,496 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 5 2010-08-05 16:39:33,501 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode. 2010-08-05 16:39:33,502 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode. 2010-08-05 16:39:34,465 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2010-08-05 16:39:34,559 [Thread-12] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2010-08-05 16:39:35,585 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Cannot get jobid for this job 2010-08-05 16:39:42,847 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2010-08-05 16:39:55,415 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 5% complete 2010-08-05 16:39:56,464 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 10% complete 2010-08-05 16:40:08,548 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 11% complete 2010-08-05 16:40:11,561 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 14% complete 2010-08-05 16:40:26,650 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 19% complete 2010-08-05 16:40:48,382 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2010-08-05 16:40:48,415 [Thread-23] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2010-08-05 16:40:49,444 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Cannot get jobid for this job 2010-08-05 16:55:41,640 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 39% complete 2010-08-05 16:57:17,243 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2010-08-05 16:57:17,275 [Thread-33] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2010-08-05 16:57:18,274 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Cannot get jobid for this job 2010-08-05 17:32:47,304 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 54% complete 2010-08-05 17:35:02,343 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 56% complete 2010-08-05 17:38:33,767 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 60% complete 2010-08-05 17:38:37,285 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete 2010-08-05 17:38:37,286 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map reduce job(s) failed! 2010-08-05 17:38:37,772 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Some jobs have failed! Stop running all dependent jobs 2010-08-05 17:38:37,775 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 6015: During execution, encountered a Hadoop error. Details at logfile: /mnt/var/lib/hadoop/steps/2/pig_1281026370670.log Command exiting with ret '255' Thanks for any enlightenment you can provide -- View this message in context: http://old.nabble.com/Problems-running-hadoop-on-Amazon-Elastic-MapReduce-tp29356384p29356384.html Sent from the Hadoop core-user mailing list archive at Nabble.com.