Hello Experts, I am using WebHCat REST interface to launch Pig jobs.
When I invoke Pig scripts using Curl, WebHCat launches a TempletonControllerJob which has one map task as expected. This job in turn is NOT launching the actual job from the REST API call. In the Resource Manager page, I only see the controller job (parent job), but PigLatin job (child job) could not be seen. However, the controller job is completed and the status getting succeeded, while looking inside this parent job, the actual Pig scripts are getting executed in the local instance. I am expecting the child jobs to be executed as a separate MR job in Hadoop cluster. Why is the controller job not launching a separate MR job for the pig scripts? Am I missing any configuration changes? Note : The pig scripts are executing fine in both Local and Mapreduce mode when I run it from command line. Templeton Controller Job - Map task logs HadoopVersion PigVersion UserId StartedAt FinishedAt Features 2.6.0 0.15.0 hduser 2016-04-16 20:56:54 2016-04-16 20:57:10 LIMIT Success! Job Stats (time in seconds): JobId Maps Reduces MaxMapTime MinMapTime AvgMapTime MedianMapTime MaxReduceTime MinReduceTime AvgReduceTime MedianReducetime Alias Feature Outputs job_local1288160920_0002 1 1 n/a n/a n/a n/a n/a n/a n/a n/a /tempo, job_local384530329_0001 1 1 n/a n/a n/a n/a n/a n/a n/a n/a a,b Input(s): Successfully read 1 records (10500218 bytes) from: "/pigouttest" Output(s): Successfully stored 1 records (20999942 bytes) in: "/tempo" Counters: Total records written : 1 Total bytes written : 20999942 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0 Job DAG: job_local384530329_0001 -> job_local1288160920_0002, job_local1288160920_0002 2016-04-16 20:57:10,759 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-04-16 20:57:10,766 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-04-16 20:57:10,768 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-04-16 20:57:10,827 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-04-16 20:57:10,829 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-04-16 20:57:10,830 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-04-16 20:57:10,839 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success! 2016-04-16 20:57:10,886 [main] INFO org.apache.pig.Main - Pig script completed in 22 seconds and 866 milliseconds (22866 ms) Thanks!