Hello, I managed to create a properties file with the following contents :
fs.defaultFS=hdfs://hadoopnamenode:9000 mapreduce.framework.name=yarn yarn.resourcemanager.address=hadoopresourcemanager:8032 It is now submitting the jobs to the cluster. I also set the HADOOP_HOME on my laptop to point to the same version of hadoop that is running on the cluster (2.7.0). I am running pig version 0.17 Then a main class not found error happened on the yarn nodes where the job was scheduled to run. I had to add the following to yarn-site.xml and restart yarn and the nodes : <property> <name>mapreduce.application.classpath</name> <value>/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/*,/home/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/*,/home/hadoop/hadoop-2.7.0/share/hadoop/common/*,/home/hadoop/hadoop-2.7.0/share/hadoop/common/lib/*,/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/*,/home/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/*,/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/*,/home/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/*</value> </property> After this change, the script ran. But the pig command only returned after the job finished. Does anyone know how to launch the script and exit immediately to the shell ? If the job takes a long time I will have to keep the terminal open. Thanks, Regards > On 11 Feb 2020, at 05:25, Vinod Kumar Vavilapalli <vino...@apache.org> wrote: > > It’s running the job in local mode (LocalJobRunner), that’s why. Please check > your configuration files and make sure that the right directories are on the > classpath. Also look in mapred-site.xml for mapreduce.framework.name (should > be yarn). > > Thanks > +Vinod > >> On Feb 11, 2020, at 2:09 AM, Daniel Santos <daniel.d...@gmail.com> wrote: >> >> Hello all, >> >> I have developed a script in my laptop. The script is now ready to be >> unleashed in a non secured cluster. >> But when I do : pig -x mapreduce <script name> it doesn’t return to the >> shell immediately. It prints stuff like [LocalJobRunner Map Task Executor #0] >> >> I have exported the PIG_CLASSPATH shell variable to point to a directory >> with the cluster’s configuration and its operating on the files located >> there. >> But I would expect the job to be launched, the shell prompt returned and the >> job would be left executing independently on the cluster. >> >> Another thing I noticed while developing the script and running it both >> locally and on the cluster, is that the web page for there resource manager >> does not show the map reduce jobs that pig generates. What do I have to do >> to be able to see them ? >> >> Thanks, >> Regards >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: user-h...@hadoop.apache.org >> > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org