Hi Mozhdeh, As you can see from the pipeline.pl source [0], you are required to override the '-tmp' flag [1] on the pipeline.pl execution. I would highly encourage you to also scope out the entire parameter and flag options to geta better feel for the pipeline execution. HTH Lewis
[0] https://github.com/apache/incubator-joshua/blob/master/scripts/training/pipeline.pl#L228-L229 [1] https://github.com/apache/incubator-joshua/blob/master/scripts/training/pipeline.pl#L305 On Sat, Feb 3, 2018 at 10:53 AM, < user-digest-h...@joshua.incubator.apache.org> wrote: > > From: Mozhdeh Gheini <ghe...@isi.edu> > To: "user@joshua.incubator.apache.org" <user@joshua.incubator.apache.org> > Cc: > Bcc: > Date: Sat, 3 Feb 2018 18:52:55 +0000 > Subject: Hadoop tmp Directory Issue > > Hi, > > > I'm working with joshua on a grid computing system. The problem I face is > that although I modify hadoop.tmp.dir and set it to the temporary directory > assigned to me, Hadoop still ends up in /tmp. This causes serious problems > because the next user that wants to run joshua and is assigned to the same > machine that I was on, is not the owner of /tmp/hadoop which was created > during my last run and cannot proceed. In a nutshell, Hadoop doesn't end up > where it's supposed to. > > > This is the command that gets run: > > > hadoop jar /nas/home/gheini/libs/joshua/thrax/bin/thrax.jar -D > hadoop.tmp.dir=/lfs_ssd/uge-tmpdir/138005.1.all.q -D > mapreduce.task.timeout=0 -D mapreduce.child.java.opts='- > Djava.io.tmpdir=/lfs_ssd/uge-tmpdir/138005.1.all.q' -D > mapreduce.map.java.opts='-Xmx4g' -D mapreduce.reduce.java.opts='-Xmx4g' > thrax-phrase.conf > pipeline-es-en-phrase-_nas_home_gheini_experiments_joshua_tmp_setting_test_2 > > thrax.log 2>&1; rm -f grammar grammar.gz; hadoop fs -cat > pipeline-es-en-phrase-_nas_home_gheini_experiments_ > joshua_tmp_setting_test_2/final/* | gzip -cd | > /nas/home/gheini/libs/joshua/scripts/training/filter-rules.pl -t 100 | > gzip -9n > grammar.gz > > > I'm using Hadoop 3.0.0. I think something is happening within Thrax but > cannot figure it out. I'd appreciate it if anyone could help. > > > Thanks, > > Mozhdeh > > > -- http://home.apache.org/~lewismc/ http://people.apache.org/keys/committer/lewismc
