I found the following scripts in $SCRIPTS_ROOTDIR with "use FindBin qw($Bin);": $SCRIPTS_ROOTDIR/training/wrappers/parse-de-berkeley.perl $SCRIPTS_ROOTDIR/training/wrappers/parse-de-bitpar.perl $SCRIPTS_ROOTDIR/training/train-model.perl $SCRIPTS_ROOTDIR/training/filter-model-given-input.pl $SCRIPTS_ROOTDIR/training/mert-moses.pl $SCRIPTS_ROOTDIR/training/mert-moses-multi.pl $SCRIPTS_ROOTDIR/training/zmert-moses.pl $SCRIPTS_ROOTDIR/analysis/weight-scan.pl $SCRIPTS_ROOTDIR/ems/support/split-sentences.perl $SCRIPTS_ROOTDIR/ems/experiment.perl $SCRIPTS_ROOTDIR/generic/trainlm-irst.perl $SCRIPTS_ROOTDIR/tokenizer/tokenizer.perl
I tested the following with "use FindBin qw($RealBin);" and associated updates: tokenizer.perl mert-moses.pl train-model.perl When a user calls the scripts directly in a terminal/console, $Bin and $RealBin versions function identically. If a user calls the $RealBin version via a symlink, the script resolves the scripts' real path, finds the relative paths to dependencies and runs fine. The $Bin version resolves to the symlink's path, can't find the relative paths to dependencies and fails. I'd like to propose that with the planned changes to eliminate references to $SCRIPTS_ROOTDIR, the Moses Code Guide and/or the Style Guide include an update for scripts to reference their $RealBin vice the $Bin to support the use of symlinks for all users. On Fri, 22 Jun 2012 08:39:56 +0700, Tom Hoar <tah...@precisiontranslationtools.com> wrote: > Ok, I see now. train-model.perl uses: > > use FindBin qw($Bin); > my $SCRIPTS_ROOTDIR = $Bin; > > We use symlinks to flatten the scripts in $SCRIPTS_ROOTDIR into the > $prefix/bin folder. By default $prefix resolves to /usr/local. > Therefore, $prefix/bin is in $PATH. Our approach confuses the > relative > path references in the script that rely on $Bin (without a separate > $SCRIPTS_ROOTDIR). In this case, the $PHRASE_EXTRACT concatenation on > line 1439 (step 5) caused my system to break because it resolved to > $PHRASE_EXTRACT = "$Bin/generic/extract-parallel.perl, which wasn't > there. > > In last night's troubleshooting, I re-referenced $SCRIPTS_ROOTDIR, > and placed symlinks to all $SCRIPTS_ROOTDIR in $prefix/bin. It was > the > latter, not the former, that enabled the script run. My bad in my > email. There are similar challenges with other scripts, such as > tokenizer.perl and detokenizer.perl which reference subfolders > relative to their location. > > Also, thanks for sharing the goal. Effectively, you'll have $prefixA > and $prefixB. Our goals are a little different. We're trying to > install moses & all components into a hierarchy complaint to the > Linux > Foundation's Filesystem Hierarchy Standard (FHA), in such a way it's > usable across most other Posix systems. Here's what we've come up > with: > > $IRSTLM, $RANDLM & $SRILM all point to $prefix. This way, their > resources, as well as the moses and MGIZA++ "make install" paths, > share the same $prefix/bin, $prefix/lib, $prefix/include (etc) > subfolders. We had to play some tricks with $SRILM to support > $prefix/sbin and the $MACHINE_TYPE references, but that support has > been there a long time and works well. Amazingly, we've been lucky > that there are no filename conflicts except for mkcls in GIZA++ and > MGIZA++. > > We place all component's original scripts under > $prefix/lib/<component>, as laid out by the component's authors. For > example, we move MGIZA++ $prefix/scripts to $prefix/lib/mgizapp and > configure moses' $SCRIPTS_ROOTDIR becomes $prefix/lib/mosesdecoder, > etc. We use of symlinks in $prefix/bin to reference various scripts > in > $PATH (as above). > > According to http://perldoc.perl.org/FindBin.html, it looks like > changing to the script's "real" location vice command line reference > is possible with: > > use FindBin qw($RealBin); > > This change will eliminate our need to symlink subfolders in > $prefix/bin, and still allow other Moses users to move $prefixA tree > anywhere they like. However, it might require a bit more editing in > each script to verify/resolve any relative references. > > For now, I'll continue using folder symlinks. I'll give you access to > a preview copy of our new binary install program via FTP with some > instructions how to make $prefix a private user folder instead of a > system level. Our approaches might give you some ideas about how > moses > can support "linguistic programs over time". For example, in addition > to the "standard" Moses components (giza-pp, mgizapp, irstlm, randlm, > srilm), we currently install DoMY CE (corpus preparation and > translation workflow), BerkeleyAligner (phrase alignment), > Champollion > Toolkit (sentence aligner), Stanford Aligner (Chinese/Arabic word > seg), MeCab (Japanese word seg), SWATH (open source Thai word seg), > and Langmatch (language ID) add-ons with this approach. We've also > mapped out support for m4loc, Okapi Framework, and the moses team's > sentence aligner. > > Regards, > Tom > > > On Thu, 21 Jun 2012 23:49:35 +0100, Hieu Hoang > <fishandfrol...@gmail.com> wrote: >> On 21/06/2012 16:46, Tom Hoar wrote: >>> Hieu, >>> >>> We're implementing these changes into DoMY. Some of these broke our >>> layout, but that's okay. We're adapt to your changes. >> thanks, much appreciated. >>> Updating -external-bin-dir was easy. Then, we scrapped our >>> references >>> to $SCRIPTS_ROOTDIR based on your comments in train-model.perl. >>> This, >>> however, caused step 5 to break. On closer inspection, a reference >>> to >>> $SCRIPTS_ROOTDIR is still necessary at this point. >> that's odd. The variable $SCRIPTS_ROOTDIR is still there but it's >> set in >> line 16-20. I didn't change these lines, I just removed the ability >> to >> override it with some other value from the command line. >> >> are you sure step 5 breaks? >>> >>> How do you see the layout evolving without the $SCRIPTS_ROOTDIR >>> value? >>> Since all of the scripts are in subfolders from $SCRIPTS_ROOTDIR, >>> do >>> you think it's possible or feasible to set $SCRIPTS_ROOTDIR == >>> $_EXTERNAL_BINDIR? That's possible today by manually configuring >>> bjam >>> for a build. However, if you have another layout in mind, would >>> this >>> cause conflicts? >> I'm aiming for everyone to set up moses like so >> [directory A]/scripts >> [directory A]/bin >> [directory B] = external bin directory >> the external bin directory has giza/mgiza (and hopefully linguistic >> programs over time). >> >> when you update moses, just replace scripts/ and bin/ . The external >> bin >> directory can stay constant >> >>> >>> Tom >>> >>> >>> On Thu, 31 May 2012 20:42:56 +0100, Hieu Hoang >>> <fishandfrol...@gmail.com> wrote: >>>> Hi all >>>> >>>> if you're checking out the latest github code, there are some >>>> changes >>>> you should be aware of: >>>> 1. There is a new argument to train-model.perl >>>> -external-bin-dir [path] >>>> This points to the directory where Giza++/mgiza lives. >>>> Setting >>>> this is MANDATORY if you're using train-model.perl to do the word >>>> alignment. It used to be hardcoded in the perl code itself. >>>> 2. All the training programs have been moved into the >>>> directory >>>> [MOSES-ROOT]/bin >>>> They should be run from there, not from wherever the >>>> source >>>> code is. >>>> 3. To roll out, simply copy the 2 directories >>>> [MOSES-ROOT]/bin >>>> [MOSES-ROOT]/scripts >>>> to wherever you want, eg. >>>> /home/hieu/moses/bin >>>> /home/hieu/moses/scripts >>>> 4. If you don't want to move it anywhere, you can run it from >>>> where >>>> you downloaded. >>>> 5. The EMS and example files have been updated. >>>> >>>> Hope this is ok for everyone. It may break some people's setup. If >>>> possible, please change your setup. It's gonna help us all in the >>>> long >>>> run. If not, flame me & i'll see what I can do >>>> >>>> HH >>>> >>>> >>>> _______________________________________________ >>>> Moses-support mailing list >>>> Moses-support@mit.edu >>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> >> _______________________________________________ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
