Re: [Moses-support] Moses: Prepare Data, Build Language Model and Train Model

[Eric Nichols]

Greetings,

For anyone still having trouble compiling SRILM under linux, here is
the set of patches I use in building my srilm package.
As mentioned earlier in this thread, if you are running any recent
version of Ubuntu, you can avoid the hassle of compiling
and manually installing dependencies by installing my moses package
from Ubuntu NLP (http://cl.naist.jp/~eric-n/ubuntu-nlp/).
Moses and SRILM have been updated to the latest release, and I have
also packaged both giza-pp and mgiza++ for those
who prefer them to the original GIZA++.

Eric Nichols

On Thu, Aug 14, 2008 at 11:12 AM, Anung Ariwibowo <[EMAIL PROTECTED]> wrote:
> Hi Llio,
>
> Have you commented the TCL part in common/Makefile.machine.i686* ?
>
> In my machine, I commented out these lines:
>
> NO_TCL = X
> #TCL_INCLUDE =
> #TCL_LIBRARY = -ltcl
> TCL_INCLUDE =
> TCL_LIBRARY =
>
> On my Ubuntu Linux, I was using i686 machine type.
>
> Anung
>
> On Wed, Aug 13, 2008 at 9:33 PM, Llio Humphreys <[EMAIL PROTECTED]>
> wrote:
>>
>> Dear Josh,
>> thanks for the links.  I had already found this information, and it
>> helped me compile SRILM on the Mac.  Here, the problem was finding the
>> most appropriate Makefile for the Linux/Ubuntu machine I'm working on:
>> amd athlon x2 dual core x86_64.  $SRILM/common.Makefile.i686_m64
>> seemed the most appropriate, and the CC and  CXX variables are
>> correct, but I still ended up with a lot of errors, unfortunately.
>> Llio
>>
>> On Wed, Aug 13, 2008 at 1:46 PM, Josh Schroeder <[EMAIL PROTECTED]>
>> wrote:
>> > You can also check out the SRILM documentation:
>> > http://www.speech.sri.com/projects/srilm/manpages/
>> > FAQ: http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html
>> >
>> > Or search the SRILM mailing list archives:
>> > http://www.speech.sri.com/projects/srilm/mail-archive/srilm-user/
>> >
>> > -Josh
>> >
>> > On 13 Aug 2008, at 13:37, Anung Ariwibowo wrote:
>> >
>> >> Hi Llio,
>> >>
>> >> I can compile SRILM in Linux Ubuntu without problem. Can you post the
>> >> error message here, maybe we can help.
>> >>
>> >> Cheers,
>> >> Anung
>> >>
>> >> On Wed, Aug 13, 2008 at 8:29 PM, Llio Humphreys <[EMAIL PROTECTED]>
>> >> wrote:
>> >> Dear Josh/Hieu,
>> >> many thanks for your replies.  The default shell is bash, and updating
>> >> the .profile file worked - thanks for that tip.  I look forward to
>> >> hearing more from you about the ./model/extract.0-0.o.part* problem.
>> >> My apologies for my ignorance of Unix matters: I'd like to think of
>> >> myself as a newbie rather than one who is averse to learning about
>> >> these things, and the further information you have provided has been
>> >> useful and interesting.  Hieu mentioned that Anung Ariwibowo got Moses
>> >> to work when he transferred to a Linux machine.  A colleague has
>> >> kindly let me borrow a Linux/Ubuntu machine, but I have already run
>> >> into problems compiling SRILM!  So, I'll see if Eric Nichols's
>> >> packages will take care of that:
>> >> http://cl.naist.jp/~eric-n/ubuntu-nlp/dists/feisty/nlp/
>> >> Best regards,
>> >> Llio
>> >>
>> >>
>> >>
>> >> On 8/13/08, Josh Schroeder <[EMAIL PROTECTED]> wrote:
>> >> > Hi Llio,
>> >> >
>> >> >
>> >> > > you may have already received my email on the following problem
>> >> > > when
>> >> > > building the language model:
>> >> > >
>> >> > > Executing: cat ./model/extract.0-0.o.part* > ./model/extract.0-0.o
>> >> > > cat: ./model/extract.0-0.o.part*: No such file or directory
>> >> > > Exit code: 1
>> >> > >
>> >> >
>> >> >  That's building the phrase table, not the language model. It seems
>> >> > like
>> >> > several people on the list are having problems with this step, so I'm
>> >> > going
>> >> > to take a look at the training process and post something to the list
>> >> > in
>> >> > the
>> >> > next day or two.
>> >> >
>> >> >
>> >> > >
>> >> > > 1. You mention that Moses does not use environment variables.
>> >> > > However, in order to get SRILM to work, I found it necessary to
>> >> > > create
>> >> > > environment variables and pass these on to SRILM's make:
>> >> > >
>> >> > > make SRILM=$PWD MACHINE_TYPE=macosx
>> >> > >
>> >> >
>> >> >
>> >> > PATH=/bin:/sbin:/usr/bin:/usr/sbin:/Users/lliohumphreys/MT/MOSESSUITE/srilm:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin/macosx:/sw/bin/gawk
>> >> > > MANPATH=/Users/lliohumphreys/MT/MOSESSUITE/srilm/man
>> >> > LC_NUMERIC=C
>> >> > >
>> >> > > In addition, I was also required to type in the following command
>> >> > > for
>> >> > > moses-scripts:
>> >> > >
>> >> > > export
>> >> >
>> >> >
>> >> > SCRIPTS_ROOTDIR=/Users/lliohumphreys/MT/MOSESSUITE/bin/moses-scripts/scripts-20080811-1801
>> >> > >
>> >> > >
>> >> >
>> >> >  Sorry, I should have been more clear. Moses itself, the decoder that
>> >> > loads
>> >> > a trained phrase table and language model and translates text, is a
>> >> > self-contained command-line program that doesn't require environment
>> >> > variables.
>

Re: [Moses-support] Moses: Prepare Data, Build Language Model and Train Model

[Anung Ariwibowo]

Hi Llio,

Have you commented the TCL part in common/Makefile.machine.i686* ?

In my machine, I commented out these lines:

NO_TCL = X
#TCL_INCLUDE =
#TCL_LIBRARY = -ltcl
TCL_INCLUDE =
TCL_LIBRARY =

On my Ubuntu Linux, I was using i686 machine type.

Anung

On Wed, Aug 13, 2008 at 9:33 PM, Llio Humphreys <[EMAIL PROTECTED]>wrote:

> Dear Josh,
> thanks for the links.  I had already found this information, and it
> helped me compile SRILM on the Mac.  Here, the problem was finding the
> most appropriate Makefile for the Linux/Ubuntu machine I'm working on:
> amd athlon x2 dual core x86_64.  $SRILM/common.Makefile.i686_m64
> seemed the most appropriate, and the CC and  CXX variables are
> correct, but I still ended up with a lot of errors, unfortunately.
> Llio
>
> On Wed, Aug 13, 2008 at 1:46 PM, Josh Schroeder <[EMAIL PROTECTED]>
> wrote:
> > You can also check out the SRILM documentation:
> > http://www.speech.sri.com/projects/srilm/manpages/
> > FAQ: http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html
> >
> > Or search the SRILM mailing list archives:
> > http://www.speech.sri.com/projects/srilm/mail-archive/srilm-user/
> >
> > -Josh
> >
> > On 13 Aug 2008, at 13:37, Anung Ariwibowo wrote:
> >
> >> Hi Llio,
> >>
> >> I can compile SRILM in Linux Ubuntu without problem. Can you post the
> >> error message here, maybe we can help.
> >>
> >> Cheers,
> >> Anung
> >>
> >> On Wed, Aug 13, 2008 at 8:29 PM, Llio Humphreys <[EMAIL PROTECTED]>
> >> wrote:
> >> Dear Josh/Hieu,
> >> many thanks for your replies.  The default shell is bash, and updating
> >> the .profile file worked - thanks for that tip.  I look forward to
> >> hearing more from you about the ./model/extract.0-0.o.part* problem.
> >> My apologies for my ignorance of Unix matters: I'd like to think of
> >> myself as a newbie rather than one who is averse to learning about
> >> these things, and the further information you have provided has been
> >> useful and interesting.  Hieu mentioned that Anung Ariwibowo got Moses
> >> to work when he transferred to a Linux machine.  A colleague has
> >> kindly let me borrow a Linux/Ubuntu machine, but I have already run
> >> into problems compiling SRILM!  So, I'll see if Eric Nichols's
> >> packages will take care of that:
> >> http://cl.naist.jp/~eric-n/ubuntu-nlp/dists/feisty/nlp/
> >> Best regards,
> >> Llio
> >>
> >>
> >>
> >> On 8/13/08, Josh Schroeder <[EMAIL PROTECTED]> wrote:
> >> > Hi Llio,
> >> >
> >> >
> >> > > you may have already received my email on the following problem when
> >> > > building the language model:
> >> > >
> >> > > Executing: cat ./model/extract.0-0.o.part* > ./model/extract.0-0.o
> >> > > cat: ./model/extract.0-0.o.part*: No such file or directory
> >> > > Exit code: 1
> >> > >
> >> >
> >> >  That's building the phrase table, not the language model. It seems
> like
> >> > several people on the list are having problems with this step, so I'm
> >> > going
> >> > to take a look at the training process and post something to the list
> in
> >> > the
> >> > next day or two.
> >> >
> >> >
> >> > >
> >> > > 1. You mention that Moses does not use environment variables.
> >> > > However, in order to get SRILM to work, I found it necessary to
> create
> >> > > environment variables and pass these on to SRILM's make:
> >> > >
> >> > > make SRILM=$PWD MACHINE_TYPE=macosx
> >> > >
> >> >
> >> >
> PATH=/bin:/sbin:/usr/bin:/usr/sbin:/Users/lliohumphreys/MT/MOSESSUITE/srilm:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin/macosx:/sw/bin/gawk
> >> > > MANPATH=/Users/lliohumphreys/MT/MOSESSUITE/srilm/man
> >> > LC_NUMERIC=C
> >> > >
> >> > > In addition, I was also required to type in the following command
> for
> >> > > moses-scripts:
> >> > >
> >> > > export
> >> >
> >> >
> SCRIPTS_ROOTDIR=/Users/lliohumphreys/MT/MOSESSUITE/bin/moses-scripts/scripts-20080811-1801
> >> > >
> >> > >
> >> >
> >> >  Sorry, I should have been more clear. Moses itself, the decoder that
> >> > loads
> >> > a trained phrase table and language model and translates text, is a
> >> > self-contained command-line program that doesn't require environment
> >> > variables.
> >> >
> >> >  Your first example is compiling SRILM. This is not part of the Moses
> >> > toolkit: it's a toolkit of its own for language modeling and a ton of
> >> > other
> >> > stuff. We use it as one of two possible integrated language models
> (the
> >> > other is IRSTLM) with Moses.
> >> >
> >> >  Your second example is part of the training regime. Yes, there is
> some
> >> > use
> >> > of the SCRIPTS_ROOTDIR in the
> >> > train-factored-phrase-model.perl, but for most training
> >> > support scripts that come with moses there is a flag that lets you
> >> > specify
> >> > SCRIPTS_ROOTDIR at the command line instead of storing it as an
> >> > environment
> >> > variable. In train-factored-phrase-model it's "-scripts-root-dir",
> which
> >> 

Re: [Moses-support] Moses: Prepare Data, Build Language Model and Train Model

[Miles Osborne]

an ugly hack is to simply create a soft link to the i686-m64 directory (as i
recently did on a new 64 bit machine)

Miles

2008/8/13 Sara Stymne <[EMAIL PROTECTED]>

> Hi!
>
> When we installed SRILM and Moses on our 64-bit Ubuntu machine we had
> some troubles with getting the machine type right. What solved it in the
> end was to hack the machine-type script (found in srilm/sbin), so that
> it gave the correct machine type, from i686 to i686-m64:
>
>else if (`uname -m` == x86_64) then
>set MACHINE_TYPE = i686-m64
>#set MACHINE_TYPE = i686
>
> After doing that we could compile SRILM without specifying the
> MACHINE_TYPE.
>
> /Sara
>
>
> Llio Humphreys skrev:
> > Dear Josh,
> > thanks for the links.  I had already found this information, and it
> > helped me compile SRILM on the Mac.  Here, the problem was finding the
> > most appropriate Makefile for the Linux/Ubuntu machine I'm working on:
> > amd athlon x2 dual core x86_64.  $SRILM/common.Makefile.i686_m64
> > seemed the most appropriate, and the CC and  CXX variables are
> > correct, but I still ended up with a lot of errors, unfortunately.
> > Llio
> >
> > On Wed, Aug 13, 2008 at 1:46 PM, Josh Schroeder <[EMAIL PROTECTED]>
> wrote:
> >> You can also check out the SRILM documentation:
> >> http://www.speech.sri.com/projects/srilm/manpages/
> >> FAQ: http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html
> >>
> >> Or search the SRILM mailing list archives:
> >> http://www.speech.sri.com/projects/srilm/mail-archive/srilm-user/
> >>
> >> -Josh
> >>
> >> On 13 Aug 2008, at 13:37, Anung Ariwibowo wrote:
> >>
> >>> Hi Llio,
> >>>
> >>> I can compile SRILM in Linux Ubuntu without problem. Can you post the
> >>> error message here, maybe we can help.
> >>>
> >>> Cheers,
> >>> Anung
> >>>
> >>> On Wed, Aug 13, 2008 at 8:29 PM, Llio Humphreys <[EMAIL PROTECTED]
> >
> >>> wrote:
> >>> Dear Josh/Hieu,
> >>> many thanks for your replies.  The default shell is bash, and updating
> >>> the .profile file worked - thanks for that tip.  I look forward to
> >>> hearing more from you about the ./model/extract.0-0.o.part* problem.
> >>> My apologies for my ignorance of Unix matters: I'd like to think of
> >>> myself as a newbie rather than one who is averse to learning about
> >>> these things, and the further information you have provided has been
> >>> useful and interesting.  Hieu mentioned that Anung Ariwibowo got Moses
> >>> to work when he transferred to a Linux machine.  A colleague has
> >>> kindly let me borrow a Linux/Ubuntu machine, but I have already run
> >>> into problems compiling SRILM!  So, I'll see if Eric Nichols's
> >>> packages will take care of that:
> >>> http://cl.naist.jp/~eric-n/ubuntu-nlp/dists/feisty/nlp/
> >>> Best regards,
> >>> Llio
> >>>
> >>>
> >>>
> >>> On 8/13/08, Josh Schroeder <[EMAIL PROTECTED]> wrote:
>  Hi Llio,
> 
> 
> > you may have already received my email on the following problem when
> > building the language model:
> >
> > Executing: cat ./model/extract.0-0.o.part* > ./model/extract.0-0.o
> > cat: ./model/extract.0-0.o.part*: No such file or directory
> > Exit code: 1
> >
>   That's building the phrase table, not the language model. It seems
> like
>  several people on the list are having problems with this step, so I'm
>  going
>  to take a look at the training process and post something to the list
> in
>  the
>  next day or two.
> 
> 
> > 1. You mention that Moses does not use environment variables.
> > However, in order to get SRILM to work, I found it necessary to
> create
> > environment variables and pass these on to SRILM's make:
> >
> > make SRILM=$PWD MACHINE_TYPE=macosx
> >
> 
> PATH=/bin:/sbin:/usr/bin:/usr/sbin:/Users/lliohumphreys/MT/MOSESSUITE/srilm:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin/macosx:/sw/bin/gawk
> > MANPATH=/Users/lliohumphreys/MT/MOSESSUITE/srilm/man
>  LC_NUMERIC=C
> > In addition, I was also required to type in the following command for
> > moses-scripts:
> >
> > export
> 
> SCRIPTS_ROOTDIR=/Users/lliohumphreys/MT/MOSESSUITE/bin/moses-scripts/scripts-20080811-1801
> >
>   Sorry, I should have been more clear. Moses itself, the decoder that
>  loads
>  a trained phrase table and language model and translates text, is a
>  self-contained command-line program that doesn't require environment
>  variables.
> 
>   Your first example is compiling SRILM. This is not part of the Moses
>  toolkit: it's a toolkit of its own for language modeling and a ton of
>  other
>  stuff. We use it as one of two possible integrated language models
> (the
>  other is IRSTLM) with Moses.
> 
>   Your second example is part of the training regime. Yes, there is
> some
>  use
>  of the SCRIPTS_RO

Re: [Moses-support] Moses: Prepare Data, Build Language Model and Train Model

[Sara Stymne]

Hi!

When we installed SRILM and Moses on our 64-bit Ubuntu machine we had
some troubles with getting the machine type right. What solved it in the
end was to hack the machine-type script (found in srilm/sbin), so that
it gave the correct machine type, from i686 to i686-m64:

else if (`uname -m` == x86_64) then
set MACHINE_TYPE = i686-m64
#set MACHINE_TYPE = i686

After doing that we could compile SRILM without specifying the MACHINE_TYPE.

/Sara


Llio Humphreys skrev:
> Dear Josh,
> thanks for the links.  I had already found this information, and it
> helped me compile SRILM on the Mac.  Here, the problem was finding the
> most appropriate Makefile for the Linux/Ubuntu machine I'm working on:
> amd athlon x2 dual core x86_64.  $SRILM/common.Makefile.i686_m64
> seemed the most appropriate, and the CC and  CXX variables are
> correct, but I still ended up with a lot of errors, unfortunately.
> Llio
> 
> On Wed, Aug 13, 2008 at 1:46 PM, Josh Schroeder <[EMAIL PROTECTED]> wrote:
>> You can also check out the SRILM documentation:
>> http://www.speech.sri.com/projects/srilm/manpages/
>> FAQ: http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html
>>
>> Or search the SRILM mailing list archives:
>> http://www.speech.sri.com/projects/srilm/mail-archive/srilm-user/
>>
>> -Josh
>>
>> On 13 Aug 2008, at 13:37, Anung Ariwibowo wrote:
>>
>>> Hi Llio,
>>>
>>> I can compile SRILM in Linux Ubuntu without problem. Can you post the
>>> error message here, maybe we can help.
>>>
>>> Cheers,
>>> Anung
>>>
>>> On Wed, Aug 13, 2008 at 8:29 PM, Llio Humphreys <[EMAIL PROTECTED]>
>>> wrote:
>>> Dear Josh/Hieu,
>>> many thanks for your replies.  The default shell is bash, and updating
>>> the .profile file worked - thanks for that tip.  I look forward to
>>> hearing more from you about the ./model/extract.0-0.o.part* problem.
>>> My apologies for my ignorance of Unix matters: I'd like to think of
>>> myself as a newbie rather than one who is averse to learning about
>>> these things, and the further information you have provided has been
>>> useful and interesting.  Hieu mentioned that Anung Ariwibowo got Moses
>>> to work when he transferred to a Linux machine.  A colleague has
>>> kindly let me borrow a Linux/Ubuntu machine, but I have already run
>>> into problems compiling SRILM!  So, I'll see if Eric Nichols's
>>> packages will take care of that:
>>> http://cl.naist.jp/~eric-n/ubuntu-nlp/dists/feisty/nlp/
>>> Best regards,
>>> Llio
>>>
>>>
>>>
>>> On 8/13/08, Josh Schroeder <[EMAIL PROTECTED]> wrote:
 Hi Llio,


> you may have already received my email on the following problem when
> building the language model:
>
> Executing: cat ./model/extract.0-0.o.part* > ./model/extract.0-0.o
> cat: ./model/extract.0-0.o.part*: No such file or directory
> Exit code: 1
>
  That's building the phrase table, not the language model. It seems like
 several people on the list are having problems with this step, so I'm
 going
 to take a look at the training process and post something to the list in
 the
 next day or two.


> 1. You mention that Moses does not use environment variables.
> However, in order to get SRILM to work, I found it necessary to create
> environment variables and pass these on to SRILM's make:
>
> make SRILM=$PWD MACHINE_TYPE=macosx
>
 PATH=/bin:/sbin:/usr/bin:/usr/sbin:/Users/lliohumphreys/MT/MOSESSUITE/srilm:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin/macosx:/sw/bin/gawk
> MANPATH=/Users/lliohumphreys/MT/MOSESSUITE/srilm/man
 LC_NUMERIC=C
> In addition, I was also required to type in the following command for
> moses-scripts:
>
> export
 SCRIPTS_ROOTDIR=/Users/lliohumphreys/MT/MOSESSUITE/bin/moses-scripts/scripts-20080811-1801
>
  Sorry, I should have been more clear. Moses itself, the decoder that
 loads
 a trained phrase table and language model and translates text, is a
 self-contained command-line program that doesn't require environment
 variables.

  Your first example is compiling SRILM. This is not part of the Moses
 toolkit: it's a toolkit of its own for language modeling and a ton of
 other
 stuff. We use it as one of two possible integrated language models (the
 other is IRSTLM) with Moses.

  Your second example is part of the training regime. Yes, there is some
 use
 of the SCRIPTS_ROOTDIR in the
 train-factored-phrase-model.perl, but for most training
 support scripts that come with moses there is a flag that lets you
 specify
 SCRIPTS_ROOTDIR at the command line instead of storing it as an
 environment
 variable. In train-factored-phrase-model it's "-scripts-root-dir", which
 I
 think you've actually used in one of your other emails.



> If I open a new terminal and echo these variables, mos

Re: [Moses-support] Moses: Prepare Data, Build Language Model and Train Model

[Llio Humphreys]

Dear Josh,
thanks for the links.  I had already found this information, and it
helped me compile SRILM on the Mac.  Here, the problem was finding the
most appropriate Makefile for the Linux/Ubuntu machine I'm working on:
amd athlon x2 dual core x86_64.  $SRILM/common.Makefile.i686_m64
seemed the most appropriate, and the CC and  CXX variables are
correct, but I still ended up with a lot of errors, unfortunately.
Llio

On Wed, Aug 13, 2008 at 1:46 PM, Josh Schroeder <[EMAIL PROTECTED]> wrote:
> You can also check out the SRILM documentation:
> http://www.speech.sri.com/projects/srilm/manpages/
> FAQ: http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html
>
> Or search the SRILM mailing list archives:
> http://www.speech.sri.com/projects/srilm/mail-archive/srilm-user/
>
> -Josh
>
> On 13 Aug 2008, at 13:37, Anung Ariwibowo wrote:
>
>> Hi Llio,
>>
>> I can compile SRILM in Linux Ubuntu without problem. Can you post the
>> error message here, maybe we can help.
>>
>> Cheers,
>> Anung
>>
>> On Wed, Aug 13, 2008 at 8:29 PM, Llio Humphreys <[EMAIL PROTECTED]>
>> wrote:
>> Dear Josh/Hieu,
>> many thanks for your replies.  The default shell is bash, and updating
>> the .profile file worked - thanks for that tip.  I look forward to
>> hearing more from you about the ./model/extract.0-0.o.part* problem.
>> My apologies for my ignorance of Unix matters: I'd like to think of
>> myself as a newbie rather than one who is averse to learning about
>> these things, and the further information you have provided has been
>> useful and interesting.  Hieu mentioned that Anung Ariwibowo got Moses
>> to work when he transferred to a Linux machine.  A colleague has
>> kindly let me borrow a Linux/Ubuntu machine, but I have already run
>> into problems compiling SRILM!  So, I'll see if Eric Nichols's
>> packages will take care of that:
>> http://cl.naist.jp/~eric-n/ubuntu-nlp/dists/feisty/nlp/
>> Best regards,
>> Llio
>>
>>
>>
>> On 8/13/08, Josh Schroeder <[EMAIL PROTECTED]> wrote:
>> > Hi Llio,
>> >
>> >
>> > > you may have already received my email on the following problem when
>> > > building the language model:
>> > >
>> > > Executing: cat ./model/extract.0-0.o.part* > ./model/extract.0-0.o
>> > > cat: ./model/extract.0-0.o.part*: No such file or directory
>> > > Exit code: 1
>> > >
>> >
>> >  That's building the phrase table, not the language model. It seems like
>> > several people on the list are having problems with this step, so I'm
>> > going
>> > to take a look at the training process and post something to the list in
>> > the
>> > next day or two.
>> >
>> >
>> > >
>> > > 1. You mention that Moses does not use environment variables.
>> > > However, in order to get SRILM to work, I found it necessary to create
>> > > environment variables and pass these on to SRILM's make:
>> > >
>> > > make SRILM=$PWD MACHINE_TYPE=macosx
>> > >
>> >
>> > PATH=/bin:/sbin:/usr/bin:/usr/sbin:/Users/lliohumphreys/MT/MOSESSUITE/srilm:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin/macosx:/sw/bin/gawk
>> > > MANPATH=/Users/lliohumphreys/MT/MOSESSUITE/srilm/man
>> > LC_NUMERIC=C
>> > >
>> > > In addition, I was also required to type in the following command for
>> > > moses-scripts:
>> > >
>> > > export
>> >
>> > SCRIPTS_ROOTDIR=/Users/lliohumphreys/MT/MOSESSUITE/bin/moses-scripts/scripts-20080811-1801
>> > >
>> > >
>> >
>> >  Sorry, I should have been more clear. Moses itself, the decoder that
>> > loads
>> > a trained phrase table and language model and translates text, is a
>> > self-contained command-line program that doesn't require environment
>> > variables.
>> >
>> >  Your first example is compiling SRILM. This is not part of the Moses
>> > toolkit: it's a toolkit of its own for language modeling and a ton of
>> > other
>> > stuff. We use it as one of two possible integrated language models (the
>> > other is IRSTLM) with Moses.
>> >
>> >  Your second example is part of the training regime. Yes, there is some
>> > use
>> > of the SCRIPTS_ROOTDIR in the
>> > train-factored-phrase-model.perl, but for most training
>> > support scripts that come with moses there is a flag that lets you
>> > specify
>> > SCRIPTS_ROOTDIR at the command line instead of storing it as an
>> > environment
>> > variable. In train-factored-phrase-model it's "-scripts-root-dir", which
>> > I
>> > think you've actually used in one of your other emails.
>> >
>> >
>> >
>> > > If I open a new terminal and echo these variables, most of them are
>> > > blank, and PATH just gives the default bin paths.
>> > >
>> > > So, how do I make them permanent?  I assume that if I want to use
>> > > Moses again, it needs to have access to these variables?  How can I
>> > > ensure that I can close the terminal, go home, open a new terminal the
>> > > next day and get Moses working again?  A colleague suggested I update
>> > > the .bashrc file to update each new terminal session with these
>> > > environment variables. However, my 

Re: [Moses-support] hypergraph-prototype

[Chris Callison-Burch]

Hi Joao,

The hypergraph prototype is something that we were experimenting with  
at the open source convention at this year's MT marathon.  We're  
developing an open source implementation of Hiero at Hopkins, which  
will be released at the end of this summer.  I'll make an announcement  
on this list once it's ready.

--Chris

On Aug 13, 2008, at 2:17 PM, João Graça wrote:

> Hello,
>
> I just saw in the new moses release that there is hierarchical  
> decoder.
> But i did not found any readme file or how to use it.
> Is it ready to be use? Can anyone point me out how to use it, and  
> its relation to Hiero.
>
> Thanks a lot,
>
> João Graça
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Moses: Prepare Data, Build Language Model and Train Model

[Llio Humphreys]

Hi Anung,
there are lots of errors!  I ran the following command:

make World MACHINE_TYPE=i686-m64 > errors.txt

My previous command was:

find . -print0 | xargs -r0 touch

so that make would compile everything anew.

The output screen ended with:

Trie.cc:253: error: 'done' was not declared in this scope
make[2]: *** [../obj/i686-m64/SArrayTrie.o] Error 1
make[1]: *** [release-libraries] Error 1
make: *** [World] Error 2

and the errors file contained the following:-

mkdir include lib bin
make init
for subdir in misc dstruct lm flm lattice utils; do \
(cd $subdir/src; make SRILM=/home/llio/MT/MOSESSUITE/srilm
MACHINE_TYPE=i686-m64 OPTION= init) || exit 1; \
done
make[2]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/misc/src'
cd ..; /home/llio/MT/MOSESSUITE/srilm/sbin/make-standard-directories
make ../obj/i686-m64/STAMP ../bin/i686-m64/STAMP
make[3]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/misc/src'
make[3]: `../obj/i686-m64/STAMP' is up to date.
make[3]: `../bin/i686-m64/STAMP' is up to date.
make[3]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm/misc/src'
make[2]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm/misc/src'
make[2]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/dstruct/src'
cd ..; /home/llio/MT/MOSESSUITE/srilm/sbin/make-standard-directories
make ../obj/i686-m64/STAMP ../bin/i686-m64/STAMP
make[3]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/dstruct/src'
make[3]: `../obj/i686-m64/STAMP' is up to date.
make[3]: `../bin/i686-m64/STAMP' is up to date.
make[3]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm/dstruct/src'
make[2]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm/dstruct/src'
make[2]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/lm/src'
cd ..; /home/llio/MT/MOSESSUITE/srilm/sbin/make-standard-directories
make ../obj/i686-m64/STAMP ../bin/i686-m64/STAMP
make[3]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/lm/src'
make[3]: `../obj/i686-m64/STAMP' is up to date.
make[3]: `../bin/i686-m64/STAMP' is up to date.
make[3]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm/lm/src'
make[2]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm/lm/src'
make[2]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/flm/src'
cd ..; /home/llio/MT/MOSESSUITE/srilm/sbin/make-standard-directories
make ../obj/i686-m64/STAMP ../bin/i686-m64/STAMP
make[3]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/flm/src'
make[3]: `../obj/i686-m64/STAMP' is up to date.
make[3]: `../bin/i686-m64/STAMP' is up to date.
make[3]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm/flm/src'
make[2]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm/flm/src'
make[2]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/lattice/src'
cd ..; /home/llio/MT/MOSESSUITE/srilm/sbin/make-standard-directories
make ../obj/i686-m64/STAMP ../bin/i686-m64/STAMP
make[3]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/lattice/src'
make[3]: `../obj/i686-m64/STAMP' is up to date.
make[3]: `../bin/i686-m64/STAMP' is up to date.
make[3]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm/lattice/src'
make[2]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm/lattice/src'
make[2]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/utils/src'
cd ..; /home/llio/MT/MOSESSUITE/srilm/sbin/make-standard-directories
make ../obj/i686-m64/STAMP ../bin/i686-m64/STAMP
make[3]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/utils/src'
make[3]: `../obj/i686-m64/STAMP' is up to date.
make[3]: `../bin/i686-m64/STAMP' is up to date.
make[3]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm/utils/src'
make[2]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm/utils/src'
make[1]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm'
make release-headers
for subdir in misc dstruct lm flm lattice utils; do \
(cd $subdir/src; make SRILM=/home/llio/MT/MOSESSUITE/srilm
MACHINE_TYPE=i686-m64 OPTION= release-headers) || exit 1; \
done
make[2]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/misc/src'
/home/llio/MT/MOSESSUITE/srilm/sbin/decipher-install 0444 cfuncproto.h
../../include
/home/llio/MT/MOSESSUITE/srilm/sbin/decipher-install 0444 option.h ../../include
/home/llio/MT/MOSESSUITE/srilm/sbin/decipher-install 0444 zio.h ../../include
/home/llio/MT/MOSESSUITE/srilm/sbin/decipher-install 0444 mkdir.h ../../include
/home/llio/MT/MOSESSUITE/srilm/sbin/decipher-install 0444 Boolean.h
../../include
/home/llio/MT/MOSESSUITE/srilm/sbin/decipher-install 0444 Debug.h ../../include
/home/llio/MT/MOSESSUITE/srilm/sbin/decipher-install 0444 File.h ../../include
/home/llio/MT/MOSESSUITE/srilm/sbin/decipher-install 0444 version.h
../../include
make[2]: Leaving directory `/home/llio/MT/MOSESSUITE/srilm/misc/src'
make[2]: Entering directory `/home/llio/MT/MOSESSUITE/srilm/dstruct/src'
/home/llio/MT/MOSESSUITE/srilm/sbin/decipher-install 0444 Array.h ../../include
/home/llio/MT/MOSESSUITE/srilm/sbin/decipher-install 0444
IntervalHeap.h ../../include
/home/llio/MT/MOSESSUITE

[Moses-support] hypergraph-prototype

[João Graça]

Hello,

I just saw in the new moses release that there is hierarchical decoder.
But i did not found any readme file or how to use it.
Is it ready to be use? Can anyone point me out how to use it, and its
relation to Hiero.

Thanks a lot,

João Graça
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Moses: Prepare Data, Build Language Model and Train Model

[Josh Schroeder]

You can also check out the SRILM documentation:
http://www.speech.sri.com/projects/srilm/manpages/
FAQ: http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html

Or search the SRILM mailing list archives:
http://www.speech.sri.com/projects/srilm/mail-archive/srilm-user/

-Josh

On 13 Aug 2008, at 13:37, Anung Ariwibowo wrote:

> Hi Llio,
>
> I can compile SRILM in Linux Ubuntu without problem. Can you post  
> the error message here, maybe we can help.
>
> Cheers,
> Anung
>
> On Wed, Aug 13, 2008 at 8:29 PM, Llio Humphreys  
> <[EMAIL PROTECTED]> wrote:
> Dear Josh/Hieu,
> many thanks for your replies.  The default shell is bash, and updating
> the .profile file worked - thanks for that tip.  I look forward to
> hearing more from you about the ./model/extract.0-0.o.part* problem.
> My apologies for my ignorance of Unix matters: I'd like to think of
> myself as a newbie rather than one who is averse to learning about
> these things, and the further information you have provided has been
> useful and interesting.  Hieu mentioned that Anung Ariwibowo got Moses
> to work when he transferred to a Linux machine.  A colleague has
> kindly let me borrow a Linux/Ubuntu machine, but I have already run
> into problems compiling SRILM!  So, I'll see if Eric Nichols's
> packages will take care of that:
> http://cl.naist.jp/~eric-n/ubuntu-nlp/dists/feisty/nlp/
> Best regards,
> Llio
>
>
>
> On 8/13/08, Josh Schroeder <[EMAIL PROTECTED]> wrote:
> > Hi Llio,
> >
> >
> > > you may have already received my email on the following problem  
> when
> > > building the language model:
> > >
> > > Executing: cat ./model/extract.0-0.o.part* > ./model/extract.0-0.o
> > > cat: ./model/extract.0-0.o.part*: No such file or directory
> > > Exit code: 1
> > >
> >
> >  That's building the phrase table, not the language model. It  
> seems like
> > several people on the list are having problems with this step, so  
> I'm going
> > to take a look at the training process and post something to the  
> list in the
> > next day or two.
> >
> >
> > >
> > > 1. You mention that Moses does not use environment variables.
> > > However, in order to get SRILM to work, I found it necessary to  
> create
> > > environment variables and pass these on to SRILM's make:
> > >
> > > make SRILM=$PWD MACHINE_TYPE=macosx
> > >
> > PATH=/bin:/sbin:/usr/bin:/usr/sbin:/Users/lliohumphreys/MT/ 
> MOSESSUITE/srilm:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin:/Users/ 
> lliohumphreys/MT/MOSESSUITE/srilm/bin/macosx:/sw/bin/gawk
> > > MANPATH=/Users/lliohumphreys/MT/MOSESSUITE/srilm/man
> > LC_NUMERIC=C
> > >
> > > In addition, I was also required to type in the following  
> command for
> > > moses-scripts:
> > >
> > > export
> > SCRIPTS_ROOTDIR=/Users/lliohumphreys/MT/MOSESSUITE/bin/moses- 
> scripts/scripts-20080811-1801
> > >
> > >
> >
> >  Sorry, I should have been more clear. Moses itself, the decoder  
> that loads
> > a trained phrase table and language model and translates text, is a
> > self-contained command-line program that doesn't require environment
> > variables.
> >
> >  Your first example is compiling SRILM. This is not part of the  
> Moses
> > toolkit: it's a toolkit of its own for language modeling and a ton  
> of other
> > stuff. We use it as one of two possible integrated language models  
> (the
> > other is IRSTLM) with Moses.
> >
> >  Your second example is part of the training regime. Yes, there is  
> some use
> > of the SCRIPTS_ROOTDIR in the
> > train-factored-phrase-model.perl, but for most training
> > support scripts that come with moses there is a flag that lets you  
> specify
> > SCRIPTS_ROOTDIR at the command line instead of storing it as an  
> environment
> > variable. In train-factored-phrase-model it's "-scripts-root-dir",  
> which I
> > think you've actually used in one of your other emails.
> >
> >
> >
> > > If I open a new terminal and echo these variables, most of them  
> are
> > > blank, and PATH just gives the default bin paths.
> > >
> > > So, how do I make them permanent?  I assume that if I want to use
> > > Moses again, it needs to have access to these variables?  How  
> can I
> > > ensure that I can close the terminal, go home, open a new  
> terminal the
> > > next day and get Moses working again?  A colleague suggested I  
> update
> > > the .bashrc file to update each new terminal session with these
> > > environment variables. However, my Mac system does not appear to  
> have
> > > a .bashrc system as a default, and when I created one in my home
> > > directory and opened a new terminal, it did not access the .bashrc
> > > file.
> > >
> >
> >  Here's some info on environment variables on the Mac, found with  
> a quick
> > Google search:
> >  http://www.macdevcenter.com/pub/a/mac/2004/02/24/bash.html
> >
> >  I tried it with .profile, that worked fine. Are you sure you're  
> set to use
> > the bash shell? Try ' echo $SHELL ' in Terminal.
> >
> >
> > > 2. You say that you ran the decoder on your la

Re: [Moses-support] Moses: Prepare Data, Build Language Model and Train Model

[Anung Ariwibowo]

Hi Llio,

I can compile SRILM in Linux Ubuntu without problem. Can you post the error
message here, maybe we can help.

Cheers,
Anung

On Wed, Aug 13, 2008 at 8:29 PM, Llio Humphreys <[EMAIL PROTECTED]>wrote:

> Dear Josh/Hieu,
> many thanks for your replies.  The default shell is bash, and updating
> the .profile file worked - thanks for that tip.  I look forward to
> hearing more from you about the ./model/extract.0-0.o.part* problem.
> My apologies for my ignorance of Unix matters: I'd like to think of
> myself as a newbie rather than one who is averse to learning about
> these things, and the further information you have provided has been
> useful and interesting.  Hieu mentioned that Anung Ariwibowo got Moses
> to work when he transferred to a Linux machine.  A colleague has
> kindly let me borrow a Linux/Ubuntu machine, but I have already run
> into problems compiling SRILM!  So, I'll see if Eric Nichols's
> packages will take care of that:
> http://cl.naist.jp/~eric-n/ubuntu-nlp/dists/feisty/nlp/
> Best regards,
> Llio
>
>
>
> On 8/13/08, Josh Schroeder <[EMAIL PROTECTED]> wrote:
> > Hi Llio,
> >
> >
> > > you may have already received my email on the following problem when
> > > building the language model:
> > >
> > > Executing: cat ./model/extract.0-0.o.part* > ./model/extract.0-0.o
> > > cat: ./model/extract.0-0.o.part*: No such file or directory
> > > Exit code: 1
> > >
> >
> >  That's building the phrase table, not the language model. It seems like
> > several people on the list are having problems with this step, so I'm
> going
> > to take a look at the training process and post something to the list in
> the
> > next day or two.
> >
> >
> > >
> > > 1. You mention that Moses does not use environment variables.
> > > However, in order to get SRILM to work, I found it necessary to create
> > > environment variables and pass these on to SRILM's make:
> > >
> > > make SRILM=$PWD MACHINE_TYPE=macosx
> > >
> >
> PATH=/bin:/sbin:/usr/bin:/usr/sbin:/Users/lliohumphreys/MT/MOSESSUITE/srilm:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin/macosx:/sw/bin/gawk
> > > MANPATH=/Users/lliohumphreys/MT/MOSESSUITE/srilm/man
> > LC_NUMERIC=C
> > >
> > > In addition, I was also required to type in the following command for
> > > moses-scripts:
> > >
> > > export
> >
> SCRIPTS_ROOTDIR=/Users/lliohumphreys/MT/MOSESSUITE/bin/moses-scripts/scripts-20080811-1801
> > >
> > >
> >
> >  Sorry, I should have been more clear. Moses itself, the decoder that
> loads
> > a trained phrase table and language model and translates text, is a
> > self-contained command-line program that doesn't require environment
> > variables.
> >
> >  Your first example is compiling SRILM. This is not part of the Moses
> > toolkit: it's a toolkit of its own for language modeling and a ton of
> other
> > stuff. We use it as one of two possible integrated language models (the
> > other is IRSTLM) with Moses.
> >
> >  Your second example is part of the training regime. Yes, there is some
> use
> > of the SCRIPTS_ROOTDIR in the
> > train-factored-phrase-model.perl, but for most training
> > support scripts that come with moses there is a flag that lets you
> specify
> > SCRIPTS_ROOTDIR at the command line instead of storing it as an
> environment
> > variable. In train-factored-phrase-model it's "-scripts-root-dir", which
> I
> > think you've actually used in one of your other emails.
> >
> >
> >
> > > If I open a new terminal and echo these variables, most of them are
> > > blank, and PATH just gives the default bin paths.
> > >
> > > So, how do I make them permanent?  I assume that if I want to use
> > > Moses again, it needs to have access to these variables?  How can I
> > > ensure that I can close the terminal, go home, open a new terminal the
> > > next day and get Moses working again?  A colleague suggested I update
> > > the .bashrc file to update each new terminal session with these
> > > environment variables. However, my Mac system does not appear to have
> > > a .bashrc system as a default, and when I created one in my home
> > > directory and opened a new terminal, it did not access the .bashrc
> > > file.
> > >
> >
> >  Here's some info on environment variables on the Mac, found with a quick
> > Google search:
> >  http://www.macdevcenter.com/pub/a/mac/2004/02/24/bash.html
> >
> >  I tried it with .profile, that worked fine. Are you sure you're set to
> use
> > the bash shell? Try ' echo $SHELL ' in Terminal.
> >
> >
> > > 2. You say that you ran the decoder on your laptop just fine, but had
> > > to change a few scripts for training.  I have very basic knowledge of
> > > Unix systems and installing open-source software: would it be possible
> > > for you to detail the changes you did to the scripts to get it to run
> > > on a Mac?  Although I need this information urgently, it may also be
> > > useful for other students who are install

Re: [Moses-support] Moses: Prepare Data, Build Language Model and Train Model

[Llio Humphreys]

Dear Josh/Hieu,
many thanks for your replies.  The default shell is bash, and updating
the .profile file worked - thanks for that tip.  I look forward to
hearing more from you about the ./model/extract.0-0.o.part* problem.
My apologies for my ignorance of Unix matters: I'd like to think of
myself as a newbie rather than one who is averse to learning about
these things, and the further information you have provided has been
useful and interesting.  Hieu mentioned that Anung Ariwibowo got Moses
to work when he transferred to a Linux machine.  A colleague has
kindly let me borrow a Linux/Ubuntu machine, but I have already run
into problems compiling SRILM!  So, I'll see if Eric Nichols's
packages will take care of that:
http://cl.naist.jp/~eric-n/ubuntu-nlp/dists/feisty/nlp/
Best regards,
Llio



On 8/13/08, Josh Schroeder <[EMAIL PROTECTED]> wrote:
> Hi Llio,
>
>
> > you may have already received my email on the following problem when
> > building the language model:
> >
> > Executing: cat ./model/extract.0-0.o.part* > ./model/extract.0-0.o
> > cat: ./model/extract.0-0.o.part*: No such file or directory
> > Exit code: 1
> >
>
>  That's building the phrase table, not the language model. It seems like
> several people on the list are having problems with this step, so I'm going
> to take a look at the training process and post something to the list in the
> next day or two.
>
>
> >
> > 1. You mention that Moses does not use environment variables.
> > However, in order to get SRILM to work, I found it necessary to create
> > environment variables and pass these on to SRILM's make:
> >
> > make SRILM=$PWD MACHINE_TYPE=macosx
> >
> PATH=/bin:/sbin:/usr/bin:/usr/sbin:/Users/lliohumphreys/MT/MOSESSUITE/srilm:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin/macosx:/sw/bin/gawk
> > MANPATH=/Users/lliohumphreys/MT/MOSESSUITE/srilm/man
> LC_NUMERIC=C
> >
> > In addition, I was also required to type in the following command for
> > moses-scripts:
> >
> > export
> SCRIPTS_ROOTDIR=/Users/lliohumphreys/MT/MOSESSUITE/bin/moses-scripts/scripts-20080811-1801
> >
> >
>
>  Sorry, I should have been more clear. Moses itself, the decoder that loads
> a trained phrase table and language model and translates text, is a
> self-contained command-line program that doesn't require environment
> variables.
>
>  Your first example is compiling SRILM. This is not part of the Moses
> toolkit: it's a toolkit of its own for language modeling and a ton of other
> stuff. We use it as one of two possible integrated language models (the
> other is IRSTLM) with Moses.
>
>  Your second example is part of the training regime. Yes, there is some use
> of the SCRIPTS_ROOTDIR in the
> train-factored-phrase-model.perl, but for most training
> support scripts that come with moses there is a flag that lets you specify
> SCRIPTS_ROOTDIR at the command line instead of storing it as an environment
> variable. In train-factored-phrase-model it's "-scripts-root-dir", which I
> think you've actually used in one of your other emails.
>
>
>
> > If I open a new terminal and echo these variables, most of them are
> > blank, and PATH just gives the default bin paths.
> >
> > So, how do I make them permanent?  I assume that if I want to use
> > Moses again, it needs to have access to these variables?  How can I
> > ensure that I can close the terminal, go home, open a new terminal the
> > next day and get Moses working again?  A colleague suggested I update
> > the .bashrc file to update each new terminal session with these
> > environment variables. However, my Mac system does not appear to have
> > a .bashrc system as a default, and when I created one in my home
> > directory and opened a new terminal, it did not access the .bashrc
> > file.
> >
>
>  Here's some info on environment variables on the Mac, found with a quick
> Google search:
>  http://www.macdevcenter.com/pub/a/mac/2004/02/24/bash.html
>
>  I tried it with .profile, that worked fine. Are you sure you're set to use
> the bash shell? Try ' echo $SHELL ' in Terminal.
>
>
> > 2. You say that you ran the decoder on your laptop just fine, but had
> > to change a few scripts for training.  I have very basic knowledge of
> > Unix systems and installing open-source software: would it be possible
> > for you to detail the changes you did to the scripts to get it to run
> > on a Mac?  Although I need this information urgently, it may also be
> > useful for other students who are installing Moses on a Mac and who
> > may also have basic knowledge of Unix installation procedures.
> >
>
>  I'll look into this. Mac isn't really the platform of choice for training a
> Moses model and I do most of my work on linux. If I recall correctly, an
> Intel-based Mac should be easier to get working than a PowerPC one. The
> *decoder* does work on my Intel-based laptop, but I haven't run a full
> training setup locally in some time -- most of the time we're working with
> so much data t

Re: [Moses-support] Moses: Prepare Data, Build Language Model and Train Model

[Josh Schroeder]

Hi Llio,

> you may have already received my email on the following problem when
> building the language model:
>
> Executing: cat ./model/extract.0-0.o.part* > ./model/extract.0-0.o
> cat: ./model/extract.0-0.o.part*: No such file or directory
> Exit code: 1

That's building the phrase table, not the language model. It seems  
like several people on the list are having problems with this step, so  
I'm going to take a look at the training process and post something to  
the list in the next day or two.

>
> 1. You mention that Moses does not use environment variables.
> However, in order to get SRILM to work, I found it necessary to create
> environment variables and pass these on to SRILM's make:
>
> make SRILM=$PWD MACHINE_TYPE=macosx
> PATH=/bin:/sbin:/usr/bin:/usr/sbin:/Users/lliohumphreys/MT/ 
> MOSESSUITE/srilm:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin:/Users/ 
> lliohumphreys/MT/MOSESSUITE/srilm/bin/macosx:/sw/bin/gawk
> MANPATH=/Users/lliohumphreys/MT/MOSESSUITE/srilm/man LC_NUMERIC=C
>
> In addition, I was also required to type in the following command for
> moses-scripts:
>
> export SCRIPTS_ROOTDIR=/Users/lliohumphreys/MT/MOSESSUITE/bin/moses- 
> scripts/scripts-20080811-1801
>

Sorry, I should have been more clear. Moses itself, the decoder that  
loads a trained phrase table and language model and translates text,  
is a self-contained command-line program that doesn't require  
environment variables.

Your first example is compiling SRILM. This is not part of the Moses  
toolkit: it's a toolkit of its own for language modeling and a ton of  
other stuff. We use it as one of two possible integrated language  
models (the other is IRSTLM) with Moses.

Your second example is part of the training regime. Yes, there is some  
use of the SCRIPTS_ROOTDIR in the train-factored-phrase-model.perl,  
but for most training support scripts that come with moses there is a  
flag that lets you specify SCRIPTS_ROOTDIR at the command line instead  
of storing it as an environment variable. In train-factored-phrase- 
model it's "-scripts-root-dir", which I think you've actually used in  
one of your other emails.


> If I open a new terminal and echo these variables, most of them are
> blank, and PATH just gives the default bin paths.
>
> So, how do I make them permanent?  I assume that if I want to use
> Moses again, it needs to have access to these variables?  How can I
> ensure that I can close the terminal, go home, open a new terminal the
> next day and get Moses working again?  A colleague suggested I update
> the .bashrc file to update each new terminal session with these
> environment variables. However, my Mac system does not appear to have
> a .bashrc system as a default, and when I created one in my home
> directory and opened a new terminal, it did not access the .bashrc
> file.

Here's some info on environment variables on the Mac, found with a  
quick Google search:
http://www.macdevcenter.com/pub/a/mac/2004/02/24/bash.html

I tried it with .profile, that worked fine. Are you sure you're set to  
use the bash shell? Try ' echo $SHELL ' in Terminal.

> 2. You say that you ran the decoder on your laptop just fine, but had
> to change a few scripts for training.  I have very basic knowledge of
> Unix systems and installing open-source software: would it be possible
> for you to detail the changes you did to the scripts to get it to run
> on a Mac?  Although I need this information urgently, it may also be
> useful for other students who are installing Moses on a Mac and who
> may also have basic knowledge of Unix installation procedures.

I'll look into this. Mac isn't really the platform of choice for  
training a Moses model and I do most of my work on linux. If I recall  
correctly, an Intel-based Mac should be easier to get working than a  
PowerPC one. The *decoder* does work on my Intel-based laptop, but I  
haven't run a full training setup locally in some time -- most of the  
time we're working with so much data that I use a cluster of linux  
machines instead of my Mac.

As a word of caution: Moses isn't an out-of-the box translation  
solution for end users. It's research software undergoing active  
development, so almost every user -- on any platform --  will need to  
muck around in the scripts at some point, or face a compile error or  
runtime crash. The ability to deal with unix/linux command line tools,  
and debug code and scripts when necessary, is really important. That  
being said, I'll see what I can do about highlighting where the  
scripts might have problems on the Mac.

> 3. My final question: which is embarrasingly basic...can I use the one
> installation of Moses for different corpora, or do I need to do a
> separate installation for each one?  Can I have separate installations
> of SRILM, Giza and mckls, or should they all reference the same
> libraries?

All you need to do to have moses use different corpora is point it to  
a different moses.ini file. Assuming you