Re: [Moses-support] ERROR

[Hieu Hoang]

you should use experiment.perl rather trying to run every stage of training
by hand.
   http://www2.statmt.org/moses/?n=FactoredTraining.EMS
Adapt the basic example config to your own data and machine:

https://github.com/moses-smt/mosesdecoder/blob/master/scripts/ems/example/config.basic

Hieu Hoang
https://hieuhoang.github.io/


On Sat, 28 Sept 2024 at 06:23, Dhiraj Pratim Barman <
dhirajpratimbar...@gmail.com> wrote:

>
> Sir,
> I was trying to train the translation model and after writing the commands
> as per the manual I got the following errors.
>
> Error: 'sVok[z] Error: 'sVok[z] Error: 'sVok[z] Error: 'sVok[z] Error: 'sVok[z] Error: 'sVok[z] Error: 'sVok[z] Error: 'sVok[z] ERROR: Execution of: /home/dhiraj/mosesdecoder/tools/mkcls -c50 -n2
> -p/home/dhiraj/corpus/news-commentary-v8.fr-en.clean.fr
> -V/home/dhiraj/working/train/corpus/fr.vcb.classes opt
>   died with signal 11, with coredump
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> https://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
https://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] ERROR

[Dhiraj Pratim Barman]

Sir,
I was trying to train the translation model and after writing the commands
as per the manual I got the following errors.

Error: 'sVok[z]___
Moses-support mailing list
Moses-support@mit.edu
https://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error: No such file or directory while opening /home/xyz/working/train/model/extract.o.sorted.gz

[Sainik Kumar Mahata]

I installed python-dev-is-python3, and it solved the problem. Thanks.

On Mon, 9 Jan, 2023, 8:15 pm Oleg Parashchenko,  wrote:

> Hello,
>
> in my docker images I install "python3-dev" and later create a symlink:
>
> ```
> ln -s /usr/bin/python3 /usr/bin/python
> ```
>
> Regards,
> Oleg
>
> On Wed, 4 Jan 2023 17:35:41 +0530
> Sainik  Kumar Mahata  wrote:
>
> > I think, the issue is with the "python-dev" package that is installed as
> a
> > dependency.
> >
> > Executing: /home/sainik/mosesdecoder/tools/merge_alignment.py
> >
> /home/sainik/working/train/giza.bn-en/bn-en.A3.final.part*>/home/sainik/working/train/giza.bn-en/bn-en.A3.final
> > /usr/bin/env: 'python': No such file or directory
> > Exit code: 127
> >
> > When I am trying to install python-dev, ubuntu says that there is no
> > installation certificate, and instead tells me to
> > install python2-dev/python2/python-dev-is-python3
> >
> > Can you kindly suggest, what to install instead of python-dev?
> >
> >
> >
> >
> > [image: Mailtrack]
> > <
> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&;
> >
> > Sender
> > notified by
> > Mailtrack
> > <
> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&;
> >
> > 01/04/23,
> > 05:35:23 PM
> >
> > On Wed, Jan 4, 2023 at 5:26 PM Hieu Hoang  wrote:
> >
> > > it may be that a previous step failed, eg due to running out of disk
> > > space, choking on bad data.
> > >
> > > Check the logs for the previous steps
> > >
> > > Hieu Hoang
> > > http://statmt.org/hieu
> > >
> > >
> > > On Wed, 4 Jan 2023 at 11:42, Sainik Kumar Mahata <
> sainik.mah...@gmail.com>
> > > wrote:
> > >
> > >> Dear All,
> > >>
> > >> I have trained my translation models using Moses Toolkit multiple
> times
> > >> before, but recently, when I tried training a fresh model, I am
> receiving
> > >> the following error message.
> > >>
> > >> No such file or directory while opening
> > >> /home/sainik/working/train/model/extract.o.sorted.gz
> > >> Aborted
> > >> Exit code: 134
> > >>
> > >> Also, when I am looking into the log file, I can find two more
> > >> warning/error messages.
> > >>
> > >> Use of uninitialized value in substitution (s///) at
> > >> /home/xyx/mosesdecoder/scripts/training/LexicalTranslationModel.pm
> line 40.
> > >> Use of uninitialized value $a in split at
> > >> /home/xyz/mosesdecoder/scripts/training/LexicalTranslationModel.pm
> line 109.
> > >>
> > >> This never happened before and it would be very kind of anyone, who
> can
> > >> guide me on this issue.
> > >>
> > >> Thanks in advance.
> > >>
> > >>
> > >> --
> > >>
> > >> Regards,
> > >>
> > >> Sainik
> > >> Phone/WA: +91-9038141146
> > >>
> > >>
> > >> [image: Mailtrack]
> > >> <
> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&;>
> Sender
> > >> notified by
> > >> Mailtrack
> > >> <
> https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&;>
> 01/04/23,
> > >> 05:05:11 PM
> > >> ___
> > >> Moses-support mailing list
> > >> Moses-support@mit.edu
> > >> https://mailman.mit.edu/mailman/listinfo/moses-support
> > >>
> > >
> >
> > --
> >
> > Regards,
> >
> > Sainik
> > Phone/WA: +91-9038141146
>
>
> --
> Mit freundlichen Grüßen,
> Oleg Parashchenko
>
___
Moses-support mailing list
Moses-support@mit.edu
https://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error: No such file or directory while opening /home/xyz/working/train/model/extract.o.sorted.gz

[Oleg Parashchenko]

Hello,

in my docker images I install "python3-dev" and later create a symlink:

```
ln -s /usr/bin/python3 /usr/bin/python
```

Regards,
Oleg

On Wed, 4 Jan 2023 17:35:41 +0530
Sainik  Kumar Mahata  wrote:

> I think, the issue is with the "python-dev" package that is installed as a
> dependency.
> 
> Executing: /home/sainik/mosesdecoder/tools/merge_alignment.py
>  
> /home/sainik/working/train/giza.bn-en/bn-en.A3.final.part*>/home/sainik/working/train/giza.bn-en/bn-en.A3.final
> /usr/bin/env: 'python': No such file or directory
> Exit code: 127
> 
> When I am trying to install python-dev, ubuntu says that there is no
> installation certificate, and instead tells me to
> install python2-dev/python2/python-dev-is-python3
> 
> Can you kindly suggest, what to install instead of python-dev?
> 
> 
> 
> 
> [image: Mailtrack]
> 
> Sender
> notified by
> Mailtrack
> 
> 01/04/23,
> 05:35:23 PM
> 
> On Wed, Jan 4, 2023 at 5:26 PM Hieu Hoang  wrote:
> 
> > it may be that a previous step failed, eg due to running out of disk
> > space, choking on bad data.
> >
> > Check the logs for the previous steps
> >
> > Hieu Hoang
> > http://statmt.org/hieu
> >
> >
> > On Wed, 4 Jan 2023 at 11:42, Sainik Kumar Mahata 
> > wrote:
> >
> >> Dear All,
> >>
> >> I have trained my translation models using Moses Toolkit multiple times
> >> before, but recently, when I tried training a fresh model, I am receiving
> >> the following error message.
> >>
> >> No such file or directory while opening
> >> /home/sainik/working/train/model/extract.o.sorted.gz
> >> Aborted
> >> Exit code: 134
> >>
> >> Also, when I am looking into the log file, I can find two more
> >> warning/error messages.
> >>
> >> Use of uninitialized value in substitution (s///) at
> >> /home/xyx/mosesdecoder/scripts/training/LexicalTranslationModel.pm line 40.
> >> Use of uninitialized value $a in split at
> >> /home/xyz/mosesdecoder/scripts/training/LexicalTranslationModel.pm line 
> >> 109.
> >>
> >> This never happened before and it would be very kind of anyone, who can
> >> guide me on this issue.
> >>
> >> Thanks in advance.
> >>
> >>
> >> --
> >>
> >> Regards,
> >>
> >> Sainik
> >> Phone/WA: +91-9038141146
> >>
> >>
> >> [image: Mailtrack]
> >> 
> >>  Sender
> >> notified by
> >> Mailtrack
> >> 
> >>  01/04/23,
> >> 05:05:11 PM
> >> ___
> >> Moses-support mailing list
> >> Moses-support@mit.edu
> >> https://mailman.mit.edu/mailman/listinfo/moses-support
> >>
> >
> 
> -- 
> 
> Regards,
> 
> Sainik
> Phone/WA: +91-9038141146


-- 
Mit freundlichen Grüßen,
Oleg Parashchenko
___
Moses-support mailing list
Moses-support@mit.edu
https://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error: No such file or directory while opening /home/xyz/working/train/model/extract.o.sorted.gz

[Sainik Kumar Mahata]

I think, the issue is with the "python-dev" package that is installed as a
dependency.

Executing: /home/sainik/mosesdecoder/tools/merge_alignment.py
 
/home/sainik/working/train/giza.bn-en/bn-en.A3.final.part*>/home/sainik/working/train/giza.bn-en/bn-en.A3.final
/usr/bin/env: 'python': No such file or directory
Exit code: 127

When I am trying to install python-dev, ubuntu says that there is no
installation certificate, and instead tells me to
install python2-dev/python2/python-dev-is-python3

Can you kindly suggest, what to install instead of python-dev?




[image: Mailtrack]

Sender
notified by
Mailtrack

01/04/23,
05:35:23 PM

On Wed, Jan 4, 2023 at 5:26 PM Hieu Hoang  wrote:

> it may be that a previous step failed, eg due to running out of disk
> space, choking on bad data.
>
> Check the logs for the previous steps
>
> Hieu Hoang
> http://statmt.org/hieu
>
>
> On Wed, 4 Jan 2023 at 11:42, Sainik Kumar Mahata 
> wrote:
>
>> Dear All,
>>
>> I have trained my translation models using Moses Toolkit multiple times
>> before, but recently, when I tried training a fresh model, I am receiving
>> the following error message.
>>
>> No such file or directory while opening
>> /home/sainik/working/train/model/extract.o.sorted.gz
>> Aborted
>> Exit code: 134
>>
>> Also, when I am looking into the log file, I can find two more
>> warning/error messages.
>>
>> Use of uninitialized value in substitution (s///) at
>> /home/xyx/mosesdecoder/scripts/training/LexicalTranslationModel.pm line 40.
>> Use of uninitialized value $a in split at
>> /home/xyz/mosesdecoder/scripts/training/LexicalTranslationModel.pm line 109.
>>
>> This never happened before and it would be very kind of anyone, who can
>> guide me on this issue.
>>
>> Thanks in advance.
>>
>>
>> --
>>
>> Regards,
>>
>> Sainik
>> Phone/WA: +91-9038141146
>>
>>
>> [image: Mailtrack]
>> 
>>  Sender
>> notified by
>> Mailtrack
>> 
>>  01/04/23,
>> 05:05:11 PM
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> https://mailman.mit.edu/mailman/listinfo/moses-support
>>
>

-- 

Regards,

Sainik
Phone/WA: +91-9038141146
___
Moses-support mailing list
Moses-support@mit.edu
https://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error: No such file or directory while opening /home/xyz/working/train/model/extract.o.sorted.gz

[Hieu Hoang]

it may be that a previous step failed, eg due to running out of disk space,
choking on bad data.

Check the logs for the previous steps

Hieu Hoang
http://statmt.org/hieu


On Wed, 4 Jan 2023 at 11:42, Sainik Kumar Mahata 
wrote:

> Dear All,
>
> I have trained my translation models using Moses Toolkit multiple times
> before, but recently, when I tried training a fresh model, I am receiving
> the following error message.
>
> No such file or directory while opening
> /home/sainik/working/train/model/extract.o.sorted.gz
> Aborted
> Exit code: 134
>
> Also, when I am looking into the log file, I can find two more
> warning/error messages.
>
> Use of uninitialized value in substitution (s///) at
> /home/xyx/mosesdecoder/scripts/training/LexicalTranslationModel.pm line 40.
> Use of uninitialized value $a in split at
> /home/xyz/mosesdecoder/scripts/training/LexicalTranslationModel.pm line 109.
>
> This never happened before and it would be very kind of anyone, who can
> guide me on this issue.
>
> Thanks in advance.
>
>
> --
>
> Regards,
>
> Sainik
> Phone/WA: +91-9038141146
>
>
> [image: Mailtrack]
> 
>  Sender
> notified by
> Mailtrack
> 
>  01/04/23,
> 05:05:11 PM
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> https://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
https://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error: No such file or directory while opening /home/xyz/working/train/model/extract.o.sorted.gz

[Sainik Kumar Mahata]

Dear All,

I have trained my translation models using Moses Toolkit multiple times
before, but recently, when I tried training a fresh model, I am receiving
the following error message.

No such file or directory while opening
/home/sainik/working/train/model/extract.o.sorted.gz
Aborted
Exit code: 134

Also, when I am looking into the log file, I can find two more
warning/error messages.

Use of uninitialized value in substitution (s///) at
/home/xyx/mosesdecoder/scripts/training/LexicalTranslationModel.pm line 40.
Use of uninitialized value $a in split at
/home/xyz/mosesdecoder/scripts/training/LexicalTranslationModel.pm line 109.

This never happened before and it would be very kind of anyone, who can
guide me on this issue.

Thanks in advance.


-- 

Regards,

Sainik
Phone/WA: +91-9038141146


[image: Mailtrack]

Sender
notified by
Mailtrack

01/04/23,
05:05:11 PM
___
Moses-support mailing list
Moses-support@mit.edu
https://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error While Compiling Moses

[Marwa Gaser]

I managed to solve the problem by updating the boost version used to 1_64_0
This resulted in a SUCCESS status!


On Sat, 30 Jan 2021 at 14:26, Marwa Gaser  wrote:

> Now when I tried to compile mosesdecoder with boost, cmph, xmlrpc, and
> irstlm using the below command on windows *wsl*:
>
>  ./bjam --with-boost=/c/Users/HP/SMT_trial1/boost_1_55_0
> --with-cmph=/c/Users/HP/SMT_trial1/cmph-2.0
> --with-irstlm=/c/Users/HP/SMT_trial1/irstlm-5.80.08
> --with-xmlrpc-c=/c/Users/HP/SMT_trial1/xmlrpc-c-1.33.14 -j7
>
> I got the error attached. Is that because of the difference in file ending
> in linux and windows?
>
> On Fri, 29 Jan 2021 at 16:10, Hieu Hoang  wrote:
>
>> the unit tests are failing, but the compile seems to have succeeded.
>>
>> I get a similar error compiling on a WSL2 environment but i know the
>> compile itself has succeeded:
>>
>> No such file or directory in
>> /mnt/c/Users/hihoan/workspace/github/mosesdecoder/test.binaryDlFBwO
>> (deleted) while resizing to 0 bytes Byte: 62
>>
>> This is because the Windows file system doesn't support some reporting
>> functionality,
>>
>> Check that your disk is not full, check that bin/moses exists. If it
>> does, then just ignore the unit test errors.
>>
>>
>> On Fri Jan 29, 21 12:12 AM, Marwa Gaser wrote:
>>
>> Dear Moses Support Team,
>>
>> I am using *wsl *on windows to run mosesdecoder for the first time.
>>
>> What I did is that I cloned the mosesdecoder repo. Then I opened the cmd
>> inside the mosesdecoder repo and ran the following command *bjam -j4 
>> *However,
>> I got the error attached in the file.
>>
>> Thank you in advance.
>>
>> Respectfully,
>> Marwa
>>
>> ___
>> Moses-support mailing 
>> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>
>> --
>> Hieu Hoanghttp://statmt.org/hieu
>>
>>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error While Compiling Moses

[Marwa Gaser]

Now when I tried to compile mosesdecoder with boost, cmph, xmlrpc, and
irstlm using the below command on windows *wsl*:

 ./bjam --with-boost=/c/Users/HP/SMT_trial1/boost_1_55_0
--with-cmph=/c/Users/HP/SMT_trial1/cmph-2.0
--with-irstlm=/c/Users/HP/SMT_trial1/irstlm-5.80.08
--with-xmlrpc-c=/c/Users/HP/SMT_trial1/xmlrpc-c-1.33.14 -j7

I got the error attached. Is that because of the difference in file ending
in linux and windows?

On Fri, 29 Jan 2021 at 16:10, Hieu Hoang  wrote:

> the unit tests are failing, but the compile seems to have succeeded.
>
> I get a similar error compiling on a WSL2 environment but i know the
> compile itself has succeeded:
>
> No such file or directory in
> /mnt/c/Users/hihoan/workspace/github/mosesdecoder/test.binaryDlFBwO
> (deleted) while resizing to 0 bytes Byte: 62
>
> This is because the Windows file system doesn't support some reporting
> functionality,
>
> Check that your disk is not full, check that bin/moses exists. If it does,
> then just ignore the unit test errors.
>
>
> On Fri Jan 29, 21 12:12 AM, Marwa Gaser wrote:
>
> Dear Moses Support Team,
>
> I am using *wsl *on windows to run mosesdecoder for the first time.
>
> What I did is that I cloned the mosesdecoder repo. Then I opened the cmd
> inside the mosesdecoder repo and ran the following command *bjam -j4 *However,
> I got the error attached in the file.
>
> Thank you in advance.
>
> Respectfully,
> Marwa
>
> ___
> Moses-support mailing 
> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
> --
> Hieu Hoanghttp://statmt.org/hieu
>
>


build.log.gz
Description: GNU Zip compressed data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error While Compiling Moses

[Hieu Hoang]

the unit tests are failing, but the compile seems to have succeeded.

I get a similar error compiling on a WSL2 environment but i know the 
compile itself has succeeded:


No such file or directory in 
/mnt/c/Users/hihoan/workspace/github/mosesdecoder/test.binaryDlFBwO 
(deleted) while resizing to 0 bytes Byte: 62


This is because the Windows file system doesn't support some reporting 
functionality,


Check that your disk is not full, check that bin/moses exists. If it 
does, then just ignore the unit test errors.



On Fri Jan 29, 21 12:12 AM, Marwa Gaser wrote:

Dear Moses Support Team,

I am using *wsl *on windows to run mosesdecoder for the first time.

What I did is that I cloned the mosesdecoder repo. Then I opened the 
cmd inside the mosesdecoder repo and ran the following command /*bjam 
-j4* /However, I got the error attached in the file.


Thank you in advance.

Respectfully,
Marwa

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


--
Hieu Hoang
http://statmt.org/hieu

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error While Compiling Moses

[Marwa Gaser]

Dear Moses Support Team,

I am using *wsl *on windows to run mosesdecoder for the first time.

What I did is that I cloned the mosesdecoder repo. Then I opened the cmd
inside the mosesdecoder repo and ran the following command *bjam -j4 *However,
I got the error attached in the file.

Thank you in advance.

Respectfully,
Marwa


build.log.gz
Description: GNU Zip compressed data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] ERROR: Filename is not absolute:

[IFEOLUWANIMI ADEBARA]

Hi Moses-Support,I'm trying to use Moses, but i'm having an error: ERROR: 
Filename is not absolute: when i run the training command. I'm certain of my 
file location, so i'm very confused about what this error means.Please can you 
help? I'm running on a linux server.
nohup: ignoring inputUsing SCRIPTS_ROOTDIR: 
/scratch/XXX/fairseq/examples/translation/mosesdecoder/scriptsUsing 
single-thread GIZAusing pigzERROR: Filename is not absolute: ~/lm/train.blm at 
/home/XXX/scratch/fairseq/examples/translation/mosesdecoder/scripts/training/train-model.perl
 line 607.~
Regards,Ife___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error when N-Gram exceeds 6

[fadi ghawanmeh]

Dears,
I am a PhD student and I am experimenting with Moses SMT. I noticed that when 
the N-Gram exceeds six, I get an error. I hope this message helps explaining 
the issue:
RROR: Failed to run'/home/fadiko/mosesdecoder/bin/moses -threads 8 
-configfiltered/moses.ini -show-weights'. 
at/home/fadiko/mosesdecoder/scripts/training/mert-moses.pl line 1775.

testing ... 

computing BLEU ... 

Use of uninitialized value$length_reference in numeric eq (==) 
at/home/fadiko/mosesdecoder/scripts/generic/multi-bleu.perl line 148. 
Do you have any idea on howto handle this issue?
Best,
Fadi AL-GHAWANMEHPhD student at the University of Lorraine___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error when installing Moses in MacOS Catalina

[Sainik Kumar Mahata]

Thanks for your response. I successfully installed moses in Ubuntu.
Installed Ubuntu on an external hard drive.

On Fri, 21 Aug 2020, 2:30 am Hieu Hoang,  wrote:

> the log file doesn't show the errors.
>
> However, there are issues with compiling on MacOS now due to diverging
> compilers. See my reply to a previous email
>https://www.mail-archive.com/moses-support@mit.edu/msg16190.html
>
> My suggestion - use Linux
>
> Hieu Hoang
> http://statmt.org/hieu
>
>
> On Thu, 20 Aug 2020 at 13:30, Sainik Kumar Mahata 
> wrote:
>
>> Respected Sir,
>>
>> I have tried many ways to install moses in my macbook but to no avail. I
>> have tried manually installing boost, installing boost through macports and
>> the latest one being, installing boost using homebrew. But, every time I
>> run into an error and the installation does not succeed. Attached herewith
>> is the log file, as suggested in the error message. Kindly have a look into
>> the same and kindly help me in this regard.
>>
>> --
>>
>> Regards,
>>
>> Mr. Sainik Kumar Mahata
>>
>> Visvesvaraya Ph.D Fellow
>>
>> Jadavpur University
>> Contact: +91-9038141146
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error when installing Moses in MacOS Catalina

[Hieu Hoang]

the log file doesn't show the errors.

However, there are issues with compiling on MacOS now due to diverging
compilers. See my reply to a previous email
   https://www.mail-archive.com/moses-support@mit.edu/msg16190.html

My suggestion - use Linux

Hieu Hoang
http://statmt.org/hieu


On Thu, 20 Aug 2020 at 13:30, Sainik Kumar Mahata 
wrote:

> Respected Sir,
>
> I have tried many ways to install moses in my macbook but to no avail. I
> have tried manually installing boost, installing boost through macports and
> the latest one being, installing boost using homebrew. But, every time I
> run into an error and the installation does not succeed. Attached herewith
> is the log file, as suggested in the error message. Kindly have a look into
> the same and kindly help me in this regard.
>
> --
>
> Regards,
>
> Mr. Sainik Kumar Mahata
>
> Visvesvaraya Ph.D Fellow
>
> Jadavpur University
> Contact: +91-9038141146
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error when installing Moses in MacOS Catalina

[Sainik Kumar Mahata]

Respected Sir,

I have tried many ways to install moses in my macbook but to no avail. I
have tried manually installing boost, installing boost through macports and
the latest one being, installing boost using homebrew. But, every time I
run into an error and the installation does not succeed. Attached herewith
is the log file, as suggested in the error message. Kindly have a look into
the same and kindly help me in this regard.

-- 

Regards,

Mr. Sainik Kumar Mahata

Visvesvaraya Ph.D Fellow

Jadavpur University
Contact: +91-9038141146


build.log.gz
Description: GNU Zip compressed data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error cleaning corpus

[IFEOLUWANIMI ADEBARA]

I'm getting the error below when I try to clean my corpus. Please what is wrong 
here?

Error:  ~/scratch/es-pt/tokenized/train.tok.pt does not exist at 
/home/fenimi/scratch/mosesdecoder/scripts/training/clean-corpus-n.perl line 58.
Regards,Ife___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error when compiling Moses with xmlrpc-c support

[Hieu Hoang]

2 things:
  1. From the error message, it seems to need the open ssl library. So I
guess you have to install openssl libssl-dev
 2. it's --with-cmph not --withcmph

Hieu Hoang
http://statmt.org/hieu


On Fri, 29 May 2020 at 12:03, Hank Kuo  wrote:

> Hello,
>
> I am encountering a compile error when building Moses with xmlrpc-c
> support.
>
> I downloaded, compiled, and installed xmlrpc-c-1.51.06 into a Ubuntu 18.04
> LTS VM.
>
> After installing xmlrpc-c-1.51.06, the command that I executed to compile
> Moses is:
>
> cd ~/mosesdecoder/
> ./bjam -a -j4 -withcmph=/usr/include
>
> See attached build.log.gz for the build output.
>
> Any help greatly appreciated.
>
> Thanks,
>
> Hank Kuo
> MITRE Corporation
> McLean, VA
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error when compiling Moses with xmlrpc-c support

[Hank Kuo]

Hello,

I am encountering a compile error when building Moses with xmlrpc-c support.

I downloaded, compiled, and installed xmlrpc-c-1.51.06 into a Ubuntu 18.04 LTS 
VM.

After installing xmlrpc-c-1.51.06, the command that I executed to compile Moses 
is:

cd ~/mosesdecoder/
./bjam -a -j4 -withcmph=/usr/include

See attached build.log.gz for the build output.

Any help greatly appreciated.

Thanks,

Hank Kuo
MITRE Corporation
McLean, VA


build.log.gz
Description: build.log.gz
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error

[Syed Basit]

Madam/Sir
 The command i executed is*  ./bjam --with-irstlm *
 The log file is also attached with this email.
THANKS & REGARDS,

SYED ABDUL BASIT

RESEARCH SCHOLAR -

Department of CS&IT
MANUU HYDERABAD India, 500032
Mobile No:9796719494
Alt email_id: sbasit.11...@manuu.edu.in


build.log.gz
Description: application/gzip
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] ERROR: Wrong format of --lm. Expected: --lm factor:order:filename //train-model.perl line 605.

[Aaron Poet]

Hi Hieu
Thanks for the reply. But, after I added the colon there, it still have
some problem in running.
*When I tried to run directly on the note of server it shows below:*

(tensorflow) aaronhan@grove2:~/Moses/working$ nohup nice
/home/aaronhan/Moses/mosesdecoder/scripts/training/train-model.perl
-root-dir train -corpus
/home/aaronhan/NMT/THUMT/corpora/preprocessed_wmt18en5maa.true -f zh -e en
-alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm
0:3:$HOME/Moses/lm/wmt18_5maaZhEn.blm.en:8 -external-bin-dir
/home/aaronhan/Moses/mosesdecoder/tools/ >& training.out &

[1] 26177

(tensorflow) aaronhan@grove2:~/Moses/working$

It produced a folder 'train' and sub-folder 'corpus' and the empty
files "en.vcb.classes
en.vcb.classes.cats  zh.vcb.classes  zh.vcb.classes.cats"

(tensorflow) aaronhan@grove2:~/Moses/working$ ls

slurm-23199.out  slurm-23201.out  *train* *train_Moses.job*

slurm-23200.out  slurm-23202.out  training.out

(tensorflow) aaronhan@grove2:~/Moses/working$ cd train/

(tensorflow) aaronhan@grove2:~/Moses/working/train$ ls

*corpus*

(tensorflow) aaronhan@grove2:~/Moses/working/train$ ls corpus/

en.vcb.classes  en.vcb.classes.cats  zh.vcb.classes  zh.vcb.classes.cats

(tensorflow) aaronhan@grove2:~/Moses/working/train$


*When I submitted this command as job file to server, it does not appear in
the job list/ doesn't to look like running.*

Do you have any clue for this, please?

Kind regards,



On Tue, Jul 30, 2019 at 8:45 AM Hieu Hoang  wrote:

>
>
> On Mon, 29 Jul 2019, 19:58 Aaron Poet,  wrote:
>
>> Hi Moses-Support
>>
>> I firstly build the lm as below:
>>
>> (base) aaronhan@grove2:~/Moses$ head -20
>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.arpa.en
>>
>> \data\
>>
>> ngram 1=553697
>>
>> ngram 2=8790963
>>
>> ngram 3=28838697
>>
>>
>> \1-grams:
>>
>> -6.993945  0
>>
>> 0-1.6929523
>>
>> -2.0611665  0
>>
>> -2.3942525  the -0.9461817
>>
>> -3.6470015  show-0.6162386
>>
>> -4.1808434  stars   -0.5792577
>>
>> -4.1823444  X   -0.53191173
>>
>> -5.0213814  Girls   -0.33374956
>>
>> -2.5064495  -   -0.6326866
>>
>> -2.5754578  a   -0.9741243
>>
>> -5.1265607  troupe  -0.3527482
>>
>> -2.2843323  of  -0.91076595
>>
>> -4.663717   talented-0.3983
>>
>> -5.2061567  topless -0.27003714
>>
>> Then, I binary it as bellow:
>>
>> /home/aaronhan/Moses/mosesdecoder/bin/build_binary
>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.arpa.en
>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>>
>> (base) aaronhan@grove2:~/Moses$ ls -lh
>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>>
>> -rw-r--r-- 1 aaronhan users 715M Jul 29 18:38
>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>>
>> when I test it, it looks working out:
>>
>> (tensorflow) aaronhan@grove2:~/Moses/working$ echo "is this an English
>> sentence ?"   \
>>
>> > | /home/aaronhan/Moses/mosesdecoder/bin/query
>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>>
>> is=23 2 -2.5766807  this=321 3 -1.0483611   an=108 3 -2.1032705
>> English=1731 2 -2.8970845   sentence=3266 3 -2.8317716  ?=815 2
>> -2.0824053  =2 3 -0.035550244   Total: -13.575124 OOV: 0
>>
>> Perplexity including OOVs:  86.9567697150713
>>
>> Perplexity excluding OOVs:  86.9567697150713
>>
>> OOVs:   0
>>
>> Tokens: 7
>>
>> Name:query  VmPeak:748300 kBVmRSS:4052 kB   RSSMax:503644 kB
>>   user:0.008  sys:0.652   CPU:0.66real:61.9797
>>
>> However, when I run the overall Moses training, it produced error as
>> below:  *Wrong format of --lm. Expected: --lm factor:order:filename*
>>
>> (tensorflow) aaronhan@grove2:~/Moses/working$ nohup nice
>> /home/aaronhan/Moses/mosesdecoder/scripts/training/train-model.perl
>> -root-dir train -corpus
>> /home/aaronhan/NMT/THUMT/corpora/preprocessed_wmt18en5maa.true -f zh -e en
>> -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm
>> 0:3$HOME/Moses/lm/wmt18_5maaZhEn.blm.en:8 -external-bin-dir
>> /home/aaronhan/Moses/mosesdecoder/tools/ >& training.out &
>>
>> [1] 23346
>>
>> You forgot a colon between 3 and $HOME
>
>>
>>  GNU nano 2.7.4
>>   File: training.out
>>
>>
>>
>> nohup: ignoring input
>>
>> Using SCRIPTS_ROOTDIR: /home/aaronhan/Moses/mosesdecoder/scripts
>>
>> Using single-thread GIZA
>>
>> using gzip
>>
>> ERROR: Wrong format of --lm. Expected: --lm factor:order:filename at
>> /home/aaronhan/Moses/mosesdecoder/scripts/training/train-model.perl line
>> 605.
>>
>>
>>
>> Can you let me know how to solve this, please?
>>
>> Kind regards,
>> Aaron
>>
>> --
>>
>> PhD student, Msc. (excellent), Bsc. (Maths)
>>
>> Google-Scholar
>>  ,
>> Presentation -slides,
>> Research-Gate 
>>
>> Google-site  Linkedin
>> 

Re: [Moses-support] ERROR: Wrong format of --lm. Expected: --lm factor:order:filename //train-model.perl line 605.

[Hieu Hoang]

you're gonna have to do some debugging yourself. Perhaps look inside
train_Moses.job and training.out

Hieu Hoang
http://statmt.org/hieu


On Tue, 30 Jul 2019 at 11:48, Aaron Poet  wrote:

> Hi Hieu
> Thanks for the reply. But, after I added the colon there, it still have
> some problem in running.
> *When I tried to run directly on the note of server it shows below:*
>
> (tensorflow) aaronhan@grove2:~/Moses/working$ nohup nice
> /home/aaronhan/Moses/mosesdecoder/scripts/training/train-model.perl
> -root-dir train -corpus
> /home/aaronhan/NMT/THUMT/corpora/preprocessed_wmt18en5maa.true -f zh -e en
> -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm
> 0:3:$HOME/Moses/lm/wmt18_5maaZhEn.blm.en:8 -external-bin-dir
> /home/aaronhan/Moses/mosesdecoder/tools/ >& training.out &
>
> [1] 26177
>
> (tensorflow) aaronhan@grove2:~/Moses/working$
>
> It produced a folder 'train' and sub-folder 'corpus' and the empty files 
> "en.vcb.classes
> en.vcb.classes.cats  zh.vcb.classes  zh.vcb.classes.cats"
>
> (tensorflow) aaronhan@grove2:~/Moses/working$ ls
>
> slurm-23199.out  slurm-23201.out  *train* *train_Moses.job*
>
> slurm-23200.out  slurm-23202.out  training.out
>
> (tensorflow) aaronhan@grove2:~/Moses/working$ cd train/
>
> (tensorflow) aaronhan@grove2:~/Moses/working/train$ ls
>
> *corpus*
>
> (tensorflow) aaronhan@grove2:~/Moses/working/train$ ls corpus/
>
> en.vcb.classes  en.vcb.classes.cats  zh.vcb.classes  zh.vcb.classes.cats
>
> (tensorflow) aaronhan@grove2:~/Moses/working/train$
>
>
> *When I submitted this command as job file to server, it does not appear
> in the job list/ doesn't to look like running.*
>
> Do you have any clue for this, please?
>
> Kind regards,
>
>
>
> On Tue, Jul 30, 2019 at 8:45 AM Hieu Hoang  wrote:
>
>>
>>
>> On Mon, 29 Jul 2019, 19:58 Aaron Poet,  wrote:
>>
>>> Hi Moses-Support
>>>
>>> I firstly build the lm as below:
>>>
>>> (base) aaronhan@grove2:~/Moses$ head -20
>>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.arpa.en
>>>
>>> \data\
>>>
>>> ngram 1=553697
>>>
>>> ngram 2=8790963
>>>
>>> ngram 3=28838697
>>>
>>>
>>> \1-grams:
>>>
>>> -6.993945  0
>>>
>>> 0-1.6929523
>>>
>>> -2.0611665  0
>>>
>>> -2.3942525  the -0.9461817
>>>
>>> -3.6470015  show-0.6162386
>>>
>>> -4.1808434  stars   -0.5792577
>>>
>>> -4.1823444  X   -0.53191173
>>>
>>> -5.0213814  Girls   -0.33374956
>>>
>>> -2.5064495  -   -0.6326866
>>>
>>> -2.5754578  a   -0.9741243
>>>
>>> -5.1265607  troupe  -0.3527482
>>>
>>> -2.2843323  of  -0.91076595
>>>
>>> -4.663717   talented-0.3983
>>>
>>> -5.2061567  topless -0.27003714
>>>
>>> Then, I binary it as bellow:
>>>
>>> /home/aaronhan/Moses/mosesdecoder/bin/build_binary
>>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.arpa.en
>>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>>>
>>> (base) aaronhan@grove2:~/Moses$ ls -lh
>>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>>>
>>> -rw-r--r-- 1 aaronhan users 715M Jul 29 18:38
>>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>>>
>>> when I test it, it looks working out:
>>>
>>> (tensorflow) aaronhan@grove2:~/Moses/working$ echo "is this an English
>>> sentence ?"   \
>>>
>>> > | /home/aaronhan/Moses/mosesdecoder/bin/query
>>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>>>
>>> is=23 2 -2.5766807  this=321 3 -1.0483611   an=108 3 -2.1032705
>>> English=1731 2 -2.8970845   sentence=3266 3 -2.8317716  ?=815 2
>>> -2.0824053  =2 3 -0.035550244   Total: -13.575124 OOV: 0
>>>
>>> Perplexity including OOVs:  86.9567697150713
>>>
>>> Perplexity excluding OOVs:  86.9567697150713
>>>
>>> OOVs:   0
>>>
>>> Tokens: 7
>>>
>>> Name:query  VmPeak:748300 kBVmRSS:4052 kB   RSSMax:503644
>>> kBuser:0.008  sys:0.652   CPU:0.66real:61.9797
>>>
>>> However, when I run the overall Moses training, it produced error as
>>> below:  *Wrong format of --lm. Expected: --lm factor:order:filename*
>>>
>>> (tensorflow) aaronhan@grove2:~/Moses/working$ nohup nice
>>> /home/aaronhan/Moses/mosesdecoder/scripts/training/train-model.perl
>>> -root-dir train -corpus
>>> /home/aaronhan/NMT/THUMT/corpora/preprocessed_wmt18en5maa.true -f zh -e en
>>> -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm
>>> 0:3$HOME/Moses/lm/wmt18_5maaZhEn.blm.en:8 -external-bin-dir
>>> /home/aaronhan/Moses/mosesdecoder/tools/ >& training.out &
>>>
>>> [1] 23346
>>>
>>> You forgot a colon between 3 and $HOME
>>
>>>
>>>  GNU nano 2.7.4
>>>   File: training.out
>>>
>>>
>>>
>>> nohup: ignoring input
>>>
>>> Using SCRIPTS_ROOTDIR: /home/aaronhan/Moses/mosesdecoder/scripts
>>>
>>> Using single-thread GIZA
>>>
>>> using gzip
>>>
>>> ERROR: Wrong format of --lm. Expected: --lm factor:order:filename at
>>> /home/aaronhan/Moses/mosesdecoder/scripts/training/train-model.perl line
>>> 605.
>>>
>>>
>>>
>>> Can you let me know how to solve this, please?
>>>
>>> Kind regards,
>>> 

Re: [Moses-support] ERROR: Wrong format of --lm. Expected: --lm factor:order:filename //train-model.perl line 605.

[Hieu Hoang]

On Mon, 29 Jul 2019, 19:58 Aaron Poet,  wrote:

> Hi Moses-Support
>
> I firstly build the lm as below:
>
> (base) aaronhan@grove2:~/Moses$ head -20
> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.arpa.en
>
> \data\
>
> ngram 1=553697
>
> ngram 2=8790963
>
> ngram 3=28838697
>
>
> \1-grams:
>
> -6.993945  0
>
> 0-1.6929523
>
> -2.0611665  0
>
> -2.3942525  the -0.9461817
>
> -3.6470015  show-0.6162386
>
> -4.1808434  stars   -0.5792577
>
> -4.1823444  X   -0.53191173
>
> -5.0213814  Girls   -0.33374956
>
> -2.5064495  -   -0.6326866
>
> -2.5754578  a   -0.9741243
>
> -5.1265607  troupe  -0.3527482
>
> -2.2843323  of  -0.91076595
>
> -4.663717   talented-0.3983
>
> -5.2061567  topless -0.27003714
>
> Then, I binary it as bellow:
>
> /home/aaronhan/Moses/mosesdecoder/bin/build_binary
> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.arpa.en
> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>
> (base) aaronhan@grove2:~/Moses$ ls -lh
> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>
> -rw-r--r-- 1 aaronhan users 715M Jul 29 18:38
> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>
> when I test it, it looks working out:
>
> (tensorflow) aaronhan@grove2:~/Moses/working$ echo "is this an English
> sentence ?"   \
>
> > | /home/aaronhan/Moses/mosesdecoder/bin/query
> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>
> is=23 2 -2.5766807  this=321 3 -1.0483611   an=108 3 -2.1032705
> English=1731 2 -2.8970845   sentence=3266 3 -2.8317716  ?=815 2
> -2.0824053  =2 3 -0.035550244   Total: -13.575124 OOV: 0
>
> Perplexity including OOVs:  86.9567697150713
>
> Perplexity excluding OOVs:  86.9567697150713
>
> OOVs:   0
>
> Tokens: 7
>
> Name:query  VmPeak:748300 kBVmRSS:4052 kB   RSSMax:503644 kB
>   user:0.008  sys:0.652   CPU:0.66real:61.9797
>
> However, when I run the overall Moses training, it produced error as
> below:  *Wrong format of --lm. Expected: --lm factor:order:filename*
>
> (tensorflow) aaronhan@grove2:~/Moses/working$ nohup nice
> /home/aaronhan/Moses/mosesdecoder/scripts/training/train-model.perl
> -root-dir train -corpus
> /home/aaronhan/NMT/THUMT/corpora/preprocessed_wmt18en5maa.true -f zh -e en
> -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm
> 0:3$HOME/Moses/lm/wmt18_5maaZhEn.blm.en:8 -external-bin-dir
> /home/aaronhan/Moses/mosesdecoder/tools/ >& training.out &
>
> [1] 23346
>
> You forgot a colon between 3 and $HOME

>
>  GNU nano 2.7.4
> File: training.out
>
>
>
> nohup: ignoring input
>
> Using SCRIPTS_ROOTDIR: /home/aaronhan/Moses/mosesdecoder/scripts
>
> Using single-thread GIZA
>
> using gzip
>
> ERROR: Wrong format of --lm. Expected: --lm factor:order:filename at
> /home/aaronhan/Moses/mosesdecoder/scripts/training/train-model.perl line
> 605.
>
>
>
> Can you let me know how to solve this, please?
>
> Kind regards,
> Aaron
>
> --
>
> PhD student, Msc. (excellent), Bsc. (Maths)
>
> Google-Scholar
>  ,
> Presentation -slides,
> Research-Gate 
>
> Google-site  Linkedin
> ,
>
> ADAPT  ResearchCentre &
>
> School of Computing, DCU,
>
> Dublin, Ireland
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] ERROR: Wrong format of --lm. Expected: --lm factor:order:filename //train-model.perl line 605.

[Aaron Poet]

Hi Moses-Support

I firstly build the lm as below:

(base) aaronhan@grove2:~/Moses$ head -20
/home/aaronhan/Moses/lm/wmt18_5maaZhEn.arpa.en

\data\

ngram 1=553697

ngram 2=8790963

ngram 3=28838697


\1-grams:

-6.993945  0

0-1.6929523

-2.0611665  0

-2.3942525  the -0.9461817

-3.6470015  show-0.6162386

-4.1808434  stars   -0.5792577

-4.1823444  X   -0.53191173

-5.0213814  Girls   -0.33374956

-2.5064495  -   -0.6326866

-2.5754578  a   -0.9741243

-5.1265607  troupe  -0.3527482

-2.2843323  of  -0.91076595

-4.663717   talented-0.3983

-5.2061567  topless -0.27003714

Then, I binary it as bellow:

/home/aaronhan/Moses/mosesdecoder/bin/build_binary
/home/aaronhan/Moses/lm/wmt18_5maaZhEn.arpa.en
/home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en

(base) aaronhan@grove2:~/Moses$ ls -lh
/home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en

-rw-r--r-- 1 aaronhan users 715M Jul 29 18:38
/home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en

when I test it, it looks working out:

(tensorflow) aaronhan@grove2:~/Moses/working$ echo "is this an English
sentence ?"   \

> | /home/aaronhan/Moses/mosesdecoder/bin/query
/home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en

is=23 2 -2.5766807  this=321 3 -1.0483611   an=108 3 -2.1032705
English=1731 2 -2.8970845   sentence=3266 3 -2.8317716  ?=815 2
-2.0824053  =2 3 -0.035550244   Total: -13.575124 OOV: 0

Perplexity including OOVs:  86.9567697150713

Perplexity excluding OOVs:  86.9567697150713

OOVs:   0

Tokens: 7

Name:query  VmPeak:748300 kBVmRSS:4052 kB   RSSMax:503644 kB
user:0.008  sys:0.652   CPU:0.66real:61.9797

However, when I run the overall Moses training, it produced error as below:
 *Wrong format of --lm. Expected: --lm factor:order:filename*

(tensorflow) aaronhan@grove2:~/Moses/working$ nohup nice
/home/aaronhan/Moses/mosesdecoder/scripts/training/train-model.perl
-root-dir train -corpus
/home/aaronhan/NMT/THUMT/corpora/preprocessed_wmt18en5maa.true -f zh -e en
-alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm
0:3$HOME/Moses/lm/wmt18_5maaZhEn.blm.en:8 -external-bin-dir
/home/aaronhan/Moses/mosesdecoder/tools/ >& training.out &

[1] 23346


 GNU nano 2.7.4
File: training.out



nohup: ignoring input

Using SCRIPTS_ROOTDIR: /home/aaronhan/Moses/mosesdecoder/scripts

Using single-thread GIZA

using gzip

ERROR: Wrong format of --lm. Expected: --lm factor:order:filename at
/home/aaronhan/Moses/mosesdecoder/scripts/training/train-model.perl line
605.



Can you let me know how to solve this, please?

Kind regards,
Aaron

-- 

PhD student, Msc. (excellent), Bsc. (Maths)

Google-Scholar  ,
Presentation -slides,
Research-Gate 

Google-site  Linkedin
,

ADAPT  ResearchCentre &

School of Computing, DCU,

Dublin, Ireland
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] ERROR: you need to define GENERAL:settings

[Usman Mohy ud Din]

Hi Everyone,

i am trying to execute tree-to-string model using EMS, however i got an
error *"ERROR: you need to define GENERAL:settings"* on *parse-input* step.
Phrase-based models have been executed successfully on this environment,
then what can be reasons of the error.
input-parser command is

input-parser = "$moses-script-dir/tokenizer/deescape-special-chars.perl \
| python
$moses-script-dir/training/wrappers/parse-en-stanford.py --stanford
$corenlp --java /home/administrator/jdk1.8/bin/java \
| java -jar $maltparser \
| python
$moses-script-dir/training/wrappers/conll2mosesxml.py"

where

maltparser = $home-dir/maltparser/maltparser-1.9.2.jar
corenlp = $home-dir/stanford-corenlp-3.9.2


Kind Regards,

*Usman*
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error

[melaku muka]

 ./bjam --with-irstlm=/home/tsegay/moses/irstlm/trunk
--with-giza=/home/tsegay/moses/tools


build.log.gz
Description: application/gzip
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error in installing moses

[Dawit Tessema]

Pls refer the attachment


build.log.gz
Description: GNU Zip compressed data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error

[Hieu Hoang]

it looks like the xmlrpc-c library has not been compiled correctly.

If you don't need this library (required by moses server and moses2) then
add the argument
--no-xmlrpc-c
to the bjam command when you compile moses

Hieu Hoang
http://moses-smt.org/


2018-04-20 2:04 GMT+01:00 Gedour Abdoullah :

>
> ./bjam --with-srilm=/home/kader/Bureau/workMoses/srilm
> --with-irstlm=/home/kader/Bureau/workMoses/irstlm/trunk
> --with-giza=/home/kader/Bureau/workMoses/tools
> --
> Sidi Abdellahi Abdelkader
> Master en Systeme d'infomation et nouvelle technologie
> Faculté des Sciences Économiques et de Gestion de Sfax (FSEG)
> Département Informatique
> Université de Sfax
> Tunisie
> Tél : 00216 52 303 538
> Site web : http://sidiabdelghader7.wixsite.com/sidi
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error

[Gedour Abdoullah]

./bjam --with-srilm=/home/kader/Bureau/workMoses/srilm
--with-irstlm=/home/kader/Bureau/workMoses/irstlm/trunk
--with-giza=/home/kader/Bureau/workMoses/tools
-- 
Sidi Abdellahi Abdelkader
Master en Systeme d'infomation et nouvelle technologie
Faculté des Sciences Économiques et de Gestion de Sfax (FSEG)
Département Informatique
Université de Sfax
Tunisie
Tél : 00216 52 303 538
Site web : http://sidiabdelghader7.wixsite.com/sidi


build.log.gz
Description: application/gzip
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error installing moses

[Kenneth Heafield]

Moses doesn't use NPLM option, so there's no point in compiling with
--with-nplm .  For what it's worth, it's meant to compile against this
fork https://github.com/kpu/nplm and NPLM has since changed. 


On 03/17/2018 07:52 PM, krishna chaitanya gudipati wrote:
> Hi,
> I am getting some error while installing Moses. I have attached the
> build log to this mail.
> Please help me in resolving this error.
>
> Exact command used : ./bjam -j12
> --with-cmph=/home/development/krishna1997/smtgec2017/software/cmph-2.0
> --with-nplm=/home/development/krishna1997/smtgec2017/software/nplm
> --max-kenlm-order=9
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error installing moses

[krishna chaitanya gudipati]

Hi,
I am getting some error while installing Moses. I have attached the build
log to this mail.
Please help me in resolving this error.

Exact command used : ./bjam -j12
--with-cmph=/home/development/krishna1997/smtgec2017/software/cmph-2.0
--with-nplm=/home/development/krishna1997/smtgec2017/software/nplm
--max-kenlm-order=9


build.log.gz
Description: application/gzip
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error during training moses

[Philipp Koehn]

Hi,

something went wrong during the extraction step.

Try to run it by itself, and check for error messages. Several things are
possible. Previous steps may have failed, it may have run out of memory,
etc.

-phi

On Wed, Feb 21, 2018 at 1:06 AM, Kamal Deep Garg 
wrote:

> Dear sir
>
> During training it stops. Exception is attached here.
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error on Moses

[Redentor Periabras]

Im using Ubuntu 17.10, thank you for your response.

On Thu, Jan 11, 2018 at 5:27 PM Hieu Hoang  wrote:

> You can probably ignore that error.
>
> There seems to be a problem with a unit test but that may be spurious.
> What OS/distribution are you running?
>
> On 11/01/18 08:15, Redentor Periabras wrote:
>
> Hi, I got this error as I run:
>
> *`./bjam --with-boost=~/Projects/mosesdecoder/boost_1_64_0 -j8`*
>
> Your response is very much appreciated. Thank you!
>
>
> --
> *Redentor Periabras*
> BS Computer Science
> Department of Computer Science
> College of Computer and Information Sciences
> Polytechnic University of the Philippines, Manila
>
>
> ___
> Moses-support mailing 
> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> --
> Hieu Hoanghttp://moses-smt.org/
>
>

-- 
*Redentor Periabras*
BS Computer Science
Department of Computer Science
College of Computer and Information Sciences
Polytechnic University of the Philippines, Manila
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error on Moses

[Hieu Hoang]

You can probably ignore that error.

There seems to be a problem with a unit test but that may be spurious. 
What OS/distribution are you running?



On 11/01/18 08:15, Redentor Periabras wrote:

Hi, I got this error as I run:

*`./bjam --with-boost=~/Projects/mosesdecoder/boost_1_64_0 -j8`*

Your response is very much appreciated. Thank you!


--
*Redentor Periabras*
BS Computer Science
Department of Computer Science
College of Computer and Information Sciences
Polytechnic University of the Philippines, Manila


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


--
Hieu Hoang
http://moses-smt.org/

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error on Moses

[Redentor Periabras]

Hi, I got this error as I run:

*`./bjam --with-boost=~/Projects/mosesdecoder/boost_1_64_0 -j8`*

Your response is very much appreciated. Thank you!


-- 
*Redentor Periabras*
BS Computer Science
Department of Computer Science
College of Computer and Information Sciences
Polytechnic University of the Philippines, Manila


build.log.gz
Description: application/gzip
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error: Cannot allocate memory

[Yasmin Moslem]

Hello!

I got an error when running Moses or Moses2 without binarizing (Command:
~/mosesdecoder/bin/moses -f ~/working/mert-work/moses.ini)

>> Cannot allocate memory for 262144 bytes in malloc
>> Aborted (core dumped)


Binarizing solved the issue for Moses (however for Moses2, I could not
binarize; will send another message about that.)

My questions is: should I increase machine resources? I have the following
setup:

Amazon EC2 instance with the following specifications: OS: Ubuntu Server
16.04 LTS, type: c3.large, vCPUs: 2, Memory (GiB): 3.75, Instance Storage
(GB): 2 x 16 (SSD).


Many thanks,
Yasmin
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error while giving lattice with epsilon as input to moses.

[Sanket Gandhare]

I am trying to build a Spoken MT system where I am first using Kaldi for
recognizing text and then then moses for translation.

kaldi is giving me output as confusion networks(with epsilons) which needs
to be feeded as input to moses in PLF fomat. Hence, I am converting the
confusion network to  PLF format.

Chris's link : http://www.cdec-decoder.org/concepts/lattices.html says plf
format accepts epsilon.

((('*EPS*',1,1),),(('there',0.9747331,1),('it',0.02526694,1)
,),(('*EPS*',1,1),),(('was',0.9987841,1),('is',0.00121591,1)
,),(('*EPS*',1,1),),(('a',1,1),),(('*EPS*',1,1),),(('
beautiful',1,1),),(('*EPS*',1,1),),(('fish',0.8245394,1),('
bench',0.1042382,1),('fetch',0.03937767,1),('fetched',0.
02650477,1),('edge',0.005339933,1),),(('*EPS*',1,1),),)



On 8 September 2017 at 01:22, Kenneth Heafield  wrote:

> It seems nobody implemented epsilons.  You're welcome to implement them.
>
>
> On 09/07/2017 09:40 PM, Sanket Gandhare wrote:
> > I am trying to give input to moses as word lattice having epsilons as
> > well, '*EPS*'. but it is giving this result :
> >
> > terminate called after throwing an instance of 'util::Exception'
> >   what():  moses/TranslationOptionCollectionLattice.cpp:48 in
> > Moses::TranslationOptionCollectionLattice::
> TranslationOptionCollectionLattice(const
> > ttasksptr&, const Moses::WordLattice&) threw util::Exception because
> > `word.IsEpsilon()'.
> > Epsilon not supported
> > Aborted (core dumped)
> >
> > Can anyone help?
> >
> >
> >
> > ___
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error while giving lattice with epsilon as input to moses.

[Kenneth Heafield]

It seems nobody implemented epsilons.  You're welcome to implement them. 


On 09/07/2017 09:40 PM, Sanket Gandhare wrote:
> I am trying to give input to moses as word lattice having epsilons as
> well, '*EPS*'. but it is giving this result :
>
> terminate called after throwing an instance of 'util::Exception'
>   what():  moses/TranslationOptionCollectionLattice.cpp:48 in
> Moses::TranslationOptionCollectionLattice::TranslationOptionCollectionLattice(const
> ttasksptr&, const Moses::WordLattice&) threw util::Exception because
> `word.IsEpsilon()'.
> Epsilon not supported
> Aborted (core dumped)
>
> Can anyone help?
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error while giving lattice with epsilon as input to moses.

[Sanket Gandhare]

I am trying to give input to moses as word lattice having epsilons as well,
'*EPS*'. but it is giving this result :

terminate called after throwing an instance of 'util::Exception'
  what():  moses/TranslationOptionCollectionLattice.cpp:48 in
Moses::TranslationOptionCollectionLattice::TranslationOptionCollectionLattice(const
ttasksptr&, const Moses::WordLattice&) threw util::Exception because
`word.IsEpsilon()'.
Epsilon not supported
Aborted (core dumped)

Can anyone help?
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] error in Run Moses for the first time

[Arezoo Arjomand]

hi, MT is runned successfully in "Run Moses for the first time" according to 
"http://www.statmt.org/moses/?n=Development.GetStarted";. but it has an error 
when running both command in "chart decoder".
arezoo1@arezoo1-X450CC:/mosesdecoder/sample-models$ /mosesdecoder/bin/moses -f 
/mosesdecoder/sample-models/string-to-tree/moses.ini< 
/mosesdecoder/sample-models/tree-to-tree/in.xml> /home/arezoo1/out.ttt
Defined parameters (per moses.ini or switch):
    config: /mosesdecoder/sample-models/string-to-tree/moses.ini 
    cube-pruning-pop-limit: 1000 
    feature: KENLM name=LM factor=0 order=3 num-features=1 
path=lm/europarl.srilm.gz WordPenalty UnknownWordPenalty PhraseDictionaryMemory 
input-factor=0 output-factor=0 path=string-to-tree/rule-table num-features=1 
table-limit=20 
    input-factors: 0 
    inputtype: 3 
    mapping: 0 T 0 
    max-chart-span: 20 1000 
    non-terminals: X S 
    search-algorithm: 3 
    translation-details: translation-details.log 
    weight: WordPenalty0= 0 LM= 0.5 PhraseDictionaryMemory0= 0.5 
line=KENLM name=LM factor=0 order=3 num-features=1 path=lm/europarl.srilm.gz
Loading the LM will be faster if you build a binary file.
Reading lm/europarl.srilm.gz
5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
**The ARPA file is missing .  Substituting log10 probability -100.000.
**
FeatureFunction: LM start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 2 end: 2
line=PhraseDictionaryMemory input-factor=0 output-factor=0 
path=string-to-tree/rule-table num-features=1 table-limit=20
FeatureFunction: PhraseDictionaryMemory0 start: 3 end: 3
Loading LM
Loading WordPenalty0
Loading UnknownWordPenalty0
Loading PhraseDictionaryMemory0
Start loading text phrase table. Moses format : [0.752] seconds
Reading string-to-tree/rule-table
5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

max-chart-span: 20
Created input-output object : [0.753] seconds
Exception: ./moses/OutputCollector.h:65 in 
Moses::OutputCollector::OutputCollector(std::__cxx11::string, 
std::__cxx11::string) threw util::Exception because `!m_outStream->good()'.
Failed to open output filetranslation-details.log

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error while Training at Step 5

[Hieu Hoang]

in the directory
   /Users/dcs/Moses/tools/mosesdecoder/bin
do you see a file 'extract'. If not, you must make sure moses compiles
successfully. Follow these instructions
   http://www.statmt.org/moses/?n=Development.GetStarted

Hieu Hoang
http://moses-smt.org/


On 21 August 2017 at 01:52, Aileen Joan Vicente  wrote:

> Thank you for your advice.
>
> I installed coreutils via macports but still get the same result:
>
> (5) extract phrases @ Mon Aug 21 08:45:08 PHT 2017
>
> MAX 7 0 0
>
> /Users/dcs/Moses/tools/mosesdecoder/scripts/generic/extract-parallel.perl
> 4 gsplit "gsort" 
> /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract
> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en
> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk
> /Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final
> /Users/dcs/Desktop/Corpus/training/model/extract 7 --GZOutput
>
> Executing: 
> /Users/dcs/Moses/tools/mosesdecoder/scripts/generic/extract-parallel.perl
> 4 gsplit "gsort" 
> /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract
> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en
> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk
> /Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final
> /Users/dcs/Desktop/Corpus/training/model/extract 7 --GZOutput
>
> Started Mon Aug 21 08:45:08 2017
>
> using gzip
>
> isBSDSplit=0
>
> Executing: mkdir -p /Users/dcs/Desktop/Corpus/training/model/tmp.24478;
> ls -l /Users/dcs/Desktop/Corpus/training/model/tmp.24478
>
> total=2942 line-per-split=736
>
> gsplit -d -l 736 -a 7 /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/target.gsplit -d -l
> 736 -a 7 /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/source.gsplit -d -l
> 736 -a 7 /Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/align.sh:
> /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such file
> or directory
>
> sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such
> file or directory
>
> sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such
> file or directory
>
> sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such
> file or directory
>
> merging extract / extract.inv
>
> gunzip -c 
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.000.gz
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.001.gz
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.002.gz
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.003.gz  |
> LC_ALL=C gsort -T /Users/dcs/Desktop/Corpus/training/model/tmp.24478
> 2>> /dev/stderr | gzip -c > /Users/dcs/Desktop/Corpus/
> training/model/extract.sorted.gz 2>> /dev/stderr
>
> gunzip -c 
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.000.inv.gz
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.001.inv.gz
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.002.inv.gz
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.003.inv.gz
> | LC_ALL=C gsort -T /Users/dcs/Desktop/Corpus/training/model/tmp.24478
> 2>> /dev/stderr | gzip -c > /Users/dcs/Desktop/Corpus/
> training/model/extract.inv.sorted.gz 2>> /dev/stderr
>
> gunzip: can't stat: 
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.000.inv.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.000.inv.gz.gz):
> No such file or directory
>
> gunzip: can't stat: 
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.001.inv.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.001.inv.gz.gz):
> No such file or directory
>
> gunzip: can't stat: 
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.002.inv.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.002.inv.gz.gz):
> No such file or directory
>
> gunzip: can't stat: 
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.003.inv.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.003.inv.gz.gz):
> No such file or directory
>
> gunzip: can't stat: 
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.000.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.000.gz.gz):
> No such file or directory
>
> gunzip: can't stat: 
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.001.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.001.gz.gz):
> No such file or directory
>
> gunzip: can't stat: 
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.002.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.002.gz.gz):
> No such file or directory
>
> gunzip: can't stat: 
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.003.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/

Re: [Moses-support] Error while Training at Step 5

[Hieu Hoang]

it looks like you're using Mac OSX. You must make sure that the GNU 
version of sort and split, not the BSD version. Moses has been tested on 
OSX using MacPorts to install sort and split so you should install MacPorts


    https://www.macports.org/



On 20/08/2017 13:22, Aileen Joan Vicente wrote:




Hi! I ran training.perl, succeeded but found an empty phrase table. 
I googled around to look for answer and some guy advised to check 
the logs. I wonder if the problem is caused by this error in Step 5. 
Please help.


Using SCRIPTS_ROOTDIR: /Users/dcs/Moses/tools/mosesdecoder/scripts
using gzip
(5) extract phrases @ Sun Aug 20 19:26:48 PHT 2017
MAX 7 0 0
/Users/dcs/Moses/tools/mosesdecoder/scripts/generic/extract-parallel.perl 
4 split "sort    " 
/Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract 
/Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en 
/Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk 
/Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final 
/Users/dcs/Desktop/Corpus/training/model/extract 7 --GZOutput
Executing: 
/Users/dcs/Moses/tools/mosesdecoder/scripts/generic/extract-parallel.perl 
4 split "sort    " 
/Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract 
/Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en 
/Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk 
/Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final 
/Users/dcs/Desktop/Corpus/training/model/extract 7 --GZOutput

Started Sun Aug 20 19:26:48 2017
using gzip
split: illegal option -- -
usage: split [-a sufflen] [-b byte_count] [-l line_count] [-p pattern]
[file [prefix]]
isBSDSplit=1
Executing: mkdir -p 
/Users/dcs/Desktop/Corpus/training/model/tmp.798; ls -l 
/Users/dcs/Desktop/Corpus/training/model/tmp.798

total=2942 line-per-split=736
split  -l 736 -a 7 
/Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/target.split  -l 
736 -a 7 /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/source.split  -l 
736 -a 7 
/Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/align.sh: 
/Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such 
file or directory
sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No 
such file or directory
sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No 
such file or directory
sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No 
such file or directory

merging extract / extract.inv
gunzip -c 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaa.gz 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aab.gz 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aac.gz 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aad.gz  
| LC_ALL=C sort     -T 
/Users/dcs/Desktop/Corpus/training/model/tmp.798 2>> /dev/stderr | 
gzip -c > /Users/dcs/Desktop/Corpus/training/model/extract.sorted.gz 
2>> /dev/stderr
gunzip -c 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaa.inv.gz 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aab.inv.gz 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aac.inv.gz 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aad.inv.gz 
| LC_ALL=C sort     -T 
/Users/dcs/Desktop/Corpus/training/model/tmp.798 2>> /dev/stderr | 
gzip -c > 
/Users/dcs/Desktop/Corpus/training/model/extract.inv.sorted.gz 2>> 
/dev/stderr
gunzip: can't stat: 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaa.inv.gz 
(/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaa.inv.gz.gz): 
No such file or directory
gunzip: can't stat: 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aab.inv.gz 
(/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aab.inv.gz.gz): 
No such file or directory
gunzip: can't stat: 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aac.inv.gz 
(/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aac.inv.gz.gz): 
No such file or directory
gunzip: can't stat: 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aad.inv.gz 
(/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aad.inv.gz.gz): 
No such file or directory
gunzip: can't stat: 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaa.gz 
(/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaa.gz.gz): 
No such file or directory
gunzip: can't stat: 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aab.gz 
(/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aab.gz.gz): 
No such file or directory
gunzip: can't stat: 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aac.gz 
(/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aac.gz.gz): 
No such file or directory
gunzip: can't stat: 
/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaa

[Moses-support] Error while Training at Step 5

[Aileen Joan Vicente]

>> 
>> Hi! I ran training.perl, succeeded but found an empty phrase table. I 
>> googled around to look for answer and some guy advised to check the logs. I 
>> wonder if the problem is caused by this error in Step 5. Please help.
>> 
>> Using SCRIPTS_ROOTDIR: /Users/dcs/Moses/tools/mosesdecoder/scripts
>> using gzip 
>> (5) extract phrases @ Sun Aug 20 19:26:48 PHT 2017
>> MAX 7 0 0
>> /Users/dcs/Moses/tools/mosesdecoder/scripts/generic/extract-parallel.perl 4 
>> split "sort" /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract 
>> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en 
>> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk 
>> /Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final 
>> /Users/dcs/Desktop/Corpus/training/model/extract 7 --GZOutput 
>> Executing: 
>> /Users/dcs/Moses/tools/mosesdecoder/scripts/generic/extract-parallel.perl 4 
>> split "sort" /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract 
>> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en 
>> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk 
>> /Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final 
>> /Users/dcs/Desktop/Corpus/training/model/extract 7 --GZOutput 
>> Started Sun Aug 20 19:26:48 2017
>> using gzip 
>> split: illegal option -- -
>> usage: split [-a sufflen] [-b byte_count] [-l line_count] [-p pattern]
>>  [file [prefix]]
>> isBSDSplit=1 
>> Executing: mkdir -p /Users/dcs/Desktop/Corpus/training/model/tmp.798; ls -l 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798 
>> total=2942 line-per-split=736 
>> split  -l 736 -a 7 /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/target.split  -l 736 -a 7 
>> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/source.split  -l 736 -a 7 
>> /Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/align.sh: 
>> /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such file or 
>> directory
>> sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such file 
>> or directory
>> sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such file 
>> or directory
>> sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such file 
>> or directory
>> merging extract / extract.inv
>> gunzip -c 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaa.gz 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aab.gz 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aac.gz 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aad.gz  | 
>> LC_ALL=C sort -T /Users/dcs/Desktop/Corpus/training/model/tmp.798 2>> 
>> /dev/stderr | gzip -c > 
>> /Users/dcs/Desktop/Corpus/training/model/extract.sorted.gz 2>> /dev/stderr 
>> gunzip -c 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaa.inv.gz 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aab.inv.gz 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aac.inv.gz 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aad.inv.gz  | 
>> LC_ALL=C sort -T /Users/dcs/Desktop/Corpus/training/model/tmp.798 2>> 
>> /dev/stderr | gzip -c > 
>> /Users/dcs/Desktop/Corpus/training/model/extract.inv.sorted.gz 2>> 
>> /dev/stderr 
>> gunzip: can't stat: 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaa.inv.gz 
>> (/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaa.inv.gz.gz):
>>  No such file or directory
>> gunzip: can't stat: 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aab.inv.gz 
>> (/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aab.inv.gz.gz):
>>  No such file or directory
>> gunzip: can't stat: 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aac.inv.gz 
>> (/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aac.inv.gz.gz):
>>  No such file or directory
>> gunzip: can't stat: 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aad.inv.gz 
>> (/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aad.inv.gz.gz):
>>  No such file or directory
>> gunzip: can't stat: 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaa.gz 
>> (/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaa.gz.gz): No 
>> such file or directory
>> gunzip: can't stat: 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aab.gz 
>> (/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aab.gz.gz): No 
>> such file or directory
>> gunzip: can't stat: 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aac.gz 
>> (/Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aac.gz.gz): No 
>> such file or directory
>> gunzip: can't stat: 
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aad.gz 
>> (/

[Moses-support] error

[Arezoo Arjomand]

Hi,I want to install moses on my PC. I follow "Getting Started with Moses" 
toturial to compile it. but in " make -f 
contrib/Makefiles/install-dependencies.gmake" step the following error is 
appeared. how can i fix the error?
Thank you 


In file included from lmContainer.cpp:23:0:
/usr/include/c++/6/cstdlib:75:25: fatal error: stdlib.h: No such file or 
directory
 #include_next 
 ^
compilation terminated.
Makefile:725: recipe for target 'lmContainer.lo' failed
make[3]: *** [lmContainer.lo] Error 1
make[3]: *** Waiting for unfinished jobs
In file included from /usr/include/c++/6/ext/string_conversions.h:41:0,
 from /usr/include/c++/6/bits/basic_string.h:5417,
 from /usr/include/c++/6/string:52,
 from /usr/include/c++/6/bits/locale_classes.h:40,
 from /usr/include/c++/6/bits/ios_base.h:41,
 from /usr/include/c++/6/ios:42,
 from /usr/include/c++/6/ostream:38,
 from /usr/include/c++/6/iostream:39,
 from htable.cpp:26:
/usr/include/c++/6/cstdlib:75:25: fatal error: stdlib.h: No such file or 
directory
 #include_next 
 ^
compilation terminated.
Makefile:725: recipe for target 'htable.lo' failed
make[3]: *** [htable.lo] Error 1
In file included from /usr/include/c++/6/ext/string_conversions.h:41:0,
 from /usr/include/c++/6/bits/basic_string.h:5417,
 from /usr/include/c++/6/string:52,
 from /usr/include/c++/6/bits/locale_classes.h:40,
 from /usr/include/c++/6/bits/ios_base.h:41,
 from /usr/include/c++/6/iomanip:40,
 from dictionary.cpp:25:
/usr/include/c++/6/cstdlib:75:25: fatal error: stdlib.h: No such file or 
directory
 #include_next 
 ^
compilation terminated.
Makefile:725: recipe for target 'dictionary.lo' failed
make[3]: *** [dictionary.lo] Error 1
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I.. -DTRACE_LEVEL=1 
-DMY_ASSERT_FLAG -g -O2 -MT cmd.lo -MD -MP -MF .deps/cmd.Tpo -c cmd.c -o cmd.o 
>/dev/null 2>&1
mv -f .deps/cmd.Tpo .deps/cmd.Plo
make[3]: Leaving directory '/mosesdecoder/build/irstlm/irstlm-5.80.08/trunk/src'
Makefile:407: recipe for target 'all-recursive' failed
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory '/mosesdecoder/build/irstlm/irstlm-5.80.08/trunk'
Makefile:339: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/mosesdecoder/build/irstlm/irstlm-5.80.08/trunk'
contrib/Makefiles/install-dependencies.gmake:88: recipe for target 
'/mosesdecoder/opt/irstlm-5.80.08/bin/build-lm.sh' failed
make: *** [/mosesdecoder/opt/irstlm-5.80.08/bin/build-lm.sh] Error 2

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error display japanese from output of mecab, please me!

[Ngô Thị Vinh]

I have solve my problem with guide:
http://qiita.com/junpow/items/0a7d13addc0acad10606
if anyone have them same my problem can see that.

$ tar zxfv mecab-ipadic-2.7.0-.tar.gz
$ cd mecab-ipadic-2.7.0-
$ nkf -w - overwrite *. csv
$ nkf -w - overwrite * .def

Edit dicrc

- config-charset = EUC-JP -->replace by Config-charset = UTF-8

Installation

$ ./configure --with-charset=utf8
$ make
$ sudo make install



2017-06-22 14:55 GMT+07:00 Ngô Thị Vinh :

> Thank Hieu Hoang!
>
> 2017-06-22 14:05 GMT+07:00 Hieu Hoang :
>
>> You have to find out what encoding the output is in. This page says MeCab
>> output in EUC-JP
>>https://forum.koohii.com/thread-8952.html
>> But running
>>iconv -f EUCJP  -t UTF-8 temp_rs
>> doesn't seem to fix it.
>>
>> You should ask the Mecab developers to get a definite answer
>>
>> Hieu Hoang
>> http://moses-smt.org/
>>
>>
>> On 22 June 2017 at 05:44, Ngô Thị Vinh  wrote:
>>
>>> Hi all,
>>> Does anyone has ever done experiments with Japanese-English and using
>>> mecab for tokenization japanese?
>>> I test mecab with a simple file temp from terminal such as:
>>> mecab -O wakati temp -o temp_rs
>>> but temp_rs is error with font, although I have installed font for
>>> Japanese?
>>>  Please help me to solve the problem if you know.
>>> Thank you!
>>>
>>> --
>>> *Ngô Thị Vinh*
>>> Faculty of Electronics and Communications,
>>> Thai Nguyen University of Information and Communication Technology
>>> (ICTU).
>>> TEL: 0987 706 830
>>> Email:   *ntv...@ictu.edu.vn *
>>>
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>
>
> --
> *Ngô Thị Vinh*
> Faculty of Electronics and Communications,
> Thai Nguyen University of Information and Communication Technology (ICTU).
> TEL: 0987 706 830
> Email:   *ntv...@ictu.edu.vn *
>



-- 
*Ngô Thị Vinh*
Faculty of Electronics and Communications,
Thai Nguyen University of Information and Communication Technology (ICTU).
TEL: 0987 706 830
Email:   *ntv...@ictu.edu.vn *
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error display japanese from output of mecab, please me!

[Ngô Thị Vinh]

Thank Hieu Hoang!

2017-06-22 14:05 GMT+07:00 Hieu Hoang :

> You have to find out what encoding the output is in. This page says MeCab
> output in EUC-JP
>https://forum.koohii.com/thread-8952.html
> But running
>iconv -f EUCJP  -t UTF-8 temp_rs
> doesn't seem to fix it.
>
> You should ask the Mecab developers to get a definite answer
>
> Hieu Hoang
> http://moses-smt.org/
>
>
> On 22 June 2017 at 05:44, Ngô Thị Vinh  wrote:
>
>> Hi all,
>> Does anyone has ever done experiments with Japanese-English and using
>> mecab for tokenization japanese?
>> I test mecab with a simple file temp from terminal such as:
>> mecab -O wakati temp -o temp_rs
>> but temp_rs is error with font, although I have installed font for
>> Japanese?
>>  Please help me to solve the problem if you know.
>> Thank you!
>>
>> --
>> *Ngô Thị Vinh*
>> Faculty of Electronics and Communications,
>> Thai Nguyen University of Information and Communication Technology (ICTU).
>> TEL: 0987 706 830
>> Email:   *ntv...@ictu.edu.vn *
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>


-- 
*Ngô Thị Vinh*
Faculty of Electronics and Communications,
Thai Nguyen University of Information and Communication Technology (ICTU).
TEL: 0987 706 830
Email:   *ntv...@ictu.edu.vn *
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error display japanese from output of mecab, please me!

[Hieu Hoang]

You have to find out what encoding the output is in. This page says MeCab
output in EUC-JP
   https://forum.koohii.com/thread-8952.html
But running
   iconv -f EUCJP  -t UTF-8 temp_rs
doesn't seem to fix it.

You should ask the Mecab developers to get a definite answer

Hieu Hoang
http://moses-smt.org/


On 22 June 2017 at 05:44, Ngô Thị Vinh  wrote:

> Hi all,
> Does anyone has ever done experiments with Japanese-English and using
> mecab for tokenization japanese?
> I test mecab with a simple file temp from terminal such as:
> mecab -O wakati temp -o temp_rs
> but temp_rs is error with font, although I have installed font for
> Japanese?
>  Please help me to solve the problem if you know.
> Thank you!
>
> --
> *Ngô Thị Vinh*
> Faculty of Electronics and Communications,
> Thai Nguyen University of Information and Communication Technology (ICTU).
> TEL: 0987 706 830
> Email:   *ntv...@ictu.edu.vn *
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error display japanese from output of mecab, please me!

[Ngô Thị Vinh]

Hi all,
Does anyone has ever done experiments with Japanese-English and using mecab
for tokenization japanese?
I test mecab with a simple file temp from terminal such as:
mecab -O wakati temp -o temp_rs
but temp_rs is error with font, although I have installed font for Japanese?
 Please help me to solve the problem if you know.
Thank you!

-- 
*Ngô Thị Vinh*
Faculty of Electronics and Communications,
Thai Nguyen University of Information and Communication Technology (ICTU).
TEL: 0987 706 830
Email:   *ntv...@ictu.edu.vn *


temp
Description: Binary data


temp_rs
Description: Binary data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error installing Moses

[Hieu Hoang]

looks like you don't have the boost libraries installed on your system.

And Redhat usually have out of date boost which won't work with moses. 
It's better to compile your own. Instructions are here:


   http://www.statmt.org/moses/?n=Development.GetStarted


On 19/06/2017 16:07, Vudi, Sai Mahit (Ext) wrote:


Hello Team,

Trying to deploy Moses on our system(Properties: MAPR(Bigdata 
cluster)version-5.1.0 ,RHEL6.9,gcc-4.8.2,).It’s throwing a 
failure.Please help us on resolving this issue.


Thank you,

Sai Mahit,

Data Engineer.



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


--
Hieu Hoang
http://moses-smt.org/

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error when building with --with-srilm

[Hieu Hoang]

make sure you have compiled srilm successfully and the library files it 
generates are in


   /home/nikhilesh/srilm/lib/i686-m64


On 19/06/2017 18:31, Varun Goel wrote:

Hello,
I am trying to build Moses on my Ubuntu 16.04, x86_64 machine.
I can build successfully using the normal compilation.
However, on using --with-srilm the build fails. Following is the EXACT 
command used for compilation :

./compile.sh --with-srilm=/home/nikhilesh/srilm

PFA build.log.gz


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


--
Hieu Hoang
http://moses-smt.org/

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error when building with --with-srilm

[Varun Goel]

Hello,
I am trying to build Moses on my Ubuntu 16.04, x86_64 machine.
I can build successfully using the normal compilation.
However, on using --with-srilm the build fails. Following is the EXACT
command used for compilation :
./compile.sh --with-srilm=/home/nikhilesh/srilm

PFA build.log.gz


build.log.gz
Description: GNU Zip compressed data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error installing Moses

[Vudi, Sai Mahit (Ext)]

./bjam is the command I used to execute.

From: Vudi, Sai Mahit (Ext)
Sent: Monday, June 19, 2017 10:07 AM
To: 'moses-support@mit.edu'
Subject: Error installing Moses

Hello Team,
Trying to deploy Moses on our system(Properties: MAPR(Bigdata 
cluster)version-5.1.0 ,RHEL6.9,gcc-4.8.2,).It's throwing a failure.Please help 
us on resolving this issue.


Thank you,
Sai Mahit,
Data Engineer.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error installing Moses

[Vudi, Sai Mahit (Ext)]

Hello Team,
Trying to deploy Moses on our system(Properties: MAPR(Bigdata 
cluster)version-5.1.0 ,RHEL6.9,gcc-4.8.2,).It's throwing a failure.Please help 
us on resolving this issue.


Thank you,
Sai Mahit,
Data Engineer.


build.log.gz
Description: build.log.gz
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error while testing data

[Pritesh Ranjan]

Dear Sir/Ma'am,

I am trying to run moses decoder for english to hindi translation but moses is
unable to load the phrase table as it runs out of memory and process
exits.Also when I tried to use the binarized form of  phrase table I was
getting the following error.

>~/smt/mosesdecoder/bin/moses -f ~/set_eng_to_hindi/tune/mert-work/moses.ini
test.out.hi.text

   "Loading PhrasePenalty0Loading LexicalReordering0
Loading table into memory...Exception: vector::_M_range_check
"
I have also attached the moses.ini file.

It would be very helpful if someone can help.

 # MERT optimized configuration
  # decoder /home/iit/mosesdecoder/bin/moses
  # BLEU 0.00292786 on dev /home/iit/corpora/parallel/dev.en.clean.txt.txt
  # We were before running iteration 2
  # finished Tue Jun  6 12:55:43 IST 2017
  ### MOSES CONFIG FILE ###
  #

  # input factors
  [input-factors]
  0

  # mapping steps
  [mapping]
  0 T 0

 [distortion-limit]
 6

 # feature functions
[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryCompact name=TranslationModel0 num-features=4
path=/home/iit/corpora/parallel/binarised-model/phrase-table.binphr
input-factor=0  output-factor=0
LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-
  msd-bidirectional-fe-allff input-factor=0
output-factor=0 path=/home

/iit/corpora/parallel/binarised-model/reordering-table.binlexr
Distortion
IRSTLM name=LM0 factor=0 path=/home/iit/corpora/parallel
   /lm/corpora_train.lm.arpa.binary order=3

# dense weights for feature functions
[weight]

LexicalReordering0= 0.0652174 0.0652174 0.0652174 0.0652174 0.0652174
0.0652174
Distortion0= 0.0652174
LM0= 0.108696
WordPenalty0= -0.217391
PhrasePenalty0= 0.0434783
TranslationModel0= 0.0434783 0.0434783 0.0434783 0.0434783
 UnknownWordPenalty0= 1


Thanks and Regards

_Pritesh Ranjan


moses.ini
Description: Binary data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error during tuning

[Hieu Hoang]

Use absolute paths for everything

On 2 Jun 2017 7:07 a.m., "huda noor"  wrote:

> yes the compilation of moses was successful. I ran the command below in
> the working directory made in mosesdecoder directory.
>
> nohup nice ../scripts/training/mert-moses.pl ../corpus2/dev/dev.clean.en
> ../corpus2/dev/dev.clean.tok.ur ../bin/moses train/model/moses.ini
> --mertdir ../bin/ &> mert.out &
>
> mert.out file is attached
>
>
> On Fri, Jun 2, 2017 at 6:56 AM, Hieu Hoang  wrote:
>
>> what was the exact command you ran and was the compilation of moses
>> successful?
>>
>> * Looking for MT/NLP opportunities *
>> Hieu Hoang
>> http://moses-smt.org/
>>
>>
>> On 31 May 2017 at 13:48, huda noor  wrote:
>>
>>> Dear Moses Experts!!!
>>> we are building Statistical Machine Translation system for English-Urdu
>>> language pair and we get an error after tuning i.e. no moses.ini file build.
>>>
>>> exec: /home/huda/mosesdecoder/working/mert-work/extractor.sh
>>> Executing: /home/huda/mosesdecoder/working/mert-work/extractor.sh >
>>> extract.out 2> extract.err
>>> Exit code: 127
>>> ERROR: Failed to run 
>>> '/home/huda/mosesdecoder/working/mert-work/extractor.sh'.
>>> at ../scripts/training/mert-moses.pl line 1748.
>>>
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error during tuning

[Hieu Hoang]

what was the exact command you ran and was the compilation of moses
successful?

* Looking for MT/NLP opportunities *
Hieu Hoang
http://moses-smt.org/


On 31 May 2017 at 13:48, huda noor  wrote:

> Dear Moses Experts!!!
> we are building Statistical Machine Translation system for English-Urdu
> language pair and we get an error after tuning i.e. no moses.ini file build.
>
> exec: /home/huda/mosesdecoder/working/mert-work/extractor.sh
> Executing: /home/huda/mosesdecoder/working/mert-work/extractor.sh >
> extract.out 2> extract.err
> Exit code: 127
> ERROR: Failed to run '/home/huda/mosesdecoder/working/mert-work/extractor.sh'.
> at ../scripts/training/mert-moses.pl line 1748.
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error during tuning

[huda noor]

Dear Moses Experts!!!
we are building Statistical Machine Translation system for English-Urdu
language pair and we get an error after tuning i.e. no moses.ini file build.

exec: /home/huda/mosesdecoder/working/mert-work/extractor.sh
Executing: /home/huda/mosesdecoder/working/mert-work/extractor.sh >
extract.out 2> extract.err
Exit code: 127
ERROR: Failed to run
'/home/huda/mosesdecoder/working/mert-work/extractor.sh'. at
../scripts/training/mert-moses.pl line 1748.


extract.err
Description: Binary data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error in language model training.

[Philipp Koehn]

Hi,

this is ac common error when the data is "odd", such as
heavily duplicated segments, reduced vocabulary, etc.

To avoid the error, just add the switch "--discount_fallback"
when invoking lmplz.

-phi

On Mon, May 15, 2017 at 7:44 AM, rmogla  wrote:

> Hello,
>
> While using KenLM for language model training for Hindi I got this error.
>
>
> mosesdecoder/lm/builder/adjust_counts.cc:61 in void
> lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
> lm::builder::DiscountConfig&) threw BadDiscountException because
> `discounts_[i].amount[j] < 0.0 || discounts_[i].amount[j] > j'.
>
> ERROR: 1-gram discount out of range for adjusted count 2: -0.0625
> Aborted (core dumped)
>
>
> Please suggest how to remove this error.
>
> Thanking in advance.
> Best Regards
>
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error in language model training.

[rmogla]

Hello,

While using KenLM for language model training for Hindi I got this error.


mosesdecoder/lm/builder/adjust_counts.cc:61 in void
lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
lm::builder::DiscountConfig&) threw BadDiscountException because
`discounts_[i].amount[j] < 0.0 || discounts_[i].amount[j] > j'.

ERROR: 1-gram discount out of range for adjusted count 2: -0.0625
Aborted (core dumped)


Please suggest how to remove this error.

Thanking in advance.
Best Regards
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] error Can't read /phrase-model/moses.ini.

[Philipp Koehn]

Hi,

it is very unlikely that "/phrase-model/moses.ini"
is the correct full path to the configuration file.

Could you check that?

-phi

On Sun, Apr 9, 2017 at 1:40 AM, Kirti Agrawal 
wrote:

> I have just installed moses , I am a beginner . While trying the sample I
> am getting error Can't read /phrase-model/moses.ini.
>
>
> Command I am sending is :
> echo 'das ist ein kleines haus' | 
> /Users/kagrawal/Desktop/tools/moses/bin/moses
> -f /phrase-model/moses.ini < phrase-model/in > out  from my sample-models
> folder
>
> I am using mac.  Thanks for support .I am using moses for my project.
>
> Thanks
> Kirti
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] error when tuning

[dai xin]

Dear Moses experts,

I got an error when I am doing tuning. It is from the extractor.sh.

It shows Binary write mode is not selected. Anyone has some ideas about
that?

Thanks in advance and hoping for reply.

Best regards

Xin


extract.err
Description: Binary data


extractor.sh
Description: Bourne shell script
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] error Can't read /phrase-model/moses.ini.

[Kirti Agrawal]

I have just installed moses , I am a beginner . While trying the sample I
am getting error Can't read /phrase-model/moses.ini.


Command I am sending is :
echo 'das ist ein kleines haus' | /Users/kagrawal/Desktop/tools/moses/bin/moses
-f /phrase-model/moses.ini < phrase-model/in > out  from my sample-models
folder

I am using mac.  Thanks for support .I am using moses for my project.

Thanks
Kirti
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] ERROR: Lexical reordering scoring failed

[Philipp Koehn]

Hi,

can you check if the line

inhaled corticosteroids ||| inhalative Kortikoichaemic ||| ischämische |||
mono other

really occurs in the extract.o file?

It should only have source / target / reordering status, not 4 entries.

Did something go wrong when the extract file was created (out of disk
space?).

Do you have "|||" in the corpus?

Did you try re-running extract & build reorder?

-phi


On Tue, Mar 28, 2017 at 10:03 AM, Anna Kupś  wrote:

>
> Hi,
>
> I got the following error while using Experiment Management System:
>
> ==> TRAINING_build-reordering.3 <==
>
> PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/
> sbin:/bin:/snap/bin:/snap/bin"
> cd /root/working/experiments
> echo 'starting at '`date`' on '`hostname`
> mkdir -p /root/working/experiments/training
>
> /root/mosesdecoder/scripts/training/train-model.perl -mgiza -mgiza-cpus 8
> -dont-zip -first-step 7 -last-step 7 -external-bin-dir
> /root/mosesdecoder/tools -f en -e de -alignment grow-diag-final-and
> -reordering msd-bidirectional-fe -extract-file 
> /root/working/experiments/model/extract.2
> -reordering-table /root/working/experiments/model/reordering-table.3
>
> echo 'finished at '`date`
> touch /root/working/experiments/steps/3/TRAINING_build-reordering.3.DONE
>
> ==> TRAINING_build-reordering.3.INFO <==
> lexicalized-reordering = msd-bidirectional-fe
> INPUT = USED /root/working/experiments/model/extract.2
> # reuse run 2 for TRAINING:extract-phrases
>
> ==> TRAINING_build-reordering.3.STDERR <==
> Using SCRIPTS_ROOTDIR: /root/mosesdecoder/scripts
> using gzip
> (7) learn reordering model @ Tue Mar 28 12:19:55 CEST 2017
> (7.1) [no factors] learn reordering model @ Tue Mar 28 12:19:55 CEST 2017
> (7.2) building tables @ Tue Mar 28 12:19:55 CEST 2017
> Executing: /root/mosesdecoder/scripts/../bin/lexical-reordering-score
> /root/working/experiments/model/extract.2.o.sorted.gz 0.5
> /root/working/experiments/model/reordering-table.3. --model "wbe msd
> wbe-msd-bidirectional-fe"
> Lexical Reordering Scorer
> scores lexical reordering models of several types (hierarchical,
> phrase-based and word-based-extraction
>
> ==> TRAINING_build-reordering.3.STDOUT <==
> starting at Di 28. Mär 12:19:54 CEST 2017 on machine-VirtualBox
>
> ==> TRAINING_build-reordering.3.STDERR <==
> terminate called after throwing an instance of 'FileFormatException'
>   what():  phrase-extract/lexical-reordering/score.cpp:260 in void
> split_line(const StringPiece&, StringPiece&, StringPiece&, StringPiece&,
> StringPiece&, StringPiece&, float&) threw FileFormatException because
> `errIndex == next.data()'.
> Invalid extract file format: inhaled corticosteroids ||| inhalative
> Kortikoichaemic ||| ischämische ||| mono other
> Aborted (core dumped)
> Exit code: 134
> *ERROR: Lexical reordering scoring failed at
> /root/mosesdecoder/scripts/training/train-model.perl line 1924.*
>
> Does someone know how to get rid of that problem?
>
> Thanks in advance and best regards,
> Anna
>
> --
> Anna Kupś
> +48 782 823 850 <+48%20782%20823%20850> | www.linkedin.com/in/annakups
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error in mteval-v13a.pl

[Dingyuan Wang]

Hi,

The OS is Debian 9 (stretch/testing), and perl is v5.24.1.


2017年3月29日 上午12:53，"liling tan" 寫道：

Interesting, I've just took a look at the perluniprop documentation
carefully, generic \p{Line_Break} didn't exist but Ubuntu and Mac OS let me
slip pass without error/warning. I supposed  \p{Line_Break} automatically
goes to  \p{Line_Break: Alphabetic} or \p{Line_Break: Ambiguous}

Maybe the perluniprop handling is different for different machines/OS/perl
versions.

\p{Line_Break: Hyphen} might be closer to \p{Hyphen} than
alphbetic/ambiguous \p{Line_Break}


I think we can change it to \p{Line_Break: Hyphen} safely.But first,
@Dingyuan, please do tell us your OS and perl versions and see whether
it's just that the newer perl version handles the  \p{Line_Break}
automatically while the older perl doesn't


On Wed, Mar 29, 2017 at 12:36 AM, liling tan  wrote:

> Hi Dingyuan, Hieu,
>
> Thanks for highlighting the issue.
>
> The deprecation warning from mteval has been there since early 2015 on
> https://www.mail-archive.com/moses-support@mit.edu/msg12057.html
>
> The fix at https://github.com/moses-smt/mosesdecoder/pull/170 was
> adhering to unicode annex on 2016-06-01.
>
> Actually \p{Line_Break:Hyphen} is a little more restricting than 
> \p{Line_Break}.
> Technically, if the script didn't break with  \p{Line_Break:Hyphen}  ,
> the superset  \p{Line_Break} shouldn't break. Perl unicode properties
> documentation has the detailed explanations: http://perl11.or
> g/pod/perluniprops.html
>
> The current mteval-v13a.pl works fine on:
> - Ubuntu 16.04.01 with perl v5.22.1
> - Mac OS Sierra with perl v5.18.2
>
> @Dingyuan, a couple of short questions.
>
> (i) Which OS are you using? Which version of the OS?
>
> (ii) What is your perl version? use command "perl -v"
>
> That'll help us to know which OS and/or perl version caused that error.
>
> Regards,
> Liling
>
>
> On Tue, Mar 28, 2017 at 8:52 PM, Hieu Hoang  wrote:
>
>> If you find that {Line_Break:Hyphen} works, please consider checking it
>> in.
>>
>> These compatibility issues are difficult to debug alone and depends on
>> the exact perl/OS version you're running. Your fix will add a little to the
>> body of knowledge
>>
>> * Looking for MT/NLP opportunities *
>> Hieu Hoang
>> http://moses-smt.org/
>>
>>
>> On 28 March 2017 at 02:48, Dingyuan Wang  wrote:
>>
>>> -BEGIN PGP SIGNED MESSAGE-
>>> Hash: SHA256
>>>
>>> Recently mteval-v13a.pl stopped working, printing:
>>>
>>> Can't find Unicode property definition "Line_Break" in regex; marked
>>> by <-- HERE in m/\p{Line_Break} <-- HERE \p{Zl}/ at
>>> /home/gumble/software/moses/scripts/generic/mteval-v13a.pl line 953.
>>>
>>> I see this commit
>>> >> 2cbc6b708f55a9ecd3
>>> 
>>> >.
>>> I found that changing this to \p{Line_Break: Hyphen} worked. Is this
>>> the equivalent of \p{Hyphen}?
>>>
>>> - --
>>> Dingyuan Wang
>>> -BEGIN PGP SIGNATURE-
>>>
>>> iQEzBAEBCAAdFiEEjE4PLbCEqfvlC0rjs+TYPj8+X9wFAljZwNYACgkQs+TYPj8+
>>> X9xShggAhSjPEEYXsiRPT9wVljRV7XjBmexe/E7EKzl9b/PEnuxlSNSrz/0Estr5
>>> 8/H4s+lwKdv9xx1jTxOGOVkToiVC95QkuppXX3WS+BCDjajE8fqWc2Y0IhUWRaqf
>>> PAhhotEZmoWAhQC/qVM7lILf29N9OhQ2FStQH9rn+LpD2dkSZweZ0XGJ+CFpCdaP
>>> VA7XPWJCJZeEBUsBqrSxl1Cwzr+KQ4pw/NFP6yxJ+smmTkUSyp2FfYCtvalx/L0d
>>> 2UZ1fiujzco7NHeJW/0ZYwsb+NNMuM7CljBMhQAWIN+D0f6Wz1/bHH8jhFyxUw0B
>>> +/hN/chrAmYX+Kz2j/MKc7eXZuPtmA==
>>> =C+jT
>>> -END PGP SIGNATURE-
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>
>>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error in mteval-v13a.pl

[liling tan]

Interesting, I've just took a look at the perluniprop documentation
carefully, generic \p{Line_Break} didn't exist but Ubuntu and Mac OS let me
slip pass without error/warning. I supposed  \p{Line_Break} automatically
goes to  \p{Line_Break: Alphabetic} or \p{Line_Break: Ambiguous}

Maybe the perluniprop handling is different for different machines/OS/perl
versions.

\p{Line_Break: Hyphen} might be closer to \p{Hyphen} than
alphbetic/ambiguous \p{Line_Break}


I think we can change it to \p{Line_Break: Hyphen} safely.But first,
@Dingyuan, please do tell us your OS and perl versions and see whether
it's just that the newer perl version handles the  \p{Line_Break}
automatically while the older perl doesn't


On Wed, Mar 29, 2017 at 12:36 AM, liling tan  wrote:

> Hi Dingyuan, Hieu,
>
> Thanks for highlighting the issue.
>
> The deprecation warning from mteval has been there since early 2015 on
> https://www.mail-archive.com/moses-support@mit.edu/msg12057.html
>
> The fix at https://github.com/moses-smt/mosesdecoder/pull/170 was
> adhering to unicode annex on 2016-06-01.
>
> Actually \p{Line_Break:Hyphen} is a little more restricting than 
> \p{Line_Break}.
> Technically, if the script didn't break with  \p{Line_Break:Hyphen}  ,
> the superset  \p{Line_Break} shouldn't break. Perl unicode properties
> documentation has the detailed explanations: http://perl11.
> org/pod/perluniprops.html
>
> The current mteval-v13a.pl works fine on:
> - Ubuntu 16.04.01 with perl v5.22.1
> - Mac OS Sierra with perl v5.18.2
>
> @Dingyuan, a couple of short questions.
>
> (i) Which OS are you using? Which version of the OS?
>
> (ii) What is your perl version? use command "perl -v"
>
> That'll help us to know which OS and/or perl version caused that error.
>
> Regards,
> Liling
>
>
> On Tue, Mar 28, 2017 at 8:52 PM, Hieu Hoang  wrote:
>
>> If you find that {Line_Break:Hyphen} works, please consider checking it
>> in.
>>
>> These compatibility issues are difficult to debug alone and depends on
>> the exact perl/OS version you're running. Your fix will add a little to the
>> body of knowledge
>>
>> * Looking for MT/NLP opportunities *
>> Hieu Hoang
>> http://moses-smt.org/
>>
>>
>> On 28 March 2017 at 02:48, Dingyuan Wang  wrote:
>>
>>> -BEGIN PGP SIGNED MESSAGE-
>>> Hash: SHA256
>>>
>>> Recently mteval-v13a.pl stopped working, printing:
>>>
>>> Can't find Unicode property definition "Line_Break" in regex; marked
>>> by <-- HERE in m/\p{Line_Break} <-- HERE \p{Zl}/ at
>>> /home/gumble/software/moses/scripts/generic/mteval-v13a.pl line 953.
>>>
>>> I see this commit
>>> >> 2cbc6b708f55a9ecd3
>>> 
>>> >.
>>> I found that changing this to \p{Line_Break: Hyphen} worked. Is this
>>> the equivalent of \p{Hyphen}?
>>>
>>> - --
>>> Dingyuan Wang
>>> -BEGIN PGP SIGNATURE-
>>>
>>> iQEzBAEBCAAdFiEEjE4PLbCEqfvlC0rjs+TYPj8+X9wFAljZwNYACgkQs+TYPj8+
>>> X9xShggAhSjPEEYXsiRPT9wVljRV7XjBmexe/E7EKzl9b/PEnuxlSNSrz/0Estr5
>>> 8/H4s+lwKdv9xx1jTxOGOVkToiVC95QkuppXX3WS+BCDjajE8fqWc2Y0IhUWRaqf
>>> PAhhotEZmoWAhQC/qVM7lILf29N9OhQ2FStQH9rn+LpD2dkSZweZ0XGJ+CFpCdaP
>>> VA7XPWJCJZeEBUsBqrSxl1Cwzr+KQ4pw/NFP6yxJ+smmTkUSyp2FfYCtvalx/L0d
>>> 2UZ1fiujzco7NHeJW/0ZYwsb+NNMuM7CljBMhQAWIN+D0f6Wz1/bHH8jhFyxUw0B
>>> +/hN/chrAmYX+Kz2j/MKc7eXZuPtmA==
>>> =C+jT
>>> -END PGP SIGNATURE-
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>
>>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error in mteval-v13a.pl

[liling tan]

Hi Dingyuan, Hieu,

Thanks for highlighting the issue.

The deprecation warning from mteval has been there since early 2015 on
https://www.mail-archive.com/moses-support@mit.edu/msg12057.html

The fix at https://github.com/moses-smt/mosesdecoder/pull/170 was adhering
to unicode annex on 2016-06-01.

Actually \p{Line_Break:Hyphen} is a little more restricting than
\p{Line_Break}.
Technically, if the script didn't break with  \p{Line_Break:Hyphen}  , the
superset  \p{Line_Break} shouldn't break. Perl unicode properties
documentation has the detailed explanations:
http://perl11.org/pod/perluniprops.html

The current mteval-v13a.pl works fine on:
- Ubuntu 16.04.01 with perl v5.22.1
- Mac OS Sierra with perl v5.18.2

@Dingyuan, a couple of short questions.

(i) Which OS are you using? Which version of the OS?

(ii) What is your perl version? use command "perl -v"

That'll help us to know which OS and/or perl version caused that error.

Regards,
Liling


On Tue, Mar 28, 2017 at 8:52 PM, Hieu Hoang  wrote:

> If you find that {Line_Break:Hyphen} works, please consider checking it in.
>
> These compatibility issues are difficult to debug alone and depends on the
> exact perl/OS version you're running. Your fix will add a little to the
> body of knowledge
>
> * Looking for MT/NLP opportunities *
> Hieu Hoang
> http://moses-smt.org/
>
>
> On 28 March 2017 at 02:48, Dingyuan Wang  wrote:
>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA256
>>
>> Recently mteval-v13a.pl stopped working, printing:
>>
>> Can't find Unicode property definition "Line_Break" in regex; marked
>> by <-- HERE in m/\p{Line_Break} <-- HERE \p{Zl}/ at
>> /home/gumble/software/moses/scripts/generic/mteval-v13a.pl line 953.
>>
>> I see this commit
>> > 2cbc6b708f55a9ecd3
>> 
>> >.
>> I found that changing this to \p{Line_Break: Hyphen} worked. Is this
>> the equivalent of \p{Hyphen}?
>>
>> - --
>> Dingyuan Wang
>> -BEGIN PGP SIGNATURE-
>>
>> iQEzBAEBCAAdFiEEjE4PLbCEqfvlC0rjs+TYPj8+X9wFAljZwNYACgkQs+TYPj8+
>> X9xShggAhSjPEEYXsiRPT9wVljRV7XjBmexe/E7EKzl9b/PEnuxlSNSrz/0Estr5
>> 8/H4s+lwKdv9xx1jTxOGOVkToiVC95QkuppXX3WS+BCDjajE8fqWc2Y0IhUWRaqf
>> PAhhotEZmoWAhQC/qVM7lILf29N9OhQ2FStQH9rn+LpD2dkSZweZ0XGJ+CFpCdaP
>> VA7XPWJCJZeEBUsBqrSxl1Cwzr+KQ4pw/NFP6yxJ+smmTkUSyp2FfYCtvalx/L0d
>> 2UZ1fiujzco7NHeJW/0ZYwsb+NNMuM7CljBMhQAWIN+D0f6Wz1/bHH8jhFyxUw0B
>> +/hN/chrAmYX+Kz2j/MKc7eXZuPtmA==
>> =C+jT
>> -END PGP SIGNATURE-
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] ERROR: Lexical reordering scoring failed

[Anna Kupś]

Hi,

I got the following error while using Experiment Management System:

==> TRAINING_build-reordering.3 <==

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/snap/bin"
cd /root/working/experiments
echo 'starting at '`date`' on '`hostname`
mkdir -p /root/working/experiments/training

/root/mosesdecoder/scripts/training/train-model.perl -mgiza -mgiza-cpus 8
-dont-zip -first-step 7 -last-step 7 -external-bin-dir
/root/mosesdecoder/tools -f en -e de -alignment grow-diag-final-and
-reordering msd-bidirectional-fe -extract-file
/root/working/experiments/model/extract.2 -reordering-table
/root/working/experiments/model/reordering-table.3

echo 'finished at '`date`
touch /root/working/experiments/steps/3/TRAINING_build-reordering.3.DONE

==> TRAINING_build-reordering.3.INFO <==
lexicalized-reordering = msd-bidirectional-fe
INPUT = USED /root/working/experiments/model/extract.2
# reuse run 2 for TRAINING:extract-phrases

==> TRAINING_build-reordering.3.STDERR <==
Using SCRIPTS_ROOTDIR: /root/mosesdecoder/scripts
using gzip
(7) learn reordering model @ Tue Mar 28 12:19:55 CEST 2017
(7.1) [no factors] learn reordering model @ Tue Mar 28 12:19:55 CEST 2017
(7.2) building tables @ Tue Mar 28 12:19:55 CEST 2017
Executing: /root/mosesdecoder/scripts/../bin/lexical-reordering-score
/root/working/experiments/model/extract.2.o.sorted.gz 0.5
/root/working/experiments/model/reordering-table.3. --model "wbe msd
wbe-msd-bidirectional-fe"
Lexical Reordering Scorer
scores lexical reordering models of several types (hierarchical,
phrase-based and word-based-extraction

==> TRAINING_build-reordering.3.STDOUT <==
starting at Di 28. Mär 12:19:54 CEST 2017 on machine-VirtualBox

==> TRAINING_build-reordering.3.STDERR <==
terminate called after throwing an instance of 'FileFormatException'
  what():  phrase-extract/lexical-reordering/score.cpp:260 in void
split_line(const StringPiece&, StringPiece&, StringPiece&, StringPiece&,
StringPiece&, StringPiece&, float&) threw FileFormatException because
`errIndex == next.data()'.
Invalid extract file format: inhaled corticosteroids ||| inhalative
Kortikoichaemic ||| ischämische ||| mono other
Aborted (core dumped)
Exit code: 134
*ERROR: Lexical reordering scoring failed at
/root/mosesdecoder/scripts/training/train-model.perl line 1924.*

Does someone know how to get rid of that problem?

Thanks in advance and best regards,
Anna

-- 
Anna Kupś
+48 782 823 850 | www.linkedin.com/in/annakups
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error in mteval-v13a.pl

[Hieu Hoang]

If you find that {Line_Break:Hyphen} works, please consider checking it in.

These compatibility issues are difficult to debug alone and depends on the
exact perl/OS version you're running. Your fix will add a little to the
body of knowledge

* Looking for MT/NLP opportunities *
Hieu Hoang
http://moses-smt.org/


On 28 March 2017 at 02:48, Dingyuan Wang  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Recently mteval-v13a.pl stopped working, printing:
>
> Can't find Unicode property definition "Line_Break" in regex; marked
> by <-- HERE in m/\p{Line_Break} <-- HERE \p{Zl}/ at
> /home/gumble/software/moses/scripts/generic/mteval-v13a.pl line 953.
>
> I see this commit
>  2cbc6b708f55a9ecd3>.
> I found that changing this to \p{Line_Break: Hyphen} worked. Is this
> the equivalent of \p{Hyphen}?
>
> - --
> Dingyuan Wang
> -BEGIN PGP SIGNATURE-
>
> iQEzBAEBCAAdFiEEjE4PLbCEqfvlC0rjs+TYPj8+X9wFAljZwNYACgkQs+TYPj8+
> X9xShggAhSjPEEYXsiRPT9wVljRV7XjBmexe/E7EKzl9b/PEnuxlSNSrz/0Estr5
> 8/H4s+lwKdv9xx1jTxOGOVkToiVC95QkuppXX3WS+BCDjajE8fqWc2Y0IhUWRaqf
> PAhhotEZmoWAhQC/qVM7lILf29N9OhQ2FStQH9rn+LpD2dkSZweZ0XGJ+CFpCdaP
> VA7XPWJCJZeEBUsBqrSxl1Cwzr+KQ4pw/NFP6yxJ+smmTkUSyp2FfYCtvalx/L0d
> 2UZ1fiujzco7NHeJW/0ZYwsb+NNMuM7CljBMhQAWIN+D0f6Wz1/bHH8jhFyxUw0B
> +/hN/chrAmYX+Kz2j/MKc7eXZuPtmA==
> =C+jT
> -END PGP SIGNATURE-
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] ERROR: Lexical reordering scoring failed

[Anna Kupś]

Hi,

I got the following error while using Experiment Management System:

==> TRAINING_build-reordering.3 <==

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/snap/bin"
cd /root/working/experiments
echo 'starting at '`date`' on '`hostname`
mkdir -p /root/working/experiments/training

/root/mosesdecoder/scripts/training/train-model.perl -mgiza -mgiza-cpus 8
-dont-zip -first-step 7 -last-step 7 -external-bin-dir
/root/mosesdecoder/tools -f en -e de -alignment grow-diag-final-and
-reordering msd-bidirectional-fe -extract-file
/root/working/experiments/model/extract.2 -reordering-table
/root/working/experiments/model/reordering-table.3

echo 'finished at '`date`
touch /root/working/experiments/steps/3/TRAINING_build-reordering.3.DONE

==> TRAINING_build-reordering.3.INFO <==
lexicalized-reordering = msd-bidirectional-fe
INPUT = USED /root/working/experiments/model/extract.2
# reuse run 2 for TRAINING:extract-phrases

==> TRAINING_build-reordering.3.STDERR <==
Using SCRIPTS_ROOTDIR: /root/mosesdecoder/scripts
using gzip
(7) learn reordering model @ Tue Mar 28 12:19:55 CEST 2017
(7.1) [no factors] learn reordering model @ Tue Mar 28 12:19:55 CEST 2017
(7.2) building tables @ Tue Mar 28 12:19:55 CEST 2017
Executing: /root/mosesdecoder/scripts/../bin/lexical-reordering-score
/root/working/experiments/model/extract.2.o.sorted.gz 0.5
/root/working/experiments/model/reordering-table.3. --model "wbe msd
wbe-msd-bidirectional-fe"
Lexical Reordering Scorer
scores lexical reordering models of several types (hierarchical,
phrase-based and word-based-extraction

==> TRAINING_build-reordering.3.STDOUT <==
starting at Di 28. Mär 12:19:54 CEST 2017 on machine-VirtualBox

==> TRAINING_build-reordering.3.STDERR <==
terminate called after throwing an instance of 'FileFormatException'
  what():  phrase-extract/lexical-reordering/score.cpp:260 in void
split_line(const StringPiece&, StringPiece&, StringPiece&, StringPiece&,
StringPiece&, StringPiece&, float&) threw FileFormatException because
`errIndex == next.data()'.
Invalid extract file format: inhaled corticosteroids ||| inhalative
Kortikoichaemic ||| ischämische ||| mono other
Aborted (core dumped)
Exit code: 134
*ERROR: Lexical reordering scoring failed at
/root/mosesdecoder/scripts/training/train-model.perl line 1924.*

Does someone know how to get rid of that problem?

Thanks in advance and best regards,
Anna

-- 
Anna Kupś
+48 782 823 850 | www.linkedin.com/in/annakups
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error in mteval-v13a.pl

[Dingyuan Wang]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Recently mteval-v13a.pl stopped working, printing:

Can't find Unicode property definition "Line_Break" in regex; marked
by <-- HERE in m/\p{Line_Break} <-- HERE \p{Zl}/ at
/home/gumble/software/moses/scripts/generic/mteval-v13a.pl line 953.

I see this commit
.
I found that changing this to \p{Line_Break: Hyphen} worked. Is this
the equivalent of \p{Hyphen}?

- -- 
Dingyuan Wang
-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEjE4PLbCEqfvlC0rjs+TYPj8+X9wFAljZwNYACgkQs+TYPj8+
X9xShggAhSjPEEYXsiRPT9wVljRV7XjBmexe/E7EKzl9b/PEnuxlSNSrz/0Estr5
8/H4s+lwKdv9xx1jTxOGOVkToiVC95QkuppXX3WS+BCDjajE8fqWc2Y0IhUWRaqf
PAhhotEZmoWAhQC/qVM7lILf29N9OhQ2FStQH9rn+LpD2dkSZweZ0XGJ+CFpCdaP
VA7XPWJCJZeEBUsBqrSxl1Cwzr+KQ4pw/NFP6yxJ+smmTkUSyp2FfYCtvalx/L0d
2UZ1fiujzco7NHeJW/0ZYwsb+NNMuM7CljBMhQAWIN+D0f6Wz1/bHH8jhFyxUw0B
+/hN/chrAmYX+Kz2j/MKc7eXZuPtmA==
=C+jT
-END PGP SIGNATURE-
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error Installing Moses

[Hieu Hoang]

You don't need to compile moses if all you want is to use the tokenizer 
- it's just a perl script.

ps. please subscribe to the Moses mailing list before posting to it. You 
can subscribe here:

http://mailman.mit.edu/mailman/listinfo/moses-support


On 27/02/2017 21:12, Sriharini Pingali wrote:
> I've attached the build log to this email. I currently have OS 
> X 10.11.6 and had a couple failed targets when installing Boost on my 
> computer.
>
> I just want to be able to use the Moses tokenizer in my Python code. I 
> already have NLTK. Please let me know if there is some easy way to set 
> this up.

-- 
Hieu Hoang
http://moses-smt.org/

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error: died with signal 11, with coredump

[Arpita Dutta]

Hello everyone'

Hmm: Iteration 1
ERROR: Execution of: /home/nits/Desktop/ad/mosesdecoder/tools/GIZA++
-CoocurrenceFile /home/nits/Desktop/ad/demo1/giza.ben-en/ben-en.cooc -c
/home/nits/Desktop/ad/demo1/corpus/ben-en-int-train.snt -m1 5 -m2 0
-m3 3 -m4 3 -model1dumpfrequency 1
model4smoothfactor 0.4 -nodumps 1 -nsmooth 4 -o
/home/nits/Desktop/ad/demo1/giza.ben-en/ben-en
-onlyaldumps 1 -p0 0.999 -s /home/nits/Desktop/ad/demo1/corpus/en.vcb
-t /home/nits/Desktop/ad/demo1/corpus/ben.vcb
  died with signal 11, with coredump

Can anyone please explain to me why this error is coming?I am using a
parallel corpora of 5000 sentences for training,3500 sentences for
tuning and 4GB RAM.

I have tried this command once to remove segmentation fault -S 50%.I
have got result that time but now this command  is not working.

Is there any other command to remove this error?

Please help me.

Arpita Dutta

PhD Scholar

Nit Silchar

Assam-788010

India
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error in Training the Translation Model

[Sonal khosla]

Hi all,

Thank you to those who helped me in the installation. I am able to install 
Moses through your help.

Now I have an error while training the translation model. 
I am using the following command for Truecasing. 
/home/sonal/mosesdecoder/scripts/recaser/truecase.perl --model  
/home/sonal/corpus/truecase-model.hi  < /home/sonal/corpus/CORPUS.tok.hi > 
/home/sonal/corpus/CORPUS.true.hi

There was no error and the file is generated. 

But before doing this I had run the same command with different file name.
/home/sonal/mosesdecoder/scripts/recaser/truecase.perl --model  
/home/sonal/corpus/truecase-model.hi  < /home/sonal/corpus/CORPUS.tok.hi > 
/home/sonal/corpus/CORPUS.hi.true.hi


Now while training the translation model with this command 
nohup nice /home/sonal/mosesdecoder/scripts/training/train-model.perl -root-dir 
train -corpus /home/sonal/corpus/CORPUS.hi.true -f hi -e mr -alignment 
grow-diag-final-and -reordering msd-bidirectional-fe -lm 
0:3:/home/sonal/corpus/CORPUS.hi.blm.mr:8 -external-bin-dir 
/home/sonal/mosesdecoder/tools >training.out

I am getting the following error in the training.out file. 
Using SCRIPTS_ROOTDIR: /home/sonal/mosesdecoder/scripts
Using single-thread GIZA
using gzip 
(1) preparing corpus @ Mon Jan 30 11:02:20 IST 2017
Executing: mkdir -p /home/sonal/corpus/train/corpus
(1.0) selecting factors @ Mon Jan 30 11:02:20 IST 2017
(1.1) running mkcls  @ Mon Jan 30 11:02:20 IST 2017
/home/sonal/mosesdecoder/tools/mkcls -c50 -n2 
-p/home/sonal/corpus/CORPUS.hi.true.hi 
-V/home/sonal/corpus/train/corpus/hi.vcb.classes opt
Executing: /home/sonal/mosesdecoder/tools/mkcls -c50 -n2 
-p/home/sonal/corpus/CORPUS.hi.true.hi 
-V/home/sonal/corpus/train/corpus/hi.vcb.classes opt
ERROR: can not open file /home/sonal/corpus/CORPUS.hi.true.hi.
Error: Could not read the file '/home/sonal/corpus/CORPUS.hi.true.hi'.


Why is it referring to the old file which I have deleted. How do I correct this 
error. 

Thanks in advance

Sonal

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error while installing moses

[Hieu Hoang]

I think there's a problem with your xmlrpc-c library, the default on 
Ubuntu 14.04 may be too old for moses.


If you don't need moses server or Moses2, then add this argument when 
compiling:

   --no-xmlrpc-c
If you do need it, then download and compiled the xmlrpc-c library 
separately and add the argument:

  --with-xmlrpc-c=[path to xmlrpc-c lib]
eg.
  ./bjam --with-xmlrpc-c=/home/hieu/workspace/xmlrpc-c/xmlrpc-c-1.39.12


On 21/01/2017 12:49, Sonal khosla wrote:

Hi all,

I am trying installing Moses on Ubuntu 14.04. this is the sequence of 
commands I executed for the installation.


sudo apt-get install g++ git subversion automake libtool zlib1g-dev 
libboost-all-dev libbz2-dev liblzma-dev python-dev libtcmalloc-minimal4


wget 
http://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.gzr=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fboost%2Ffiles%2Fboost%2F1.55.0%2F&ts=13

tar zxvf boost_1_55_0.tar.gz
cd boost_1_55_0/
./bootstrap.sh
./b2 -j5 --prefix=$PWD --libdir=$PWD/lib64 --layout=tagged link=static 
threading=multi,single install || echo FAILURE




git clone https://github.com/moses-smt/mosesdecoder.git
cd mosesdecoder/
./bjam -j5

After this command I got the error "Build Failed"
Also sharing the error logs



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


--
Hieu Hoang
http://moses-smt.org/

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error while installing moses

[Sonal khosla]

Hi all,

I am trying installing Moses on Ubuntu 14.04. this is the sequence of commands 
I executed for the installation. 

sudo apt-get install g++ git subversion automake libtool zlib1g-dev 
libboost-all-dev libbz2-dev liblzma-dev python-dev libtcmalloc-minimal4

wget 
http://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.gzr=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fboost%2Ffiles%2Fboost%2F1.55.0%2F&ts=13
tar zxvf boost_1_55_0.tar.gz
cd boost_1_55_0/
./bootstrap.sh
./b2 -j5 --prefix=$PWD --libdir=$PWD/lib64 --layout=tagged link=static 
threading=multi,single install || echo FAILURE



git clone https://github.com/moses-smt/mosesdecoder.git
cd mosesdecoder/
./bjam -j5

After this command I got the error "Build Failed"
Also sharing the error logs


build.log.gz
Description: application/gzip
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] error in mosesdecoder tuning

[Tom Hoar]

Without a start-to-finish log file, it's impossible to debug. Based on 
your description, however, I'd say you probably (a) ran out of free hard 
drive space for temp files somewhere during step 2, or your very large 
corpus has invalid character. Both will cause empty GIZA++ alignment 
files and empty phrase tables.


Have you ever create a smaller model to validate your corpus preparation?



On 11/2/2016 10:32 PM, moses-support-requ...@mit.edu wrote:

Date: Wed, 2 Nov 2016 17:32:48 +0200
From: Hasan Sait ARSLAN
Subject: Re: [Moses-support] error in mosesdecoder tuning
To: Barry Haddow
Cc: moses-support
Message-ID:

Content-Type: text/plain; charset="utf-8"

Hi Barry,

I have rerun training starting from step 5, and I kept log file. I have
checked log file, and it doesn't show any bug about creating phrase-table,
but phrase-table is still not being created. I am attaching log file, could
you please check it, and show me where is the problem?

Thanks,?
  log.gz
<https://drive.google.com/file/d/0BxvJK3H5ZKsnYzJiZmhjUWI0Qlk/view?usp=drive_web>
?


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] error in mosesdecoder tuning

[Barry Haddow]

Hi Hasan

This is full of messages about incorrect alignments. It looks like 
something went wrong at an earlier stage. I would start again from the 
beginning with a small data set, in order to debug your setup,


cheers - Barry

On 02/11/16 15:32, Hasan Sait ARSLAN wrote:

Hi Barry,

I have rerun training starting from step 5, and I kept log file. I 
have checked log file, and it doesn't show any bug about creating 
phrase-table, but phrase-table is still not being created. I am 
attaching log file, could you please check it, and show me where is 
the problem?


Thanks,​
log.gz 


​

2016-11-02 14:28 GMT+02:00 Barry Haddow >:


Adding
-first-step 5 -last-step 5
will just run step 5 (phrase extraction)


On 02/11/16 12:01, Hasan Sait ARSLAN wrote:

For instance, could you show me an example?

Thanks,

2016-11-02 13:57 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

You can use the first-step and last-step arguments to run
steps manually.

cheers - Barry


On 02/11/16 11:46, Hasan Sait ARSLAN wrote:

Sorry,

as I see my model folder, I can say that Lexical translation
part also went well.

so, the only steps I have to run manually are'

  * 5 Extract phrases


  * 6 Score phrases


  * 7 Reordering model



  * 8 Generation model



  * 9 Configuration file








2016-11-02 13:43 GMT+02:00 Hasan Sait ARSLAN
mailto:hasan.sait.ars...@gmail.com>>:

Hi Barry,

Actually I know where I it stuck.

As I check my train folder, I can see the all the files
which show that

  * 1 Prepare data


  * 2 Run GIZA


  * 3 Align words



went well.

So, is there any way to run the remaining


  * 4 Lexical translation



  * 5 Extract phrases


  * 6 Score phrases


  * 7 Reordering model



  * 8 Generation model



  * 9 Configuration file




steps manually?


2016-11-02 13:27 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

It's not really possible to debug without a log
file, so I think that unfortunately you should start
again.

I suggest you start with a much smaller corpus to
try to debug your setup. If you use 100,000
sentences it should train within hours.

cheers - Barry


On 02/11/16 11:13, Hasan Sait ARSLAN wrote:

Hi Barry,

Unfortunately I didn't keep the log file. Is it
really a hopeless situation?

2016-11-02 13:10 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

You should have run train_model.perl something
like this:

nohup nice /path/to/train_model.perl
ARGS_FOR_TRAINING &> log &

You just need the log file

cheers - Barry


On 02/11/16 11:00, Hasan Sait ARSLAN wrote:

Hi Barry,

I don't have any idea about where it is saved.

Cheers,

2016-11-02 12:55 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

If your phrase table is empty, that would
explain why tuning didn't work

Re: [Moses-support] error in mosesdecoder tuning

[Hasan Sait ARSLAN]

Hi Barry,

I have rerun training starting from step 5, and I kept log file. I have
checked log file, and it doesn't show any bug about creating phrase-table,
but phrase-table is still not being created. I am attaching log file, could
you please check it, and show me where is the problem?

Thanks,​
 log.gz

​

2016-11-02 14:28 GMT+02:00 Barry Haddow :

> Adding
> -first-step 5 -last-step 5
> will just run step 5 (phrase extraction)
>
>
> On 02/11/16 12:01, Hasan Sait ARSLAN wrote:
>
> For instance, could you show me an example?
>
> Thanks,
>
> 2016-11-02 13:57 GMT+02:00 Barry Haddow :
>
>> Hi Hasan
>>
>> You can use the first-step and last-step arguments to run steps manually.
>>
>> cheers - Barry
>>
>>
>> On 02/11/16 11:46, Hasan Sait ARSLAN wrote:
>>
>> Sorry,
>>
>> as I see my model folder, I can say that Lexical translation part also
>> went well.
>>
>> so, the only steps I have to run manually are'
>>
>>
>>- 5 Extract phrases
>>
>>- 6 Score phrases
>>
>>- 7 Reordering model
>>
>>- 8 Generation model
>>
>>- 9 Configuration file
>>
>>
>>
>>
>>
>>
>> 2016-11-02 13:43 GMT+02:00 Hasan Sait ARSLAN > >:
>>
>>> Hi Barry,
>>>
>>> Actually I know where I it stuck.
>>>
>>> As I check my train folder, I can see the all the files which show that
>>>
>>>
>>>- 1 Prepare data
>>>
>>>- 2 Run GIZA
>>>
>>>- 3 Align words
>>>
>>>
>>> went well.
>>>
>>> So, is there any way to run the remaining
>>>
>>>
>>>
>>>- 4 Lexical translation
>>>
>>> 
>>>- 5 Extract phrases
>>>
>>>- 6 Score phrases
>>>
>>>- 7 Reordering model
>>>
>>>- 8 Generation model
>>>
>>>- 9 Configuration file
>>>
>>>
>>> steps manually?
>>>
>>> 2016-11-02 13:27 GMT+02:00 Barry Haddow :
>>>
 Hi Hasan

 It's not really possible to debug without a log file, so I think that
 unfortunately you should start again.

 I suggest you start with a much smaller corpus to try to debug your
 setup. If you use 100,000 sentences it should train within hours.

 cheers - Barry


 On 02/11/16 11:13, Hasan Sait ARSLAN wrote:

 Hi Barry,

 Unfortunately I didn't keep the log file. Is it really a hopeless
 situation?

 2016-11-02 13:10 GMT+02:00 Barry Haddow :

> Hi Hasan
>
> You should have run train_model.perl something like this:
>
> nohup nice /path/to/train_model.perl ARGS_FOR_TRAINING &> log &
>
> You just need the log file
>
> cheers - Barry
>
>
> On 02/11/16 11:00, Hasan Sait ARSLAN wrote:
>
> Hi Barry,
>
> I don't have any idea about where it is saved.
>
> Cheers,
>
> 2016-11-02 12:55 GMT+02:00 Barry Haddow :
>
>> Hi Hasan
>>
>> If your phrase table is empty, that would explain why tuning didn't
>> work. Something went wrong earlier in the process. Could you post your 
>> log
>> file from train_model.perl,
>>
>> cheers - Barry
>>
>>
>> On 02/11/16 10:45, Hasan Sait ARSLAN wrote:
>>
>> Now currently I have a bigger problem. I have trained my data for 5
>> days, and the folder "train" is 39 G, but there is no any phrases saved 
>> on
>> phrase table. It is annoying. What should I do now? I hope I won't need 
>> to
>> rerun everything from the scratch
>>
>> 2016-11-02 12:26 GMT+02:00 Barry Haddow :
>>
>>> Hi Hasan
>>>
>>> The error message should be written into filterphrases.err, inside
>>> your working directory,
>>>
>>> cheers - Barry
>>>
>>>
>>> On 02/11/16 10:02, Hasan Sait ARSLAN wrote:
>>>
>>> Hi Hieu,
>>>
>>> I did in the way, you want. Plus, I am sure the path and file names
>>> are correctly spelt.
>>>  But still, I get the same error.
>>>
>>> 2016-11-01 21:41 GMT+02:00 Hasan Sait ARSLAN <
>>> hasan.sait.ars...@gmail.com>:
>>>
 Thank yo

Re: [Moses-support] error in mosesdecoder tuning

[Barry Haddow]

Adding
-first-step 5 -last-step 5
will just run step 5 (phrase extraction)

On 02/11/16 12:01, Hasan Sait ARSLAN wrote:

For instance, could you show me an example?

Thanks,

2016-11-02 13:57 GMT+02:00 Barry Haddow >:


Hi Hasan

You can use the first-step and last-step arguments to run steps
manually.

cheers - Barry


On 02/11/16 11:46, Hasan Sait ARSLAN wrote:

Sorry,

as I see my model folder, I can say that Lexical translation part
also went well.

so, the only steps I have to run manually are'

  * 5 Extract phrases

  * 6 Score phrases

  * 7 Reordering model


  * 8 Generation model


  * 9 Configuration file








2016-11-02 13:43 GMT+02:00 Hasan Sait ARSLAN
mailto:hasan.sait.ars...@gmail.com>>:

Hi Barry,

Actually I know where I it stuck.

As I check my train folder, I can see the all the files which
show that

  * 1 Prepare data


  * 2 Run GIZA

  * 3 Align words


went well.

So, is there any way to run the remaining


  * 4 Lexical translation



  * 5 Extract phrases


  * 6 Score phrases


  * 7 Reordering model



  * 8 Generation model



  * 9 Configuration file




steps manually?


2016-11-02 13:27 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

It's not really possible to debug without a log file, so
I think that unfortunately you should start again.

I suggest you start with a much smaller corpus to try to
debug your setup. If you use 100,000 sentences it should
train within hours.

cheers - Barry


On 02/11/16 11:13, Hasan Sait ARSLAN wrote:

Hi Barry,

Unfortunately I didn't keep the log file. Is it really a
hopeless situation?

2016-11-02 13:10 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

You should have run train_model.perl something like
this:

nohup nice /path/to/train_model.perl
ARGS_FOR_TRAINING &> log &

You just need the log file

cheers - Barry


On 02/11/16 11:00, Hasan Sait ARSLAN wrote:

Hi Barry,

I don't have any idea about where it is saved.

Cheers,

2016-11-02 12:55 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

If your phrase table is empty, that would
explain why tuning didn't work. Something went
wrong earlier in the process. Could you post
your log file from train_model.perl,

cheers - Barry


On 02/11/16 10:45, Hasan Sait ARSLAN wrote:

Now currently I have a bigger problem. I have
trained my data for 5 days, and the folder
"train" is 39 G, but there is no any phrases
saved on phrase table. It is annoying. What
should I do now? I hope I won't need to rerun
everything from the scratch

2016-11-02 12:26 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

The error message should be written into
filterphrases.err, inside your working
directory,

cheers - Barry


On 02/11/16 10:02, Hasan Sait ARSLAN wrote:

Hi Hieu,

I did in the way

Re: [Moses-support] error in mosesdecoder tuning

[Hasan Sait ARSLAN]

For instance, could you show me an example?

Thanks,

2016-11-02 13:57 GMT+02:00 Barry Haddow :

> Hi Hasan
>
> You can use the first-step and last-step arguments to run steps manually.
>
> cheers - Barry
>
>
> On 02/11/16 11:46, Hasan Sait ARSLAN wrote:
>
> Sorry,
>
> as I see my model folder, I can say that Lexical translation part also
> went well.
>
> so, the only steps I have to run manually are'
>
>
>- 5 Extract phrases
>
>- 6 Score phrases
>
>- 7 Reordering model
>
>- 8 Generation model
>
>- 9 Configuration file
>
>
>
>
>
>
> 2016-11-02 13:43 GMT+02:00 Hasan Sait ARSLAN 
> :
>
>> Hi Barry,
>>
>> Actually I know where I it stuck.
>>
>> As I check my train folder, I can see the all the files which show that
>>
>>
>>- 1 Prepare data
>>
>>- 2 Run GIZA 
>>- 3 Align words
>>
>>
>> went well.
>>
>> So, is there any way to run the remaining
>>
>>
>>
>>- 4 Lexical translation
>>
>> 
>>- 5 Extract phrases
>>
>>- 6 Score phrases
>>
>>- 7 Reordering model
>>
>>- 8 Generation model
>>
>>- 9 Configuration file
>>
>>
>> steps manually?
>>
>> 2016-11-02 13:27 GMT+02:00 Barry Haddow :
>>
>>> Hi Hasan
>>>
>>> It's not really possible to debug without a log file, so I think that
>>> unfortunately you should start again.
>>>
>>> I suggest you start with a much smaller corpus to try to debug your
>>> setup. If you use 100,000 sentences it should train within hours.
>>>
>>> cheers - Barry
>>>
>>>
>>> On 02/11/16 11:13, Hasan Sait ARSLAN wrote:
>>>
>>> Hi Barry,
>>>
>>> Unfortunately I didn't keep the log file. Is it really a hopeless
>>> situation?
>>>
>>> 2016-11-02 13:10 GMT+02:00 Barry Haddow :
>>>
 Hi Hasan

 You should have run train_model.perl something like this:

 nohup nice /path/to/train_model.perl ARGS_FOR_TRAINING &> log &

 You just need the log file

 cheers - Barry


 On 02/11/16 11:00, Hasan Sait ARSLAN wrote:

 Hi Barry,

 I don't have any idea about where it is saved.

 Cheers,

 2016-11-02 12:55 GMT+02:00 Barry Haddow :

> Hi Hasan
>
> If your phrase table is empty, that would explain why tuning didn't
> work. Something went wrong earlier in the process. Could you post your log
> file from train_model.perl,
>
> cheers - Barry
>
>
> On 02/11/16 10:45, Hasan Sait ARSLAN wrote:
>
> Now currently I have a bigger problem. I have trained my data for 5
> days, and the folder "train" is 39 G, but there is no any phrases saved on
> phrase table. It is annoying. What should I do now? I hope I won't need to
> rerun everything from the scratch
>
> 2016-11-02 12:26 GMT+02:00 Barry Haddow :
>
>> Hi Hasan
>>
>> The error message should be written into filterphrases.err, inside
>> your working directory,
>>
>> cheers - Barry
>>
>>
>> On 02/11/16 10:02, Hasan Sait ARSLAN wrote:
>>
>> Hi Hieu,
>>
>> I did in the way, you want. Plus, I am sure the path and file names
>> are correctly spelt.
>>  But still, I get the same error.
>>
>> 2016-11-01 21:41 GMT+02:00 Hasan Sait ARSLAN <
>> hasan.sait.ars...@gmail.com>:
>>
>>> Thank you Hieu, I will do it, and give you feedback about it.
>>>
>>> Kind Regards,
>>>
>>> 2016-11-01 21:37 GMT+02:00 Hieu Hoang :
>>>
 the command you execute looks ok. the problem is likely to be with
 your using tuning data.

 You should look at your tuning data, make sure you've spelt the
 path and filename correctly, use a very small subset (eg. start with 1
 parallel sentence) and increase the number of sentences until you find 
 the
 problem



 On 01/11/2016 12:30, Hasan Sait ARSLAN wrote:

 Hello,

 I have trained dataset for 5 days, and for 2-3 days I am dealing
 with the bug in tuning, but s

Re: [Moses-support] error in mosesdecoder tuning

[Barry Haddow]

Hi Hasan

You can use the first-step and last-step arguments to run steps manually.

cheers - Barry

On 02/11/16 11:46, Hasan Sait ARSLAN wrote:

Sorry,

as I see my model folder, I can say that Lexical translation part also 
went well.


so, the only steps I have to run manually are'

  * 5 Extract phrases

  * 6 Score phrases

  * 7 Reordering model


  * 8 Generation model


  * 9 Configuration file







2016-11-02 13:43 GMT+02:00 Hasan Sait ARSLAN 
mailto:hasan.sait.ars...@gmail.com>>:


Hi Barry,

Actually I know where I it stuck.

As I check my train folder, I can see the all the files which show
that

  * 1 Prepare data

  * 2 Run GIZA

  * 3 Align words


went well.

So, is there any way to run the remaining


  * 4 Lexical translation



  * 5 Extract phrases

  * 6 Score phrases

  * 7 Reordering model


  * 8 Generation model


  * 9 Configuration file




steps manually?


2016-11-02 13:27 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

It's not really possible to debug without a log file, so I
think that unfortunately you should start again.

I suggest you start with a much smaller corpus to try to debug
your setup. If you use 100,000 sentences it should train
within hours.

cheers - Barry


On 02/11/16 11:13, Hasan Sait ARSLAN wrote:

Hi Barry,

Unfortunately I didn't keep the log file. Is it really a
hopeless situation?

2016-11-02 13:10 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

You should have run train_model.perl something like this:

nohup nice /path/to/train_model.perl ARGS_FOR_TRAINING &>
log &

You just need the log file

cheers - Barry


On 02/11/16 11:00, Hasan Sait ARSLAN wrote:

Hi Barry,

I don't have any idea about where it is saved.

Cheers,

2016-11-02 12:55 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

If your phrase table is empty, that would explain
why tuning didn't work. Something went wrong earlier
in the process. Could you post your log file from
train_model.perl,

cheers - Barry


On 02/11/16 10:45, Hasan Sait ARSLAN wrote:

Now currently I have a bigger problem. I have
trained my data for 5 days, and the folder "train"
is 39 G, but there is no any phrases saved on
phrase table. It is annoying. What should I do now?
I hope I won't need to rerun everything from the
scratch

2016-11-02 12:26 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

The error message should be written into
filterphrases.err, inside your working directory,

cheers - Barry


On 02/11/16 10:02, Hasan Sait ARSLAN wrote:

Hi Hieu,

I did in the way, you want. Plus, I am sure
the path and file names are correctly spelt.
 But still, I get the same error.

2016-11-01 21:41 GMT+02:00 Hasan Sait ARSLAN
mailto:hasan.sait.ars...@gmail.com>>:

Thank you Hieu, I will do it, and give you
feedback about it.

Kind Regards,

2016-11-01 21:37 GMT+02:00 Hieu Hoang
mailto:hieuho...@gmail.com>>:

the command you execute looks ok. the
problem is likely to be with your
   

Re: [Moses-support] error in mosesdecoder tuning

[Barry Haddow]

Hi Hasan

It's not really possible to debug without a log file, so I think that 
unfortunately you should start again.


I suggest you start with a much smaller corpus to try to debug your 
setup. If you use 100,000 sentences it should train within hours.


cheers - Barry

On 02/11/16 11:13, Hasan Sait ARSLAN wrote:

Hi Barry,

Unfortunately I didn't keep the log file. Is it really a hopeless 
situation?


2016-11-02 13:10 GMT+02:00 Barry Haddow >:


Hi Hasan

You should have run train_model.perl something like this:

nohup nice /path/to/train_model.perl ARGS_FOR_TRAINING &> log &

You just need the log file

cheers - Barry


On 02/11/16 11:00, Hasan Sait ARSLAN wrote:

Hi Barry,

I don't have any idea about where it is saved.

Cheers,

2016-11-02 12:55 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

If your phrase table is empty, that would explain why tuning
didn't work. Something went wrong earlier in the process.
Could you post your log file from train_model.perl,

cheers - Barry


On 02/11/16 10:45, Hasan Sait ARSLAN wrote:

Now currently I have a bigger problem. I have trained my
data for 5 days, and the folder "train" is 39 G, but there
is no any phrases saved on phrase table. It is annoying.
What should I do now? I hope I won't need to rerun
everything from the scratch

2016-11-02 12:26 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

The error message should be written into
filterphrases.err, inside your working directory,

cheers - Barry


On 02/11/16 10:02, Hasan Sait ARSLAN wrote:

Hi Hieu,

I did in the way, you want. Plus, I am sure the path
and file names are correctly spelt.
 But still, I get the same error.

2016-11-01 21:41 GMT+02:00 Hasan Sait ARSLAN
mailto:hasan.sait.ars...@gmail.com>>:

Thank you Hieu, I will do it, and give you feedback
about it.

Kind Regards,

2016-11-01 21:37 GMT+02:00 Hieu Hoang
mailto:hieuho...@gmail.com>>:

the command you execute looks ok. the problem
is likely to be with your using tuning data.

You should look at your tuning data, make sure
you've spelt the path and filename correctly,
use a very small subset (eg. start with 1
parallel sentence) and increase the number of
sentences until you find the problem



On 01/11/2016 12:30, Hasan Sait ARSLAN wrote:

Hello,

I have trained dataset for 5 days, and for 2-3
days I am dealing with the bug in tuning, but
still couldn't solve the problem.

I use the following command to run:

*/home/sait/mosesdecoder/scripts/training/mert-moses.pl

/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr

/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
/home/sait/mosesdecoder/bin/moses
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
--working-dir
/home/sait/Kairit/Task3/working_tr-en/mert
--decoder-flags "--threads 32"

*
Then I get the following error:

*Using SCRIPTS_ROOTDIR:
/home/sait/mosesdecoder/scripts
Assuming --mertdir=/home/sait/mosesdecoder/bin
filtering the phrase tables... T nov1
19:17:39 EET 2016
exec:

/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl

./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr

Executing:

/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl

./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
 > filterphrases.out 2>
filterphrases.err
Exit code: 255
ERROR: Failed to run
 

Re: [Moses-support] error in mosesdecoder tuning

[Hasan Sait ARSLAN]

Hi Barry,

Unfortunately I didn't keep the log file. Is it really a hopeless situation?

2016-11-02 13:10 GMT+02:00 Barry Haddow :

> Hi Hasan
>
> You should have run train_model.perl something like this:
>
> nohup nice /path/to/train_model.perl ARGS_FOR_TRAINING &> log &
>
> You just need the log file
>
> cheers - Barry
>
>
> On 02/11/16 11:00, Hasan Sait ARSLAN wrote:
>
> Hi Barry,
>
> I don't have any idea about where it is saved.
>
> Cheers,
>
> 2016-11-02 12:55 GMT+02:00 Barry Haddow :
>
>> Hi Hasan
>>
>> If your phrase table is empty, that would explain why tuning didn't work.
>> Something went wrong earlier in the process. Could you post your log file
>> from train_model.perl,
>>
>> cheers - Barry
>>
>>
>> On 02/11/16 10:45, Hasan Sait ARSLAN wrote:
>>
>> Now currently I have a bigger problem. I have trained my data for 5 days,
>> and the folder "train" is 39 G, but there is no any phrases saved on phrase
>> table. It is annoying. What should I do now? I hope I won't need to rerun
>> everything from the scratch
>>
>> 2016-11-02 12:26 GMT+02:00 Barry Haddow :
>>
>>> Hi Hasan
>>>
>>> The error message should be written into filterphrases.err, inside your
>>> working directory,
>>>
>>> cheers - Barry
>>>
>>>
>>> On 02/11/16 10:02, Hasan Sait ARSLAN wrote:
>>>
>>> Hi Hieu,
>>>
>>> I did in the way, you want. Plus, I am sure the path and file names are
>>> correctly spelt.
>>>  But still, I get the same error.
>>>
>>> 2016-11-01 21:41 GMT+02:00 Hasan Sait ARSLAN <
>>> hasan.sait.ars...@gmail.com>:
>>>
 Thank you Hieu, I will do it, and give you feedback about it.

 Kind Regards,

 2016-11-01 21:37 GMT+02:00 Hieu Hoang :

> the command you execute looks ok. the problem is likely to be with
> your using tuning data.
>
> You should look at your tuning data, make sure you've spelt the path
> and filename correctly, use a very small subset (eg. start with 1 parallel
> sentence) and increase the number of sentences until you find the problem
>
>
>
> On 01/11/2016 12:30, Hasan Sait ARSLAN wrote:
>
> Hello,
>
> I have trained dataset for 5 days, and for 2-3 days I am dealing with
> the bug in tuning, but still couldn't solve the problem.
>
> I use the following command to run:
>
>
>
> */home/sait/mosesdecoder/scripts/training/mert-moses.pl
>  /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>  /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
> /home/sait/mosesdecoder/bin/moses
> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini --working-dir
> /home/sait/Kairit/Task3/working_tr-en/mert --decoder-flags "--threads 32" 
> *
> Then I get the following error:
>
>
>
>
>
>
>
>
>
> *Using SCRIPTS_ROOTDIR: /home/sait/mosesdecoder/scripts Assuming
> --mertdir=/home/sait/mosesdecoder/bin filtering the phrase tables... T
> nov1 19:17:39 EET 2016 exec:
> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>  ./filtered
> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr 
> Executing:
> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>  ./filtered
> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr  
> >
> filterphrases.out 2> filterphrases.err Exit code: 255 ERROR: Failed to run
> '/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>  ./filtered
> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr 
> '.
> at /home/sait/mosesdecoder/scripts/training/mert-moses.pl
>  line 1748. *
> I have searched over the internet about the error, but unfortunately I
> couldn't find any beneficial solution.
>
> It is so annoying, and eats my time, please help me to solve my
> problem.
>
> Thanks,
>
>
> ___
> Moses-support mailing 
> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>

>>>
>>>
>>> ___
>>> Moses-support mailing 
>>> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>>
>>> The University of Edinburgh is a charitable body, registered in
>>> Scotland, with registration number SC005336.
>>>
>>>
>>
>>
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>>
>>
>
>
> The University of Edinburgh is a charitable 

Re: [Moses-support] error in mosesdecoder tuning

[Barry Haddow]

Hi Hasan

You should have run train_model.perl something like this:

nohup nice /path/to/train_model.perl ARGS_FOR_TRAINING &> log &

You just need the log file

cheers - Barry

On 02/11/16 11:00, Hasan Sait ARSLAN wrote:

Hi Barry,

I don't have any idea about where it is saved.

Cheers,

2016-11-02 12:55 GMT+02:00 Barry Haddow >:


Hi Hasan

If your phrase table is empty, that would explain why tuning
didn't work. Something went wrong earlier in the process. Could
you post your log file from train_model.perl,

cheers - Barry


On 02/11/16 10:45, Hasan Sait ARSLAN wrote:

Now currently I have a bigger problem. I have trained my data for
5 days, and the folder "train" is 39 G, but there is no any
phrases saved on phrase table. It is annoying. What should I do
now? I hope I won't need to rerun everything from the scratch

2016-11-02 12:26 GMT+02:00 Barry Haddow
mailto:bhad...@staffmail.ed.ac.uk>>:

Hi Hasan

The error message should be written into filterphrases.err,
inside your working directory,

cheers - Barry


On 02/11/16 10:02, Hasan Sait ARSLAN wrote:

Hi Hieu,

I did in the way, you want. Plus, I am sure the path and
file names are correctly spelt.
 But still, I get the same error.

2016-11-01 21:41 GMT+02:00 Hasan Sait ARSLAN
mailto:hasan.sait.ars...@gmail.com>>:

Thank you Hieu, I will do it, and give you feedback
about it.

Kind Regards,

2016-11-01 21:37 GMT+02:00 Hieu Hoang
mailto:hieuho...@gmail.com>>:

the command you execute looks ok. the problem is
likely to be with your using tuning data.

You should look at your tuning data, make sure
you've spelt the path and filename correctly, use a
very small subset (eg. start with 1 parallel
sentence) and increase the number of sentences until
you find the problem



On 01/11/2016 12:30, Hasan Sait ARSLAN wrote:

Hello,

I have trained dataset for 5 days, and for 2-3 days
I am dealing with the bug in tuning, but still
couldn't solve the problem.

I use the following command to run:

*/home/sait/mosesdecoder/scripts/training/mert-moses.pl

/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr

/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
/home/sait/mosesdecoder/bin/moses
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
--working-dir
/home/sait/Kairit/Task3/working_tr-en/mert
--decoder-flags "--threads 32"

*
Then I get the following error:

*Using SCRIPTS_ROOTDIR: /home/sait/mosesdecoder/scripts
Assuming --mertdir=/home/sait/mosesdecoder/bin
filtering the phrase tables... T nov1 19:17:39
EET 2016
exec:

/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr

Executing:

/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
 > filterphrases.out 2>
filterphrases.err
Exit code: 255
ERROR: Failed to run

'/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
'. at
/home/sait/mosesdecoder/scripts/training/mert-moses.pl
 line 1748.

*
I have searched over the internet about the error,
but unfortunately I couldn't find any beneficial
solution.

It is so annoying, and eats my time, please help me
to solve my problem.

Thanks,


___
Moses-support mailing list
Moses-support@

Re: [Moses-support] error in mosesdecoder tuning

[Hasan Sait ARSLAN]

Hi Barry,

I don't have any idea about where it is saved.

Cheers,

2016-11-02 12:55 GMT+02:00 Barry Haddow :

> Hi Hasan
>
> If your phrase table is empty, that would explain why tuning didn't work.
> Something went wrong earlier in the process. Could you post your log file
> from train_model.perl,
>
> cheers - Barry
>
>
> On 02/11/16 10:45, Hasan Sait ARSLAN wrote:
>
> Now currently I have a bigger problem. I have trained my data for 5 days,
> and the folder "train" is 39 G, but there is no any phrases saved on phrase
> table. It is annoying. What should I do now? I hope I won't need to rerun
> everything from the scratch
>
> 2016-11-02 12:26 GMT+02:00 Barry Haddow :
>
>> Hi Hasan
>>
>> The error message should be written into filterphrases.err, inside your
>> working directory,
>>
>> cheers - Barry
>>
>>
>> On 02/11/16 10:02, Hasan Sait ARSLAN wrote:
>>
>> Hi Hieu,
>>
>> I did in the way, you want. Plus, I am sure the path and file names are
>> correctly spelt.
>>  But still, I get the same error.
>>
>> 2016-11-01 21:41 GMT+02:00 Hasan Sait ARSLAN > >:
>>
>>> Thank you Hieu, I will do it, and give you feedback about it.
>>>
>>> Kind Regards,
>>>
>>> 2016-11-01 21:37 GMT+02:00 Hieu Hoang :
>>>
 the command you execute looks ok. the problem is likely to be with your
 using tuning data.

 You should look at your tuning data, make sure you've spelt the path
 and filename correctly, use a very small subset (eg. start with 1 parallel
 sentence) and increase the number of sentences until you find the problem



 On 01/11/2016 12:30, Hasan Sait ARSLAN wrote:

 Hello,

 I have trained dataset for 5 days, and for 2-3 days I am dealing with
 the bug in tuning, but still couldn't solve the problem.

 I use the following command to run:



 */home/sait/mosesdecoder/scripts/training/mert-moses.pl
  /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
  /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
 /home/sait/mosesdecoder/bin/moses
 /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini --working-dir
 /home/sait/Kairit/Task3/working_tr-en/mert --decoder-flags "--threads 32" *
 Then I get the following error:









 *Using SCRIPTS_ROOTDIR: /home/sait/mosesdecoder/scripts Assuming
 --mertdir=/home/sait/mosesdecoder/bin filtering the phrase tables... T
 nov1 19:17:39 EET 2016 exec:
 /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
  ./filtered
 /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
 /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr 
 Executing:
 /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
  ./filtered
 /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
 /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr  >
 filterphrases.out 2> filterphrases.err Exit code: 255 ERROR: Failed to run
 '/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
  ./filtered
 /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
 /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr '.
 at /home/sait/mosesdecoder/scripts/training/mert-moses.pl
  line 1748. *
 I have searched over the internet about the error, but unfortunately I
 couldn't find any beneficial solution.

 It is so annoying, and eats my time, please help me to solve my problem.

 Thanks,


 ___
 Moses-support mailing 
 listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support



>>>
>>
>>
>> ___
>> Moses-support mailing 
>> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>>
>>
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] error in mosesdecoder tuning

[Barry Haddow]

Hi Hasan

If your phrase table is empty, that would explain why tuning didn't 
work. Something went wrong earlier in the process. Could you post your 
log file from train_model.perl,


cheers - Barry

On 02/11/16 10:45, Hasan Sait ARSLAN wrote:
Now currently I have a bigger problem. I have trained my data for 5 
days, and the folder "train" is 39 G, but there is no any phrases 
saved on phrase table. It is annoying. What should I do now? I hope I 
won't need to rerun everything from the scratch


2016-11-02 12:26 GMT+02:00 Barry Haddow >:


Hi Hasan

The error message should be written into filterphrases.err, inside
your working directory,

cheers - Barry


On 02/11/16 10:02, Hasan Sait ARSLAN wrote:

Hi Hieu,

I did in the way, you want. Plus, I am sure the path and file
names are correctly spelt.
 But still, I get the same error.

2016-11-01 21:41 GMT+02:00 Hasan Sait ARSLAN
mailto:hasan.sait.ars...@gmail.com>>:

Thank you Hieu, I will do it, and give you feedback about it.

Kind Regards,

2016-11-01 21:37 GMT+02:00 Hieu Hoang mailto:hieuho...@gmail.com>>:

the command you execute looks ok. the problem is likely
to be with your using tuning data.

You should look at your tuning data, make sure you've
spelt the path and filename correctly, use a very small
subset (eg. start with 1 parallel sentence) and increase
the number of sentences until you find the problem



On 01/11/2016 12:30, Hasan Sait ARSLAN wrote:

Hello,

I have trained dataset for 5 days, and for 2-3 days I am
dealing with the bug in tuning, but still couldn't solve
the problem.

I use the following command to run:

*/home/sait/mosesdecoder/scripts/training/mert-moses.pl

/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr

/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
/home/sait/mosesdecoder/bin/moses
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
--working-dir /home/sait/Kairit/Task3/working_tr-en/mert
--decoder-flags "--threads 32"

*
Then I get the following error:

*Using SCRIPTS_ROOTDIR: /home/sait/mosesdecoder/scripts
Assuming --mertdir=/home/sait/mosesdecoder/bin
filtering the phrase tables... T nov1 19:17:39 EET 2016
exec:
/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr

Executing:
/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
 > filterphrases.out 2>
filterphrases.err
Exit code: 255
ERROR: Failed to run

'/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
'. at
/home/sait/mosesdecoder/scripts/training/mert-moses.pl
 line 1748.

*
I have searched over the internet about the error, but
unfortunately I couldn't find any beneficial solution.

It is so annoying, and eats my time, please help me to
solve my problem.

Thanks,


___
Moses-support mailing list
Moses-support@mit.edu  
http://mailman.mit.edu/mailman/listinfo/moses-support  







___
Moses-support mailing list
Moses-support@mit.edu  
http://mailman.mit.edu/mailman/listinfo/moses-support  




The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.




The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailma

Re: [Moses-support] error in mosesdecoder tuning

[Hasan Sait ARSLAN]

Now currently I have a bigger problem. I have trained my data for 5 days,
and the folder "train" is 39 G, but there is no any phrases saved on phrase
table. It is annoying. What should I do now? I hope I won't need to rerun
everything from the scratch

2016-11-02 12:26 GMT+02:00 Barry Haddow :

> Hi Hasan
>
> The error message should be written into filterphrases.err, inside your
> working directory,
>
> cheers - Barry
>
>
> On 02/11/16 10:02, Hasan Sait ARSLAN wrote:
>
> Hi Hieu,
>
> I did in the way, you want. Plus, I am sure the path and file names are
> correctly spelt.
>  But still, I get the same error.
>
> 2016-11-01 21:41 GMT+02:00 Hasan Sait ARSLAN 
> :
>
>> Thank you Hieu, I will do it, and give you feedback about it.
>>
>> Kind Regards,
>>
>> 2016-11-01 21:37 GMT+02:00 Hieu Hoang :
>>
>>> the command you execute looks ok. the problem is likely to be with your
>>> using tuning data.
>>>
>>> You should look at your tuning data, make sure you've spelt the path and
>>> filename correctly, use a very small subset (eg. start with 1 parallel
>>> sentence) and increase the number of sentences until you find the problem
>>>
>>>
>>>
>>> On 01/11/2016 12:30, Hasan Sait ARSLAN wrote:
>>>
>>> Hello,
>>>
>>> I have trained dataset for 5 days, and for 2-3 days I am dealing with
>>> the bug in tuning, but still couldn't solve the problem.
>>>
>>> I use the following command to run:
>>>
>>>
>>>
>>> */home/sait/mosesdecoder/scripts/training/mert-moses.pl
>>>  /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>  /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
>>> /home/sait/mosesdecoder/bin/moses
>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini --working-dir
>>> /home/sait/Kairit/Task3/working_tr-en/mert --decoder-flags "--threads 32" *
>>> Then I get the following error:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Using SCRIPTS_ROOTDIR: /home/sait/mosesdecoder/scripts Assuming
>>> --mertdir=/home/sait/mosesdecoder/bin filtering the phrase tables... T
>>> nov1 19:17:39 EET 2016 exec:
>>> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>  ./filtered
>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr 
>>> Executing:
>>> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>  ./filtered
>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr  >
>>> filterphrases.out 2> filterphrases.err Exit code: 255 ERROR: Failed to run
>>> '/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>  ./filtered
>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr '.
>>> at /home/sait/mosesdecoder/scripts/training/mert-moses.pl
>>>  line 1748. *
>>> I have searched over the internet about the error, but unfortunately I
>>> couldn't find any beneficial solution.
>>>
>>> It is so annoying, and eats my time, please help me to solve my problem.
>>>
>>> Thanks,
>>>
>>>
>>> ___
>>> Moses-support mailing 
>>> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>>
>>
>
>
> ___
> Moses-support mailing 
> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] error in mosesdecoder tuning

[Barry Haddow]

Hi Hasan

The error message should be written into filterphrases.err, inside your 
working directory,


cheers - Barry

On 02/11/16 10:02, Hasan Sait ARSLAN wrote:

Hi Hieu,

I did in the way, you want. Plus, I am sure the path and file names 
are correctly spelt.

 But still, I get the same error.

2016-11-01 21:41 GMT+02:00 Hasan Sait ARSLAN 
mailto:hasan.sait.ars...@gmail.com>>:


Thank you Hieu, I will do it, and give you feedback about it.

Kind Regards,

2016-11-01 21:37 GMT+02:00 Hieu Hoang mailto:hieuho...@gmail.com>>:

the command you execute looks ok. the problem is likely to be
with your using tuning data.

You should look at your tuning data, make sure you've spelt
the path and filename correctly, use a very small subset (eg.
start with 1 parallel sentence) and increase the number of
sentences until you find the problem



On 01/11/2016 12:30, Hasan Sait ARSLAN wrote:

Hello,

I have trained dataset for 5 days, and for 2-3 days I am
dealing with the bug in tuning, but still couldn't solve the
problem.

I use the following command to run:

*/home/sait/mosesdecoder/scripts/training/mert-moses.pl

/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr

/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
/home/sait/mosesdecoder/bin/moses
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
--working-dir /home/sait/Kairit/Task3/working_tr-en/mert
--decoder-flags "--threads 32"

*
Then I get the following error:

*Using SCRIPTS_ROOTDIR: /home/sait/mosesdecoder/scripts
Assuming --mertdir=/home/sait/mosesdecoder/bin
filtering the phrase tables... T nov1 19:17:39 EET 2016
exec:
/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr

Executing:
/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
 > filterphrases.out 2> filterphrases.err
Exit code: 255
ERROR: Failed to run
'/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
'. at
/home/sait/mosesdecoder/scripts/training/mert-moses.pl
 line 1748.

*
I have searched over the internet about the error, but
unfortunately I couldn't find any beneficial solution.

It is so annoying, and eats my time, please help me to solve
my problem.

Thanks,


___
Moses-support mailing list
Moses-support@mit.edu  
http://mailman.mit.edu/mailman/listinfo/moses-support  







___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] error in mosesdecoder tuning

[Hasan Sait ARSLAN]

Hi Hieu,

I did in the way, you want. Plus, I am sure the path and file names are
correctly spelt.
 But still, I get the same error.

2016-11-01 21:41 GMT+02:00 Hasan Sait ARSLAN :

> Thank you Hieu, I will do it, and give you feedback about it.
>
> Kind Regards,
>
> 2016-11-01 21:37 GMT+02:00 Hieu Hoang :
>
>> the command you execute looks ok. the problem is likely to be with your
>> using tuning data.
>>
>> You should look at your tuning data, make sure you've spelt the path and
>> filename correctly, use a very small subset (eg. start with 1 parallel
>> sentence) and increase the number of sentences until you find the problem
>>
>>
>>
>> On 01/11/2016 12:30, Hasan Sait ARSLAN wrote:
>>
>> Hello,
>>
>> I have trained dataset for 5 days, and for 2-3 days I am dealing with the
>> bug in tuning, but still couldn't solve the problem.
>>
>> I use the following command to run:
>>
>>
>>
>> */home/sait/mosesdecoder/scripts/training/mert-moses.pl
>>  /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>  /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
>> /home/sait/mosesdecoder/bin/moses
>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini --working-dir
>> /home/sait/Kairit/Task3/working_tr-en/mert --decoder-flags "--threads 32" *
>> Then I get the following error:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Using SCRIPTS_ROOTDIR: /home/sait/mosesdecoder/scripts Assuming
>> --mertdir=/home/sait/mosesdecoder/bin filtering the phrase tables... T
>> nov1 19:17:39 EET 2016 exec:
>> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>  ./filtered
>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr 
>> Executing:
>> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>  ./filtered
>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr  >
>> filterphrases.out 2> filterphrases.err Exit code: 255 ERROR: Failed to run
>> '/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>  ./filtered
>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr '.
>> at /home/sait/mosesdecoder/scripts/training/mert-moses.pl
>>  line 1748. *
>> I have searched over the internet about the error, but unfortunately I
>> couldn't find any beneficial solution.
>>
>> It is so annoying, and eats my time, please help me to solve my problem.
>>
>> Thanks,
>>
>>
>> ___
>> Moses-support mailing 
>> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] error in mosesdecoder tuning

[Hasan Sait ARSLAN]

Thank you Hieu, I will do it, and give you feedback about it.

Kind Regards,

2016-11-01 21:37 GMT+02:00 Hieu Hoang :

> the command you execute looks ok. the problem is likely to be with your
> using tuning data.
>
> You should look at your tuning data, make sure you've spelt the path and
> filename correctly, use a very small subset (eg. start with 1 parallel
> sentence) and increase the number of sentences until you find the problem
>
>
>
> On 01/11/2016 12:30, Hasan Sait ARSLAN wrote:
>
> Hello,
>
> I have trained dataset for 5 days, and for 2-3 days I am dealing with the
> bug in tuning, but still couldn't solve the problem.
>
> I use the following command to run:
>
>
>
> */home/sait/mosesdecoder/scripts/training/mert-moses.pl
>  /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>  /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
> /home/sait/mosesdecoder/bin/moses
> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini --working-dir
> /home/sait/Kairit/Task3/working_tr-en/mert --decoder-flags "--threads 32" *
> Then I get the following error:
>
>
>
>
>
>
>
>
>
> *Using SCRIPTS_ROOTDIR: /home/sait/mosesdecoder/scripts Assuming
> --mertdir=/home/sait/mosesdecoder/bin filtering the phrase tables... T
> nov1 19:17:39 EET 2016 exec:
> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>  ./filtered
> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr 
> Executing:
> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>  ./filtered
> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr  >
> filterphrases.out 2> filterphrases.err Exit code: 255 ERROR: Failed to run
> '/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>  ./filtered
> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr '.
> at /home/sait/mosesdecoder/scripts/training/mert-moses.pl
>  line 1748. *
> I have searched over the internet about the error, but unfortunately I
> couldn't find any beneficial solution.
>
> It is so annoying, and eats my time, please help me to solve my problem.
>
> Thanks,
>
>
> ___
> Moses-support mailing 
> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] error in mosesdecoder tuning

[Hieu Hoang]

the command you execute looks ok. the problem is likely to be with your 
using tuning data.


You should look at your tuning data, make sure you've spelt the path and 
filename correctly, use a very small subset (eg. start with 1 parallel 
sentence) and increase the number of sentences until you find the problem




On 01/11/2016 12:30, Hasan Sait ARSLAN wrote:

Hello,

I have trained dataset for 5 days, and for 2-3 days I am dealing with 
the bug in tuning, but still couldn't solve the problem.


I use the following command to run:

*/home/sait/mosesdecoder/scripts/training/mert-moses.pl 
 
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr 
 
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en 
/home/sait/mosesdecoder/bin/moses 
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini 
--working-dir /home/sait/Kairit/Task3/working_tr-en/mert 
--decoder-flags "--threads 32"


*
Then I get the following error:

*Using SCRIPTS_ROOTDIR: /home/sait/mosesdecoder/scripts
Assuming --mertdir=/home/sait/mosesdecoder/bin
filtering the phrase tables... T nov1 19:17:39 EET 2016
exec: 
/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl 
 ./filtered 
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini 
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr 
Executing: 
/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl 
 ./filtered 
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini 
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr 
 > filterphrases.out 2> filterphrases.err

Exit code: 255
ERROR: Failed to run 
'/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl 
 ./filtered 
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini 
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr 
'. at 
/home/sait/mosesdecoder/scripts/training/mert-moses.pl 
 line 1748.


*
I have searched over the internet about the error, but unfortunately I 
couldn't find any beneficial solution.


It is so annoying, and eats my time, please help me to solve my problem.

Thanks,


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] error in mosesdecoder tuning

[Hasan Sait ARSLAN]

Hello,

I have trained dataset for 5 days, and for 2-3 days I am dealing with the
bug in tuning, but still couldn't solve the problem.

I use the following command to run:



*/home/sait/mosesdecoder/scripts/training/mert-moses.pl
 /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
 /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
/home/sait/mosesdecoder/bin/moses
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini --working-dir
/home/sait/Kairit/Task3/working_tr-en/mert --decoder-flags "--threads 32"*
Then I get the following error:









*Using SCRIPTS_ROOTDIR: /home/sait/mosesdecoder/scriptsAssuming
--mertdir=/home/sait/mosesdecoder/binfiltering the phrase tables... T
nov1 19:17:39 EET 2016exec:
/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
Executing:
/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr  >
filterphrases.out 2> filterphrases.errExit code: 255ERROR: Failed to run
'/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered
/home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
/home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr '.
at /home/sait/mosesdecoder/scripts/training/mert-moses.pl
 line 1748.*
I have searched over the internet about the error, but unfortunately I
couldn't find any beneficial solution.

It is so annoying, and eats my time, please help me to solve my problem.

Thanks,
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error while running sample cases

[Hieu Hoang]

what was the exact command you used to create the moses binaries? If you 
compiled your own boost libraries, what was the exact command you used 
to create the boost libraries?


The error means you don't have the RIGHT boost library. I would 
recommend creating static boost libraries only so that they are included 
in the moses executable, rather than having to be loaded separately.


The install page here tells you how to do that
   http://www.statmt.org/moses/?n=Development.GetStarted

On 21/09/2016 13:59, Pablo Gonzalez wrote:
Thank you very much for your quick response last time, I decided to 
switch to Linux and could finally compile Moses, however, after 
installing all of the dependencies, when I try to run the tutorial

~/mosesdecoder/bin/moses -f phrase-model/moses.ini < phrase-model/in > out

It fails with error message

/home/pablo/mosesdecoder/bin/moses: error while loading shared libraries: 
libboost_serialization.so.1.59.0:
cannot open shared object file: No such file or directory

I thought this meant I didn't have the whole libboost installed or up to date 
but when I run

$ sudo apt-get install libboost-all-dev

I get

Reading package lists... Done
Building dependency tree
Reading state information... Done
libboost-all-dev is already the newest version (1.58.0.1ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 153 not upgraded.

So it seems to be all there and up to date, am I supposed to tell Moses how to 
find it and if so how?
Anyone know how to deal with this? I would really appreciate it
Thanks
Pablo Gonzalez


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error while running sample cases

[Pablo Gonzalez]

Thank you very much for your quick response last time, I decided to switch
to Linux and could finally compile Moses, however, after installing all of
the dependencies, when I try to run the tutorial

~/mosesdecoder/bin/moses -f phrase-model/moses.ini < phrase-model/in > out

It fails with error message

/home/pablo/mosesdecoder/bin/moses: error while loading shared
libraries: libboost_serialization.so.1.59.0:
cannot open shared object file: No such file or directory

I thought this meant I didn't have the whole libboost installed or up
to date but when I run

$ sudo apt-get install libboost-all-dev

I get

Reading package lists... Done
Building dependency tree
Reading state information... Done
libboost-all-dev is already the newest version (1.58.0.1ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 153 not upgraded.

So it seems to be all there and up to date, am I supposed to tell
Moses how to find it and if so how?
Anyone know how to deal with this? I would really appreciate it


Thanks

Pablo Gonzalez
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Error when Filtering Binarised Model

[Steve Braich]

Everything checked out.  BLEU is 23.33, which is close enough to what the
instructions say (23.5).

Thank you!
Steve

On Fri, Sep 16, 2016 at 2:20 AM, Steve Braich  wrote:

> Ah, that makes sense.  However, the instructions seem to lead you down the
> path I took of changing the moses.ini file to use the binary phrase table,
> to test how fast it is, THEN it does the filtering step.
>
> Anyway, that resolved it.  I am running the BLEU on it now.
>
> Thanks Hieu!
> Steve
>
> On Fri, Sep 16, 2016 at 2:17 AM, Hieu Hoang  wrote:
>
>> it will binarize AFTER it filters. It can't filter already binarized
>> tables
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 16 September 2016 at 11:10, Steve Braich  wrote:
>>
>>> Hi Hieu,
>>>
>>> OK, but then why does the filtering script have an option for using the
>>> binary "-Binarizer" Phrase Table?
>>>
>>> Thanks,
>>> Steve
>>>
>>> *~/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>  filtered-newstest2011
>>> mert-work/moses.ini ~/corpus/newstest2011.true.fr
>>>  -Binarizer
>>> ~/mosesdecoder/bin/processPhraseTableMin*
>>>
>>> On Fri, Sep 16, 2016 at 1:49 AM, Hieu Hoang  wrote:
>>>
 ah. from the moses.ini file, it looks like you've binarized the
 phrase-table (and the re-ordering model?) already.

 The filtering script only works with moses.ini which has the original
 text pt and reordering files

 Hieu Hoang
 http://www.hoang.co.uk/hieu

 On 16 September 2016 at 09:13, Steve Braich 
 wrote:

> I included the moses.ini file in the original post.
>
> I checked in VIM using :set list.  Everything looks good.  I repost it
> here:
>
> # MERT optimized configuration
> # decoder /home/steve/mosesdecoder/bin/moses
> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr
> # We were before running iteration 5
> # finished Thu Sep 15 01:53:22 PDT 2016
> ### MOSES CONFIG FILE ###
> #
>
> # input factors
> [input-factors]
> 0
>
> # mapping steps
> [mapping]
> 0 T 0
>
> [distortion-limit]
> 6
>
> # feature functions
> [feature]
> UnknownWordPenalty
> WordPenalty
> PhrasePenalty
> PhraseDictionaryCompact name=TranslationModel0 num-features=4
> path=/home/steve/working/binarised-model/phrase-table.minphr
> input-factor=0 output-factor=0
> LexicalReordering name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/home/steve/working/binarised-model/reordering-table
> Distortion
> KENLM name=LM0 factor=0 
> path=/home/steve/lm/news-commentary-v8.fr-en.blm.en
> order=3
>
> # dense weights for feature functions
>
> [threads]
> 4
> [weight]
>
> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898
> 0.0348721
> Distortion0= 0.0721205
> LM0= 0.0908638
> WordPenalty0= -0.153776
> PhrasePenalty0= 0.0874059
> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132
> UnknownWordPenalty0= 1
>
> On Fri, Sep 16, 2016 at 12:10 AM, Hieu Hoang 
> wrote:
>
>> look inside the file
>>mert-work/moses.ini
>> is there anything strange going on?
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 16 September 2016 at 09:02, Steve Braich 
>> wrote:
>>
>>> Hi Hieu,
>>> I changed from relative path to absolute.  Still same error.
>>>
>>> Thanks,
>>> Steve
>>>
>>> On Thu, Sep 15, 2016 at 8:54 PM, Hieu Hoang 
>>> wrote:
>>>
 don't use relative paths, ie. change
filtered-newstest2011 -> /whatever/filtered-newstest2011
   mert-work/moses.ini -> /whatever/mert-work/moses.ini


 Hieu Hoang
 http://www.hoang.co.uk/hieu

 On 16 September 2016 at 05:42, Steve Braich 
 wrote:

> Hello,
>
> I am going thru the steps described in the baseline system
> instructions.  I have no problems until the very end when I try to 
> filter
> out the test data from the binarised model.
>
>
>
> Here is the step in the instructions where the error appears:
>
> *The model that we've trained can then be filtered for this test
> set, meaning that we only retain the entries needed translate the 
> test set.
> This will make the translation a lot faster.*
>
>
>
> *cd ~/working*
>
> *~/mosesdecoder/scripts/training/filter-model-given-input.pl
>  filtered-newstest2011
> mert-work/moses.ini ~/corpus/newstest2011.true.fr
>  -Binarizer
> ~/moses

Re: [Moses-support] Error when Filtering Binarised Model

[Steve Braich]

Ah, that makes sense.  However, the instructions seem to lead you down the
path I took of changing the moses.ini file to use the binary phrase table,
to test how fast it is, THEN it does the filtering step.

Anyway, that resolved it.  I am running the BLEU on it now.

Thanks Hieu!
Steve

On Fri, Sep 16, 2016 at 2:17 AM, Hieu Hoang  wrote:

> it will binarize AFTER it filters. It can't filter already binarized tables
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 16 September 2016 at 11:10, Steve Braich  wrote:
>
>> Hi Hieu,
>>
>> OK, but then why does the filtering script have an option for using the
>> binary "-Binarizer" Phrase Table?
>>
>> Thanks,
>> Steve
>>
>> *~/mosesdecoder/scripts/training/filter-model-given-input.pl
>>  filtered-newstest2011
>> mert-work/moses.ini ~/corpus/newstest2011.true.fr
>>  -Binarizer
>> ~/mosesdecoder/bin/processPhraseTableMin*
>>
>> On Fri, Sep 16, 2016 at 1:49 AM, Hieu Hoang  wrote:
>>
>>> ah. from the moses.ini file, it looks like you've binarized the
>>> phrase-table (and the re-ordering model?) already.
>>>
>>> The filtering script only works with moses.ini which has the original
>>> text pt and reordering files
>>>
>>> Hieu Hoang
>>> http://www.hoang.co.uk/hieu
>>>
>>> On 16 September 2016 at 09:13, Steve Braich  wrote:
>>>
 I included the moses.ini file in the original post.

 I checked in VIM using :set list.  Everything looks good.  I repost it
 here:

 # MERT optimized configuration
 # decoder /home/steve/mosesdecoder/bin/moses
 # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr
 # We were before running iteration 5
 # finished Thu Sep 15 01:53:22 PDT 2016
 ### MOSES CONFIG FILE ###
 #

 # input factors
 [input-factors]
 0

 # mapping steps
 [mapping]
 0 T 0

 [distortion-limit]
 6

 # feature functions
 [feature]
 UnknownWordPenalty
 WordPenalty
 PhrasePenalty
 PhraseDictionaryCompact name=TranslationModel0 num-features=4
 path=/home/steve/working/binarised-model/phrase-table.minphr
 input-factor=0 output-factor=0
 LexicalReordering name=LexicalReordering0 num-features=6
 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
 path=/home/steve/working/binarised-model/reordering-table
 Distortion
 KENLM name=LM0 factor=0 path=/home/steve/lm/news-commentary-v8.fr-en.blm.en
 order=3

 # dense weights for feature functions

 [threads]
 4
 [weight]

 LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898
 0.0348721
 Distortion0= 0.0721205
 LM0= 0.0908638
 WordPenalty0= -0.153776
 PhrasePenalty0= 0.0874059
 TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132
 UnknownWordPenalty0= 1

 On Fri, Sep 16, 2016 at 12:10 AM, Hieu Hoang 
 wrote:

> look inside the file
>mert-work/moses.ini
> is there anything strange going on?
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 16 September 2016 at 09:02, Steve Braich 
> wrote:
>
>> Hi Hieu,
>> I changed from relative path to absolute.  Still same error.
>>
>> Thanks,
>> Steve
>>
>> On Thu, Sep 15, 2016 at 8:54 PM, Hieu Hoang 
>> wrote:
>>
>>> don't use relative paths, ie. change
>>>filtered-newstest2011 -> /whatever/filtered-newstest2011
>>>   mert-work/moses.ini -> /whatever/mert-work/moses.ini
>>>
>>>
>>> Hieu Hoang
>>> http://www.hoang.co.uk/hieu
>>>
>>> On 16 September 2016 at 05:42, Steve Braich 
>>> wrote:
>>>
 Hello,

 I am going thru the steps described in the baseline system
 instructions.  I have no problems until the very end when I try to 
 filter
 out the test data from the binarised model.



 Here is the step in the instructions where the error appears:

 *The model that we've trained can then be filtered for this test
 set, meaning that we only retain the entries needed translate the test 
 set.
 This will make the translation a lot faster.*



 *cd ~/working*

 *~/mosesdecoder/scripts/training/filter-model-given-input.pl
  filtered-newstest2011
 mert-work/moses.ini ~/corpus/newstest2011.true.fr
  -Binarizer
 ~/mosesdecoder/bin/processPhraseTableMin*



 Here is the error message:

 *Can't open '<
 /home/steve/working/binarised-model/reordering-table' at
 /home/steve/mosesdecoder/scripts/training/filter-model-given-input.pl
  line 439.*


>

Re: [Moses-support] Error when Filtering Binarised Model

[Hieu Hoang]

it will binarize AFTER it filters. It can't filter already binarized tables


On 16/09/2016 11:10, Steve Braich wrote:

Hi Hieu,

OK, but then why does the filtering script have an option for using 
the binary "-Binarizer" Phrase Table?


Thanks,
Steve

*~/mosesdecoder/scripts/training/filter-model-given-input.pl 
 filtered-newstest2011 
mert-work/moses.ini ~/corpus/newstest2011.true.fr 
 -Binarizer 
~/mosesdecoder/bin/processPhraseTableMin*


On Fri, Sep 16, 2016 at 1:49 AM, Hieu Hoang > wrote:


ah. from the moses.ini file, it looks like you've binarized the
phrase-table (and the re-ordering model?) already.

The filtering script only works with moses.ini which has the
original text pt and reordering files

Hieu Hoang
http://www.hoang.co.uk/hieu

On 16 September 2016 at 09:13, Steve Braich mailto:steveb...@gmail.com>> wrote:

I included the moses.ini file in the original post.

I checked in VIM using :set list. Everything looks good.  I
repost it here:

# MERT optimized configuration
# decoder /home/steve/mosesdecoder/bin/moses
# BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr

# We were before running iteration 5
# finished Thu Sep 15 01:53:22 PDT 2016
### MOSES CONFIG FILE ###
#

# input factors
[input-factors]
0

# mapping steps
[mapping]
0 T 0

[distortion-limit]
6

# feature functions
[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryCompact name=TranslationModel0 num-features=4
path=/home/steve/working/binarised-model/phrase-table.minphr
input-factor=0 output-factor=0
LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0
output-factor=0
path=/home/steve/working/binarised-model/reordering-table
Distortion
KENLM name=LM0 factor=0
path=/home/steve/lm/news-commentary-v8.fr-en.blm.en order=3

# dense weights for feature functions

[threads]
4
[weight]

LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621
0.0141898 0.0348721
Distortion0= 0.0721205
LM0= 0.0908638
WordPenalty0= -0.153776
PhrasePenalty0= 0.0874059
TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132
UnknownWordPenalty0= 1

On Fri, Sep 16, 2016 at 12:10 AM, Hieu Hoang
mailto:hieuho...@gmail.com>> wrote:

look inside the file
mert-work/moses.ini
is there anything strange going on? *
*

Hieu Hoang
http://www.hoang.co.uk/hieu

On 16 September 2016 at 09:02, Steve Braich
mailto:steveb...@gmail.com>> wrote:

Hi Hieu,
I changed from relative path to absolute.  Still same
error.

Thanks,
Steve

On Thu, Sep 15, 2016 at 8:54 PM, Hieu Hoang
mailto:hieuho...@gmail.com>> wrote:

don't use relative paths, ie. change
filtered-newstest2011 ->
/whatever/filtered-newstest2011
mert-work/moses.ini -> /whatever/mert-work/moses.ini
*
*

Hieu Hoang
http://www.hoang.co.uk/hieu

On 16 September 2016 at 05:42, Steve Braich
mailto:steveb...@gmail.com>>
wrote:

Hello,

I am going thru the steps described in the
baseline system instructions. I have no
problems until the very end when I try to
filter out the test data from the binarised model.

Here is the step in the instructions where the
error appears:

*The model that we've trained can then be
filtered for this test set, meaning that we
only retain the entries needed translate the
test set. This will make the translation a lot
faster.*

**

*cd ~/working*


*~/mosesdecoder/scripts/training/filter-model-given-input.pl

filtered-newstest2011 mert-work/moses.ini
~/corpus/newstest2011.true.fr
 -Binarizer
~/mosesdeco

Re: [Moses-support] Error when Filtering Binarised Model

[Hieu Hoang]

it will binarize AFTER it filters. It can't filter already binarized tables

Hieu Hoang
http://www.hoang.co.uk/hieu

On 16 September 2016 at 11:10, Steve Braich  wrote:

> Hi Hieu,
>
> OK, but then why does the filtering script have an option for using the
> binary "-Binarizer" Phrase Table?
>
> Thanks,
> Steve
>
> *~/mosesdecoder/scripts/training/filter-model-given-input.pl
>  filtered-newstest2011
> mert-work/moses.ini ~/corpus/newstest2011.true.fr
>  -Binarizer
> ~/mosesdecoder/bin/processPhraseTableMin*
>
> On Fri, Sep 16, 2016 at 1:49 AM, Hieu Hoang  wrote:
>
>> ah. from the moses.ini file, it looks like you've binarized the
>> phrase-table (and the re-ordering model?) already.
>>
>> The filtering script only works with moses.ini which has the original
>> text pt and reordering files
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 16 September 2016 at 09:13, Steve Braich  wrote:
>>
>>> I included the moses.ini file in the original post.
>>>
>>> I checked in VIM using :set list.  Everything looks good.  I repost it
>>> here:
>>>
>>> # MERT optimized configuration
>>> # decoder /home/steve/mosesdecoder/bin/moses
>>> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr
>>> # We were before running iteration 5
>>> # finished Thu Sep 15 01:53:22 PDT 2016
>>> ### MOSES CONFIG FILE ###
>>> #
>>>
>>> # input factors
>>> [input-factors]
>>> 0
>>>
>>> # mapping steps
>>> [mapping]
>>> 0 T 0
>>>
>>> [distortion-limit]
>>> 6
>>>
>>> # feature functions
>>> [feature]
>>> UnknownWordPenalty
>>> WordPenalty
>>> PhrasePenalty
>>> PhraseDictionaryCompact name=TranslationModel0 num-features=4
>>> path=/home/steve/working/binarised-model/phrase-table.minphr
>>> input-factor=0 output-factor=0
>>> LexicalReordering name=LexicalReordering0 num-features=6
>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>>> path=/home/steve/working/binarised-model/reordering-table
>>> Distortion
>>> KENLM name=LM0 factor=0 path=/home/steve/lm/news-commentary-v8.fr-en.blm.en
>>> order=3
>>>
>>> # dense weights for feature functions
>>>
>>> [threads]
>>> 4
>>> [weight]
>>>
>>> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898
>>> 0.0348721
>>> Distortion0= 0.0721205
>>> LM0= 0.0908638
>>> WordPenalty0= -0.153776
>>> PhrasePenalty0= 0.0874059
>>> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132
>>> UnknownWordPenalty0= 1
>>>
>>> On Fri, Sep 16, 2016 at 12:10 AM, Hieu Hoang 
>>> wrote:
>>>
 look inside the file
mert-work/moses.ini
 is there anything strange going on?

 Hieu Hoang
 http://www.hoang.co.uk/hieu

 On 16 September 2016 at 09:02, Steve Braich 
 wrote:

> Hi Hieu,
> I changed from relative path to absolute.  Still same error.
>
> Thanks,
> Steve
>
> On Thu, Sep 15, 2016 at 8:54 PM, Hieu Hoang 
> wrote:
>
>> don't use relative paths, ie. change
>>filtered-newstest2011 -> /whatever/filtered-newstest2011
>>   mert-work/moses.ini -> /whatever/mert-work/moses.ini
>>
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 16 September 2016 at 05:42, Steve Braich 
>> wrote:
>>
>>> Hello,
>>>
>>> I am going thru the steps described in the baseline system
>>> instructions.  I have no problems until the very end when I try to 
>>> filter
>>> out the test data from the binarised model.
>>>
>>>
>>>
>>> Here is the step in the instructions where the error appears:
>>>
>>> *The model that we've trained can then be filtered for this test
>>> set, meaning that we only retain the entries needed translate the test 
>>> set.
>>> This will make the translation a lot faster.*
>>>
>>>
>>>
>>> *cd ~/working*
>>>
>>> *~/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>  filtered-newstest2011
>>> mert-work/moses.ini ~/corpus/newstest2011.true.fr
>>>  -Binarizer
>>> ~/mosesdecoder/bin/processPhraseTableMin*
>>>
>>>
>>>
>>> Here is the error message:
>>>
>>> *Can't open '< /home/steve/working/binarised-model/reordering-table'
>>> at /home/steve/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>  line 439.*
>>>
>>>
>>>
>>> Keep in mind the following:
>>>
>>>- I compiled moses with the following parameters.  It is the
>>>same compilation.  I did not recompile moses at any point during the
>>>process.
>>>./bjam -a --with-boost=/home/steve/mosesdecoder/boost_1_55_0
>>>--with-cmph=$HOME/mosesdecoder/cmph/cmph-2.0
>>>NOTE: I did try recompiling with the absolute path for cmph.
>>>Same error.
>>>- I checked all paths first to make sure they exist.
>

Re: [Moses-support] Error when Filtering Binarised Model

[Steve Braich]

Hi Hieu,

OK, but then why does the filtering script have an option for using the
binary "-Binarizer" Phrase Table?

Thanks,
Steve

*~/mosesdecoder/scripts/training/filter-model-given-input.pl
 filtered-newstest2011
mert-work/moses.ini ~/corpus/newstest2011.true.fr
 -Binarizer
~/mosesdecoder/bin/processPhraseTableMin*

On Fri, Sep 16, 2016 at 1:49 AM, Hieu Hoang  wrote:

> ah. from the moses.ini file, it looks like you've binarized the
> phrase-table (and the re-ordering model?) already.
>
> The filtering script only works with moses.ini which has the original text
> pt and reordering files
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 16 September 2016 at 09:13, Steve Braich  wrote:
>
>> I included the moses.ini file in the original post.
>>
>> I checked in VIM using :set list.  Everything looks good.  I repost it
>> here:
>>
>> # MERT optimized configuration
>> # decoder /home/steve/mosesdecoder/bin/moses
>> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr
>> # We were before running iteration 5
>> # finished Thu Sep 15 01:53:22 PDT 2016
>> ### MOSES CONFIG FILE ###
>> #
>>
>> # input factors
>> [input-factors]
>> 0
>>
>> # mapping steps
>> [mapping]
>> 0 T 0
>>
>> [distortion-limit]
>> 6
>>
>> # feature functions
>> [feature]
>> UnknownWordPenalty
>> WordPenalty
>> PhrasePenalty
>> PhraseDictionaryCompact name=TranslationModel0 num-features=4
>> path=/home/steve/working/binarised-model/phrase-table.minphr
>> input-factor=0 output-factor=0
>> LexicalReordering name=LexicalReordering0 num-features=6
>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>> path=/home/steve/working/binarised-model/reordering-table
>> Distortion
>> KENLM name=LM0 factor=0 path=/home/steve/lm/news-commentary-v8.fr-en.blm.en
>> order=3
>>
>> # dense weights for feature functions
>>
>> [threads]
>> 4
>> [weight]
>>
>> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898
>> 0.0348721
>> Distortion0= 0.0721205
>> LM0= 0.0908638
>> WordPenalty0= -0.153776
>> PhrasePenalty0= 0.0874059
>> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132
>> UnknownWordPenalty0= 1
>>
>> On Fri, Sep 16, 2016 at 12:10 AM, Hieu Hoang  wrote:
>>
>>> look inside the file
>>>mert-work/moses.ini
>>> is there anything strange going on?
>>>
>>> Hieu Hoang
>>> http://www.hoang.co.uk/hieu
>>>
>>> On 16 September 2016 at 09:02, Steve Braich  wrote:
>>>
 Hi Hieu,
 I changed from relative path to absolute.  Still same error.

 Thanks,
 Steve

 On Thu, Sep 15, 2016 at 8:54 PM, Hieu Hoang 
 wrote:

> don't use relative paths, ie. change
>filtered-newstest2011 -> /whatever/filtered-newstest2011
>   mert-work/moses.ini -> /whatever/mert-work/moses.ini
>
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 16 September 2016 at 05:42, Steve Braich 
> wrote:
>
>> Hello,
>>
>> I am going thru the steps described in the baseline system
>> instructions.  I have no problems until the very end when I try to filter
>> out the test data from the binarised model.
>>
>>
>>
>> Here is the step in the instructions where the error appears:
>>
>> *The model that we've trained can then be filtered for this test set,
>> meaning that we only retain the entries needed translate the test set. 
>> This
>> will make the translation a lot faster.*
>>
>>
>>
>> *cd ~/working*
>>
>> *~/mosesdecoder/scripts/training/filter-model-given-input.pl
>>  filtered-newstest2011
>> mert-work/moses.ini ~/corpus/newstest2011.true.fr
>>  -Binarizer
>> ~/mosesdecoder/bin/processPhraseTableMin*
>>
>>
>>
>> Here is the error message:
>>
>> *Can't open '< /home/steve/working/binarised-model/reordering-table'
>> at /home/steve/mosesdecoder/scripts/training/filter-model-given-input.pl
>>  line 439.*
>>
>>
>>
>> Keep in mind the following:
>>
>>- I compiled moses with the following parameters.  It is the same
>>compilation.  I did not recompile moses at any point during the 
>> process.
>>./bjam -a --with-boost=/home/steve/mosesdecoder/boost_1_55_0
>>--with-cmph=$HOME/mosesdecoder/cmph/cmph-2.0
>>NOTE: I did try recompiling with the absolute path for cmph.
>>Same error.
>>- I checked all paths first to make sure they exist.
>>- I am able to translate segments using the binarised model with
>>no problem
>>- I provided my moses.ini file below.
>>
>> Questions:
>>
>>- Can I recompile moses after training and/or tuning, and expect
>>that it should work for testing?  I ask because I want to know if the
>>solution is to mod

Re: [Moses-support] Error when Filtering Binarised Model

[Hieu Hoang]

ah. from the moses.ini file, it looks like you've binarized the
phrase-table (and the re-ordering model?) already.

The filtering script only works with moses.ini which has the original text
pt and reordering files

Hieu Hoang
http://www.hoang.co.uk/hieu

On 16 September 2016 at 09:13, Steve Braich  wrote:

> I included the moses.ini file in the original post.
>
> I checked in VIM using :set list.  Everything looks good.  I repost it
> here:
>
> # MERT optimized configuration
> # decoder /home/steve/mosesdecoder/bin/moses
> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr
> # We were before running iteration 5
> # finished Thu Sep 15 01:53:22 PDT 2016
> ### MOSES CONFIG FILE ###
> #
>
> # input factors
> [input-factors]
> 0
>
> # mapping steps
> [mapping]
> 0 T 0
>
> [distortion-limit]
> 6
>
> # feature functions
> [feature]
> UnknownWordPenalty
> WordPenalty
> PhrasePenalty
> PhraseDictionaryCompact name=TranslationModel0 num-features=4
> path=/home/steve/working/binarised-model/phrase-table.minphr
> input-factor=0 output-factor=0
> LexicalReordering name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/home/steve/working/binarised-model/reordering-table
> Distortion
> KENLM name=LM0 factor=0 path=/home/steve/lm/news-commentary-v8.fr-en.blm.en
> order=3
>
> # dense weights for feature functions
>
> [threads]
> 4
> [weight]
>
> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898
> 0.0348721
> Distortion0= 0.0721205
> LM0= 0.0908638
> WordPenalty0= -0.153776
> PhrasePenalty0= 0.0874059
> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132
> UnknownWordPenalty0= 1
>
> On Fri, Sep 16, 2016 at 12:10 AM, Hieu Hoang  wrote:
>
>> look inside the file
>>mert-work/moses.ini
>> is there anything strange going on?
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 16 September 2016 at 09:02, Steve Braich  wrote:
>>
>>> Hi Hieu,
>>> I changed from relative path to absolute.  Still same error.
>>>
>>> Thanks,
>>> Steve
>>>
>>> On Thu, Sep 15, 2016 at 8:54 PM, Hieu Hoang  wrote:
>>>
 don't use relative paths, ie. change
filtered-newstest2011 -> /whatever/filtered-newstest2011
   mert-work/moses.ini -> /whatever/mert-work/moses.ini


 Hieu Hoang
 http://www.hoang.co.uk/hieu

 On 16 September 2016 at 05:42, Steve Braich 
 wrote:

> Hello,
>
> I am going thru the steps described in the baseline system
> instructions.  I have no problems until the very end when I try to filter
> out the test data from the binarised model.
>
>
>
> Here is the step in the instructions where the error appears:
>
> *The model that we've trained can then be filtered for this test set,
> meaning that we only retain the entries needed translate the test set. 
> This
> will make the translation a lot faster.*
>
>
>
> *cd ~/working*
>
> *~/mosesdecoder/scripts/training/filter-model-given-input.pl
>  filtered-newstest2011
> mert-work/moses.ini ~/corpus/newstest2011.true.fr
>  -Binarizer
> ~/mosesdecoder/bin/processPhraseTableMin*
>
>
>
> Here is the error message:
>
> *Can't open '< /home/steve/working/binarised-model/reordering-table'
> at /home/steve/mosesdecoder/scripts/training/filter-model-given-input.pl
>  line 439.*
>
>
>
> Keep in mind the following:
>
>- I compiled moses with the following parameters.  It is the same
>compilation.  I did not recompile moses at any point during the 
> process.
>./bjam -a --with-boost=/home/steve/mosesdecoder/boost_1_55_0
>--with-cmph=$HOME/mosesdecoder/cmph/cmph-2.0
>NOTE: I did try recompiling with the absolute path for cmph.  Same
>error.
>- I checked all paths first to make sure they exist.
>- I am able to translate segments using the binarised model with
>no problem
>- I provided my moses.ini file below.
>
> Questions:
>
>- Can I recompile moses after training and/or tuning, and expect
>that it should work for testing?  I ask because I want to know if the
>solution is to modify the compile options, do I have to start from 
> scratch
>again?  (Can you filter a binarised model using moses with different
>compile options that were used to train the model?)
>
> Thanks,
> Steve
>
>
> Moses.ini file
>
> # MERT optimized configuration
>
> # decoder /home/steve/mosesdecoder/bin/moses
>
> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr
>
> # We were before running iteration 5
>
> # finished Thu Sep 15 01:53:22 PDT 2016
>
> ### MOSES CONFIG FILE ###
>
> ###