Re: [Moses-support] apostrophe: detokenization or corpus issue ?

[Philipp Koehn]

Hi,

I do not think that the detokenizer would cause conversion of ' to ".
You can check the raw output of the decoder, and see how it is
changed by the detokenizer.

-phi

On Wed, Mar 9, 2016 at 11:44 AM, Vincent Nguyen  wrote:

> Hi,
>
> I got the following situation:
>
> This group age
> is translated sometimes in:
> ce groupe d'âge (correct)
> ce groupe d" âge (incorrect)
> ce groupe d "âge (incorrect)
>
> I am wondering if this is more a detokenizer issue or a corpus issue, or
> both.
>
> Technically in French, there shouldn't be any space before or after the
> apostrophe.
> In the Europarl Corpus, as well as in the News2014 one, there are some
> instances with a space before or after.
>
> Then I have the feeling that the decoder gets a ' with surrounding
> spaces leading to the detokenizer to transform into "
>
> Anyone with a similar issue ?
>
> thanks.
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] (no subject)

[Philipp Koehn]

Hi,

it's probably a good idea to use the full path instead of "-root-dir
train". But what is in "training.out"? That should give some clues. Are any
files created?

-phi

On Wed, Mar 9, 2016 at 11:57 PM, BIRENDRA CHAUHAN SINGH <
birendrachauh...@gmail.com> wrote:

> on running this command:
>
> mkdir ~/working
>  cd ~/working
>  nohup nice ~/mosesdecoder/scripts/training/train-model.perl -root-dir train \
>  -corpus ~/corpus/news-commentary-v8.fr-en.clean \
>  -f fr -e en -alignment grow-diag-final-and -reordering msd-bidirectional-fe \
>  -lm 0:3:$HOME/lm/news-commentary-v8.fr-en.blm.en:8  \
>  -external-bin-dir ~/mosesdecoder/tools >& training.out &
>
>
> output:
> [1] 2217
> and this output comes within a sec.It does not take hours to train.I think
> its not do training may be due to some reasons.and not directory 
> ~/working/train/model
> is formed.
> So there is no moses.ini file formed.
>
> Can any one tell me what is happenging and why?
> Thanks.
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Error in setting the path for boost lib

[BIRENDRA CHAUHAN SINGH]

Error:
berry@berry-inspiron-3521:~$ ./bjam
--with-boost=~/workspace/temp/boost_1_55_0 -j4
bash: ./bjam: No such file or directory

before this i ran this command :

wget 
http://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.gz
   tar zxvf boost_1_55_0.tar.gz
   cd boost_1_55_0/
   ./bootstrap.sh
   ./b2 -j4 --prefix=$PWD --libdir=$PWD/lib64 --layout=system
link=static install || echo FAILURE

As there is no directory with this path /workspace/temp/boost_1_55_0
but in home i do have directory with path ~/boost_1_55_0

so what should i do now to tell Moses where boost is with the --with-boost flag.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] apostrophe: detokenization or corpus issue ?

[Vincent Nguyen]

this age group
is decoded as
ce groupe d " âge

I'll check my corpus and see why it got " instead of ' in there.

thanks.


Le 10/03/2016 13:00, Philipp Koehn a écrit :

Hi,

I do not think that the detokenizer would cause conversion of ' to ".
You can check the raw output of the decoder, and see how it is
changed by the detokenizer.

-phi

On Wed, Mar 9, 2016 at 11:44 AM, Vincent Nguyen > wrote:


Hi,

I got the following situation:

This group age
is translated sometimes in:
ce groupe d'âge (correct)
ce groupe d" âge (incorrect)
ce groupe d "âge (incorrect)

I am wondering if this is more a detokenizer issue or a corpus
issue, or
both.

Technically in French, there shouldn't be any space before or
after the
apostrophe.
In the Europarl Corpus, as well as in the News2014 one, there are some
instances with a space before or after.

Then I have the feeling that the decoder gets a ' with
surrounding
spaces leading to the detokenizer to transform into "

Anyone with a similar issue ?

thanks.
___
Moses-support mailing list
Moses-support@mit.edu 
http://mailman.mit.edu/mailman/listinfo/moses-support




___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Missing /usr/lib64/lib64/libboost_thread-mt.so.5 for multithreaded GIZA

[Hieu Hoang]

not sure, to be honest.

If you tire of fighting cmake/make problems, you can try
   ./manual-compile/compile.sh
It has it's own problems, but it's problems that you can see and fix
yourself

Hieu Hoang
http://www.hoang.co.uk/hieu

On 9 March 2016 at 18:20, Sergey A.  wrote:

> Hi and thank you for your time. I also just cloned it from the path you've
> provided, and ran these commands, from the mgizapp directory inside the
> repo. I'm getting this:
>
> [ 94%] Building CXX object src/CMakeFiles/mgiza_lib.dir/vocab.cpp.o
> Linking CXX static library ../lib/libmgiza.a
> [ 94%] Built target mgiza_lib
> Scanning dependencies of target d4norm
> Scanning dependencies of target hmmnorm
> Scanning dependencies of target mgiza
> make[2]: *** No rule to make target
> `/usr/lib64/lib64/libboost_thread-mt.so.5', needed by `bin/d4norm'.  Stop.
> make[2]: *** Waiting for unfinished jobs
> make[2]: *** No rule to make target
> `/usr/lib64/lib64/libboost_thread-mt.so.5', needed by `bin/hmmnorm'.  Stop.
> make[2]: *** Waiting for unfinished jobs
> [ 96%] make[2]: *** No rule to make target
> `/usr/lib64/lib64/libboost_thread-mt.so.5', needed by `bin/mgiza'.  Stop.
> make[2]: *** Waiting for unfinished jobs
> [ 98%] [100%] Building CXX object src/CMakeFiles/d4norm.dir/d4norm.cxx.o
> Building CXX object src/CMakeFiles/hmmnorm.dir/hmmnorm.cxx.o
> Building CXX object src/CMakeFiles/mgiza.dir/main.cpp.o
> make[1]: *** [src/CMakeFiles/hmmnorm.dir/all] Error 2
> make[1]: *** Waiting for unfinished jobs
> make[1]: *** [src/CMakeFiles/d4norm.dir/all] Error 2
> make[1]: *** [src/CMakeFiles/mgiza.dir/all] Error 2
> make: *** [all] Error 2
>
>
> 2016-03-07 11:33 GMT+02:00 Hieu Hoang :
>
>> where did you get the code for mgiza from and what was the exact commands
>> you used to compile?
>>
>> I just tested mgiza compilation with the code from
>>https://github.com/moses-smt/mgiza
>> using the commands
>>cmake .
>>cmake .
>>make -j4
>> It seems to compile ok
>>
>>
>> On 06/03/2016 20:02, Sergey A. wrote:
>>
>> Hello.
>>
>> I'm trying to compile mgiza
>> , but getting
>> the below error. What am I doing wrong? Note that I don't have root access
>> on the machine, so if libraries needed I'd like to know how to download and
>> use them without installing on the server. BTW, I couldn't find anything in
>> the archives
>> .
>>
>> make[2]: *** No rule to make target 
>> `/usr/lib64/lib64/libboost_thread-mt.so.5', needed by `bin/hmmnorm'.  Stop.
>> make[2]: *** Waiting for unfinished jobs
>> make[2]: *** No rule to make target 
>> `/usr/lib64/lib64/libboost_thread-mt.so.5', needed by `bin/d4norm'.  Stop.
>> make[2]: *** Waiting for unfinished jobs
>> make[2]: *** No rule to make target 
>> `/usr/lib64/lib64/libboost_thread-mt.so.5', needed by `bin/mgiza'.  Stop.
>> make[2]: *** Waiting for unfinished jobs
>> [ 96%] [ 98%] [100%] Building CXX object 
>> src/CMakeFiles/hmmnorm.dir/hmmnorm.cxx.o
>> Building CXX object src/CMakeFiles/d4norm.dir/d4norm.cxx.o
>> Building CXX object src/CMakeFiles/mgiza.dir/main.cpp.o
>> make[1]: *** [src/CMakeFiles/hmmnorm.dir/all] Error 2
>> make[1]: *** Waiting for unfinished jobs
>> make[1]: *** [src/CMakeFiles/d4norm.dir/all] Error 2
>> make[1]: *** [src/CMakeFiles/mgiza.dir/all] Error 2make: *** [all] Error 2
>>
>>
>> Thanks in advance, I'll really appreciate any help here.
>>
>> Sergey
>>
>>
>> ___
>> Moses-support mailing 
>> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] problem with return alignment with mosesserver

[Maria Braga]

When using mosesserver, with the -align and -topt flags noticed that the
first element of the list of alignments does not make sense.

Example from en-es: "my blue car"
Alignment:
{'src-start': -1, 'tgt-start': -1, 'src-end': -1, 'tgt-end': -1},
{'src-start': 0, 'tgt-start': 0, 'src-end': 0, 'tgt-end': 1}

When into the TranslationRequest.cpp printed the target GetStartPos() in
add_phrase_an_info(). The mosesserver output is:

[moses/server/TranslationRequest.cpp:288] Input:  my blue car
Translating: my blue car
Line 0: Collecting options took 0.009 seconds at moses/Manager.cpp Line 141
Line 0: Search took 0.091 seconds
[moses/server/TranslationRequest.cpp:374] BEST TRANSLATION: mi coche azul
[111]  [total=-1.508] core= ...

*[moses/server/TranslationRequest.cpp:87] H: [000]  [total=0.000]
core=(0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,*
*0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000*
*)*
*[moses/server/TranslationRequest.cpp:95] ALIGN: -1*
*[moses/server/TranslationRequest.cpp:97] TARGET:
[18446744073709551615..18446744073709551615]*

[moses/server/TranslationRequest.cpp:87] H: mi [100]  [total=-1.322]
core=(0.000,-1.000,1.000,0.000,-0.245,-0.257,-0.435,-0.565,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.0
00,0.000,0.000,0.000,-0.264,0.000,0.000,0.000,0.000,0.000,-0.247,0.000,0.000,0.000,0.000,0.000,-0.185,0.000,0.000,0.000,0.000,0.000,-0.068,0.000,0.000,0.000,0.000,0.000,-6.101,-7.
162,-7.741,-10.450)
[moses/server/TranslationRequest.cpp:95] ALIGN: 0
[moses/server/TranslationRequest.cpp:97] TARGET: [0..0]

[moses/server/TranslationRequest.cpp:87] H: mi coche azul [111]
 [total=-1.508]
core=(0.000,-3.000,2.000,0.000,-0.756,-1.081,-1.712,-2.459,0.000,0.000,0.000,0.000,0.000,0.000,0.00
0,0.000,0.000,0.000,0.000,0.000,-0.341,0.000,0.000,-0.093,0.000,0.000,-0.247,0.000,0.000,-0.320,0.000,0.000,-0.185,0.000,0.000,-0.103,0.000,0.000,-0.068,0.000,0.000,-0.141,0.000,0
.000,-26.601,-29.405,-36.306,-35.151)
[moses/server/TranslationRequest.cpp:95] ALIGN: 1
[moses/server/TranslationRequest.cpp:97] TARGET: [1..2]

In bold the problematic output. Why does it return alignment -1 ?

-- 
Maria Braga
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Installing/compiling Moses - the build failed

[Anna Garbar]

Dear Moses support team,

I tried to install Moses on Windows via cygwin following the instructions
in http://www.statmt.org/moses/?n=Development.GetStarted but have problems
compiling Moses:

The build.log.gz is attached, the exact command I executed when I got the
error was ./bjam -j4

Thanks a lot for your help,
Anna


build.log.gz
Description: GNU Zip compressed data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Training backward LM?

[Michael Denkowski]

Hi Moses LM enthusiasts,

This may be a silly question.  If I want to use the BackwardLanguageModel
feature, what (if anything) should I do to let KenLM know that it's a
backward model?

If I just reverse the words in each sentence and run lmplz/build_binary,
will Moses do the right thing with the model?  Do I need to do anything
special with  and ?

Best,
Michael
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Training backward LM?

[Michael Denkowski]

As a follow-up, how exactly does one activate the BackwardLanguageModel
feature?  It doesn't seem to be registered anywhere although it's been in
Moses for awhile.  Still work in progress?

Best,
Michael

On Thu, Mar 10, 2016 at 4:33 PM, Michael Denkowski <
michael.j.denkow...@gmail.com> wrote:

> Hi Moses LM enthusiasts,
>
> This may be a silly question.  If I want to use the BackwardLanguageModel
> feature, what (if anything) should I do to let KenLM know that it's a
> backward model?
>
> If I just reverse the words in each sentence and run lmplz/build_binary,
> will Moses do the right thing with the model?  Do I need to do anything
> special with  and ?
>
> Best,
> Michael
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] problem while testing moses

[sofiane bouzaher]

Hi,we are trying use the moses toolkit for a Rapid Construction of
Supervised Word Sense Disambiguation System for Lesser-resourced Languages
(arabic) we use the virtual machine (ubuntu 14.04 32-bit.ova30-Jan-2015
16:41 7.2G), we we have compiled moses withs cmph2.0 and boost_ 1_57_0  as
recommanded alla steps passed succufully, but when we try to run Moses with
compact phrase table using
"/home/hieu/workspace/github/mosesdecoder/bin/moses -f
/home/hieu/working/binarised-model/moses.ini" we get this error

hieu@hieu-VirtualBox:~$ /home/hieu/workspace/github/mosesdecoder/bin/moses
-f /home/hieu/working/binarised-model/moses.ini
Defined parameters (per moses.ini or switch):
config: /home/hieu/working/binarised-model/moses.ini
distortion-limit: 6
feature: UnknownWordPenalty WordPenalty PhrasePenalty
PhraseDictionaryCompact name=TranslationModel0 num-features=4
path=/home/hieu/working/binarised-model/phrase-table.minphr input-factor=0
output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
path=/home/hieu/working/binarised-model/reordering-table.minlexr Distortion
KENLM lazyken=0 name=LM0 factor=0
path=/home/hieu/lm/news-commentary-v8.fr-en.blm.en order=3
input-factors: 0
mapping: 0 T 0
weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3
0.3 Distortion0= 0.3 LM0= 0.5
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=PhrasePenalty
FeatureFunction: PhrasePenalty0 start: 2 end: 2
line=PhraseDictionaryCompact name=TranslationModel0 num-features=4
path=/home/hieu/working/binarised-model/phrase-table.minphr input-factor=0
output-factor=0
Exception: moses/FF/Factory.cpp:321 in void
Moses::FeatureRegistry::Construct(const string&, const string&) threw
UnknownFeatureException because `i == registry_.end()'.
Feature name PhraseDictionaryCompact is not registered.


we need help please ,
 and also an idea for the possibility to use of arabic language ?

the moses.ini file :
#
### MOSES CONFIG FILE ###
#

# input factors
[input-factors]
0

# mapping steps
[mapping]
0 T 0

[distortion-limit]
6

# feature functions
[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryCompact name=TranslationModel0 num-features=4
path=/home/hieu/working/binarised-model/phrase-table.minphr input-factor=0
output-factor=0
LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
path=/home/hieu/working/binarised-model/reordering-table.minlexr
Distortion
KENLM lazyken=0 name=LM0 factor=0
path=/home/hieu/lm/news-commentary-v8.fr-en.blm.en order=3

# dense weights for feature functions
[weight]
UnknownWordPenalty0= 1
WordPenalty0= -1
PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2
LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
Distortion0= 0.3
LM0= 0.5
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] problem while testing moses

[Hieu Hoang]

What was the exact command you used to compile moses? ie. ./bjam 

It doesn't look like the cmph library was linked to moses so the compact 
phrase-table wasn't compiled.

On 10/03/2016 21:33, sofiane bouzaher wrote:
> Hi,we are trying use the moses toolkit for a Rapid Construction of 
> Supervised Word Sense Disambiguation System for Lesser-resourced 
> Languages (arabic) we use the virtual machine (ubuntu 14.04 
> 32-bit.ova30-Jan-2015 16:41 7.2G), we we have compiled moses withs 
> cmph2.0 and boost_ 1_57_0  as recommanded alla steps passed 
> succufully, but when we try to run Moses with compact phrase table 
> using "/home/hieu/workspace/github/mosesdecoder/bin/moses -f 
> /home/hieu/working/binarised-model/moses.ini" we get this error
>
> hieu@hieu-VirtualBox:~$ 
> /home/hieu/workspace/github/mosesdecoder/bin/moses -f 
> /home/hieu/working/binarised-model/moses.ini
> Defined parameters (per moses.ini or switch):
> config: /home/hieu/working/binarised-model/moses.ini
> distortion-limit: 6
> feature: UnknownWordPenalty WordPenalty PhrasePenalty 
> PhraseDictionaryCompact name=TranslationModel0 num-features=4 
> path=/home/hieu/working/binarised-model/phrase-table.minphr 
> input-factor=0 output-factor=0 LexicalReordering 
> name=LexicalReordering0 num-features=6 
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 
> path=/home/hieu/working/binarised-model/reordering-table.minlexr 
> Distortion KENLM lazyken=0 name=LM0 factor=0 
> path=/home/hieu/lm/news-commentary-v8.fr-en.blm.en order=3
> input-factors: 0
> mapping: 0 T 0
> weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 
> 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 
> 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5
> line=UnknownWordPenalty
> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
> line=WordPenalty
> FeatureFunction: WordPenalty0 start: 1 end: 1
> line=PhrasePenalty
> FeatureFunction: PhrasePenalty0 start: 2 end: 2
> line=PhraseDictionaryCompact name=TranslationModel0 num-features=4 
> path=/home/hieu/working/binarised-model/phrase-table.minphr 
> input-factor=0 output-factor=0
> Exception: moses/FF/Factory.cpp:321 in void 
> Moses::FeatureRegistry::Construct(const string&, const string&) threw 
> UnknownFeatureException because `i == registry_.end()'.
> Feature name PhraseDictionaryCompact is not registered.
>
>
> we need help please ,
>  and also an idea for the possibility to use of arabic language ?
>
> the moses.ini file :
> #
> ### MOSES CONFIG FILE ###
> #
>
> # input factors
> [input-factors]
> 0
>
> # mapping steps
> [mapping]
> 0 T 0
>
> [distortion-limit]
> 6
>
> # feature functions
> [feature]
> UnknownWordPenalty
> WordPenalty
> PhrasePenalty
> PhraseDictionaryCompact name=TranslationModel0 num-features=4 
> path=/home/hieu/working/binarised-model/phrase-table.minphr 
> input-factor=0 output-factor=0
> LexicalReordering name=LexicalReordering0 num-features=6 
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 
> path=/home/hieu/working/binarised-model/reordering-table.minlexr
> Distortion
> KENLM lazyken=0 name=LM0 factor=0 
> path=/home/hieu/lm/news-commentary-v8.fr-en.blm.en order=3
>
> # dense weights for feature functions
> [weight]
> UnknownWordPenalty0= 1
> WordPenalty0= -1
> PhrasePenalty0= 0.2
> TranslationModel0= 0.2 0.2 0.2 0.2
> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
> Distortion0= 0.3
> LM0= 0.5

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support