[Moses-support] First Call for Participation: WMT17 Machine Translated related Shared Tasks

[Barry Haddow]

EMNLP 2017 SECOND CONFERENCE ON MACHINE TRANSLATION (WMT17)
Shared Tasks on translation, evaluation, training and automated 
post-editing.


http://www.statmt.org/wmt17/index.html
September 7-8 2017, in conjunction with EMNLP 2017 in Copenhagen, Denmark

As part of WMT, as in previous years, we will be organising a collection 
of shared tasks related to machine translation.  We hope that both 
beginners and established research groups will participate. This year we 
have so far confirmed the following tasks


- Translation tasks
- News
- Biomedical
- Multimodal
- Evaluation tasks
- Metrics
- Quality estimation
- Other tasks
- Bandit learning
- Neural MT training
- Automatic post-editing

Further information, including task rationale, timetables and data will 
be posted on the WMT17 website, in time for the task launches in 
January/February.  Intending participants are encouraged to register 
with the mailing list for further announcements 
(https://groups.google.com/forum/#!forum/wmt-tasks)


For all tasks,  participants will also be  invited to submit a short 
paper describing their system.



Best wishes
Barry Haddow
(On behalf of the organisers)







The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Tuning for factored phrase based systems

[Angli Liu]

Thank you!

On Tue, Dec 6, 2016 at 12:55 AM Sašo Kuntaric 
wrote:

> Hi Angli,
>
> Here is an excerpt of Hieu's answers regarding this topic when I was doing
> research in factored models, might be of some help:
>
> On 30/06/2016 21:44, Sašo Kuntaric wrote:
>
> Hi all,
>
> I would like to ask one more question. When you say that my reference only
> has the surface form, are you talking about the "tuning corpus", which in
> the case of my command
>
> ~/mosesdecoder/scripts/training/mert-moses.pl
> ~/working/IT_corpus/TMX/txt/factored_corpus/singles/tuning_corpus.tagged.clean.en
> ~/working/IT_corpus/TMX/txt/factored_corpus/singles/
> tuning_corpus.tagged.clean.sl ~/mosesdecoder/bin/moses
> ~/working/IT_corpus/TMX/txt/factored_corpus/singles/test/model/moses.ini
> --mertdir ~/mosesdecoder/bin/ --decoder-flags="-threads all"
>
> are tuning_corpus.tagged.clean.en and tuning_corpus.tagged.clean.sl? Can
> tuning be done with files that only contains surface forms?
>
> it's usual that the reference tuning data does not have factors, even if
> there are factors in the phrase table. After all, you don't care if the
> output surface form is correct but the other factors are wrong.
>
> Will the results be compatible with tuning done with a factored tuning
> corpus?
>
> yes
>
> Best regards,
>
> Sašo
>
> 2016-12-04 1:37 GMT+01:00 Hieu Hoang :
>
>
>
> Hieu
> Sent while bumping into things
>
> On 1 Dec 2016 07:01, "Angli Liu"  wrote:
>
> Hi, what's the major difference between the tuning process for a factored
> phrase based system (i.e., surface+pos data) and a simple baseline phrase
> based system?
>
>
> Nothing, the tuning just optimise weights for feature functions.
>
> If you decompose your translation so that it has multiple phrase tables
> and generation models, then they are just extra feature functions with
> weights to be tuned
>
> Do I need to organize the dev set the same way as the training set (i.e.,
> surface|pos)?
>
> Yes
>
> Is there a tutorial on the moses website on this topic?
>
> Maybe this
> http://www.statmt.org/moses/?n=FactoredTraining.FactoredTraining
>
>
> Thanks!
>
> -Angli
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> --
> lp,
>
> Sašo
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Run fast_align with multithreading

[Lane Schwartz]

OK. Thanks!

On Tue, Dec 6, 2016 at 11:18 AM, Michael Denkowski <
michael.j.denkow...@gmail.com> wrote:

> Hi Lane,
>
> I had to track this one down too.  The multithreading uses OpenMP, so if
> you install OpenMP on your machine and build fast_align, multithreading
> should be baked in.  By default, it should use all available CPUs.  You can
> override this by setting the OMP_NUM_THREADS environment variable (export
> OMP_NUM_THREADS=8).
>
> Best,
> Michael
>
> On Tue, Dec 6, 2016 at 12:06 PM, Lane Schwartz  wrote:
>
>> Hi,
>>
>> Looking at the change log in the fast_align github repo, it appears that
>> there is now multithreading in fast_align.
>>
>> This should be simple, but I haven't been able to find it documented
>> anywhere. Can anyone verify that fast_align does indeed support
>> multithreading, and if it does, how do you enable it?
>>
>> Thanks,
>> Lane
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>


-- 
When a place gets crowded enough to require ID's, social collapse is not
far away.  It is time to go elsewhere.  The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Run fast_align with multithreading

[Michael Denkowski]

Hi Lane,

I had to track this one down too.  The multithreading uses OpenMP, so if
you install OpenMP on your machine and build fast_align, multithreading
should be baked in.  By default, it should use all available CPUs.  You can
override this by setting the OMP_NUM_THREADS environment variable (export
OMP_NUM_THREADS=8).

Best,
Michael

On Tue, Dec 6, 2016 at 12:06 PM, Lane Schwartz  wrote:

> Hi,
>
> Looking at the change log in the fast_align github repo, it appears that
> there is now multithreading in fast_align.
>
> This should be simple, but I haven't been able to find it documented
> anywhere. Can anyone verify that fast_align does indeed support
> multithreading, and if it does, how do you enable it?
>
> Thanks,
> Lane
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Run fast_align with multithreading

[Lane Schwartz]

Hi,

Looking at the change log in the fast_align github repo, it appears that
there is now multithreading in fast_align.

This should be simple, but I haven't been able to find it documented
anywhere. Can anyone verify that fast_align does indeed support
multithreading, and if it does, how do you enable it?

Thanks,
Lane
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS dry-run flag?

[Lane Schwartz]

Thanks, Philipp. That was it. I had a CORPUS section, but I did not have a
CORPUS:specificdata.

On Mon, Dec 5, 2016 at 10:56 PM, Philipp Koehn  wrote:

> Hi,
>
> I guess this error implies that you did not define any CORPUS section.
>
> -phi
>
> On Mon, Dec 5, 2016 at 6:51 PM, Lane Schwartz  wrote:
>
>> Ah, indeed there is an error.
>>
>> ERROR: Step TRAINING:consolidate requires input from prior steps, but
>> none defined
>>
>> On Mon, Dec 5, 2016 at 5:08 PM, Barry Haddow 
>> wrote:
>>
>>> Hi Lane
>>>
>>> That's odd. It looks like EMS has decided it does not need to run
>>> anything. What did it report on the console?
>>>
>>> cheers - Barry
>>>
>>>
>>> On 05/12/16 22:55, Lane Schwartz wrote:
>>>
>>> Hmm... My steps/0 folder contains the following:
>>>
>>> parameter.0:
>>>
 CORPUS:max-sentence-length = 80
 CORPUS:raw-stem = training
 EVALUATION:analysis = /opt/moses/scripts/ems/support/analysis.perl
 EVALUATION:analyze-coverage = yes
 EVALUATION:decoder-settings =  -search-algorithm 1
 -cube-pruning-pop-limit 5000 -s 5000 -threads 4
>>>
>>> ...
>>>
>>> TUNING:tuning-script = /opt/moses/scripts/training/mert-moses.pl
 TUNING:tuning-settings = -mertdir /opt/moses/bin
>>>
>>>
>>> config.0 (a copy of my ems config file)
>>>
>>> re-use.0 (an empty file)
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Dec 5, 2016 at 4:42 PM, Barry Haddow >> > wrote:
>>>
 In steps/0

 On 05/12/16 22:36, Fred Blain wrote:

 hi Lane,

 if you omit the '-exec' in your call to experiment.perl, it will only
 generate the required scripts without running anything. you will find
 the
 scripts under the steps/ folder.

 best,


 ___
 Moses-support mailing 
 listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support

 The University of Edinburgh is a charitable body, registered in
 Scotland, with registration number SC005336.
>>>
>>> --
>>> When a place gets crowded enough to require ID's, social collapse is not
>>> far away.  It is time to go elsewhere.  The best thing about space travel
>>> is that it made it possible to go elsewhere. -- R.A.
>>> Heinlein, "Time Enough For Love"
>>>
>>>
>>> The University of Edinburgh is a charitable body, registered in
>>> Scotland, with registration number SC005336.
>>>
>>>
>>
>>
>> --
>> When a place gets crowded enough to require ID's, social collapse is not
>> far away.  It is time to go elsewhere.  The best thing about space travel
>> is that it made it possible to go elsewhere.
>> -- R.A. Heinlein, "Time Enough For Love"
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>


-- 
When a place gets crowded enough to require ID's, social collapse is not
far away.  It is time to go elsewhere.  The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Regarding Decoding Time

[Hieu Hoang]

have a look at this page
   http://www.statmt.org/moses/?n=Moses.Optimize

Hieu Hoang
http://www.hoang.co.uk/hieu

On 5 December 2016 at 14:15, Shubham Khandelwal  wrote:

> Hello,
>
> I have created one translation model (German to English) and converted it
> to Compact format.
> Now when I run the command:
> ~/mosesdecoder/bin/moses2 -f  moses.ini
> Then, It loads the model very quickly but when I pass the German Text
> (around 4-5 lines) for the translation then it translates the text in 10
> minutes.
> Can you please tell me that how can I reduce this time for the translation
> ?
>
> Thanking You.
>
> Regards,
> Shubham
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] too few factors error in mert

[Matthias Huck]

Hi,

Maybe your moses.ini lets the decoder expect five input factors, wherea
s there are only four present in the data?

I see this in your log file:

input-factors: 0 1 2 3 4

Cheers,
Matthias


On Tue, 2016-12-06 at 11:18 +0200, Hasan Sait ARSLAN wrote:
> Hi,
> 
> I have a factored dataset. It involves 4 factors,
> factor1|factor2|factor3|factor4. I have trained my model with such a
> dataset.
> 
> Now when I want to tune my model, I encounter with the following error:
> 
> 
> 
> 
> *Exception: moses/Word.cpp:159 in void
> Moses::Word::CreateFromString(Moses::FactorDirection, const
> std::vector&, const StringPiece&, bool, bool) threw
> util::Exception because `!isNonTerminal && i < factorOrder.size()'.Too few
> factors in string '-|-|Punc|Punc*
> The details of the error is in mert.txt file, which is attached to this
> e-mail.
> 
> Thanks,
> 
> Kind Regards,
> Hasan Sait Arslan
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] too few factors error in mert

[Hasan Sait ARSLAN]

Yes, but my dev set is also factored. It is at the same format with
training set. There is no difference at all.

2016-12-06 12:15 GMT+02:00 Sašo Kuntaric :

> Please see my reply to another thread below. I believe you need your
> source part of the tuning set factored as well.
>
> On 30/06/2016 21:44, Sašo Kuntaric wrote:
>
> Hi all,
>
> I would like to ask one more question. When you say that my reference only
> has the surface form, are you talking about the "tuning corpus", which in
> the case of my command
>
> ~/mosesdecoder/scripts/training/mert-moses.pl
> ~/working/IT_corpus/TMX/txt/factored_corpus/singles/tuning_corpus.tagged.clean.en
> ~/working/IT_corpus/TMX/txt/factored_corpus/singles/tuning_
> corpus.tagged.clean.sl ~/mosesdecoder/bin/moses
> ~/working/IT_corpus/TMX/txt/factored_corpus/singles/test/model/moses.ini
> --mertdir ~/mosesdecoder/bin/ --decoder-flags="-threads all"
>
> are tuning_corpus.tagged.clean.en and tuning_corpus.tagged.clean.sl? Can
> tuning be done with files that only contains surface forms?
>
> it's usual that the reference tuning data does not have factors, even if
> there are factors in the phrase table. After all, you don't care if the
> output surface form is correct but the other factors are wrong.
>
> Will the results be compatible with tuning done with a factored tuning
> corpus?
>
> yes
>
> Best regards,
>
> Sašo
>
> 2016-12-06 10:18 GMT+01:00 Hasan Sait ARSLAN 
> :
>
>> Hi,
>>
>> I have a factored dataset. It involves 4 factors,
>> factor1|factor2|factor3|factor4. I have trained my model with such a
>> dataset.
>>
>> Now when I want to tune my model, I encounter with the following error:
>>
>>
>>
>>
>> *Exception: moses/Word.cpp:159 in void
>> Moses::Word::CreateFromString(Moses::FactorDirection, const
>> std::vector&, const StringPiece&, bool, bool) threw
>> util::Exception because `!isNonTerminal && i < factorOrder.size()'.Too few
>> factors in string '-|-|Punc|Punc*
>> The details of the error is in mert.txt file, which is attached to this
>> e-mail.
>>
>> Thanks,
>>
>> Kind Regards,
>> Hasan Sait Arslan
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
>
> --
> lp,
>
> Sašo
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] too few factors error in mert

[Sašo Kuntaric]

Please see my reply to another thread below. I believe you need your source
part of the tuning set factored as well.

On 30/06/2016 21:44, Sašo Kuntaric wrote:

Hi all,

I would like to ask one more question. When you say that my reference only
has the surface form, are you talking about the "tuning corpus", which in
the case of my command

~/mosesdecoder/scripts/training/mert-moses.pl ~/working/IT_corpus/TMX/txt/
factored_corpus/singles/tuning_corpus.tagged.clean.en
~/working/IT_corpus/TMX/txt/factored_corpus/singles/tuning
_corpus.tagged.clean.sl ~/mosesdecoder/bin/moses
~/working/IT_corpus/TMX/txt/factored_corpus/singles/test/model/moses.ini
--mertdir ~/mosesdecoder/bin/ --decoder-flags="-threads all"

are tuning_corpus.tagged.clean.en and tuning_corpus.tagged.clean.sl? Can
tuning be done with files that only contains surface forms?

it's usual that the reference tuning data does not have factors, even if
there are factors in the phrase table. After all, you don't care if the
output surface form is correct but the other factors are wrong.

Will the results be compatible with tuning done with a factored tuning
corpus?

yes

Best regards,

Sašo

2016-12-06 10:18 GMT+01:00 Hasan Sait ARSLAN :

> Hi,
>
> I have a factored dataset. It involves 4 factors, 
> factor1|factor2|factor3|factor4.
> I have trained my model with such a dataset.
>
> Now when I want to tune my model, I encounter with the following error:
>
>
>
>
> *Exception: moses/Word.cpp:159 in void
> Moses::Word::CreateFromString(Moses::FactorDirection, const
> std::vector&, const StringPiece&, bool, bool) threw
> util::Exception because `!isNonTerminal && i < factorOrder.size()'.Too few
> factors in string '-|-|Punc|Punc*
> The details of the error is in mert.txt file, which is attached to this
> e-mail.
>
> Thanks,
>
> Kind Regards,
> Hasan Sait Arslan
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


-- 
lp,

Sašo
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] too few factors error in mert

[Hasan Sait ARSLAN]

Hi,

I have a factored dataset. It involves 4 factors,
factor1|factor2|factor3|factor4. I have trained my model with such a
dataset.

Now when I want to tune my model, I encounter with the following error:




*Exception: moses/Word.cpp:159 in void
Moses::Word::CreateFromString(Moses::FactorDirection, const
std::vector&, const StringPiece&, bool, bool) threw
util::Exception because `!isNonTerminal && i < factorOrder.size()'.Too few
factors in string '-|-|Punc|Punc*
The details of the error is in mert.txt file, which is attached to this
e-mail.

Thanks,

Kind Regards,
Hasan Sait Arslan
Using SCRIPTS_ROOTDIR: /export/students/sait/build/mosesdecoder/scripts
filtering the phrase tables... T dets   6 10:59:05 EET 2016
exec: 
/export/students/sait/build/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered 
/export/students/sait/factored_translation/working/unfactored/model/moses.ini 
/export/students/sait/factored_translation/dataset/dev.tr
Executing: 
/export/students/sait/build/mosesdecoder/scripts/training/filter-model-given-input.pl
 ./filtered 
/export/students/sait/factored_translation/working/unfactored/model/moses.ini 
/export/students/sait/factored_translation/dataset/dev.tr > filterphrases.out 
2> filterphrases.err
Asking moses for feature names and values from filtered/moses.ini
Executing: /export/students/sait/build/mosesdecoder/bin/moses -threads all 
-config filtered/moses.ini -show-weights
exec: /export/students/sait/build/mosesdecoder/bin/moses -threads all -config 
filtered/moses.ini -show-weights
Executing: /export/students/sait/build/mosesdecoder/bin/moses -threads all 
-config filtered/moses.ini -show-weights > ./features.list 2> /dev/null
MERT starting values and ranges for random generation:
  Distortion0 =   0.300 ( 0.00 ..  1.00)
LM0 =   0.500 ( 0.00 ..  1.00)
  WordPenalty0 =  -1.000 ( 0.00 ..  1.00)
  PhrasePenalty0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
featlist: Distortion0=0.30 
featlist: LM0=0.50 
featlist: WordPenalty0=-1.00 
featlist: PhrasePenalty0=0.20 
featlist: TranslationModel0=0.20 
featlist: TranslationModel0=0.20 
featlist: TranslationModel0=0.20 
featlist: TranslationModel0=0.20 
run 1 start at T dets   6 11:01:28 EET 2016
Parsing --decoder-flags: |-threads all|
Saving new config to: ./run1.moses.ini
Saved: ./run1.moses.ini
Normalizing lambdas: 0.30 0.50 -1.00 0.20 0.20 0.20 
0.20 0.20
DECODER_CFG = -weight-overwrite 'TranslationModel0= 0.071429 0.071429 0.071429 
0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 0.107143 
WordPenalty0= -0.357143'
Executing: /export/students/sait/build/mosesdecoder/bin/moses -threads all  
-config filtered/moses.ini -weight-overwrite 'TranslationModel0= 0.071429 
0.071429 0.071429 0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 
0.107143 WordPenalty0= -0.357143'  -n-best-list run1.best100.out 100 distinct  
-input-file /export/students/sait/factored_translation/dataset/dev.tr > 
run1.out 
Executing: /export/students/sait/build/mosesdecoder/bin/moses -threads all  
-config filtered/moses.ini -weight-overwrite 'TranslationModel0= 0.071429 
0.071429 0.071429 0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 
0.107143 WordPenalty0= -0.357143'  -n-best-list run1.best100.out 100 distinct  
-input-file /export/students/sait/factored_translation/dataset/dev.tr > run1.out
1-10.20.2 0.2 0.2 0.20.30.5(1) run decoder to produce n-best lists
params = -threads all
decoder_config = -weight-overwrite 'TranslationModel0= 0.071429 0.071429 
0.071429 0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 0.107143 
WordPenalty0= -0.357143'
Defined parameters (per moses.ini or switch):
config: filtered/moses.ini 
distortion-limit: 6 
feature: UnknownWordPenalty WordPenalty PhrasePenalty 
PhraseDictionaryMemory name=TranslationModel0 num-features=4 
path=/export/students/sait/factored_translation/working/mert-work/filtered/phrase-table.0-0.1.1.gz
 input-factor=0 output-factor=0 Distortion KENLM name=LM0 factor=0 
path=/export/students/sait/factored_translation/lm/surface.lm order=3 
input-factors: 0 1 2 3 4 
input-file: /export/students/sait/factored_translation/dataset/dev.tr 
mapping: 0 T 0 
n-best-list: run1.best100.out 100 distinct 
threads: all 
weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 
TranslationModel0= 0.2 0.2 0.2 0.2 Distortion0= 0.3 LM0= 0.5 
weight-overwrite: TranslationModel0= 0.071429 0.071429 0.071429 
0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 0.107143 
WordPenalty0= -0.357143 
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=Phras

[Moses-support] Biconcor: Permission denied

[Fathima Farhath Farook]

Dear all,

I have been doing some experiments with EMS for some time and i was able to
check the statistics over 'biconcor'.
For very recent, I couldn't get the statistic out of the concorder in the
EMS test result page and it returned permission denied.

So to check if something went wrong with the permission, i tried the
following command in the terminal. But I work fine and gave methe dfesired
result.

sudo -u apache LD_LIBRARY_PATH=/home/farhath/moses/mosesdecoder/opt/lib/
/var/www/html/EMS/biconcor -html -l
/var/www/html/test/ems/ST2/model/biconcor.13 -Q 4La74LeP4Lai4LeK4oCN4La6


Can anyone guide me in this regard.

-- 
Regards,

Farhath Farook



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Tuning for factored phrase based systems

[Sašo Kuntaric]

Hi Angli,

Here is an excerpt of Hieu's answers regarding this topic when I was doing
research in factored models, might be of some help:

On 30/06/2016 21:44, Sašo Kuntaric wrote:

Hi all,

I would like to ask one more question. When you say that my reference only
has the surface form, are you talking about the "tuning corpus", which in
the case of my command

~/mosesdecoder/scripts/training/mert-moses.pl ~/working/IT_corpus/TMX/txt/
factored_corpus/singles/tuning_corpus.tagged.clean.en
~/working/IT_corpus/TMX/txt/factored_corpus/singles/tuning
_corpus.tagged.clean.sl ~/mosesdecoder/bin/moses
~/working/IT_corpus/TMX/txt/factored_corpus/singles/test/model/moses.ini
--mertdir ~/mosesdecoder/bin/ --decoder-flags="-threads all"

are tuning_corpus.tagged.clean.en and tuning_corpus.tagged.clean.sl? Can
tuning be done with files that only contains surface forms?

it's usual that the reference tuning data does not have factors, even if
there are factors in the phrase table. After all, you don't care if the
output surface form is correct but the other factors are wrong.

Will the results be compatible with tuning done with a factored tuning
corpus?

yes

Best regards,

Sašo

2016-12-04 1:37 GMT+01:00 Hieu Hoang :

>
>
> Hieu
> Sent while bumping into things
>
> On 1 Dec 2016 07:01, "Angli Liu"  wrote:
>
> Hi, what's the major difference between the tuning process for a factored
> phrase based system (i.e., surface+pos data) and a simple baseline phrase
> based system?
>
>
> Nothing, the tuning just optimise weights for feature functions.
>
> If you decompose your translation so that it has multiple phrase tables
> and generation models, then they are just extra feature functions with
> weights to be tuned
>
> Do I need to organize the dev set the same way as the training set (i.e.,
> surface|pos)?
>
> Yes
>
> Is there a tutorial on the moses website on this topic?
>
> Maybe this
> http://www.statmt.org/moses/?n=FactoredTraining.FactoredTraining
>
>
> Thanks!
>
> -Angli
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


-- 
lp,

Sašo
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support