Re: [Moses-support] 10 years of OPUS

[Read, James C]

I wonder if there would be a demand for ready made phrase tables generated from 
the data. 

From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on behalf 
of Jorg Tiedemann [tiede...@gmail.com]
Sent: 02 November 2013 18:54
To: moses-support@MIT.EDU
Subject: [Moses-support] 10 years of OPUS

After attending the 20-years-of-bitext workshop at EMNLP I suddenly realized 
that OPUS (http://opus.lingfil.uu.se) also has its 10-years anniversary this 
year (send me some champagne if you like). I will celebrate this anniversary by 
sending out this e-mail with some recent news and highlights.

OPUS is a growing collection of parallel corpora for many languages and various 
domains. The collection becomes pretty big and includes a variety of data sets 
and tools that are not only useful for statistical machine translation. OPUS 
has been extended a lot since its first appearance in 2003. Actually the best 
birthday present would be if anyone would decide to start a mirror of OPUS. Let 
me know if you are interested.


Here some of the highlights:

- over 150 languages and language variants
- over 5 billion aligned translation units
- downloads in XML/XCES, plain text (Moses/SMT) and TMX
- raw, tokenized and machine-annotated data
- monolingual data sets (for language modeling)
- search interfaces


Some recent news and data sets:

- EUbookshop: a large but noisy corpus (converted from PDF)
- Tatoeba: a small but clean corpus with many languages
- OpenSubtitles2012: an improved version of the 2011 version
- coming soon: OpenSubtitles2013 - an extension of OpenSubtitles2012
- UN, MultiUN, Europarl v7: aligned for all language combinations
- word alignments and phrase tables for the majority of bitexts


The Web Site: http://opus.lingfil.uu.se
More information: http://opus.lingfil.uu.se/trac/wiki

Feedback is very welcome!
And, be nice to our server!


Jörg Tiedemann
tiede...@gmail.com





___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] test sets

[Read, James C]

Hi all,

does anybody know about efforts made to create large test sets from a vast 
array of genres such that a researcher could get a general idea of how his 
system performs in the open domain (e.g. speech2speech or webpage translation)?

thanks,
James

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] gappy phrases

[Read, James C]

Could you tell me a little bit more? When you say basic decoder what do you 
mean? What components does it have? Would I need to first train my translation 
models in the usual way? Will phrase tables generated with the Moses training 
script be in the right format for it?

What I really need is to perhaps implement my own translation model. I want to 
experiment with different units of translation and observe whether the 
translation model ranks better translations higher in isolation. For this 
reason I need to eliminate other variables like the effects of reordering 
models, language models and beam searches.

James


From: hieuho...@gmail.com [hieuho...@gmail.com] on behalf of Hieu Hoang 
[hieu.ho...@ed.ac.uk]
Sent: 05 November 2013 12:12
To: Read, James C
Cc: Daniel Cer; Kenneth Heafield; moses-support@mit.edu
Subject: Re: [Moses-support] gappy phrases

If you're gonna throw most of Moses away, you may consider using and extending 
the basic decoder I've just added to github
https://github.com/moses-smt/mosesdecoder/tree/master/contrib/basic-decoder

Or write your own. You're welcome to add it to moses too


On 5 November 2013 07:29, Read, James C 
mailto:jcr...@essex.ac.uk>> wrote:
Interesting.

This isn't going to be an issue for the kind of experiments I will be running, 
though. I won't be using a language model or a reordering model or a beam 
search during decoding (I know, not much of Moses left). My main usage of 
'Moses' at the moment is training translation models so I can run experiments 
with 'units of translation' in isolation from other variables in the system 
(language model, reordering model, beam search). I would really like to be able 
to run some experiments also with discontinuous phrases both source side and 
target side.

Any idea what kind of changes I would need to make to the training process to 
be able to learn these kind of transformations? I suppose I'm also going to 
need to modify the operation of the translation model to get these working as 
well.

thanks,
James


From: moses-support-boun...@mit.edu 
[moses-support-boun...@mit.edu] on behalf 
of Daniel Cer [daniel@gmail.com]
Sent: 04 November 2013 19:43
To: Kenneth Heafield
Cc: moses-support@mit.edu
Subject: Re: [Moses-support] gappy phrases

Hi everyone,

Ken and I just spoke about this.

Here's a quick synopsis of our semi-recent experience at Stanford with 
discontinuous/gappy phrases:

  *   Source side gaps are effectively free and don't really degrade decoding 
time.
  *   Target side gaps are fine for smaller beam sizes (e.g., < 200).
  *   When using large beams, our current implementation slows down 
dramatically. For example, with a stack size of 500, I think it was sometimes 
taking over an hour to translate some sentences.

While discontinuous phrases can moderately increase the BLEU score, but we get 
a bigger increase by just using very large beam sizes.

Dan



On Sun, Nov 3, 2013 at 8:34 PM, Kenneth Heafield 
mailto:mo...@kheafield.com>>>
 wrote:
Hi,

I'll throw in the anecdote that gappy phrases are currently not in use
at Stanford.  My predecessor told me that it took a lot longer and only
improved BLEU slightly on Chinese-English.  But it's also possible that
something didn't get passed down correctly from Michel to my predecessor
to me. . .

Kenneth

On 11/03/13 14:18, Read, James C wrote:
> My understanding is that they used a similar approach as the grammar 
> extraction to extract the gappy phrases. Would it be a massive undertaking to 
> get Moses to support this?
>
> James
> 
> From: Barry Haddow 
> [bhad...@staffmail.ed.ac.uk>]
> Sent: 30 October 2013 09:26
> To: Read, James C
> Cc: 
> moses-support@mit.edu>
> Subject: Re: [Moses-support] gappy phrases
>
> No, but it does support hiero and syntax models.
>
> On 29/10/13 22:23, Read, James C wrote:
>> Hi,
>>
>> does anybody know if Moses supports gappy phrases 
>> http://www-nlp.stanford.edu/pubs/naacl10-discontinuous_phrases.pdf
>>
>> James
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu<

Re: [Moses-support] Moses installation terminated with IRSTLM and RANDLM

[Guchun Zhang]

Thanks, James for the quick reply.

I probably should have read READMEs more carefully. It's the prefix.

Cheers,
Guchun


On 5 November 2013 19:14, Read, James C  wrote:

> I just wrestled with thing kind of thing. Did you install IRSTLM and
> RANDLM to your home directory? You need to set compile options
> --with-randlm and --with-irstlm to whereever you installed them to. If you
> did ./configure --prefix=$HOME then this will be your home directory.
>
> James
>
>
>
> 
> From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on
> behalf of Guchun Zhang [gzh...@alphacrc.com]
> Sent: 05 November 2013 17:44
> To: moses-support@MIT.EDU
> Subject: [Moses-support] Moses installation terminated with IRSTLM and
> RANDLM
>
> Hi,
>
> Trying to install a new Moses and link it with a few LMs. But I got the
> following errors, regarding IRSTLM and RANDLM. The directories for search
> are incorrect, but I don't know where to change them. Or I am expected to
> create those directories and copy the files into there?
>
> "g++"  -ftemplate-depth-128 -O3 -finline-functions -Wno-inline -Wall -g
> -pthread  -DMAX_NUM_FACTORS=4 -DMPI_ENABLE -DNDEBUG -DTRACE_ENABLE=1
> -DWITH_THREADS -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES  -I"."
> -I"/home/alpha_mt/mt-grid/tools/mt/moses/external-tools/irstlm/include"
> -I"/home/alpha_mt/mt-grid/tools/mt/moses/external-tools/irstlm/include/irstlm"
> -I"/usr/lib/openmpi/include" -I"/usr/lib/openmpi/include/openmpi"
> -I"util/double-conversion" -c -o
> "moses/LM/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/IRST.o"
> "moses/LM/IRST.cpp"
>
> moses/LM/IRST.cpp:25:24: fatal error: dictionary.h: No such file or
> directory
> compilation terminated.
> ...failed gcc.compile.c++
> moses/LM/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/IRST.o...
> gcc.compile.c++
> moses/LM/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/Rand.o
>
> "g++"  -ftemplate-depth-128 -O3 -finline-functions -Wno-inline -Wall
> -g -pthread  -DMAX_NUM_FACTORS=4 -DMPI_ENABLE -DNDEBUG -DTRACE_ENABLE=1
> -DWITH_THREADS -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES  -I"."
> -I"/home/alpha_mt/mt-grid/tools/mt/moses/external-tools/randlm/include"
> -I"/home/alpha_mt/mt-grid/tools/mt/moses/external-tools/randlm/include/RandLM"
> -I"/usr/lib/openmpi/include" -I"/usr/lib/openmpi/include/openmpi"
> -I"util/double-conversion" -c -o
> "moses/LM/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/Rand.o"
> "moses/LM/Rand.cpp"
>
> moses/LM/Rand.cpp:38:20: fatal error: RandLM.h: No such file or directory
>
> The build log is attached. Many thanks for the help.
>
> Cheers,
> Guchun
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] alignment info

[Jorg Tiedemann]

Hi again,

I have another question about the alignment info option. My output is different 
from what I expect according to 
http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc9

I get only something like
…. ||| 0-0 1-1 2-2 3-3 

How do I get both phrase and word alignment in my output?
(like described here: 
http://article.gmane.org/gmane.comp.nlp.moses.user/3776/match=alignment+info)

Thanks!
Jörg



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] gappy phrases

[Matthias Huck]

Hi James,

I tried Phrasal Beta2 and Beta3 a couple of months ago. Both worked for
me with some minor hassle. You should follow the instructions from
http://www-nlp.stanford.edu/wiki/Software/Phrasal#Phrasal in order to
set up your environment.

I'm also sure that the Phrasal developers are able to assist you with
the installation and usage of their toolkit in case you're running into
trouble.

Cheers,
Matthias


On Tue, 2013-11-05 at 18:52 +, Read, James C wrote:
> I just downloaded and installed (I think) a version called Beta4. I
> followed the instructions on the quick start and got to running the
> align script which then bombs out because the VERSION variable isn't
> set to what it should be. Setting that to something sensible it now
> looks like the Berkeley aligner is aligning the suggested data from
> WMT-08.
> 
> I note that the instructions guide you to download an old version of
> corenlp and don't give any compilation instructions. Just wget and tar
> xzf and that's it. I'm not convinced that this means corenlp is
> actually 'installed' or that phrasal is using it in any way. Please
> correct me if you know better.
> 
> thanks,
> James



-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Moses installation terminated with IRSTLM and RANDLM

[Read, James C]

I just wrestled with thing kind of thing. Did you install IRSTLM and RANDLM to 
your home directory? You need to set compile options --with-randlm and 
--with-irstlm to whereever you installed them to. If you did ./configure 
--prefix=$HOME then this will be your home directory.

James




From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on behalf 
of Guchun Zhang [gzh...@alphacrc.com]
Sent: 05 November 2013 17:44
To: moses-support@MIT.EDU
Subject: [Moses-support] Moses installation terminated with IRSTLM and RANDLM

Hi,

Trying to install a new Moses and link it with a few LMs. But I got the 
following errors, regarding IRSTLM and RANDLM. The directories for search are 
incorrect, but I don't know where to change them. Or I am expected to create 
those directories and copy the files into there?

"g++"  -ftemplate-depth-128 -O3 -finline-functions -Wno-inline -Wall -g 
-pthread  -DMAX_NUM_FACTORS=4 -DMPI_ENABLE -DNDEBUG -DTRACE_ENABLE=1 
-DWITH_THREADS -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES  -I"." 
-I"/home/alpha_mt/mt-grid/tools/mt/moses/external-tools/irstlm/include" 
-I"/home/alpha_mt/mt-grid/tools/mt/moses/external-tools/irstlm/include/irstlm" 
-I"/usr/lib/openmpi/include" -I"/usr/lib/openmpi/include/openmpi" 
-I"util/double-conversion" -c -o 
"moses/LM/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/IRST.o"
 "moses/LM/IRST.cpp"

moses/LM/IRST.cpp:25:24: fatal error: dictionary.h: No such file or directory
compilation terminated.
...failed gcc.compile.c++ 
moses/LM/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/IRST.o...
gcc.compile.c++ 
moses/LM/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/Rand.o

"g++"  -ftemplate-depth-128 -O3 -finline-functions -Wno-inline -Wall -g 
-pthread  -DMAX_NUM_FACTORS=4 -DMPI_ENABLE -DNDEBUG -DTRACE_ENABLE=1 
-DWITH_THREADS -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES  -I"." 
-I"/home/alpha_mt/mt-grid/tools/mt/moses/external-tools/randlm/include" 
-I"/home/alpha_mt/mt-grid/tools/mt/moses/external-tools/randlm/include/RandLM" 
-I"/usr/lib/openmpi/include" -I"/usr/lib/openmpi/include/openmpi" 
-I"util/double-conversion" -c -o 
"moses/LM/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/Rand.o"
 "moses/LM/Rand.cpp"

moses/LM/Rand.cpp:38:20: fatal error: RandLM.h: No such file or directory

The build log is attached. Many thanks for the help.

Cheers,
Guchun

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] gappy phrases

[Read, James C]

I just downloaded and installed (I think) a version called Beta4. I followed 
the instructions on the quick start and got to running the align script which 
then bombs out because the VERSION variable isn't set to what it should be. 
Setting that to something sensible it now looks like the Berkeley aligner is 
aligning the suggested data from WMT-08.

I note that the instructions guide you to download an old version of corenlp 
and don't give any compilation instructions. Just wget and tar xzf and that's 
it. I'm not convinced that this means corenlp is actually 'installed' or that 
phrasal is using it in any way. Please correct me if you know better.

thanks,
James


From: Daniel Cer [daniel@gmail.com]
Sent: 05 November 2013 18:20
To: Read, James C
Cc: Matthias Huck; moses-support@mit.edu
Subject: Re: [Moses-support] gappy phrases

Hi James,

> I can't get past installation. I posted on the mailing list. But nobody 
> answered.

I'm sorry your question wasn't answered on the JavaNLP mailing list.

It looks like release 3.2 of Phrasal is now known to be broken. The link should 
should be updated soon to a working version.

Dan


On Tue, Nov 5, 2013 at 5:33 AM, Read, James C 
mailto:jcr...@essex.ac.uk>> wrote:
I can't get past installation. I posted on the mailing list. But nobody 
answered.

From: moses-support-boun...@mit.edu 
[moses-support-boun...@mit.edu] on behalf 
of Matthias Huck [mh...@inf.ed.ac.uk]
Sent: 05 November 2013 13:15
To: moses-support@mit.edu
Subject: Re: [Moses-support] gappy phrases

Hi James,

the Phrasal toolkit is freely available as well 
[http://nlp.stanford.edu/phrasal/],
so why don't you consider extracting discontinuous phrases using Stanford's 
original implementation?

Cheers,
Matthias


On Tue, 2013-11-05 at 07:29 +, Read, James C wrote:
> Interesting.
>
> This isn't going to be an issue for the kind of experiments I will be
> running, though. I won't be using a language model or a reordering
> model or a beam search during decoding (I know, not much of Moses
> left). My main usage of 'Moses' at the moment is training translation
> models so I can run experiments with 'units of translation' in
> isolation from other variables in the system (language model,
> reordering model, beam search). I would really like to be able to run
> some experiments also with discontinuous phrases both source side and
> target side.
>
> Any idea what kind of changes I would need to make to the training
> process to be able to learn these kind of transformations? I suppose
> I'm also going to need to modify the operation of the translation
> model to get these working as well.
>
> thanks,
> James
>
> 
> From: moses-support-boun...@mit.edu 
> [moses-support-boun...@mit.edu] on 
> behalf of Daniel Cer [daniel@gmail.com]
> Sent: 04 November 2013 19:43
> To: Kenneth Heafield
> Cc: moses-support@mit.edu
> Subject: Re: [Moses-support] gappy phrases
>
> Hi everyone,
>
> Ken and I just spoke about this.
>
> Here's a quick synopsis of our semi-recent experience at Stanford with 
> discontinuous/gappy phrases:
>
>   *   Source side gaps are effectively free and don't really degrade decoding 
> time.
>   *   Target side gaps are fine for smaller beam sizes (e.g., < 200).
>   *   When using large beams, our current implementation slows down 
> dramatically. For example, with a stack size of 500, I think it was sometimes 
> taking over an hour to translate some sentences.
>
> While discontinuous phrases can moderately increase the BLEU score, but we 
> get a bigger increase by just using very large beam sizes.
>
> Dan
>
>
>
> On Sun, Nov 3, 2013 at 8:34 PM, Kenneth Heafield 
> mailto:mo...@kheafield.com>>>
>  wrote:
> Hi,
>
> I'll throw in the anecdote that gappy phrases are currently not in use
> at Stanford.  My predecessor told me that it took a lot longer and only
> improved BLEU slightly on Chinese-English.  But it's also possible that
> something didn't get passed down correctly from Michel to my predecessor
> to me. . .
>
> Kenneth
>
> On 11/03/13 14:18, Read, James C wrote:
> > My understanding is that they used a similar approach as the grammar 
> > extraction to extract the gappy phrases. Would it be a massive undertaking 
> > to get Moses to support this?
> >
> > James
> > 
> > From: Barry Haddow 
> > [bhad...@staffmail.ed.ac.uk>]
> > Sent: 30 October 2013 09:26
> > To: Read, James C
> > Cc: 
> > moses-support@mit.edu<

Re: [Moses-support] gappy phrases

[Daniel Cer]

Hi James,

> I can't get past installation. I posted on the mailing list. But nobody
answered.

I'm sorry your question wasn't answered on the JavaNLP mailing list.

It looks like release 3.2 of Phrasal is now known to be broken. The link
should should be updated soon to a working version.

Dan


On Tue, Nov 5, 2013 at 5:33 AM, Read, James C  wrote:

> I can't get past installation. I posted on the mailing list. But nobody
> answered.
> 
> From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on
> behalf of Matthias Huck [mh...@inf.ed.ac.uk]
> Sent: 05 November 2013 13:15
> To: moses-support@mit.edu
> Subject: Re: [Moses-support] gappy phrases
>
> Hi James,
>
> the Phrasal toolkit is freely available as well [
> http://nlp.stanford.edu/phrasal/],
> so why don't you consider extracting discontinuous phrases using
> Stanford's original implementation?
>
> Cheers,
> Matthias
>
>
> On Tue, 2013-11-05 at 07:29 +, Read, James C wrote:
> > Interesting.
> >
> > This isn't going to be an issue for the kind of experiments I will be
> > running, though. I won't be using a language model or a reordering
> > model or a beam search during decoding (I know, not much of Moses
> > left). My main usage of 'Moses' at the moment is training translation
> > models so I can run experiments with 'units of translation' in
> > isolation from other variables in the system (language model,
> > reordering model, beam search). I would really like to be able to run
> > some experiments also with discontinuous phrases both source side and
> > target side.
> >
> > Any idea what kind of changes I would need to make to the training
> > process to be able to learn these kind of transformations? I suppose
> > I'm also going to need to modify the operation of the translation
> > model to get these working as well.
> >
> > thanks,
> > James
> >
> > 
> > From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on
> behalf of Daniel Cer [daniel@gmail.com]
> > Sent: 04 November 2013 19:43
> > To: Kenneth Heafield
> > Cc: moses-support@mit.edu
> > Subject: Re: [Moses-support] gappy phrases
> >
> > Hi everyone,
> >
> > Ken and I just spoke about this.
> >
> > Here's a quick synopsis of our semi-recent experience at Stanford with
> discontinuous/gappy phrases:
> >
> >   *   Source side gaps are effectively free and don't really degrade
> decoding time.
> >   *   Target side gaps are fine for smaller beam sizes (e.g., < 200).
> >   *   When using large beams, our current implementation slows down
> dramatically. For example, with a stack size of 500, I think it was
> sometimes taking over an hour to translate some sentences.
> >
> > While discontinuous phrases can moderately increase the BLEU score, but
> we get a bigger increase by just using very large beam sizes.
> >
> > Dan
> >
> >
> >
> > On Sun, Nov 3, 2013 at 8:34 PM, Kenneth Heafield  > wrote:
> > Hi,
> >
> > I'll throw in the anecdote that gappy phrases are currently not
> in use
> > at Stanford.  My predecessor told me that it took a lot longer and only
> > improved BLEU slightly on Chinese-English.  But it's also possible that
> > something didn't get passed down correctly from Michel to my predecessor
> > to me. . .
> >
> > Kenneth
> >
> > On 11/03/13 14:18, Read, James C wrote:
> > > My understanding is that they used a similar approach as the grammar
> extraction to extract the gappy phrases. Would it be a massive undertaking
> to get Moses to support this?
> > >
> > > James
> > > 
> > > From: Barry Haddow [bhad...@staffmail.ed.ac.uk bhad...@staffmail.ed.ac.uk>]
> > > Sent: 30 October 2013 09:26
> > > To: Read, James C
> > > Cc: moses-support@mit.edu
> > > Subject: Re: [Moses-support] gappy phrases
> > >
> > > No, but it does support hiero and syntax models.
> > >
> > > On 29/10/13 22:23, Read, James C wrote:
> > >> Hi,
> > >>
> > >> does anybody know if Moses supports gappy phrases
> http://www-nlp.stanford.edu/pubs/naacl10-discontinuous_phrases.pdf
> > >>
> > >> James
> > >>
> > >> ___
> > >> Moses-support mailing list
> > >> Moses-support@mit.edu
> > >> http://mailman.mit.edu/mailman/listinfo/moses-support
> > >>
> > >
> > >
> > > --
> > > The University of Edinburgh is a charitable body, registered in
> > > Scotland, with registration number SC005336.
> > >
> > >
> > > ___
> > > Moses-support mailing list
> > > Moses-support@mit.edu
> > > http://mailman.mit.edu/mailman/listinfo/moses-support
> > >
> > ___
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> >
> > 

[Moses-support] Moses installation terminated with IRSTLM and RANDLM

[Guchun Zhang]

Hi,

Trying to install a new Moses and link it with a few LMs. But I got the
following errors, regarding IRSTLM and RANDLM. The directories for search
are incorrect, but I don't know where to change them. Or I am expected to
create those directories and copy the files into there?

*"g++"  -ftemplate-depth-128 -O3 -finline-functions -Wno-inline -Wall -g
-pthread  -DMAX_NUM_FACTORS=4 -DMPI_ENABLE -DNDEBUG -DTRACE_ENABLE=1
-DWITH_THREADS -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES  -I"."
-I"/home/alpha_mt/mt-grid/tools/mt/moses/external-tools/irstlm/include"
-I"/home/alpha_mt/mt-grid/tools/mt/moses/external-tools/irstlm/include/irstlm"
-I"/usr/lib/openmpi/include" -I"/usr/lib/openmpi/include/openmpi"
-I"util/double-conversion" -c -o
"moses/LM/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/IRST.o"
"moses/LM/IRST.cpp"*

*moses/LM/IRST.cpp:25:24: fatal error: dictionary.h: No such file or
directory*
*compilation terminated.*
*...failed gcc.compile.c++
moses/LM/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/IRST.o...*
*gcc.compile.c++
moses/LM/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/Rand.o*

*"g++"  -ftemplate-depth-128 -O3 -finline-functions -Wno-inline -Wall
-g -pthread  -DMAX_NUM_FACTORS=4 -DMPI_ENABLE -DNDEBUG -DTRACE_ENABLE=1
-DWITH_THREADS -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES  -I"."
-I"/home/alpha_mt/mt-grid/tools/mt/moses/external-tools/randlm/include"
-I"/home/alpha_mt/mt-grid/tools/mt/moses/external-tools/randlm/include/RandLM"
-I"/usr/lib/openmpi/include" -I"/usr/lib/openmpi/include/openmpi"
-I"util/double-conversion" -c -o
"moses/LM/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/Rand.o"
"moses/LM/Rand.cpp"*

*moses/LM/Rand.cpp:38:20: fatal error: RandLM.h: No such file or directory*

The build log is attached. Many thanks for the help.

Cheers,
Guchun


build.log.tgz
Description: GNU Zip compressed data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] compiling latest moses

[Jörg Tiedemann]

Thanks a lot. I used 1.46 now ...

Jörg




On 5 nov 2013, at 17:08, Barry Haddow  wrote:

> Or install boost 1.46 from the ubuntu repository. There should be 
> packages for 12.04
> 
> 
> On 05/11/13 15:53, Hieu Hoang wrote:
>> Boost 1.48 is the only version you should avoid because of this 
>> problem. It doesn't appear in other versions
>> https://svn.boost.org/trac/boost/ticket/6141
>> 
>> You can download & compile your own boost by doing
>> ./bootstrap.sh
>> ./b2 --prefix=$PWD --libdir=$PWD/lib64 --layout=tagged link=static 
>> threading=multi,single install -j6
>> then compiling moses
>> ./bjam  --with-boost=[boost/path]
>> 
>> 
>> 
>> 
>> On 5 November 2013 15:13, joerg > > wrote:
>> 
>> 
>>I got the following error when trying to compile the latest moses:
>> 
>>/usr/include/boost/thread/locks.hpp: In constructor
>>‘boost::upgrade_to_unique_lock::upgrade_to_unique_lock(boost::
>>upgrade_lock&) [with Mutex = boost::shared_mutex]’:
>>moses/FeatureVector.cpp:107:76: instantiated from here
>>/usr/include/boost/thread/locks.hpp:926:48: error: call of
>>overloaded ‘move(boost::upgrade_lock&)’ i
>>s ambiguous
>> 
>>I have boost 1.48 on my Ubuntu 12.04.2 LTS system.
>>The log file is attached.
>>Thanks for helping ...
>> 
>> 
>>Jörg
>> 
>> 
>> 
>>
>> **
>>Jörg Tiedemann http://stp.lingfil.uu.se/~joerg/
>>
>> 
>> 
>> 
>> 
>>___
>>Moses-support mailing list
>>Moses-support@mit.edu 
>>http://mailman.mit.edu/mailman/listinfo/moses-support
>> 
>> 
>> 
>> 
>> -- 
>> Hieu Hoang
>> Research Associate
>> University of Edinburgh
>> http://www.hoang.co.uk/hieu
>> 
>> 
>> 
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>> 
>> 
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
> 
> 
> -- 
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
> 
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] compiling latest moses

[Barry Haddow]

Or install boost 1.46 from the ubuntu repository. There should be 
packages for 12.04


On 05/11/13 15:53, Hieu Hoang wrote:
> Boost 1.48 is the only version you should avoid because of this 
> problem. It doesn't appear in other versions
> https://svn.boost.org/trac/boost/ticket/6141
>
> You can download & compile your own boost by doing
> ./bootstrap.sh
> ./b2 --prefix=$PWD --libdir=$PWD/lib64 --layout=tagged link=static 
> threading=multi,single install -j6
> then compiling moses
> ./bjam  --with-boost=[boost/path]
>
>
>
>
> On 5 November 2013 15:13, joerg  > wrote:
>
>
> I got the following error when trying to compile the latest moses:
>
> /usr/include/boost/thread/locks.hpp: In constructor
> ‘boost::upgrade_to_unique_lock::upgrade_to_unique_lock(boost::
> upgrade_lock&) [with Mutex = boost::shared_mutex]’:
> moses/FeatureVector.cpp:107:76: instantiated from here
> /usr/include/boost/thread/locks.hpp:926:48: error: call of
> overloaded ‘move(boost::upgrade_lock&)’ i
> s ambiguous
>
> I have boost 1.48 on my Ubuntu 12.04.2 LTS system.
> The log file is attached.
> Thanks for helping ...
>
>
> Jörg
>
>
>
> 
> **
> Jörg Tiedemann http://stp.lingfil.uu.se/~joerg/
> 
>
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu 
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> -- 
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] compiling latest moses

[Hieu Hoang]

Boost 1.48 is the only version you should avoid because of this problem. It
doesn't appear in other versions
   https://svn.boost.org/trac/boost/ticket/6141

You can download & compile your own boost by doing
   ./bootstrap.sh
   ./b2 --prefix=$PWD --libdir=$PWD/lib64  --layout=tagged link=static
threading=multi,single install -j6
then compiling moses
   ./bjam  --with-boost=[boost/path]




On 5 November 2013 15:13, joerg  wrote:

>
> I got the following error when trying to compile the latest moses:
>
> /usr/include/boost/thread/locks.hpp: In constructor
> ‘boost::upgrade_to_unique_lock::upgrade_to_unique_lock(boost::
> upgrade_lock&) [with Mutex = boost::shared_mutex]’:
> moses/FeatureVector.cpp:107:76:   instantiated from here
> /usr/include/boost/thread/locks.hpp:926:48: error: call of overloaded
> ‘move(boost::upgrade_lock&)’ i
> s ambiguous
>
> I have boost 1.48 on my Ubuntu 12.04.2 LTS system.
> The log file is attached.
> Thanks for helping ...
>
>
> Jörg
>
>
>
>
> **
> Jörg Tiedemann
> http://stp.lingfil.uu.se/~joerg/
>
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


-- 
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] removing identical pairs in phrase table

[Andrew Shin]

Hi,
I'm using Moses in dialogue setting, where both source and target language are 
in English.This results in parroting back of the input, so I need to remove 
phrase-pairs where one phrase is a substring of the other. That'll require 
editing the phrase table after GIZA++ but before Moses.Is there a way to 
automate this? If not, how can I access the phrase table generated by GIZA++ 
and run Moses on that after editing?
I appreciate your time in advance.___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] compiling latest moses

[joerg]

I got the following error when trying to compile the latest moses:/usr/include/boost/thread/locks.hpp: In constructor ‘boost::upgrade_to_unique_lock::upgrade_to_unique_lock(boost::upgrade_lock&) [with Mutex = boost::shared_mutex]’:moses/FeatureVector.cpp:107:76:   instantiated from here/usr/include/boost/thread/locks.hpp:926:48: error: call of overloaded ‘move(boost::upgrade_lock&)’ is ambiguousI have boost 1.48 on my Ubuntu 12.04.2 LTS system.The log file is attached.Thanks for helping ...
Jörg

build.log
Description: Binary data
**Jörg Tiedemann http://stp.lingfil.uu.se/~joerg/


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Issue with EMS web interface

[Daniel Valenzuela]

Hi,

First of all: I love working with moses and I was able to handle all errors
which occured during decoding.

But to get a better overview I wanted to try the web interface which is offered
by the EMS. In order to do this I installed on Ubuntu the newest version of
LAMPP and stored the whole ems/web folder in /opt/lampp/htdocs/ and added the
experiments to /setup .

The problem which occurs, is that (beside the overview page) every page contains
a lot of warnings, also the "img" graph cannot be shown, so actually it's no
help at all. I tried using different browsers and older firefox versions, since
I assumed it has to do something with the PHP version. Since I'm not a php
programmer I wanted to ask if someone has a hint on how to proceed.

Here a few examples:

/?setup=3

Warning: Creating default object from empty value in /opt/lampp/htdocs/lib.php
on line 126

Warning: file(comment): failed to open stream: No such file or directory in
/opt/lampp/htdocs/lib.php on line 96

Warning: Variable passed to each() is not an array or object in
/opt/lampp/htdocs/lib.php on line 97


Task: First Experiment (DanielV)

Wiki Notes  | Overview of
experiments  | /home/moses/working/experiments
ID  start   end newstest2010
cfg  |par
 |img
  [3-1]
Warning: array_key_exists() expects parameter 2 to be array, null given in
/opt/lampp/htdocs/overview.php on line 78
14 Okt  14 Okt
6: 0.163936 17.14 (1.005)



-

/?setup=4


Warning: Creating default object from empty value in /opt/lampp/htdocs/lib.php
on line 126

Warning: uksort() expects parameter 1 to be array, null given in
/opt/lampp/htdocs/lib.php on line 68

Warning: file(comment): failed to open stream: No such file or directory in
/opt/lampp/htdocs/lib.php on line 96

Warning: Variable passed to each() is not an array or object in
/opt/lampp/htdocs/lib.php on line 97


Task: Europarl Experiment ( )

Wiki Notes  | Overview of
experiments  |
/home/moses/project_europarl/experiments_de
Warning: reset() expects parameter 1 to be array, null given in
/opt/lampp/htdocs/overview.php on line 37

Warning: Variable passed to each() is not an array or object in
/opt/lampp/htdocs/overview.php on line 38

Warning: reset() expects parameter 1 to be array, null given in
/opt/lampp/htdocs/overview.php on line 57

Warning: Variable passed to each() is not an array or object in
/opt/lampp/htdocs/overview.php on line 59

Warning: reset() expects parameter 1 to be array, null given in
/opt/lampp/htdocs/overview.php on line 282

Warning: Variable passed to each() is not an array or object in
/opt/lampp/htdocs/overview.php on line 285

ID  start   end
cfg  |par
 |img
  [2-1]
Warning: array_key_exists() expects parameter 2 to be array, null given in
/opt/lampp/htdocs/overview.php on line 78
11:47:44
crashed


---


clicking at "img" yields:



This image cannot be displayed because it contains errors


(in german language, using firefox)

It seems that every document has wrong syntax (which I suppose, will not be
true) but several workarounds didn't help so I ask you.

Thanks everyone and kind regards
Daniel Valenzuela


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] getting WER metrics

[Felipe Sánchez Martínez]

Hi Andrew,

I recently implemented WER for tunnig (MERT) with Moses on an old 
version of Moses I am using. Contributing my code is on my TODO list; I 
have not done it yet because I want to be sure that it does not break 
anything in the up-to-date version. If you want to try yourself I can 
send you the source files.

Cheers
--
Felipe

El 28/10/13 16:37, Francis Tyers escribió:
> There is a package in Apertium which is a simple perl script which
> calculates WER and PER:
>
> https://svn.code.sf.net/p/apertium/svn/trunk/apertium-eval-translator
>
> http://wiki.apertium.org/wiki/Evaluation#Using_apertium-eval-translator_for_WER_and_PER
>
> Fran
>
> El dl 28 de 10 de 2013 a les 11:33 -0400, en/na Philipp Koehn va
> escriure:
>> Hi,
>>
>> Moses currently does not include a tool to measure WER.
>> It should be simple to write, so I would encourage you to
>> implement it and contribute it back.
>>
>> -phi
>>
>> On Sun, Oct 27, 2013 at 11:11 PM, Andrew Shin  wrote:
>>> Hello,
>>> sorry to ask another question..
>>>
>>> I've done getting BLEU score in the past following the baseline tutorial,
>>> but is there a way to also get WER given a reference text?
>>>
>>> Thank you very much for your help.
>>>
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>

-- 
Felipe Sánchez Martínez
Dep. de Llenguatges i Sistemes Informàtics
Universitat d'Alacant, E-03071 Alacant (Spain)
Tel.: +34 965 903 400, ext: 2966 Fax: +34 965 909 326
http://www.dlsi.ua.es/~fsanchez
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] gappy phrases

[Read, James C]

Yes. That's the guide I'd been following. I got hung up on the CoreNLP 
dependency somewhere. It was taking too much time up and nobody on the mailing 
list was answering so I just decided to go with Moses instead.

James

From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on behalf 
of Matthias Huck [mh...@inf.ed.ac.uk]
Sent: 05 November 2013 13:41
To: moses-support@mit.edu
Subject: Re: [Moses-support] gappy phrases

Their Quick Start Guide is helpful:
http://www-nlp.stanford.edu/wiki/Software/Phrasal


On Tue, 2013-11-05 at 13:33 +, Read, James C wrote:
> I can't get past installation. I posted on the mailing list. But nobody 
> answered.
> 
> From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on behalf 
> of Matthias Huck [mh...@inf.ed.ac.uk]
> Sent: 05 November 2013 13:15
> To: moses-support@mit.edu
> Subject: Re: [Moses-support] gappy phrases
>
> Hi James,
>
> the Phrasal toolkit is freely available as well 
> [http://nlp.stanford.edu/phrasal/],
> so why don't you consider extracting discontinuous phrases using Stanford's 
> original implementation?
>
> Cheers,
> Matthias



--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] gappy phrases

[Read, James C]

My basic plan for experimentation was to use a translation model only. Of 
course, with the current implementation (as I understand it) many permutations 
generated by the translation model alone will have the same phrases in a 
different order and thus have the same probability. I was thinking about 
perhaps modifying the translation model such that it generates sentences from 
phrases that overlap on both the source side and target side and scoring in a 
way that gives a higher rank to translations generated with greater degrees of 
overlap.

Do you think it would be easier to write the new translation model from scratch 
or adapt the existing one?

James


From: hieuho...@gmail.com [hieuho...@gmail.com] on behalf of Hieu Hoang 
[hieu.ho...@ed.ac.uk]
Sent: 05 November 2013 12:12
To: Read, James C
Cc: Daniel Cer; Kenneth Heafield; moses-support@mit.edu
Subject: Re: [Moses-support] gappy phrases

If you're gonna throw most of Moses away, you may consider using and extending 
the basic decoder I've just added to github
https://github.com/moses-smt/mosesdecoder/tree/master/contrib/basic-decoder

Or write your own. You're welcome to add it to moses too


On 5 November 2013 07:29, Read, James C 
mailto:jcr...@essex.ac.uk>> wrote:
Interesting.

This isn't going to be an issue for the kind of experiments I will be running, 
though. I won't be using a language model or a reordering model or a beam 
search during decoding (I know, not much of Moses left). My main usage of 
'Moses' at the moment is training translation models so I can run experiments 
with 'units of translation' in isolation from other variables in the system 
(language model, reordering model, beam search). I would really like to be able 
to run some experiments also with discontinuous phrases both source side and 
target side.

Any idea what kind of changes I would need to make to the training process to 
be able to learn these kind of transformations? I suppose I'm also going to 
need to modify the operation of the translation model to get these working as 
well.

thanks,
James


From: moses-support-boun...@mit.edu 
[moses-support-boun...@mit.edu] on behalf 
of Daniel Cer [daniel@gmail.com]
Sent: 04 November 2013 19:43
To: Kenneth Heafield
Cc: moses-support@mit.edu
Subject: Re: [Moses-support] gappy phrases

Hi everyone,

Ken and I just spoke about this.

Here's a quick synopsis of our semi-recent experience at Stanford with 
discontinuous/gappy phrases:

  *   Source side gaps are effectively free and don't really degrade decoding 
time.
  *   Target side gaps are fine for smaller beam sizes (e.g., < 200).
  *   When using large beams, our current implementation slows down 
dramatically. For example, with a stack size of 500, I think it was sometimes 
taking over an hour to translate some sentences.

While discontinuous phrases can moderately increase the BLEU score, but we get 
a bigger increase by just using very large beam sizes.

Dan



On Sun, Nov 3, 2013 at 8:34 PM, Kenneth Heafield 
mailto:mo...@kheafield.com>>>
 wrote:
Hi,

I'll throw in the anecdote that gappy phrases are currently not in use
at Stanford.  My predecessor told me that it took a lot longer and only
improved BLEU slightly on Chinese-English.  But it's also possible that
something didn't get passed down correctly from Michel to my predecessor
to me. . .

Kenneth

On 11/03/13 14:18, Read, James C wrote:
> My understanding is that they used a similar approach as the grammar 
> extraction to extract the gappy phrases. Would it be a massive undertaking to 
> get Moses to support this?
>
> James
> 
> From: Barry Haddow 
> [bhad...@staffmail.ed.ac.uk>]
> Sent: 30 October 2013 09:26
> To: Read, James C
> Cc: 
> moses-support@mit.edu>
> Subject: Re: [Moses-support] gappy phrases
>
> No, but it does support hiero and syntax models.
>
> On 29/10/13 22:23, Read, James C wrote:
>> Hi,
>>
>> does anybody know if Moses supports gappy phrases 
>> http://www-nlp.stanford.edu/pubs/naacl10-discontinuous_phrases.pdf
>>
>> James
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
> ___
> Moses-support mailing list
> Moses-support

Re: [Moses-support] gappy phrases

[Matthias Huck]

Their Quick Start Guide is helpful:
http://www-nlp.stanford.edu/wiki/Software/Phrasal


On Tue, 2013-11-05 at 13:33 +, Read, James C wrote:
> I can't get past installation. I posted on the mailing list. But nobody 
> answered.
> 
> From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on behalf 
> of Matthias Huck [mh...@inf.ed.ac.uk]
> Sent: 05 November 2013 13:15
> To: moses-support@mit.edu
> Subject: Re: [Moses-support] gappy phrases
> 
> Hi James,
> 
> the Phrasal toolkit is freely available as well 
> [http://nlp.stanford.edu/phrasal/],
> so why don't you consider extracting discontinuous phrases using Stanford's 
> original implementation?
> 
> Cheers,
> Matthias



-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] gappy phrases

[Read, James C]

I can't get past installation. I posted on the mailing list. But nobody 
answered.

From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on behalf 
of Matthias Huck [mh...@inf.ed.ac.uk]
Sent: 05 November 2013 13:15
To: moses-support@mit.edu
Subject: Re: [Moses-support] gappy phrases

Hi James,

the Phrasal toolkit is freely available as well 
[http://nlp.stanford.edu/phrasal/],
so why don't you consider extracting discontinuous phrases using Stanford's 
original implementation?

Cheers,
Matthias


On Tue, 2013-11-05 at 07:29 +, Read, James C wrote:
> Interesting.
>
> This isn't going to be an issue for the kind of experiments I will be
> running, though. I won't be using a language model or a reordering
> model or a beam search during decoding (I know, not much of Moses
> left). My main usage of 'Moses' at the moment is training translation
> models so I can run experiments with 'units of translation' in
> isolation from other variables in the system (language model,
> reordering model, beam search). I would really like to be able to run
> some experiments also with discontinuous phrases both source side and
> target side.
>
> Any idea what kind of changes I would need to make to the training
> process to be able to learn these kind of transformations? I suppose
> I'm also going to need to modify the operation of the translation
> model to get these working as well.
>
> thanks,
> James
>
> 
> From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on behalf 
> of Daniel Cer [daniel@gmail.com]
> Sent: 04 November 2013 19:43
> To: Kenneth Heafield
> Cc: moses-support@mit.edu
> Subject: Re: [Moses-support] gappy phrases
>
> Hi everyone,
>
> Ken and I just spoke about this.
>
> Here's a quick synopsis of our semi-recent experience at Stanford with 
> discontinuous/gappy phrases:
>
>   *   Source side gaps are effectively free and don't really degrade decoding 
> time.
>   *   Target side gaps are fine for smaller beam sizes (e.g., < 200).
>   *   When using large beams, our current implementation slows down 
> dramatically. For example, with a stack size of 500, I think it was sometimes 
> taking over an hour to translate some sentences.
>
> While discontinuous phrases can moderately increase the BLEU score, but we 
> get a bigger increase by just using very large beam sizes.
>
> Dan
>
>
>
> On Sun, Nov 3, 2013 at 8:34 PM, Kenneth Heafield 
> mailto:mo...@kheafield.com>> wrote:
> Hi,
>
> I'll throw in the anecdote that gappy phrases are currently not in use
> at Stanford.  My predecessor told me that it took a lot longer and only
> improved BLEU slightly on Chinese-English.  But it's also possible that
> something didn't get passed down correctly from Michel to my predecessor
> to me. . .
>
> Kenneth
>
> On 11/03/13 14:18, Read, James C wrote:
> > My understanding is that they used a similar approach as the grammar 
> > extraction to extract the gappy phrases. Would it be a massive undertaking 
> > to get Moses to support this?
> >
> > James
> > 
> > From: Barry Haddow 
> > [bhad...@staffmail.ed.ac.uk]
> > Sent: 30 October 2013 09:26
> > To: Read, James C
> > Cc: moses-support@mit.edu
> > Subject: Re: [Moses-support] gappy phrases
> >
> > No, but it does support hiero and syntax models.
> >
> > On 29/10/13 22:23, Read, James C wrote:
> >> Hi,
> >>
> >> does anybody know if Moses supports gappy phrases 
> >> http://www-nlp.stanford.edu/pubs/naacl10-discontinuous_phrases.pdf
> >>
> >> James
> >>
> >> ___
> >> Moses-support mailing list
> >> Moses-support@mit.edu
> >> http://mailman.mit.edu/mailman/listinfo/moses-support
> >>
> >
> >
> > --
> > The University of Edinburgh is a charitable body, registered in
> > Scotland, with registration number SC005336.
> >
> >
> > ___
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] gappy phrases

[Matthias Huck]

Hi James,

the Phrasal toolkit is freely available as well 
[http://nlp.stanford.edu/phrasal/], 
so why don't you consider extracting discontinuous phrases using Stanford's 
original implementation?

Cheers,
Matthias


On Tue, 2013-11-05 at 07:29 +, Read, James C wrote:
> Interesting.
> 
> This isn't going to be an issue for the kind of experiments I will be
> running, though. I won't be using a language model or a reordering
> model or a beam search during decoding (I know, not much of Moses
> left). My main usage of 'Moses' at the moment is training translation
> models so I can run experiments with 'units of translation' in
> isolation from other variables in the system (language model,
> reordering model, beam search). I would really like to be able to run
> some experiments also with discontinuous phrases both source side and
> target side.
> 
> Any idea what kind of changes I would need to make to the training
> process to be able to learn these kind of transformations? I suppose
> I'm also going to need to modify the operation of the translation
> model to get these working as well.
> 
> thanks,
> James
> 
> 
> From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on behalf 
> of Daniel Cer [daniel@gmail.com]
> Sent: 04 November 2013 19:43
> To: Kenneth Heafield
> Cc: moses-support@mit.edu
> Subject: Re: [Moses-support] gappy phrases
> 
> Hi everyone,
> 
> Ken and I just spoke about this.
> 
> Here's a quick synopsis of our semi-recent experience at Stanford with 
> discontinuous/gappy phrases:
> 
>   *   Source side gaps are effectively free and don't really degrade decoding 
> time.
>   *   Target side gaps are fine for smaller beam sizes (e.g., < 200).
>   *   When using large beams, our current implementation slows down 
> dramatically. For example, with a stack size of 500, I think it was sometimes 
> taking over an hour to translate some sentences.
> 
> While discontinuous phrases can moderately increase the BLEU score, but we 
> get a bigger increase by just using very large beam sizes.
> 
> Dan
> 
> 
> 
> On Sun, Nov 3, 2013 at 8:34 PM, Kenneth Heafield 
> mailto:mo...@kheafield.com>> wrote:
> Hi,
> 
> I'll throw in the anecdote that gappy phrases are currently not in use
> at Stanford.  My predecessor told me that it took a lot longer and only
> improved BLEU slightly on Chinese-English.  But it's also possible that
> something didn't get passed down correctly from Michel to my predecessor
> to me. . .
> 
> Kenneth
> 
> On 11/03/13 14:18, Read, James C wrote:
> > My understanding is that they used a similar approach as the grammar 
> > extraction to extract the gappy phrases. Would it be a massive undertaking 
> > to get Moses to support this?
> >
> > James
> > 
> > From: Barry Haddow 
> > [bhad...@staffmail.ed.ac.uk]
> > Sent: 30 October 2013 09:26
> > To: Read, James C
> > Cc: moses-support@mit.edu
> > Subject: Re: [Moses-support] gappy phrases
> >
> > No, but it does support hiero and syntax models.
> >
> > On 29/10/13 22:23, Read, James C wrote:
> >> Hi,
> >>
> >> does anybody know if Moses supports gappy phrases 
> >> http://www-nlp.stanford.edu/pubs/naacl10-discontinuous_phrases.pdf
> >>
> >> James
> >>
> >> ___
> >> Moses-support mailing list
> >> Moses-support@mit.edu
> >> http://mailman.mit.edu/mailman/listinfo/moses-support
> >>
> >
> >
> > --
> > The University of Edinburgh is a charitable body, registered in
> > Scotland, with registration number SC005336.
> >
> >
> > ___
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
> 
> 
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
> 



-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] gappy phrases

[Hieu Hoang]

If you're gonna throw most of Moses away, you may consider using and
extending the basic decoder I've just added to github

https://github.com/moses-smt/mosesdecoder/tree/master/contrib/basic-decoder

Or write your own. You're welcome to add it to moses too


On 5 November 2013 07:29, Read, James C  wrote:

> Interesting.
>
> This isn't going to be an issue for the kind of experiments I will be
> running, though. I won't be using a language model or a reordering model or
> a beam search during decoding (I know, not much of Moses left). My main
> usage of 'Moses' at the moment is training translation models so I can run
> experiments with 'units of translation' in isolation from other variables
> in the system (language model, reordering model, beam search). I would
> really like to be able to run some experiments also with discontinuous
> phrases both source side and target side.
>
> Any idea what kind of changes I would need to make to the training process
> to be able to learn these kind of transformations? I suppose I'm also going
> to need to modify the operation of the translation model to get these
> working as well.
>
> thanks,
> James
>
> 
> From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on
> behalf of Daniel Cer [daniel@gmail.com]
> Sent: 04 November 2013 19:43
> To: Kenneth Heafield
> Cc: moses-support@mit.edu
> Subject: Re: [Moses-support] gappy phrases
>
> Hi everyone,
>
> Ken and I just spoke about this.
>
> Here's a quick synopsis of our semi-recent experience at Stanford with
> discontinuous/gappy phrases:
>
>   *   Source side gaps are effectively free and don't really degrade
> decoding time.
>   *   Target side gaps are fine for smaller beam sizes (e.g., < 200).
>   *   When using large beams, our current implementation slows down
> dramatically. For example, with a stack size of 500, I think it was
> sometimes taking over an hour to translate some sentences.
>
> While discontinuous phrases can moderately increase the BLEU score, but we
> get a bigger increase by just using very large beam sizes.
>
> Dan
>
>
>
> On Sun, Nov 3, 2013 at 8:34 PM, Kenneth Heafield  > wrote:
> Hi,
>
> I'll throw in the anecdote that gappy phrases are currently not in
> use
> at Stanford.  My predecessor told me that it took a lot longer and only
> improved BLEU slightly on Chinese-English.  But it's also possible that
> something didn't get passed down correctly from Michel to my predecessor
> to me. . .
>
> Kenneth
>
> On 11/03/13 14:18, Read, James C wrote:
> > My understanding is that they used a similar approach as the grammar
> extraction to extract the gappy phrases. Would it be a massive undertaking
> to get Moses to support this?
> >
> > James
> > 
> > From: Barry Haddow [bhad...@staffmail.ed.ac.uk bhad...@staffmail.ed.ac.uk>]
> > Sent: 30 October 2013 09:26
> > To: Read, James C
> > Cc: moses-support@mit.edu
> > Subject: Re: [Moses-support] gappy phrases
> >
> > No, but it does support hiero and syntax models.
> >
> > On 29/10/13 22:23, Read, James C wrote:
> >> Hi,
> >>
> >> does anybody know if Moses supports gappy phrases
> http://www-nlp.stanford.edu/pubs/naacl10-discontinuous_phrases.pdf
> >>
> >> James
> >>
> >> ___
> >> Moses-support mailing list
> >> Moses-support@mit.edu
> >> http://mailman.mit.edu/mailman/listinfo/moses-support
> >>
> >
> >
> > --
> > The University of Edinburgh is a charitable body, registered in
> > Scotland, with registration number SC005336.
> >
> >
> > ___
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



-- 
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] gappy phrases

[Read, James C]

Interesting.

This isn't going to be an issue for the kind of experiments I will be running, 
though. I won't be using a language model or a reordering model or a beam 
search during decoding (I know, not much of Moses left). My main usage of 
'Moses' at the moment is training translation models so I can run experiments 
with 'units of translation' in isolation from other variables in the system 
(language model, reordering model, beam search). I would really like to be able 
to run some experiments also with discontinuous phrases both source side and 
target side.

Any idea what kind of changes I would need to make to the training process to 
be able to learn these kind of transformations? I suppose I'm also going to 
need to modify the operation of the translation model to get these working as 
well.

thanks,
James


From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on behalf 
of Daniel Cer [daniel@gmail.com]
Sent: 04 November 2013 19:43
To: Kenneth Heafield
Cc: moses-support@mit.edu
Subject: Re: [Moses-support] gappy phrases

Hi everyone,

Ken and I just spoke about this.

Here's a quick synopsis of our semi-recent experience at Stanford with 
discontinuous/gappy phrases:

  *   Source side gaps are effectively free and don't really degrade decoding 
time.
  *   Target side gaps are fine for smaller beam sizes (e.g., < 200).
  *   When using large beams, our current implementation slows down 
dramatically. For example, with a stack size of 500, I think it was sometimes 
taking over an hour to translate some sentences.

While discontinuous phrases can moderately increase the BLEU score, but we get 
a bigger increase by just using very large beam sizes.

Dan



On Sun, Nov 3, 2013 at 8:34 PM, Kenneth Heafield 
mailto:mo...@kheafield.com>> wrote:
Hi,

I'll throw in the anecdote that gappy phrases are currently not in use
at Stanford.  My predecessor told me that it took a lot longer and only
improved BLEU slightly on Chinese-English.  But it's also possible that
something didn't get passed down correctly from Michel to my predecessor
to me. . .

Kenneth

On 11/03/13 14:18, Read, James C wrote:
> My understanding is that they used a similar approach as the grammar 
> extraction to extract the gappy phrases. Would it be a massive undertaking to 
> get Moses to support this?
>
> James
> 
> From: Barry Haddow 
> [bhad...@staffmail.ed.ac.uk]
> Sent: 30 October 2013 09:26
> To: Read, James C
> Cc: moses-support@mit.edu
> Subject: Re: [Moses-support] gappy phrases
>
> No, but it does support hiero and syntax models.
>
> On 29/10/13 22:23, Read, James C wrote:
>> Hi,
>>
>> does anybody know if Moses supports gappy phrases 
>> http://www-nlp.stanford.edu/pubs/naacl10-discontinuous_phrases.pdf
>>
>> James
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support