Re: [Moses-support] Language model interpolation without SRILM

[Mathias Müller]

Thanks Philipp and Kenneth!

So, does this mean that finding the weights and log-linear interpolation of
LMs is actually implemented in KenLM, but there is no ready-made,
higher-level script to use this functionality, as there is for SRILM
(interpolate-lm.perl)?

@Kenneth Since KenLM is already distributed with Moses, why do you
recommend that I compile the code separately again? Does
github.com/kpu/kenlm have different code than what comes with Moses?

Thanks again,
Mathias

On Tue, Jun 28, 2016 at 6:08 PM, Kenneth Heafield 
wrote:

> Log-linear interpolation is in KenLM in the lm/interpolate directory.
> You'll want to get KenLM from github.com/kpu/kenlm and compile with Eigen.
>
> Tuning log-linear weights is super slow, but applying them is reasonably
> fast.  In total the tuning + applying weights time is comparable to SRILM.
>
> https://kheafield.com/professional/edinburgh/interpolate_paper.pdf
>
> Kenneth
>
> On 06/28/2016 03:27 PM, Philipp Koehn wrote:
> > Hi,
> >
> > unfortunately, the interpolation of language models requires two pieces
> > of code that only exist in SRILM: The EM training method to find weights
> > for the language models, and the linear interpolation of the language
> > models.
> >
> > Maybe Ken and Lane can weigh in, if/when a replacement in KENLM will be
> > available.
> >
> > -phi
> >
> > On Tue, Jun 28, 2016 at 10:10 AM, Mathias Müller  > > wrote:
> >
> > Hi all
> >
> > I have trained several language models and would like to combine
> > them with interpolate-lm.perl:
> >
> >
> https://github.com/moses-smt/mosesdecoder/blob/master/scripts/ems/support/interpolate-lm.perl
> >
> > As the language model tool, I always use KenLM, but looking at the
> > code of interpolate-lm.perl, it seems that the use of SRILM is
> > hard-coded in the script. I would like to avoid SRILM because, if I
> > understand correctly, its license does not permit use in commercial
> > products.
> >
> > My question is:
> >
> > Can I simply replace the call to SRILM with KenLM in my copy of
> > interpolate-lm.perl? Does KenLM have the functionality necessary for
> > language model combination, e.g. a substitute for SRILM's
> > "compute-best-mix"?
> >
> > Thanks for your help.
> > Mathias
> >
> > —
> >
> > Mathias Müller
> > AND-2-20
> > Institute of Computational Linguistics
> > University of Zurich
> > +41 44 635 75 81 
> > mathias.muel...@uzh.ch 
> >
> > ___
> > Moses-support mailing list
> > Moses-support@mit.edu 
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> >
> >
> >
> > ___
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Language model interpolation without SRILM

[Kenneth Heafield]

It's new.  There are some rough edges like memory budgeting.  Also, I'd
argue there is less need for a script since there is one integrated
program that takes models, tunes, and generates the combined model
(though you can split it into steps if you'd like).

Another thing to note: you'll need to generate the component models
using lmplz with the "intermediate" binary format.  This is done by
adding "--intermediate $file_name" to the lmplz arguments.  I'd like to
kill the intermediate format and have lmplz generate a trie, but it's a
matter of getting code bandwidth.

Kenneth

On 06/30/2016 01:30 PM, Mathias Müller wrote:
> Thanks Philipp and Kenneth!
> 
> So, does this mean that finding the weights and log-linear interpolation
> of LMs is actually implemented in KenLM, but there is no ready-made,
> higher-level script to use this functionality, as there is for SRILM
> (interpolate-lm.perl)?
> 
> @Kenneth Since KenLM is already distributed with Moses, why do you
> recommend that I compile the code separately again? Does
> github.com/kpu/kenlm  have different code
> than what comes with Moses?
> 
> Thanks again,
> Mathias
> 
> On Tue, Jun 28, 2016 at 6:08 PM, Kenneth Heafield  > wrote:
> 
> Log-linear interpolation is in KenLM in the lm/interpolate directory.
> You'll want to get KenLM from github.com/kpu/kenlm
>  and compile with Eigen.
> 
> Tuning log-linear weights is super slow, but applying them is reasonably
> fast.  In total the tuning + applying weights time is comparable to
> SRILM.
> 
> https://kheafield.com/professional/edinburgh/interpolate_paper.pdf
> 
> Kenneth
> 
> On 06/28/2016 03:27 PM, Philipp Koehn wrote:
> > Hi,
> >
> > unfortunately, the interpolation of language models requires two pieces
> > of code that only exist in SRILM: The EM training method to find weights
> > for the language models, and the linear interpolation of the language
> > models.
> >
> > Maybe Ken and Lane can weigh in, if/when a replacement in KENLM will be
> > available.
> >
> > -phi
> >
> > On Tue, Jun 28, 2016 at 10:10 AM, Mathias Müller 
> mailto:mathias.muel...@uzh.ch>
> > >> wrote:
> >
> > Hi all
> >
> > I have trained several language models and would like to combine
> > them with interpolate-lm.perl:
> >
> > 
> https://github.com/moses-smt/mosesdecoder/blob/master/scripts/ems/support/interpolate-lm.perl
> >
> > As the language model tool, I always use KenLM, but looking at the
> > code of interpolate-lm.perl, it seems that the use of SRILM is
> > hard-coded in the script. I would like to avoid SRILM because, if I
> > understand correctly, its license does not permit use in commercial
> > products.
> >
> > My question is:
> >
> > Can I simply replace the call to SRILM with KenLM in my copy of
> > interpolate-lm.perl? Does KenLM have the functionality necessary for
> > language model combination, e.g. a substitute for SRILM's
> > "compute-best-mix"?
> >
> > Thanks for your help.
> > Mathias
> >
> > —
> >
> > Mathias Müller
> > AND-2-20
> > Institute of Computational Linguistics
> > University of Zurich
> > +41 44 635 75 81 
> 
> > mathias.muel...@uzh.ch 
> >
> >
> > ___
> > Moses-support mailing list
> > Moses-support@mit.edu 
> >
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> >
> >
> >
> > ___
> > Moses-support mailing list
> > Moses-support@mit.edu 
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> ___
> Moses-support mailing list
> Moses-support@mit.edu 
> http://mailman.mit.edu/mailman/listinfo/moses-support
> 
> 
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Language model interpolation without SRILM

[Kenneth Heafield]

Oh also, use a small -S argument to the interpolate program because it
doesn't quite budget memory properly yet.

On 06/28/2016 05:08 PM, Kenneth Heafield wrote:
> Log-linear interpolation is in KenLM in the lm/interpolate directory.
> You'll want to get KenLM from github.com/kpu/kenlm and compile with Eigen.
> 
> Tuning log-linear weights is super slow, but applying them is reasonably
> fast.  In total the tuning + applying weights time is comparable to SRILM.
> 
> https://kheafield.com/professional/edinburgh/interpolate_paper.pdf
> 
> Kenneth
> 
> On 06/28/2016 03:27 PM, Philipp Koehn wrote:
>> Hi,
>>
>> unfortunately, the interpolation of language models requires two pieces
>> of code that only exist in SRILM: The EM training method to find weights
>> for the language models, and the linear interpolation of the language
>> models.
>>
>> Maybe Ken and Lane can weigh in, if/when a replacement in KENLM will be
>> available.
>>
>> -phi
>>
>> On Tue, Jun 28, 2016 at 10:10 AM, Mathias Müller > > wrote:
>>
>> Hi all
>>
>> I have trained several language models and would like to combine
>> them with interpolate-lm.perl:
>>
>> 
>> https://github.com/moses-smt/mosesdecoder/blob/master/scripts/ems/support/interpolate-lm.perl
>>
>> As the language model tool, I always use KenLM, but looking at the
>> code of interpolate-lm.perl, it seems that the use of SRILM is
>> hard-coded in the script. I would like to avoid SRILM because, if I
>> understand correctly, its license does not permit use in commercial
>> products.
>>
>> My question is:
>>
>> Can I simply replace the call to SRILM with KenLM in my copy of
>> interpolate-lm.perl? Does KenLM have the functionality necessary for
>> language model combination, e.g. a substitute for SRILM's
>> "compute-best-mix"?
>>
>> Thanks for your help.
>> Mathias
>>
>> —
>>
>> Mathias Müller
>> AND-2-20
>> Institute of Computational Linguistics
>> University of Zurich
>> +41 44 635 75 81 
>> mathias.muel...@uzh.ch 
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu 
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Language model interpolation without SRILM

[Kenneth Heafield]

Log-linear interpolation is in KenLM in the lm/interpolate directory.
You'll want to get KenLM from github.com/kpu/kenlm and compile with Eigen.

Tuning log-linear weights is super slow, but applying them is reasonably
fast.  In total the tuning + applying weights time is comparable to SRILM.

https://kheafield.com/professional/edinburgh/interpolate_paper.pdf

Kenneth

On 06/28/2016 03:27 PM, Philipp Koehn wrote:
> Hi,
> 
> unfortunately, the interpolation of language models requires two pieces
> of code that only exist in SRILM: The EM training method to find weights
> for the language models, and the linear interpolation of the language
> models.
> 
> Maybe Ken and Lane can weigh in, if/when a replacement in KENLM will be
> available.
> 
> -phi
> 
> On Tue, Jun 28, 2016 at 10:10 AM, Mathias Müller  > wrote:
> 
> Hi all
> 
> I have trained several language models and would like to combine
> them with interpolate-lm.perl:
> 
> 
> https://github.com/moses-smt/mosesdecoder/blob/master/scripts/ems/support/interpolate-lm.perl
> 
> As the language model tool, I always use KenLM, but looking at the
> code of interpolate-lm.perl, it seems that the use of SRILM is
> hard-coded in the script. I would like to avoid SRILM because, if I
> understand correctly, its license does not permit use in commercial
> products.
> 
> My question is:
> 
> Can I simply replace the call to SRILM with KenLM in my copy of
> interpolate-lm.perl? Does KenLM have the functionality necessary for
> language model combination, e.g. a substitute for SRILM's
> "compute-best-mix"?
> 
> Thanks for your help.
> Mathias
> 
> —
> 
> Mathias Müller
> AND-2-20
> Institute of Computational Linguistics
> University of Zurich
> +41 44 635 75 81 
> mathias.muel...@uzh.ch 
> 
> ___
> Moses-support mailing list
> Moses-support@mit.edu 
> http://mailman.mit.edu/mailman/listinfo/moses-support
> 
> 
> 
> 
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
> 
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Language model interpolation without SRILM

[Philipp Koehn]

Hi,

unfortunately, the interpolation of language models requires two pieces of
code that only exist in SRILM: The EM training method to find weights for
the language models, and the linear interpolation of the language models.

Maybe Ken and Lane can weigh in, if/when a replacement in KENLM will be
available.

-phi

On Tue, Jun 28, 2016 at 10:10 AM, Mathias Müller 
wrote:

> Hi all
>
> I have trained several language models and would like to combine them with
> interpolate-lm.perl:
>
>
> https://github.com/moses-smt/mosesdecoder/blob/master/scripts/ems/support/interpolate-lm.perl
>
> As the language model tool, I always use KenLM, but looking at the code of
> interpolate-lm.perl, it seems that the use of SRILM is hard-coded in the
> script. I would like to avoid SRILM because, if I understand correctly, its
> license does not permit use in commercial products.
>
> My question is:
>
> Can I simply replace the call to SRILM with KenLM in my copy of
> interpolate-lm.perl? Does KenLM have the functionality necessary for
> language model combination, e.g. a substitute for SRILM's
> "compute-best-mix"?
>
> Thanks for your help.
> Mathias
>
> —
>
> Mathias Müller
> AND-2-20
> Institute of Computational Linguistics
> University of Zurich
> +41 44 635 75 81
> mathias.muel...@uzh.ch
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Language model interpolation without SRILM

[Mathias Müller]

Hi all

I have trained several language models and would like to combine them with
interpolate-lm.perl:

https://github.com/moses-smt/mosesdecoder/blob/master/scripts/ems/support/interpolate-lm.perl

As the language model tool, I always use KenLM, but looking at the code of
interpolate-lm.perl, it seems that the use of SRILM is hard-coded in the
script. I would like to avoid SRILM because, if I understand correctly, its
license does not permit use in commercial products.

My question is:

Can I simply replace the call to SRILM with KenLM in my copy of
interpolate-lm.perl? Does KenLM have the functionality necessary for
language model combination, e.g. a substitute for SRILM's "compute-best-mix
"?

Thanks for your help.
Mathias

—

Mathias Müller
AND-2-20
Institute of Computational Linguistics
University of Zurich
+41 44 635 75 81
mathias.muel...@uzh.ch
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support