date:20150801

Re: [Moses-support] nplm ngram total order in ems

2015-08-01 Thread John Joseph Morgan

Thanks Nick
> On Aug 1, 2015, at 2:57 PM, Nikolay Bogoychev  wrote:
> 
> Hey John,
> 
> This is correct. So imagine the situation of order 5 and source window 4:
>  
>  is aligned to  and your source window is 4: 4 tokens before and 
> after s0, which results in a 14gram in total.
> 
> Cheers,
> 
> Nick
> 
> On Sat, Aug 1, 2015 at 4:30 PM, John Joseph Morgan 
> mailto:johnjosephmor...@gmail.com>> wrote:
> I’m trying to run the toy bilingualnplm example with ems.
> The ngram order gets computed in experiment.perl on line 1868.
> The formula is:
> $order + 2 * $source_window + 1
> If $order is 5 and $source_window is 4 this formula gives 14.
> Is this correct?
> It doesn't seem right.
> 
> John
> ___
> Moses-support mailing list
> Moses-support@mit.edu 
> http://mailman.mit.edu/mailman/listinfo/moses-support 
> 
> 

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] nplm ngram total order in ems

2015-08-01 Thread Nikolay Bogoychev

Hey John,

This is correct. So imagine the situation of order 5 and source window 4:

 is aligned to  and your source window is 4: 4 tokens before and
after s0, which results in a 14gram in total.

Cheers,

Nick

On Sat, Aug 1, 2015 at 4:30 PM, John Joseph Morgan <
johnjosephmor...@gmail.com> wrote:

> I’m trying to run the toy bilingualnplm example with ems.
> The ngram order gets computed in experiment.perl on line 1868.
> The formula is:
> $order + 2 * $source_window + 1
> If $order is 5 and $source_window is 4 this formula gives 14.
> Is this correct?
> It doesn't seem right.
>
> John
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] nplm ngram total order in ems

2015-08-01 Thread John Joseph Morgan

I’m trying to run the toy bilingualnplm example with ems.
The ngram order gets computed in experiment.perl on line 1868.
The formula is:
$order + 2 * $source_window + 1
If $order is 5 and $source_window is 4 this formula gives 14.
Is this correct?
It doesn't seem right.

John
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Parallelizer multi core

2015-08-01 Thread Hieu Hoang



On 01/08/2015 12:51, Vincent Nguyen wrote:
>
> I am confused
> LM section type = 8 is just for KenLM in model creation / training, 
> right ?
no. type = 8 specify what LM is used during decoding - it puts the 
'KENLM' into the moses.ini

KENLM and IRSTLM also have binarizing programs to make the decoding use 
less memory. eg. if you want to binarize with kenlm, then add
   [LM]
   lm-binarizer = $moses-src-dir/bin/build_binary
> If I want to try IRSTLM for model creation training , then the LM 
> section should not contain any KenLM / type stuff
to creating a LM

with SRILM:
   lm-training = $srilm-bin-dir/ngram-count
with IRSLM:
   lm-training = "$moses-script-dir/generic/trainlm-irst2.perl -cores 
$cores -irst-dir $irstlm-dir -temp-dir $working-dir/tmp"
   settings = "-s msb -p 0"
with KenLM:
   lm-training = "$moses-script-dir/ems/support/lmplz-wrapper.perl -bin 
$moses-bin-dir/lmplz"
   settings = "--prune '0 0 1' -T $working-dir/lm -S 20%"

> BUT
> in the tuning, section it will not use the KenLM since the moses.ini 
> will have been generated with IRSTLM
it uses whatever type you specified in type = ??
>
> Am I wrong ?
>
>
>
> Le 01/08/2015 09:53, Hieu Hoang a écrit :
>> in the [LM] section, make sure
>> type = 8
>>
>>
>> On 01/08/2015 11:48, Vincent Nguyen wrote:
>>> fair enough.
>>> One thing though 
>>> When you use irstlm for LM and training, then the EMS crashes in multi
>>> thread at tuning (decoder)
>>>
>>> What is the easiest way so that the tuning part uses KenLM each time
>>> multi threads is activated ? (I mean in EMS).
>>>
>>> Vincent
>>>
>>>
>>>
>>>
>>> Le 01/08/2015 09:38, Hieu Hoang a écrit :
 oh alright. I've made it 4 cores. The example config files are 
 aimed at
 beginners with laptops.

 On 01/08/2015 10:35, Marcin Junczys-Dowmunt wrote:
> Hi, I agree with Nick. I am using a 64-core machine. "-threads 
> all" will
> grind to a still-stand. I am however fine with a few more threads, 
> say 16.
> Best,
> Marcin
>
> On 01.08.2015 00:35, Nikolay Bogoychev wrote:
>> Hey,
>>
>> I have opposed this change in the past for two reasons:
>>
>> Using more than 4 threads doesn't help unless the user is using
>> PhraseDictionaryCompact. See this issue
>> https://github.com/moses-smt/mosesdecoder/issues/39 in fact on most
>> machines you rarely want to run moses on all available threads.
>>
>> Also - threads all picks up virtual (hyper) threads which are in 
>> fact
>> harmful to performance.
>>
>> If you want to change the default I think it would be better to 
>> have a
>> sane default like 4.. It would boost performance for most people and
>> if you run it on machines with less available cores it would be not
>> too bad.
>>
>> Cheers,
>>
>> Nick
>>
>> On 31 Jul 2015 7:31 pm, "Hieu Hoang" > > wrote:
>>
>> good suggestion. Changed:
>> https://github.com/moses-smt/mosesdecoder/commit/f894dec0fd8d5b15eb16c35d3d2599338894ee9d
>> if you have any more suggestions, it's best if you can 
>> just me a
>> patch and I'll check it in
>>
>> On 31/07/2015 15:59, Vincent Nguyen wrote:
>>> for inexperienced people like me :)
>>> Add |--decoder-flags="-threads 4"|  is key
>>>
>>> if EMS config.basic had "-threads all" by default we 
>>> would gain A
>>> LOT of time.
>>>
>>> cheers,
>>>
>>> Vincent
>>>
>>>
>>> Le 29/07/2015 22:05, Vincent Nguyen a écrit :
 Hi,

 I am wondering what tasks of the EMS are really 
 parallelized.
 I activated the script line + 8 cores.

 Training / binarizing / Tuning all make only one core 
 to actually work.

 Am I correct ?
 ___
 Moses-support mailing list
 Moses-support@mit.edu 
 http://mailman.mit.edu/mailman/listinfo/moses-support
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu 
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>> --
>> Hieu Hoang
>> Researcher
>> New York University, Abu Dhabi
>> http://www.hoang.co.uk/hieu
>>
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu 
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>> ___
>> Moses-support mailing list
>> Moses-support@

Re: [Moses-support] Parallelizer multi core

2015-08-01 Thread Vincent Nguyen


I am confused
LM section type = 8 is just for KenLM in model creation / training, right ?
If I want to try IRSTLM for model creation training , then the LM 
section should not contain any KenLM / type stuff

BUT
in the tuning, section it will not use the KenLM since the moses.ini 
will have been generated with IRSTLM

Am I wrong ?



Le 01/08/2015 09:53, Hieu Hoang a écrit :
> in the [LM] section, make sure
> type = 8
>
>
> On 01/08/2015 11:48, Vincent Nguyen wrote:
>> fair enough.
>> One thing though 
>> When you use irstlm for LM and training, then the EMS crashes in multi
>> thread at tuning (decoder)
>>
>> What is the easiest way so that the tuning part uses KenLM each time
>> multi threads is activated ? (I mean in EMS).
>>
>> Vincent
>>
>>
>>
>>
>> Le 01/08/2015 09:38, Hieu Hoang a écrit :
>>> oh alright. I've made it 4 cores. The example config files are aimed at
>>> beginners with laptops.
>>>
>>> On 01/08/2015 10:35, Marcin Junczys-Dowmunt wrote:
 Hi, I agree with Nick. I am using a 64-core machine. "-threads all" will
 grind to a still-stand. I am however fine with a few more threads, say 16.
 Best,
 Marcin

 On 01.08.2015 00:35, Nikolay Bogoychev wrote:
> Hey,
>
> I have opposed this change in the past for two reasons:
>
> Using more than 4 threads doesn't help unless the user is using
> PhraseDictionaryCompact. See this issue
> https://github.com/moses-smt/mosesdecoder/issues/39 in fact on most
> machines you rarely want to run moses on all available threads.
>
> Also - threads all picks up virtual (hyper) threads which are in fact
> harmful to performance.
>
> If you want to change the default I think it would be better to have a
> sane default like 4.. It would boost performance for most people and
> if you run it on machines with less available cores it would be not
> too bad.
>
> Cheers,
>
> Nick
>
> On 31 Jul 2015 7:31 pm, "Hieu Hoang"  > wrote:
>
> good suggestion. Changed:
> 
> https://github.com/moses-smt/mosesdecoder/commit/f894dec0fd8d5b15eb16c35d3d2599338894ee9d
> if you have any more suggestions, it's best if you can just me a
> patch and I'll check it in
>
> On 31/07/2015 15:59, Vincent Nguyen wrote:
>> for inexperienced people like me :)
>> Add |--decoder-flags="-threads 4"|  is key
>>
>> if EMS config.basic had "-threads all" by default we would gain A
>> LOT of time.
>>
>> cheers,
>>
>> Vincent
>>
>>
>> Le 29/07/2015 22:05, Vincent Nguyen a écrit :
>>> Hi,
>>>
>>> I am wondering what tasks of the EMS are really parallelized.
>>> I activated the script line + 8 cores.
>>>
>>> Training / binarizing / Tuning all make only one core to 
>>> actually work.
>>>
>>> Am I correct ?
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu  
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu  
>> http://mailman.mit.edu/mailman/listinfo/moses-support
> --
> Hieu Hoang
> Researcher
> New York University, Abu Dhabi
> http://www.hoang.co.uk/hieu
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu 
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support

>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Language model creation error

2015-08-01 Thread kalu mera

Dear Members,
I am trying to create a language model creation, I entered this command
kalumera@kalumera-Satellite-C50-A534:~/mosesdecoder$ ./bjam
--with-boost=~/workspace/temp/boost_1_55_0 -j4

but the build failed

Please check the attachment for the command i entered and the error, and
help advise me on how to rectify the problem

Christine


language model creation.odt
Description: application/vnd.oasis.opendocument.text
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Parallelizer multi core

2015-08-01 Thread Hieu Hoang

in the [LM] section, make sure
   type = 8


On 01/08/2015 11:48, Vincent Nguyen wrote:
> fair enough.
> One thing though 
> When you use irstlm for LM and training, then the EMS crashes in multi
> thread at tuning (decoder)
>
> What is the easiest way so that the tuning part uses KenLM each time
> multi threads is activated ? (I mean in EMS).
>
> Vincent
>
>
>
>
> Le 01/08/2015 09:38, Hieu Hoang a écrit :
>> oh alright. I've made it 4 cores. The example config files are aimed at
>> beginners with laptops.
>>
>> On 01/08/2015 10:35, Marcin Junczys-Dowmunt wrote:
>>> Hi, I agree with Nick. I am using a 64-core machine. "-threads all" will
>>> grind to a still-stand. I am however fine with a few more threads, say 16.
>>> Best,
>>> Marcin
>>>
>>> On 01.08.2015 00:35, Nikolay Bogoychev wrote:
 Hey,

 I have opposed this change in the past for two reasons:

 Using more than 4 threads doesn't help unless the user is using
 PhraseDictionaryCompact. See this issue
 https://github.com/moses-smt/mosesdecoder/issues/39 in fact on most
 machines you rarely want to run moses on all available threads.

 Also - threads all picks up virtual (hyper) threads which are in fact
 harmful to performance.

 If you want to change the default I think it would be better to have a
 sane default like 4.. It would boost performance for most people and
 if you run it on machines with less available cores it would be not
 too bad.

 Cheers,

 Nick

 On 31 Jul 2015 7:31 pm, "Hieu Hoang" >>> > wrote:

good suggestion. Changed:

 https://github.com/moses-smt/mosesdecoder/commit/f894dec0fd8d5b15eb16c35d3d2599338894ee9d
if you have any more suggestions, it's best if you can just me a
patch and I'll check it in

On 31/07/2015 15:59, Vincent Nguyen wrote:
>for inexperienced people like me :)
>Add |--decoder-flags="-threads 4"|  is key
>
>if EMS config.basic had "-threads all" by default we would gain A
>LOT of time.
>
>cheers,
>
>Vincent
>
>
>Le 29/07/2015 22:05, Vincent Nguyen a écrit :
>>Hi,
>>
>>I am wondering what tasks of the EMS are really parallelized.
>>I activated the script line + 8 cores.
>>
>>Training / binarizing / Tuning all make only one core to actually 
>> work.
>>
>>Am I correct ?
>>___
>>Moses-support mailing list
>>Moses-support@mit.edu  
>>http://mailman.mit.edu/mailman/listinfo/moses-support
>___
>Moses-support mailing list
>Moses-support@mit.edu  
>http://mailman.mit.edu/mailman/listinfo/moses-support
--
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu


___
Moses-support mailing list
Moses-support@mit.edu 
http://mailman.mit.edu/mailman/listinfo/moses-support



 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>

-- 
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Parallelizer multi core

2015-08-01 Thread Vincent Nguyen

fair enough.
One thing though 
When you use irstlm for LM and training, then the EMS crashes in multi 
thread at tuning (decoder)

What is the easiest way so that the tuning part uses KenLM each time 
multi threads is activated ? (I mean in EMS).

Vincent




Le 01/08/2015 09:38, Hieu Hoang a écrit :
> oh alright. I've made it 4 cores. The example config files are aimed at
> beginners with laptops.
>
> On 01/08/2015 10:35, Marcin Junczys-Dowmunt wrote:
>> Hi, I agree with Nick. I am using a 64-core machine. "-threads all" will
>> grind to a still-stand. I am however fine with a few more threads, say 16.
>> Best,
>> Marcin
>>
>> On 01.08.2015 00:35, Nikolay Bogoychev wrote:
>>> Hey,
>>>
>>> I have opposed this change in the past for two reasons:
>>>
>>> Using more than 4 threads doesn't help unless the user is using
>>> PhraseDictionaryCompact. See this issue
>>> https://github.com/moses-smt/mosesdecoder/issues/39 in fact on most
>>> machines you rarely want to run moses on all available threads.
>>>
>>> Also - threads all picks up virtual (hyper) threads which are in fact
>>> harmful to performance.
>>>
>>> If you want to change the default I think it would be better to have a
>>> sane default like 4.. It would boost performance for most people and
>>> if you run it on machines with less available cores it would be not
>>> too bad.
>>>
>>> Cheers,
>>>
>>> Nick
>>>
>>> On 31 Jul 2015 7:31 pm, "Hieu Hoang" >> > wrote:
>>>
>>>   good suggestion. Changed:
>>>   
>>> https://github.com/moses-smt/mosesdecoder/commit/f894dec0fd8d5b15eb16c35d3d2599338894ee9d
>>>   if you have any more suggestions, it's best if you can just me a
>>>   patch and I'll check it in
>>>
>>>   On 31/07/2015 15:59, Vincent Nguyen wrote:
   for inexperienced people like me :)
   Add |--decoder-flags="-threads 4"|  is key

   if EMS config.basic had "-threads all" by default we would gain A
   LOT of time.

   cheers,

   Vincent


   Le 29/07/2015 22:05, Vincent Nguyen a écrit :
>   Hi,
>
>   I am wondering what tasks of the EMS are really parallelized.
>   I activated the script line + 8 cores.
>
>   Training / binarizing / Tuning all make only one core to actually 
> work.
>
>   Am I correct ?
>   ___
>   Moses-support mailing list
>   Moses-support@mit.edu  
>   http://mailman.mit.edu/mailman/listinfo/moses-support

   ___
   Moses-support mailing list
   Moses-support@mit.edu  
   http://mailman.mit.edu/mailman/listinfo/moses-support
>>>   --
>>>   Hieu Hoang
>>>   Researcher
>>>   New York University, Abu Dhabi
>>>   http://www.hoang.co.uk/hieu
>>>
>>>
>>>   ___
>>>   Moses-support mailing list
>>>   Moses-support@mit.edu 
>>>   http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>>
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Parallelizer multi core

2015-08-01 Thread Hieu Hoang

oh alright. I've made it 4 cores. The example config files are aimed at 
beginners with laptops.

On 01/08/2015 10:35, Marcin Junczys-Dowmunt wrote:
> Hi, I agree with Nick. I am using a 64-core machine. "-threads all" will
> grind to a still-stand. I am however fine with a few more threads, say 16.
> Best,
> Marcin
>
> On 01.08.2015 00:35, Nikolay Bogoychev wrote:
>> Hey,
>>
>> I have opposed this change in the past for two reasons:
>>
>> Using more than 4 threads doesn't help unless the user is using
>> PhraseDictionaryCompact. See this issue
>> https://github.com/moses-smt/mosesdecoder/issues/39 in fact on most
>> machines you rarely want to run moses on all available threads.
>>
>> Also - threads all picks up virtual (hyper) threads which are in fact
>> harmful to performance.
>>
>> If you want to change the default I think it would be better to have a
>> sane default like 4.. It would boost performance for most people and
>> if you run it on machines with less available cores it would be not
>> too bad.
>>
>> Cheers,
>>
>> Nick
>>
>> On 31 Jul 2015 7:31 pm, "Hieu Hoang" > > wrote:
>>
>>  good suggestion. Changed:
>>  
>> https://github.com/moses-smt/mosesdecoder/commit/f894dec0fd8d5b15eb16c35d3d2599338894ee9d
>>  if you have any more suggestions, it's best if you can just me a
>>  patch and I'll check it in
>>
>>  On 31/07/2015 15:59, Vincent Nguyen wrote:
>>>  for inexperienced people like me :)
>>>  Add |--decoder-flags="-threads 4"|  is key
>>>
>>>  if EMS config.basic had "-threads all" by default we would gain A
>>>  LOT of time.
>>>
>>>  cheers,
>>>
>>>  Vincent
>>>
>>>
>>>  Le 29/07/2015 22:05, Vincent Nguyen a écrit :
  Hi,

  I am wondering what tasks of the EMS are really parallelized.
  I activated the script line + 8 cores.

  Training / binarizing / Tuning all make only one core to actually 
 work.

  Am I correct ?
  ___
  Moses-support mailing list
  Moses-support@mit.edu  
  http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>>  ___
>>>  Moses-support mailing list
>>>  Moses-support@mit.edu  
>>>  http://mailman.mit.edu/mailman/listinfo/moses-support
>>  --
>>  Hieu Hoang
>>  Researcher
>>  New York University, Abu Dhabi
>>  http://www.hoang.co.uk/hieu
>>
>>
>>  ___
>>  Moses-support mailing list
>>  Moses-support@mit.edu 
>>  http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>

-- 
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] nplm ngram total order in ems

Re: [Moses-support] nplm ngram total order in ems

[Moses-support] nplm ngram total order in ems

Re: [Moses-support] Parallelizer multi core

Re: [Moses-support] Parallelizer multi core

[Moses-support] Language model creation error

Re: [Moses-support] Parallelizer multi core

Re: [Moses-support] Parallelizer multi core

Re: [Moses-support] Parallelizer multi core

9 matches

Site Navigation

Mail list logo

Footer information