Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Shubham Khandelwal]

Hello,

I was using http://www.statmt.org/moses/RELEASE-3.0/models/fr-
en/tuning/moses.tuned.ini.5
There I had converted phrase table and reordering table to their compact
format and also made some changes in this moses.tuned.ini.5 file (attached
in this email) to decode fastly.

Now, this time, I am using *moses *binary file which I just directly
downloaded from here:
http://www.statmt.org/moses/RELEASE-3.0/binaries/linux-64bit/bin/
(just to be sure that I am using correct binary moses file)

And just run
-> ~/moses -f moses.tuned.ini.5 -threads all
Then after creating input-output object, it took 60 seconds to decode just
for 3 words. Am I missing or doing something wrong here.

Also, I do not wish to run this command ( ~/moses -f moses.tuned.ini.5
-threads all) every time for every input. Is there any way in Moses by
which all models will load in memory for forever and I can just pass a
input and get output in real time without using this command again and
again.

Looking forward for your response.

Thanking You.

On Thu, Dec 15, 2016 at 4:14 PM, Hieu Hoang  wrote:

> there is no limit to the number of words.
>
> if you are using the premade models, make sure you are using tuned
> moses.ini file in
>   tuning/moses.tuned.ini.?
>
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 13 December 2016 at 09:32, Shubham Khandelwal 
> wrote:
>
>> Thanks. Hieu. I understood now.
>> Also, Is there any limit of number of words for the translation because
>> when I use this command: *~/mosesdecoder/bin/moses -f moses.ini*
>> Then, it translates only some words which are in the beginning. I mean,
>> it does not consume all words for the translation after Created
>> input-output object.
>> Is there any way by which I can control or remove this limit.
>>
>> Thanks.
>>
>> On Mon, Dec 12, 2016 at 7:17 PM, Hieu Hoang  wrote:
>>
>>> there are actually 7 different configurations. You have to look at the
>>> config file in
>>>steps/?/config.?
>>> For fr-en:
>>>   1. phrase-based, truecased
>>>   2. phrase-based, lowercased then recased
>>>   3. hierarchical model, lowercased then recased
>>>   4. phrase-based, lowercased then recased. Using target side word + pos
>>> factors
>>>   5. Like (2) but using batch-mira to tune
>>>   6. Like (2) but using PRO to tune
>>>   7. Like (2) but using CreateOnDiskPt to create binary phrase table
>>> You can see the BLEU scores in
>>>   evaluation/report.*
>>>
>>> Hieu Hoang
>>> http://www.hoang.co.uk/hieu
>>>
>>> On 12 December 2016 at 13:28, Shubham Khandelwal 
>>> wrote:
>>>
 Okay Thanks Hieu. I will try it with 1TB HD-memory machine.
 Btw I can see there are 4 pre-made models available for fr-en and de-en
 (http://www.statmt.org/moses/RELEASE-3.0/models/fr-en/model/   and
 http://www.statmt.org/moses/RELEASE-3.0/models/de-en/model/). Can you
 please tell me among these 4, which one is better model(in terms of bleu
 score) except the huge model which is already there in both, as I can not
 understand how analysis is shown in steps folder.
 Also, Are all these pre-made models hierarchical model ?


 On Mon, Dec 12, 2016 at 6:09 PM, Hieu Hoang 
 wrote:

>
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 10 December 2016 at 14:06, Shubham Khandelwal 
> wrote:
>
>> Yes, CreateOnDiskPt command executed without any error.
>>
>> There are 5 files in this phrase-table.3.folder: Misc.dat
>> , Source.dat, TargetColl.dat, TargetInd.dat, Vocab.dat
>> *Misc.dat  and Vocab.dat files are empty. *
>> I just checked that my hard-disk memory is full as this folder
>> took 165G already. So may be, due to this reason those 2 files are empty.
>> But CreateOnDiskPt command should throw an error of *No space left
>> on machine *when it stopped.
>> Let me know if no space on my machine, is the issue or not so that I
>> can go for better device having more hard-disk memory.
>>
> Good idea. Not sure who's going to do it but if you do it, please send
> me a patch & I'll check it in
>
>>
>> Also May I know that How much memory phrase-table.3.folder has in
>> general when CreateOnDiskPt command executes completely
>> as phrase-table.3.gz size is only 23GB.
>>
> I'm not too sure. Try it on a disk with 1TB and please report back
> what you find for future reference
>
>>
>> Thanking You.
>>
>>
>> On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang 
>> wrote:
>>
>>> strange, did the CreateOnDiskPt command execute ok, ie. with no
>>> error?
>>>
>>> Does this file exist:
>>>/home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat
>>> If you do
>>>   cat Misc.dat
>>> what does it say?
>>>
>>> Hieu Hoang
>>> 

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Hieu Hoang]

there is no limit to the number of words.

if you are using the premade models, make sure you are using tuned
moses.ini file in
  tuning/moses.tuned.ini.?


Hieu Hoang
http://www.hoang.co.uk/hieu

On 13 December 2016 at 09:32, Shubham Khandelwal 
wrote:

> Thanks. Hieu. I understood now.
> Also, Is there any limit of number of words for the translation because
> when I use this command: *~/mosesdecoder/bin/moses -f moses.ini*
> Then, it translates only some words which are in the beginning. I mean, it
> does not consume all words for the translation after Created input-output
> object.
> Is there any way by which I can control or remove this limit.
>
> Thanks.
>
> On Mon, Dec 12, 2016 at 7:17 PM, Hieu Hoang  wrote:
>
>> there are actually 7 different configurations. You have to look at the
>> config file in
>>steps/?/config.?
>> For fr-en:
>>   1. phrase-based, truecased
>>   2. phrase-based, lowercased then recased
>>   3. hierarchical model, lowercased then recased
>>   4. phrase-based, lowercased then recased. Using target side word + pos
>> factors
>>   5. Like (2) but using batch-mira to tune
>>   6. Like (2) but using PRO to tune
>>   7. Like (2) but using CreateOnDiskPt to create binary phrase table
>> You can see the BLEU scores in
>>   evaluation/report.*
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 12 December 2016 at 13:28, Shubham Khandelwal 
>> wrote:
>>
>>> Okay Thanks Hieu. I will try it with 1TB HD-memory machine.
>>> Btw I can see there are 4 pre-made models available for fr-en and de-en (
>>> http://www.statmt.org/moses/RELEASE-3.0/models/fr-en/model/   and
>>> http://www.statmt.org/moses/RELEASE-3.0/models/de-en/model/). Can you
>>> please tell me among these 4, which one is better model(in terms of bleu
>>> score) except the huge model which is already there in both, as I can not
>>> understand how analysis is shown in steps folder.
>>> Also, Are all these pre-made models hierarchical model ?
>>>
>>>
>>> On Mon, Dec 12, 2016 at 6:09 PM, Hieu Hoang  wrote:
>>>


 Hieu Hoang
 http://www.hoang.co.uk/hieu

 On 10 December 2016 at 14:06, Shubham Khandelwal 
 wrote:

> Yes, CreateOnDiskPt command executed without any error.
>
> There are 5 files in this phrase-table.3.folder: Misc.dat
> , Source.dat, TargetColl.dat, TargetInd.dat, Vocab.dat
> *Misc.dat  and Vocab.dat files are empty. *
> I just checked that my hard-disk memory is full as this folder
> took 165G already. So may be, due to this reason those 2 files are empty.
> But CreateOnDiskPt command should throw an error of *No space left on
> machine *when it stopped.
> Let me know if no space on my machine, is the issue or not so that I
> can go for better device having more hard-disk memory.
>
 Good idea. Not sure who's going to do it but if you do it, please send
 me a patch & I'll check it in

>
> Also May I know that How much memory phrase-table.3.folder has in
> general when CreateOnDiskPt command executes completely
> as phrase-table.3.gz size is only 23GB.
>
 I'm not too sure. Try it on a disk with 1TB and please report back what
 you find for future reference

>
> Thanking You.
>
>
> On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang 
> wrote:
>
>> strange, did the CreateOnDiskPt command execute ok, ie. with no error?
>>
>> Does this file exist:
>>/home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat
>> If you do
>>   cat Misc.dat
>> what does it say?
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 10 December 2016 at 11:30, Shubham Khandelwal > > wrote:
>>
>>> Hi,
>>>
>>> Thanks Hieu for your reply.
>>> I have used *CreateOnDiskPt* to binarize the model and stored in
>>> *phrase-table.3.folder *using following command:
>>>
>>>
>>> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2  phrase-table.3.gz
>>> phrase-table.3.folder*
>>>
>>> Also I have made changes in *moses.ini.3 (*i.e. I have converted
>>> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*)
>>>  *
>>>
>>> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
>>> path=/home/shubham/models/fr-en/phrase-table.3.folder
>>> input-factor=0 output-factor=0
>>>
>>> Now, when I run it using :* ~/mosesdecoder/bin/moses -f
>>> moses.ini.3 * , it gave following error after *Created input-output
>>> object*:
>>>
>>> *terminate called after throwing an instance of 'util::Exception'*
>>> *  what():  OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t
>>> OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw 
>>> util::Exception
>>> because `iter == m_miscInfo.end()'.*
>>> *Couldn't 

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Shubham Khandelwal]

Hi,
This is just the reminder of my previous email.

Thanks.

Regards,
Shubham

On Dec 13, 2016 3:02 PM, "Shubham Khandelwal"  wrote:

> Thanks. Hieu. I understood now.
> Also, Is there any limit of number of words for the translation because
> when I use this command: *~/mosesdecoder/bin/moses -f moses.ini*
> Then, it translates only some words which are in the beginning. I mean, it
> does not consume all words for the translation after Created input-output
> object.
> Is there any way by which I can control or remove this limit.
>
> Thanks.
>
> On Mon, Dec 12, 2016 at 7:17 PM, Hieu Hoang  wrote:
>
>> there are actually 7 different configurations. You have to look at the
>> config file in
>>steps/?/config.?
>> For fr-en:
>>   1. phrase-based, truecased
>>   2. phrase-based, lowercased then recased
>>   3. hierarchical model, lowercased then recased
>>   4. phrase-based, lowercased then recased. Using target side word + pos
>> factors
>>   5. Like (2) but using batch-mira to tune
>>   6. Like (2) but using PRO to tune
>>   7. Like (2) but using CreateOnDiskPt to create binary phrase table
>> You can see the BLEU scores in
>>   evaluation/report.*
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 12 December 2016 at 13:28, Shubham Khandelwal 
>> wrote:
>>
>>> Okay Thanks Hieu. I will try it with 1TB HD-memory machine.
>>> Btw I can see there are 4 pre-made models available for fr-en and de-en (
>>> http://www.statmt.org/moses/RELEASE-3.0/models/fr-en/model/   and
>>> http://www.statmt.org/moses/RELEASE-3.0/models/de-en/model/). Can you
>>> please tell me among these 4, which one is better model(in terms of bleu
>>> score) except the huge model which is already there in both, as I can not
>>> understand how analysis is shown in steps folder.
>>> Also, Are all these pre-made models hierarchical model ?
>>>
>>>
>>> On Mon, Dec 12, 2016 at 6:09 PM, Hieu Hoang  wrote:
>>>


 Hieu Hoang
 http://www.hoang.co.uk/hieu

 On 10 December 2016 at 14:06, Shubham Khandelwal 
 wrote:

> Yes, CreateOnDiskPt command executed without any error.
>
> There are 5 files in this phrase-table.3.folder: Misc.dat
> , Source.dat, TargetColl.dat, TargetInd.dat, Vocab.dat
> *Misc.dat  and Vocab.dat files are empty. *
> I just checked that my hard-disk memory is full as this folder
> took 165G already. So may be, due to this reason those 2 files are empty.
> But CreateOnDiskPt command should throw an error of *No space left on
> machine *when it stopped.
> Let me know if no space on my machine, is the issue or not so that I
> can go for better device having more hard-disk memory.
>
 Good idea. Not sure who's going to do it but if you do it, please send
 me a patch & I'll check it in

>
> Also May I know that How much memory phrase-table.3.folder has in
> general when CreateOnDiskPt command executes completely
> as phrase-table.3.gz size is only 23GB.
>
 I'm not too sure. Try it on a disk with 1TB and please report back what
 you find for future reference

>
> Thanking You.
>
>
> On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang 
> wrote:
>
>> strange, did the CreateOnDiskPt command execute ok, ie. with no error?
>>
>> Does this file exist:
>>/home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat
>> If you do
>>   cat Misc.dat
>> what does it say?
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 10 December 2016 at 11:30, Shubham Khandelwal > > wrote:
>>
>>> Hi,
>>>
>>> Thanks Hieu for your reply.
>>> I have used *CreateOnDiskPt* to binarize the model and stored in
>>> *phrase-table.3.folder *using following command:
>>>
>>>
>>> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2  phrase-table.3.gz
>>> phrase-table.3.folder*
>>>
>>> Also I have made changes in *moses.ini.3 (*i.e. I have converted
>>> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*)
>>>  *
>>>
>>> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
>>> path=/home/shubham/models/fr-en/phrase-table.3.folder
>>> input-factor=0 output-factor=0
>>>
>>> Now, when I run it using :* ~/mosesdecoder/bin/moses -f
>>> moses.ini.3 * , it gave following error after *Created input-output
>>> object*:
>>>
>>> *terminate called after throwing an instance of 'util::Exception'*
>>> *  what():  OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t
>>> OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw 
>>> util::Exception
>>> because `iter == m_miscInfo.end()'.*
>>> *Couldn't find value for key NumSourceFactors*
>>> *Aborted (core dumped)*
>>>
>>> Here, I do not know that what key 

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Shubham Khandelwal]

Thanks. Hieu. I understood now.
Also, Is there any limit of number of words for the translation because
when I use this command: *~/mosesdecoder/bin/moses -f moses.ini*
Then, it translates only some words which are in the beginning. I mean, it
does not consume all words for the translation after Created input-output
object.
Is there any way by which I can control or remove this limit.

Thanks.

On Mon, Dec 12, 2016 at 7:17 PM, Hieu Hoang  wrote:

> there are actually 7 different configurations. You have to look at the
> config file in
>steps/?/config.?
> For fr-en:
>   1. phrase-based, truecased
>   2. phrase-based, lowercased then recased
>   3. hierarchical model, lowercased then recased
>   4. phrase-based, lowercased then recased. Using target side word + pos
> factors
>   5. Like (2) but using batch-mira to tune
>   6. Like (2) but using PRO to tune
>   7. Like (2) but using CreateOnDiskPt to create binary phrase table
> You can see the BLEU scores in
>   evaluation/report.*
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 12 December 2016 at 13:28, Shubham Khandelwal 
> wrote:
>
>> Okay Thanks Hieu. I will try it with 1TB HD-memory machine.
>> Btw I can see there are 4 pre-made models available for fr-en and de-en (
>> http://www.statmt.org/moses/RELEASE-3.0/models/fr-en/model/   and
>> http://www.statmt.org/moses/RELEASE-3.0/models/de-en/model/). Can you
>> please tell me among these 4, which one is better model(in terms of bleu
>> score) except the huge model which is already there in both, as I can not
>> understand how analysis is shown in steps folder.
>> Also, Are all these pre-made models hierarchical model ?
>>
>>
>> On Mon, Dec 12, 2016 at 6:09 PM, Hieu Hoang  wrote:
>>
>>>
>>>
>>> Hieu Hoang
>>> http://www.hoang.co.uk/hieu
>>>
>>> On 10 December 2016 at 14:06, Shubham Khandelwal 
>>> wrote:
>>>
 Yes, CreateOnDiskPt command executed without any error.

 There are 5 files in this phrase-table.3.folder: Misc.dat
 , Source.dat, TargetColl.dat, TargetInd.dat, Vocab.dat
 *Misc.dat  and Vocab.dat files are empty. *
 I just checked that my hard-disk memory is full as this folder
 took 165G already. So may be, due to this reason those 2 files are empty.
 But CreateOnDiskPt command should throw an error of *No space left on
 machine *when it stopped.
 Let me know if no space on my machine, is the issue or not so that I
 can go for better device having more hard-disk memory.

>>> Good idea. Not sure who's going to do it but if you do it, please send
>>> me a patch & I'll check it in
>>>

 Also May I know that How much memory phrase-table.3.folder has in
 general when CreateOnDiskPt command executes completely
 as phrase-table.3.gz size is only 23GB.

>>> I'm not too sure. Try it on a disk with 1TB and please report back what
>>> you find for future reference
>>>

 Thanking You.


 On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang 
 wrote:

> strange, did the CreateOnDiskPt command execute ok, ie. with no error?
>
> Does this file exist:
>/home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat
> If you do
>   cat Misc.dat
> what does it say?
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 10 December 2016 at 11:30, Shubham Khandelwal 
> wrote:
>
>> Hi,
>>
>> Thanks Hieu for your reply.
>> I have used *CreateOnDiskPt* to binarize the model and stored in
>> *phrase-table.3.folder *using following command:
>>
>>
>> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2  phrase-table.3.gz
>> phrase-table.3.folder*
>>
>> Also I have made changes in *moses.ini.3 (*i.e. I have converted
>> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*)  *
>>
>> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
>> path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0
>> output-factor=0
>>
>> Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * ,
>> it gave following error after *Created input-output object*:
>>
>> *terminate called after throwing an instance of 'util::Exception'*
>> *  what():  OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t
>> OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw 
>> util::Exception
>> because `iter == m_miscInfo.end()'.*
>> *Couldn't find value for key NumSourceFactors*
>> *Aborted (core dumped)*
>>
>> Here, I do not know that what key value should I pass and how ? Can
>> you please help me in this regard.
>>
>> Thank you so much for your help.
>>
>> Regards,
>> Shubham
>>
>> On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang 
>> wrote:
>>
>>> This is a hierarchical model. You must 

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Shubham Khandelwal]

Okay Thanks Hieu. I will try it with 1TB HD-memory machine.
Btw I can see there are 4 pre-made models available for fr-en and de-en (
http://www.statmt.org/moses/RELEASE-3.0/models/fr-en/model/   and
http://www.statmt.org/moses/RELEASE-3.0/models/de-en/model/). Can you
please tell me among these 4, which one is better model(in terms of bleu
score) except the huge model which is already there in both, as I can not
understand how analysis is shown in steps folder.
Also, Are all these pre-made models hierarchical model ?


On Mon, Dec 12, 2016 at 6:09 PM, Hieu Hoang  wrote:

>
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 10 December 2016 at 14:06, Shubham Khandelwal 
> wrote:
>
>> Yes, CreateOnDiskPt command executed without any error.
>>
>> There are 5 files in this phrase-table.3.folder: Misc.dat , Source.dat,
>> TargetColl.dat, TargetInd.dat, Vocab.dat
>> *Misc.dat  and Vocab.dat files are empty. *
>> I just checked that my hard-disk memory is full as this folder took 165G
>> already. So may be, due to this reason those 2 files are empty. But 
>> CreateOnDiskPt
>> command should throw an error of *No space left on machine *when it
>> stopped.
>> Let me know if no space on my machine, is the issue or not so that I can
>> go for better device having more hard-disk memory.
>>
> Good idea. Not sure who's going to do it but if you do it, please send me
> a patch & I'll check it in
>
>>
>> Also May I know that How much memory phrase-table.3.folder has in
>> general when CreateOnDiskPt command executes completely
>> as phrase-table.3.gz size is only 23GB.
>>
> I'm not too sure. Try it on a disk with 1TB and please report back what
> you find for future reference
>
>>
>> Thanking You.
>>
>>
>> On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang  wrote:
>>
>>> strange, did the CreateOnDiskPt command execute ok, ie. with no error?
>>>
>>> Does this file exist:
>>>/home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat
>>> If you do
>>>   cat Misc.dat
>>> what does it say?
>>>
>>> Hieu Hoang
>>> http://www.hoang.co.uk/hieu
>>>
>>> On 10 December 2016 at 11:30, Shubham Khandelwal 
>>> wrote:
>>>
 Hi,

 Thanks Hieu for your reply.
 I have used *CreateOnDiskPt* to binarize the model and stored in
 *phrase-table.3.folder *using following command:


 *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2  phrase-table.3.gz
 phrase-table.3.folder*

 Also I have made changes in *moses.ini.3 (*i.e. I have converted
 *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*)  *

 PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
 path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0
 output-factor=0

 Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * ,
 it gave following error after *Created input-output object*:

 *terminate called after throwing an instance of 'util::Exception'*
 *  what():  OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t
 OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw util::Exception
 because `iter == m_miscInfo.end()'.*
 *Couldn't find value for key NumSourceFactors*
 *Aborted (core dumped)*

 Here, I do not know that what key value should I pass and how ? Can you
 please help me in this regard.

 Thank you so much for your help.

 Regards,
 Shubham

 On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang  wrote:

> This is a hierarchical model. You must binarize with CreateOnDiskPt
> for this model
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 9 December 2016 at 08:18, Shubham Khandelwal 
> wrote:
>
>> Hello,
>>
>> Thanks. It worked now. I have created compact phrase table.
>> Now, when I am running it using following command:
>>
>> *~/mosesdecoder/bin/moses
>> -f ~/Translate/models/de-en/model/moses.ini.2 -threads all*
>>
>> Then, after creating input-output object, it gave following
>> segmentation fault   error:
>>
>> Created input-output object : [14.796] seconds
>> Ich bin ein Student
>> Line 0: Initialize search took 0.000 seconds total
>> Translating:  Ich bin ein Student   ||| [0,0]=X (1) [0,1]=X
>> (1) [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X 
>> (1)
>> [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1)
>> [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1)
>> [5,5]=X (1)
>>
>> Segmentation fault (core dumped)
>>
>> In my machine, I have 40GB RAM but still I am confused why it gave
>> this error.
>> Can you please help me in this regard. I have attached moses.ini.2
>> for your reference.
>>
>> Thanks.
>>
>> Regards,
>> Shubham
>>
>> On 

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Hieu Hoang]

there are actually 7 different configurations. You have to look at the
config file in
   steps/?/config.?
For fr-en:
  1. phrase-based, truecased
  2. phrase-based, lowercased then recased
  3. hierarchical model, lowercased then recased
  4. phrase-based, lowercased then recased. Using target side word + pos
factors
  5. Like (2) but using batch-mira to tune
  6. Like (2) but using PRO to tune
  7. Like (2) but using CreateOnDiskPt to create binary phrase table
You can see the BLEU scores in
  evaluation/report.*

Hieu Hoang
http://www.hoang.co.uk/hieu

On 12 December 2016 at 13:28, Shubham Khandelwal 
wrote:

> Okay Thanks Hieu. I will try it with 1TB HD-memory machine.
> Btw I can see there are 4 pre-made models available for fr-en and de-en (
> http://www.statmt.org/moses/RELEASE-3.0/models/fr-en/model/   and
> http://www.statmt.org/moses/RELEASE-3.0/models/de-en/model/). Can you
> please tell me among these 4, which one is better model(in terms of bleu
> score) except the huge model which is already there in both, as I can not
> understand how analysis is shown in steps folder.
> Also, Are all these pre-made models hierarchical model ?
>
>
> On Mon, Dec 12, 2016 at 6:09 PM, Hieu Hoang  wrote:
>
>>
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 10 December 2016 at 14:06, Shubham Khandelwal 
>> wrote:
>>
>>> Yes, CreateOnDiskPt command executed without any error.
>>>
>>> There are 5 files in this phrase-table.3.folder: Misc.dat , Source.dat,
>>> TargetColl.dat, TargetInd.dat, Vocab.dat
>>> *Misc.dat  and Vocab.dat files are empty. *
>>> I just checked that my hard-disk memory is full as this folder took 165G
>>> already. So may be, due to this reason those 2 files are empty. But 
>>> CreateOnDiskPt
>>> command should throw an error of *No space left on machine *when it
>>> stopped.
>>> Let me know if no space on my machine, is the issue or not so that I can
>>> go for better device having more hard-disk memory.
>>>
>> Good idea. Not sure who's going to do it but if you do it, please send me
>> a patch & I'll check it in
>>
>>>
>>> Also May I know that How much memory phrase-table.3.folder has in
>>> general when CreateOnDiskPt command executes completely
>>> as phrase-table.3.gz size is only 23GB.
>>>
>> I'm not too sure. Try it on a disk with 1TB and please report back what
>> you find for future reference
>>
>>>
>>> Thanking You.
>>>
>>>
>>> On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang  wrote:
>>>
 strange, did the CreateOnDiskPt command execute ok, ie. with no error?

 Does this file exist:
/home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat
 If you do
   cat Misc.dat
 what does it say?

 Hieu Hoang
 http://www.hoang.co.uk/hieu

 On 10 December 2016 at 11:30, Shubham Khandelwal 
 wrote:

> Hi,
>
> Thanks Hieu for your reply.
> I have used *CreateOnDiskPt* to binarize the model and stored in
> *phrase-table.3.folder *using following command:
>
>
> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2  phrase-table.3.gz
> phrase-table.3.folder*
>
> Also I have made changes in *moses.ini.3 (*i.e. I have converted
> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*)  *
>
> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
> path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0
> output-factor=0
>
> Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * ,
> it gave following error after *Created input-output object*:
>
> *terminate called after throwing an instance of 'util::Exception'*
> *  what():  OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t
> OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw 
> util::Exception
> because `iter == m_miscInfo.end()'.*
> *Couldn't find value for key NumSourceFactors*
> *Aborted (core dumped)*
>
> Here, I do not know that what key value should I pass and how ? Can
> you please help me in this regard.
>
> Thank you so much for your help.
>
> Regards,
> Shubham
>
> On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang 
> wrote:
>
>> This is a hierarchical model. You must binarize with CreateOnDiskPt
>> for this model
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 9 December 2016 at 08:18, Shubham Khandelwal 
>> wrote:
>>
>>> Hello,
>>>
>>> Thanks. It worked now. I have created compact phrase table.
>>> Now, when I am running it using following command:
>>>
>>> *~/mosesdecoder/bin/moses
>>> -f ~/Translate/models/de-en/model/moses.ini.2 -threads all*
>>>
>>> Then, after creating input-output object, it gave following
>>> segmentation fault   error:
>>>
>>> Created 

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Hieu Hoang]

Hieu Hoang
http://www.hoang.co.uk/hieu

On 10 December 2016 at 14:06, Shubham Khandelwal 
wrote:

> Yes, CreateOnDiskPt command executed without any error.
>
> There are 5 files in this phrase-table.3.folder: Misc.dat , Source.dat,
> TargetColl.dat, TargetInd.dat, Vocab.dat
> *Misc.dat  and Vocab.dat files are empty. *
> I just checked that my hard-disk memory is full as this folder took 165G
> already. So may be, due to this reason those 2 files are empty. But 
> CreateOnDiskPt
> command should throw an error of *No space left on machine *when it
> stopped.
> Let me know if no space on my machine, is the issue or not so that I can
> go for better device having more hard-disk memory.
>
Good idea. Not sure who's going to do it but if you do it, please send me a
patch & I'll check it in

>
> Also May I know that How much memory phrase-table.3.folder has in general
> when CreateOnDiskPt command executes completely as phrase-table.3.gz size
> is only 23GB.
>
I'm not too sure. Try it on a disk with 1TB and please report back what you
find for future reference

>
> Thanking You.
>
>
> On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang  wrote:
>
>> strange, did the CreateOnDiskPt command execute ok, ie. with no error?
>>
>> Does this file exist:
>>/home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat
>> If you do
>>   cat Misc.dat
>> what does it say?
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 10 December 2016 at 11:30, Shubham Khandelwal 
>> wrote:
>>
>>> Hi,
>>>
>>> Thanks Hieu for your reply.
>>> I have used *CreateOnDiskPt* to binarize the model and stored in
>>> *phrase-table.3.folder *using following command:
>>>
>>>
>>> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2  phrase-table.3.gz
>>> phrase-table.3.folder*
>>>
>>> Also I have made changes in *moses.ini.3 (*i.e. I have converted
>>> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*)  *
>>>
>>> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
>>> path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0
>>> output-factor=0
>>>
>>> Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * ,
>>> it gave following error after *Created input-output object*:
>>>
>>> *terminate called after throwing an instance of 'util::Exception'*
>>> *  what():  OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t
>>> OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw util::Exception
>>> because `iter == m_miscInfo.end()'.*
>>> *Couldn't find value for key NumSourceFactors*
>>> *Aborted (core dumped)*
>>>
>>> Here, I do not know that what key value should I pass and how ? Can you
>>> please help me in this regard.
>>>
>>> Thank you so much for your help.
>>>
>>> Regards,
>>> Shubham
>>>
>>> On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang  wrote:
>>>
 This is a hierarchical model. You must binarize with CreateOnDiskPt for
 this model

 Hieu Hoang
 http://www.hoang.co.uk/hieu

 On 9 December 2016 at 08:18, Shubham Khandelwal 
 wrote:

> Hello,
>
> Thanks. It worked now. I have created compact phrase table.
> Now, when I am running it using following command:
>
> *~/mosesdecoder/bin/moses
> -f ~/Translate/models/de-en/model/moses.ini.2 -threads all*
>
> Then, after creating input-output object, it gave following
> segmentation fault   error:
>
> Created input-output object : [14.796] seconds
> Ich bin ein Student
> Line 0: Initialize search took 0.000 seconds total
> Translating:  Ich bin ein Student   ||| [0,0]=X (1) [0,1]=X (1)
> [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X (1)
> [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1)
> [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1)
> [5,5]=X (1)
>
> Segmentation fault (core dumped)
>
> In my machine, I have 40GB RAM but still I am confused why it gave
> this error.
> Can you please help me in this regard. I have attached moses.ini.2 for
> your reference.
>
> Thanks.
>
> Regards,
> Shubham
>
> On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang 
> wrote:
>
>> maybe try
>>
>>   -encoding None
>>
>> On 08/12/2016 19:44, Shubham Khandelwal wrote:
>>
>> Hi Hieu,
>>
>> Thanks for your reply.
>> Yes, I have used the absolute path and also I tried with -T but it
>> did not work.
>> Is there any other solution to this problem.
>>
>> Btw, Can anybody please upload the compact model of all pre-made
>> models as this will take less space and also it will be very fast during
>> decoding.
>>
>> Thanks.
>>
>> On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang 
>> wrote:
>>
>>> the previous email you referred to says that 

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Shubham Khandelwal]

Yes, CreateOnDiskPt command executed without any error.

There are 5 files in this phrase-table.3.folder: Misc.dat , Source.dat,
TargetColl.dat, TargetInd.dat, Vocab.dat
*Misc.dat  and Vocab.dat files are empty. *
I just checked that my hard-disk memory is full as this folder took 165G
already. So may be, due to this reason those 2 files are empty. But
CreateOnDiskPt
command should throw an error of *No space left on machine *when it stopped.

Let me know if no space on my machine, is the issue or not so that I can go
for better device having more hard-disk memory.

Also May I know that How much memory phrase-table.3.folder has in general
when CreateOnDiskPt command executes completely as phrase-table.3.gz size
is only 23GB.

Thanking You.


On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang  wrote:

> strange, did the CreateOnDiskPt command execute ok, ie. with no error?
>
> Does this file exist:
>/home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat
> If you do
>   cat Misc.dat
> what does it say?
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 10 December 2016 at 11:30, Shubham Khandelwal 
> wrote:
>
>> Hi,
>>
>> Thanks Hieu for your reply.
>> I have used *CreateOnDiskPt* to binarize the model and stored in
>> *phrase-table.3.folder *using following command:
>>
>>
>> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2  phrase-table.3.gz
>> phrase-table.3.folder*
>>
>> Also I have made changes in *moses.ini.3 (*i.e. I have converted
>> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*)  *
>>
>> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
>> path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0
>> output-factor=0
>>
>> Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * ,
>> it gave following error after *Created input-output object*:
>>
>> *terminate called after throwing an instance of 'util::Exception'*
>> *  what():  OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t
>> OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw util::Exception
>> because `iter == m_miscInfo.end()'.*
>> *Couldn't find value for key NumSourceFactors*
>> *Aborted (core dumped)*
>>
>> Here, I do not know that what key value should I pass and how ? Can you
>> please help me in this regard.
>>
>> Thank you so much for your help.
>>
>> Regards,
>> Shubham
>>
>> On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang  wrote:
>>
>>> This is a hierarchical model. You must binarize with CreateOnDiskPt for
>>> this model
>>>
>>> Hieu Hoang
>>> http://www.hoang.co.uk/hieu
>>>
>>> On 9 December 2016 at 08:18, Shubham Khandelwal 
>>> wrote:
>>>
 Hello,

 Thanks. It worked now. I have created compact phrase table.
 Now, when I am running it using following command:

 *~/mosesdecoder/bin/moses
 -f ~/Translate/models/de-en/model/moses.ini.2 -threads all*

 Then, after creating input-output object, it gave following
 segmentation fault   error:

 Created input-output object : [14.796] seconds
 Ich bin ein Student
 Line 0: Initialize search took 0.000 seconds total
 Translating:  Ich bin ein Student   ||| [0,0]=X (1) [0,1]=X (1)
 [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X (1)
 [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1)
 [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1)
 [5,5]=X (1)

 Segmentation fault (core dumped)

 In my machine, I have 40GB RAM but still I am confused why it gave this
 error.
 Can you please help me in this regard. I have attached moses.ini.2 for
 your reference.

 Thanks.

 Regards,
 Shubham

 On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang  wrote:

> maybe try
>
>   -encoding None
>
> On 08/12/2016 19:44, Shubham Khandelwal wrote:
>
> Hi Hieu,
>
> Thanks for your reply.
> Yes, I have used the absolute path and also I tried with -T but it did
> not work.
> Is there any other solution to this problem.
>
> Btw, Can anybody please upload the compact model of all pre-made
> models as this will take less space and also it will be very fast during
> decoding.
>
> Thanks.
>
> On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang 
> wrote:
>
>> the previous email you referred to says that the directory
>>
>> * binarised-model/ *
>>
>> *must exist before you run it, otherwise it will segfault. I would
>> also use absolute path to make sure, ie. not*
>> *binarised-model/phrase-table *
>>
>> *but *
>>
>>
>> *  /home/shubham/moses/binarised-model/phrase-table *
>>
>> *The previous email exchange also says you should try to add the
>> argument *
>>
>>
>> *   -T . *
>>
>> Hieu Hoang
>> 

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Hieu Hoang]

strange, did the CreateOnDiskPt command execute ok, ie. with no error?

Does this file exist:
   /home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat
If you do
  cat Misc.dat
what does it say?

Hieu Hoang
http://www.hoang.co.uk/hieu

On 10 December 2016 at 11:30, Shubham Khandelwal 
wrote:

> Hi,
>
> Thanks Hieu for your reply.
> I have used *CreateOnDiskPt* to binarize the model and stored in
> *phrase-table.3.folder *using following command:
>
>
> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2  phrase-table.3.gz
> phrase-table.3.folder*
>
> Also I have made changes in *moses.ini.3 (*i.e. I have converted
> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*)  *
>
> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
> path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0
> output-factor=0
>
> Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * ,
> it gave following error after *Created input-output object*:
>
> *terminate called after throwing an instance of 'util::Exception'*
> *  what():  OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t
> OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw util::Exception
> because `iter == m_miscInfo.end()'.*
> *Couldn't find value for key NumSourceFactors*
> *Aborted (core dumped)*
>
> Here, I do not know that what key value should I pass and how ? Can you
> please help me in this regard.
>
> Thank you so much for your help.
>
> Regards,
> Shubham
>
> On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang  wrote:
>
>> This is a hierarchical model. You must binarize with CreateOnDiskPt for
>> this model
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 9 December 2016 at 08:18, Shubham Khandelwal 
>> wrote:
>>
>>> Hello,
>>>
>>> Thanks. It worked now. I have created compact phrase table.
>>> Now, when I am running it using following command:
>>>
>>> *~/mosesdecoder/bin/moses
>>> -f ~/Translate/models/de-en/model/moses.ini.2 -threads all*
>>>
>>> Then, after creating input-output object, it gave following segmentation
>>> fault   error:
>>>
>>> Created input-output object : [14.796] seconds
>>> Ich bin ein Student
>>> Line 0: Initialize search took 0.000 seconds total
>>> Translating:  Ich bin ein Student   ||| [0,0]=X (1) [0,1]=X (1)
>>> [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X (1)
>>> [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1)
>>> [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1)
>>> [5,5]=X (1)
>>>
>>> Segmentation fault (core dumped)
>>>
>>> In my machine, I have 40GB RAM but still I am confused why it gave this
>>> error.
>>> Can you please help me in this regard. I have attached moses.ini.2 for
>>> your reference.
>>>
>>> Thanks.
>>>
>>> Regards,
>>> Shubham
>>>
>>> On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang  wrote:
>>>
 maybe try

   -encoding None

 On 08/12/2016 19:44, Shubham Khandelwal wrote:

 Hi Hieu,

 Thanks for your reply.
 Yes, I have used the absolute path and also I tried with -T but it did
 not work.
 Is there any other solution to this problem.

 Btw, Can anybody please upload the compact model of all pre-made models
 as this will take less space and also it will be very fast during decoding.

 Thanks.

 On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang 
 wrote:

> the previous email you referred to says that the directory
>
> * binarised-model/ *
>
> *must exist before you run it, otherwise it will segfault. I would
> also use absolute path to make sure, ie. not*
> *binarised-model/phrase-table *
>
> *but *
>
>
> *  /home/shubham/moses/binarised-model/phrase-table *
>
> *The previous email exchange also says you should try to add the
> argument *
>
>
> *   -T . *
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 8 December 2016 at 15:52, Shubham Khandelwal 
> wrote:
>
>> Hello,
>>
>> This is just the reminder of my previous email.
>>
>> Thanking You.
>>
>> Regards,
>> Shubham
>>
>> On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal <
>> skhlnm...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I have just downloaded phrase-table.2.gz (18GB) de-en model
>>> and phrase-table.3.gz (22GB) fr-en model from the available pre-made
>>> models.
>>> Now, I am converting them to PhraseDictionaryCompact using following
>>> command (for exmaple):
>>>
>>>
>>> *~/mosesdecoder/bin/processPhraseTableMin -threads all -in
>>> ~/model/phrase-table.3.gz -nscores 4  -out binarised-model/phrase-table 
>>>  *
>>>
>>> But after passing 1/3, it gave following segementation fault error:
>>>
>>> *Pass 1/3: Creating hash function for 

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Shubham Khandelwal]

Hi,

Thanks Hieu for your reply.
I have used *CreateOnDiskPt* to binarize the model and stored in
*phrase-table.3.folder *using following command:


*~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2  phrase-table.3.gz
phrase-table.3.folder*

Also I have made changes in *moses.ini.3 (*i.e. I have converted
*PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*)  *

PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0
output-factor=0

Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * , it
gave following error after *Created input-output object*:

*terminate called after throwing an instance of 'util::Exception'*
*  what():  OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t
OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw util::Exception
because `iter == m_miscInfo.end()'.*
*Couldn't find value for key NumSourceFactors*
*Aborted (core dumped)*

Here, I do not know that what key value should I pass and how ? Can you
please help me in this regard.

Thank you so much for your help.

Regards,
Shubham

On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang  wrote:

> This is a hierarchical model. You must binarize with CreateOnDiskPt for
> this model
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 9 December 2016 at 08:18, Shubham Khandelwal 
> wrote:
>
>> Hello,
>>
>> Thanks. It worked now. I have created compact phrase table.
>> Now, when I am running it using following command:
>>
>> *~/mosesdecoder/bin/moses
>> -f ~/Translate/models/de-en/model/moses.ini.2 -threads all*
>>
>> Then, after creating input-output object, it gave following segmentation
>> fault   error:
>>
>> Created input-output object : [14.796] seconds
>> Ich bin ein Student
>> Line 0: Initialize search took 0.000 seconds total
>> Translating:  Ich bin ein Student   ||| [0,0]=X (1) [0,1]=X (1)
>> [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X (1)
>> [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1)
>> [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1)
>> [5,5]=X (1)
>>
>> Segmentation fault (core dumped)
>>
>> In my machine, I have 40GB RAM but still I am confused why it gave this
>> error.
>> Can you please help me in this regard. I have attached moses.ini.2 for
>> your reference.
>>
>> Thanks.
>>
>> Regards,
>> Shubham
>>
>> On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang  wrote:
>>
>>> maybe try
>>>
>>>   -encoding None
>>>
>>> On 08/12/2016 19:44, Shubham Khandelwal wrote:
>>>
>>> Hi Hieu,
>>>
>>> Thanks for your reply.
>>> Yes, I have used the absolute path and also I tried with -T but it did
>>> not work.
>>> Is there any other solution to this problem.
>>>
>>> Btw, Can anybody please upload the compact model of all pre-made models
>>> as this will take less space and also it will be very fast during decoding.
>>>
>>> Thanks.
>>>
>>> On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang  wrote:
>>>
 the previous email you referred to says that the directory

 * binarised-model/ *

 *must exist before you run it, otherwise it will segfault. I would also
 use absolute path to make sure, ie. not*
 *binarised-model/phrase-table *

 *but *


 *  /home/shubham/moses/binarised-model/phrase-table *

 *The previous email exchange also says you should try to add the
 argument *


 *   -T . *

 Hieu Hoang
 http://www.hoang.co.uk/hieu

 On 8 December 2016 at 15:52, Shubham Khandelwal 
 wrote:

> Hello,
>
> This is just the reminder of my previous email.
>
> Thanking You.
>
> Regards,
> Shubham
>
> On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal <
> skhlnm...@gmail.com> wrote:
>
>> Hello,
>>
>> I have just downloaded phrase-table.2.gz (18GB) de-en model
>> and phrase-table.3.gz (22GB) fr-en model from the available pre-made
>> models.
>> Now, I am converting them to PhraseDictionaryCompact using following
>> command (for exmaple):
>>
>>
>> *~/mosesdecoder/bin/processPhraseTableMin -threads all -in
>> ~/model/phrase-table.3.gz -nscores 4  -out binarised-model/phrase-table  
>> *
>>
>> But after passing 1/3, it gave following segementation fault error:
>>
>> *Pass 1/3: Creating hash function for rank assignment*
>> *Segmentation fault (core dumped)*
>>
>> I have found almost same issue on this thread:
>> http://comments.gmane.org/gmane.comp.nlp.moses.user/13033
>> However, I have provided the existing *binarised-model *folder in
>> the command. Also, I have the write-access in /tmp but still it gave
>> sementation fault.
>>
>> Can you please tell me what could be wrong here ?
>>
>> Thanking You.
>>
>> Regards,
>> Shubham
>>
>

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Hieu Hoang]

This is a hierarchical model. You must binarize with CreateOnDiskPt for
this model

Hieu Hoang
http://www.hoang.co.uk/hieu

On 9 December 2016 at 08:18, Shubham Khandelwal  wrote:

> Hello,
>
> Thanks. It worked now. I have created compact phrase table.
> Now, when I am running it using following command:
>
> *~/mosesdecoder/bin/moses
> -f ~/Translate/models/de-en/model/moses.ini.2 -threads all*
>
> Then, after creating input-output object, it gave following segmentation
> fault   error:
>
> Created input-output object : [14.796] seconds
> Ich bin ein Student
> Line 0: Initialize search took 0.000 seconds total
> Translating:  Ich bin ein Student   ||| [0,0]=X (1) [0,1]=X (1)
> [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X (1)
> [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1)
> [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1)
> [5,5]=X (1)
>
> Segmentation fault (core dumped)
>
> In my machine, I have 40GB RAM but still I am confused why it gave this
> error.
> Can you please help me in this regard. I have attached moses.ini.2 for
> your reference.
>
> Thanks.
>
> Regards,
> Shubham
>
> On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang  wrote:
>
>> maybe try
>>
>>   -encoding None
>>
>> On 08/12/2016 19:44, Shubham Khandelwal wrote:
>>
>> Hi Hieu,
>>
>> Thanks for your reply.
>> Yes, I have used the absolute path and also I tried with -T but it did
>> not work.
>> Is there any other solution to this problem.
>>
>> Btw, Can anybody please upload the compact model of all pre-made models
>> as this will take less space and also it will be very fast during decoding.
>>
>> Thanks.
>>
>> On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang  wrote:
>>
>>> the previous email you referred to says that the directory
>>>
>>> * binarised-model/ *
>>>
>>> *must exist before you run it, otherwise it will segfault. I would also
>>> use absolute path to make sure, ie. not*
>>> *binarised-model/phrase-table *
>>>
>>> *but *
>>>
>>>
>>> *  /home/shubham/moses/binarised-model/phrase-table *
>>>
>>> *The previous email exchange also says you should try to add the
>>> argument *
>>>
>>>
>>> *   -T . *
>>>
>>> Hieu Hoang
>>> http://www.hoang.co.uk/hieu
>>>
>>> On 8 December 2016 at 15:52, Shubham Khandelwal 
>>> wrote:
>>>
 Hello,

 This is just the reminder of my previous email.

 Thanking You.

 Regards,
 Shubham

 On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal  wrote:

> Hello,
>
> I have just downloaded phrase-table.2.gz (18GB) de-en model
> and phrase-table.3.gz (22GB) fr-en model from the available pre-made
> models.
> Now, I am converting them to PhraseDictionaryCompact using following
> command (for exmaple):
>
>
> *~/mosesdecoder/bin/processPhraseTableMin -threads all -in
> ~/model/phrase-table.3.gz -nscores 4  -out binarised-model/phrase-table  *
>
> But after passing 1/3, it gave following segementation fault error:
>
> *Pass 1/3: Creating hash function for rank assignment*
> *Segmentation fault (core dumped)*
>
> I have found almost same issue on this thread:
> http://comments.gmane.org/gmane.comp.nlp.moses.user/13033
> However, I have provided the existing *binarised-model *folder in the
> command. Also, I have the write-access in /tmp but still it gave
> sementation fault.
>
> Can you please tell me what could be wrong here ?
>
> Thanking You.
>
> Regards,
> Shubham
>





 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support


>>>
>>
>>
>>
>>
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Shubham Khandelwal]

Hello,

Thanks. It worked now. I have created compact phrase table.
Now, when I am running it using following command:

*~/mosesdecoder/bin/moses
-f ~/Translate/models/de-en/model/moses.ini.2 -threads all*

Then, after creating input-output object, it gave following segmentation
fault   error:

Created input-output object : [14.796] seconds
Ich bin ein Student
Line 0: Initialize search took 0.000 seconds total
Translating:  Ich bin ein Student   ||| [0,0]=X (1) [0,1]=X (1)
[0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X (1)
[1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1)
[2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1)
[5,5]=X (1)

Segmentation fault (core dumped)

In my machine, I have 40GB RAM but still I am confused why it gave this
error.
Can you please help me in this regard. I have attached moses.ini.2 for your
reference.

Thanks.

Regards,
Shubham

On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang  wrote:

> maybe try
>
>   -encoding None
>
> On 08/12/2016 19:44, Shubham Khandelwal wrote:
>
> Hi Hieu,
>
> Thanks for your reply.
> Yes, I have used the absolute path and also I tried with -T but it did not
> work.
> Is there any other solution to this problem.
>
> Btw, Can anybody please upload the compact model of all pre-made models as
> this will take less space and also it will be very fast during decoding.
>
> Thanks.
>
> On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang  wrote:
>
>> the previous email you referred to says that the directory
>>
>> * binarised-model/ *
>>
>> *must exist before you run it, otherwise it will segfault. I would also
>> use absolute path to make sure, ie. not*
>> *binarised-model/phrase-table *
>>
>> *but *
>>
>>
>> *  /home/shubham/moses/binarised-model/phrase-table *
>>
>> *The previous email exchange also says you should try to add the argument
>> *
>>
>>
>> *   -T . *
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 8 December 2016 at 15:52, Shubham Khandelwal 
>> wrote:
>>
>>> Hello,
>>>
>>> This is just the reminder of my previous email.
>>>
>>> Thanking You.
>>>
>>> Regards,
>>> Shubham
>>>
>>> On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal 
>>> wrote:
>>>
 Hello,

 I have just downloaded phrase-table.2.gz (18GB) de-en model
 and phrase-table.3.gz (22GB) fr-en model from the available pre-made
 models.
 Now, I am converting them to PhraseDictionaryCompact using following
 command (for exmaple):


 *~/mosesdecoder/bin/processPhraseTableMin -threads all -in
 ~/model/phrase-table.3.gz -nscores 4  -out binarised-model/phrase-table  *

 But after passing 1/3, it gave following segementation fault error:

 *Pass 1/3: Creating hash function for rank assignment*
 *Segmentation fault (core dumped)*

 I have found almost same issue on this thread:
 http://comments.gmane.org/gmane.comp.nlp.moses.user/13033
 However, I have provided the existing *binarised-model *folder in the
 command. Also, I have the write-access in /tmp but still it gave
 sementation fault.

 Can you please tell me what could be wrong here ?

 Thanking You.

 Regards,
 Shubham

>>>
>>>
>>>
>>>
>>>
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>
>
>
>


moses.ini.2
Description: Binary data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Hieu Hoang]

maybe try

  -encoding None


On 08/12/2016 19:44, Shubham Khandelwal wrote:

Hi Hieu,

Thanks for your reply.
Yes, I have used the absolute path and also I tried with -T but it did 
not work.

Is there any other solution to this problem.

Btw, Can anybody please upload the compact model of all pre-made 
models as this will take less space and also it will be very fast 
during decoding.


Thanks.

On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang > wrote:


the previous email you referred to says that the directory
/binarised-model/
/
/must exist before you run it, otherwise it will segfault. I would
also use absolute path to make sure, ie. not
binarised-model/phrase-table
///
/but
/
/  /home/shubham/moses/binarised-model/phrase-table

/
/The previous email exchange also says you should try to add the
argument
/
/   -T .

/

Hieu Hoang
http://www.hoang.co.uk/hieu

On 8 December 2016 at 15:52, Shubham Khandelwal
> wrote:

Hello,

This is just the reminder of my previous email.

Thanking You.

Regards,
Shubham

On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal
> wrote:

Hello,

I have just downloaded phrase-table.2.gz (18GB) de-en
model and phrase-table.3.gz (22GB) fr-en model from the
available pre-made models.
Now, I am converting them to PhraseDictionaryCompact using
following command (for exmaple):
/
/
/~/mosesdecoder/bin/processPhraseTableMin -threads all -in
~/model/phrase-table.3.gz -nscores 4  -out
binarised-model/phrase-table
/

But after passing 1/3, it gave following segementation
fault error:
/
/
/Pass 1/3: Creating hash function for rank assignment/
/Segmentation fault (core dumped)/

I have found almost same issue on this thread:
http://comments.gmane.org/gmane.comp.nlp.moses.user/13033

However, I have provided the existing /binarised-model
/folder in the command. Also, I have the write-access in
/tmp but still it gave sementation fault.

Can you please tell me what could be wrong here ?

Thanking You.

Regards,
Shubham






___
Moses-support mailing list
Moses-support@mit.edu 
http://mailman.mit.edu/mailman/listinfo/moses-support








___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Shubham Khandelwal]

Hi Hieu,

Thanks for your reply.
Yes, I have used the absolute path and also I tried with -T but it did not
work.
Is there any other solution to this problem.

Btw, Can anybody please upload the compact model of all pre-made models as
this will take less space and also it will be very fast during decoding.

Thanks.

On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang  wrote:

> the previous email you referred to says that the directory
>
> * binarised-model/*
>
> *must exist before you run it, otherwise it will segfault. I would also
> use absolute path to make sure, ie. not   *
> *binarised-model/phrase-table *
>
> *but*
>
>
> *  /home/shubham/moses/binarised-model/phrase-table *
>
> *The previous email exchange also says you should try to add the argument*
>
>
> *   -T .*
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 8 December 2016 at 15:52, Shubham Khandelwal 
> wrote:
>
>> Hello,
>>
>> This is just the reminder of my previous email.
>>
>> Thanking You.
>>
>> Regards,
>> Shubham
>>
>> On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal 
>> wrote:
>>
>>> Hello,
>>>
>>> I have just downloaded phrase-table.2.gz (18GB) de-en model
>>> and phrase-table.3.gz (22GB) fr-en model from the available pre-made
>>> models.
>>> Now, I am converting them to PhraseDictionaryCompact using following
>>> command (for exmaple):
>>>
>>>
>>> *~/mosesdecoder/bin/processPhraseTableMin -threads all -in
>>> ~/model/phrase-table.3.gz -nscores 4  -out binarised-model/phrase-table *
>>>
>>> But after passing 1/3, it gave following segementation fault error:
>>>
>>> *Pass 1/3: Creating hash function for rank assignment*
>>> *Segmentation fault (core dumped)*
>>>
>>> I have found almost same issue on this thread:
>>> http://comments.gmane.org/gmane.comp.nlp.moses.user/13033
>>> However, I have provided the existing *binarised-model *folder in the
>>> command. Also, I have the write-access in /tmp but still it gave
>>> sementation fault.
>>>
>>> Can you please tell me what could be wrong here ?
>>>
>>> Thanking You.
>>>
>>> Regards,
>>> Shubham
>>>
>>
>>
>>
>>
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Hieu Hoang]

the previous email you referred to says that the directory

* binarised-model/*

*must exist before you run it, otherwise it will segfault. I would also use
absolute path to make sure, ie. not   *
*binarised-model/phrase-table *

*but*


*  /home/shubham/moses/binarised-model/phrase-table *

*The previous email exchange also says you should try to add the argument*


*   -T .*

Hieu Hoang
http://www.hoang.co.uk/hieu

On 8 December 2016 at 15:52, Shubham Khandelwal  wrote:

> Hello,
>
> This is just the reminder of my previous email.
>
> Thanking You.
>
> Regards,
> Shubham
>
> On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal 
> wrote:
>
>> Hello,
>>
>> I have just downloaded phrase-table.2.gz (18GB) de-en model
>> and phrase-table.3.gz (22GB) fr-en model from the available pre-made
>> models.
>> Now, I am converting them to PhraseDictionaryCompact using following
>> command (for exmaple):
>>
>>
>> *~/mosesdecoder/bin/processPhraseTableMin -threads all -in
>> ~/model/phrase-table.3.gz -nscores 4  -out binarised-model/phrase-table *
>>
>> But after passing 1/3, it gave following segementation fault error:
>>
>> *Pass 1/3: Creating hash function for rank assignment*
>> *Segmentation fault (core dumped)*
>>
>> I have found almost same issue on this thread:
>> http://comments.gmane.org/gmane.comp.nlp.moses.user/13033
>> However, I have provided the existing *binarised-model *folder in the
>> command. Also, I have the write-access in /tmp but still it gave
>> sementation fault.
>>
>> Can you please tell me what could be wrong here ?
>>
>> Thanking You.
>>
>> Regards,
>> Shubham
>>
>
>
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact

[Shubham Khandelwal]

Hello,

This is just the reminder of my previous email.

Thanking You.

Regards,
Shubham

On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal 
wrote:

> Hello,
>
> I have just downloaded phrase-table.2.gz (18GB) de-en model
> and phrase-table.3.gz (22GB) fr-en model from the available pre-made
> models.
> Now, I am converting them to PhraseDictionaryCompact using following
> command (for exmaple):
>
>
> *~/mosesdecoder/bin/processPhraseTableMin -threads all -in
> ~/model/phrase-table.3.gz -nscores 4  -out binarised-model/phrase-table *
>
> But after passing 1/3, it gave following segementation fault error:
>
> *Pass 1/3: Creating hash function for rank assignment*
> *Segmentation fault (core dumped)*
>
> I have found almost same issue on this thread:
> http://comments.gmane.org/gmane.comp.nlp.moses.user/13033
> However, I have provided the existing *binarised-model *folder in the
> command. Also, I have the write-access in /tmp but still it gave
> sementation fault.
>
> Can you please tell me what could be wrong here ?
>
> Thanking You.
>
> Regards,
> Shubham
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support