[Moses-support] Need help for parallelisation in mosesserver
Hello, As mosesserver accepts only one sentence at a time. So I am creating one another component in front of mosesserver to handle tokenisation, casing and splitting taking care of parallelisation. Following is my procedure to do it, let me know whether am I heading correctly or not to do this: *---* *So suppose, if I have 5 different sentences (as a paragraph) to translate at once (fr-en). So I will be creating mosesserver on 5 different ports firstly and pass those 5 different sentences after doing parallely tokenisaton, casing and splitting on those different ports and then concatenate the output after recasing and detokenisation parallely. * *--* Let me know whether this is correct or not ? If no, then please suggest me better solution to do this. Also, I have one more question in this that if a sentence is composed of around 10 words. Then when I pass this sentence to translate as follows: -> ~/mosesdecoder/bin/mosesserver -f moses.ini -threads 16 -b 0.1 then it takes around 10 seconds to translate. To make it fast, I can run this on different ports but that is not a good idea I think, as splitting a single sentence to multiple group of sentence and then translate them on different ports separately, can give different meaning rather than translate the whole single sentence at single port. So basically, my doubt is how to make better splitting in such cases which can take care of parallelisation aswell ? -- Yours Sincerely, Shubham Khandelwal ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Moses-support Digest, Vol 122, Issue 29
Hello, Currently, I have created one fr-en translation model (size of phrase-table.minphr and reordering-table.minlexr are 13 GB and 6.6 GB respectively) by following the tutorial of Moses baseline system on a big dataset. I have also used Cube Pruning method as suggested by Thomas. Now, I use mosesserver and getting response. Now it is taking little bit less time to decode the input sentences. However, the decoding is still *not *in real time. I have attached moses.ini for your reference. To make it fast, I just found an infrastructure: https://github.com/ufal/mtmonkey which makes decoding faster by distributed way. So, before trying this (mtmonkey) out, I would like to know that Is there any other solution or way now by which I can get this decoding in real time using Moses ? Is it possible on GPU ? Looking forward for your response. Thanking You. Regards, Shubham Khandelwal On Fri, Dec 16, 2016 at 4:29 PM, Mathias Müller wrote: > Hi Shubham > > You could start Moses in server mode: > > $ moses -f /path/to/moses.ini --server --server-port 12345 --server-log > /path/to/log > > This will load the models, keep them in memory and the server will wait > for client requests and serve them until you terminate the process. > Translating is a bit different in this case, you have to send an XML-RPC > request to the server. > > But first you'd have to make sure Moses is built with XML-RPC. > > Regards and good luck > Mathias > — > > Mathias Müller > AND-2-20 > Institute of Computational Linguistics > University of Zurich > Switzerland > +41 44 635 75 81 > mathias.muel...@uzh.ch > > On Fri, Dec 16, 2016 at 10:32 AM, Shubham Khandelwal > wrote: > >> Hey Thomas, >> >> Thanks for your reply. >> Using Cube Pruning, the speed is littile bit high, but not that much. I >> will try to play with these parameters. >> >> I have binary moses2 which supports it aswell but it is taking more time >> to than moses. Can you please send/share somewhere your binary moses2 file >> if possible ? >> >> Also, I do not wish to run this command ( ~/mosesdecoder/bin/moses >> -f moses.ini -threads all) every time for every input. Is there any way in >> Moses by which all models will load in memory for forever and I can just >> pass a input and get output in real time without using this command again >> and again. >> >> Looking forward for your response. >> >> Thanks again. >> >> On Fri, Dec 16, 2016 at 1:20 PM, Tomasz Gawryl < >> tomasz.gaw...@skrivanek.pl> wrote: >> >>> Hi, >>> If you want to speed up decoding time maybe you should consider changing >>> searching algorithm. I'm also using compact phrase tables and after some >>> test I realised that cube pruning gives almost exactly the same quality >>> but >>> is much faster. For example you can add something like this to your >>> config >>> file: >>> >>> # Cube Pruning >>> [search-algorithm] >>> 1 >>> [cube-pruning-pop-limit] >>> 1000 >>> [stack] >>> 50 >>> >>> If your model allows you may also try moses2 binary which is faster than >>> original. >>> >>> Regards, >>> Thomas >>> >>> -- >>> >>> Message: 1 >>> Date: Thu, 15 Dec 2016 19:12:01 +0530 >>> From: Shubham Khandelwal >>> Subject: Re: [Moses-support] Regarding Decoding Time >>> To: Hieu Hoang >>> Cc: moses-support >>> Message-ID: >>> >> ail.com> >>> Content-Type: text/plain; charset="utf-8" >>> >>> Hello, >>> >>> Currently, I am using phrase-table.minphr , reordering-table.minlexr and >>> language model (total size of these 3 are 6 GB). Now, I tried to decode >>> on >>> two different machines (8 core-16GB RAM *&* 4 core-40GB RAM) using them. >>> So, During decoding of around 500 words, it took 90 seconds and 100 >>> seconds >>> respectively on those machines. I am already using compact phrase and >>> reordering table representations for faster decoding. Is there any other >>> way >>> to reduce this decoding time. >>> >>> Also, In Moses, Do we have distributed way of decoding on multiple >>> machines >>> ? >>> >>> Looking forward for your response. >>> >>> ___ >>> Moses-support mailing list >>> Moses-support@mit.edu >>> http://
Re: [Moses-support] Moses-support Digest, Vol 122, Issue 29
Hey Thomas, Thanks for your reply. Using Cube Pruning, the speed is littile bit high, but not that much. I will try to play with these parameters. I have binary moses2 which supports it aswell but it is taking more time to than moses. Can you please send/share somewhere your binary moses2 file if possible ? Also, I do not wish to run this command ( ~/mosesdecoder/bin/moses -f moses.ini -threads all) every time for every input. Is there any way in Moses by which all models will load in memory for forever and I can just pass a input and get output in real time without using this command again and again. Looking forward for your response. Thanks again. On Fri, Dec 16, 2016 at 1:20 PM, Tomasz Gawryl wrote: > Hi, > If you want to speed up decoding time maybe you should consider changing > searching algorithm. I'm also using compact phrase tables and after some > test I realised that cube pruning gives almost exactly the same quality but > is much faster. For example you can add something like this to your config > file: > > # Cube Pruning > [search-algorithm] > 1 > [cube-pruning-pop-limit] > 1000 > [stack] > 50 > > If your model allows you may also try moses2 binary which is faster than > original. > > Regards, > Thomas > > ------ > > Message: 1 > Date: Thu, 15 Dec 2016 19:12:01 +0530 > From: Shubham Khandelwal > Subject: Re: [Moses-support] Regarding Decoding Time > To: Hieu Hoang > Cc: moses-support > Message-ID: > gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > Currently, I am using phrase-table.minphr , reordering-table.minlexr and > language model (total size of these 3 are 6 GB). Now, I tried to decode on > two different machines (8 core-16GB RAM *&* 4 core-40GB RAM) using them. > So, During decoding of around 500 words, it took 90 seconds and 100 seconds > respectively on those machines. I am already using compact phrase and > reordering table representations for faster decoding. Is there any other > way > to reduce this decoding time. > > Also, In Moses, Do we have distributed way of decoding on multiple machines > ? > > Looking forward for your response. > > _______ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > -- Yours Sincerely, Shubham Khandelwal Masters in Informatics (M2-MoSIG), University Joseph Fourier-Grenoble INP, Grenoble, France Webpage: https://sites.google.com/site/skhandelwl21/ ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact
Hello, I was using http://www.statmt.org/moses/RELEASE-3.0/models/fr- en/tuning/moses.tuned.ini.5 There I had converted phrase table and reordering table to their compact format and also made some changes in this moses.tuned.ini.5 file (attached in this email) to decode fastly. Now, this time, I am using *moses *binary file which I just directly downloaded from here: http://www.statmt.org/moses/RELEASE-3.0/binaries/linux-64bit/bin/ (just to be sure that I am using correct binary moses file) And just run -> ~/moses -f moses.tuned.ini.5 -threads all Then after creating input-output object, it took 60 seconds to decode just for 3 words. Am I missing or doing something wrong here. Also, I do not wish to run this command ( ~/moses -f moses.tuned.ini.5 -threads all) every time for every input. Is there any way in Moses by which all models will load in memory for forever and I can just pass a input and get output in real time without using this command again and again. Looking forward for your response. Thanking You. On Thu, Dec 15, 2016 at 4:14 PM, Hieu Hoang wrote: > there is no limit to the number of words. > > if you are using the premade models, make sure you are using tuned > moses.ini file in > tuning/moses.tuned.ini.? > > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 13 December 2016 at 09:32, Shubham Khandelwal > wrote: > >> Thanks. Hieu. I understood now. >> Also, Is there any limit of number of words for the translation because >> when I use this command: *~/mosesdecoder/bin/moses -f moses.ini* >> Then, it translates only some words which are in the beginning. I mean, >> it does not consume all words for the translation after Created >> input-output object. >> Is there any way by which I can control or remove this limit. >> >> Thanks. >> >> On Mon, Dec 12, 2016 at 7:17 PM, Hieu Hoang wrote: >> >>> there are actually 7 different configurations. You have to look at the >>> config file in >>>steps/?/config.? >>> For fr-en: >>> 1. phrase-based, truecased >>> 2. phrase-based, lowercased then recased >>> 3. hierarchical model, lowercased then recased >>> 4. phrase-based, lowercased then recased. Using target side word + pos >>> factors >>> 5. Like (2) but using batch-mira to tune >>> 6. Like (2) but using PRO to tune >>> 7. Like (2) but using CreateOnDiskPt to create binary phrase table >>> You can see the BLEU scores in >>> evaluation/report.* >>> >>> Hieu Hoang >>> http://www.hoang.co.uk/hieu >>> >>> On 12 December 2016 at 13:28, Shubham Khandelwal >>> wrote: >>> >>>> Okay Thanks Hieu. I will try it with 1TB HD-memory machine. >>>> Btw I can see there are 4 pre-made models available for fr-en and de-en >>>> (http://www.statmt.org/moses/RELEASE-3.0/models/fr-en/model/ and >>>> http://www.statmt.org/moses/RELEASE-3.0/models/de-en/model/). Can you >>>> please tell me among these 4, which one is better model(in terms of bleu >>>> score) except the huge model which is already there in both, as I can not >>>> understand how analysis is shown in steps folder. >>>> Also, Are all these pre-made models hierarchical model ? >>>> >>>> >>>> On Mon, Dec 12, 2016 at 6:09 PM, Hieu Hoang >>>> wrote: >>>> >>>>> >>>>> >>>>> Hieu Hoang >>>>> http://www.hoang.co.uk/hieu >>>>> >>>>> On 10 December 2016 at 14:06, Shubham Khandelwal >>>>> wrote: >>>>> >>>>>> Yes, CreateOnDiskPt command executed without any error. >>>>>> >>>>>> There are 5 files in this phrase-table.3.folder: Misc.dat >>>>>> , Source.dat, TargetColl.dat, TargetInd.dat, Vocab.dat >>>>>> *Misc.dat and Vocab.dat files are empty. * >>>>>> I just checked that my hard-disk memory is full as this folder >>>>>> took 165G already. So may be, due to this reason those 2 files are empty. >>>>>> But CreateOnDiskPt command should throw an error of *No space left >>>>>> on machine *when it stopped. >>>>>> Let me know if no space on my machine, is the issue or not so that I >>>>>> can go for better device having more hard-disk memory. >>>>>> >>>>> Good idea. Not sure who's going to do it but if you do it, please send >>>>> me a patch & I'll check it in >>>>> >>>>>>
Re: [Moses-support] Regarding Decoding Time
Hello, Currently, I am using phrase-table.minphr , reordering-table.minlexr and language model (total size of these 3 are 6 GB). Now, I tried to decode on two different machines (8 core-16GB RAM *&* 4 core-40GB RAM) using them. So, During decoding of around 500 words, it took 90 seconds and 100 seconds respectively on those machines. I am already using compact phrase and reordering table representations for faster decoding. Is there any other way to reduce this decoding time. Also, In Moses, Do we have distributed way of decoding on multiple machines ? Looking forward for your response. On Tue, Dec 6, 2016 at 4:25 PM, Hieu Hoang wrote: > have a look at this page >http://www.statmt.org/moses/?n=Moses.Optimize > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 5 December 2016 at 14:15, Shubham Khandelwal > wrote: > >> Hello, >> >> I have created one translation model (German to English) and converted it >> to Compact format. >> Now when I run the command: >> ~/mosesdecoder/bin/moses2 -f moses.ini >> Then, It loads the model very quickly but when I pass the German Text >> (around 4-5 lines) for the translation then it translates the text in 10 >> minutes. >> Can you please tell me that how can I reduce this time for the >> translation ? >> >> Thanking You. >> >> Regards, >> Shubham >> >> ___ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> > -- Yours Sincerely, Shubham Khandelwal Masters in Informatics (M2-MoSIG), University Joseph Fourier-Grenoble INP, Grenoble, France Webpage: https://sites.google.com/site/skhandelwl21/ ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact
Hi, This is just the reminder of my previous email. Thanks. Regards, Shubham On Dec 13, 2016 3:02 PM, "Shubham Khandelwal" wrote: > Thanks. Hieu. I understood now. > Also, Is there any limit of number of words for the translation because > when I use this command: *~/mosesdecoder/bin/moses -f moses.ini* > Then, it translates only some words which are in the beginning. I mean, it > does not consume all words for the translation after Created input-output > object. > Is there any way by which I can control or remove this limit. > > Thanks. > > On Mon, Dec 12, 2016 at 7:17 PM, Hieu Hoang wrote: > >> there are actually 7 different configurations. You have to look at the >> config file in >>steps/?/config.? >> For fr-en: >> 1. phrase-based, truecased >> 2. phrase-based, lowercased then recased >> 3. hierarchical model, lowercased then recased >> 4. phrase-based, lowercased then recased. Using target side word + pos >> factors >> 5. Like (2) but using batch-mira to tune >> 6. Like (2) but using PRO to tune >> 7. Like (2) but using CreateOnDiskPt to create binary phrase table >> You can see the BLEU scores in >> evaluation/report.* >> >> Hieu Hoang >> http://www.hoang.co.uk/hieu >> >> On 12 December 2016 at 13:28, Shubham Khandelwal >> wrote: >> >>> Okay Thanks Hieu. I will try it with 1TB HD-memory machine. >>> Btw I can see there are 4 pre-made models available for fr-en and de-en ( >>> http://www.statmt.org/moses/RELEASE-3.0/models/fr-en/model/ and >>> http://www.statmt.org/moses/RELEASE-3.0/models/de-en/model/). Can you >>> please tell me among these 4, which one is better model(in terms of bleu >>> score) except the huge model which is already there in both, as I can not >>> understand how analysis is shown in steps folder. >>> Also, Are all these pre-made models hierarchical model ? >>> >>> >>> On Mon, Dec 12, 2016 at 6:09 PM, Hieu Hoang wrote: >>> >>>> >>>> >>>> Hieu Hoang >>>> http://www.hoang.co.uk/hieu >>>> >>>> On 10 December 2016 at 14:06, Shubham Khandelwal >>>> wrote: >>>> >>>>> Yes, CreateOnDiskPt command executed without any error. >>>>> >>>>> There are 5 files in this phrase-table.3.folder: Misc.dat >>>>> , Source.dat, TargetColl.dat, TargetInd.dat, Vocab.dat >>>>> *Misc.dat and Vocab.dat files are empty. * >>>>> I just checked that my hard-disk memory is full as this folder >>>>> took 165G already. So may be, due to this reason those 2 files are empty. >>>>> But CreateOnDiskPt command should throw an error of *No space left on >>>>> machine *when it stopped. >>>>> Let me know if no space on my machine, is the issue or not so that I >>>>> can go for better device having more hard-disk memory. >>>>> >>>> Good idea. Not sure who's going to do it but if you do it, please send >>>> me a patch & I'll check it in >>>> >>>>> >>>>> Also May I know that How much memory phrase-table.3.folder has in >>>>> general when CreateOnDiskPt command executes completely >>>>> as phrase-table.3.gz size is only 23GB. >>>>> >>>> I'm not too sure. Try it on a disk with 1TB and please report back what >>>> you find for future reference >>>> >>>>> >>>>> Thanking You. >>>>> >>>>> >>>>> On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang >>>>> wrote: >>>>> >>>>>> strange, did the CreateOnDiskPt command execute ok, ie. with no error? >>>>>> >>>>>> Does this file exist: >>>>>>/home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat >>>>>> If you do >>>>>> cat Misc.dat >>>>>> what does it say? >>>>>> >>>>>> Hieu Hoang >>>>>> http://www.hoang.co.uk/hieu >>>>>> >>>>>> On 10 December 2016 at 11:30, Shubham Khandelwal >>>>> > wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Thanks Hieu for your reply. >>>>>>> I have used *CreateOnDiskPt* to binarize the model and stored in >>>>>>> *phrase-table.3.folder *using following c
Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact
Thanks. Hieu. I understood now. Also, Is there any limit of number of words for the translation because when I use this command: *~/mosesdecoder/bin/moses -f moses.ini* Then, it translates only some words which are in the beginning. I mean, it does not consume all words for the translation after Created input-output object. Is there any way by which I can control or remove this limit. Thanks. On Mon, Dec 12, 2016 at 7:17 PM, Hieu Hoang wrote: > there are actually 7 different configurations. You have to look at the > config file in >steps/?/config.? > For fr-en: > 1. phrase-based, truecased > 2. phrase-based, lowercased then recased > 3. hierarchical model, lowercased then recased > 4. phrase-based, lowercased then recased. Using target side word + pos > factors > 5. Like (2) but using batch-mira to tune > 6. Like (2) but using PRO to tune > 7. Like (2) but using CreateOnDiskPt to create binary phrase table > You can see the BLEU scores in > evaluation/report.* > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 12 December 2016 at 13:28, Shubham Khandelwal > wrote: > >> Okay Thanks Hieu. I will try it with 1TB HD-memory machine. >> Btw I can see there are 4 pre-made models available for fr-en and de-en ( >> http://www.statmt.org/moses/RELEASE-3.0/models/fr-en/model/ and >> http://www.statmt.org/moses/RELEASE-3.0/models/de-en/model/). Can you >> please tell me among these 4, which one is better model(in terms of bleu >> score) except the huge model which is already there in both, as I can not >> understand how analysis is shown in steps folder. >> Also, Are all these pre-made models hierarchical model ? >> >> >> On Mon, Dec 12, 2016 at 6:09 PM, Hieu Hoang wrote: >> >>> >>> >>> Hieu Hoang >>> http://www.hoang.co.uk/hieu >>> >>> On 10 December 2016 at 14:06, Shubham Khandelwal >>> wrote: >>> >>>> Yes, CreateOnDiskPt command executed without any error. >>>> >>>> There are 5 files in this phrase-table.3.folder: Misc.dat >>>> , Source.dat, TargetColl.dat, TargetInd.dat, Vocab.dat >>>> *Misc.dat and Vocab.dat files are empty. * >>>> I just checked that my hard-disk memory is full as this folder >>>> took 165G already. So may be, due to this reason those 2 files are empty. >>>> But CreateOnDiskPt command should throw an error of *No space left on >>>> machine *when it stopped. >>>> Let me know if no space on my machine, is the issue or not so that I >>>> can go for better device having more hard-disk memory. >>>> >>> Good idea. Not sure who's going to do it but if you do it, please send >>> me a patch & I'll check it in >>> >>>> >>>> Also May I know that How much memory phrase-table.3.folder has in >>>> general when CreateOnDiskPt command executes completely >>>> as phrase-table.3.gz size is only 23GB. >>>> >>> I'm not too sure. Try it on a disk with 1TB and please report back what >>> you find for future reference >>> >>>> >>>> Thanking You. >>>> >>>> >>>> On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang >>>> wrote: >>>> >>>>> strange, did the CreateOnDiskPt command execute ok, ie. with no error? >>>>> >>>>> Does this file exist: >>>>>/home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat >>>>> If you do >>>>> cat Misc.dat >>>>> what does it say? >>>>> >>>>> Hieu Hoang >>>>> http://www.hoang.co.uk/hieu >>>>> >>>>> On 10 December 2016 at 11:30, Shubham Khandelwal >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Thanks Hieu for your reply. >>>>>> I have used *CreateOnDiskPt* to binarize the model and stored in >>>>>> *phrase-table.3.folder *using following command: >>>>>> >>>>>> >>>>>> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2 phrase-table.3.gz >>>>>> phrase-table.3.folder* >>>>>> >>>>>> Also I have made changes in *moses.ini.3 (*i.e. I have converted >>>>>> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*) * >>>>>> >>>>>> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4 >>>>>> path=/home/shubham/models/fr-en/phrase-t
Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact
Okay Thanks Hieu. I will try it with 1TB HD-memory machine. Btw I can see there are 4 pre-made models available for fr-en and de-en ( http://www.statmt.org/moses/RELEASE-3.0/models/fr-en/model/ and http://www.statmt.org/moses/RELEASE-3.0/models/de-en/model/). Can you please tell me among these 4, which one is better model(in terms of bleu score) except the huge model which is already there in both, as I can not understand how analysis is shown in steps folder. Also, Are all these pre-made models hierarchical model ? On Mon, Dec 12, 2016 at 6:09 PM, Hieu Hoang wrote: > > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 10 December 2016 at 14:06, Shubham Khandelwal > wrote: > >> Yes, CreateOnDiskPt command executed without any error. >> >> There are 5 files in this phrase-table.3.folder: Misc.dat , Source.dat, >> TargetColl.dat, TargetInd.dat, Vocab.dat >> *Misc.dat and Vocab.dat files are empty. * >> I just checked that my hard-disk memory is full as this folder took 165G >> already. So may be, due to this reason those 2 files are empty. But >> CreateOnDiskPt >> command should throw an error of *No space left on machine *when it >> stopped. >> Let me know if no space on my machine, is the issue or not so that I can >> go for better device having more hard-disk memory. >> > Good idea. Not sure who's going to do it but if you do it, please send me > a patch & I'll check it in > >> >> Also May I know that How much memory phrase-table.3.folder has in >> general when CreateOnDiskPt command executes completely >> as phrase-table.3.gz size is only 23GB. >> > I'm not too sure. Try it on a disk with 1TB and please report back what > you find for future reference > >> >> Thanking You. >> >> >> On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang wrote: >> >>> strange, did the CreateOnDiskPt command execute ok, ie. with no error? >>> >>> Does this file exist: >>>/home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat >>> If you do >>> cat Misc.dat >>> what does it say? >>> >>> Hieu Hoang >>> http://www.hoang.co.uk/hieu >>> >>> On 10 December 2016 at 11:30, Shubham Khandelwal >>> wrote: >>> >>>> Hi, >>>> >>>> Thanks Hieu for your reply. >>>> I have used *CreateOnDiskPt* to binarize the model and stored in >>>> *phrase-table.3.folder *using following command: >>>> >>>> >>>> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2 phrase-table.3.gz >>>> phrase-table.3.folder* >>>> >>>> Also I have made changes in *moses.ini.3 (*i.e. I have converted >>>> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*) * >>>> >>>> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4 >>>> path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0 >>>> output-factor=0 >>>> >>>> Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * , >>>> it gave following error after *Created input-output object*: >>>> >>>> *terminate called after throwing an instance of 'util::Exception'* >>>> * what(): OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t >>>> OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw util::Exception >>>> because `iter == m_miscInfo.end()'.* >>>> *Couldn't find value for key NumSourceFactors* >>>> *Aborted (core dumped)* >>>> >>>> Here, I do not know that what key value should I pass and how ? Can you >>>> please help me in this regard. >>>> >>>> Thank you so much for your help. >>>> >>>> Regards, >>>> Shubham >>>> >>>> On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang wrote: >>>> >>>>> This is a hierarchical model. You must binarize with CreateOnDiskPt >>>>> for this model >>>>> >>>>> Hieu Hoang >>>>> http://www.hoang.co.uk/hieu >>>>> >>>>> On 9 December 2016 at 08:18, Shubham Khandelwal >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> Thanks. It worked now. I have created compact phrase table. >>>>>> Now, when I am running it using following command: >>>>>> >>>>>> *~/mosesdecoder/bin/moses >>>>>> -f ~/Translate/models/de-en
Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact
Yes, CreateOnDiskPt command executed without any error. There are 5 files in this phrase-table.3.folder: Misc.dat , Source.dat, TargetColl.dat, TargetInd.dat, Vocab.dat *Misc.dat and Vocab.dat files are empty. * I just checked that my hard-disk memory is full as this folder took 165G already. So may be, due to this reason those 2 files are empty. But CreateOnDiskPt command should throw an error of *No space left on machine *when it stopped. Let me know if no space on my machine, is the issue or not so that I can go for better device having more hard-disk memory. Also May I know that How much memory phrase-table.3.folder has in general when CreateOnDiskPt command executes completely as phrase-table.3.gz size is only 23GB. Thanking You. On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang wrote: > strange, did the CreateOnDiskPt command execute ok, ie. with no error? > > Does this file exist: >/home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat > If you do > cat Misc.dat > what does it say? > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 10 December 2016 at 11:30, Shubham Khandelwal > wrote: > >> Hi, >> >> Thanks Hieu for your reply. >> I have used *CreateOnDiskPt* to binarize the model and stored in >> *phrase-table.3.folder *using following command: >> >> >> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2 phrase-table.3.gz >> phrase-table.3.folder* >> >> Also I have made changes in *moses.ini.3 (*i.e. I have converted >> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*) * >> >> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4 >> path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0 >> output-factor=0 >> >> Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * , >> it gave following error after *Created input-output object*: >> >> *terminate called after throwing an instance of 'util::Exception'* >> * what(): OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t >> OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw util::Exception >> because `iter == m_miscInfo.end()'.* >> *Couldn't find value for key NumSourceFactors* >> *Aborted (core dumped)* >> >> Here, I do not know that what key value should I pass and how ? Can you >> please help me in this regard. >> >> Thank you so much for your help. >> >> Regards, >> Shubham >> >> On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang wrote: >> >>> This is a hierarchical model. You must binarize with CreateOnDiskPt for >>> this model >>> >>> Hieu Hoang >>> http://www.hoang.co.uk/hieu >>> >>> On 9 December 2016 at 08:18, Shubham Khandelwal >>> wrote: >>> >>>> Hello, >>>> >>>> Thanks. It worked now. I have created compact phrase table. >>>> Now, when I am running it using following command: >>>> >>>> *~/mosesdecoder/bin/moses >>>> -f ~/Translate/models/de-en/model/moses.ini.2 -threads all* >>>> >>>> Then, after creating input-output object, it gave following >>>> segmentation fault error: >>>> >>>> Created input-output object : [14.796] seconds >>>> Ich bin ein Student >>>> Line 0: Initialize search took 0.000 seconds total >>>> Translating: Ich bin ein Student ||| [0,0]=X (1) [0,1]=X (1) >>>> [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X (1) >>>> [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1) >>>> [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1) >>>> [5,5]=X (1) >>>> >>>> Segmentation fault (core dumped) >>>> >>>> In my machine, I have 40GB RAM but still I am confused why it gave this >>>> error. >>>> Can you please help me in this regard. I have attached moses.ini.2 for >>>> your reference. >>>> >>>> Thanks. >>>> >>>> Regards, >>>> Shubham >>>> >>>> On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang wrote: >>>> >>>>> maybe try >>>>> >>>>> -encoding None >>>>> >>>>> On 08/12/2016 19:44, Shubham Khandelwal wrote: >>>>> >>>>> Hi Hieu, >>>>> >>>>> Thanks for your reply. >>>>> Yes, I have used the absolute path and also I tried with -T but it did >>>>> not work. >>>>> Is there any oth
Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact
Hi, Thanks Hieu for your reply. I have used *CreateOnDiskPt* to binarize the model and stored in *phrase-table.3.folder *using following command: *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2 phrase-table.3.gz phrase-table.3.folder* Also I have made changes in *moses.ini.3 (*i.e. I have converted *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*) * PhraseDictionaryOnDisk name=TranslationModel0 num-features=4 path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0 output-factor=0 Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * , it gave following error after *Created input-output object*: *terminate called after throwing an instance of 'util::Exception'* * what(): OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw util::Exception because `iter == m_miscInfo.end()'.* *Couldn't find value for key NumSourceFactors* *Aborted (core dumped)* Here, I do not know that what key value should I pass and how ? Can you please help me in this regard. Thank you so much for your help. Regards, Shubham On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang wrote: > This is a hierarchical model. You must binarize with CreateOnDiskPt for > this model > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 9 December 2016 at 08:18, Shubham Khandelwal > wrote: > >> Hello, >> >> Thanks. It worked now. I have created compact phrase table. >> Now, when I am running it using following command: >> >> *~/mosesdecoder/bin/moses >> -f ~/Translate/models/de-en/model/moses.ini.2 -threads all* >> >> Then, after creating input-output object, it gave following segmentation >> fault error: >> >> Created input-output object : [14.796] seconds >> Ich bin ein Student >> Line 0: Initialize search took 0.000 seconds total >> Translating: Ich bin ein Student ||| [0,0]=X (1) [0,1]=X (1) >> [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X (1) >> [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1) >> [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1) >> [5,5]=X (1) >> >> Segmentation fault (core dumped) >> >> In my machine, I have 40GB RAM but still I am confused why it gave this >> error. >> Can you please help me in this regard. I have attached moses.ini.2 for >> your reference. >> >> Thanks. >> >> Regards, >> Shubham >> >> On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang wrote: >> >>> maybe try >>> >>> -encoding None >>> >>> On 08/12/2016 19:44, Shubham Khandelwal wrote: >>> >>> Hi Hieu, >>> >>> Thanks for your reply. >>> Yes, I have used the absolute path and also I tried with -T but it did >>> not work. >>> Is there any other solution to this problem. >>> >>> Btw, Can anybody please upload the compact model of all pre-made models >>> as this will take less space and also it will be very fast during decoding. >>> >>> Thanks. >>> >>> On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang wrote: >>> >>>> the previous email you referred to says that the directory >>>> >>>> * binarised-model/ * >>>> >>>> *must exist before you run it, otherwise it will segfault. I would also >>>> use absolute path to make sure, ie. not* >>>> *binarised-model/phrase-table * >>>> >>>> *but * >>>> >>>> >>>> * /home/shubham/moses/binarised-model/phrase-table * >>>> >>>> *The previous email exchange also says you should try to add the >>>> argument * >>>> >>>> >>>> * -T . * >>>> >>>> Hieu Hoang >>>> http://www.hoang.co.uk/hieu >>>> >>>> On 8 December 2016 at 15:52, Shubham Khandelwal >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> This is just the reminder of my previous email. >>>>> >>>>> Thanking You. >>>>> >>>>> Regards, >>>>> Shubham >>>>> >>>>> On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal < >>>>> skhlnm...@gmail.com> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I have just downloaded phrase-table.2.gz (18GB) de-en model >>>>>> and phrase-table.3.gz (22GB) fr-en model from the available pre-made >>>>>> models.
Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact
Hello, Thanks. It worked now. I have created compact phrase table. Now, when I am running it using following command: *~/mosesdecoder/bin/moses -f ~/Translate/models/de-en/model/moses.ini.2 -threads all* Then, after creating input-output object, it gave following segmentation fault error: Created input-output object : [14.796] seconds Ich bin ein Student Line 0: Initialize search took 0.000 seconds total Translating: Ich bin ein Student ||| [0,0]=X (1) [0,1]=X (1) [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X (1) [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1) [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1) [5,5]=X (1) Segmentation fault (core dumped) In my machine, I have 40GB RAM but still I am confused why it gave this error. Can you please help me in this regard. I have attached moses.ini.2 for your reference. Thanks. Regards, Shubham On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang wrote: > maybe try > > -encoding None > > On 08/12/2016 19:44, Shubham Khandelwal wrote: > > Hi Hieu, > > Thanks for your reply. > Yes, I have used the absolute path and also I tried with -T but it did not > work. > Is there any other solution to this problem. > > Btw, Can anybody please upload the compact model of all pre-made models as > this will take less space and also it will be very fast during decoding. > > Thanks. > > On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang wrote: > >> the previous email you referred to says that the directory >> >> * binarised-model/ * >> >> *must exist before you run it, otherwise it will segfault. I would also >> use absolute path to make sure, ie. not* >> *binarised-model/phrase-table * >> >> *but * >> >> >> * /home/shubham/moses/binarised-model/phrase-table * >> >> *The previous email exchange also says you should try to add the argument >> * >> >> >> * -T . * >> >> Hieu Hoang >> http://www.hoang.co.uk/hieu >> >> On 8 December 2016 at 15:52, Shubham Khandelwal >> wrote: >> >>> Hello, >>> >>> This is just the reminder of my previous email. >>> >>> Thanking You. >>> >>> Regards, >>> Shubham >>> >>> On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal >>> wrote: >>> >>>> Hello, >>>> >>>> I have just downloaded phrase-table.2.gz (18GB) de-en model >>>> and phrase-table.3.gz (22GB) fr-en model from the available pre-made >>>> models. >>>> Now, I am converting them to PhraseDictionaryCompact using following >>>> command (for exmaple): >>>> >>>> >>>> *~/mosesdecoder/bin/processPhraseTableMin -threads all -in >>>> ~/model/phrase-table.3.gz -nscores 4 -out binarised-model/phrase-table * >>>> >>>> But after passing 1/3, it gave following segementation fault error: >>>> >>>> *Pass 1/3: Creating hash function for rank assignment* >>>> *Segmentation fault (core dumped)* >>>> >>>> I have found almost same issue on this thread: >>>> http://comments.gmane.org/gmane.comp.nlp.moses.user/13033 >>>> However, I have provided the existing *binarised-model *folder in the >>>> command. Also, I have the write-access in /tmp but still it gave >>>> sementation fault. >>>> >>>> Can you please tell me what could be wrong here ? >>>> >>>> Thanking You. >>>> >>>> Regards, >>>> Shubham >>>> >>> >>> >>> >>> >>> >>> ___ >>> Moses-support mailing list >>> Moses-support@mit.edu >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> >> > > > > moses.ini.2 Description: Binary data ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact
Hi Hieu, Thanks for your reply. Yes, I have used the absolute path and also I tried with -T but it did not work. Is there any other solution to this problem. Btw, Can anybody please upload the compact model of all pre-made models as this will take less space and also it will be very fast during decoding. Thanks. On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang wrote: > the previous email you referred to says that the directory > > * binarised-model/* > > *must exist before you run it, otherwise it will segfault. I would also > use absolute path to make sure, ie. not * > *binarised-model/phrase-table * > > *but* > > > * /home/shubham/moses/binarised-model/phrase-table * > > *The previous email exchange also says you should try to add the argument* > > > * -T .* > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 8 December 2016 at 15:52, Shubham Khandelwal > wrote: > >> Hello, >> >> This is just the reminder of my previous email. >> >> Thanking You. >> >> Regards, >> Shubham >> >> On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal >> wrote: >> >>> Hello, >>> >>> I have just downloaded phrase-table.2.gz (18GB) de-en model >>> and phrase-table.3.gz (22GB) fr-en model from the available pre-made >>> models. >>> Now, I am converting them to PhraseDictionaryCompact using following >>> command (for exmaple): >>> >>> >>> *~/mosesdecoder/bin/processPhraseTableMin -threads all -in >>> ~/model/phrase-table.3.gz -nscores 4 -out binarised-model/phrase-table * >>> >>> But after passing 1/3, it gave following segementation fault error: >>> >>> *Pass 1/3: Creating hash function for rank assignment* >>> *Segmentation fault (core dumped)* >>> >>> I have found almost same issue on this thread: >>> http://comments.gmane.org/gmane.comp.nlp.moses.user/13033 >>> However, I have provided the existing *binarised-model *folder in the >>> command. Also, I have the write-access in /tmp but still it gave >>> sementation fault. >>> >>> Can you please tell me what could be wrong here ? >>> >>> Thanking You. >>> >>> Regards, >>> Shubham >>> >> >> >> >> >> >> ___ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Conversion of phrase model to PhraseDictionaryCompact
Hello, This is just the reminder of my previous email. Thanking You. Regards, Shubham On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal wrote: > Hello, > > I have just downloaded phrase-table.2.gz (18GB) de-en model > and phrase-table.3.gz (22GB) fr-en model from the available pre-made > models. > Now, I am converting them to PhraseDictionaryCompact using following > command (for exmaple): > > > *~/mosesdecoder/bin/processPhraseTableMin -threads all -in > ~/model/phrase-table.3.gz -nscores 4 -out binarised-model/phrase-table * > > But after passing 1/3, it gave following segementation fault error: > > *Pass 1/3: Creating hash function for rank assignment* > *Segmentation fault (core dumped)* > > I have found almost same issue on this thread: > http://comments.gmane.org/gmane.comp.nlp.moses.user/13033 > However, I have provided the existing *binarised-model *folder in the > command. Also, I have the write-access in /tmp but still it gave > sementation fault. > > Can you please tell me what could be wrong here ? > > Thanking You. > > Regards, > Shubham > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Conversion of phrase model to PhraseDictionaryCompact
Hello, I have just downloaded phrase-table.2.gz (18GB) de-en model and phrase-table.3.gz (22GB) fr-en model from the available pre-made models. Now, I am converting them to PhraseDictionaryCompact using following command (for exmaple): *~/mosesdecoder/bin/processPhraseTableMin -threads all -in ~/model/phrase-table.3.gz -nscores 4 -out binarised-model/phrase-table * But after passing 1/3, it gave following segementation fault error: *Pass 1/3: Creating hash function for rank assignment* *Segmentation fault (core dumped)* I have found almost same issue on this thread: http://comments.gmane.org/gmane.comp.nlp.moses.user/13033 However, I have provided the existing *binarised-model *folder in the command. Also, I have the write-access in /tmp but still it gave sementation fault. Can you please tell me what could be wrong here ? Thanking You. Regards, Shubham ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Regarding Decoding Time
Hello, I have created one translation model (German to English) and converted it to Compact format. Now when I run the command: ~/mosesdecoder/bin/moses2 -f moses.ini Then, It loads the model very quickly but when I pass the German Text (around 4-5 lines) for the translation then it translates the text in 10 minutes. Can you please tell me that how can I reduce this time for the translation ? Thanking You. Regards, Shubham ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
