Hi, This is just the reminder of my previous email. Thanks.
Regards, Shubham On Dec 13, 2016 3:02 PM, "Shubham Khandelwal" <skhlnm...@gmail.com> wrote: > Thanks. Hieu. I understood now. > Also, Is there any limit of number of words for the translation because > when I use this command: *~/mosesdecoder/bin/moses -f moses.ini* > Then, it translates only some words which are in the beginning. I mean, it > does not consume all words for the translation after Created input-output > object. > Is there any way by which I can control or remove this limit. > > Thanks. > > On Mon, Dec 12, 2016 at 7:17 PM, Hieu Hoang <hieuho...@gmail.com> wrote: > >> there are actually 7 different configurations. You have to look at the >> config file in >> steps/?/config.? >> For fr-en: >> 1. phrase-based, truecased >> 2. phrase-based, lowercased then recased >> 3. hierarchical model, lowercased then recased >> 4. phrase-based, lowercased then recased. Using target side word + pos >> factors >> 5. Like (2) but using batch-mira to tune >> 6. Like (2) but using PRO to tune >> 7. Like (2) but using CreateOnDiskPt to create binary phrase table >> You can see the BLEU scores in >> evaluation/report.* >> >> Hieu Hoang >> http://www.hoang.co.uk/hieu >> >> On 12 December 2016 at 13:28, Shubham Khandelwal <skhlnm...@gmail.com> >> wrote: >> >>> Okay Thanks Hieu. I will try it with 1TB HD-memory machine. >>> Btw I can see there are 4 pre-made models available for fr-en and de-en ( >>> http://www.statmt.org/moses/RELEASE-3.0/models/fr-en/model/ and >>> http://www.statmt.org/moses/RELEASE-3.0/models/de-en/model/). Can you >>> please tell me among these 4, which one is better model(in terms of bleu >>> score) except the huge model which is already there in both, as I can not >>> understand how analysis is shown in steps folder. >>> Also, Are all these pre-made models hierarchical model ? >>> >>> >>> On Mon, Dec 12, 2016 at 6:09 PM, Hieu Hoang <hieuho...@gmail.com> wrote: >>> >>>> >>>> >>>> Hieu Hoang >>>> http://www.hoang.co.uk/hieu >>>> >>>> On 10 December 2016 at 14:06, Shubham Khandelwal <skhlnm...@gmail.com> >>>> wrote: >>>> >>>>> Yes, CreateOnDiskPt command executed without any error. >>>>> >>>>> There are 5 files in this phrase-table.3.folder: Misc.dat >>>>> , Source.dat, TargetColl.dat, TargetInd.dat, Vocab.dat >>>>> *Misc.dat and Vocab.dat files are empty. * >>>>> I just checked that my hard-disk memory is full as this folder >>>>> took 165G already. So may be, due to this reason those 2 files are empty. >>>>> But CreateOnDiskPt command should throw an error of *No space left on >>>>> machine *when it stopped. >>>>> Let me know if no space on my machine, is the issue or not so that I >>>>> can go for better device having more hard-disk memory. >>>>> >>>> Good idea. Not sure who's going to do it but if you do it, please send >>>> me a patch & I'll check it in >>>> >>>>> >>>>> Also May I know that How much memory phrase-table.3.folder has in >>>>> general when CreateOnDiskPt command executes completely >>>>> as phrase-table.3.gz size is only 23GB. >>>>> >>>> I'm not too sure. Try it on a disk with 1TB and please report back what >>>> you find for future reference >>>> >>>>> >>>>> Thanking You. >>>>> >>>>> >>>>> On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang <hieuho...@gmail.com> >>>>> wrote: >>>>> >>>>>> strange, did the CreateOnDiskPt command execute ok, ie. with no error? >>>>>> >>>>>> Does this file exist: >>>>>> /home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat >>>>>> If you do >>>>>> cat Misc.dat >>>>>> what does it say? >>>>>> >>>>>> Hieu Hoang >>>>>> http://www.hoang.co.uk/hieu >>>>>> >>>>>> On 10 December 2016 at 11:30, Shubham Khandelwal <skhlnm...@gmail.com >>>>>> > wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Thanks Hieu for your reply. >>>>>>> I have used *CreateOnDiskPt* to binarize the model and stored in >>>>>>> *phrase-table.3.folder *using following command: >>>>>>> >>>>>>> >>>>>>> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2 phrase-table.3.gz >>>>>>> phrase-table.3.folder* >>>>>>> >>>>>>> Also I have made changes in *moses.ini.3 (*i.e. I have converted >>>>>>> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*) >>>>>>> * >>>>>>> >>>>>>> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4 >>>>>>> path=/home/shubham/models/fr-en/phrase-table.3.folder >>>>>>> input-factor=0 output-factor=0 >>>>>>> >>>>>>> Now, when I run it using :* ~/mosesdecoder/bin/moses -f >>>>>>> moses.ini.3 * , it gave following error after *Created input-output >>>>>>> object*: >>>>>>> >>>>>>> *terminate called after throwing an instance of 'util::Exception'* >>>>>>> * what(): OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t >>>>>>> OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw >>>>>>> util::Exception >>>>>>> because `iter == m_miscInfo.end()'.* >>>>>>> *Couldn't find value for key NumSourceFactors* >>>>>>> *Aborted (core dumped)* >>>>>>> >>>>>>> Here, I do not know that what key value should I pass and how ? Can >>>>>>> you please help me in this regard. >>>>>>> >>>>>>> Thank you so much for your help. >>>>>>> >>>>>>> Regards, >>>>>>> Shubham >>>>>>> >>>>>>> On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang <hieuho...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> This is a hierarchical model. You must binarize with CreateOnDiskPt >>>>>>>> for this model >>>>>>>> >>>>>>>> Hieu Hoang >>>>>>>> http://www.hoang.co.uk/hieu >>>>>>>> >>>>>>>> On 9 December 2016 at 08:18, Shubham Khandelwal < >>>>>>>> skhlnm...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> Thanks. It worked now. I have created compact phrase table. >>>>>>>>> Now, when I am running it using following command: >>>>>>>>> >>>>>>>>> *~/mosesdecoder/bin/moses >>>>>>>>> -f ~/Translate/models/de-en/model/moses.ini.2 -threads all* >>>>>>>>> >>>>>>>>> Then, after creating input-output object, it gave following >>>>>>>>> segmentation fault error: >>>>>>>>> >>>>>>>>> Created input-output object : [14.796] seconds >>>>>>>>> Ich bin ein Student >>>>>>>>> Line 0: Initialize search took 0.000 seconds total >>>>>>>>> Translating: <s> Ich bin ein Student </s> ||| [0,0]=X (1) [0,1]=X >>>>>>>>> (1) [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) >>>>>>>>> [1,2]=X (1) >>>>>>>>> [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X >>>>>>>>> (1) >>>>>>>>> [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X >>>>>>>>> (1) >>>>>>>>> [5,5]=X (1) >>>>>>>>> >>>>>>>>> Segmentation fault (core dumped) >>>>>>>>> >>>>>>>>> In my machine, I have 40GB RAM but still I am confused why it gave >>>>>>>>> this error. >>>>>>>>> Can you please help me in this regard. I have attached moses.ini.2 >>>>>>>>> for your reference. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Shubham >>>>>>>>> >>>>>>>>> On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang <hieuho...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> maybe try >>>>>>>>>> >>>>>>>>>> -encoding None >>>>>>>>>> >>>>>>>>>> On 08/12/2016 19:44, Shubham Khandelwal wrote: >>>>>>>>>> >>>>>>>>>> Hi Hieu, >>>>>>>>>> >>>>>>>>>> Thanks for your reply. >>>>>>>>>> Yes, I have used the absolute path and also I tried with -T but >>>>>>>>>> it did not work. >>>>>>>>>> Is there any other solution to this problem. >>>>>>>>>> >>>>>>>>>> Btw, Can anybody please upload the compact model of all pre-made >>>>>>>>>> models as this will take less space and also it will be very fast >>>>>>>>>> during >>>>>>>>>> decoding. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang <hieuho...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> the previous email you referred to says that the directory >>>>>>>>>>> >>>>>>>>>>> * binarised-model/ * >>>>>>>>>>> >>>>>>>>>>> *must exist before you run it, otherwise it will segfault. I >>>>>>>>>>> would also use absolute path to make sure, ie. not * >>>>>>>>>>> *binarised-model/phrase-table * >>>>>>>>>>> >>>>>>>>>>> *but * >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> * /home/shubham/moses/binarised-model/phrase-table * >>>>>>>>>>> >>>>>>>>>>> *The previous email exchange also says you should try to add the >>>>>>>>>>> argument * >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> * -T . * >>>>>>>>>>> >>>>>>>>>>> Hieu Hoang >>>>>>>>>>> http://www.hoang.co.uk/hieu >>>>>>>>>>> >>>>>>>>>>> On 8 December 2016 at 15:52, Shubham Khandelwal < >>>>>>>>>>> skhlnm...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> This is just the reminder of my previous email. >>>>>>>>>>>> >>>>>>>>>>>> Thanking You. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Shubham >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal < >>>>>>>>>>>> skhlnm...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> >>>>>>>>>>>>> I have just downloaded phrase-table.2.gz (18GB) de-en model >>>>>>>>>>>>> and phrase-table.3.gz (22GB) fr-en model from the available >>>>>>>>>>>>> pre-made >>>>>>>>>>>>> models. >>>>>>>>>>>>> Now, I am converting them to PhraseDictionaryCompact using >>>>>>>>>>>>> following command (for exmaple): >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> *~/mosesdecoder/bin/processPhraseTableMin -threads all -in >>>>>>>>>>>>> ~/model/phrase-table.3.gz -nscores 4 -out >>>>>>>>>>>>> binarised-model/phrase-table * >>>>>>>>>>>>> >>>>>>>>>>>>> But after passing 1/3, it gave following segementation fault >>>>>>>>>>>>> error: >>>>>>>>>>>>> >>>>>>>>>>>>> *Pass 1/3: Creating hash function for rank assignment* >>>>>>>>>>>>> *Segmentation fault (core dumped)* >>>>>>>>>>>>> >>>>>>>>>>>>> I have found almost same issue on this thread: >>>>>>>>>>>>> http://comments.gmane.org/gmane.comp.nlp.moses.user/13033 >>>>>>>>>>>>> However, I have provided the existing *binarised-model *folder >>>>>>>>>>>>> in the command. Also, I have the write-access in /tmp but >>>>>>>>>>>>> still it gave sementation fault. >>>>>>>>>>>>> >>>>>>>>>>>>> Can you please tell me what could be wrong here ? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanking You. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Shubham >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Moses-support mailing list >>>>>>>>>>>> Moses-support@mit.edu >>>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> Yours Sincerely, >>> >>> Shubham Khandelwal >>> Masters in Informatics (M2-MoSIG), >>> University Joseph Fourier-Grenoble INP, >>> Grenoble, France >>> Webpage: https://sites.google.com/site/skhandelwl21/ >>> >> >> > > > -- > Yours Sincerely, > > Shubham Khandelwal > Masters in Informatics (M2-MoSIG), > University Joseph Fourier-Grenoble INP, > Grenoble, France > Webpage: https://sites.google.com/site/skhandelwl21/ >
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
