Hi Hieu, Just to provide more info, I had compiled moses using the following command: "./bjam -j8 -q --with-cmph=/cs/natlang-user/jasneet/softwares/cmph-2.0/ --with-boost=/cs/natlang-user/jasneet/softwares/boost/ --max-kenlm-order=8 -a --with-mm --with-probing-pt”.
Following are some more translation times from the logs using the command: $ grep “Translation took” mert.log Line 53: Translation took 9504.886 seconds total Line 25: Translation took 16931.106 seconds total Line 20: Translation took 17477.958 seconds total Line 34: Translation took 18409.183 seconds total Line 36: Translation took 20495.204 seconds total Line 48: Translation took 16093.966 seconds total Line 68: Translation took 4773.139 seconds total Line 18: Translation took 22165.429 seconds total Line 10: Translation took 23794.930 seconds total Line 11: Translation took 26313.130 seconds total Line 74: Translation took 6238.326 seconds total Line 66: Translation took 14968.715 seconds total Line 3: Translation took 28973.902 seconds total Line 45: Translation took 27619.088 seconds total Line 81: Translation took 4666.394 seconds total Line 37: Translation took 36502.892 seconds total Line 83: Translation took 3143.882 seconds total Line 70: Translation took 20143.743 seconds total Line 1: Translation took 38498.391 seconds total Line 19: Translation took 39683.472 seconds total Line 15: Translation took 39903.566 seconds total Line 33: Translation took 40047.447 seconds total The times are extremely high and I’m not really sure why it is taking so much time. Regards, Jasneet > On Feb 18, 2016, at 11:04 AM, Jasneet Sabharwal <jasneet.sabhar...@sfu.ca> > wrote: > > Hi, > > I was able to solve the segmentation fault issue. It was because of OOVs. I’m > currently trying to tune the parameters using mert, but it is running > extremely slow. For example, from the logs: > > Translating: 美国 之 音 记者 伏 来 库 斯 从 布宜诺斯艾利斯 发 来 的 另 一 篇 报导 说 , 几 名 美国 国会 议员 星期二 > 把 这 一 争论 带 到 了 布宜诺斯艾利斯 的 会议 大厅 。 > Line 43: Initialize search took 0.007 seconds total > Line 43: Collecting options took 0.191 seconds at moses/Manager.cpp:117 > Line 38: Search took 1092.075 seconds > Line 38: Decision rule took 0.000 seconds total > Line 38: Additional reporting took 0.041 seconds total > Line 38: Translation took 1092.132 seconds total > > I tried to time the functions in my feature function > <https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp> > using clock_t but all of them show up as 0.000. I’m not sure why tuning is > taking too much time. My moses.ini is attached in this email. > > Any suggestions would be helpful. > > Regards, > Jasneet > > <moses.ini> >> On Feb 12, 2016, at 3:58 PM, Hieu Hoang <hieuho...@gmail.com >> <mailto:hieuho...@gmail.com>> wrote: >> >> I think it's >> FeatureFunction::GetScoreProducerDescription() >> >> On 12/02/16 23:56, Jasneet Sabharwal wrote: >>> Thanks, will give that a try. >>> >>> Also, is it possible to get the value of feature name inside the feature >>> function. I’m specifically talking about “name” parameter in moses.ini. I’m >>> running multiple copies of my feature function with different parameter as >>> follows: >>> CoarseBiLM name=CoarseBiLM tgtWordId... >>> CoarseBiLM name=CoarseLM100 tgtWordId… >>> CoarseBiLM name=CoarseLM1600 tgtWordId... >>> CoarseBiLM name=CoarseBiLMWithoutClustering tgtWordId… >>> >>> Thanks, >>> Jasneet >>>> On Feb 12, 2016, at 3:39 PM, Hieu Hoang < >>>> <mailto:hieuho...@gmail.com>hieuho...@gmail.com >>>> <mailto:hieuho...@gmail.com>> wrote: >>>> >>>> you can run the decoder >>>> ./moses -v 3 >>>> however, you should put debugging messages in your feature functions to >>>> find out where the problem is. It looks like its in the Load() method so >>>> add lots of debugging message in there and all functions it calls >>>> >>>> On 12/02/16 23:34, Jasneet Sabharwal wrote: >>>>> Thanks Hieu for your reply. >>>>> >>>>> Is it possible to do a verbose output of what’s happening, so that I can >>>>> identify when it’s going out of memory? I’m only running it for 1928 >>>>> sentences. I have almost 170gb of free memory and additional 400gb memory >>>>> in buffer. >>>>> >>>>> Thanks, >>>>> Jasneet >>>>> >>>>>> On Feb 12, 2016, at 2:36 PM, Hieu Hoang <hieuho...@gmail.com >>>>>> <mailto:hieuho...@gmail.com>> wrote: >>>>>> >>>>>> looks like it's run out of memory. >>>>>> >>>>>> On 11/02/16 23:23, Jasneet Sabharwal wrote: >>>>>>> Hi, >>>>>>> >>>>>>> I was adding a new feature function in Moses ( >>>>>>> <https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp>https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp >>>>>>> >>>>>>> <https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp>). >>>>>>> It works fine when I test it for 1-2 sentences, but when I’m trying to >>>>>>> tune my parameters, I’m getting segmentation faults or sometimes it is >>>>>>> bad_alloc. Following was one of the commands that was executed during >>>>>>> the tuning process which caused the Segmentation Fault or bad_alloc: >>>>>>> >>>>>>> moses -threads 40 -v 0 -config filtered/moses.ini -weight-overwrite >>>>>>> 'CoarseLM100= 0.075758 LM0= 0.075758 CoarseBiLMNotClustered= 0.075758 >>>>>>> WordPenalty0= -0.151515 PhrasePenalty0= 0.030303 CoarseBiLMClustered= >>>>>>> 0.075758 TranslationModel0= 0.030303 0.030303 0.030303 0.030303 >>>>>>> Distortion0= 0.045455 CoarseLM1600= 0.075758 LexicalReordering0= >>>>>>> 0.045455 0.045455 0.045455 0.045455 0.045455 0.045455' -n-best-list >>>>>>> run1.best100.out 100 distinct -input-file tune.word.lc.cn >>>>>>> <http://tune.word.lc.cn/> >>>>>>> >>>>>>> The log is enclosed in this email. >>>>>>> >>>>>>> Any pointers would be very useful. >>>>>>> >>>>>>> Thanks, >>>>>>> Jasneet >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Moses-support mailing list >>>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu> >>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>>> <http://mailman.mit.edu/mailman/listinfo/moses-support> >>>>>> >>>>>> -- >>>>>> Hieu Hoang >>>>>> http://www.hoang.co.uk/hieu <http://www.hoang.co.uk/hieu> >>>> >>>> -- >>>> Hieu Hoang >>>> http://www.hoang.co.uk/hieu <http://www.hoang.co.uk/hieu> >> >> -- >> Hieu Hoang >> http://www.hoang.co.uk/hieu <http://www.hoang.co.uk/hieu>
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
