On Fri, 2016-01-29 at 21:26 +0000, Hieu Hoang wrote: > The decoder should handle no translation without falling over. But > yes, the model is too toy
Normally the decoder would always produce some translation. (The translation could be an empty sentence, of course.) If it's misconfigured, it should tell you about it. But maybe not with a segmentation fault. :-) > On 29 Jan 2016 9:15 pm, "Matthias Huck" <mh...@inf.ed.ac.uk> wrote: > > Hi, > > > > It seems to me that this toy string-to-tree setup is either > > outdated, > > or it always had issues. It should be replaced. > > > > Under real-world conditions, the decoder should always be able to > > produce some hypothesis. We would therefore usually extract a whole > > set > > of glue rules. And we would typically also add an [unknown-lhs] > > section > > to the moses.ini that would tell the decoder which left-hand side > > non > > -terminal labels to use for out-of-vocabulary words. To my > > knowledge, > > these two techniques are crucial for being able to parse any input > > sentence provided to the chart decoder in syntax-based translation. > > > > So, in my opinion, the problem is most likely neither the server > > implementation nor the syntax-based decoder, but a problematic > > setup. > > I would consider it okay for the server to crash (or at least print > > a > > warning) under such circumstances. You don't want it to silently > > not > > translate complete sentences. > > > > (I must admit that I didn't look into it in too much detail, but it > > sho > > uld be easy to confirm.) > > > > Cheers, > > Matthias > > > > > > On Fri, 2016-01-29 at 20:28 +0000, Barry Haddow wrote: > > > Hi All > > > > > > I think I see what happened now. > > > > > > When you give the input "dies ist ein haus" to the sample model, > > the > > > "dies" is unknown, and there is no translation. The server did > > not check > > > for this condition, and got a seg fault. I have added a check, so > > if you > > > pull and try again it should not crash. > > > > > > In the log pasted by Martin, he passed "das ist ein haus" to > > > command-line Moses, which works, and gives a translation. > > > > > > I think ideally the sample models should handle unknown words, > > and give > > > a translation. Maybe adding a glue rule would be sufficient? > > > > > > cheers - Barry > > > > > > On 29/01/16 11:13, Barry Haddow wrote: > > > > Hi > > > > > > > > When I run command-line Moses, I get the output below - i.e. no > > best > > > > translation. The server crashes for me since it does not check > > for the > > > > null pointer, but the command-line version does. > > > > > > > > I think there should be a translation for this example. > > > > > > > > cheers - Barry > > > > > > > > [gna]bhaddow: echo 'dies ist ein haus' | ~/moses.new/bin/moses > > -f > > > > string-to-tree/moses.ini > > > > Defined parameters (per moses.ini or switch): > > > > config: string-to-tree/moses.ini > > > > cube-pruning-pop-limit: 1000 > > > > feature: KENLM name=LM factor=0 order=3 num > > -features=1 > > > > path=lm/europarl.srilm.gz WordPenalty UnknownWordPenalty > > > > PhraseDictionaryMemory input-factor=0 output-factor=0 > > > > path=string-to-tree/rule-table num-features=1 table-limit=20 > > > > input-factors: 0 > > > > inputtype: 3 > > > > mapping: 0 T 0 > > > > max-chart-span: 20 1000 > > > > non-terminals: X S > > > > search-algorithm: 3 > > > > translation-details: translation-details.log > > > > weight: WordPenalty0= 0 LM= 0.5 > > PhraseDictionaryMemory0= 0.5 > > > > line=KENLM name=LM factor=0 order=3 num-features=1 > > path=lm/europarl.srilm.gz > > > > Loading the LM will be faster if you build a binary file. > > > > Reading lm/europarl.srilm.gz > > > > ----5---10---15---20---25---30---35---40---45---50---55---60-- > > -65---70---75---80---85---90---95--100 > > > > **The ARPA file is missing . Substituting log10 probability > > -100.000. > > > > > > ******************************************************************* > > ******************************* > > > > FeatureFunction: LM start: 0 end: 0 > > > > line=WordPenalty > > > > FeatureFunction: WordPenalty0 start: 1 end: 1 > > > > line=UnknownWordPenalty > > > > FeatureFunction: UnknownWordPenalty0 start: 2 end: 2 > > > > line=PhraseDictionaryMemory input-factor=0 output-factor=0 > > > > path=string-to-tree/rule-table num-features=1 table-limit=20 > > > > FeatureFunction: PhraseDictionaryMemory0 start: 3 end: 3 > > > > Loading LM > > > > Loading WordPenalty0 > > > > Loading UnknownWordPenalty0 > > > > Loading PhraseDictionaryMemory0 > > > > Start loading text phrase table. Moses format : [3.038] seconds > > > > Reading string-to-tree/rule-table > > > > ----5---10---15---20---25---30---35---40---45---50---55---60-- > > -65---70---75---80---85---90---95--100 > > > > > > ******************************************************************* > > ********************************* > > > > max-chart-span: 20 > > > > Created input-output object : [3.041] seconds > > > > Line 0: Initialize search took 0.000 seconds total > > > > Translating: dies ist ein haus ||| [0,0]=X (1) [0,1]=X (1) > > > > [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) > > [1,2]=X (1) > > > > [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) > > [2,4]=X (1) > > > > [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) > > [4,5]=X (1) > > > > [5,5]=X (1) > > > > > > > > 0 1 2 3 4 5 > > > > 0 1 2 2 1 0 > > > > 0 0 0 2 0 > > > > 0 0 4 0 > > > > 0 0 0 > > > > 0 0 > > > > 0 > > > > Line 0: Additional reporting took 0.000 seconds total > > > > Line 0: Translation took 0.002 seconds total > > > > Translation took 0.000 seconds > > > > Name:moses VmPeak:74024 kB VmRSS:11084 kB RSSMax:36832 kB > > > > user:2.972 sys:0.048 CPU:3.020 real:3.058 > > > > > > > > > > > > On 29/01/16 00:40, Hieu Hoang wrote: > > > > > If it works ok on the command line but crashes when using the > > server, > > > > > then that suggest a server issue. > > > > > > > > > > I don't know much about the server code, to be honest. > > > > > > > > > > > > > > > > > > > -- > > The University of Edinburgh is a charitable body, registered in > > Scotland, with registration number SC005336. > > > > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
