Hi, the reason is to make the results a (little bit) more comparable. Tokenization drastically changes the absolute values of metrics like BLEU, NIST, ... By converting to "the output that humans want" and then using a fixed evaluation chain, you somewhat reduce the problem.
Internally, we evaluate in our tokenization and before recasing. It is definitely ok for checking the progress within a given MT type and a particular toolkit implementation. Cheers, Ondrej. On 06/08/2013 07:34 PM, Cyrine NASRI wrote: > Dear all, > > I have a question concern evaluation step. > > I see in the tutorial that we have to Recase and Detokenize the output. > > But i don't know exactly the reason for doing this > > Any explanation please? > > Thank you > > Best regards > > -- > /Cyrine > Ph.D. Student in Computer Science/ > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- Ondrej Bojar (mailto:[email protected] / [email protected]) http://www.cuni.cz/~obo _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
