Hi Marcin > Which one do you prefer for sparse features? How do they cope with > optimizer instability compared to mert? We have been using kbmira. It seems a bit more stable than mert, and pro can have problems with the sentence length -- see some recent papers by Preslav Nakov et al on this problem,
cheers - Barry On 09/02/14 20:03, Marcin Junczys-Dowmunt wrote: > Hi Barry, > OK, thanks for the confirmation, so there is sense to try it. I will see > whether I can manage to add my metric (which by itself is not > particularly useful to the community) and maybe I will manage to > convince pro or kbmira by the way to use the general Scorer classes from > mert. > > Which one do you prefer for sparse features? How do they cope with > optimizer instability compared to mert? > Best, > Marcin > > W dniu 09.02.2014 20:53, Barry Haddow pisze: >> Hi Marcin >> >> There was a project at MTM2012 for this, but I have not seen any >> outputs from it >> http://www.statmt.org/mtm12/index.php%3Fn=Projects.NewDevelopmentFuncionalityForTheAsiyaSuiteParameterOptimizationWithMert >> >> I am not aware of anyone working on new metrics for pro and kbmira. >> >> In principle I don't think it would be hard to implement. The current >> implementations of pro and kbmira make use of the sufficient >> statistics in the same way that mert does. The main difference is that >> they require evaluations of single sentences, as opposed to mert which >> can optimise a corpus metric. kbmira uses Chiang's technique (from his >> 2008 mira paper) to approximate corpus bleu, but pro just optimises >> sentence bleu. However it could (and perhaps should) also employ >> Chiang's technique. Both pro and kbmira use methods from BleuScorer to >> score the sentences -- smoothedSentenceBleu() and >> sentenceLevelBackgroundBleu() respectively. >> >> cheers - Barry >> >> On 09/02/14 09:19, Marcin Junczys-Dowmunt wrote: >>> Hi list, >>> It seems that currently for both, pro and kbmira, optimization of BLEU >>> is hardwired into the code. I managed to add my custom metric to mert, >>> but would like to experiment with it and sparse features, too. >>> >>> I see custom metrics is on a TODO list in the mert folder, is someone >>> working on custom metrics for sparse features? >>> Are pro and/or kbmira in principle compatible with this "sufficient >>> statistics per sentence" approach as it is done for mert? Any pointers >>> how I could best attack this? >>> >>> Best, >>> Marcin >>> _______________________________________________ >>> Moses-support mailing list >>> Moses-support@mit.edu >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
