Re: [Moses-support] sparse features, optimization, custom metrics
Hi Barry, OK, thanks for the confirmation, so there is sense to try it. I will see whether I can manage to add my metric (which by itself is not particularly useful to the community) and maybe I will manage to convince pro or kbmira by the way to use the general Scorer classes from mert. Which one do you prefer for sparse features? How do they cope with optimizer instability compared to mert? Best, Marcin W dniu 09.02.2014 20:53, Barry Haddow pisze: > Hi Marcin > > There was a project at MTM2012 for this, but I have not seen any > outputs from it > http://www.statmt.org/mtm12/index.php%3Fn=Projects.NewDevelopmentFuncionalityForTheAsiyaSuiteParameterOptimizationWithMert > > > I am not aware of anyone working on new metrics for pro and kbmira. > > In principle I don't think it would be hard to implement. The current > implementations of pro and kbmira make use of the sufficient > statistics in the same way that mert does. The main difference is that > they require evaluations of single sentences, as opposed to mert which > can optimise a corpus metric. kbmira uses Chiang's technique (from his > 2008 mira paper) to approximate corpus bleu, but pro just optimises > sentence bleu. However it could (and perhaps should) also employ > Chiang's technique. Both pro and kbmira use methods from BleuScorer to > score the sentences -- smoothedSentenceBleu() and > sentenceLevelBackgroundBleu() respectively. > > cheers - Barry > > On 09/02/14 09:19, Marcin Junczys-Dowmunt wrote: >> Hi list, >> It seems that currently for both, pro and kbmira, optimization of BLEU >> is hardwired into the code. I managed to add my custom metric to mert, >> but would like to experiment with it and sparse features, too. >> >> I see custom metrics is on a TODO list in the mert folder, is someone >> working on custom metrics for sparse features? >> Are pro and/or kbmira in principle compatible with this "sufficient >> statistics per sentence" approach as it is done for mert? Any pointers >> how I could best attack this? >> >> Best, >> Marcin >> ___ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] sparse features, optimization, custom metrics
Hi Marcin There was a project at MTM2012 for this, but I have not seen any outputs from it http://www.statmt.org/mtm12/index.php%3Fn=Projects.NewDevelopmentFuncionalityForTheAsiyaSuiteParameterOptimizationWithMert I am not aware of anyone working on new metrics for pro and kbmira. In principle I don't think it would be hard to implement. The current implementations of pro and kbmira make use of the sufficient statistics in the same way that mert does. The main difference is that they require evaluations of single sentences, as opposed to mert which can optimise a corpus metric. kbmira uses Chiang's technique (from his 2008 mira paper) to approximate corpus bleu, but pro just optimises sentence bleu. However it could (and perhaps should) also employ Chiang's technique. Both pro and kbmira use methods from BleuScorer to score the sentences -- smoothedSentenceBleu() and sentenceLevelBackgroundBleu() respectively. cheers - Barry On 09/02/14 09:19, Marcin Junczys-Dowmunt wrote: > Hi list, > It seems that currently for both, pro and kbmira, optimization of BLEU > is hardwired into the code. I managed to add my custom metric to mert, > but would like to experiment with it and sparse features, too. > > I see custom metrics is on a TODO list in the mert folder, is someone > working on custom metrics for sparse features? > Are pro and/or kbmira in principle compatible with this "sufficient > statistics per sentence" approach as it is done for mert? Any pointers > how I could best attack this? > > Best, > Marcin > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] exception during tuning step
Hello all when I run moses EMS, in the tuning step, it gives this exception: Exception: moses/FF/Factory.cpp:235 in void Moses::FeatureRegistry::Construct(const string&, const string&) threw UnknownFeatureException because `i == registry_.end()'. Feature name IRSTLM is not registered. Exit code: 1 Failed to run moses with the config /opt/working/ems/tuning/moses.filtered.ini.1 at /opt/tools/mosesdecoder/scripts/training/mert-moses.pl line 1271. cp: cannot stat '/opt/working/ems/tuning/tmp.1/moses.ini': No such file or directory I will be thankful if you could help me to solve this problem. Regards amir ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] sparse features, optimization, custom metrics
Hi list, It seems that currently for both, pro and kbmira, optimization of BLEU is hardwired into the code. I managed to add my custom metric to mert, but would like to experiment with it and sparse features, too. I see custom metrics is on a TODO list in the mert folder, is someone working on custom metrics for sparse features? Are pro and/or kbmira in principle compatible with this "sufficient statistics per sentence" approach as it is done for mert? Any pointers how I could best attack this? Best, Marcin ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
