On Tue, Mar 21, 2017 at 05:09:46PM +0500, Kirill Mishchenko wrote:
> Ryan,
>
> I’m working on a proposal for the idea, and wondering whether
> hyper-parameter module should be flexible enough to support metrics
> with different correlations. E.g., if we use accuracy as a metric,
> then we want to find a model that maximises this metric; on the other
> hand, if we want to use some kind of error as a metric (like mean
> squared error), then we need to find a model that minimises the
> metric. So, again, the question is whether hyper-parameter module
> should be flexible enough to maximise some metrics and minimise
> others?
Hi Kirill,
I agree, this would be nice support. Although it's true that you can
maximize, e.g., (1 / RMSE), that might be a bit awkward. One idea might
be to use traits for the individual metrics being optimized, like:
/**
* I'm just making up this interface, maybe you have a better idea.
*/
class RMSE
{
// The signature almost certainly needs to be different.
template<typename Model>
static double Evaluate(Model& m,
arma::mat& testData,
arma::Mat<size_t> testLabels);
// This boolean can be used with SFINAE to determine whether or not
// minimization or maximization is necessary. Maybe it could have a
// better name too.
static const bool NeedsMinimization = true;
};
What do you think? You could easily automatically generate a wrapper
class for any type where NeedsMinimization was false, so that you could
use the mlpack optimizers with it. (Probably most of the optimizers
won't be useful, but in the end any optimizers you implemented for
hyper-parameter tuning, like grid search, could probably follow the same
API as the rest of the other mlpack optimizers, and later be used for
other tasks too.)
--
Ryan Curtin | "You can think about it... but don't do it."
[email protected] | - Sheriff Justice
_______________________________________________
mlpack mailing list
[email protected]
http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack
