Hi Brown, Thanks for the email. There is a working PR here at https://github.com/scikit-learn/scikit-learn/pull/7388
Would you be kind to take a look at it and comment how helpful the proposed API is for your use case? Thanks On Mon, Oct 3, 2016 at 6:05 AM, Brown J.B. <[email protected]> wrote: > Hello community, > > Congratulations on the release of 0.19 ! > While I'm merely a casual user and wish I could contribute more often, I > thank everyone for their time and efforts! > > 2016-10-01 1:58 GMT+09:00 Andreas Mueller <[email protected]>: > > We've got a lot in the works already for 0.19. >>> >>> * multiple metrics for cross validation (#7388 et al.) >>> >> > I've done something like this in my internal model building and selection > libraries. > My solution has been to have > -each metric object be able to explain a "distance from optimal" > -a metric collection object, which can be built by either explicit > instantiation or calculation using data > -a pareto curve calculation object > -a ranker for the points on the pareto curve, with the ability to select > the N-best points. > > While there are certainly smarter interfaces and implementations, here is > an example of one of my doctests that may help get this PR started. > My apologies that my old docstring argument notation doesn't match the > commonly used standards. > > Hope this helps, > J.B. Brown > Kyoto University > > 26 class TrialRanker(object): > > 27 """An object for handling the generic mechanism of selecting > optimal > 28 trials from a colletion of trials.""" > > 43 def SelectBest(self, metricSets, paretoAlg, > > 44 preProcessor=None): > > 45 """Select the best [metricSets] by using > the > 46 [paretoAlg] pareto selection object. Note that it is > actually > 47 the [paretoAlg] that specifies how many optimal [metricSets] > to > 48 select. > > 49 > > 50 Data may be pre-processed into a form necessary for the > [paretoAlg] > 51 by using the [preProcessor] that is a > MetricSetConverter. > 52 > > 53 Return: an EvaluatedMetricSet if [paretoAlg] selects only > one > 54 metric set, otherwise a list of EvaluatedMetricSet > objects. > 55 > > 56 >>> from pareto.paretoDecorators import > MinNormSelector > 57 >>> from pareto import OriginBasePareto > > 58 >>> pAlg = MinNormSelector(OriginBasePare > to()) > 59 > > 60 >>> from metrics.TwoClassMetrics import Accuracy, > Sensitivity > 61 >>> from metrics.metricSet import > EvaluatedMetricSet > 62 >>> met1 = EvaluatedMetricSet.BuildByExpl > icitValue( > 63 ... [(Accuracy, 0.7), (Sensitivity, > 0.9)]) > 64 >>> met1.SetTitle("Example1") > > 65 >>> met1.associatedData = range(5) # property > set/get > 66 >>> met2 = EvaluatedMetricSet.BuildByExpl > icitValue( > 67 ... [(Accuracy, 0.8), (Sensitivity, > 0.6)]) > 68 >>> met2.SetTitle("Example2") > > 69 >>> met2.SetAssociatedData("abcdef") # explicit method > call > 70 >>> met3 = EvaluatedMetricSet.BuildByExpl > icitValue( > 71 ... [(Accuracy, 0.5), (Sensitivity, > 0.5)]) > 72 >>> met3.SetTitle("Example3") > > 73 >>> met3.associatedData = float > > 74 > > 75 >>> from metrics.metricSet.converters import > OptDistConverter > 76 > > 77 >>> ranker = TrialRanker() # pAlg selects > met1 > 78 >>> best = ranker.SelectBest((met1,met2,m > et3), > 79 ... pAlg, > OptDistConverter()) > 80 >>> best.VerboseDescription(True) > > 81 >>> str(best) > > 82 'Example1: 2 metrics; Accuracy=0.700; > Sensitivity=0.900' > 83 >>> best.associatedData > > 84 [0, 1, 2, 3, 4] > > 85 > > 86 >>> pAlg = MinNormSelector(OriginBasePareto(), > nSelect=2) > 87 >>> best = ranker.SelectBest((met1,met2,m > et3), > 88 ... pAlg, > OptDistConverter()) > 89 >>> for metSet in best: > > 90 ... metSet.VerboseDescription(True > ) > 91 ... str(metSet) > > 92 ... str(metSet.associatedData) > > 93 'Example1: 2 metrics; Accuracy=0.700; > Sensitivity=0.900' > 94 '[0, 1, 2, 3, 4]' > > 95 'Example2: 2 metrics; Accuracy=0.800; > Sensitivity=0.600' > 96 'abcdef' > > 97 > > 98 >>> from metrics.TwoClassMetrics import > PositivePredictiveValue > 99 >>> met4 = EvaluatedMetricSet.BuildByExpl > icitValue( > 100 ... [(Accuracy, 0.7), (PositivePredictiveValue, > 0.5)]) > 101 >>> best = ranker.SelectBest((met1,met2,m > et3,met4), > 102 ... pAlg, > OptDistConverter()) > 103 Traceback (most recent call last): > > 104 ... > > 105 ValueError: Metric sets contain differing > Metrics. > > > > > _______________________________________________ > scikit-learn mailing list > [email protected] > https://mail.python.org/mailman/listinfo/scikit-learn > >
_______________________________________________ scikit-learn mailing list [email protected] https://mail.python.org/mailman/listinfo/scikit-learn
