Most or all of those should have entries in sklearn.metrics.scorer.SCORERS,
but this is all pretty new stuff and may not be sufficiently documented.
The ground truth and predictions should look like your `y`.
A scorer is just a function mapping (estimator, X, y) -> real, and the
Scorer implementation does little more than decide whether to call
`predict` or `decision_function` on the estimator.
For multilabel, estimator.predict() will return a sequence of sequences (or
a label indicator matrix if that was the fit() input) which can be passed
directly to the multi-label-supporting metrics.
estimator.decision_function() will return a matrix of scores for each
possible label, for each sample.
- Joel
On Thu, Jul 18, 2013 at 1:08 PM, Josh Wasserstein <[email protected]>wrote:
> Thanks Joel. I take that those metrics are supported by GridSearchCV. I
> think my confusion comes from the fact that some names under "metrics" such
> as "classification_report" cannot be used for model selection (I think I
> have read that only things under metrics that end in "_score" or "_error"
> work for that).
>
> I was also looking at how to implement your own Scorer
> objects<http://scikit-learn.org/dev/modules/model_evaluation.html#scoring-objects-defining-your-scoring-rules>.
> I read the interface of
> Scorer<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.Scorer.html>
> but
> I am still not clear on how to define your own scorers for multi-label
> problems, specifically when working with unthresholded scores. What do *
> ground_truth* and *predictions *hold when we want to implement a Scorer
> that takes unthresholded predictions in multi-label classification? (I
> presume they are not 1D arrays anymore, correct?)
>
> def my_custom_loss_func(ground_truth, predictions):
>
> Thanks again for providing this fantastic Python package. I look forward
> to start contributing to the project soon.
>
> Josh
>
>
>
> On Wed, Jul 17, 2013 at 7:53 PM, Joel Nothman <
> [email protected]> wrote:
>
>> To be clear,
>> http://scikit-learn.org/dev<http://scikit-learn.org/dev/whats_new.html>
>> corresponds
>> to master (with a little lag).
>>
>>
>> On Thu, Jul 18, 2013 at 9:41 AM, Joel Nothman <
>> [email protected]> wrote:
>>
>>> whats_new.html says:
>>>
>>>
>>> - Multi-label classification output is now supported by
>>>
>>> metrics.accuracy_score<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score>
>>>
>>> ,metrics.zero_one_loss<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.zero_one_loss.html#sklearn.metrics.zero_one_loss>
>>> ,
>>> metrics.f1_score<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score>
>>> ,
>>> metrics.fbeta_score<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.fbeta_score.html#sklearn.metrics.fbeta_score>
>>>
>>> ,metrics.classification_report<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.classification_report.html#sklearn.metrics.classification_report>
>>> ,
>>> metrics.precision_score<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score>
>>>
>>> andmetrics.recall_score<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score>
>>> by Arnaud Joly <http://www.ajoly.org/>.
>>>
>>>
>>> - Two new metrics
>>> metrics.hamming_loss<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.hamming_loss.html#sklearn.metrics.hamming_loss>
>>> and
>>> metrics.jaccard_similarity_score<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.jaccard_similarity_score.html#sklearn.metrics.jaccard_similarity_score>
>>> are
>>> added with multi-label support by Arnaud Joly <http://www.ajoly.org/>
>>> .
>>>
>>>
>>>
>>> On Thu, Jul 18, 2013 at 9:29 AM, Josh Wasserstein <
>>> [email protected]> wrote:
>>>
>>>> Thanks Joel. I am working with the master branch from GitHub. Do you
>>>> know which multi-label 'scoring' methods are supported by GridSearchCV in
>>>> for now?
>>>>
>>>> Josh
>>>>
>>>>
>>>> On Wed, Jul 17, 2013 at 7:23 PM, Joel Nothman <
>>>> [email protected]> wrote:
>>>>
>>>>> It's a work in progress. See
>>>>> http://scikit-learn.org/dev/whats_new.html
>>>>>
>>>>> I think the classes of problems handled by different types of metrics
>>>>> should be enumerated
>>>>> http://scikit-learn.org/dev/modules/model_evaluation.html but
>>>>> currently they aren't.
>>>>>
>>>>> - Joel
>>>>>
>>>>>
>>>>> On Thu, Jul 18, 2013 at 8:49 AM, Josh Wasserstein <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> What 'scoring' methods (strings or callables) does GridSearchCV
>>>>>> support for multi-label classification?
>>>>>>
>>>>>> I haven't been able to find much on the documentation, but I saw this
>>>>>> issue open: Make GridSearchCV and cross_val_score work with
>>>>>> multilabel
>>>>>> data<https://github.com/scikit-learn/scikit-learn/issues/1683>: by
>>>>>> Andreas. Does that mean that there is currently no multi-label scoring
>>>>>> method supported by GridSearchCV?
>>>>>>
>>>>>> Josh
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> See everything from the browser to the database with AppDynamics
>>>>>> Get end-to-end visibility with application monitoring from AppDynamics
>>>>>> Isolate bottlenecks and diagnose root cause in seconds.
>>>>>> Start your free trial of AppDynamics Pro today!
>>>>>>
>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>>>>> _______________________________________________
>>>>>> Scikit-learn-general mailing list
>>>>>> [email protected]
>>>>>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> See everything from the browser to the database with AppDynamics
>>>>> Get end-to-end visibility with application monitoring from AppDynamics
>>>>> Isolate bottlenecks and diagnose root cause in seconds.
>>>>> Start your free trial of AppDynamics Pro today!
>>>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>>>> _______________________________________________
>>>>> Scikit-learn-general mailing list
>>>>> [email protected]
>>>>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> See everything from the browser to the database with AppDynamics
>>>> Get end-to-end visibility with application monitoring from AppDynamics
>>>> Isolate bottlenecks and diagnose root cause in seconds.
>>>> Start your free trial of AppDynamics Pro today!
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Scikit-learn-general mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>>>
>>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> See everything from the browser to the database with AppDynamics
>> Get end-to-end visibility with application monitoring from AppDynamics
>> Isolate bottlenecks and diagnose root cause in seconds.
>> Start your free trial of AppDynamics Pro today!
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Scikit-learn-general mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
