Thanks Joel. I take that those metrics are supported by GridSearchCV. I
think my confusion comes from the fact that some names under "metrics" such
as "classification_report" cannot be used for model selection (I think I
have read that only things under metrics that end in "_score" or "_error"
work for that).
I was also looking at how to implement your own Scorer
objects<http://scikit-learn.org/dev/modules/model_evaluation.html#scoring-objects-defining-your-scoring-rules>.
I read the interface of
Scorer<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.Scorer.html>
but
I am still not clear on how to define your own scorers for multi-label
problems, specifically when working with unthresholded scores. What do *
ground_truth* and *predictions *hold when we want to implement a Scorer
that takes unthresholded predictions in multi-label classification? (I
presume they are not 1D arrays anymore, correct?)
def my_custom_loss_func(ground_truth, predictions):
Thanks again for providing this fantastic Python package. I look forward to
start contributing to the project soon.
Josh
On Wed, Jul 17, 2013 at 7:53 PM, Joel Nothman
<[email protected]>wrote:
> To be clear,
> http://scikit-learn.org/dev<http://scikit-learn.org/dev/whats_new.html>
> corresponds
> to master (with a little lag).
>
>
> On Thu, Jul 18, 2013 at 9:41 AM, Joel Nothman <
> [email protected]> wrote:
>
>> whats_new.html says:
>>
>>
>> - Multi-label classification output is now supported by
>>
>> metrics.accuracy_score<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score>
>>
>> ,metrics.zero_one_loss<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.zero_one_loss.html#sklearn.metrics.zero_one_loss>
>> ,
>> metrics.f1_score<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score>
>> ,
>> metrics.fbeta_score<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.fbeta_score.html#sklearn.metrics.fbeta_score>
>>
>> ,metrics.classification_report<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.classification_report.html#sklearn.metrics.classification_report>
>> ,
>> metrics.precision_score<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score>
>>
>> andmetrics.recall_score<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score>
>> by Arnaud Joly <http://www.ajoly.org/>.
>>
>>
>> - Two new metrics
>> metrics.hamming_loss<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.hamming_loss.html#sklearn.metrics.hamming_loss>
>> and
>> metrics.jaccard_similarity_score<http://scikit-learn.org/dev/modules/generated/sklearn.metrics.jaccard_similarity_score.html#sklearn.metrics.jaccard_similarity_score>
>> are
>> added with multi-label support by Arnaud Joly <http://www.ajoly.org/>.
>>
>>
>>
>> On Thu, Jul 18, 2013 at 9:29 AM, Josh Wasserstein <[email protected]
>> > wrote:
>>
>>> Thanks Joel. I am working with the master branch from GitHub. Do you
>>> know which multi-label 'scoring' methods are supported by GridSearchCV in
>>> for now?
>>>
>>> Josh
>>>
>>>
>>> On Wed, Jul 17, 2013 at 7:23 PM, Joel Nothman <
>>> [email protected]> wrote:
>>>
>>>> It's a work in progress. See http://scikit-learn.org/dev/whats_new.html
>>>>
>>>> I think the classes of problems handled by different types of metrics
>>>> should be enumerated
>>>> http://scikit-learn.org/dev/modules/model_evaluation.html but
>>>> currently they aren't.
>>>>
>>>> - Joel
>>>>
>>>>
>>>> On Thu, Jul 18, 2013 at 8:49 AM, Josh Wasserstein <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> What 'scoring' methods (strings or callables) does GridSearchCV
>>>>> support for multi-label classification?
>>>>>
>>>>> I haven't been able to find much on the documentation, but I saw this
>>>>> issue open: Make GridSearchCV and cross_val_score work with
>>>>> multilabel
>>>>> data<https://github.com/scikit-learn/scikit-learn/issues/1683>: by
>>>>> Andreas. Does that mean that there is currently no multi-label scoring
>>>>> method supported by GridSearchCV?
>>>>>
>>>>> Josh
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> See everything from the browser to the database with AppDynamics
>>>>> Get end-to-end visibility with application monitoring from AppDynamics
>>>>> Isolate bottlenecks and diagnose root cause in seconds.
>>>>> Start your free trial of AppDynamics Pro today!
>>>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>>>> _______________________________________________
>>>>> Scikit-learn-general mailing list
>>>>> [email protected]
>>>>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> See everything from the browser to the database with AppDynamics
>>>> Get end-to-end visibility with application monitoring from AppDynamics
>>>> Isolate bottlenecks and diagnose root cause in seconds.
>>>> Start your free trial of AppDynamics Pro today!
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Scikit-learn-general mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>>>
>>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> See everything from the browser to the database with AppDynamics
>>> Get end-to-end visibility with application monitoring from AppDynamics
>>> Isolate bottlenecks and diagnose root cause in seconds.
>>> Start your free trial of AppDynamics Pro today!
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Scikit-learn-general mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>>
>>>
>>
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
