Got it, I just realized that the
dev<http://scikit-learn.org/dev/modules/model_evaluation.html#the-scoring-parameter-defining-model-evaluation-rules>documentation
is outdated (looking at the code I noticed that make_scorer
has been replaced by Scorer).
Thanks.
Josh
On Thu, Jul 25, 2013 at 8:24 AM, Josh Wasserstein <[email protected]>wrote:
> Thanks. I am having problems when using the micro/macro variants for
> GridSearchCV. I tried creating the corresponding scorer objects, but I got
> the error:
>
> > cannot import name make_scorer
>
> This is with 0.14 git (from master) that I checked out about a week ago.
> Here is the code in more detail
> ============================
> from sklearn.metrics import fbeta_score, f1_score, make_scorer
> f1_micro = make_scorer(f1_score, average='micro')
> f1_macro = make_scorer(f1_score, average='macro')
> f1_weighted = make_scorer(f1_score, average='weighted')
>
> score_functions = [f1_micro, f1_macro, f1_weighted]
>
> for score_func in score_functions:
> clf = GridSearchCV(SVC(C=1, cache_size=5000),
> tuned_parameters,
> scoring=score_func,
> verbose=1, n_jobs=1, cv=cv_method)
> clf.fit(X, y)
> ...
> ============================
>
> Josh
>
>
> On Thu, Jul 25, 2013 at 8:07 AM, Olivier Grisel
> <[email protected]>wrote:
>
>> 2013/7/25 Josh Wasserstein <[email protected]>:
>> > Thank you Olivier. I went through that paper and I agree, it looks like
>> > implementing micro-AUC or macro-AUC should not be that hard. I will
>> try to
>> > implement within the next week. I have have never contributed to a
>> project
>> > in GitHub, so I am not sure to what extent my code would meet the
>> standards
>> > but I am happy to try.
>> >
>> > In the mean time, is there anything similar to an AUC metric that scikit
>> > supports when working with GridSearchCV in a multi-label setting? I am
>> > looking for some compromise between precision and recall that indirectly
>> > optimizes for the AUC score of each label .
>>
>> You can try the f1 score that is a balanced score (a tradeoff between
>> precision and recall) that is a reasonable score for imbalanced
>> multiclass dataset.
>>
>> It supports both micro and macro averaging.
>>
>>
>> --
>> Olivier
>> http://twitter.com/ogrisel - http://github.com/ogrisel
>>
>>
>> ------------------------------------------------------------------------------
>> See everything from the browser to the database with AppDynamics
>> Get end-to-end visibility with application monitoring from AppDynamics
>> Isolate bottlenecks and diagnose root cause in seconds.
>> Start your free trial of AppDynamics Pro today!
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Scikit-learn-general mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>
>
------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
