Thanks Joel.
Before posting my post I did the following, so I don't know why I didn't
get the very latest version:
$ pip install git+git://github.com/scikit-learn/scikit-learn@master
Downloading/unpacking git+git://github.com/scikit-learn/scikit-learn@master
Cloning git://github.com/scikit-learn/scikit-learn (to master) to
/tmp/pip-PEFOLU-build
Running setup.py egg_info for package from git+git://
github.com/scikit-learn/scikit-learn@master
Partial import of sklearn during the build process.
My understanding was that pip install would update the package (even when
working with git
repos<http://stackoverflow.com/questions/17710947/pip-pulling-updates-from-remote-git-repository>
).
I tried again and I seem to be using the older version.
Hmm..
Josh
On Thu, Jul 25, 2013 at 8:35 AM, Joel Nothman
<[email protected]>wrote:
> On the contrary, make_scorer, replacing Scorer, was merged into master in
> the last couple of days. Try pulling the latest changes.
>
>
> On Thu, Jul 25, 2013 at 10:33 PM, Josh Wasserstein <[email protected]
> > wrote:
>
>> Got it, I just realized that the
>> dev<http://scikit-learn.org/dev/modules/model_evaluation.html#the-scoring-parameter-defining-model-evaluation-rules>documentation
>> is outdated (looking at the code I noticed that make_scorer
>> has been replaced by Scorer).
>>
>> Thanks.
>>
>> Josh
>>
>>
>> On Thu, Jul 25, 2013 at 8:24 AM, Josh Wasserstein <[email protected]
>> > wrote:
>>
>>> Thanks. I am having problems when using the micro/macro variants for
>>> GridSearchCV. I tried creating the corresponding scorer objects, but I got
>>> the error:
>>>
>>> > cannot import name make_scorer
>>>
>>> This is with 0.14 git (from master) that I checked out about a week ago.
>>> Here is the code in more detail
>>> ============================
>>> from sklearn.metrics import fbeta_score, f1_score, make_scorer
>>> f1_micro = make_scorer(f1_score, average='micro')
>>> f1_macro = make_scorer(f1_score, average='macro')
>>> f1_weighted = make_scorer(f1_score, average='weighted')
>>>
>>> score_functions = [f1_micro, f1_macro, f1_weighted]
>>>
>>> for score_func in score_functions:
>>> clf = GridSearchCV(SVC(C=1, cache_size=5000),
>>> tuned_parameters,
>>> scoring=score_func,
>>> verbose=1, n_jobs=1, cv=cv_method)
>>> clf.fit(X, y)
>>> ...
>>> ============================
>>>
>>> Josh
>>>
>>>
>>> On Thu, Jul 25, 2013 at 8:07 AM, Olivier Grisel <
>>> [email protected]> wrote:
>>>
>>>> 2013/7/25 Josh Wasserstein <[email protected]>:
>>>> > Thank you Olivier. I went through that paper and I agree, it looks
>>>> like
>>>> > implementing micro-AUC or macro-AUC should not be that hard. I will
>>>> try to
>>>> > implement within the next week. I have have never contributed to a
>>>> project
>>>> > in GitHub, so I am not sure to what extent my code would meet the
>>>> standards
>>>> > but I am happy to try.
>>>> >
>>>> > In the mean time, is there anything similar to an AUC metric that
>>>> scikit
>>>> > supports when working with GridSearchCV in a multi-label setting? I am
>>>> > looking for some compromise between precision and recall that
>>>> indirectly
>>>> > optimizes for the AUC score of each label .
>>>>
>>>> You can try the f1 score that is a balanced score (a tradeoff between
>>>> precision and recall) that is a reasonable score for imbalanced
>>>> multiclass dataset.
>>>>
>>>> It supports both micro and macro averaging.
>>>>
>>>>
>>>> --
>>>> Olivier
>>>> http://twitter.com/ogrisel - http://github.com/ogrisel
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> See everything from the browser to the database with AppDynamics
>>>> Get end-to-end visibility with application monitoring from AppDynamics
>>>> Isolate bottlenecks and diagnose root cause in seconds.
>>>> Start your free trial of AppDynamics Pro today!
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Scikit-learn-general mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>>>
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> See everything from the browser to the database with AppDynamics
>> Get end-to-end visibility with application monitoring from AppDynamics
>> Isolate bottlenecks and diagnose root cause in seconds.
>> Start your free trial of AppDynamics Pro today!
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Scikit-learn-general mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Scikit-learn-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
