On Fri, May 3, 2013 at 4:33 PM, Andreas Mueller <[email protected]>wrote:
>
> Both. But multiclass logistic regression only supports l1 / l2 (group
> lasso) penalty. If you want to use l2 or l1 penalty, you need to use binary
> logistic regression + OVR.
>
> Is there any problem with having a pure l2 penalty or did you just not
> implement it?
>
I just didn't implement it.
Some authors have argued that, compared to indirect formulations such as
OVR, direct multiclass formulations are not worth the additional complexity
(see e.g. "In Defense of one-vs-all classification", by Rifkin). I agree
with that point of view but only in the case of l1 (lasso) or l2 (ridge)
penalties. Indeed, in practice there's no big difference in terms of
accuracy, and, using OVR, each binary classification problem is
independent, hence OVR is embarrasingly parallel with respect to the
classes. With l1 / l2 penalty (group lasso), the situation is different.
Even if you use OVR, you *must* solve all binary classification problems
simultaneously (since the goal is to force the weights corresponding to a
feature to zero across all classes). Hence, it is not embarrassingly
parallel anymore (in that case, OVR can be seen as a multi-task
classification problem). So, with l1 / l2 penalty, it is worth using a
direct multiclass formulation instead of OVR, in my opinion.
>
> I hope I find the time to study your code, it sounds really great!
> You said this was related to a paper? When can we have a read ;) (also
> which conference ?)
>
Yes my paper "Block Coordinate Descent Algorithms for Large-scale Sparse
Multiclass Classification." (accepted to "Machine Learning" journal)
proposes a more efficient direct multiclass formulation than multiclass
logistic regression (and I combine it with l1 / l2 group lasso penalty).
The camera ready paper was sent to the journal and should be online soon (I
think I sent you a draft a few months ago, though).
Mathieu
------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite
It's a free troubleshooting tool designed for production
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://p.sf.net/sfu/appdyn_d2d_ap2
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
