On Tue, Dec 27, 2011 at 3:24 PM, Tom Pierce<t...@cloudera.com> wrote:
...
They discover Mahout, which does specifically bill itself as scalable
(from http://mahout.apache.org, in some of the largest letters: "What
is Apache Mahout? The Apache Mahout™ machine learning library's goal
is to build scalable machine learning libraries."). They sniff check
it by massaging some moderately-sized data set into the same format as
an example from the wiki and they fail to get a result - often because
their problem has some very different properties (more classes, much
larger feature space, etc.) and the implementation has some limitation
that they trip over.
I have worked with users of Mahout who had 10^9 possible features and
others who are classifying
into 60,000 categories.
Neither of these implementations uses Naive Bayes. Both work very well.
They will usually try one of the simplest methods available under the
assumption "well, if this doesn't scale well, the more complex methods
are surely no better".
Silly assumption.
This may not be entirely fair, but since the
docs they're encountering on the main website and wiki don't warn them
that certain implementations don't necessarily scale in different
ways, it's certainly not unreasonable.
Well, it is actually silly.
Clearly the docs can be better. Clearly the code quality can be better
especially in terms of nuking capabilities that have not found an audience.
But clearly also just trying one technique without asking anybody what the
limitations are isn't going to work as an evaluation technique. This is
exactly analogous to somebody finding that a matrix in R doesn't do what a
data frame is supposed to do. It doesn't and you aren't going to find out
why or how from the documentation very quickly.
In both cases of investigating Mahout or investigating R you will find out
plenty if you ask somebody who knows what they are talking about.
They're at best going to
conclude the scalability will be hit-and-miss when a simple method
doesn't work. Perhaps they'll check in again in 6-12 months.
Maybe so. Maybe not. I have little sympathy with people who make
scatter-shot decisions like this.
...
I see your analogy to R or sciPy - and I don't disagree. But those
projects do not put scaling front and center; if Mahout is going to
keep scalability as a "headline feature" (which I would like to see!),
I think prominently acknowledging how different methods fail to scale
would really help its credibility. For what it's worth, of the people
I know who've tried Mahout 100% of them were using R and/or sciPy
already, but were curious about Mahout specifically for better
scalability.
Did they ask on the mailing list?
I'm not sure where this information is best placed - it would be great
to see it on the Wiki along with the examples, at least.
Sounds OK. Maybe we should put it in the book.
(oh... wait, we already did that)
It would be
awesome to see warnings at runtime ("Warning: You just trained a model
that you cannot load without at least 20GB of RAM"), but I'm not sure
how realistic that is.
I think it is fine that loading the model fails with a fine error message
but putting yellow warning tape all over the user's keyboard isn't going to
help anything.
I would like it to be easier to determine, at some very high level, why
something didn't work when an experiment fails. Ideally, without having to
dive into the code at all.
How about you ask an expert?
That really is easier. It helps the community to hear about what other
people need and it helps the new user to hear what other people have done.
