Not sure why you address this to me. I agree with most of your statements.
I think Ted’s intent was to find a simple consensus statement that addresses
where the project is going in a general way. I look at it as something to
communicate to the outside world. Why? We are rejecting new mapreduce code.
This was announced as a project-wide rule and has already been used to reject
one contribution I know of. OK, what replaces Hadoop mapreduce? What therefore
should contributors look to as a model if not Hadoop mapreduce? Do we give no
advice or comment on this question?
For example, I’m doing drivers that read and write text files. This is quite
tightly coupled to Spark. Possible contributors should know that this is OK,
that it will not be rejected and is indeed where most of the engine specific
work is being done by committers. You are right, most of us know what we are
doing, but simply to say “no more mapreduce” without offering an alternative
isn’t quite fair to everyone else.
You are abstracting your code away from a specific engine, and that is great, but in
practice anyone running it currently must run Spark. This also needs to be
communicated. It’s as practical as answering, “What do I need to install to make
Mahout 1.0-snapshot work?"
On May 15, 2014, at 7:17 AM, Dmitriy Lyubimov <dlie...@gmail.com> wrote:
Pat, it can't be as high-level or as dteailed as it can be, I don't care,
as long as it doesn't contain misstatements. It simply can state "we adhere
to the "Apache's power of doing" principle and accept new contributions".
This is ok with me. But, as offered, it does try to enumerate strategic
directions, and in doing so, its wording is either vague, or incomplete, or
just wrong.
For example, it says "it is clear that what the committers are working on
is Spark". This is less than accurate.
First, if I interpret it literally, it is wrong, as our committers for most
part are not working on Spark, and even if they do, to whatever negligible
degree it esxists, why Mahout would care.
Second, if it is meant to say "we develop algorithms for Spark", this is
also wrong, because whatever algorithms we have added to day, have 0 Spark
dependencies.
Third, if it is meant to say that majority of what we are working on is
Spark bindings, this is still incorrect. Head count-wise, Mahout-math
tweaks and Scala enablement were at least a big effort. Hadoop 2.0 stuff
was at least as big. Documentation and tutorial work engagement was
absolute leader headcount-wise to date.
The problem i am trying to explain here is that we obviously internally
know what we are doing; but this is for external consumption so we have to
be careful to avoid miscommunication here. It is easy for us to pass on
less than accurate info delivery exactly because we already know what we
are doing and therefore our brain is happy to jump to conclusions and make
up the missing connections between stated and implied as we see it. But for
an outsider, this would sound vague or make him make wrong connections.
On Wed, May 7, 2014 at 9:54 AM, Pat Ferrel <pat.fer...@gmail.com> wrote:
This doesn’t seem to be a vision statement. I was +1 to a simple consensus
statement.
The vision is up to you.
We have an interactive shell that scales to huge datasets without
resorting to massive subsampling. One that allows you to deal with the
exact data your black box algos work on. Every data tool has an interactive
mode except Mahout—now it does. Virtually every complex transform as well
as basic linear algebra works on massive datasets. The interactivity will
allow people to do things with Mahout they could never do before.
We also have the building blocks to make the fastest most flexible cutting
edge collaborative filtering+metadata recommenders in the world. Honestly I
don’t see anything like this elsewhere. We will also be able to fit into
virtually any workflow and directly consume data produced in those systems
with no intermediate scrubbing. This has never happened before in Mahout
and I don’t see it in MLlib either. Even the interactive shell will benefit
from this.
Other feature champions will be able to add to this list.
Seems like the vision comes from feature champions. I may not use Mahout
in the same way you do but I rely on your code. Maybe I serve a different
user type than you. I don’t see a problem with that, do you?
On May 6, 2014, at 2:32 PM, Dmitriy Lyubimov <dlie...@gmail.com> wrote:
Pat et. al,
The whole problem with original suggested consensus statement is that it
reads as "we are building MLLib for Spark (oh wait, there's already such a
thing)" and then "we are building MLLib for 0xdata" and then perhaps for
something else. Which can't be farther from the true philosophy of what has
been done. If not it, then at best it reads as "we don't know what it is we
are building, but we are including some Spark dependencies now". So it is
either misleading, or sufficiently vague, not sure which is worse.
If a collection of backend-specific separated MLLibs is the new consensus,
i can't say i can share it. In fact, the only motivation for me to do
anything within this project was to fix everything that (per my perhaps
lopsided perception) is less than ideal with the approach of building ML
projects as backend-specific collections of black-box trainers and solvers
and bring in an ideology similar to Julia and R to the jvm-based big data
ML .
If users are to love us, somehow i think it will not be because we ported
yet another flavor of K-means to Spark.
At this point I think it is a little premature to talk about an existing
consensus, it seems.
On Tue, May 6, 2014 at 12:41 PM, Pat Ferrel <p...@occamsmachete.com> wrote:
+1
I personally won’t spend a lot of time generalizing right now.
Contributors can help with that if they want or make suggestions.
On May 6, 2014, at 9:23 AM, Ted Dunning <ted.dunn...@gmail.com> wrote:
As a bit of commentary, it is clear that what the committers are working
on
is Spark
Mahout committers, with very rare exceptions, are not working on Spark.
Spark committers and contributors are working on Spark.
