I am sure the project needs to refactor and unify the Hadoop-related code. There's a lot of copy and paste at this stage. That would go some way towards abstracting away Hadoop -- would tend to centralize the dependency.
I think there's a lot more to it -- abstracting away contacting a cluster? running a job? storing and reading data? Then you're also learning how to configure Mahout's layer, as well as your underlying infrastructure. My gut says it's hard, compared to the value it could add. Given that Hadoop is the de facto standard and big clouds like Amazon directly support it, it seems unlikely someone would not be able to use Hadoop. It's all just my guess given my impressions... My meta-concern is that we don't really have a polished, finished approach to using even Hadoop (which is again to be expected given it's early, and given Hadoop is evolving fast too) -- so would rather focus on tying up loose ends, or documenting and testing, before reaching too much farther. On Mon, Sep 7, 2009 at 9:02 AM, Lukáš Vlček<lukas.vl...@gmail.com> wrote: > May be there is no direct equivalent but there are many ways how one can > build MapReduce architecture into existing system without Hadoop. And there > is something all these systems have in common at high level. I can see many > existing systems are adding MapReduce paradigm into their stack (e.g.: > Aster, GigaSpaces, ... to name a few). Do you think it would be too dificult > or impractical at this point to target clean design of algorithms in Mahout > and make then pure MapReduce as opposed to coupled with Hadoop? MapReduce > API can be just set of few interfaces (and I think there are already such > interfaces in Hadoop but I don't think you can get then as a separated JAR). > The rest of the Hadoop dependencies (like using HDFS) can be abstracted > later if needed. > Think of a developer who would like to use Mahout but can not use Hadoop. > For such developer it would be "just" a matter of adapting Mahout to his/her > proprietar MapReduce system. I am not saying Mahout should have this > capability now but would be a nice goal. > > Regards, > Lukas
