On Wed, Nov 12, 2014 at 2:04 PM, Ted Dunning <[email protected]> wrote:
> On Wed, Nov 12, 2014 at 9:53 AM, Dmitriy Lyubimov <[email protected]> > wrote: > > > once we start mapping aggregate, there's no reason not to > > map other engine specific capabilities, which are vast. At this point > > dilemma is, no matter what we do we are losing coherency: if we map it > all, > > then other engines will have trouble supporting all of it. If we don't > map > > it all, then we are forcing capability reduction compared to what the > > engine actually can do. > > > > It is obvious to me that all-reduce aggregate will make a lot of sense -- > > even if it means math checkpoint. but then where do we stop in mapping > > those. E.g. do we do fold? cartesian? And what is that true reason we are > > remapping everything if it is already natively available? etc. etc. For > > myself, I still haven't figured a good answer to those . > > > > Actually, I disagree with the premise here. > > There *is* a reason not to map all other engine specific capabilities. > That reason is we don't need them. Yet. > First, i am, sort of, advocating the same thing. I am not in favor of mapping anything beyond map. I am in favor of going to native capabilities directly. Second, actually "we don't need them" is not a good argument because if the goal is to build an ML environment offering and not just a collection of ML methodologies, then we must assume 3rd parties will (and in fact, do, on my own behalf) build much more than there is ever planned or done in Mahout. And such 3rd parties can attest to the fact that the issue of all-reduce aggregation is indeed coming up every second full moon.
