On Wed, Nov 12, 2014 at 9:53 AM, Dmitriy Lyubimov <[email protected]> wrote:
> once we start mapping aggregate, there's no reason not to > map other engine specific capabilities, which are vast. At this point > dilemma is, no matter what we do we are losing coherency: if we map it all, > then other engines will have trouble supporting all of it. If we don't map > it all, then we are forcing capability reduction compared to what the > engine actually can do. > > It is obvious to me that all-reduce aggregate will make a lot of sense -- > even if it means math checkpoint. but then where do we stop in mapping > those. E.g. do we do fold? cartesian? And what is that true reason we are > remapping everything if it is already natively available? etc. etc. For > myself, I still haven't figured a good answer to those . > Actually, I disagree with the premise here. There *is* a reason not to map all other engine specific capabilities. That reason is we don't need them. Yet. So far, we *clearly* need some sort of block aggregate for a host of hog-wild sorts of algorithms. That doesn't imply that we need all kinds of mapping aggregates. It just means that we are clear on one need for now. So let's get this one in and see how far we can go. Also, having one kind of aggregation in the DSL does not restrict anyone from using engine specific capabilities. It just means that one kind of idiom can be done without engine specificity.
