Ted, Can we easily integrate t-digest for descriptives once we have block aggregates? This might count one more reason.
Gokhan On Thu, Nov 13, 2014 at 12:04 AM, Ted Dunning <ted.dunn...@gmail.com> wrote: > On Wed, Nov 12, 2014 at 9:53 AM, Dmitriy Lyubimov <dlie...@gmail.com> > wrote: > > > once we start mapping aggregate, there's no reason not to > > map other engine specific capabilities, which are vast. At this point > > dilemma is, no matter what we do we are losing coherency: if we map it > all, > > then other engines will have trouble supporting all of it. If we don't > map > > it all, then we are forcing capability reduction compared to what the > > engine actually can do. > > > > It is obvious to me that all-reduce aggregate will make a lot of sense -- > > even if it means math checkpoint. but then where do we stop in mapping > > those. E.g. do we do fold? cartesian? And what is that true reason we are > > remapping everything if it is already natively available? etc. etc. For > > myself, I still haven't figured a good answer to those . > > > > Actually, I disagree with the premise here. > > There *is* a reason not to map all other engine specific capabilities. > That reason is we don't need them. Yet. > > So far, we *clearly* need some sort of block aggregate for a host of > hog-wild sorts of algorithms. That doesn't imply that we need all kinds of > mapping aggregates. It just means that we are clear on one need for now. > > So let's get this one in and see how far we can go. > > Also, having one kind of aggregation in the DSL does not restrict anyone > from using engine specific capabilities. It just means that one kind of > idiom can be done without engine specificity. >
