Hi CPC, we had several requests in the past to add this features. However, adding select/split for DataSet is much! more work than you would expect. As you pointed out, we have to go through the optimizer, which assumes that the outputs of a function are replicated. This is pretty much wired in and you would have to touch a lot of code.
I'm sorry, but am not comfortable doing such a big change. IMO, the potential gains are not worth the effort of implementation and verification and the risk of breaking something. Best, Fabian 2017-03-02 16:31 GMT+01:00 CPC <acha...@gmail.com>: > Hi all, > > We will try to implement select/split functionality for batch api. We > looked at streaming side and understand how it works but since streaming > side does not include an optimizer it was easier. Since adding such a > runtime operator will affect optimizer layer as well, is there a part that > you want us to pay particular attention to? > > Thanks... >
