I have added you as a contributor. On Fri, Apr 19, 2019 at 9:56 AM jincheng sun <sunjincheng...@gmail.com> wrote:
> Hi Lukasz, > > Thanks for your affirmation and provide more contextual information. :) > > Would you please give me the contributor permission? My JIRA ID is > sunjincheng121. > > I would like to create/assign tickets for this work. > > Thanks, > Jincheng > > Lukasz Cwik <lc...@google.com> 于2019年4月20日周六 上午12:26写道： > >> Since I don't think this is a contentious change. >> >> On Fri, Apr 19, 2019 at 9:25 AM Lukasz Cwik <lc...@google.com> wrote: >> >>> Yes, using T makes sense. >>> >>> The WindowedValue was meant to be a context object in the SDK harness >>> that propagates various information about the current element. We have >>> discussed in the past about: >>> * making optimizations which would pass around less of the context >>> information if we know that the DoFns don't need it (for example, all the >>> values share the same window). >>> * versioning the encoding separately from the WindowedValue context >>> object (see recent discussion about element timestamp precision [1]) >>> * the runner may want its own representation of a context object that >>> makes sense for it which isn't a WindowedValue necessarily. >>> >>> Feel free to cut a JIRA about this and start working on a change towards >>> this. >>> >>> 1: >>> https://lists.apache.org/thread.html/221b06e81bba335d0ea8d770212cc7ee047dba65bec7978368a51473@%3Cdev.beam.apache.org%3E >>> >>> On Fri, Apr 19, 2019 at 3:18 AM jincheng sun <sunjincheng...@gmail.com> >>> wrote: >>> >>>> Hi Beam devs, >>>> >>>> I read some of the docs about `Communicating over the Fn API` in Beam. >>>> I feel that Beam has a very good design for Control Plane/Data Plane/State >>>> Plane/Logging services, and it is described in <How to send and receive >>>> data> document. When communicating between Runner and SDK Harness, the >>>> DataPlane API will be WindowedValue(An immutable triple of value, >>>> timestamp, and windows.) As a contract object between Runner and SDK >>>> Harness. I see the interface definitions for sending and receiving data in >>>> the code as follows: >>>> >>>> - org.apache.beam.runners.fnexecution.data.FnDataService >>>> >>>> public interface FnDataService { >>>>> <T> InboundDataClient receive(LogicalEndpoint inputLocation, >>>>> Coder<WindowedValue<T>> coder, FnDataReceiver<WindowedValue<T>> listener); >>>>> <T> CloseableFnDataReceiver<WindowedValue<T>> send( >>>>> LogicalEndpoint outputLocation, Coder<WindowedValue<T>> coder); >>>>> } >>>> >>>> >>>> >>>> - org.apache.beam.fn.harness.data.BeamFnDataClient >>>> >>>> public interface BeamFnDataClient { >>>>> <T> InboundDataClient receive(ApiServiceDescriptor >>>>> apiServiceDescriptor, LogicalEndpoint inputLocation, >>>>> Coder<WindowedValue<T>> coder, FnDataReceiver<WindowedValue<T>> receiver); >>>>> <T> CloseableFnDataReceiver<WindowedValue<T>> >>>>> send(BeamFnDataGrpcClient Endpoints.ApiServiceDescriptor >>>>> apiServiceDescriptor, LogicalEndpoint outputLocation, >>>>> Coder<WindowedValue<T>> coder); >>>>> } >>>> >>>> >>>> Both `Coder<WindowedValue<T>>` and `FnDataReceiver<WindowedValue<T>>` >>>> use `WindowedValue` as the data structure that both sides of Runner and SDK >>>> Harness know each other. Control Plane/Data Plane/State Plane/Logging is a >>>> highly abstraction, such as Control Plane and Logging, these are common >>>> requirements for all multi-language platforms. For example, the Flink >>>> community is also discussing how to support Python UDF, as well as how to >>>> deal with docker environment. how to data transfer, how to state access, >>>> how to logging etc. If Beam can further abstract these service interfaces, >>>> i.e., interface definitions are compatible with multiple engines, and >>>> finally provided to other projects in the form of class libraries, it >>>> definitely will help other platforms that want to support multiple >>>> languages. So could beam can further abstract the interface definition of >>>> FnDataService's BeamFnDataClient? Here I am to throw out a minnow to catch >>>> a whale, take the FnDataService#receive interface as an example, and turn >>>> `WindowedValue<T>` into `T` so that other platforms can be extended >>>> arbitrarily, as follows: >>>> >>>> <T> InboundDataClient receive(LogicalEndpoint inputLocation, Coder<T> >>>> coder, FnDataReceiver<T>> listener); >>>> >>>> What do you think? >>>> >>>> Feel free to correct me if there any incorrect understanding. And >>>> welcome any feedback! >>>> >>>> >>>> Regards, >>>> Jincheng >>>> >>>
