Hey Sandesh, I like your idea, but here is what I think. API Stability is VERY important! The API we have to build a dag has been used for a long time. If the new change doesn't help a lot, I won't recommend to change the API
so +0 :) On Mon, Apr 11, 2016 at 12:02 PM, Sandesh Hegde <[email protected]> wrote: > Hello Chinmay, > > Idea is to follow natural order. Suppose you want to set attribute on a > port, you will have to write something like below > > dag.setInputPortAttribute(deserializeJSON.input, > Context.PortContext.PARTITION_PARALLEL, true); > > How about setting it directly on the ports? > deserializeJSON.input.setAttribute(Context.PortContext.PARTITION_PARALLEL, > true)); > > Same thing goes for attributes of the Operators. > > These new set of APIs will support all the features of our > existing API, but in a more natural way. > It is not similar to High-Level API, it is just moonlighting idea about our > existing API. > > Thanks > > > > > > > > On Sun, Apr 10, 2016 at 11:20 PM Chinmay Kolhatkar <[email protected]> > wrote: > > > I feel that adding all operators to dag makes it explicit for a developer > > who is looking at the code for the first time. > > > > Besides, one might want to do dag.getMeta on operators and ports to set > > some attributes. > > I'm not sure how much explicit will that be if we bypass adding operators > > to the DAG. > > > > Though, I totally agree with the intention of simplifying the APIs for > > which the proposal from Siyuan in another mail is a nice first step. > > > > Regards, > > Chinmay. > > > > > > > > On Mon, Apr 11, 2016 at 10:48 AM, Sandesh Hegde <[email protected] > > > > wrote: > > > > > Yes, that is the idea. > > > > > > On Sun, Apr 10, 2016, 9:28 PM Sandeep Deshmukh < > [email protected]> > > > wrote: > > > > > > > Are you suggesting we add only the input adapters to the dag and then > > all > > > > the connected ones are added by simply traversing the DAG? > > > > > > > > Regards, > > > > Sandeep > > > > > > > > On Fri, Apr 8, 2016 at 11:06 PM, Sandesh Hegde < > > [email protected]> > > > > wrote: > > > > > > > > > Hello Team, > > > > > > > > > > As we were talking about High Level api in another thread, here is > a > > > > small > > > > > variation proposal for our existing apis, > > > > > > > > > > Take look at the current code and the proposed code > > > > > > > > > > Current Style > > > > > --------------------------- > > > > > KafkaSinglePortStringInputOperator kafkaInput = > > > > > dag.addOperator("kafkaInput", new > > > KafkaSinglePortStringInputOperator()); > > > > > DeserializeJSON deserializeJSON = > dag.addOperator("deserialize", > > > new > > > > > DeserializeJSON()); > > > > > RedisJoin redisJoin = dag.addOperator("redisJoin", new > > > RedisJoin()); > > > > > CampaignProcessorWithApexWindow campaignProcessor = > > > > > dag.addOperator("campaignProcessor", new > > > > > CampaignProcessorWithApexWindow()); > > > > > > > > > > // Connect the Ports in the Operators > > > > > dag.addStream("deserialize", kafkaInput.outputPort, > > > > > deserializeJSON.input) ; > > > > > dag.addStream("redisJoin", filterFields.output, > redisJoin.input) > > ; > > > > > dag.addStream("output", redisJoin.output, > > campaignProcessor.input); > > > > > ------------------------------------- > > > > > > > > > > > > > > > Proposed Change ( Just create Operators and connect the output port > > to > > > > > input ports and then add the input operators to DAG ) > > > > > ------------------------------------------- > > > > > KafkaSinglePortStringInputOperator > > > > kafkaSinglePortStringInputOperator = > > > > > new KafkaSinglePortStringInputOperator() ; > > > > > DeserializeJSON deserializeJSON = new DeserializeJSON(); > > > > > RedisJoin redisJoin = new RedisJoin() ; > > > > > CampaignProcessor campaignProcessor = new CampaignProcessor() > ; > > > > > > > > > > kafkaInput.outputPort.connect(deserializeJSON.input); > > > > > filterFields.output.connect(redisJoin.input) ; > > > > > redisJoin.output.connect(campaignProcessor.input) ; > > > > > > > > > > dag.add(kafkaInput); > > > > > ----------------------------------------- > > > > > > > > > > Things to note : Name can be generated internally or user can > > specify a > > > > > name property of the operator ( like Dag.AddOperator("name", ...) ) > > > > > > > > > > Let me know your thoughts. > > > > > > > > > > Thanks > > > > > > > > > > > > > > >
