Hi, You can create your stream codec to define custom routing. Default is sticky key partitioning.
See this example Codec3.java here: https://github.com/DataTorrent/examples/tree/master/tutorials/partition/src/main/java/com/example/myapexapp For load balancing, the getPartition method can keep returning incremental value for every call for partition. So, it will become round robin instead of sticky key. In case of sticky key, it returns the hashCode of the object. More documentation here: https://apex.apache.org/docs/apex/application_development/#partitioning Regards, Ashwin. On Fri, Jun 10, 2016 at 2:39 PM, Junguk Cho <[email protected]> wrote: > Hi. > > Thank you for replies. > > In Storm, storm APIs provides several grouping methods( > http://storm.apache.org/releases/current/Concepts.html). > For example, app developers specify the number of replica and routing > rules like shuffleGrouping and fieldsGrouping. > > builder.setSpout("spout", new RandomSentenceSpout(), 5); > builder.setBolt("split", new SplitSentence(), 8).shuffleGrouping("spout"); > builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new > Fields("word")); > > In apex, we can set the number of replica in property files and for > partitions except for "sticky" we can use "Partition" and StreamCodec. > > Is my understanding correct? Thanks in advance. Junguk > > > > > 2016-06-10 16:16 GMT-04:00 Munagala Ramanath <[email protected]>: > >> There is no mechanism for automatic load balancing across the board; some >> individual operators >> support it (e.g. Kafka input operator) out of the box. You can implement >> it in your own operator by using a >> custom partitioner. >> >> Keys are sticky by default since the hashcode is used to distribute >> tuples to partitions. There is >> no special setting for this. You can gain more control over how the >> distribution happens by >> using a StreamCodec. >> >> Ram >> >> On Fri, Jun 10, 2016 at 11:36 AM, Junguk Cho <[email protected]> wrote: >> >>> Hi, Ram. >>> >>> Thank you for reply. >>> I have following question. >>> >>> 1. Where do we specify policy to use "Load balance" or "Sticky Key"? >>> >>> 2. This is just general question. >>> For example, I would like to run A class now as one operator and then >>> later I need other tasks which I do not think. >>> At that time, could I load this new class in current DAG even though it >>> is not included in *.apa file. >>> >>> >>> Thanks, >>> Junguk >>> >>> >>> 2016-06-10 12:03 GMT-04:00 Munagala Ramanath <[email protected]>: >>> >>>> You don't need dynamic partitioning to achieve that topology. You can >>>> simply create your DAG as: A --> X --> Y and then set the *PARTITIONER* >>>> attribute on X >>>> as discussed in the "Advanced Features" section of the TopN words >>>> tutorial at: >>>> http://docs.datatorrent.com/tutorials/topnwords-c7/ >>>> >>>> The stanza in your properties file should look something like this >>>> (with suitable changes >>>> for the application and operator names): >>>> >>>> *<property>* >>>> * >>>> <name>dt.application.TopNWordsWithQueries.operator.wordReader.attr.PARTITIONER</name>* >>>> * >>>> <value>com.datatorrent.common.partitioner.StatelessPartitioner:2</value>* >>>> *</property>* >>>> >>>> The platform will then create two replicas of X. >>>> >>>> For the rest, could you provide additional details of what you are >>>> trying to accomplish, what >>>> you tried and what sort of problems you encountered ? That will help us >>>> provide more precise answers. >>>> >>>> Ram >>>> >>>> On Fri, Jun 10, 2016 at 8:37 AM, Junguk Cho <[email protected]> wrote: >>>> >>>>> Hi. >>>>> >>>>> I explore "Dynamic Application Modification" with this example( >>>>> https://github.com/DataTorrent/examples/tree/master/tutorials/dynamic-partition >>>>> ). >>>>> >>>>> >>>>> To be clear, basically, application developers carefully design a >>>>> program. >>>>> For Dynamic Partitioning, in "dynamic partition" example, it uses >>>>> threadhold to adjust partition. In addition, to use Modification of DAG >>>>> structure, app developers should make program like java classes to use >>>>> them >>>>> later. >>>>> Do I interpret that they are determined "compiling time"? >>>>> >>>>> How do I use "Load balance and Sticky Key" in Apex? >>>>> I would like to create this topology for word counting. >>>>> For example, >>>>> B operator >>>>> A operator -> -> one >>>>> onperator >>>>> C Operator >>>>> >>>>> A sends the same values like "word" to the same operator ( B or C )? >>>>> Can we specify this rule in code and how to make two process for B and >>>>> C with one class? >>>>> >>>>> Thanks, >>>>> Junguk >>>>> >>>> >>>> >>> >> > -- Regards, Ashwin.
