Hi. Thank you for replies.
In Storm, storm APIs provides several grouping methods( http://storm.apache.org/releases/current/Concepts.html). For example, app developers specify the number of replica and routing rules like shuffleGrouping and fieldsGrouping. builder.setSpout("spout", new RandomSentenceSpout(), 5); builder.setBolt("split", new SplitSentence(), 8).shuffleGrouping("spout"); builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word")); In apex, we can set the number of replica in property files and for partitions except for "sticky" we can use "Partition" and StreamCodec. Is my understanding correct? Thanks in advance. Junguk 2016-06-10 16:16 GMT-04:00 Munagala Ramanath <[email protected]>: > There is no mechanism for automatic load balancing across the board; some > individual operators > support it (e.g. Kafka input operator) out of the box. You can implement > it in your own operator by using a > custom partitioner. > > Keys are sticky by default since the hashcode is used to distribute tuples > to partitions. There is > no special setting for this. You can gain more control over how the > distribution happens by > using a StreamCodec. > > Ram > > On Fri, Jun 10, 2016 at 11:36 AM, Junguk Cho <[email protected]> wrote: > >> Hi, Ram. >> >> Thank you for reply. >> I have following question. >> >> 1. Where do we specify policy to use "Load balance" or "Sticky Key"? >> >> 2. This is just general question. >> For example, I would like to run A class now as one operator and then >> later I need other tasks which I do not think. >> At that time, could I load this new class in current DAG even though it >> is not included in *.apa file. >> >> >> Thanks, >> Junguk >> >> >> 2016-06-10 12:03 GMT-04:00 Munagala Ramanath <[email protected]>: >> >>> You don't need dynamic partitioning to achieve that topology. You can >>> simply create your DAG as: A --> X --> Y and then set the *PARTITIONER* >>> attribute on X >>> as discussed in the "Advanced Features" section of the TopN words >>> tutorial at: >>> http://docs.datatorrent.com/tutorials/topnwords-c7/ >>> >>> The stanza in your properties file should look something like this (with >>> suitable changes >>> for the application and operator names): >>> >>> *<property>* >>> * >>> <name>dt.application.TopNWordsWithQueries.operator.wordReader.attr.PARTITIONER</name>* >>> * >>> <value>com.datatorrent.common.partitioner.StatelessPartitioner:2</value>* >>> *</property>* >>> >>> The platform will then create two replicas of X. >>> >>> For the rest, could you provide additional details of what you are >>> trying to accomplish, what >>> you tried and what sort of problems you encountered ? That will help us >>> provide more precise answers. >>> >>> Ram >>> >>> On Fri, Jun 10, 2016 at 8:37 AM, Junguk Cho <[email protected]> wrote: >>> >>>> Hi. >>>> >>>> I explore "Dynamic Application Modification" with this example( >>>> https://github.com/DataTorrent/examples/tree/master/tutorials/dynamic-partition >>>> ). >>>> >>>> >>>> To be clear, basically, application developers carefully design a >>>> program. >>>> For Dynamic Partitioning, in "dynamic partition" example, it uses >>>> threadhold to adjust partition. In addition, to use Modification of DAG >>>> structure, app developers should make program like java classes to use them >>>> later. >>>> Do I interpret that they are determined "compiling time"? >>>> >>>> How do I use "Load balance and Sticky Key" in Apex? >>>> I would like to create this topology for word counting. >>>> For example, >>>> B operator >>>> A operator -> -> one >>>> onperator >>>> C Operator >>>> >>>> A sends the same values like "word" to the same operator ( B or C )? >>>> Can we specify this rule in code and how to make two process for B and >>>> C with one class? >>>> >>>> Thanks, >>>> Junguk >>>> >>> >>> >> >
