You can set the platform Attributes & operators properties either from the property file or from the application. ( dag.setAttribute.., dag.setInputPortAttribute, dag.setOutputPortAttribute ) StreamCodec can be used to - Route the tuples flowing through the streams and also to change the serializer used for the tuples, ( default is Kryo )
On Fri, Jun 10, 2016 at 2:40 PM Junguk Cho <[email protected]> wrote: > Hi. > > Thank you for replies. > > In Storm, storm APIs provides several grouping methods( > http://storm.apache.org/releases/current/Concepts.html). > For example, app developers specify the number of replica and routing > rules like shuffleGrouping and fieldsGrouping. > > builder.setSpout("spout", new RandomSentenceSpout(), 5); > builder.setBolt("split", new SplitSentence(), 8).shuffleGrouping("spout"); > builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new > Fields("word")); > > In apex, we can set the number of replica in property files and for > partitions except for "sticky" we can use "Partition" and StreamCodec. > > Is my understanding correct? Thanks in advance. Junguk > > > > > 2016-06-10 16:16 GMT-04:00 Munagala Ramanath <[email protected]>: > >> There is no mechanism for automatic load balancing across the board; some >> individual operators >> support it (e.g. Kafka input operator) out of the box. You can implement >> it in your own operator by using a >> custom partitioner. >> >> Keys are sticky by default since the hashcode is used to distribute >> tuples to partitions. There is >> no special setting for this. You can gain more control over how the >> distribution happens by >> using a StreamCodec. >> >> Ram >> >> On Fri, Jun 10, 2016 at 11:36 AM, Junguk Cho <[email protected]> wrote: >> >>> Hi, Ram. >>> >>> Thank you for reply. >>> I have following question. >>> >>> 1. Where do we specify policy to use "Load balance" or "Sticky Key"? >>> >>> 2. This is just general question. >>> For example, I would like to run A class now as one operator and then >>> later I need other tasks which I do not think. >>> At that time, could I load this new class in current DAG even though it >>> is not included in *.apa file. >>> >>> >>> Thanks, >>> Junguk >>> >>> >>> 2016-06-10 12:03 GMT-04:00 Munagala Ramanath <[email protected]>: >>> >>>> You don't need dynamic partitioning to achieve that topology. You can >>>> simply create your DAG as: A --> X --> Y and then set the *PARTITIONER* >>>> attribute on X >>>> as discussed in the "Advanced Features" section of the TopN words >>>> tutorial at: >>>> http://docs.datatorrent.com/tutorials/topnwords-c7/ >>>> >>>> The stanza in your properties file should look something like this >>>> (with suitable changes >>>> for the application and operator names): >>>> >>>> *<property>* >>>> * >>>> <name>dt.application.TopNWordsWithQueries.operator.wordReader.attr.PARTITIONER</name>* >>>> * >>>> <value>com.datatorrent.common.partitioner.StatelessPartitioner:2</value>* >>>> *</property>* >>>> >>>> The platform will then create two replicas of X. >>>> >>>> For the rest, could you provide additional details of what you are >>>> trying to accomplish, what >>>> you tried and what sort of problems you encountered ? That will help us >>>> provide more precise answers. >>>> >>>> Ram >>>> >>>> On Fri, Jun 10, 2016 at 8:37 AM, Junguk Cho <[email protected]> wrote: >>>> >>>>> Hi. >>>>> >>>>> I explore "Dynamic Application Modification" with this example( >>>>> https://github.com/DataTorrent/examples/tree/master/tutorials/dynamic-partition >>>>> ). >>>>> >>>>> >>>>> To be clear, basically, application developers carefully design a >>>>> program. >>>>> For Dynamic Partitioning, in "dynamic partition" example, it uses >>>>> threadhold to adjust partition. In addition, to use Modification of DAG >>>>> structure, app developers should make program like java classes to use >>>>> them >>>>> later. >>>>> Do I interpret that they are determined "compiling time"? >>>>> >>>>> How do I use "Load balance and Sticky Key" in Apex? >>>>> I would like to create this topology for word counting. >>>>> For example, >>>>> B operator >>>>> A operator -> -> one >>>>> onperator >>>>> C Operator >>>>> >>>>> A sends the same values like "word" to the same operator ( B or C )? >>>>> Can we specify this rule in code and how to make two process for B and >>>>> C with one class? >>>>> >>>>> Thanks, >>>>> Junguk >>>>> >>>> >>>> >>> >> >
