If you use localOrShuffleGrouping, the tuple will stay in process if possible. The solution, assuming your bolts are distributed, is to use shuffleGrouping to send tuples from spout->bolt so that the tuples are distributed to all of your workers, then use localOrShuffleGrouping from bolt->bolt to reduce inter-worker communication overhead.
On Thu, Jun 9, 2016 at 3:00 PM, Kazansky, Michael < [email protected]> wrote: > We are using localOrShuffleGrouping > > > > Thanks, > > Michael Kazansky > > > > *From:* Satish Duggana [mailto:[email protected]] > *Sent:* Thursday, June 09, 2016 1:54 PM > *To:* [email protected] > *Subject:* Re: not all the servers in storm cluster processing messages > > > > Hi Michael, > > Kafka spout is given parallelism-hint as 1 and the topic has only one > partition. So, only one kafka spout task is run in one of the workers of > the topology. Are you asking that bolts connected directly/indirectly from > that kafka spout are all executed in the same worker as kafka spout? If so, > What are the groupings and parallellism-hint that you have configured on > those bolts? > > > > Thanks, > > Satish. > > > > > > On Thu, Jun 9, 2016 at 10:41 PM, Kazansky, Michael < > [email protected]> wrote: > > Hi All, > > We have 3 servers in storm cluster. To prevent the order of the messages > which come to Kafka the Kafka partition was set to 1. KafkaSpout > parallelism hint is set to 1 as well. When messages flow through the Storm > we noticed that only one server in the cluster is busy processing messages > other 2 are idle although the number of workers is set to 3 in Config. Is > it excepted behavior or something is missing? > > If it is expected behavior is there any way to have all 3 servers in the > cluster processing messages? > > Storm version is 0.9.5 > > Thanks, > > Michael Kazansky > > > > This communication is for informational purposes only. It is not intended > as an offer or solicitation for the purchase or sale of any financial > instrument or as an official confirmation of any transaction. All market > prices, data and other information are not warranted as to completeness or > accuracy and are subject to change without notice. Any comments or > statements made herein do not necessarily reflect those of JPMorgan Chase & > Co., its subsidiaries and affiliates (collectively, "JPMC"). This > transmission may contain information that is proprietary, privileged, > confidential and/or exempt from disclosure under applicable law. If you are > not the intended recipient, you are hereby notified that any disclosure, > copying, distribution, or use of the information contained herein > (including any reliance thereon) is STRICTLY PROHIBITED. If you received > this transmission in error, please immediately contact the sender and > destroy the material in its entirety, whether in electronic or hard copy > format. Although this transmission and any attachments are believed to be > free of any virus or other defect that might affect any computer system > into which it is received and opened, it is the responsibility of the > recipient to ensure that it is virus free and no responsibility is accepted > by JPMC for any loss or damage arising in any way from its use. Please note > that any electronic communication that is conducted within or through > JPMC's systems is subject to interception, monitoring, review, retention > and external production in accordance with JPMC's policy and local laws, > rules and regulations; may be stored or otherwise processed in countries > other than the country in which you are located; and will be treated in > accordance with JPMC policies and applicable laws and regulations. Please > refer to http://www.jpmorgan.com/pages/disclosures for disclosures > relating to European legal entities. > > > > This communication is for informational purposes only. It is not intended > as an offer or solicitation for the purchase or sale of any financial > instrument or as an official confirmation of any transaction. All market > prices, data and other information are not warranted as to completeness or > accuracy and are subject to change without notice. Any comments or > statements made herein do not necessarily reflect those of JPMorgan Chase & > Co., its subsidiaries and affiliates (collectively, "JPMC"). This > transmission may contain information that is proprietary, privileged, > confidential and/or exempt from disclosure under applicable law. If you are > not the intended recipient, you are hereby notified that any disclosure, > copying, distribution, or use of the information contained herein > (including any reliance thereon) is STRICTLY PROHIBITED. If you received > this transmission in error, please immediately contact the sender and > destroy the material in its entirety, whether in electronic or hard copy > format. Although this transmission and any attachments are believed to be > free of any virus or other defect that might affect any computer system > into which it is received and opened, it is the responsibility of the > recipient to ensure that it is virus free and no responsibility is accepted > by JPMC for any loss or damage arising in any way from its use. Please note > that any electronic communication that is conducted within or through > JPMC's systems is subject to interception, monitoring, review, retention > and external production in accordance with JPMC's policy and local laws, > rules and regulations; may be stored or otherwise processed in countries > other than the country in which you are located; and will be treated in > accordance with JPMC policies and applicable laws and regulations. Please > refer to http://www.jpmorgan.com/pages/disclosures for disclosures > relating to European legal entities. >
