Thanks I will try it out. Thanks, Michael Kazansky
From: Nathan Leung [mailto:[email protected]] Sent: Thursday, June 09, 2016 3:07 PM To: user Subject: Re: not all the servers in storm cluster processing messages If you use localOrShuffleGrouping, the tuple will stay in process if possible. The solution, assuming your bolts are distributed, is to use shuffleGrouping to send tuples from spout->bolt so that the tuples are distributed to all of your workers, then use localOrShuffleGrouping from bolt->bolt to reduce inter-worker communication overhead. On Thu, Jun 9, 2016 at 3:00 PM, Kazansky, Michael <[email protected]<mailto:[email protected]>> wrote: We are using localOrShuffleGrouping Thanks, Michael Kazansky From: Satish Duggana [mailto:[email protected]<mailto:[email protected]>] Sent: Thursday, June 09, 2016 1:54 PM To: [email protected]<mailto:[email protected]> Subject: Re: not all the servers in storm cluster processing messages Hi Michael, Kafka spout is given parallelism-hint as 1 and the topic has only one partition. So, only one kafka spout task is run in one of the workers of the topology. Are you asking that bolts connected directly/indirectly from that kafka spout are all executed in the same worker as kafka spout? If so, What are the groupings and parallellism-hint that you have configured on those bolts? Thanks, Satish. On Thu, Jun 9, 2016 at 10:41 PM, Kazansky, Michael <[email protected]<mailto:[email protected]>> wrote: Hi All, We have 3 servers in storm cluster. To prevent the order of the messages which come to Kafka the Kafka partition was set to 1. KafkaSpout parallelism hint is set to 1 as well. When messages flow through the Storm we noticed that only one server in the cluster is busy processing messages other 2 are idle although the number of workers is set to 3 in Config. Is it excepted behavior or something is missing? If it is expected behavior is there any way to have all 3 servers in the cluster processing messages? Storm version is 0.9.5 Thanks, Michael Kazansky This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities. This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities. This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.
