Re: Significant Bug

[Thomas L. Redman]

What’s the word on this? I sent this out some time ago, including a GitHub 
project that clearly demonstrates the brokenness, yet I have not heard a word. 
Is there anybody supporting Storm?

> On Sep 30, 2020, at 9:03 AM, Thomas L. Redman  wrote:
> 
> I believe I have encountered a significant bug. It seems topologies employing 
> anchored tuples do not distribute across multiple nodes, regardless of the 
> computation demands of the bolts. It works fine on a single node, but when 
> throwing multiple nodes into the mix, only one machine gets pegged. When we 
> disable anchoring, it will distribute across all nodes just fine, pegging 
> each machine appropriately.
> 
> This bug manifests from version 2.1 forward. I first encountered this issue 
> with my own production cluster on an app that does significant NLP 
> computation across hundreds of millions of documents. This topology is fairly 
> complex, so I developed a very simple exemplar that demonstrates the issue 
> with only one spout and bolt. I pushed this demonstration up to github to 
> provide the developers with a mechanism to easily isolate the bug, and maybe 
> provide some workaround. I used gradle to build this simple topology and 
> software and package the results. This code is well documented, so it should 
> be fairly simple to reproduce the issue. I first encountered this issue on 3 
> 32 core nodes, but when I started experimenting, I set up a test cluster with 
> 8 cores, and then I increased each node to 16 cores, and plenty of memory in 
> every case.
> 
> The topology can be accessed from github at 
> https://github.com/cowchipkid/storm-issue.git 
> . Please feel free to respond 
> to me directory if you have any questions that are beyond the scope of this 
> mail list.

Re: Significant Bug

[Kishor Patil]

Hello Thomas,

Apologies for delay in responding here. I tested the topology code provided in 
storm-issue repo. 
*only one machine gets peggeg*: Although it appears, his is not a bug. This is 
related to Locality Awareness. Please refer to 
https://github.com/apache/storm/blob/master/docs/LocalityAwareness.md
It appears spout to bolt ratio is 200, so if there are enough bolt's on single 
node to handle events generated by the spout, it won't send events out to 
another node unless it runs out of capacity on single node. If you do not like 
this and want to distribute events evenly, you can try disabling this feature. 
You can turn off LoadAwareShuffleGrouping by setting 
topology.disable.loadaware.messaging to true.
-Kishor

On 2020/10/28 15:21:54, "Thomas L. Redman"  wrote: 
> What’s the word on this? I sent this out some time ago, including a GitHub 
> project that clearly demonstrates the brokenness, yet I have not heard a 
> word. Is there anybody supporting Storm?
> 
> > On Sep 30, 2020, at 9:03 AM, Thomas L. Redman  wrote:
> > 
> > I believe I have encountered a significant bug. It seems topologies 
> > employing anchored tuples do not distribute across multiple nodes, 
> > regardless of the computation demands of the bolts. It works fine on a 
> > single node, but when throwing multiple nodes into the mix, only one 
> > machine gets pegged. When we disable anchoring, it will distribute across 
> > all nodes just fine, pegging each machine appropriately.
> > 
> > This bug manifests from version 2.1 forward. I first encountered this issue 
> > with my own production cluster on an app that does significant NLP 
> > computation across hundreds of millions of documents. This topology is 
> > fairly complex, so I developed a very simple exemplar that demonstrates the 
> > issue with only one spout and bolt. I pushed this demonstration up to 
> > github to provide the developers with a mechanism to easily isolate the 
> > bug, and maybe provide some workaround. I used gradle to build this simple 
> > topology and software and package the results. This code is well 
> > documented, so it should be fairly simple to reproduce the issue. I first 
> > encountered this issue on 3 32 core nodes, but when I started 
> > experimenting, I set up a test cluster with 8 cores, and then I increased 
> > each node to 16 cores, and plenty of memory in every case.
> > 
> > The topology can be accessed from github at 
> > https://github.com/cowchipkid/storm-issue.git 
> > . Please feel free to 
> > respond to me directory if you have any questions that are beyond the scope 
> > of this mail list.
> 
> 

Re: Significant Bug

[Thomas L. Redman]

Thanks so much. I will look into this and it’s good to hear there is a team 
intact, as we have just migrated to Storm on a big project, my boss would tar 
and feather me if it went south. I will looking into this and post back what I 
find.


> On Oct 29, 2020, at 8:07 AM, Kishor Patil  wrote:
> 
> Hello Thomas,
> 
> Apologies for delay in responding here. I tested the topology code provided 
> in storm-issue repo. 
> *only one machine gets peggeg*: Although it appears, his is not a bug. This 
> is related to Locality Awareness. Please refer to 
> https://github.com/apache/storm/blob/master/docs/LocalityAwareness.md
> It appears spout to bolt ratio is 200, so if there are enough bolt's on 
> single node to handle events generated by the spout, it won't send events out 
> to another node unless it runs out of capacity on single node. If you do not 
> like this and want to distribute events evenly, you can try disabling this 
> feature. You can turn off LoadAwareShuffleGrouping by setting 
> topology.disable.loadaware.messaging to true.
> -Kishor
> 
> On 2020/10/28 15:21:54, "Thomas L. Redman"  wrote: 
>> What’s the word on this? I sent this out some time ago, including a GitHub 
>> project that clearly demonstrates the brokenness, yet I have not heard a 
>> word. Is there anybody supporting Storm?
>> 
>>> On Sep 30, 2020, at 9:03 AM, Thomas L. Redman  wrote:
>>> 
>>> I believe I have encountered a significant bug. It seems topologies 
>>> employing anchored tuples do not distribute across multiple nodes, 
>>> regardless of the computation demands of the bolts. It works fine on a 
>>> single node, but when throwing multiple nodes into the mix, only one 
>>> machine gets pegged. When we disable anchoring, it will distribute across 
>>> all nodes just fine, pegging each machine appropriately.
>>> 
>>> This bug manifests from version 2.1 forward. I first encountered this issue 
>>> with my own production cluster on an app that does significant NLP 
>>> computation across hundreds of millions of documents. This topology is 
>>> fairly complex, so I developed a very simple exemplar that demonstrates the 
>>> issue with only one spout and bolt. I pushed this demonstration up to 
>>> github to provide the developers with a mechanism to easily isolate the 
>>> bug, and maybe provide some workaround. I used gradle to build this simple 
>>> topology and software and package the results. This code is well 
>>> documented, so it should be fairly simple to reproduce the issue. I first 
>>> encountered this issue on 3 32 core nodes, but when I started 
>>> experimenting, I set up a test cluster with 8 cores, and then I increased 
>>> each node to 16 cores, and plenty of memory in every case.
>>> 
>>> The topology can be accessed from github at 
>>> https://github.com/cowchipkid/storm-issue.git 
>>> . Please feel free to 
>>> respond to me directory if you have any questions that are beyond the scope 
>>> of this mail list.
>> 
>>