Github user roshannaik commented on a diff in the pull request:
https://github.com/apache/storm/pull/2241#discussion_r135695333
--- Diff: conf/defaults.yaml ---
@@ -231,20 +234,35 @@ topology.multilang.serializer:
"org.apache.storm.multilang.JsonSerializer"
topology.shellbolt.max.pending: 100
topology.skip.missing.kryo.registrations: false
topology.max.task.parallelism: null
-topology.max.spout.pending: null
+topology.max.spout.pending: 995 # Used in ACK mode only. Should be less
than `topology.executor.receive.buffer.size` to avoid deadlocked cycles
involving ACKer
--- End diff --
I agree and yes I am still looking to avoid relying on this.
Unfortunately discovered a corner case during testing that requires it as
of now ... which i alluded to in that comment.
**Details:** The issue is that ACKer bolt creates a cycle in the topology.
So if ACKer bolt's recvQ is full, and also the spout's recvQ is full. Then
spout & ACKer can end up in a deadlock while trying to send msg to each other.
Setting m.s.p to a value slightly smaller than the recvQ size avoids this
situation.
One option is to make this an internal setting that is automatically set
based on the revQ size and users dont have to set.
I am investigating an alternative that allows the spout to process incoming
ACKs if outbound Q is full...and thus break the deadlock.
Hope to revert that setting back to null soon.
I do not plan to remove that setting as now. Will remove it after ensuring
@revans2's prior noted concerns in relation to removing it have been addressed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
