Your max spout pending probably doesn't need to be that high. I've seen 10s of thousands of tuples / s with max spout pending of 6k. Also, I wouldn't use the fieldsGrouping unless you intend to dispatch to a particular task within a bolt. For example, if you're aggregating by some user id. ShuffleGrouping should be sufficient to load balance (though it's not a very sophisticated distributor of load). And if your # bolt tasks is greater than the # of workers, storm should distribute them evenly across the workers so localOrShuffleGrouping will work fine. The only thing to be careful if is the upstream component has less tasks than the number of workers.
For example, spout S (1 task), bolt B (localOrShuffleGrouping on S, 2 tasks), across 2 workers will have S in 1 worker, and 1 task for B in each worker. If you use localOrShuffleGrouping, then the bolt task in the worker without the spout task will be starved. Though I wouldn't think that storm would be capped at 16k tuples / s, especially in the complete tree. How many workers are you using? How many supervisor nodes? One other thing you can do is start with just the spout, test its throughput, add a bolt, test throughput, etc. If any bolt is a bottleneck, then this should be able to identify it. You can also stub the business logic in your bolts to test what the limits of the storm framework are with your hardware and topology configuration. On Fri, Dec 19, 2014 at 2:54 PM, Jens-U. Mozdzen <[email protected]> wrote: > > Hi Nathan, > > thank you for your immediate response! > > Zitat von Nathan Leung <[email protected]>: > >> Do you have max spout pending set in the topology? It could be that your >> spout is throttling data since there are too many tuples in flight >> simultaneously, which will have the effect of reducing your maximum >> throughput. >> > > Yes, we have that set, but at approx 13 times our per-second throughput. > As we see a "complete latency" of about 200ms, those 13 seconds of data > should give us enough tuples? > > Whether you are reaching framework communications limits depends on your >> application (how many tuples you are sending), and your network. If you >> suspect this to be the case, you can turn some shuffleGroupings (if you >> have any) into localOrShuffleGroupings, which try to keep the data in >> process if possible, thereby reducing network congestion. >> > > Currently, we're processing about 16k tuples per second in the complete > tree (so including those emitted by intermediate bolts), which shouldn't be > "that high" a number - we're aiming at 10 times of that, at least. All > across a local Gigabit Ethernet segment, no routers or WAN links in between. > > I'll give your suggestion a try - those shuffleGroupings are 50% of the > groupings, the other half is fieldsGrouping (to balance load across > aggregators). > > Are there any Storm-specific options/tools to monitor inter-worker queues? > > With regards, > Jens > >
