How are you measuring output speed? Is it possible that you are experiencing problems with HBase?
On Sun, Nov 1, 2015 at 9:22 PM, Renjie Liu <[email protected]> wrote: > The result of jstat shows that it's not in full gc cycle but the minor gc > takes more than 1s each time. However, the frequence of minor gc is quite > low, which happens once every few seconds. > > On Mon, Nov 2, 2015 at 12:29 AM, Nathan Leung <[email protected]> wrote: > >> The box with no throughput might be in a gc loop. Check your heap >> utilization and maybe increase worker heap if necessary. Also consider >> decreasing the max spout pending, even without further details 20k seems >> high. >> On Nov 1, 2015 10:50 AM, "Harsha" <[email protected]> wrote: >> >>> Do you have any calls to external data sources which might be increasing >>> the latency and causing tuple timeout? >>> >>> >>> On Sun, Nov 1, 2015, at 04:49 AM, Renjie Liu wrote: >>> >>> Yes, I've set it to 20000 >>> >>> On Sun, Nov 1, 2015 at 6:40 PM, Santosh Pingale < >>> [email protected]> wrote: >>> >>> Have you set 'topology.*max*.*spout*.*pending'?* >>> >>> On Sun, Nov 1, 2015 at 2:26 PM, Renjie Liu <[email protected]> >>> wrote: >>> >>> Hi, storm community: >>> >>> We have a storm cluster deployed with 15 workers and recently we often >>> experience failure since ack timeout. Our input source is kafka and we used >>> ganglia to monitor our cluster. Recently we experience failures every 12 >>> hours and following are my observations from some monitoring tools when >>> problem happens: >>> >>> 1. Topology page shows that no worker was down since uptime of each >>> task are nearly equal to topology uptime >>> 2. I've checked ganglia, the cpu report and mem report does not give >>> any clue about the problem. But network report shows something unusual: >>> the >>> in speed decreases a little while the out speed decreases to nearly zero >>> on >>> some workers. >>> 3. I've logged in to one of machines mentioned above, and found out >>> that one of the survivor areas always remains 100% full. >>> 4. dstat show that csw turns to 4k+ every few seconds while it >>> remains around 400 in normal condition. >>> >>> Can anyone give us some hint about this problem? >>> >>> >>> >>> >>> >>> >>> -- >>> Renjie Liu >>> Department of Computer Science & Engineering >>> Shanghai JiaoTong University >>> >>> >>> >> > > > -- > Renjie Liu > Department of Computer Science & Engineering > Shanghai JiaoTong University >
