Hi Pradeep, Your config looks fine to me. Could you please check the log and see what causes the failure?
Regards, Li Sent from my iPhone > On 10 Dec 2016, at 11:26 AM, pradeep s <[email protected]> wrote: > > Hi, > We are running a 5 node cluster(2x large ec2) with below config. Topology > is consuming message from SQS and writing to RDS and S3. Even if there are > no bolt failures , we are seeing many spout failures. > Can you please help in checking the config.Also i am setting tasks as > parallelism count * 2. Is this fine? > > TOPOLOGY_NAME=MDP_STORM_PRD > MARIA_BOLT_PARALLELISM=50 > S3_BOLT_PARALLELISM=50 > SQS_DELETE_BOLT_PARALLELISM=100 > SPOUT_PARALLELISM=50 > NUMBER_OF_WORKERS=5 > NUMBER_OF_ACKERS=5 > SPOUT_MAX_PENDING=5000 > MESSAGE_TIMEOUT_SECONDS=240 > > > Topology Code > ============= > > Config config = new Config(); > config.setNumWorkers(numWorkers); > config.setDebug(false); > config.setNumAckers(numAckers); > config.setMaxSpoutPending(maxSpoutPending); > config.setMessageTimeoutSecs(messageTimeoutSecs); > > > topologyBuilder.setSpout(spoutId, new > SQSMessageReaderSpout(sqsUtils.getSQSUrl(dataQueue), properties), > spoutParallelism).*setNumTasks(spoutParallelism * TWO);* > topologyBuilder.setBolt(mariaBoltId, new MariaDbBolt(properties), > mariaBoltParallelism) > .*setNumTasks(mariaBoltParallelism * TWO)*.fieldsGrouping(spoutId, > new Fields(MESSAGE_ID)); > > topologyBuilder.setBolt(s3BoltId, new S3WriteBolt(properties, > s3Properties), s3BoltParallelism) > .*setNumTasks(s3BoltParallelism * TWO).* > shuffleGrouping(mariaBoltId); > > topologyBuilder > .setBolt(sqsDeleteBoltId, new > SQSMessageDeleteBolt(sqsUtils.getSQSUrl(dataQueue)), sqsBoltParallelism) > .*setNumTasks(sqsBoltParallelism * TWO)*.shuffleGrouping(s3BoltId); > > StormSubmitter.submitTopology(topologyName, config, > topologyBuilder.createTopology()); > > > Regards > > Pradeep S
