Hi, I haven't really looked at optimizing storm-kafka-client for Trident, but I'd suggest you start by making a topology that only contains a spout and no bolts and see what that'll push through. That'll let you figure out whether the bottleneck is in your processing or in the spout, or in the part of the code that saves to your DB. You may also be able to tell from Storm UI's capacity metric which part of the topology is the bottleneck.
2018-07-30 19:58 GMT+02:00 Shaik Asifullah <[email protected]>: > Hi, > I am using Apache Storm Trident (1.2.1) for micro-batching. The main idea > of my topology is to modify messages from Kafka a little and push it to > Database. Each event of mine in Kafka is around 3 KB which is to be > inserted in one of the tables in DB. I want 10K inserts per second in the > database. Currently I am able to send around 1K messages per second with > the current configuration. > > .setOffsetCommitPeriodMs(1_000) > > .setFirstPollOffsetStrategy(FirstPollOffsetStrategy.UNCOMMITTED_EARLIEST) > > .setMaxUncommittedOffsets(1000) > > .setPollTimeoutMs(2000) > > .setEmitNullTuples(false) > > .setProp(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest") > > .setProcessingGuarantee(ProcessingGuarantee.AT_LEAST_ONCE) > > .setProp("fetch.min.bytes",10*1024*1024) > > .setProp("max.partition.fetch.bytes",20*1024*1024) > > .setProp("fetch.max.wait.ms",5* 100) > > > I am using 8 worker nodes spread over 2 machines of 4 cores each and I am > using tumbling window of batch size 10K. I am using batch since writes to > DB takes around 1-2 seconds. Can you please help me in increasing it > performance ? > > > Thanks, > > Shaik Asifullah > >
