Hi, I am using storm kafka spout (Version 1.1.0 ) and having multiple topics configured using named subscription. There are 5 topics in one group and each topic has 50 partitions. What should be the spout parallelism to have maximum performance .
Is it 50*5 = 250 as the spout parallelism.Also auto commit for the kafka spout is set as false . Avro is the message format. Also any suggestions to improve the deserialization performance. *Code snippet for topology* *config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 16384);* *config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 16384);* *topicNames.forEach(name -> topicNameSet.add((String) name));* * NamedSubscription namedSubscription = new NamedSubscription(topicNameSet);* *Builder<String, MdpData> builder builder = new KafkaSpoutConfig.Builder<String, MdpData>(kafkaBootstrapServers, StringDeserializer.class, AvroDeserializer.class, namedSubscription);* * KafkaSpoutConfig<String, MdpData> spoutConfig =* * builder.setGroupId(kafkaGroupId).setProp(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false").setFirstPollOffsetStrategy(FirstPollOffsetStrategy.UNCOMMITTED_EARLIEST).build();* Regards Pradeep S
