[ https://issues.apache.org/jira/browse/KAFKA-7749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16737038#comment-16737038 ]
Sönke Liebau commented on KAFKA-7749: ------------------------------------- Hi [~pdavidson], can you perhaps elaborate why you'd want to set this at task level? Shouldn't tasks just be for load balancing and hence write to the same cluster? I think Connect currently doesn't have the concept of task-level setting, so that would probably be a larger change.. > confluent does not provide option to set consumer properties at connector > level > ------------------------------------------------------------------------------- > > Key: KAFKA-7749 > URL: https://issues.apache.org/jira/browse/KAFKA-7749 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect > Reporter: Manjeet Duhan > Priority: Major > > _We want to increase consumer.max.poll.record to increase performance but > this value can only be set in worker properties which is applicable to all > connectors given cluster._ > __ > _Operative Situation :- We have one project which is communicating with > Elasticsearch and we set consumer.max.poll.record=500 after multiple > performance tests which worked fine for an year._ > _Then one more project onboarded in the same cluster which required > consumer.max.poll.record=5000 based on their performance tests. This > configuration is moved to production._ > _Admetric started failing as it was taking more than 5 minutes to process > 5000 polled records and started throwing commitfailed exception which is > vicious cycle as it will process same data over and over again._ > __ > _We can control above if start consumer using plain java but this control was > not available at each consumer level in confluent connector._ > _I have overridden kafka code to accept connector properties which will be > applied to single connector and others will keep on using default properties > . These changes are already running in production for more than 5 months._ > _Some of the properties which were useful for us._ > max.poll.records > max.poll.interval.ms > request.timeout.ms > key.deserializer > value.deserializer > heartbeat.interval.ms > session.timeout.ms > auto.offset.reset > connections.max.idle.ms > enable.auto.commit > > auto.commit.interval.ms > -- This message was sent by Atlassian JIRA (v7.6.3#76005)