[ https://issues.apache.org/jira/browse/SPARK-22991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shixiong Zhu updated SPARK-22991: --------------------------------- Component/s: (was: Structured Streaming) (was: Spark Core) DStreams > High read latency with spark streaming 2.2.1 and kafka 0.10.0.1 > --------------------------------------------------------------- > > Key: SPARK-22991 > URL: https://issues.apache.org/jira/browse/SPARK-22991 > Project: Spark > Issue Type: Bug > Components: DStreams > Affects Versions: 2.2.1 > Reporter: Kiran Shivappa Japannavar > Priority: Critical > > Spark 2.2.1 + Kafka 0.10 + Spark streaming. > Batch duration is 1s, Max rate per partition is 500, poll interval is 120 > seconds, max poll records is 500 and no of partitions in Kafka is 500, > enabled cache consumer. > While trying to read data from Kafka we are observing very high read > latencies intermittently.The high latencies results in Kafka consumer session > expiration and hence the Kafka brokers removes the consumer from the group. > The consumer keeps retrying and finally fails with the > [org.apache.kafka.clients.NetworkClient] - Disconnecting from node 12 due to > request timeout > [org.apache.kafka.clients.NetworkClient] - Cancelled request ClientRequest > [org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient] - > Cancelled FETCH request ClientRequest.** > Due to this a lot of batches are in the queued state. > The high read latencies are occurring whenever multiple clients are > parallelly trying to read the data from the same Kafka cluster. The Kafka > cluster is having a large number of brokers and can support high network > bandwidth. > When running with spark 1.5 and Kafka 0.8 consumer client against the same > Kafka cluster we are not seeing any read latencies. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org