[ https://issues.apache.org/jira/browse/SPARK-27002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin reassigned SPARK-27002: -------------------------------------- Assignee: Gabor Somogyi > Latest kafka delegation token not always picked up > -------------------------------------------------- > > Key: SPARK-27002 > URL: https://issues.apache.org/jira/browse/SPARK-27002 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 3.0.0 > Reporter: Gabor Somogyi > Assignee: Gabor Somogyi > Priority: Critical > > Lately KAFKA-8004 has been discovered and this triggered me to do further > tests with Spark. > Namely I've created long running Kafka to Kafka tests on 4 node cluster with > randomly thrown artificial exceptions. > Test scenario: > * 4 node cluster > * Yarn > * Kafka broker version 2.1.0 > * security.protocol = SASL_SSL > * sasl.mechanism = SCRAM-SHA-512 > Kafka broker settings: > * delegation.token.expiry.time.ms=600000 (10 min) > * delegation.token.max.lifetime.ms=1200000 (20 min) > * delegation.token.expiry.check.interval.ms=300000 (5 min) > After each 7.5 minutes new delegation token obtained from Kafka broker (10 > min * 0.75). > But when token expired after 10 minutes (Spark obtains new one and doesn't > renew the old), the brokers expiring thread comes after each 5 minutes > (invalidates expired tokens) and artificial exception has been thrown inside > the Spark application (such case Spark closes connection), then the latest > delegation token not always picked up. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org