Thanks for the reply. We are on Spark 2.4. Is there no way to get this fixed in Spark 2.4?
On Mon, Nov 2, 2020 at 8:32 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > Which Spark version do you use? There's a known issue on Kafka producer > pool in Spark 2.x which was fixed in Spark 3.0, so you'd like to check > whether your case is bound to the known issue or not. > > https://issues.apache.org/jira/browse/SPARK-21869 > > > On Tue, Nov 3, 2020 at 1:53 AM Eric Beabes <mailinglist...@gmail.com> > wrote: > >> I know this is related to Kafka but it happens during the Spark >> Structured Streaming job that's why I am asking on this mailing list. >> >> How would you debug this or get around this in Spark Structured >> Streaming? Any tips would be appreciated. Thanks. >> >> >> java.lang.IllegalStateException: Cannot perform operation after producer >> has been closed at >> org.apache.kafka.clients.producer.KafkaProducer.throwIfProducerClosed(KafkaProducer.java:853) >> at >> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:862) >> at >> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:846) >> at >> org.apache.spark.sql.kafka010.KafkaRowWriter.sendRow(KafkaWriteTask.scala:92) >> at >> org.apache.spark.sql.kafka010.KafkaStreamDataWriter.write(KafkaStreamWriter.scala:95) >> >
