jaagupku commented on issue #36284: URL: https://github.com/apache/beam/issues/36284#issuecomment-3381459522
Found a solution to increase the Spark configuration `spark.task.maxFailures`, it had default value of 4. I observed that if the setup fails multiple times in a row then it will start reading from the KafkaUnboundedSource again and seems not to process the previously read bundle of messages. I added a sleep in setup if there are issues with database connection and increased the `spark.task.maxFailures` property. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
