Hi All, I am using Spark structured streaming with spark 2.3 running on Yarn cluster with hadoop 2.7.3 In the driver logs I see numerous lines as below.
2019-05-02 17:14:24.619 ContextCleaner Spark Context Cleaner [INFO] Cleaned accumulator 81492577 2019-05-02 17:14:24.619 ContextCleaner Spark Context Cleaner [INFO] Cleaned accumulator 81492514 2019-05-02 17:14:24.619 ContextCleaner Spark Context Cleaner [INFO] Cleaned accumulator 81492531 2019-05-02 17:14:24.620 ContextCleaner Spark Context Cleaner [INFO] Cleaned accumulator 81492524 2019-05-02 17:14:24.620 ContextCleaner Spark Context Cleaner [INFO] Cleaned accumulator 81492617 2019-05-02 17:14:24.620 ContextCleaner Spark Context Cleaner [INFO] Cleaned accumulator 81492475 2019-05-02 17:14:24.620 ContextCleaner Spark Context Cleaner [INFO] Cleaned accumulator 81492467 Could anyone explain what do these logs indicate internally? I have streaming interval of 500ms, reading data from Kafka topic with max batch size as 1000. Akshay Bhardwaj +91-97111-33849
