gengliangwang commented on PR #38567: URL: https://github.com/apache/spark/pull/38567#issuecomment-1308227478
> Or part of a larger set of changes Yes, I created Jira https://issues.apache.org/jira/browse/SPARK-41053, and this one is just the beginning. > Maintaining state at driver on disk backed store and copying that to dfs has a few things which impact it - particularly for larger applications. Yes, but it worthies starting the effort. We can keep developing it until it is ready. I run a benchmark with Protobuf serializer, and the results are promising. For large applications, the current in-memory live UI can bring memory pressure which affects the driver stability as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
