Jungtaek Lim created SPARK-34383: ------------------------------------ Summary: Optimize WAL commit phase on SS Key: SPARK-34383 URL: https://issues.apache.org/jira/browse/SPARK-34383 Project: Spark Issue Type: Improvement Components: Structured Streaming Affects Versions: 3.2.0 Reporter: Jungtaek Lim
I found there're unnecessary access / expensive operation of file system in WAL commit phase of SS. They can be optimized via caching (using driver memory a bit) & replacing FS operation. This brings reduced latency per batch, especially checkpoint against object store. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org