viirya commented on pull request #30812: URL: https://github.com/apache/spark/pull/30812#issuecomment-753254552
I put the summary of my points in the PR description. IMHO, I do not see a strong reason to object this change. This does not have back-compatibility issue and also does not change user-facing behavior. This does not change any API. Actually without non-trivial task locality setting, this is a no-op because of Spark task scheduler's characteristics. Again, preferred location is only a hint to Spark task scheduler. As one point mentioned (https://github.com/apache/spark/pull/30812#issuecomment-750544363) already is, there is no other feasible approach to deal with the issue of unstable state store locations across batches and skew state store locations without major change, and the issue is harmful to stateful streaming queries. Please consider this patch so we may move things forward. If there is other feasible approach we can take, please share and I will be happy to take. Thanks. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
