mridulm commented on pull request #29413: URL: https://github.com/apache/spark/pull/29413#issuecomment-675249974
Queue capacity is an approximation to bound memory usage in event queues; and tends to be conservative - event loss is fine (except DRA) - but driver OOM causes entire app to fail. @SaurabhChawla100 I understand the motivation, but if the 'x'% more required to avoid dropping event could have been predicted - it also could have been very well used to bump up capacity of queue to begin with along with corresponding driver memory increase. Since a reasonably simple solution exists in this case (by leveraging queue capacity configs), based on the current usecases/discussion in the PR, I agree with @tgravescs - I dont see a good reason to introduce the additional complexity. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
