squito commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks
when executors are lost
URL: https://github.com/apache/spark/pull/24462#issuecomment-520556153
The "reasonable balance" I was talking about was between extending the spark
api to cover more use cases, while still k
squito commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks
when executors are lost
URL: https://github.com/apache/spark/pull/24462#issuecomment-519542105
ok, I see what you're trying to do -- and yeah I don't think you can do it
with the api we are proposing. That is a bu
squito commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks
when executors are lost
URL: https://github.com/apache/spark/pull/24462#issuecomment-516149439
why do you want to store the data files on hdfs, but the index files on the
executors? This seems to have the worst of
squito commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks
when executors are lost
URL: https://github.com/apache/spark/pull/24462#issuecomment-511901893
I took another look at @yifeih 's changes, and I think she's right, that
will be sufficient. Now you're custom shuffle
squito commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks
when executors are lost
URL: https://github.com/apache/spark/pull/24462#issuecomment-497862712
FWIW, its at least clear to me know that this will be needed even with
SPARK-25299, as that work is orthogonal to this
squito commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks
when executors are lost
URL: https://github.com/apache/spark/pull/24462#issuecomment-488385270
> A better long-term approach could be something like: add a method to the
ShuffleManager interface that allows the imp
squito commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks
when executors are lost
URL: https://github.com/apache/spark/pull/24462#issuecomment-487145924
I understand the motivation here, but I think this should be handled as part
of the new shuffle storage plugin mechanis
