Github user ryan-williams commented on the pull request:
https://github.com/apache/spark/pull/2848#issuecomment-62074970
Thanks for having a look, I'll get to that style nit momentarily.
The tie-in to SPARK-3967 is that, on my cluster, when attempting to
redundantly copy the JAR deps, I was getting "Permission Denied" errors because
not only were the JARs already there, they were read-only and could not be
overwritten.
I did not get to the bottom of who was putting them there; it seemed like
every executor would try to copy its JARs, find them to be already present, try
to copy anyway, and fail due to "permission denied."
Simply skipping the redundant copy attempt (as this PR does) resulted in my
jobs passing, and it seemed like the original authors intended to not attempt
to redundantly copy but had a bug, so I settled for this "fix" and didn't get
fully to the bottom of this.
For example, suppose an executor E hit the code path where a given JAR is
present but its contents are not identical to the one E wants. In this case, E
would attempt to overwrite the existing JAR and fail due to the same
"Permission denied", so there could be some larger way in which it needs to be
made possible for executors to overwrite their JAR deps (maybe that is just a
cluster config issue I and @preaudc are hitting?), but if so I think that's
orthogonal to this change.
Let me know if I should just take the "SPARK-3967" label out of this PR,
since it might not fully "solve" the issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
