Github user ankuriitg commented on the issue:
https://github.com/apache/spark/pull/22504
I looked further into it and it seems that writing to yarn app dir will
mean following certain guidelines: writing filelength, acls, owner, version,
using a compression algorithm, etc. And since these guidelines are internal to
spark (not exposed via public apis), they can change and break this
functionality. So, I have limited the scope of this feature to syncing only to
hdfs.
Thanks for bringing it to notice @tgravescs
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
