[GitHub] spark issue #12203: [SPARK-14423][YARN] Avoid same name files added to distr...
Github user kishorvpatil commented on the issue: https://github.com/apache/spark/pull/12203 @vanzin, @jerryshao Sorry for breaking this functionality. I have the patch available with more unit tests added to ensure positive test case ensuring submission continues if unique files/archives are mentioned. https://github.com/apache/spark/pull/15810 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12203: [SPARK-14423][YARN] Avoid same name files added to distr...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/12203 I think there was a version of Oozie that triggered that assert; so maybe upgrading Oozie fixes the problem. It's also probably fine to remove that assert since we haven't seen many people hit it, meaning this situation should be rare. And, btw, please avoid long discussions on closed PRs. That's why we have mailing lists and JIRA. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12203: [SPARK-14423][YARN] Avoid same name files added to distr...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/12203 Maybe as you mentioned - skip adding to distributed cache and log warning - is enough, throwing exception will fail the application and this is actually not a fatal problem. I'm OK to change the current behavior for this, what do you think @vanzin ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12203: [SPARK-14423][YARN] Avoid same name files added to distr...
Github user RicoGit commented on the issue: https://github.com/apache/spark/pull/12203 Thanks, i understand this is different problems. What will you advice me? I think that this is not good solution: `require(localizedPath != null)` just fails with exception message "requirements fails".It is better skip adding to the distributed cache and log warning. How do you think it is enough to open issue? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12203: [SPARK-14423][YARN] Avoid same name files added to distr...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/12203 Can you make sure the problem you met is exactly the same as what this PR solved? Since the exception stack you pasted in the StackOverFlow is different from What I pasted here before. From you exception stack, what I could guess is that same jar (same path with same file name) added twice, this is a little different from this PR's mentioned problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12203: [SPARK-14423][YARN] Avoid same name files added to distr...
Github user RicoGit commented on the issue: https://github.com/apache/spark/pull/12203 Thanks for reply. I have [problem with running spark job with oozie](http://stackoverflow.com/questions/38144022/oozie-spark-action-requirement-failed). This patch solves my problem. I applied this path to spark 1.6, built (spark-yarn_2.10-1.6.0-cdh5.7.0.jar) and put into sharedLibs of oozie. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12203: [SPARK-14423][YARN] Avoid same name files added to distr...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/12203 @RicoGit This is a behavior change for jars uploading to distributed cache, I'm not sure if it is suitable to back-port to branch 1.6. Also this problem is not so severe in 1.6 since we do the assembly for packaging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12203: [SPARK-14423][YARN] Avoid same name files added to distr...
Github user RicoGit commented on the issue: https://github.com/apache/spark/pull/12203 Hi guys, it is possible to apply this patch to version 1.6? What can I do for this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
