[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19073 @caneGuy why do you think it is misleading? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19073 @jerryshao There is a user of our cluster who first saw `warning` he ignore it,if we print error it may be more clearly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19073 But as I remembered the same name file will be ignored when met again. This should not be a fatal issue, right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19073 Job submitting will fail since exception threw.I think this is fatal issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19073 I tried locally with same name jar uploaded twice, the spark application can be started. Can you please paste your exception here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19073 `17/08/29 10:06:26 INFO yarn.Client: Deleting staging directory /spark/xxx/staging/hdfs_admin/.sparkStaging/application_1500536001739_0003 Exception in thread "main" java.lang.IllegalArgumentException: requirement failed at scala.Predef$.require(Predef.scala:221) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6$$anonfun$apply$3.apply(Client.scala:516) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6$$anonfun$apply$3.apply(Client.scala:514) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6.apply(Client.scala:514) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6.apply(Client.scala:512) at scala.collection.immutable.List.foreach(List.scala:318) at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:512) at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:771) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:144) at org.apache.spark.deploy.yarn.Client.run(Client.scala:1088) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1148) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:749) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)` Cluster version is 1.6,i have checked the logic of master,it has not been changed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19073 This is not an error condition. 'someone ignored it' is not sufficient reason. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19073 @caneGuy can you please check this JIRA (https://issues.apache.org/jira/browse/SPARK-14423), I remembered I fixed this issue before. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19073 The actual reason i want change the log level is that,if you threw an exception ,then this indicate it is a fatal error,we should not just print a warning. @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19073 Ok i will try later,thanks @jerryshao --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19073 @jerryshao reproduce step is below: 1ãadd `--files` when submit jobs 2ãadd file with same name in `--files` then exception will throw After looking into your pr,that fix this issue. I will backport 14423 to our inner branch,and can you backport your fix to masterã --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19073 I think it is already in master branch @caneGuy . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19073 Yes from `git blame` i found that.And i found a follow-up(maybe follow-up?) pr [spark-18099](https://github.com/apache/spark/pull/15627) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19073 Yes, maybe. You can take a try locally. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
