[GitHub] spark issue #19200: Get default Locale
Github user Gschiavon commented on the issue: https://github.com/apache/spark/pull/19200 As far as I saw there are other test cases that set default locale to US. This case is not passing when your jvm default locale value differs from "US". --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19205: [SPARK-21982] Set locale to US
GitHub user Gschiavon opened a pull request: https://github.com/apache/spark/pull/19205 [SPARK-21982] Set locale to US ## What changes were proposed in this pull request? In UtilsSuite Locale was set by default to US, but at the format time it wasn't, taking by default JVM locale which could be different than US making this test fail. ## How was this patch tested? Unit test (UtilsSuite) Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Gschiavon/spark fix/test-locale Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19205.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19205 commit 22bbb924eae20b8d3f899008317f5d623c6a49ef Author: German Schiavon <germanschia...@gmail.com> Date: 2017-09-12T12:05:03Z Set locale to US --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19200: Get default Locale
Github user Gschiavon closed the pull request at: https://github.com/apache/spark/pull/19200 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19200: Get default Locale
Github user Gschiavon commented on the issue: https://github.com/apache/spark/pull/19200 Ok, I got it. I will do that then. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19200: Set default Locale
GitHub user Gschiavon opened a pull request: https://github.com/apache/spark/pull/19200 Set default Locale ## What changes were proposed in this pull request? Get default Locale in UtilsSuite.scala in order to make it work with different Locales than US. ## How was this patch tested? Running UtilsSuite.scala Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Gschiavon/spark fix/locale Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19200.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19200 commit 632526ba3e9a4d72133202cf0bfcc8a997dc9cb9 Author: German Schiavon <germanschia...@gmail.com> Date: 2017-09-12T08:33:00Z Set default Locale --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19793: [SPARK-22574] [Mesos] [Submit] Check submission request ...
Github user Gschiavon commented on the issue: https://github.com/apache/spark/pull/19793 can we retest this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19793: [SPARK-22574] [Mesos] [Submit] Check submission request ...
Github user Gschiavon commented on the issue: https://github.com/apache/spark/pull/19793 ping @ArtRand --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19793: [SPARK-22574] [Mesos] [Submit] Check submission r...
Github user Gschiavon commented on a diff in the pull request: https://github.com/apache/spark/pull/19793#discussion_r153709924 --- Diff: core/src/test/scala/org/apache/spark/deploy/rest/SubmitRestProtocolSuite.scala --- @@ -86,6 +86,8 @@ class SubmitRestProtocolSuite extends SparkFunSuite { message.clientSparkVersion = "1.2.3" message.appResource = "honey-walnut-cherry.jar" message.mainClass = "org.apache.spark.examples.SparkPie" +message.appArgs = Array("hdfs://tmp/auth") +message.environmentVariables = Map("SPARK_HOME" -> "/test") --- End diff -- @susanxhuynh Yes we have to make sure we don't break the Standalone mode. I've reviewed StandaloneSubmitRequestServlet class and i think we might be facing the same problem here, (L.126-131) You can find checks for 'appResource' and 'mainClass' but not for 'appArgs'(L.140) or 'environmentVariables'(L.143) when they could be null as they are initialised as null. I think is the same case, let me know what you think. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19793: [SPARK-22574] [Mesos] [Submit] Check submission r...
Github user Gschiavon commented on a diff in the pull request: https://github.com/apache/spark/pull/19793#discussion_r153131125 --- Diff: core/src/test/scala/org/apache/spark/deploy/rest/SubmitRestProtocolSuite.scala --- @@ -86,6 +86,8 @@ class SubmitRestProtocolSuite extends SparkFunSuite { message.clientSparkVersion = "1.2.3" message.appResource = "honey-walnut-cherry.jar" message.mainClass = "org.apache.spark.examples.SparkPie" +message.appArgs = Array("hdfs://tmp/auth") +message.environmentVariables = Map("SPARK_SCALA_VERSION" -> "2.11") --- End diff -- In this case it's just a name, I could change it in order to make it clear! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19793: [SPARK-22574] [Mesos] [Submit] Check submission request ...
Github user Gschiavon commented on the issue: https://github.com/apache/spark/pull/19793 Yes @ArtRand! I think it's not documented at all, ping me when it's done and I will review it :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19793: [SPARK-22574] [Mesos] [Submit] Check submission r...
Github user Gschiavon commented on a diff in the pull request: https://github.com/apache/spark/pull/19793#discussion_r153410217 --- Diff: core/src/main/scala/org/apache/spark/deploy/rest/SubmitRestProtocolRequest.scala --- @@ -46,6 +46,8 @@ private[rest] class CreateSubmissionRequest extends SubmitRestProtocolRequest { super.doValidate() assert(sparkProperties != null, "No Spark properties set!") assertFieldIsSet(appResource, "appResource") +assertFieldIsSet(appArgs, "appArgs") +assertFieldIsSet(environmentVariables, "environmentVariables") --- End diff -- Actually If the caller wouldn't set "appArgs" or "environmentVariables" was causing a null pointer and leaving the Dispatcher inactive. So now I think the caller should pass an empty array, I could add a test for that case @susanxhuynh . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19793: [SPARK-22574] [Mesos] [Submit] Check submission r...
Github user Gschiavon commented on a diff in the pull request: https://github.com/apache/spark/pull/19793#discussion_r153414519 --- Diff: core/src/test/scala/org/apache/spark/deploy/rest/SubmitRestProtocolSuite.scala --- @@ -86,6 +86,8 @@ class SubmitRestProtocolSuite extends SparkFunSuite { message.clientSparkVersion = "1.2.3" message.appResource = "honey-walnut-cherry.jar" message.mainClass = "org.apache.spark.examples.SparkPie" +message.appArgs = Array("hdfs://tmp/auth") +message.environmentVariables = Map("SPARK_HOME" -> "/test") --- End diff -- @felixcheung okay done :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19793: [SPARK-22574] [Mesos] [Submit] Check submission r...
Github user Gschiavon commented on a diff in the pull request: https://github.com/apache/spark/pull/19793#discussion_r153410771 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/deploy/rest/mesos/MesosRestServer.scala --- @@ -82,6 +82,12 @@ private[mesos] class MesosSubmitRequestServlet( val mainClass = Option(request.mainClass).getOrElse { throw new SubmitRestMissingFieldException("Main class is missing.") } +val appArgs = Option(request.appArgs).getOrElse { + throw new SubmitRestMissingFieldException("Application arguments are missing.") --- End diff -- Done @ArtRand --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19793: [SPARK-22574] [Mesos] [Submit] Check submission r...
Github user Gschiavon commented on a diff in the pull request: https://github.com/apache/spark/pull/19793#discussion_r153415074 --- Diff: core/src/test/scala/org/apache/spark/deploy/rest/SubmitRestProtocolSuite.scala --- @@ -86,6 +86,8 @@ class SubmitRestProtocolSuite extends SparkFunSuite { message.clientSparkVersion = "1.2.3" message.appResource = "honey-walnut-cherry.jar" message.mainClass = "org.apache.spark.examples.SparkPie" +message.appArgs = Array("hdfs://tmp/auth") +message.environmentVariables = Map("SPARK_HOME" -> "/test") --- End diff -- @susanxhuynh I've checked what you said but those variables are overwritten below, so I can't actually check it, right? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19793: [SPARK-22574] [Mesos] [Submit] Check submission r...
Github user Gschiavon commented on a diff in the pull request: https://github.com/apache/spark/pull/19793#discussion_r154072747 --- Diff: core/src/test/scala/org/apache/spark/deploy/rest/SubmitRestProtocolSuite.scala --- @@ -86,6 +86,8 @@ class SubmitRestProtocolSuite extends SparkFunSuite { message.clientSparkVersion = "1.2.3" message.appResource = "honey-walnut-cherry.jar" message.mainClass = "org.apache.spark.examples.SparkPie" +message.appArgs = Array("hdfs://tmp/auth") +message.environmentVariables = Map("SPARK_HOME" -> "/test") --- End diff -- Cool, I can do it when I have some time. Let me know if I have to something else here! Thanks @susanxhuynh --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19793: [SPARK-22574] [Mesos] [Submit] Check submission r...
Github user Gschiavon commented on a diff in the pull request: https://github.com/apache/spark/pull/19793#discussion_r153901749 --- Diff: core/src/test/scala/org/apache/spark/deploy/rest/SubmitRestProtocolSuite.scala --- @@ -86,6 +86,8 @@ class SubmitRestProtocolSuite extends SparkFunSuite { message.clientSparkVersion = "1.2.3" message.appResource = "honey-walnut-cherry.jar" message.mainClass = "org.apache.spark.examples.SparkPie" +message.appArgs = Array("hdfs://tmp/auth") +message.environmentVariables = Map("SPARK_HOME" -> "/test") --- End diff -- Perfect then @susanxhuynh . Fix it maybe in another PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19793: [SPARK-22574] [Mesos] [Submit] Check submission request ...
Github user Gschiavon commented on the issue: https://github.com/apache/spark/pull/19793 Hi @vanzin ! I just fixed it and pushed it to my branch but it's not opening the PR, maybe I need to do a new one? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19966: Fix submission request
Github user Gschiavon commented on the issue: https://github.com/apache/spark/pull/19966 Jenkins, test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19966: Fix submission request
Github user Gschiavon commented on the issue: https://github.com/apache/spark/pull/19966 @felixcheung @cloud-fan @vanzin @susanxhuynh @gatorsmile @ArtRand Here is the new PR --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19966: Fix submission request
GitHub user Gschiavon opened a pull request: https://github.com/apache/spark/pull/19966 Fix submission request ## What changes were proposed in this pull request? PR closed with all the comments -> https://github.com/apache/spark/pull/19793 It solves the problem when submitting a wrong CreateSubmissionRequest to Spark Dispatcher was causing a bad state of Dispatcher and making it inactive as a Mesos framework. https://issues.apache.org/jira/browse/SPARK-22574 ## How was this patch tested? All spark test passed successfully. It was tested sending a wrong request (without appArgs) before and after the change. The point is easy, check if the value is null before being accessed. This was before the change, leaving the dispatcher inactive: ``` Exception in thread "Thread-22" java.lang.NullPointerException at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.getDriverCommandValue(MesosClusterScheduler.scala:444) at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.buildDriverCommand(MesosClusterScheduler.scala:451) at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.org$apache$spark$scheduler$cluster$mesos$MesosClusterScheduler$$createTaskInfo(MesosClusterScheduler.scala:538) at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:570) at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:555) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.scheduleTasks(MesosClusterScheduler.scala:555) at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.resourceOffers(MesosClusterScheduler.scala:621) ``` And after: ``` "message" : "Malformed request: org.apache.spark.deploy.rest.SubmitRestProtocolException: Validation of message CreateSubmissionRequest failed!\n\torg.apache.spark.deploy.rest.SubmitRestProtocolMessage.validate(SubmitRestProtocolMessage.scala:70)\n\torg.apache.spark.deploy.rest.SubmitRequestServlet.doPost(RestSubmissionServer.scala:272)\n\tjavax.servlet.http.HttpServlet.service(HttpServlet.java:707)\n\tjavax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\torg.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)\n\torg.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)\n\torg.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\torg.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)\n\torg.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\torg.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\torg.s park_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\torg.spark_project.jetty.server.Server.handle(Server.java:524)\n\torg.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:319)\n\torg.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)\n\torg.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\n\torg.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\torg.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\torg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\torg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\torg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\torg.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool. java:671)\n\torg.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tjava.lang.Thread.run(Thread.java:745)" ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/Gschiavon/spark fix-submission-request Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19966.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19966 commit 44fd5d3921299f93d1aab7fe971078a6bce835a2 Author: German Schiavon <germanschia...@gmail.com> Date: 2017-11-21T15:32:04Z Check submission request parameters commit 14d64172500483f9e984ac28ba5f3b52db33ad9e Author: German Schiavon <germanschia...@gmail.com> Date: 2017-11-27T08
[GitHub] spark pull request #19793: [SPARK-22574] [Mesos] [Submit] Check submission r...
GitHub user Gschiavon opened a pull request: https://github.com/apache/spark/pull/19793 [SPARK-22574] [Mesos] [Submit] Check submission request parameters ## What changes were proposed in this pull request? It solving the problem when submitting a wrong CreateSubmissionRequest to Spark Dispatcher was causing a bad state of Dispatcher and making it inactive as a Mesos framework. ## How was this patch tested? It was tested sending a wrong request (without appArgs) before and after the change. The point is easy, check if the value is null before being accessed. This was before the change, leaving the dispatcher inactive: ``` Exception in thread "Thread-22" java.lang.NullPointerException at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.getDriverCommandValue(MesosClusterScheduler.scala:444) at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.buildDriverCommand(MesosClusterScheduler.scala:451) at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.org$apache$spark$scheduler$cluster$mesos$MesosClusterScheduler$$createTaskInfo(MesosClusterScheduler.scala:538) at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:570) at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:555) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.scheduleTasks(MesosClusterScheduler.scala:555) at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.resourceOffers(MesosClusterScheduler.scala:621) ``` And after: ``` "message" : "Malformed request: org.apache.spark.deploy.rest.SubmitRestProtocolException: Validation of message CreateSubmissionRequest failed!\n\torg.apache.spark.deploy.rest.SubmitRestProtocolMessage.validate(SubmitRestProtocolMessage.scala:70)\n\torg.apache.spark.deploy.rest.SubmitRequestServlet.doPost(RestSubmissionServer.scala:272)\n\tjavax.servlet.http.HttpServlet.service(HttpServlet.java:707)\n\tjavax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\torg.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)\n\torg.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)\n\torg.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\torg.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)\n\torg.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\torg.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\torg.s park_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\torg.spark_project.jetty.server.Server.handle(Server.java:524)\n\torg.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:319)\n\torg.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)\n\torg.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\n\torg.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\torg.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\torg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\torg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\torg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\torg.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool. java:671)\n\torg.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tjava.lang.Thread.run(Thread.java:745)", ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/Gschiavon/spark fix-submission-request Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19793.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19793 commit 4f28d0011892326c6ec3fd9a4fc6fb48756cd635 Author: German Schiavon <germanschia...@gmail.com> Date: 2017-11-21T15:32:04Z Check submission request parameters --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org