[jira] [Created] (SPARK-6287) Add support for dynamic allocation in the Mesos coarse-grained scheduler
Iulian Dragos created SPARK-6287: Summary: Add support for dynamic allocation in the Mesos coarse-grained scheduler Key: SPARK-6287 URL: https://issues.apache.org/jira/browse/SPARK-6287 Project: Spark Issue Type: Bug Components: Mesos Reporter: Iulian Dragos Add support inside the coarse-grained Mesos scheduler for dynamic allocation. It amounts to implementing two methods that allow scaling up and down the number of executors: {code} def doKillExecutors(executorIds: Seq[String]) def doRequestTotalExecutors(requestedTotal: Int) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6285) Duplicated code leads to errors
[ https://issues.apache.org/jira/browse/SPARK-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357297#comment-14357297 ] Iulian Dragos commented on SPARK-6285: -- According to the git commit message that introduced the duplicate: {quote} To avoid potential merge conflicts, old testing code are not removed yet. The following classes can be safely removed after most Parquet related PRs are handled: - `ParquetQuerySuite` - `ParquetTestData` {quote} I mentioned the Eclipse build problem in passing, but I can expand: the class *is* a duplicated name, so the Scala compiler is correct in refusing it. It only compiles in Sbt/Maven because the src/main and src/test are compiled in separate compiler runs, and scalac seems to not notice the duplicate name when it comes from bytecode. Eclipse builds src/main and src/test together, and when both classes originate from sources scalac issues an error message. > Duplicated code leads to errors > --- > > Key: SPARK-6285 > URL: https://issues.apache.org/jira/browse/SPARK-6285 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.0 >Reporter: Iulian Dragos > > The following class is duplicated inside > [ParquetTestData|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTestData.scala#L39] > and > [ParquetIOSuite|https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetIOSuite.scala#L44], > with exact same code and fully qualified name: > {code} > org.apache.spark.sql.parquet.TestGroupWriteSupport > {code} > The second one was introduced in > [3b395e10|https://github.com/apache/spark/commit/3b395e10510782474789c9098084503f98ca4830], > but even though it mentions that `ParquetTestData` should be removed later, > I couldn't find a corresponding Jira ticket. > This duplicate class causes the Eclipse builder to fail (since src/main and > src/test are compiled together in Eclipse, unlike Sbt). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6286) Handle TASK_ERROR in TaskState
[ https://issues.apache.org/jira/browse/SPARK-6286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357310#comment-14357310 ] Iulian Dragos commented on SPARK-6286: -- Good point. It's been [introduced in 0.21.0|http://mesos.apache.org/blog/mesos-0-21-0-released/]. According to [pom.xml|https://github.com/apache/spark/blob/master/pom.xml#L119], Spark depends on `0.21.0`, so it seems safe to handle it. Feel free to close if you think it's going to break something else. > Handle TASK_ERROR in TaskState > -- > > Key: SPARK-6286 > URL: https://issues.apache.org/jira/browse/SPARK-6286 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Iulian Dragos >Priority: Minor > Labels: mesos > > Scala warning: > {code} > match may not be exhaustive. It would fail on the following input: TASK_ERROR > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6286) Handle TASK_ERROR in TaskState
[ https://issues.apache.org/jira/browse/SPARK-6286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358905#comment-14358905 ] Iulian Dragos commented on SPARK-6286: -- Sure, I'll issue a PR for handling {{TASK_ERROR => TASK_LOST}} > Handle TASK_ERROR in TaskState > -- > > Key: SPARK-6286 > URL: https://issues.apache.org/jira/browse/SPARK-6286 > Project: Spark > Issue Type: Bug > Components: Spark Core >Reporter: Iulian Dragos >Priority: Minor > Labels: mesos > > Scala warning: > {code} > match may not be exhaustive. It would fail on the following input: TASK_ERROR > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6285) Duplicated code leads to errors
[ https://issues.apache.org/jira/browse/SPARK-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360937#comment-14360937 ] Iulian Dragos commented on SPARK-6285: -- Thanks, [~lian cheng] > Duplicated code leads to errors > --- > > Key: SPARK-6285 > URL: https://issues.apache.org/jira/browse/SPARK-6285 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.3.0 >Reporter: Iulian Dragos >Assignee: Cheng Lian > > The following class is duplicated inside > [ParquetTestData|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTestData.scala#L39] > and > [ParquetIOSuite|https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetIOSuite.scala#L44], > with exact same code and fully qualified name: > {code} > org.apache.spark.sql.parquet.TestGroupWriteSupport > {code} > The second one was introduced in > [3b395e10|https://github.com/apache/spark/commit/3b395e10510782474789c9098084503f98ca4830], > but even though it mentions that `ParquetTestData` should be removed later, > I couldn't find a corresponding Jira ticket. > This duplicate class causes the Eclipse builder to fail (since src/main and > src/test are compiled together in Eclipse, unlike Sbt). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4631) Add real unit test for MQTT
[ https://issues.apache.org/jira/browse/SPARK-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295338#comment-14295338 ] Iulian Dragos commented on SPARK-4631: -- I'll have a go at fixing it. > Add real unit test for MQTT > > > Key: SPARK-4631 > URL: https://issues.apache.org/jira/browse/SPARK-4631 > Project: Spark > Issue Type: Test > Components: Streaming >Reporter: Tathagata Das >Priority: Critical > Fix For: 1.3.0 > > > A real unit test that actually transfers data to ensure that the MQTTUtil is > functional -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4631) Add real unit test for MQTT
[ https://issues.apache.org/jira/browse/SPARK-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295316#comment-14295316 ] Iulian Dragos commented on SPARK-4631: -- This test exhibits a race condition. MQTTStreamSuite fails both on my machine and on an EC2 instance with the message below (but passes if I add a `Thread.sleep(50)` inside `publishData`). See my [comment on the PR|https://github.com/apache/spark/pull/3844#issuecomment-71852913]. {code} [info] - mqtt input stream *** FAILED *** (552 milliseconds) [info] org.eclipse.paho.client.mqttv3.MqttException: Too many publishes in progress {code} The issue comes from the hard-coded limit of 10 in-flight messages in the MQTT client. This test produces too much data too fast on some machines. > Add real unit test for MQTT > > > Key: SPARK-4631 > URL: https://issues.apache.org/jira/browse/SPARK-4631 > Project: Spark > Issue Type: Test > Components: Streaming >Reporter: Tathagata Das >Priority: Critical > Fix For: 1.3.0, 1.2.1 > > > A real unit test that actually transfers data to ensure that the MQTTUtil is > functional -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4631) Add real unit test for MQTT
[ https://issues.apache.org/jira/browse/SPARK-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296790#comment-14296790 ] Iulian Dragos commented on SPARK-4631: -- [~advancedxy], I see the same exception in the logs. It's funny, but as I mentioned in my github account, things start working if I add a `Thread.sleep(50)` inside the loop, or a longer one before the call to {{publishData}}. On my machine it takes around 250ms for the Receiver to install its MQTT callback, and I think that's where the race condition is. If {{publishData}} finishes before that, no message is ever received on the Spark receiver end. I don't know much about MQTT, so I don't know if the service is supposed to queue those 100 messages and deliver them eventually. > Add real unit test for MQTT > > > Key: SPARK-4631 > URL: https://issues.apache.org/jira/browse/SPARK-4631 > Project: Spark > Issue Type: Test > Components: Streaming >Reporter: Tathagata Das >Priority: Critical > Fix For: 1.3.0 > > > A real unit test that actually transfers data to ensure that the MQTTUtil is > functional -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4631) Add real unit test for MQTT
[ https://issues.apache.org/jira/browse/SPARK-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296840#comment-14296840 ] Iulian Dragos commented on SPARK-4631: -- [~Bilna], do you also see the {{java.net.ProtocolException}} if you run the tests with logging? ({{-Dorg.apache.logging.log4j.level=DEBUG}}) > Add real unit test for MQTT > > > Key: SPARK-4631 > URL: https://issues.apache.org/jira/browse/SPARK-4631 > Project: Spark > Issue Type: Test > Components: Streaming >Reporter: Tathagata Das >Priority: Critical > Fix For: 1.3.0 > > > A real unit test that actually transfers data to ensure that the MQTTUtil is > functional -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4631) Add real unit test for MQTT
[ https://issues.apache.org/jira/browse/SPARK-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297081#comment-14297081 ] Iulian Dragos commented on SPARK-4631: -- I opened [PR 4270|https://github.com/apache/spark/pull/4270], it would be good if someone had a look at it. [~Bilna], since you are the author of this code it would be particularly useful to have you look at it too. > Add real unit test for MQTT > > > Key: SPARK-4631 > URL: https://issues.apache.org/jira/browse/SPARK-4631 > Project: Spark > Issue Type: Test > Components: Streaming >Reporter: Tathagata Das >Priority: Critical > Fix For: 1.3.0 > > > A real unit test that actually transfers data to ensure that the MQTTUtil is > functional -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org