[jira] [Created] (SPARK-6287) Add support for dynamic allocation in the Mesos coarse-grained scheduler

2015-03-11 Thread Iulian Dragos (JIRA) 
Iulian Dragos created SPARK-6287:


 Summary: Add support for dynamic allocation in the Mesos 
coarse-grained scheduler
 Key: SPARK-6287
 URL: https://issues.apache.org/jira/browse/SPARK-6287
 Project: Spark
  Issue Type: Bug
  Components: Mesos
Reporter: Iulian Dragos


Add support inside the coarse-grained Mesos scheduler for dynamic allocation. 
It amounts to implementing two methods that allow scaling up and down the 
number of executors:

{code}
def doKillExecutors(executorIds: Seq[String])
def doRequestTotalExecutors(requestedTotal: Int)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-6285) Duplicated code leads to errors

2015-03-11 Thread Iulian Dragos (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357297#comment-14357297
 ] 

Iulian Dragos commented on SPARK-6285:
--

According to the git commit message that introduced the duplicate:

{quote}
 To avoid potential merge conflicts, old testing code are not removed yet. The 
following classes can be safely removed after most Parquet related PRs are 
handled:

- `ParquetQuerySuite`
- `ParquetTestData`
{quote}

I mentioned the Eclipse build problem in passing, but I can expand: the class 
*is* a duplicated name, so the Scala compiler is correct in refusing it. It 
only compiles in Sbt/Maven because the src/main and src/test are compiled in 
separate compiler runs, and scalac seems to not notice the duplicate name when 
it comes from bytecode. Eclipse builds src/main and src/test together, and when 
both classes originate from sources scalac issues an error message.


> Duplicated code leads to errors
> ---
>
> Key: SPARK-6285
> URL: https://issues.apache.org/jira/browse/SPARK-6285
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.3.0
>Reporter: Iulian Dragos
>
> The following class is duplicated inside 
> [ParquetTestData|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTestData.scala#L39]
>  and 
> [ParquetIOSuite|https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetIOSuite.scala#L44],
>  with exact same code and fully qualified name:
> {code}
> org.apache.spark.sql.parquet.TestGroupWriteSupport
> {code}
> The second one was introduced in 
> [3b395e10|https://github.com/apache/spark/commit/3b395e10510782474789c9098084503f98ca4830],
>  but even though it mentions that `ParquetTestData` should be removed later, 
> I couldn't find a corresponding Jira ticket.
> This duplicate class causes the Eclipse builder to fail (since src/main and 
> src/test are compiled together in Eclipse, unlike Sbt).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-6286) Handle TASK_ERROR in TaskState

2015-03-12 Thread Iulian Dragos (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-6286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357310#comment-14357310
 ] 

Iulian Dragos commented on SPARK-6286:
--

Good point. It's been [introduced in 
0.21.0|http://mesos.apache.org/blog/mesos-0-21-0-released/]. According to 
[pom.xml|https://github.com/apache/spark/blob/master/pom.xml#L119], Spark 
depends on `0.21.0`, so it seems safe to handle it. Feel free to close if you 
think it's going to break something else.

> Handle TASK_ERROR in TaskState
> --
>
> Key: SPARK-6286
> URL: https://issues.apache.org/jira/browse/SPARK-6286
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Reporter: Iulian Dragos
>Priority: Minor
>  Labels: mesos
>
> Scala warning:
> {code}
> match may not be exhaustive. It would fail on the following input: TASK_ERROR
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-6286) Handle TASK_ERROR in TaskState

2015-03-12 Thread Iulian Dragos (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-6286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358905#comment-14358905
 ] 

Iulian Dragos commented on SPARK-6286:
--

Sure, I'll issue a PR for handling {{TASK_ERROR => TASK_LOST}}

> Handle TASK_ERROR in TaskState
> --
>
> Key: SPARK-6286
> URL: https://issues.apache.org/jira/browse/SPARK-6286
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Reporter: Iulian Dragos
>Priority: Minor
>  Labels: mesos
>
> Scala warning:
> {code}
> match may not be exhaustive. It would fail on the following input: TASK_ERROR
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-6285) Duplicated code leads to errors

2015-03-13 Thread Iulian Dragos (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360937#comment-14360937
 ] 

Iulian Dragos commented on SPARK-6285:
--

Thanks, [~lian cheng]

> Duplicated code leads to errors
> ---
>
> Key: SPARK-6285
> URL: https://issues.apache.org/jira/browse/SPARK-6285
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.3.0
>Reporter: Iulian Dragos
>Assignee: Cheng Lian
>
> The following class is duplicated inside 
> [ParquetTestData|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTestData.scala#L39]
>  and 
> [ParquetIOSuite|https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetIOSuite.scala#L44],
>  with exact same code and fully qualified name:
> {code}
> org.apache.spark.sql.parquet.TestGroupWriteSupport
> {code}
> The second one was introduced in 
> [3b395e10|https://github.com/apache/spark/commit/3b395e10510782474789c9098084503f98ca4830],
>  but even though it mentions that `ParquetTestData` should be removed later, 
> I couldn't find a corresponding Jira ticket.
> This duplicate class causes the Eclipse builder to fail (since src/main and 
> src/test are compiled together in Eclipse, unlike Sbt).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-4631) Add real unit test for MQTT

2015-01-28 Thread Iulian Dragos (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295338#comment-14295338
 ] 

Iulian Dragos commented on SPARK-4631:
--

I'll have a go at fixing it.

> Add real unit test for MQTT 
> 
>
> Key: SPARK-4631
> URL: https://issues.apache.org/jira/browse/SPARK-4631
> Project: Spark
>  Issue Type: Test
>  Components: Streaming
>Reporter: Tathagata Das
>Priority: Critical
> Fix For: 1.3.0
>
>
> A real unit test that actually transfers data to ensure that the MQTTUtil is 
> functional



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-4631) Add real unit test for MQTT

2015-01-28 Thread Iulian Dragos (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295316#comment-14295316
 ] 

Iulian Dragos commented on SPARK-4631:
--

This test exhibits a race condition.

MQTTStreamSuite fails both on my machine and on an EC2 instance with the 
message below (but passes if I add a `Thread.sleep(50)` inside `publishData`). 
See my [comment on the 
PR|https://github.com/apache/spark/pull/3844#issuecomment-71852913].

{code}
[info] - mqtt input stream *** FAILED *** (552 milliseconds)
[info]   org.eclipse.paho.client.mqttv3.MqttException: Too many publishes in 
progress
{code}

The issue comes from the hard-coded limit of 10 in-flight messages in the MQTT 
client. This test produces too much data too fast on some machines.

> Add real unit test for MQTT 
> 
>
> Key: SPARK-4631
> URL: https://issues.apache.org/jira/browse/SPARK-4631
> Project: Spark
>  Issue Type: Test
>  Components: Streaming
>Reporter: Tathagata Das
>Priority: Critical
> Fix For: 1.3.0, 1.2.1
>
>
> A real unit test that actually transfers data to ensure that the MQTTUtil is 
> functional



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-4631) Add real unit test for MQTT

2015-01-29 Thread Iulian Dragos (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296790#comment-14296790
 ] 

Iulian Dragos commented on SPARK-4631:
--

[~advancedxy], I see the same exception in the logs. It's funny, but as I 
mentioned in my github account, things start working if I add a 
`Thread.sleep(50)` inside the loop, or a longer one before the call to 
{{publishData}}. On my machine it takes around 250ms for the Receiver to 
install its MQTT callback, and I think that's where the race condition is. If 
{{publishData}} finishes before that, no message is ever received on the Spark 
receiver end.

I don't know much about MQTT, so I don't know if the service is supposed to 
queue those 100 messages and deliver them eventually.

> Add real unit test for MQTT 
> 
>
> Key: SPARK-4631
> URL: https://issues.apache.org/jira/browse/SPARK-4631
> Project: Spark
>  Issue Type: Test
>  Components: Streaming
>Reporter: Tathagata Das
>Priority: Critical
> Fix For: 1.3.0
>
>
> A real unit test that actually transfers data to ensure that the MQTTUtil is 
> functional



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-4631) Add real unit test for MQTT

2015-01-29 Thread Iulian Dragos (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296840#comment-14296840
 ] 

Iulian Dragos commented on SPARK-4631:
--

[~Bilna], do you also see the {{java.net.ProtocolException}} if you run the 
tests with logging? ({{-Dorg.apache.logging.log4j.level=DEBUG}})

> Add real unit test for MQTT 
> 
>
> Key: SPARK-4631
> URL: https://issues.apache.org/jira/browse/SPARK-4631
> Project: Spark
>  Issue Type: Test
>  Components: Streaming
>Reporter: Tathagata Das
>Priority: Critical
> Fix For: 1.3.0
>
>
> A real unit test that actually transfers data to ensure that the MQTTUtil is 
> functional



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-4631) Add real unit test for MQTT

2015-01-29 Thread Iulian Dragos (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297081#comment-14297081
 ] 

Iulian Dragos commented on SPARK-4631:
--

I opened [PR 4270|https://github.com/apache/spark/pull/4270], it would be good 
if someone had a look at it. [~Bilna], since you are the author of this code it 
would be particularly useful to have you look at it too.

> Add real unit test for MQTT 
> 
>
> Key: SPARK-4631
> URL: https://issues.apache.org/jira/browse/SPARK-4631
> Project: Spark
>  Issue Type: Test
>  Components: Streaming
>Reporter: Tathagata Das
>Priority: Critical
> Fix For: 1.3.0
>
>
> A real unit test that actually transfers data to ensure that the MQTTUtil is 
> functional



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

<    1   2