subject:"\[GitHub\] spark pull request\: \[streaming\] SPARK\-10955. Disable dynamic alloc..."

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145966530
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145966515
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145968060
  
Actually looking at this again, I think our only option is to log a 
message. It is possible that the `SparkContext` was already created and passed 
to us, in which case we can't do anything about dynamic allocation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

2015-10-06 Thread JoshRosen

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145972319
  
/cc @andrewor14 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145968432
  
  [Test build #43289 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43289/consoleFull)
 for   PR 8998 at commit 
[`a4a5212`](https://github.com/apache/spark/commit/a4a521273114d3ac485592f5c4408adcc11bfc45).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

GitHub user harishreedharan opened a pull request:

https://github.com/apache/spark/pull/8998

[streaming] SPARK-10955. Disable dynamic allocation for Streaming appâ¦

â¦lications.

Dynamic allocation can be painful for streaming apps and can lose data. The 
one drawback though
is that apps which run Streaming and non-streaming form the same context 
will also end up not
being able to do dynamic allocation.

Another option would be to log a warning.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/harishreedharan/spark ss-log-error

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8998.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8998


commit a4a521273114d3ac485592f5c4408adcc11bfc45
Author: Hari Shreedharan 
Date:   2015-10-06T18:58:39Z

[streaming] SPARK-10955. Disable dynamic allocation for Streaming 
applications.

Dynamic allocation can be painful for streaming apps and can lose data. The 
one drawback though
is that apps which run Streaming and non-streaming form the same context 
will also end up not
being able to do dynamic allocation.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145966395
  
/cc @vanzin @tdas 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145968943
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145968956
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145983548
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43290/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145983545
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145997044
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145997065
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145983265
  
  [Test build #43290 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43290/console)
 for   PR 8998 at commit 
[`b3f9a95`](https://github.com/apache/spark/commit/b3f9a95542cf07f6d3c389d295f23fc9b09b945e).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145996934
  
Added config parameter to enable it if the user really wants to enable 
dynamic allocation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145980725
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43289/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145980547
  
  [Test build #43289 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43289/console)
 for   PR 8998 at commit 
[`a4a5212`](https://github.com/apache/spark/commit/a4a521273114d3ac485592f5c4408adcc11bfc45).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145980723
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

2015-10-06 Thread vanzin

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145985862
  
Yeah, looks fine to me. Perhaps an internal config option so that the check 
can be disabled? e.g. if someone wants to actually try it out and see how it 
behaves without having to recompile Spark.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-145999062
  
  [Test build #43299 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43299/consoleFull)
 for   PR 8998 at commit 
[`725f090`](https://github.com/apache/spark/commit/725f0900b708f393570f2e9fd50f4f33f4b352ee).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-146012980
  
I don't have a strong preference for the config name. @vanzin, @andrewor14 
- like the current name or the one which @markgrover suggested? Vote please :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

2015-10-06 Thread tdas

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-146016788
  
Also please update the PR title to be similar to other PRs. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-146017855
  
As I said, I don't mind either. But in this case, we need to be 
conservative. We can additional checks to see if WAL is enabled, but it is not 
possible to actually check what DStreams are being used (it could be a custom 
one, which does not break if executors go away), so that case still exists. 

How about we print the config param out or document it in that case? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

2015-10-06 Thread vanzin

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-146007357
  
> but we should document this new property 

If you document it, it becomes public, and you need to document that you 
removed it, and all sorts of things. Since the point is not to publicize it, 
instead just have a way for those tinkering with Spark itself to enable the 
feature, documentation is not necessary.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-146011686
  
  [Test build #43299 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43299/console)
 for   PR 8998 at commit 
[`725f090`](https://github.com/apache/spark/commit/725f0900b708f393570f2e9fd50f4f33f4b352ee).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-146011888
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43299/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-146011885
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

2015-10-06 Thread tdas

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-146016637

I strongly think that we should not add that require. It can break existing
streaming applications very easily. People might be using dynamic allocation
with direct kafka or kinesis or with anything + WAL, in which case it is fine
to have dynamic allocation as the system will not loose data. Upgrading Spark
will immediately break their application, and it might be hard to figure out
the solution for disabling dynamic allocation in their setting.

We have made mistakes in the past by throwing errors like this for usecases
which we didnt think would be useful in production, and people have complained
that it broke their testing stuff. See
https://issues.apache.org/jira/browse/SPARK-8630 and discussion
https://www.mail-archive.com/search?l=u...@spark.apache.org=subject:%22Re%5C%3A+QueueStream+Does+Not+Support+Checkpointing%22=newest=1

We must not break stuff. So I suggest this be turned into a warning. In
that case we dont need a new configuration.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...

2015-10-06 Thread markgrover

Github user markgrover commented on the pull request:

https://github.com/apache/spark/pull/8998#issuecomment-146006957
  
Pointing out the obvious here, but we should document this new property so 
someone doesn't have to read the source code to figure out what's the name of 
the property to enable dynamic allocation in streaming.

Also, I personally prefer the name 
`spark.streaming.dynamicAllocation.allowed` over 
`spark.streaming.dynamicAllocation.enabled` because one will still have to 
enable dynamic allocation for the entire spark context separately. And, as much 
as I don't like the idea of adding yet another property I don't really see a 
better way to do this, essentially we want to decouple folks from enabling 
dynamic allocation for streaming vs. non-streaming workloads. Currently, we use 
the same property, we could decouple by having 2 separate properties - one for 
streaming and one for the rest, or, alternatively, as is being suggested here, 
have a second property to allow/disallow dynamic allocation for streaming. The 
latter seems less hairy so I am for it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [streaming] SPARK-10955. Disable dynamic alloc...