Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1485#issuecomment-50545882
Hey there - as Aaron said, the executors should never have more than N
tasks active if there are N cores. I think there might be a bug causing this.
So I'd recommend we
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1485
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user fireflyc commented on the pull request:
https://github.com/apache/spark/pull/1485#issuecomment-49501533
My program is spark streaming over Hadoop yarn.It work for user click
stream.
I read code,number of worker threads and block?
---
If your project is set up for it,
Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/1485#issuecomment-49526386
@fireflyc Spark should not be scheduling more than N concurrent tasks on an
Executor. It appears that the tasks may be returning success but then don't
actually return
GitHub user fireflyc opened a pull request:
https://github.com/apache/spark/pull/1485
Fixed the number of worker thread
There are a lot of input Block cause too many Worker threads and will
load all data.So it should be to control the number of Worker threads
You can merge this
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1485#issuecomment-49443851
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/1485#issuecomment-49444796
Slightly bigger point: both the 'fixed' and 'cached' executors from
`Executors` have some drawbacks:
- 'fixed' always keeps the given number of threads active
Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/1485#issuecomment-49494194
The tasks launched on an Executor are controlled by the DAGScheduler, and
should not exceed the number of cores that executor is advertising. In what
situation have you
Github user fireflyc commented on the pull request:
https://github.com/apache/spark/pull/1485#issuecomment-49495043
My application have 1000+ Worker Threads.
