Github user fpompermaier commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-77586066
That would be awesome :)
I think you could talk with Markus about the Dopa scheduler..propably it's
a closed project but it could be a source of inputs to create a
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-77569539
I think it would be definitely good to have something like a job submission
queue, that accepts jobs and executes them as soon as enough as enough
resource become av
Github user fpompermaier commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-77537609
I know that in stratosphere there was an effort to write a job scheduler,
do you think that such a thing could be valuable for the future or are you
going to rely onl
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-77536080
At the moment, this is not supported yet. The easiest way to execute
multiple jobs concurrently is to start each job in a separate Flink cluster
running on YARN
Github user fpompermaier commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-77533354
That's true but what if there's not enough resources? Is there any policy
to retry the job submission automatically or give priority to waiting/queued
ones?
---
If
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-77530332
Hey,
Flink already supports running multiple jobs in parallel.
If you have 50 slots available, you can run two jobs requiring 25 slots.
The webfrontend is not r
Github user fpompermaier commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-77529565
Hi to all,
I was reading this interesting thread..is there any change that
multi-user/multi-job scenarios will come into play sooner or later? Or do you
just rely
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-77239086
Thank you for merging it!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have th
Github user StephanEwen closed the pull request at:
https://github.com/apache/flink/pull/410
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-77205854
Manually merged into `release-0.8` in
a6f9f9939ca03026baeefb3bd0876b90068b7682
---
If your project is set up for it, you can reply to this email and have your
reply a
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-76978687
@rmetzger I don't see a reason why this should not go to master as well.
After all, it's optional and quite useful if you want to run a job on the full
cluster with as many av
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-76974155
Ping ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enable
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-75542112
Are there any plans to merge this to master as well?
I need this feature to implement a testcase.
---
If your project is set up for it, you can reply to this email an
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-74661555
But currently the system does not support multi-user/multi-job scenarios so
well either. If I'm not mistaken, then the scheduler schedules the tasks
eagerly whi
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-74651790
+1
I think setup via `ExecutionConfig` is the way to go.
I agree with @rmetzger that we should merge it to master as well. The
important thing is to document
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-74643785
I agree with Fabian that it is not a good default behavior to grab
everything that is possible.
It should be an explicit request by the user. For YARN single job se
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-74643360
Using max parallelism basically prohibits to run more than one program at a
time. I don't think that would be a good default mode.
---
If your project is set up for it,
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/410#discussion_r24804242
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/JobManager.java
---
@@ -374,6 +375,8 @@ public JobSubmissionResult submitJob(JobGraph jo
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-74642763
Right now, the user has to set the parallelism to
`ExecutionConfig.PARALLELISM_AUTO_MAX`. Why not use all available task slots by
default? I understand, that we shouldn't simp
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-74639308
The user has to enable the auto parallelism explicitly, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-74638095
Cool.
Lets merge this also to master and document it there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/410#issuecomment-74637789
Looks good. That way, we can help users by preventing them from running
programs with the default degree of parallelism (=1) if more task slots are
available.
---
If your pr
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/410
Add auto-parallelism to Jobs (0.8 branch)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink autopar
Alternatively
23 matches
Mail list logo
