Re: Task slot sharing: force reallocation

Hi Le, at the moment Flink does not allow the user to specify how the individual task a being deployed (spread out across all TMs vs. filling up a TM before using another). At the moment, the current implementation uses the second strategy. That's why you see all sources being deployed to the

Re: Task slot sharing: force reallocation

Thanks Till. I switched to Flink 1.7.1 and it seems to solve part of my problem (the downstream operator does not seem to sit on the same machine anymore). But the new problem is that does Flink implicitly set all sub tasks of data sources generated by RichParallelFunction to be inside the same

Re: Task slot sharing: force reallocation

The community no longer actively supports Flink < 1.6. Therefore I would try out whether you can upgrade to one of the latest versions. However, be aware that we reworked Flink's distributed architecture which slightly affected the scheduling behavior. In your case, it should actually be

Re: Task slot sharing: force reallocation

1.3.2 -- should I update to the latest version? Thanks, Le On Wed, Mar 6, 2019 at 4:24 AM Till Rohrmann wrote: > Which version of Flink are you using? > > On Tue, Mar 5, 2019 at 10:58 PM Le Xu wrote: > >> Hi Till: >> >> Thanks for the reply. The setup of the jobs is roughly as follows: For a

Re: Task slot sharing: force reallocation

Which version of Flink are you using? On Tue, Mar 5, 2019 at 10:58 PM Le Xu wrote: > Hi Till: > > Thanks for the reply. The setup of the jobs is roughly as follows: For a > cluster with N machines, we deploy X simple map/reduce style jobs (the job > DAG and settings are exactly the same, except

Re: Task slot sharing: force reallocation

Hi Till: Thanks for the reply. The setup of the jobs is roughly as follows: For a cluster with N machines, we deploy X simple map/reduce style jobs (the job DAG and settings are exactly the same, except they consumes different data). Each job has N mappers (they are evenly distributed, one mapper

Re: Task slot sharing: force reallocation

Hard to tell whether this is related to FLINK-11815. To me the setup is not fully clear. Let me try to sum it up: According to Le Xu's description there are n jobs running on a session cluster. I assume that every TaskManager has n slots. The observed behaviour is that every job allocates the

Re: Task slot sharing: force reallocation

Hi Le, As I wrote, you can try running Flink in job mode, which spawns separate clusters per each job. Till, is this issue covered by FLINK-11815 ? Is this the same as: > Known issues: > 1. (…) > 2. if task slots are registered before slot

Re: Task slot sharing: force reallocation

Thanks Piotr. I didn't realize that the email attachment isn't working so the example I was referring to was this figure from Flink website: https://ci.apache.org/projects/flink/flink-docs-stable/fig/slot_sharing.svg So I try to run multiple jobs concurrently in a cluster -- the jobs are

Re: Task slot sharing: force reallocation

Hi, Are you asking the question if that’s the behaviour or you have actually observed this issue? I’m not entirely sure, but I would guess that the Sink tasks would be distributed randomly across the cluster, but maybe I’m mixing this issue with resource allocations for Task Managers. Maybe