Hi Ufuk - thanks for the 2016 roadmap - glad to see changing parallelism is the first bullet :) Mesos support also sounds great, we're currently running job and task managers on Mesos statically via Marathon.
Hi Stephan - thanks, that trick sounds pretty clever, I will try wrapping my head around using 2 different jobs and uids like that. -Zach On Thu, Feb 18, 2016 at 7:13 AM Stephan Ewen <se...@apache.org> wrote: > Hi Zach! > > Yes, changing parallelism is pretty high up the priority list. The good > news is that "scaling in" is the simpler part of changing the parallelism > and we are pushing to get that in soon. > > > Until then, there is only a pretty ugly trick that you can do right now to > "rescale' the state: > > 1) savepoint with high parallelism > > 2) run an intermediate job that has the state twice in two operators, > once with high parallelism, once with low. Emit the state from the first > operator, write in the second. The first operator has the operator ID of > the initial high-parallelism state. > > 3) Run the low parallelism job, and the stateful operator needs the ID > of the second (low parallelism) operator in the intermediate job. > > > Greetings, > Stephan > > > On Thu, Feb 18, 2016 at 9:24 AM, Ufuk Celebi <u...@apache.org> wrote: > >> Hey Zach! >> >> Sounds like a great use case. >> >> On Wed, Feb 17, 2016 at 3:16 PM, Zach Cox <zcox...@gmail.com> wrote: >> > However, the savepoint docs state that the job parallelism cannot be >> changed >> > over time [1]. Does this mean we need to use the same, fixed >> parallelism=n >> > during reprocessing and going forward? Are there any tricks or >> workarounds >> > we could use to still make changes to parallelism and take advantage of >> > savepoints? >> >> Yes, currently you have to keep the parallelism fixed. Dynamic scale >> in and out of programs will have very high priority after the 1.0 >> release [1]. Unfortunately, I'm not aware of any work arounds to >> overcome this at the moment. >> >> – Ufuk >> >> [1] https://flink.apache.org/news/2015/12/18/a-year-in-review.html (at >> the end of the post there is a road map for 2016) >> > >
