Thanks a lot for responding, Randall! See my comments below.
Regards, Per Steffensen
On 22/05/17 22:36, Randall Hauch wrote:
You're not doing anything wrong, but I suspect you're requesting task
reconfiguration more frequently than was originally envisioned, which
means that the current implementation is not as optimal for your case.
OK thanks for confirming
I'm not sure how much effort is required to implement this new
behavior. The logic for the standalone worker is pretty
straightforward, but the logic for the distributed worker is going to
be much more involved.
Yeah, when I realized the "problem" I had a short look at the code to
see if it was easily fixable. I never went deep into it, but it seem
like more than just an hour of work.
But we also need to be careful about changing existing behavior, since
it's not hard to imagine connectors that might expect that all tasks
be restarted when there are any changes to the task configurations.
FWIW, I think it is a little hard to imagine :-)
If there's any potential that this is the case, we'd have to be sure
to keep the existing behavior as the default but to somehow enable the
new behavior if desired.
I definitely agree!
One possibility is to add an overloaded
requestTaskReconfiguration(boolean changedOnly) that specifies whether
only changed tasks should be reconfigured. This way the existing
requestTaskReconfiguration() method could be changed to call
requestTaskReconfiguration(false), and then the implementation has to
deal with this.
Yep, or make is a optional standard-configuration that you can always
give a connector. Potato, potato
But again, the bigger challenge is to implement this new behavior in
the DistributedHerder. OTOH, perhaps it's not as complicated as I
might guess.
Well I would really like to see it happen. Anyone up for it? Am I
allowed to create a ticket on this?
What if I would like to give it a shot myself. Is there a committer that
would help review and eventually commit?
Which branch should I make a PR to?
