Re: Duplicate task IDs

2016-12-12 Thread Neil Conway
equire adding a new data structure, BoundedMultiHashMap (https://reviews.apache.org/r/54178/). That seems like overkill to me, for now. > It would also be unfortunate if we said we were dis-allowing duplicate task > ids but only catch some of the manifestations. Definitely unfortunate, but I

Re: Duplicate task IDs

2016-12-12 Thread Joris Van Remoortere
It sounds like using a multi_hashmap for now allows you to clean up the code and avoid some bugs, without changing the existing behavior. I agree that we would want a deprecation period if we changed the behavior. It would also be unfortunate if we said we were dis-allowing duplicate task ids but

Re: Duplicate task IDs

2016-12-12 Thread Neil Conway
Hi Joris, Fair point: I didn't deliberately set out to change the behavior for duplicate task IDs. Rather, it was a consequence of switching from boost::circular_buffer to using a hashmap for managing completed tasks. Using a hashmap has a few minor advantages [1], but we can certainly con

Re: Duplicate task IDs

2016-12-11 Thread Alex Rukletsov
> > I concur that using duplicate task IDs is bad practice and asking for > trouble. > > Could you please clarify *why* you want to use a hashmap? Is your goal to > remove duplicate task IDs or is this just a side-effect and you have a > different reason (e.g. performance) for us

Re: Duplicate task IDs

2016-12-09 Thread Joris Van Remoortere
Hey Neil, I concur that using duplicate task IDs is bad practice and asking for trouble. Could you please clarify *why* you want to use a hashmap? Is your goal to remove duplicate task IDs or is this just a side-effect and you have a different reason (e.g. performance) for using a hashmap? I&#

Duplicate task IDs

2016-12-09 Thread Neil Conway
Folks, The master stores a cache of metadata about recently completed tasks; for example, this information can be accessed via the "/tasks" HTTP endpoint or the "GET_TASKS" call in the new Operator API. The master currently stores this metadata using a list; this means tha