mcvsubbu commented on issue #5960:
URL:
https://github.com/apache/incubator-pinot/issues/5960#issuecomment-693104377
It is useful add some notes here on this.
The reason the semaphore was put in place was because Kafka consumers had an
issue with dynamically creating new kafka metrics when a new consumer was
created on the same partition while keeping the old consumer open. This led to
metrics proliferation.
From kafka (or any other stream) point of view, if they are trying to emit
per-partition consumer metrics, it probably makes sense that they add different
metrics for each consumer so as to get the health of the system correctly. So,
we need to handle it somehow.
One way could be that when we implement an internal "queue" of state
transitions, holding the consuming transition until the ONLINE transition is
received. Of course, we will have to respond back OK for the consuming state
transition, but just not allow our software to handle it. A little tricky to
implement, and sounds somewhat hacky to me. Also has a downside that we will be
responding OK without actually handling the state transition. Helix does not
provide a way to mark a transition in ERROR from the participant. We could ask
for that feature (useful for other things as well), and then we can mark the
consumer in ERROR state if it later fails to transition correctly.
A simple work-around in the short/medium term may be we stop realtime table
creation in the controller if the max-threads is not set to at least 2 (we
don't expect to have more than one outstanding state transition for the same
partition).
I am open to hearing other ideas on how to fix the deadlock.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
