Daryn Sharp created HADOOP-14033:
------------------------------------
Summary: Reduce fair call queue lock contention
Key: HADOOP-14033
URL: https://issues.apache.org/jira/browse/HADOOP-14033
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Affects Versions: 2.7.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Under heavy load the call queue may run dry yet clients experience high latency.
The fcq requires producers and consumers to sync via a shared lock. Polling
consumers hold the lock while scanning all sub-queues. Consumers are
serialized despite the sub-queues being thread-safe blocking queues. The
effect is to cause other producers/consumers to frequently park.
The lock is unfair, so producers/consumers attempt to barge in on the lock.
The outnumbered producers tend to remain blocked for an extended time. As load
increases and the queues fill, the barging consumers drain the queues faster
than the producers can fill it.
Server metrics provide an illusion of healthy throughput, response time, and
call queue length due to starvation on the ingress. Often as the load gets
worse, the server looks better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
