Shrihari created TS-4735:
----------------------------
Summary: Possible deadlock on traffic_server startup
Key: TS-4735
URL: https://issues.apache.org/jira/browse/TS-4735
Project: Traffic Server
Issue Type: Bug
Components: Core
Reporter: Shrihari
As part of startup, traffic_server creates two threads (to begin with).
1. (main) Thread (1) blocks till its signaled by another thread
1. Thread 2 polls for messages from traffic_manager
It is waiting for a message from traffic_manager which contains all the
configuration required for it to go ahead with initialization. Hence, it is
critical that the main Thread (1) wait till it gets the configuration.
Thread 2 which polls for message from traffic_manager works like this:
for(;;) {
if (pmgmt->require_lm) { <--- Always True (when using traffic_cop)
pmgmt->pollLMConnection(); <--- | for (count = 0; count < 10000; count ++)
| num =
mgmt_read_timeout(...) <---- Blocking call. returns 0 if nothing was received
for 1 second
| if !num: break
<--- Break out of the loop and return from function
| else: read(fd),
add_to_event_queue, continue the loop,
| Back to fetching
another message
}
pmgmt->processEventQueue(); <-- Process the messages received in
pollLMConnection()
pmgmt->processSignalQueue();
mgmt_sleep_sec(pmgmt->timeout);
}
RCA:
There are two problems here:
1. If we look into the above code, we should observe that the pollLMConnection
might not return back for a very long time if it keeps getting messages. As a
result, we may not call processEventQueue() which processes the received
messages. And unless we process the messages, we cannot signal the main Thread
(1) to continue, which is still blocked. Hence we see the issue where
traffic_server won't complete initialization for a very long time.
2. The second problem is that why is traffic_server receiving so many messages
at boot-up? The problem lies in the configuration. In 6.2.x, we replaced
'proxy.process.ssl.total_success_handshake_count' with
'proxy.process.ssl.total_success_handshake_count_in'.
In order to provide backwards compatibility, we defined the old stat in
stats.config.xml. The caveat here is that, since this statconfig is defined in
stats.config.xml, traffic_manager assumes the responsibility of updating this
stat. According to the code:
if (i_am_not_owner_of(stat)) : send traffic_server a notify message.
Ideally, this code should not be triggered because, traffic_manager does own
the stat. However, the ownership in the code is determined solely based on the
'string name'. If the name contains 'process', it is owned by traffic_server.
This leads to an interesting scenario where traffic_manger keeps updating its
own stat and sends unnecessary events to traffic_server. These updates happen
every 1 second (Thanks James for helping me understand this period) which is
the same as our timeout in traffic_server. Due to 'Problem 1' we can prevent
traffic_server from processing any messages for up to 10,000 seconds! (Just
imagine the case where the message is received just before the timout of 1
second happens)
I saw this happening with 100% on a VM but 0% on a physical box. I don't have
any other results as of now though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
