[ https://issues.apache.org/jira/browse/TS-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leif Hedstrom resolved TS-1346. ------------------------------- Resolution: Won't Fix Fix Version/s: (was: 5.2.0) > syn cookie may crash config update in cluster > --------------------------------------------- > > Key: TS-1346 > URL: https://issues.apache.org/jira/browse/TS-1346 > Project: Traffic Server > Issue Type: Bug > Components: Clustering, Manager > Affects Versions: 3.3.1, 3.0.5 > Reporter: Zhao Yongming > Assignee: Bin Chen > Labels: Crash > > in clustering, we have about 48 boxes, where the config sync and others may > trigger the 8088 syn cookie or so. which may case the manager killed: > {code} > [179378.192955] possible SYN flooding on port 8088. Sending cookies. > [179472.331827] possible SYN flooding on port 8088. Sending cookies. > [235126.729593] possible SYN flooding on port 8088. Sending cookies. > [235335.369386] possible SYN flooding on port 8088. Sending cookies. > [235483.128214] possible SYN flooding on port 8088. Sending cookies. > [235639.853147] possible SYN flooding on port 8088. Sending cookies. > [235785.611114] possible SYN flooding on port 8088. Sending cookies. > [235852.501364] possible SYN flooding on port 8088. Sending cookies. > [238275.390711] possible SYN flooding on port 8088. Sending cookies. > {code} > {code} > [Jul 12 11:37:15.254] Manager {0x7fe8b33d5700} NOTE: [drainIncomingChannel] > file req: records.config v: 8963 bytes: 29295 > [Jul 12 11:37:15.301] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8959 > [Jul 12 11:37:15.301] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:37:15.314] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8960 > [Jul 12 11:37:15.315] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:37:27.633] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8963 n: > 8964 > [Jul 12 11:37:27.892] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8964 n: > 8965 > [Jul 12 11:37:32.374] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8965 n: > 8966 > [Jul 12 11:37:39.608] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8965 > [Jul 12 11:37:39.608] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:37:49.975] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8966 n: > 8967 > [Jul 12 11:37:50.067] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8967 n: > 8968 > [Jul 12 11:38:10.051] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8968 n: > 8969 > [Jul 12 11:38:39.222] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8969 n: > 8970 > [Jul 12 11:38:40.194] Manager {0x7fe8b33d5700} NOTE: marking server on node > 222.186.49.169 as up > [Jul 12 11:38:49.250] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:49.250] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:49.250] Manager {0x7fe8b33d5700} NOTE: marking server on node > 222.186.49.165 as up > [Jul 12 11:38:49.252] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:49.252] Manager {0x7fe8b4a6f7e0} NOTE: marking server on node > 222.186.49.185 as down > [Jul 12 11:38:49.252] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:49.252] Manager {0x7fe8b33d5700} NOTE: marking server on node > 222.186.49.185 as up > [Jul 12 11:38:49.253] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:49.253] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:49.253] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:49.253] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:54.491] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.491] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:54.492] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.492] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:54.493] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.493] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:54.494] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.494] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:54.494] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.494] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:54.495] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.495] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:54.496] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.496] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:54.497] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.497] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:54.497] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.498] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:54.498] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.498] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:54.499] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.499] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:54.500] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.500] Manager {0x7fe8b33d5700} ERROR: (last system error 4: > Interrupted system call) > [Jul 12 11:38:54.500] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::sendReliableMessageReadTillClose] Unable to connect > [Jul 12 11:38:54.500] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:38:54.501] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.501] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:38:54.502] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.502] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:38:54.503] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.503] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:38:54.503] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.504] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:38:54.504] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.504] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:38:54.505] Manager {0x7fe8b33d5700} ERROR: > [ClusterCom::handleMultiCastFilePacket] Update failed > [Jul 12 11:38:54.505] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:38:54.624] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8970 n: > 8973 > [Jul 12 11:39:02.360] Manager {0x7fe8b4a6f7e0} NOTE: marking manager on node > 222.186.49.196 as down > [Jul 12 11:39:02.360] Manager {0x7fe8b4a6f7e0} NOTE: marking server on node > 222.186.49.196 as down > [Jul 12 11:39:02.388] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8973 n: > 8974 > [Jul 12 11:39:02.389] Manager {0x7fe8b33d5700} NOTE: marking manager on node > 222.186.49.196 as up > [Jul 12 11:39:03.279] Manager {0x7fe8b33d5700} NOTE: marking server on node > 222.186.49.196 as up > [Jul 12 11:39:05.471] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8974 n: > 8975 > [Jul 12 11:39:18.916] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8975 n: > 8976 > [Jul 12 11:39:18.918] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8975 > [Jul 12 11:39:18.918] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:18.999] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8976 n: > 8977 > [Jul 12 11:39:22.973] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8977 n: > 8978 > [Jul 12 11:39:28.455] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8978 n: > 8979 > [Jul 12 11:39:29.013] Manager {0x7fe8b33d5700} NOTE: [drainIncomingChannel] > file req: records.config v: 8979 bytes: 29295 > [Jul 12 11:39:32.995] Manager {0x7fe8b33d5700} NOTE: > [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8979 n: > 8980 > [Jul 12 11:39:32.997] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:37.400] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:37.400] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:37.403] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:37.403] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:37.403] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:37.404] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [Jul 12 11:39:37.404] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] > Error file req: records.config ver: 8979 > [Jul 12 11:39:37.404] Manager {0x7fe8b33d5700} ERROR: (last system error > 111: Connection refused) > [TrafficManager] ==> Cleaning up and reissuing signal #11 > [Jul 12 11:39:48.329] Manager {0x7fe8b33d5700} ERROR: [TrafficManager] ==> > Cleaning up and reissuing signal #11 > [Jul 12 11:39:48.329] Manager {0x7fe8b33d5700} ERROR: (last system error > 104: Connection reset by peer) > [E. Mgmt] log ==> [TrafficManager] using root directory '/usr' > [Jul 12 11:39:49.346] {0x7fc772ec37e0} STATUS: opened > /var/log/trafficserver/manager.log > [Jul 12 11:39:49.346] {0x7fc772ec37e0} NOTE: updated diags config > [Jul 12 11:39:49.353] Manager {0x7fc772ec37e0} NOTE: [ClusterCom::ClusterCom] > Node running on OS: 'Linux' Release: '2.6.32-220.17.1.tb619.el6.x86_6' > [Jul 12 11:39:49.356] Manager {0x7fc772ec37e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 8080 > [Jul 12 11:39:49.357] Manager {0x7fc772ec37e0} NOTE: > [LocalManager::listenForProxy] Listening on port: 81 > [Jul 12 11:39:49.357] Manager {0x7fc772ec37e0} NOTE: [TrafficManager] Setup > complete > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)