[jira] [Commented] (TS-2205) AIO caused system hang
[ https://issues.apache.org/jira/browse/TS-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13762922#comment-13762922 ] Zhao Yongming commented on TS-2205: --- this is a issue reported from ShangHai Jiao Tong Univercity http://www.sjtu.edu.cn/ > AIO caused system hang > -- > > Key: TS-2205 > URL: https://issues.apache.org/jira/browse/TS-2205 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.1 >Reporter: Zhao Yongming >Assignee: weijin > Fix For: 4.1.0 > > > the system may hang with AIO thread CPU usage rising: > {code} > top - 17:10:46 up 38 days, 22:43, 2 users, load average: 11.34, 2.97, 2.75 > Tasks: 512 total, 55 running, 457 sleeping, 0 stopped, 0 zombie > Cpu(s): 6.9%us, 54.8%sy, 0.0%ni, 37.3%id, 0.0%wa, 0.0%hi, 0.9%si, 0.0%st > Mem: 65963696k total, 64318444k used, 1645252k free, 241496k buffers > Swap: 33554424k total,20416k used, 33534008k free, 14864188k cached > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND > 32498 ats 20 0 59.3g 45g 25m R 65.8 72.1 24:44.15 [ET_AIO 5] > 3213 root 20 0 000 S 15.4 0.0 13:38.32 kondemand/7 > 3219 root 20 0 000 S 15.1 0.0 16:32.78 kondemand/13 > 4 root 20 0 000 S 13.8 0.0 33:18.13 ksoftirqd/0 >13 root 20 0 000 S 13.4 0.0 21:45.18 ksoftirqd/2 >37 root 20 0 000 S 13.4 0.0 19:42.34 ksoftirqd/8 >45 root 20 0 000 S 13.4 0.0 18:31.17 ksoftirqd/10 > 32483 ats 20 0 59.3g 45g 25m R 13.4 72.1 16:47.14 [ET_AIO 6] > 32487 ats 20 0 59.3g 45g 25m R 13.4 72.1 16:46.93 [ET_AIO 2] >25 root 20 0 000 S 13.1 0.0 19:02.18 ksoftirqd/5 >65 root 20 0 000 S 13.1 0.0 19:24.04 ksoftirqd/15 > 32477 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:32.90 [ET_AIO 0] > 32478 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:49.77 [ET_AIO 1] > 32479 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:41.77 [ET_AIO 2] > 32481 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:50.40 [ET_AIO 4] > 32482 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:47.42 [ET_AIO 5] > 32484 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:25.81 [ET_AIO 7] > 32485 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:52.71 [ET_AIO 0] > 32486 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:51.69 [ET_AIO 1] > 32491 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:50.58 [ET_AIO 6] > 32492 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:49.12 [ET_AIO 7] > 32480 ats 20 0 59.3g 45g 25m S 12.8 72.1 16:47.39 [ET_AIO 3] > 32488 ats 20 0 59.3g 45g 25m R 12.8 72.1 16:52.16 [ET_AIO 3] > 32489 ats 20 0 59.3g 45g 25m S 12.8 72.1 16:50.79 [ET_AIO 4] > 32490 ats 20 0 59.3g 45g 25m R 12.8 72.1 16:52.61 [ET_AIO 5] > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-2205) AIO caused system hang
[ https://issues.apache.org/jira/browse/TS-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13870429#comment-13870429 ] weijin commented on TS-2205: It maybe related to TS-2027, so we can close this now, reopen it if arise again. > AIO caused system hang > -- > > Key: TS-2205 > URL: https://issues.apache.org/jira/browse/TS-2205 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.1 >Reporter: Zhao Yongming >Assignee: weijin >Priority: Critical > Fix For: 4.2.0 > > > the system may hang with AIO thread CPU usage rising: > {code} > top - 17:10:46 up 38 days, 22:43, 2 users, load average: 11.34, 2.97, 2.75 > Tasks: 512 total, 55 running, 457 sleeping, 0 stopped, 0 zombie > Cpu(s): 6.9%us, 54.8%sy, 0.0%ni, 37.3%id, 0.0%wa, 0.0%hi, 0.9%si, 0.0%st > Mem: 65963696k total, 64318444k used, 1645252k free, 241496k buffers > Swap: 33554424k total,20416k used, 33534008k free, 14864188k cached > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND > 32498 ats 20 0 59.3g 45g 25m R 65.8 72.1 24:44.15 [ET_AIO 5] > 3213 root 20 0 000 S 15.4 0.0 13:38.32 kondemand/7 > 3219 root 20 0 000 S 15.1 0.0 16:32.78 kondemand/13 > 4 root 20 0 000 S 13.8 0.0 33:18.13 ksoftirqd/0 >13 root 20 0 000 S 13.4 0.0 21:45.18 ksoftirqd/2 >37 root 20 0 000 S 13.4 0.0 19:42.34 ksoftirqd/8 >45 root 20 0 000 S 13.4 0.0 18:31.17 ksoftirqd/10 > 32483 ats 20 0 59.3g 45g 25m R 13.4 72.1 16:47.14 [ET_AIO 6] > 32487 ats 20 0 59.3g 45g 25m R 13.4 72.1 16:46.93 [ET_AIO 2] >25 root 20 0 000 S 13.1 0.0 19:02.18 ksoftirqd/5 >65 root 20 0 000 S 13.1 0.0 19:24.04 ksoftirqd/15 > 32477 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:32.90 [ET_AIO 0] > 32478 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:49.77 [ET_AIO 1] > 32479 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:41.77 [ET_AIO 2] > 32481 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:50.40 [ET_AIO 4] > 32482 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:47.42 [ET_AIO 5] > 32484 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:25.81 [ET_AIO 7] > 32485 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:52.71 [ET_AIO 0] > 32486 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:51.69 [ET_AIO 1] > 32491 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:50.58 [ET_AIO 6] > 32492 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:49.12 [ET_AIO 7] > 32480 ats 20 0 59.3g 45g 25m S 12.8 72.1 16:47.39 [ET_AIO 3] > 32488 ats 20 0 59.3g 45g 25m R 12.8 72.1 16:52.16 [ET_AIO 3] > 32489 ats 20 0 59.3g 45g 25m S 12.8 72.1 16:50.79 [ET_AIO 4] > 32490 ats 20 0 59.3g 45g 25m R 12.8 72.1 16:52.61 [ET_AIO 5] > {code} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (TS-2205) AIO caused system hang
[ https://issues.apache.org/jira/browse/TS-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14365053#comment-14365053 ] Zhaonanli commented on TS-2205: --- hi , arise again, version 4.2.2 > AIO caused system hang > -- > > Key: TS-2205 > URL: https://issues.apache.org/jira/browse/TS-2205 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.1 >Reporter: Zhao Yongming >Assignee: weijin >Priority: Critical > > the system may hang with AIO thread CPU usage rising: > {code} > top - 17:10:46 up 38 days, 22:43, 2 users, load average: 11.34, 2.97, 2.75 > Tasks: 512 total, 55 running, 457 sleeping, 0 stopped, 0 zombie > Cpu(s): 6.9%us, 54.8%sy, 0.0%ni, 37.3%id, 0.0%wa, 0.0%hi, 0.9%si, 0.0%st > Mem: 65963696k total, 64318444k used, 1645252k free, 241496k buffers > Swap: 33554424k total,20416k used, 33534008k free, 14864188k cached > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND > 32498 ats 20 0 59.3g 45g 25m R 65.8 72.1 24:44.15 [ET_AIO 5] > 3213 root 20 0 000 S 15.4 0.0 13:38.32 kondemand/7 > 3219 root 20 0 000 S 15.1 0.0 16:32.78 kondemand/13 > 4 root 20 0 000 S 13.8 0.0 33:18.13 ksoftirqd/0 >13 root 20 0 000 S 13.4 0.0 21:45.18 ksoftirqd/2 >37 root 20 0 000 S 13.4 0.0 19:42.34 ksoftirqd/8 >45 root 20 0 000 S 13.4 0.0 18:31.17 ksoftirqd/10 > 32483 ats 20 0 59.3g 45g 25m R 13.4 72.1 16:47.14 [ET_AIO 6] > 32487 ats 20 0 59.3g 45g 25m R 13.4 72.1 16:46.93 [ET_AIO 2] >25 root 20 0 000 S 13.1 0.0 19:02.18 ksoftirqd/5 >65 root 20 0 000 S 13.1 0.0 19:24.04 ksoftirqd/15 > 32477 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:32.90 [ET_AIO 0] > 32478 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:49.77 [ET_AIO 1] > 32479 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:41.77 [ET_AIO 2] > 32481 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:50.40 [ET_AIO 4] > 32482 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:47.42 [ET_AIO 5] > 32484 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:25.81 [ET_AIO 7] > 32485 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:52.71 [ET_AIO 0] > 32486 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:51.69 [ET_AIO 1] > 32491 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:50.58 [ET_AIO 6] > 32492 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:49.12 [ET_AIO 7] > 32480 ats 20 0 59.3g 45g 25m S 12.8 72.1 16:47.39 [ET_AIO 3] > 32488 ats 20 0 59.3g 45g 25m R 12.8 72.1 16:52.16 [ET_AIO 3] > 32489 ats 20 0 59.3g 45g 25m S 12.8 72.1 16:50.79 [ET_AIO 4] > 32490 ats 20 0 59.3g 45g 25m R 12.8 72.1 16:52.61 [ET_AIO 5] > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-2205) AIO caused system hang
[ https://issues.apache.org/jira/browse/TS-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386140#comment-14386140 ] Zhao Yongming commented on TS-2205: --- TS-3458 is reported as a index sycing issue. things need more information, I will try. basicly we don't find anything that need to look anymore, I will let this issue open for a while and close if no further information. > AIO caused system hang > -- > > Key: TS-2205 > URL: https://issues.apache.org/jira/browse/TS-2205 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.1 >Reporter: Zhao Yongming >Assignee: weijin >Priority: Critical > Fix For: 6.0.0 > > > the system may hang with AIO thread CPU usage rising: > {code} > top - 17:10:46 up 38 days, 22:43, 2 users, load average: 11.34, 2.97, 2.75 > Tasks: 512 total, 55 running, 457 sleeping, 0 stopped, 0 zombie > Cpu(s): 6.9%us, 54.8%sy, 0.0%ni, 37.3%id, 0.0%wa, 0.0%hi, 0.9%si, 0.0%st > Mem: 65963696k total, 64318444k used, 1645252k free, 241496k buffers > Swap: 33554424k total,20416k used, 33534008k free, 14864188k cached > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND > 32498 ats 20 0 59.3g 45g 25m R 65.8 72.1 24:44.15 [ET_AIO 5] > 3213 root 20 0 000 S 15.4 0.0 13:38.32 kondemand/7 > 3219 root 20 0 000 S 15.1 0.0 16:32.78 kondemand/13 > 4 root 20 0 000 S 13.8 0.0 33:18.13 ksoftirqd/0 >13 root 20 0 000 S 13.4 0.0 21:45.18 ksoftirqd/2 >37 root 20 0 000 S 13.4 0.0 19:42.34 ksoftirqd/8 >45 root 20 0 000 S 13.4 0.0 18:31.17 ksoftirqd/10 > 32483 ats 20 0 59.3g 45g 25m R 13.4 72.1 16:47.14 [ET_AIO 6] > 32487 ats 20 0 59.3g 45g 25m R 13.4 72.1 16:46.93 [ET_AIO 2] >25 root 20 0 000 S 13.1 0.0 19:02.18 ksoftirqd/5 >65 root 20 0 000 S 13.1 0.0 19:24.04 ksoftirqd/15 > 32477 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:32.90 [ET_AIO 0] > 32478 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:49.77 [ET_AIO 1] > 32479 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:41.77 [ET_AIO 2] > 32481 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:50.40 [ET_AIO 4] > 32482 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:47.42 [ET_AIO 5] > 32484 ats 20 0 59.3g 45g 25m R 13.1 72.1 16:25.81 [ET_AIO 7] > 32485 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:52.71 [ET_AIO 0] > 32486 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:51.69 [ET_AIO 1] > 32491 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:50.58 [ET_AIO 6] > 32492 ats 20 0 59.3g 45g 25m S 13.1 72.1 16:49.12 [ET_AIO 7] > 32480 ats 20 0 59.3g 45g 25m S 12.8 72.1 16:47.39 [ET_AIO 3] > 32488 ats 20 0 59.3g 45g 25m R 12.8 72.1 16:52.16 [ET_AIO 3] > 32489 ats 20 0 59.3g 45g 25m S 12.8 72.1 16:50.79 [ET_AIO 4] > 32490 ats 20 0 59.3g 45g 25m R 12.8 72.1 16:52.61 [ET_AIO 5] > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)