[jira] [Commented] (TS-981) ATS as transparent proxy crashes under heavy load
[ https://issues.apache.org/jira/browse/TS-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240096#comment-13240096 ] Leif Hedstrom commented on TS-981: -- I'll check with John, but I think we should just remove libev entirely from the source. > ATS as transparent proxy crashes under heavy load > - > > Key: TS-981 > URL: https://issues.apache.org/jira/browse/TS-981 > Project: Traffic Server > Issue Type: Bug > Components: Core >Affects Versions: 3.0.0 > Environment: 4 core desktop. Centos 5.6, kernel 2.6.39.2 compiled > with TPROXY support. > ATS compiled as: ./configure --enable-tproxy --enable-libev >Reporter: Danny Shporer >Assignee: Leif Hedstrom >Priority: Critical > Fix For: 3.1.4 > > Attachments: records.config > > > ATS crashes under heavy load testing - around 25,000 HTTP transactions per > second. > I have tested it on both vesions 3.0.0 and 3.0.1 and the same happens. > GDB info: > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x42174940 (LWP 6663)] > 0x0068fd4b in modify (this=0x2b12c628, event= out>, e=) at P_UnixNet.h:536 > 536 fd_change(event_loop->eio, eio.fd, 0); > (gdb) bt > #0 0x0068fd4b in modify (this=0x2b12c628, event= out>, e=) at P_UnixNet.h:536 > #1 NetHandler::mainNetEvent (this=0x2b12c628, event= out>, e=) at UnixNet.cc:432 > #2 0x006bfc4f in EThread::process_event (this=0x2b12b010, > e=0xf44570, calling_code=5) at I_Continuation.h:146 > #3 0x006c055c in EThread::execute (this=0x2b12b010) at > UnixEThread.cc:262 > #4 0x006bef8e in spawn_thread_internal (a=0xf35c90) at Thread.cc:88 > #5 0x003e9dc0673d in start_thread () from /lib64/libpthread.so.0 > #6 0x003e9d0d44bd in clone () from /lib64/libc.so.6 > (gdb) info f > Stack level 0, frame at 0x42174030: > rip = 0x68fd4b in modify (P_UnixNet.h:536); saved rip 0x6bfc4f > inlined into frame 1 > source language c++. > Arglist at unknown address. > Locals at unknown address, Previous frame's sp in rsp -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-981) ATS as transparent proxy crashes under heavy load
[ https://issues.apache.org/jira/browse/TS-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182281#comment-13182281 ] Leif Hedstrom commented on TS-981: -- Ah, yeah, I see it's still there :/. Let me take a look at it, and see what is there. I'm fairly certain the person who did the libev code decided to give up, and that it was fairly pointless. > ATS as transparent proxy crashes under heavy load > - > > Key: TS-981 > URL: https://issues.apache.org/jira/browse/TS-981 > Project: Traffic Server > Issue Type: Bug > Components: Core >Affects Versions: 3.0.0 > Environment: 4 core desktop. Centos 5.6, kernel 2.6.39.2 compiled > with TPROXY support. > ATS compiled as: ./configure --enable-tproxy --enable-libev >Reporter: Danny Shporer >Priority: Critical > Fix For: 3.1.2 > > Attachments: records.config > > > ATS crashes under heavy load testing - around 25,000 HTTP transactions per > second. > I have tested it on both vesions 3.0.0 and 3.0.1 and the same happens. > GDB info: > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x42174940 (LWP 6663)] > 0x0068fd4b in modify (this=0x2b12c628, event= out>, e=) at P_UnixNet.h:536 > 536 fd_change(event_loop->eio, eio.fd, 0); > (gdb) bt > #0 0x0068fd4b in modify (this=0x2b12c628, event= out>, e=) at P_UnixNet.h:536 > #1 NetHandler::mainNetEvent (this=0x2b12c628, event= out>, e=) at UnixNet.cc:432 > #2 0x006bfc4f in EThread::process_event (this=0x2b12b010, > e=0xf44570, calling_code=5) at I_Continuation.h:146 > #3 0x006c055c in EThread::execute (this=0x2b12b010) at > UnixEThread.cc:262 > #4 0x006bef8e in spawn_thread_internal (a=0xf35c90) at Thread.cc:88 > #5 0x003e9dc0673d in start_thread () from /lib64/libpthread.so.0 > #6 0x003e9d0d44bd in clone () from /lib64/libc.so.6 > (gdb) info f > Stack level 0, frame at 0x42174030: > rip = 0x68fd4b in modify (P_UnixNet.h:536); saved rip 0x6bfc4f > inlined into frame 1 > source language c++. > Arglist at unknown address. > Locals at unknown address, Previous frame's sp in rsp -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-981) ATS as transparent proxy crashes under heavy load
[ https://issues.apache.org/jira/browse/TS-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182218#comment-13182218 ] Danny Shporer commented on TS-981: -- Yes I was. It's mentioned in the Environment field of the bug report details that ATS is compiled with --enable-libev. I did not find anywhere any mention that it has been removed. Anyway, when I realized that there was an exception in the libev code, I reverted to using epoll and that seems to work penfing bug report TS-1075. If libev is not supported by ATS then I guess you can close this bug. I would suggest to make it clear not to use it (maybe remove README.libev from the code distributable). > ATS as transparent proxy crashes under heavy load > - > > Key: TS-981 > URL: https://issues.apache.org/jira/browse/TS-981 > Project: Traffic Server > Issue Type: Bug > Components: Core >Affects Versions: 3.0.0 > Environment: 4 core desktop. Centos 5.6, kernel 2.6.39.2 compiled > with TPROXY support. > ATS compiled as: ./configure --enable-tproxy --enable-libev >Reporter: Danny Shporer >Priority: Critical > Fix For: 3.1.2 > > Attachments: records.config > > > ATS crashes under heavy load testing - around 25,000 HTTP transactions per > second. > I have tested it on both vesions 3.0.0 and 3.0.1 and the same happens. > GDB info: > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x42174940 (LWP 6663)] > 0x0068fd4b in modify (this=0x2b12c628, event= out>, e=) at P_UnixNet.h:536 > 536 fd_change(event_loop->eio, eio.fd, 0); > (gdb) bt > #0 0x0068fd4b in modify (this=0x2b12c628, event= out>, e=) at P_UnixNet.h:536 > #1 NetHandler::mainNetEvent (this=0x2b12c628, event= out>, e=) at UnixNet.cc:432 > #2 0x006bfc4f in EThread::process_event (this=0x2b12b010, > e=0xf44570, calling_code=5) at I_Continuation.h:146 > #3 0x006c055c in EThread::execute (this=0x2b12b010) at > UnixEThread.cc:262 > #4 0x006bef8e in spawn_thread_internal (a=0xf35c90) at Thread.cc:88 > #5 0x003e9dc0673d in start_thread () from /lib64/libpthread.so.0 > #6 0x003e9d0d44bd in clone () from /lib64/libc.so.6 > (gdb) info f > Stack level 0, frame at 0x42174030: > rip = 0x68fd4b in modify (P_UnixNet.h:536); saved rip 0x6bfc4f > inlined into frame 1 > source language c++. > Arglist at unknown address. > Locals at unknown address, Previous frame's sp in rsp -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-981) ATS as transparent proxy crashes under heavy load
[ https://issues.apache.org/jira/browse/TS-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182074#comment-13182074 ] Leif Hedstrom commented on TS-981: -- Not sure I understand, are you saying you are using the libev "support"? I thought we removed that, because it was not working well, and not as fast as epoll. > ATS as transparent proxy crashes under heavy load > - > > Key: TS-981 > URL: https://issues.apache.org/jira/browse/TS-981 > Project: Traffic Server > Issue Type: Bug > Components: Core >Affects Versions: 3.0.0 > Environment: 4 core desktop. Centos 5.6, kernel 2.6.39.2 compiled > with TPROXY support. > ATS compiled as: ./configure --enable-tproxy --enable-libev >Reporter: Danny Shporer >Priority: Critical > Fix For: 3.1.2 > > Attachments: records.config > > > ATS crashes under heavy load testing - around 25,000 HTTP transactions per > second. > I have tested it on both vesions 3.0.0 and 3.0.1 and the same happens. > GDB info: > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x42174940 (LWP 6663)] > 0x0068fd4b in modify (this=0x2b12c628, event= out>, e=) at P_UnixNet.h:536 > 536 fd_change(event_loop->eio, eio.fd, 0); > (gdb) bt > #0 0x0068fd4b in modify (this=0x2b12c628, event= out>, e=) at P_UnixNet.h:536 > #1 NetHandler::mainNetEvent (this=0x2b12c628, event= out>, e=) at UnixNet.cc:432 > #2 0x006bfc4f in EThread::process_event (this=0x2b12b010, > e=0xf44570, calling_code=5) at I_Continuation.h:146 > #3 0x006c055c in EThread::execute (this=0x2b12b010) at > UnixEThread.cc:262 > #4 0x006bef8e in spawn_thread_internal (a=0xf35c90) at Thread.cc:88 > #5 0x003e9dc0673d in start_thread () from /lib64/libpthread.so.0 > #6 0x003e9d0d44bd in clone () from /lib64/libc.so.6 > (gdb) info f > Stack level 0, frame at 0x42174030: > rip = 0x68fd4b in modify (P_UnixNet.h:536); saved rip 0x6bfc4f > inlined into frame 1 > source language c++. > Arglist at unknown address. > Locals at unknown address, Previous frame's sp in rsp -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-981) ATS as transparent proxy crashes under heavy load
[ https://issues.apache.org/jira/browse/TS-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181913#comment-13181913 ] Danny Shporer commented on TS-981: -- I have not tried to reproduce with reverse or forward proxy. I did try to reproduce in debug build but without success - my tool was not able to reach 25K TPS in debug build. It only reached around 16K TPS and there was no exception. The problem seems to be with the libev support in ATS. Without it in my Linux environment when plain epoll is used (./configure --enable-tproxy) it does not happen and I have been working this way since (I have reached round 34K TPS). There is another issue regarding heavy load in transparent proxy - the Linux TPROXY stack only takes into account the local addresses when using dynamic bind (bind without specifying a port). I opened a new bug for this one - TS-1075. > ATS as transparent proxy crashes under heavy load > - > > Key: TS-981 > URL: https://issues.apache.org/jira/browse/TS-981 > Project: Traffic Server > Issue Type: Bug > Components: Core >Affects Versions: 3.0.0 > Environment: 4 core desktop. Centos 5.6, kernel 2.6.39.2 compiled > with TPROXY support. > ATS compiled as: ./configure --enable-tproxy --enable-libev >Reporter: Danny Shporer >Priority: Critical > Fix For: 3.1.2 > > Attachments: records.config > > > ATS crashes under heavy load testing - around 25,000 HTTP transactions per > second. > I have tested it on both vesions 3.0.0 and 3.0.1 and the same happens. > GDB info: > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x42174940 (LWP 6663)] > 0x0068fd4b in modify (this=0x2b12c628, event= out>, e=) at P_UnixNet.h:536 > 536 fd_change(event_loop->eio, eio.fd, 0); > (gdb) bt > #0 0x0068fd4b in modify (this=0x2b12c628, event= out>, e=) at P_UnixNet.h:536 > #1 NetHandler::mainNetEvent (this=0x2b12c628, event= out>, e=) at UnixNet.cc:432 > #2 0x006bfc4f in EThread::process_event (this=0x2b12b010, > e=0xf44570, calling_code=5) at I_Continuation.h:146 > #3 0x006c055c in EThread::execute (this=0x2b12b010) at > UnixEThread.cc:262 > #4 0x006bef8e in spawn_thread_internal (a=0xf35c90) at Thread.cc:88 > #5 0x003e9dc0673d in start_thread () from /lib64/libpthread.so.0 > #6 0x003e9d0d44bd in clone () from /lib64/libc.so.6 > (gdb) info f > Stack level 0, frame at 0x42174030: > rip = 0x68fd4b in modify (P_UnixNet.h:536); saved rip 0x6bfc4f > inlined into frame 1 > source language c++. > Arglist at unknown address. > Locals at unknown address, Previous frame's sp in rsp -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-981) ATS as transparent proxy crashes under heavy load
[ https://issues.apache.org/jira/browse/TS-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181863#comment-13181863 ] Leif Hedstrom commented on TS-981: -- I'm curious if you have only got this to reproduce in transparent proxy, or if it happens in reverse or forward proxy as well? Also, can you reproduce it with a debug build? It would (hopefully) have more information. > ATS as transparent proxy crashes under heavy load > - > > Key: TS-981 > URL: https://issues.apache.org/jira/browse/TS-981 > Project: Traffic Server > Issue Type: Bug > Components: Core >Affects Versions: 3.0.0 > Environment: 4 core desktop. Centos 5.6, kernel 2.6.39.2 compiled > with TPROXY support. > ATS compiled as: ./configure --enable-tproxy --enable-libev >Reporter: Danny Shporer >Priority: Critical > Fix For: 3.1.2 > > Attachments: records.config > > > ATS crashes under heavy load testing - around 25,000 HTTP transactions per > second. > I have tested it on both vesions 3.0.0 and 3.0.1 and the same happens. > GDB info: > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x42174940 (LWP 6663)] > 0x0068fd4b in modify (this=0x2b12c628, event= out>, e=) at P_UnixNet.h:536 > 536 fd_change(event_loop->eio, eio.fd, 0); > (gdb) bt > #0 0x0068fd4b in modify (this=0x2b12c628, event= out>, e=) at P_UnixNet.h:536 > #1 NetHandler::mainNetEvent (this=0x2b12c628, event= out>, e=) at UnixNet.cc:432 > #2 0x006bfc4f in EThread::process_event (this=0x2b12b010, > e=0xf44570, calling_code=5) at I_Continuation.h:146 > #3 0x006c055c in EThread::execute (this=0x2b12b010) at > UnixEThread.cc:262 > #4 0x006bef8e in spawn_thread_internal (a=0xf35c90) at Thread.cc:88 > #5 0x003e9dc0673d in start_thread () from /lib64/libpthread.so.0 > #6 0x003e9d0d44bd in clone () from /lib64/libc.so.6 > (gdb) info f > Stack level 0, frame at 0x42174030: > rip = 0x68fd4b in modify (P_UnixNet.h:536); saved rip 0x6bfc4f > inlined into frame 1 > source language c++. > Arglist at unknown address. > Locals at unknown address, Previous frame's sp in rsp -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
