[jira] [Updated] (TS-1648) Segmentation fault in dir_clear_range()
[ https://issues.apache.org/jira/browse/TS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Kuzemko updated TS-1648: --- Affects Version/s: 3.3.0 3.2.0 > Segmentation fault in dir_clear_range() > --- > > Key: TS-1648 > URL: https://issues.apache.org/jira/browse/TS-1648 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 3.3.0, 3.2.0 >Reporter: Tomasz Kuzemko > > I use ATS as a reverse proxy. I have a fairly large disk cache consisting of > 2x 10TB raw disks. I do not use cache compression. After a few days of > running (this is a dev machine - not handling any traffic) ATS begins to > crash with a segfault shortly after start: > [Jan 11 16:11:00.690] Server {0x72bb8700} DEBUG: (rusage) took rusage > snap 1357917060690487000 > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x720ad700 (LWP 17292)] > 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) > at CacheDir.cc:382 > 382 CacheDir.cc: No such file or directory. > in CacheDir.cc > (gdb) p i > $1 = 214748365 > (gdb) l > 377 in CacheDir.cc > (gdb) p dir_index(vol, i) > $2 = (Dir *) 0x7ff997a04002 > (gdb) p dir_index(vol, i-1) > $3 = (Dir *) 0x7ffa97a03ff8 > (gdb) p *dir_index(vol, i-1) > $4 = {w = {0, 0, 0, 0, 0}} > (gdb) p *dir_index(vol, i-2) > $5 = {w = {0, 0, 52431, 52423, 0}} > (gdb) p *dir_index(vol, i) > Cannot access memory at address 0x7ff997a04002 > (gdb) p *dir_index(vol, i+2) > Cannot access memory at address 0x7ff997a04016 > (gdb) p *dir_index(vol, i+1) > Cannot access memory at address 0x7ff997a0400c > (gdb) p vol->buckets * DIR_DEPTH * vol->segments > $6 = 1246953472 > (gdb) bt > #0 0x00696a71 in dir_clear_range (start=640, end=17024, > vol=0x16057d0) at CacheDir.cc:382 > #1 0x0068aba2 in Vol::handle_recover_from_data (this=0x16057d0, > event=3900, data=0x16058a0) at Cache.cc:1384 > #2 0x004e8e1c in Continuation::handleEvent (this=0x16057d0, > event=3900, data=0x16058a0) at ../iocore/eventsystem/I_Continuation.h:146 > #3 0x00692385 in AIOCallbackInternal::io_complete (this=0x16058a0, > event=1, data=0x135afc0) at ../../iocore/aio/P_AIO.h:80 > #4 0x004e8e1c in Continuation::handleEvent (this=0x16058a0, event=1, > data=0x135afc0) at ../iocore/eventsystem/I_Continuation.h:146 > #5 0x00700fec in EThread::process_event (this=0x736c4010, > e=0x135afc0, calling_code=1) at UnixEThread.cc:142 > #6 0x007011ff in EThread::execute (this=0x736c4010) at > UnixEThread.cc:191 > #7 0x006ff8c2 in spawn_thread_internal (a=0x1356040) at Thread.cc:88 > #8 0x7797e8ca in start_thread () from /lib/libpthread.so.0 > #9 0x755c6b6d in clone () from /lib/libc.so.6 > #10 0x in ?? () > This is fixed by running "traffic_server -Kk" to clear the cache. But after a > few days the issue reappears. > I will keep the current faulty setup as-is in case you need me to provide > more data. I tried to make a core dump but it took a couple of GB even after > gzip (I can however provide it on request). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-1648) Segmentation fault in dir_clear_range()
[ https://issues.apache.org/jira/browse/TS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Kuzemko updated TS-1648: --- Environment: reverse proxy > Segmentation fault in dir_clear_range() > --- > > Key: TS-1648 > URL: https://issues.apache.org/jira/browse/TS-1648 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 3.3.0, 3.2.0 > Environment: reverse proxy >Reporter: Tomasz Kuzemko > > I use ATS as a reverse proxy. I have a fairly large disk cache consisting of > 2x 10TB raw disks. I do not use cache compression. After a few days of > running (this is a dev machine - not handling any traffic) ATS begins to > crash with a segfault shortly after start: > [Jan 11 16:11:00.690] Server {0x72bb8700} DEBUG: (rusage) took rusage > snap 1357917060690487000 > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x720ad700 (LWP 17292)] > 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) > at CacheDir.cc:382 > 382 CacheDir.cc: No such file or directory. > in CacheDir.cc > (gdb) p i > $1 = 214748365 > (gdb) l > 377 in CacheDir.cc > (gdb) p dir_index(vol, i) > $2 = (Dir *) 0x7ff997a04002 > (gdb) p dir_index(vol, i-1) > $3 = (Dir *) 0x7ffa97a03ff8 > (gdb) p *dir_index(vol, i-1) > $4 = {w = {0, 0, 0, 0, 0}} > (gdb) p *dir_index(vol, i-2) > $5 = {w = {0, 0, 52431, 52423, 0}} > (gdb) p *dir_index(vol, i) > Cannot access memory at address 0x7ff997a04002 > (gdb) p *dir_index(vol, i+2) > Cannot access memory at address 0x7ff997a04016 > (gdb) p *dir_index(vol, i+1) > Cannot access memory at address 0x7ff997a0400c > (gdb) p vol->buckets * DIR_DEPTH * vol->segments > $6 = 1246953472 > (gdb) bt > #0 0x00696a71 in dir_clear_range (start=640, end=17024, > vol=0x16057d0) at CacheDir.cc:382 > #1 0x0068aba2 in Vol::handle_recover_from_data (this=0x16057d0, > event=3900, data=0x16058a0) at Cache.cc:1384 > #2 0x004e8e1c in Continuation::handleEvent (this=0x16057d0, > event=3900, data=0x16058a0) at ../iocore/eventsystem/I_Continuation.h:146 > #3 0x00692385 in AIOCallbackInternal::io_complete (this=0x16058a0, > event=1, data=0x135afc0) at ../../iocore/aio/P_AIO.h:80 > #4 0x004e8e1c in Continuation::handleEvent (this=0x16058a0, event=1, > data=0x135afc0) at ../iocore/eventsystem/I_Continuation.h:146 > #5 0x00700fec in EThread::process_event (this=0x736c4010, > e=0x135afc0, calling_code=1) at UnixEThread.cc:142 > #6 0x007011ff in EThread::execute (this=0x736c4010) at > UnixEThread.cc:191 > #7 0x006ff8c2 in spawn_thread_internal (a=0x1356040) at Thread.cc:88 > #8 0x7797e8ca in start_thread () from /lib/libpthread.so.0 > #9 0x755c6b6d in clone () from /lib/libc.so.6 > #10 0x in ?? () > This is fixed by running "traffic_server -Kk" to clear the cache. But after a > few days the issue reappears. > I will keep the current faulty setup as-is in case you need me to provide > more data. I tried to make a core dump but it took a couple of GB even after > gzip (I can however provide it on request). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-1648) Segmentation fault in dir_clear_range()
[ https://issues.apache.org/jira/browse/TS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Kuzemko updated TS-1648: --- Description: I use ATS as a reverse proxy. I have a fairly large disk cache consisting of 2x 10TB raw disks. I do not use cache compression. After a few days of running (this is a dev machine - not handling any traffic) ATS begins to crash with a segfault shortly after start: [Jan 11 16:11:00.690] Server {0x72bb8700} DEBUG: (rusage) took rusage snap 1357917060690487000 Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x720ad700 (LWP 17292)] 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) at CacheDir.cc:382 382 CacheDir.cc: No such file or directory. in CacheDir.cc (gdb) p i $1 = 214748365 (gdb) l 377 in CacheDir.cc (gdb) p dir_index(vol, i) $2 = (Dir *) 0x7ff997a04002 (gdb) p dir_index(vol, i-1) $3 = (Dir *) 0x7ffa97a03ff8 (gdb) p *dir_index(vol, i-1) $4 = {w = {0, 0, 0, 0, 0}} (gdb) p *dir_index(vol, i-2) $5 = {w = {0, 0, 52431, 52423, 0}} (gdb) p *dir_index(vol, i) Cannot access memory at address 0x7ff997a04002 (gdb) p *dir_index(vol, i+2) Cannot access memory at address 0x7ff997a04016 (gdb) p *dir_index(vol, i+1) Cannot access memory at address 0x7ff997a0400c (gdb) p vol->buckets * DIR_DEPTH * vol->segments $6 = 1246953472 (gdb) bt #0 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) at CacheDir.cc:382 #1 0x0068aba2 in Vol::handle_recover_from_data (this=0x16057d0, event=3900, data=0x16058a0) at Cache.cc:1384 #2 0x004e8e1c in Continuation::handleEvent (this=0x16057d0, event=3900, data=0x16058a0) at ../iocore/eventsystem/I_Continuation.h:146 #3 0x00692385 in AIOCallbackInternal::io_complete (this=0x16058a0, event=1, data=0x135afc0) at ../../iocore/aio/P_AIO.h:80 #4 0x004e8e1c in Continuation::handleEvent (this=0x16058a0, event=1, data=0x135afc0) at ../iocore/eventsystem/I_Continuation.h:146 #5 0x00700fec in EThread::process_event (this=0x736c4010, e=0x135afc0, calling_code=1) at UnixEThread.cc:142 #6 0x007011ff in EThread::execute (this=0x736c4010) at UnixEThread.cc:191 #7 0x006ff8c2 in spawn_thread_internal (a=0x1356040) at Thread.cc:88 #8 0x7797e8ca in start_thread () from /lib/libpthread.so.0 #9 0x755c6b6d in clone () from /lib/libc.so.6 #10 0x in ?? () This is fixed by running "traffic_server -Kk" to clear the cache. But after a few days the issue reappears. I will keep the current faulty setup as-is in case you need me to provide more data. I tried to make a core dump but it took a couple of GB even after gzip (I can however provide it on request). *Edit* OS is Debian GNU/Linux 6.0.6 with custom built kernel 3.2.13-grsec--grs-ipv6-64 was: I use ATS as a reverse proxy. I have a fairly large disk cache consisting of 2x 10TB raw disks. I do not use cache compression. After a few days of running (this is a dev machine - not handling any traffic) ATS begins to crash with a segfault shortly after start: [Jan 11 16:11:00.690] Server {0x72bb8700} DEBUG: (rusage) took rusage snap 1357917060690487000 Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x720ad700 (LWP 17292)] 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) at CacheDir.cc:382 382 CacheDir.cc: No such file or directory. in CacheDir.cc (gdb) p i $1 = 214748365 (gdb) l 377 in CacheDir.cc (gdb) p dir_index(vol, i) $2 = (Dir *) 0x7ff997a04002 (gdb) p dir_index(vol, i-1) $3 = (Dir *) 0x7ffa97a03ff8 (gdb) p *dir_index(vol, i-1) $4 = {w = {0, 0, 0, 0, 0}} (gdb) p *dir_index(vol, i-2) $5 = {w = {0, 0, 52431, 52423, 0}} (gdb) p *dir_index(vol, i) Cannot access memory at address 0x7ff997a04002 (gdb) p *dir_index(vol, i+2) Cannot access memory at address 0x7ff997a04016 (gdb) p *dir_index(vol, i+1) Cannot access memory at address 0x7ff997a0400c (gdb) p vol->buckets * DIR_DEPTH * vol->segments $6 = 1246953472 (gdb) bt #0 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) at CacheDir.cc:382 #1 0x0068aba2 in Vol::handle_recover_from_data (this=0x16057d0, event=3900, data=0x16058a0) at Cache.cc:1384 #2 0x004e8e1c in Continuation::handleEvent (this=0x16057d0, event=3900, data=0x16058a0) at ../iocore/eventsystem/I_Continuation.h:146 #3 0x00692385 in AIOCallbackInternal::io_complete (this=0x16058a0, event=1, data=0x135afc0) at ../../iocore/aio/P_AIO.h:80 #4 0x004e8e1c in Continuation::handleEvent (this=0x16058a0, event=1, data=0x135afc0) at ../iocore/eventsystem/I_Continuation.h:146 #5 0x00700fec in EThread::process_event (this=0x736c4010, e=0x135afc0, calling_code=1) at UnixEThread.cc:142 #6 0x007011ff in EThread::execute (this=0x736c4010) at UnixEThread.cc:191 #7
[jira] [Updated] (TS-1648) Segmentation fault in dir_clear_range()
[ https://issues.apache.org/jira/browse/TS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Kuzemko updated TS-1648: --- Attachment: 0001-Fix-for-TS-1648-Segmentation-fault-in-dir_clear_rang.patch Changing index variable type from 'int' to 'long' seems to fix this. > Segmentation fault in dir_clear_range() > --- > > Key: TS-1648 > URL: https://issues.apache.org/jira/browse/TS-1648 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 3.3.0, 3.2.0 > Environment: reverse proxy >Reporter: Tomasz Kuzemko >Assignee: weijin > Attachments: > 0001-Fix-for-TS-1648-Segmentation-fault-in-dir_clear_rang.patch > > > I use ATS as a reverse proxy. I have a fairly large disk cache consisting of > 2x 10TB raw disks. I do not use cache compression. After a few days of > running (this is a dev machine - not handling any traffic) ATS begins to > crash with a segfault shortly after start: > [Jan 11 16:11:00.690] Server {0x72bb8700} DEBUG: (rusage) took rusage > snap 1357917060690487000 > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x720ad700 (LWP 17292)] > 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) > at CacheDir.cc:382 > 382 CacheDir.cc: No such file or directory. > in CacheDir.cc > (gdb) p i > $1 = 214748365 > (gdb) l > 377 in CacheDir.cc > (gdb) p dir_index(vol, i) > $2 = (Dir *) 0x7ff997a04002 > (gdb) p dir_index(vol, i-1) > $3 = (Dir *) 0x7ffa97a03ff8 > (gdb) p *dir_index(vol, i-1) > $4 = {w = {0, 0, 0, 0, 0}} > (gdb) p *dir_index(vol, i-2) > $5 = {w = {0, 0, 52431, 52423, 0}} > (gdb) p *dir_index(vol, i) > Cannot access memory at address 0x7ff997a04002 > (gdb) p *dir_index(vol, i+2) > Cannot access memory at address 0x7ff997a04016 > (gdb) p *dir_index(vol, i+1) > Cannot access memory at address 0x7ff997a0400c > (gdb) p vol->buckets * DIR_DEPTH * vol->segments > $6 = 1246953472 > (gdb) bt > #0 0x00696a71 in dir_clear_range (start=640, end=17024, > vol=0x16057d0) at CacheDir.cc:382 > #1 0x0068aba2 in Vol::handle_recover_from_data (this=0x16057d0, > event=3900, data=0x16058a0) at Cache.cc:1384 > #2 0x004e8e1c in Continuation::handleEvent (this=0x16057d0, > event=3900, data=0x16058a0) at ../iocore/eventsystem/I_Continuation.h:146 > #3 0x00692385 in AIOCallbackInternal::io_complete (this=0x16058a0, > event=1, data=0x135afc0) at ../../iocore/aio/P_AIO.h:80 > #4 0x004e8e1c in Continuation::handleEvent (this=0x16058a0, event=1, > data=0x135afc0) at ../iocore/eventsystem/I_Continuation.h:146 > #5 0x00700fec in EThread::process_event (this=0x736c4010, > e=0x135afc0, calling_code=1) at UnixEThread.cc:142 > #6 0x007011ff in EThread::execute (this=0x736c4010) at > UnixEThread.cc:191 > #7 0x006ff8c2 in spawn_thread_internal (a=0x1356040) at Thread.cc:88 > #8 0x7797e8ca in start_thread () from /lib/libpthread.so.0 > #9 0x755c6b6d in clone () from /lib/libc.so.6 > #10 0x in ?? () > This is fixed by running "traffic_server -Kk" to clear the cache. But after a > few days the issue reappears. > I will keep the current faulty setup as-is in case you need me to provide > more data. I tried to make a core dump but it took a couple of GB even after > gzip (I can however provide it on request). > *Edit* > OS is Debian GNU/Linux 6.0.6 with custom built kernel > 3.2.13-grsec--grs-ipv6-64 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-1648) Segmentation fault in dir_clear_range()
[ https://issues.apache.org/jira/browse/TS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leif Hedstrom updated TS-1648: -- Fix Version/s: 3.3.2 weijin: What should we do with this bug? I'm marking it for v3.3.2 for now. > Segmentation fault in dir_clear_range() > --- > > Key: TS-1648 > URL: https://issues.apache.org/jira/browse/TS-1648 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 3.3.0, 3.2.0 > Environment: reverse proxy >Reporter: Tomasz Kuzemko >Assignee: weijin > Fix For: 3.3.2 > > Attachments: > 0001-Fix-for-TS-1648-Segmentation-fault-in-dir_clear_rang.patch > > > I use ATS as a reverse proxy. I have a fairly large disk cache consisting of > 2x 10TB raw disks. I do not use cache compression. After a few days of > running (this is a dev machine - not handling any traffic) ATS begins to > crash with a segfault shortly after start: > [Jan 11 16:11:00.690] Server {0x72bb8700} DEBUG: (rusage) took rusage > snap 1357917060690487000 > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x720ad700 (LWP 17292)] > 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) > at CacheDir.cc:382 > 382 CacheDir.cc: No such file or directory. > in CacheDir.cc > (gdb) p i > $1 = 214748365 > (gdb) l > 377 in CacheDir.cc > (gdb) p dir_index(vol, i) > $2 = (Dir *) 0x7ff997a04002 > (gdb) p dir_index(vol, i-1) > $3 = (Dir *) 0x7ffa97a03ff8 > (gdb) p *dir_index(vol, i-1) > $4 = {w = {0, 0, 0, 0, 0}} > (gdb) p *dir_index(vol, i-2) > $5 = {w = {0, 0, 52431, 52423, 0}} > (gdb) p *dir_index(vol, i) > Cannot access memory at address 0x7ff997a04002 > (gdb) p *dir_index(vol, i+2) > Cannot access memory at address 0x7ff997a04016 > (gdb) p *dir_index(vol, i+1) > Cannot access memory at address 0x7ff997a0400c > (gdb) p vol->buckets * DIR_DEPTH * vol->segments > $6 = 1246953472 > (gdb) bt > #0 0x00696a71 in dir_clear_range (start=640, end=17024, > vol=0x16057d0) at CacheDir.cc:382 > #1 0x0068aba2 in Vol::handle_recover_from_data (this=0x16057d0, > event=3900, data=0x16058a0) at Cache.cc:1384 > #2 0x004e8e1c in Continuation::handleEvent (this=0x16057d0, > event=3900, data=0x16058a0) at ../iocore/eventsystem/I_Continuation.h:146 > #3 0x00692385 in AIOCallbackInternal::io_complete (this=0x16058a0, > event=1, data=0x135afc0) at ../../iocore/aio/P_AIO.h:80 > #4 0x004e8e1c in Continuation::handleEvent (this=0x16058a0, event=1, > data=0x135afc0) at ../iocore/eventsystem/I_Continuation.h:146 > #5 0x00700fec in EThread::process_event (this=0x736c4010, > e=0x135afc0, calling_code=1) at UnixEThread.cc:142 > #6 0x007011ff in EThread::execute (this=0x736c4010) at > UnixEThread.cc:191 > #7 0x006ff8c2 in spawn_thread_internal (a=0x1356040) at Thread.cc:88 > #8 0x7797e8ca in start_thread () from /lib/libpthread.so.0 > #9 0x755c6b6d in clone () from /lib/libc.so.6 > #10 0x in ?? () > This is fixed by running "traffic_server -Kk" to clear the cache. But after a > few days the issue reappears. > I will keep the current faulty setup as-is in case you need me to provide > more data. I tried to make a core dump but it took a couple of GB even after > gzip (I can however provide it on request). > *Edit* > OS is Debian GNU/Linux 6.0.6 with custom built kernel > 3.2.13-grsec--grs-ipv6-64 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-1648) Segmentation fault in dir_clear_range()
[ https://issues.apache.org/jira/browse/TS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leif Hedstrom updated TS-1648: -- Labels: A (was: ) > Segmentation fault in dir_clear_range() > --- > > Key: TS-1648 > URL: https://issues.apache.org/jira/browse/TS-1648 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 3.3.0, 3.2.0 > Environment: reverse proxy >Reporter: Tomasz Kuzemko >Assignee: weijin > Labels: A > Fix For: 3.3.3 > > Attachments: > 0001-Fix-for-TS-1648-Segmentation-fault-in-dir_clear_rang.patch > > > I use ATS as a reverse proxy. I have a fairly large disk cache consisting of > 2x 10TB raw disks. I do not use cache compression. After a few days of > running (this is a dev machine - not handling any traffic) ATS begins to > crash with a segfault shortly after start: > [Jan 11 16:11:00.690] Server {0x72bb8700} DEBUG: (rusage) took rusage > snap 1357917060690487000 > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x720ad700 (LWP 17292)] > 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) > at CacheDir.cc:382 > 382 CacheDir.cc: No such file or directory. > in CacheDir.cc > (gdb) p i > $1 = 214748365 > (gdb) l > 377 in CacheDir.cc > (gdb) p dir_index(vol, i) > $2 = (Dir *) 0x7ff997a04002 > (gdb) p dir_index(vol, i-1) > $3 = (Dir *) 0x7ffa97a03ff8 > (gdb) p *dir_index(vol, i-1) > $4 = {w = {0, 0, 0, 0, 0}} > (gdb) p *dir_index(vol, i-2) > $5 = {w = {0, 0, 52431, 52423, 0}} > (gdb) p *dir_index(vol, i) > Cannot access memory at address 0x7ff997a04002 > (gdb) p *dir_index(vol, i+2) > Cannot access memory at address 0x7ff997a04016 > (gdb) p *dir_index(vol, i+1) > Cannot access memory at address 0x7ff997a0400c > (gdb) p vol->buckets * DIR_DEPTH * vol->segments > $6 = 1246953472 > (gdb) bt > #0 0x00696a71 in dir_clear_range (start=640, end=17024, > vol=0x16057d0) at CacheDir.cc:382 > #1 0x0068aba2 in Vol::handle_recover_from_data (this=0x16057d0, > event=3900, data=0x16058a0) at Cache.cc:1384 > #2 0x004e8e1c in Continuation::handleEvent (this=0x16057d0, > event=3900, data=0x16058a0) at ../iocore/eventsystem/I_Continuation.h:146 > #3 0x00692385 in AIOCallbackInternal::io_complete (this=0x16058a0, > event=1, data=0x135afc0) at ../../iocore/aio/P_AIO.h:80 > #4 0x004e8e1c in Continuation::handleEvent (this=0x16058a0, event=1, > data=0x135afc0) at ../iocore/eventsystem/I_Continuation.h:146 > #5 0x00700fec in EThread::process_event (this=0x736c4010, > e=0x135afc0, calling_code=1) at UnixEThread.cc:142 > #6 0x007011ff in EThread::execute (this=0x736c4010) at > UnixEThread.cc:191 > #7 0x006ff8c2 in spawn_thread_internal (a=0x1356040) at Thread.cc:88 > #8 0x7797e8ca in start_thread () from /lib/libpthread.so.0 > #9 0x755c6b6d in clone () from /lib/libc.so.6 > #10 0x in ?? () > This is fixed by running "traffic_server -Kk" to clear the cache. But after a > few days the issue reappears. > I will keep the current faulty setup as-is in case you need me to provide > more data. I tried to make a core dump but it took a couple of GB even after > gzip (I can however provide it on request). > *Edit* > OS is Debian GNU/Linux 6.0.6 with custom built kernel > 3.2.13-grsec--grs-ipv6-64 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-1648) Segmentation fault in dir_clear_range()
[ https://issues.apache.org/jira/browse/TS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Plevyak updated TS-1648: - Attachment: cachedir_int64-jp-1.patch > Segmentation fault in dir_clear_range() > --- > > Key: TS-1648 > URL: https://issues.apache.org/jira/browse/TS-1648 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 3.3.0, 3.2.0 > Environment: reverse proxy >Reporter: Tomasz Kuzemko >Assignee: John Plevyak > Labels: A > Fix For: 3.3.3 > > Attachments: > 0001-Fix-for-TS-1648-Segmentation-fault-in-dir_clear_rang.patch, > cachedir_int64-jp-1.patch > > > I use ATS as a reverse proxy. I have a fairly large disk cache consisting of > 2x 10TB raw disks. I do not use cache compression. After a few days of > running (this is a dev machine - not handling any traffic) ATS begins to > crash with a segfault shortly after start: > [Jan 11 16:11:00.690] Server {0x72bb8700} DEBUG: (rusage) took rusage > snap 1357917060690487000 > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x720ad700 (LWP 17292)] > 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) > at CacheDir.cc:382 > 382 CacheDir.cc: No such file or directory. > in CacheDir.cc > (gdb) p i > $1 = 214748365 > (gdb) l > 377 in CacheDir.cc > (gdb) p dir_index(vol, i) > $2 = (Dir *) 0x7ff997a04002 > (gdb) p dir_index(vol, i-1) > $3 = (Dir *) 0x7ffa97a03ff8 > (gdb) p *dir_index(vol, i-1) > $4 = {w = {0, 0, 0, 0, 0}} > (gdb) p *dir_index(vol, i-2) > $5 = {w = {0, 0, 52431, 52423, 0}} > (gdb) p *dir_index(vol, i) > Cannot access memory at address 0x7ff997a04002 > (gdb) p *dir_index(vol, i+2) > Cannot access memory at address 0x7ff997a04016 > (gdb) p *dir_index(vol, i+1) > Cannot access memory at address 0x7ff997a0400c > (gdb) p vol->buckets * DIR_DEPTH * vol->segments > $6 = 1246953472 > (gdb) bt > #0 0x00696a71 in dir_clear_range (start=640, end=17024, > vol=0x16057d0) at CacheDir.cc:382 > #1 0x0068aba2 in Vol::handle_recover_from_data (this=0x16057d0, > event=3900, data=0x16058a0) at Cache.cc:1384 > #2 0x004e8e1c in Continuation::handleEvent (this=0x16057d0, > event=3900, data=0x16058a0) at ../iocore/eventsystem/I_Continuation.h:146 > #3 0x00692385 in AIOCallbackInternal::io_complete (this=0x16058a0, > event=1, data=0x135afc0) at ../../iocore/aio/P_AIO.h:80 > #4 0x004e8e1c in Continuation::handleEvent (this=0x16058a0, event=1, > data=0x135afc0) at ../iocore/eventsystem/I_Continuation.h:146 > #5 0x00700fec in EThread::process_event (this=0x736c4010, > e=0x135afc0, calling_code=1) at UnixEThread.cc:142 > #6 0x007011ff in EThread::execute (this=0x736c4010) at > UnixEThread.cc:191 > #7 0x006ff8c2 in spawn_thread_internal (a=0x1356040) at Thread.cc:88 > #8 0x7797e8ca in start_thread () from /lib/libpthread.so.0 > #9 0x755c6b6d in clone () from /lib/libc.so.6 > #10 0x in ?? () > This is fixed by running "traffic_server -Kk" to clear the cache. But after a > few days the issue reappears. > I will keep the current faulty setup as-is in case you need me to provide > more data. I tried to make a core dump but it took a couple of GB even after > gzip (I can however provide it on request). > *Edit* > OS is Debian GNU/Linux 6.0.6 with custom built kernel > 3.2.13-grsec--grs-ipv6-64 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-1648) Segmentation fault in dir_clear_range()
[ https://issues.apache.org/jira/browse/TS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Peach updated TS-1648: Description: I use ATS as a reverse proxy. I have a fairly large disk cache consisting of 2x 10TB raw disks. I do not use cache compression. After a few days of running (this is a dev machine - not handling any traffic) ATS begins to crash with a segfault shortly after start: {code} [Jan 11 16:11:00.690] Server {0x72bb8700} DEBUG: (rusage) took rusage snap 1357917060690487000 Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x720ad700 (LWP 17292)] 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) at CacheDir.cc:382 382 CacheDir.cc: No such file or directory. in CacheDir.cc (gdb) p i $1 = 214748365 (gdb) l 377 in CacheDir.cc (gdb) p dir_index(vol, i) $2 = (Dir *) 0x7ff997a04002 (gdb) p dir_index(vol, i-1) $3 = (Dir *) 0x7ffa97a03ff8 (gdb) p *dir_index(vol, i-1) $4 = {w = {0, 0, 0, 0, 0}} (gdb) p *dir_index(vol, i-2) $5 = {w = {0, 0, 52431, 52423, 0}} (gdb) p *dir_index(vol, i) Cannot access memory at address 0x7ff997a04002 (gdb) p *dir_index(vol, i+2) Cannot access memory at address 0x7ff997a04016 (gdb) p *dir_index(vol, i+1) Cannot access memory at address 0x7ff997a0400c (gdb) p vol->buckets * DIR_DEPTH * vol->segments $6 = 1246953472 (gdb) bt #0 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) at CacheDir.cc:382 #1 0x0068aba2 in Vol::handle_recover_from_data (this=0x16057d0, event=3900, data=0x16058a0) at Cache.cc:1384 #2 0x004e8e1c in Continuation::handleEvent (this=0x16057d0, event=3900, data=0x16058a0) at ../iocore/eventsystem/I_Continuation.h:146 #3 0x00692385 in AIOCallbackInternal::io_complete (this=0x16058a0, event=1, data=0x135afc0) at ../../iocore/aio/P_AIO.h:80 #4 0x004e8e1c in Continuation::handleEvent (this=0x16058a0, event=1, data=0x135afc0) at ../iocore/eventsystem/I_Continuation.h:146 #5 0x00700fec in EThread::process_event (this=0x736c4010, e=0x135afc0, calling_code=1) at UnixEThread.cc:142 #6 0x007011ff in EThread::execute (this=0x736c4010) at UnixEThread.cc:191 #7 0x006ff8c2 in spawn_thread_internal (a=0x1356040) at Thread.cc:88 #8 0x7797e8ca in start_thread () from /lib/libpthread.so.0 #9 0x755c6b6d in clone () from /lib/libc.so.6 #10 0x in ?? () {code} This is fixed by running "traffic_server -Kk" to clear the cache. But after a few days the issue reappears. I will keep the current faulty setup as-is in case you need me to provide more data. I tried to make a core dump but it took a couple of GB even after gzip (I can however provide it on request). *Edit* OS is Debian GNU/Linux 6.0.6 with custom built kernel 3.2.13-grsec--grs-ipv6-64 was: I use ATS as a reverse proxy. I have a fairly large disk cache consisting of 2x 10TB raw disks. I do not use cache compression. After a few days of running (this is a dev machine - not handling any traffic) ATS begins to crash with a segfault shortly after start: [Jan 11 16:11:00.690] Server {0x72bb8700} DEBUG: (rusage) took rusage snap 1357917060690487000 Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x720ad700 (LWP 17292)] 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) at CacheDir.cc:382 382 CacheDir.cc: No such file or directory. in CacheDir.cc (gdb) p i $1 = 214748365 (gdb) l 377 in CacheDir.cc (gdb) p dir_index(vol, i) $2 = (Dir *) 0x7ff997a04002 (gdb) p dir_index(vol, i-1) $3 = (Dir *) 0x7ffa97a03ff8 (gdb) p *dir_index(vol, i-1) $4 = {w = {0, 0, 0, 0, 0}} (gdb) p *dir_index(vol, i-2) $5 = {w = {0, 0, 52431, 52423, 0}} (gdb) p *dir_index(vol, i) Cannot access memory at address 0x7ff997a04002 (gdb) p *dir_index(vol, i+2) Cannot access memory at address 0x7ff997a04016 (gdb) p *dir_index(vol, i+1) Cannot access memory at address 0x7ff997a0400c (gdb) p vol->buckets * DIR_DEPTH * vol->segments $6 = 1246953472 (gdb) bt #0 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) at CacheDir.cc:382 #1 0x0068aba2 in Vol::handle_recover_from_data (this=0x16057d0, event=3900, data=0x16058a0) at Cache.cc:1384 #2 0x004e8e1c in Continuation::handleEvent (this=0x16057d0, event=3900, data=0x16058a0) at ../iocore/eventsystem/I_Continuation.h:146 #3 0x00692385 in AIOCallbackInternal::io_complete (this=0x16058a0, event=1, data=0x135afc0) at ../../iocore/aio/P_AIO.h:80 #4 0x004e8e1c in Continuation::handleEvent (this=0x16058a0, event=1, data=0x135afc0) at ../iocore/eventsystem/I_Continuation.h:146 #5 0x00700fec in EThread::process_event (this=0x736c4010, e=0x135afc0, calling_code=1) at UnixEThread.cc:142 #6 0x007011ff in EThread::execute (this=0x736c4010) at UnixEThread.cc:
[jira] [Updated] (TS-1648) Segmentation fault in dir_clear_range()
[ https://issues.apache.org/jira/browse/TS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Geffon updated TS-1648: - Fix Version/s: 4.1.2 > Segmentation fault in dir_clear_range() > --- > > Key: TS-1648 > URL: https://issues.apache.org/jira/browse/TS-1648 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 3.3.0, 3.2.0 > Environment: reverse proxy >Reporter: Tomasz Kuzemko >Assignee: John Plevyak > Labels: A > Fix For: 3.3.4, 4.1.2 > > Attachments: > 0001-Fix-for-TS-1648-Segmentation-fault-in-dir_clear_rang.patch, > cachedir_int64-jp-1.patch > > > I use ATS as a reverse proxy. I have a fairly large disk cache consisting of > 2x 10TB raw disks. I do not use cache compression. After a few days of > running (this is a dev machine - not handling any traffic) ATS begins to > crash with a segfault shortly after start: > {code} > [Jan 11 16:11:00.690] Server {0x72bb8700} DEBUG: (rusage) took rusage > snap 1357917060690487000 > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x720ad700 (LWP 17292)] > 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) > at CacheDir.cc:382 > 382 CacheDir.cc: No such file or directory. > in CacheDir.cc > (gdb) p i > $1 = 214748365 > (gdb) l > 377 in CacheDir.cc > (gdb) p dir_index(vol, i) > $2 = (Dir *) 0x7ff997a04002 > (gdb) p dir_index(vol, i-1) > $3 = (Dir *) 0x7ffa97a03ff8 > (gdb) p *dir_index(vol, i-1) > $4 = {w = {0, 0, 0, 0, 0}} > (gdb) p *dir_index(vol, i-2) > $5 = {w = {0, 0, 52431, 52423, 0}} > (gdb) p *dir_index(vol, i) > Cannot access memory at address 0x7ff997a04002 > (gdb) p *dir_index(vol, i+2) > Cannot access memory at address 0x7ff997a04016 > (gdb) p *dir_index(vol, i+1) > Cannot access memory at address 0x7ff997a0400c > (gdb) p vol->buckets * DIR_DEPTH * vol->segments > $6 = 1246953472 > (gdb) bt > #0 0x00696a71 in dir_clear_range (start=640, end=17024, > vol=0x16057d0) at CacheDir.cc:382 > #1 0x0068aba2 in Vol::handle_recover_from_data (this=0x16057d0, > event=3900, data=0x16058a0) at Cache.cc:1384 > #2 0x004e8e1c in Continuation::handleEvent (this=0x16057d0, > event=3900, data=0x16058a0) at ../iocore/eventsystem/I_Continuation.h:146 > #3 0x00692385 in AIOCallbackInternal::io_complete (this=0x16058a0, > event=1, data=0x135afc0) at ../../iocore/aio/P_AIO.h:80 > #4 0x004e8e1c in Continuation::handleEvent (this=0x16058a0, event=1, > data=0x135afc0) at ../iocore/eventsystem/I_Continuation.h:146 > #5 0x00700fec in EThread::process_event (this=0x736c4010, > e=0x135afc0, calling_code=1) at UnixEThread.cc:142 > #6 0x007011ff in EThread::execute (this=0x736c4010) at > UnixEThread.cc:191 > #7 0x006ff8c2 in spawn_thread_internal (a=0x1356040) at Thread.cc:88 > #8 0x7797e8ca in start_thread () from /lib/libpthread.so.0 > #9 0x755c6b6d in clone () from /lib/libc.so.6 > #10 0x in ?? () > {code} > This is fixed by running "traffic_server -Kk" to clear the cache. But after a > few days the issue reappears. > I will keep the current faulty setup as-is in case you need me to provide > more data. I tried to make a core dump but it took a couple of GB even after > gzip (I can however provide it on request). > *Edit* > OS is Debian GNU/Linux 6.0.6 with custom built kernel > 3.2.13-grsec--grs-ipv6-64 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (TS-1648) Segmentation fault in dir_clear_range()
[ https://issues.apache.org/jira/browse/TS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Geffon updated TS-1648: - Fix Version/s: (was: 4.1.2) > Segmentation fault in dir_clear_range() > --- > > Key: TS-1648 > URL: https://issues.apache.org/jira/browse/TS-1648 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 3.3.0, 3.2.0 > Environment: reverse proxy >Reporter: Tomasz Kuzemko >Assignee: John Plevyak > Labels: A > Fix For: 3.3.4 > > Attachments: > 0001-Fix-for-TS-1648-Segmentation-fault-in-dir_clear_rang.patch, > cachedir_int64-jp-1.patch > > > I use ATS as a reverse proxy. I have a fairly large disk cache consisting of > 2x 10TB raw disks. I do not use cache compression. After a few days of > running (this is a dev machine - not handling any traffic) ATS begins to > crash with a segfault shortly after start: > {code} > [Jan 11 16:11:00.690] Server {0x72bb8700} DEBUG: (rusage) took rusage > snap 1357917060690487000 > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x720ad700 (LWP 17292)] > 0x00696a71 in dir_clear_range (start=640, end=17024, vol=0x16057d0) > at CacheDir.cc:382 > 382 CacheDir.cc: No such file or directory. > in CacheDir.cc > (gdb) p i > $1 = 214748365 > (gdb) l > 377 in CacheDir.cc > (gdb) p dir_index(vol, i) > $2 = (Dir *) 0x7ff997a04002 > (gdb) p dir_index(vol, i-1) > $3 = (Dir *) 0x7ffa97a03ff8 > (gdb) p *dir_index(vol, i-1) > $4 = {w = {0, 0, 0, 0, 0}} > (gdb) p *dir_index(vol, i-2) > $5 = {w = {0, 0, 52431, 52423, 0}} > (gdb) p *dir_index(vol, i) > Cannot access memory at address 0x7ff997a04002 > (gdb) p *dir_index(vol, i+2) > Cannot access memory at address 0x7ff997a04016 > (gdb) p *dir_index(vol, i+1) > Cannot access memory at address 0x7ff997a0400c > (gdb) p vol->buckets * DIR_DEPTH * vol->segments > $6 = 1246953472 > (gdb) bt > #0 0x00696a71 in dir_clear_range (start=640, end=17024, > vol=0x16057d0) at CacheDir.cc:382 > #1 0x0068aba2 in Vol::handle_recover_from_data (this=0x16057d0, > event=3900, data=0x16058a0) at Cache.cc:1384 > #2 0x004e8e1c in Continuation::handleEvent (this=0x16057d0, > event=3900, data=0x16058a0) at ../iocore/eventsystem/I_Continuation.h:146 > #3 0x00692385 in AIOCallbackInternal::io_complete (this=0x16058a0, > event=1, data=0x135afc0) at ../../iocore/aio/P_AIO.h:80 > #4 0x004e8e1c in Continuation::handleEvent (this=0x16058a0, event=1, > data=0x135afc0) at ../iocore/eventsystem/I_Continuation.h:146 > #5 0x00700fec in EThread::process_event (this=0x736c4010, > e=0x135afc0, calling_code=1) at UnixEThread.cc:142 > #6 0x007011ff in EThread::execute (this=0x736c4010) at > UnixEThread.cc:191 > #7 0x006ff8c2 in spawn_thread_internal (a=0x1356040) at Thread.cc:88 > #8 0x7797e8ca in start_thread () from /lib/libpthread.so.0 > #9 0x755c6b6d in clone () from /lib/libc.so.6 > #10 0x in ?? () > {code} > This is fixed by running "traffic_server -Kk" to clear the cache. But after a > few days the issue reappears. > I will keep the current faulty setup as-is in case you need me to provide > more data. I tried to make a core dump but it took a couple of GB even after > gzip (I can however provide it on request). > *Edit* > OS is Debian GNU/Linux 6.0.6 with custom built kernel > 3.2.13-grsec--grs-ipv6-64 -- This message was sent by Atlassian JIRA (v6.1.5#6160)