[ https://issues.apache.org/jira/browse/TS-4279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
taoyunxing updated TS-4279: --------------------------- Description: CPU: 40 cores, Mem: 120GB, Disk: 1*300 sys + 11 * 899GB, OS: CentOS 6.6 records.config: CONFIG proxy.config.cache.min_average_object_size INT 1048576 CONFIG proxy.config.cache.ram_cache.algorithm INT 1 CONFIG proxy.config.cache.ram_cache_cutoff INT 4194304 CONFIG proxy.config.cache.ram_cache.size INT 64424509440 storage.config: /dev/sdc id=cache.disk.1 I encountered a kind of dead loop situation of ats 5.3.1 on two production hosts, a burst of warning is seen by me in the diags.log like this: {code} [Mar 16 13:04:32.730] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.732] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.733] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.735] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.737] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.739] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.742] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.744] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.747] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.750] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.753] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.756] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... {code} ats restart in every serval hours, and the TIMEWAIT count is huge above the ESTABLISH TCP connection count. the following is the current dir snapshot of disk /dev/sdc on one of the hosts: {code} Directory for [cache.disk.1 172032:109741163] Bytes: 8573600 Segments: 14 Buckets: 15310 Entries: 857360 Full: 852904 Empty: 4085 Stale: 0 Free: 371 Bucket Fullness: 4085 15800 32044 41621 42175 33137 22232 12605 Segment Fullness: 60903 60918 60914 60947 60956 60947 60872 60943 60918 60927 60858 60917 60927 60957 Freelist Fullness: 45 30 27 13 0 7 89 5 32 12 83 0 20 8 {code} I wonder why the value of freelist[4] is zero, which cause ats dead loop, anyone help me? thinks a lot. was: CPU 40 cores, Mem: 120GB, Disk: 1*300 sys + 11 * 899GB, records.config: CONFIG proxy.config.cache.min_average_object_size INT 1048576 CONFIG proxy.config.cache.ram_cache.algorithm INT 1 CONFIG proxy.config.cache.ram_cache_cutoff INT 4194304 CONFIG proxy.config.cache.ram_cache.size INT 64424509440 storage.config: /dev/sdc id=cache.disk.1 I encountered a kind of dead loop situation of ats 5.3.1 on two production hosts, a burst of warning is seen by me in the diags.log like this: {code} [Mar 16 13:04:32.730] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.732] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.733] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.735] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.737] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.739] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.742] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.744] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.747] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.750] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.753] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... [Mar 16 13:04:32.756] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... {code} ats restart in every serval hours, and the TIMEWAIT count is huge above the ESTABLISH TCP connection count. the following is the current dir snapshot of those host: {code} Directory for [cache.disk.1 172032:109741163] Bytes: 8573600 Segments: 14 Buckets: 15310 Entries: 857360 Full: 852904 Empty: 4085 Stale: 0 Free: 371 Bucket Fullness: 4085 15800 32044 41621 42175 33137 22232 12605 Segment Fullness: 60903 60918 60914 60947 60956 60947 60872 60943 60918 60927 60858 60917 60927 60957 Freelist Fullness: 45 30 27 13 0 7 89 5 32 12 83 0 20 8 {code} I wonder why, anyone help me? thinks a lot. > ats fallen into dead loop for cache directory overflow > ------------------------------------------------------ > > Key: TS-4279 > URL: https://issues.apache.org/jira/browse/TS-4279 > Project: Traffic Server > Issue Type: Bug > Components: Cache > Affects Versions: 5.3.1 > Reporter: taoyunxing > Fix For: 6.2.0 > > > CPU: 40 cores, Mem: 120GB, Disk: 1*300 sys + 11 * 899GB, OS: CentOS 6.6 > records.config: > CONFIG proxy.config.cache.min_average_object_size INT 1048576 > CONFIG proxy.config.cache.ram_cache.algorithm INT 1 > CONFIG proxy.config.cache.ram_cache_cutoff INT 4194304 > CONFIG proxy.config.cache.ram_cache.size INT 64424509440 > storage.config: > /dev/sdc id=cache.disk.1 > I encountered a kind of dead loop situation of ats 5.3.1 on two production > hosts, a burst of warning is seen by me in the diags.log like this: > {code} > [Mar 16 13:04:32.730] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 > (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... > [Mar 16 13:04:32.732] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 > (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... > [Mar 16 13:04:32.733] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 > (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... > [Mar 16 13:04:32.735] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 > (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... > [Mar 16 13:04:32.737] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 > (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... > [Mar 16 13:04:32.739] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 > (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... > [Mar 16 13:04:32.742] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 > (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... > [Mar 16 13:04:32.744] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 > (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... > [Mar 16 13:04:32.747] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 > (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... > [Mar 16 13:04:32.750] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 > (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... > [Mar 16 13:04:32.753] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 > (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... > [Mar 16 13:04:32.756] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 > (freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging... > {code} > ats restart in every serval hours, and the TIMEWAIT count is huge above the > ESTABLISH TCP connection count. > the following is the current dir snapshot of disk /dev/sdc on one of the > hosts: > {code} > Directory for [cache.disk.1 172032:109741163] > Bytes: 8573600 > Segments: 14 > Buckets: 15310 > Entries: 857360 > Full: 852904 > Empty: 4085 > Stale: 0 > Free: 371 > Bucket Fullness: 4085 15800 32044 41621 > 42175 33137 22232 12605 > > Segment Fullness: 60903 60918 60914 60947 60956 > 60947 60872 60943 60918 60927 > 60858 60917 60927 60957 > Freelist Fullness: 45 30 27 13 0 > 7 89 5 32 12 > 83 0 20 8 > {code} > I wonder why the value of freelist[4] is zero, which cause ats dead loop, > anyone help me? thinks a lot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)