[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14116551#comment-14116551
]
Nikolai Gorchilov commented on TS-3032:
---
This is to confirm, FATAL: ats_malloc: couldn't allocate bytes is due to
reaching vm.max_map_count limit. Closing the ticket as non-ATS issue.
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
Attachments: memory.d.png
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
void*)+0x22b)[0x59270b]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server[0x714a60]
/z/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x1ed)[0x7077cd]
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115168#comment-14115168
]
Nikolai Gorchilov commented on TS-3032:
---
In the mean time, the smaller box, running with --enable-reclaimable-freelist
buld, crashed with the following message:
FATAL: Failed to mmap 12288 bytes, Out of memory, or the process's maximum
number of mappings would have been exceeded(if so, you can enlarge
'vm.max_map_count' by sysctl in linux).
During the crash free memory was above 10G and vm.max_map_count is 65530.
Could vm.max_map_count be the actual reason for all crashes?
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115401#comment-14115401
]
Zhao Yongming commented on TS-3032:
---
yeah, 64k is too small for you, I'd suggest you 128K, you may use 256K I
think.
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
Attachments: memory.d.png
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
void*)+0x22b)[0x59270b]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server[0x714a60]
/z/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x1ed)[0x7077cd]
/z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x736111]
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115416#comment-14115416
]
Zhao Yongming commented on TS-3032:
---
well, looks your memory is starting from 20G, I'd think that your index memory
is nearly about 20G, that indicate you may have ~20TB storage, if you haven't
change proxy.config.cache.min_average_object_size, is this right?
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
Attachments: memory.d.png
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
void*)+0x22b)[0x59270b]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server[0x714a60]
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115442#comment-14115442
]
Nikolai Gorchilov commented on TS-3032:
---
I prefer to keep only objects smaller than 64k in RAM, thus
proxy.config.cache.ram_cache_cutoff = 65536. The question was about the
possibility this limit to trigger the un-handled out-of-memory exception I'm
facing.
Total storage is just bellow 8TB and proxy.config.cache.min_average_object_size
= 8000.
RAM cache is 1G (proxy.config.cache.ram_cache.size = 1073741824)
Here's the current ATS memory allocation on this box:
{noformat}
allocated |in-use | type size | free list name
|||--
67108864 | 10485760 |2097152 |
memory/ioBufAllocator[14]
738197504 | 679477248 |1048576 |
memory/ioBufAllocator[13]
100663296 | 58720256 | 524288 |
memory/ioBufAllocator[12]
50331648 | 34340864 | 262144 |
memory/ioBufAllocator[11]
50331648 | 35782656 | 131072 |
memory/ioBufAllocator[10]
312475648 | 270270464 | 65536 | memory/ioBufAllocator[9]
1249902592 | 1039499264 | 32768 | memory/ioBufAllocator[8]
293601280 | 261799936 | 16384 | memory/ioBufAllocator[7]
418381824 | 397361152 | 8192 | memory/ioBufAllocator[6]
1760559104 | 1741463552 | 4096 | memory/ioBufAllocator[5]
786432 | 550912 | 2048 | memory/ioBufAllocator[4]
393216 | 275456 | 1024 | memory/ioBufAllocator[3]
3997696 |3930624 |512 | memory/ioBufAllocator[2]
5537792 | 273152 |256 | memory/ioBufAllocator[1]
13795328 |1084544 |128 | memory/ioBufAllocator[0]
2580480 | 320352 | 96 | memory/eventAllocator
29511680 | 29227040 | 80 | memory/mutexAllocator
38346752 | 31291840 | 64 | memory/ioBlockAllocator
27420672 | 26394288 | 48 | memory/ioDataAllocator
97720320 | 97719360 |240 | memory/ioAllocator
0 | 0 |384 | memory/socksAllocator
0 | 0 |128 |
memory/udpReadContAllocator
0 | 0 |160 |
memory/udpPacketAllocator
256929792 | 254691360 |672 | memory/netVCAllocator
0 | 0 |128 |
memory/UDPIOEventAllocator
92160 | 2880 |720 | memory/sslNetVCAllocator
4546560 |4426496 | 64 | memory/RamCacheLRUEntry
0 | 0 | 96 |
memory/RamCacheCLFUSEntry
983040 | 964960 |160 | memory/openDirEntry
14336 |112 |112 |
memory/migrateToInterimCache
0 | 0 | 48 | memory/evacuationKey
6144 | 0 | 48 | memory/cacheRemoveCont
2691072 |2683968 | 96 | memory/evacuationBlock
7004160 |6917760 |960 | memory/cacheVConnection
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 |592 |
memory/clusterVCAllocator
0 | 0 |112 |
memory/inControlAllocator
0 | 0 |112 |
memory/outControlAllocator
0 | 0 | 48 |
memory/ClusterVConnectionCache::Entry
0 | 0 |576 |
memory/cacheContAllocator
0 | 0 | 16 |
memory/DNSRequestDataAllocator
6364928 | 33856 | 33856 | memory/dnsBufAllocator
983040 | 21760 | 1280 | memory/dnsEntryAllocator
7372800 | 46080 | 2304 |
memory/hostDBContAllocator
0 | 0 |112 |
memory/OneWayTunnelAllocator
64225280 | 64210944 | 2048 | memory/hdrStrHeap
120848384 | 117352448 | 2048 | memory/hdrHeap
163840 | 144896 |256 |
memory/httpCacheAltAllocator
0 | 0 | 48 |
memory/CongestRequestParamAllocator
0 | 0 |144 |
memory/CongestionDBContAllocator
0 | 0 |128 |
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115477#comment-14115477
]
Nikolai Gorchilov commented on TS-3032:
---
I also suspect vm.max_map_count to be the reason behind the failure. Just
checked the number of allocations at the other box and it stands at 50222
currently.
Added crontab logging facility in order to track number of ATS memory maps:
{noformat}
* * * * * cat /proc/18455/maps | wc -l | xargs echo $(date): /tmp/maps.log
{noformat}
If I'm right ATS will crash soon and the last logged value would be just bellow
vm.max_map_count (65530)
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
Attachments: memory.d.png
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115529#comment-14115529
]
Leif Hedstrom commented on TS-3032:
---
Yeah, we need to document this better. It was unclear if we still needed this
many mmap's, but apparently we do :). I wonder, does the interim cache consume
more mmap areas?
Fwiw, there's a setting in ATS as well for this, but we set it way, way high
(such that the sysctl would always become the bottleneck):
{code}
{RECT_CONFIG, proxy.config.system.mmap_max, RECD_INT, 2097152,
{code}
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
Attachments: memory.d.png
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115536#comment-14115536
]
Nikolai Gorchilov commented on TS-3032:
---
I'm can't tell what is interim cache contribution to mmap exhaust as I don't
have comparable size installations without interim storage. I can only say that
all crashing setups do use interim storage.
What is going to happen if I decrease proxy.config.system.mmap_max vm.
max_map_count?
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
Attachments: memory.d.png
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
void*)+0x22b)[0x59270b]
/z/bin/traffic_server(HttpSM::main_handler(int,
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115745#comment-14115745
]
Leif Hedstrom commented on TS-3032:
---
Doing that change would not make things better. What you need to try is to try
setting your vm.max_map_count bigger, and restart ATS. E.g. in sysctl.conf
{code}
vm.max_map_count = 2097152
{code}
or, using sysctl from command line.
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
Attachments: memory.d.png
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
void*)+0x22b)[0x59270b]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server[0x714a60]
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113762#comment-14113762
]
Zhao Yongming commented on TS-3032:
---
any update? [~ngorchilov]
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
void*)+0x22b)[0x59270b]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server[0x714a60]
/z/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x1ed)[0x7077cd]
/z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x736111]
/z/bin/traffic_server(EThread::execute()+0x4fc)[0x736bcc]
/z/bin/traffic_server[0x7353aa]
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113798#comment-14113798
]
Nikolai Gorchilov commented on TS-3032:
---
Unfortunately, I'm out of office this week. Will update you early next week.
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
void*)+0x22b)[0x59270b]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server[0x714a60]
/z/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x1ed)[0x7077cd]
/z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x736111]
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14110941#comment-14110941
]
Zhao Yongming commented on TS-3032:
---
I'd suggest you get some tool to log the memory usage and other history data. a
tool we used very often in tracing issues like this is
https://github.com/alibaba/tsar
https://blog.zymlinux.net/index.php/archives/251 , any other tool that can find
out the data to compare is great.
when we deal with TS-1006, I even make some excel sheet to point out that the
memory is a big problem, the more data the better
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109022#comment-14109022
]
Nikolai Gorchilov commented on TS-3032:
---
* /proc/meminfo
{noformat}
MemTotal: 131999852 kB
MemFree:81857040 kB
Buffers: 357000 kB
Cached: 25975284 kB
SwapCached:0 kB
Active: 30757792 kB
Inactive: 15528096 kB
Active(anon): 19953648 kB
Inactive(anon): 292 kB
Active(file): 10804144 kB
Inactive(file): 15527804 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 1800 kB
Writeback: 0 kB
AnonPages: 19956200 kB
Mapped:47440 kB
Shmem: 332 kB
Slab:1794776 kB
SReclaimable: 990372 kB
SUnreclaim: 804404 kB
KernelStack:4872 kB
PageTables:43808 kB
NFS_Unstable: 0 kB
Bounce:0 kB
WritebackTmp: 0 kB
CommitLimit:6524 kB
Committed_AS: 17297316 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 488860 kB
VmallocChunk: 34290894976 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free:0
HugePages_Rsvd:0
HugePages_Surp:0
Hugepagesize: 2048 kB
DirectMap4k: 1125372 kB
DirectMap2M:93210624 kB
DirectMap1G:41943040 kB
{noformat}
* /proc/pid/status
{noformat}
Name: [ET_NET 0]
State: S (sleeping)
Tgid: 24037
Pid:24037
PPid: 24025
TracerPid: 0
Uid:13 13 13 13
Gid:13 13 13 13
FDSize: 262144
Groups: 0
VmPeak: 25063316 kB
VmSize: 24999124 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 19672548 kB
VmRSS: 19672300 kB
VmData: 24902004 kB
VmStk: 136 kB
VmExe: 4100 kB
VmLib: 9684 kB
VmPTE: 39768 kB
VmSwap:0 kB
Threads:145
SigQ: 0/1031099
SigPnd:
ShdPnd:
SigBlk:
SigIgn: 00381000
SigCgt: 000180004e4f
CapInh:
CapPrm: 5402
CapEff: 5400
CapBnd: 001f
Seccomp:0
Cpus_allowed:
Cpus_allowed_list: 0-31
Mems_allowed: ,0003
Mems_allowed_list: 0-1
voluntary_ctxt_switches:130883732
nonvoluntary_ctxt_switches: 797084
{noformat}
* configure options
{noformat}
--with-group=proxy \
--with-xml=libxml2 \
--disable-static \
--disable-static-libts \
--disable-spdy \
--enable-interim-cache \
--enable-tproxy \
--enable-hwloc \
--enable-experimental-plugins \
--enable-example-plugins
{noformat}
* other system logs - nothing memory related available
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109133#comment-14109133
]
Zhao Yongming commented on TS-3032:
---
looks nothing unusal, I think that 'Cached: 25975284 kB' is caused by
the access logging, then we need more infomation on ATS:
1. your ram cache setting: proxy.config.cache.ram_cache.size, if not set please
tell us your storage device usage, and cache min_average_object_size.
2. let us dump some memory details in the ATS itself:
https://docs.trafficserver.apache.org/en/latest/sdk/troubleshooting-tips/debugging-memory-leaks.en.html
and we should better get all those data the breaking point too :D
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109151#comment-14109151
]
Nikolai Gorchilov commented on TS-3032:
---
* traffic_line -r proxy.config.cache.ram_cache.size
{noformat}
1073741824
{noformat}
* traffic_line -s proxy.config.dump_mem_info_frequency -v 60
{noformat}
allocated |in-use | type size | free list name
|||--
67108864 | 0 |2097152 |
memory/ioBufAllocator[14]
268435456 | 80740352 |1048576 |
memory/ioBufAllocator[13]
83886080 |7340032 | 524288 |
memory/ioBufAllocator[12]
218103808 | 11272192 | 262144 |
memory/ioBufAllocator[11]
88080384 |9568256 | 131072 |
memory/ioBufAllocator[10]
243269632 | 36831232 | 65536 | memory/ioBufAllocator[9]
2797600768 | 2661744640 | 32768 | memory/ioBufAllocator[8]
443547648 | 184287232 | 16384 | memory/ioBufAllocator[7]
612630528 | 525918208 | 8192 | memory/ioBufAllocator[6]
1208483840 | 905179136 | 4096 | memory/ioBufAllocator[5]
4456448 |4358144 | 2048 | memory/ioBufAllocator[4]
1179648 |1078272 | 1024 | memory/ioBufAllocator[3]
9043968 |9004032 |512 | memory/ioBufAllocator[2]
4521984 |1761536 |256 | memory/ioBufAllocator[1]
39108608 |1824768 |128 | memory/ioBufAllocator[0]
120840192 | 249408 | 96 | memory/eventAllocator
10270720 |7616480 | 80 | memory/mutexAllocator
46039040 | 17172480 | 64 | memory/ioBlockAllocator
28323840 | 20658912 | 48 | memory/ioDataAllocator
55449600 | 40364640 |240 | memory/ioAllocator
0 | 0 |384 | memory/socksAllocator
0 | 0 |128 |
memory/udpReadContAllocator
0 | 0 |160 |
memory/udpPacketAllocator
105627648 | 71248800 |672 | memory/netVCAllocator
0 | 0 |128 |
memory/UDPIOEventAllocator
92160 | 68400 |720 | memory/sslNetVCAllocator
0 | 0 | 64 | memory/RamCacheLRUEntry
32526336 | 32181984 | 96 |
memory/RamCacheCLFUSEntry
1454080 |1420640 |160 | memory/openDirEntry
14336 | 0 |112 |
memory/migrateToInterimCache
0 | 0 | 48 | memory/evacuationKey
6144 | 0 | 48 | memory/cacheRemoveCont
10936320 | 10875648 | 96 | memory/evacuationBlock
11673600 | 11447040 |960 | memory/cacheVConnection
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 |592 |
memory/clusterVCAllocator
0 | 0 |112 |
memory/inControlAllocator
0 | 0 |112 |
memory/outControlAllocator
0 | 0 | 48 |
memory/ClusterVConnectionCache::Entry
0 | 0 |576 |
memory/cacheContAllocator
0 | 0 | 16 |
memory/DNSRequestDataAllocator
1354240 | 33856 | 33856 | memory/dnsBufAllocator
1146880 | 20480 | 1280 | memory/dnsEntryAllocator
24477696 | 73728 | 2304 |
memory/hostDBContAllocator
0 | 0 |112 |
memory/OneWayTunnelAllocator
168034304 | 47165440 | 2048 | memory/hdrStrHeap
217841664 | 75976704 | 2048 | memory/hdrHeap
229376 | 70400 |256 |
memory/httpCacheAltAllocator
0 | 0 | 48 |
memory/CongestRequestParamAllocator
0 | 0 |144 |
memory/CongestionDBContAllocator
0 | 0 |128 | memory/RemapPluginsAlloc
0 | 0 | 7760 |
memory/httpUpdateSMAllocator
0 | 0 | 48 |
memory/CacheLookupHttpConfigAllocator
13848576 |9485280 |224 |
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109214#comment-14109214
]
Zhao Yongming commented on TS-3032:
---
well, you have 7368964608 memory in the freelist, and 4893378608 in use, that
is 66% in use. with about 8000 active connections. all sounds not so bad except
that 7G is far smaller than that 19G from the pid summary, why?
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
void*)+0x22b)[0x59270b]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server[0x714a60]
/z/bin/traffic_server(NetHandler::mainNetEvent(int,
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109216#comment-14109216
]
Zhao Yongming commented on TS-3032:
---
I'd like you keep colect those data for some more days, the same time(to get
the same load) if you can, to figure out which component is wasting more
memories.
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
void*)+0x22b)[0x59270b]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server[0x714a60]
/z/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x1ed)[0x7077cd]
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109225#comment-14109225
]
Zhao Yongming commented on TS-3032:
---
and if you have more than one boxes with that issue, please consider test one
box with the following tweak:
1. re-install with reclaim freelist enabled. and make sure reclaim is enabled
in the records.config
2. use the standard LRU: set proxy.config.cache.ram_cache.algorithm to 1
and if you have more system that can do a release test, we can identify which
release is proved to be correct. :D
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109294#comment-14109294
]
Nikolai Gorchilov commented on TS-3032:
---
Yes, I have two boxes that have this issue and one of them is available for
testing. Have in mind it has just 48G of RAM.
Here's the relevant info on this box just after restarting the new build of
traffic server with requested config changes:
* cat /proc/meminfo
{noformat}
MemTotal: 49361540 kB
MemFree:36481844 kB
MemAvailable: 37318832 kB
Buffers: 139888 kB
Cached: 1174072 kB
SwapCached:0 kB
Active: 11619048 kB
Inactive: 472492 kB
Active(anon): 10944848 kB
Inactive(anon):43404 kB
Active(file): 674200 kB
Inactive(file): 429088 kB
Unevictable: 175688 kB
Mlocked: 12852 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 10953252 kB
Mapped:61956 kB
Shmem: 44052 kB
Slab: 513432 kB
SReclaimable: 143176 kB
SUnreclaim: 370256 kB
KernelStack:7728 kB
PageTables:29912 kB
NFS_Unstable: 0 kB
Bounce:0 kB
WritebackTmp: 0 kB
CommitLimit:24680768 kB
Committed_AS: 11943052 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 285388 kB
VmallocChunk: 34359406432 kB
AnonHugePages: 10416128 kB
DirectMap4k: 13824 kB
DirectMap2M: 4171776 kB
DirectMap1G:48234496 kB
{noformat}
* /proc/pid/status after traffic_server restart
{noformat}
Name: [ET_NET 0]
State: S (sleeping)
Tgid: 24406
Ngid: 0
Pid:24406
PPid: 24397
TracerPid: 0
Uid:13 13 13 13
Gid:13 13 13 13
FDSize: 128
Groups: 0
VmPeak: 19229780 kB
VmSize: 19229272 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 10616928 kB
VmRSS: 10616928 kB
VmData: 19129068 kB
VmStk: 136 kB
VmExe: 4096 kB
VmLib: 10736 kB
VmPTE: 21840 kB
VmSwap:0 kB
Threads:162
SigQ: 0/192701
SigPnd:
ShdPnd:
SigBlk:
SigIgn: 00381000
SigCgt: 000180004e4f
CapInh:
CapPrm: 5402
CapEff: 5400
CapBnd: 001f
Cpus_allowed:
Cpus_allowed_list: 0-15
voluntary_ctxt_switches:26618
nonvoluntary_ctxt_switches: 135
{noformat}
* relevant records.config options:
{noformat}
CONFIG proxy.config.allocator.enable_reclaim INT 1
CONFIG proxy.config.cache.ram_cache_cutoff INT 65536
CONFIG proxy.config.cache.ram_cache.size INT 1073741824
CONFIG proxy.config.cache.ram_cache.algorithm INT 1
{noformat}
NB! Could it be related to ram_cache_cutoff? Seems all crashes are for
allocations above 64k.
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14108393#comment-14108393
]
Nikolai Gorchilov commented on TS-3032:
---
Another manifestation:
{noformat}
FATAL: ats_malloc: couldn't allocate 65536 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b0203d6a837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b0203d6dc50]
/z/bin/traffic_server(HdrHeap::allocate_obj(int, int)+0xa4)[0x62e414]
/z/bin/traffic_server(mime_hdr_copy_onto(MIMEHdrImpl*, HdrHeap*, MIMEHdrImpl*,
HdrHeap*, bool)+0x102)[0x634bc2]
/z/bin/traffic_server(http_hdr_copy_onto(HTTPHdrImpl*, HdrHeap*, HTTPHdrImpl*,
HdrHeap*, bool)+0x87)[0x62a427]
/z/bin/traffic_server(HttpSM::set_next_state()+0x9c1)[0x5a0cc1]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c13]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c82a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b0211806508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999d2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0538]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c13]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0538]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int, void*)+0x103)[0x59b003]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ada8]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x1d6)[0x57bc16]
/z/bin/traffic_server[0x6d56b8]
/z/bin/traffic_server(CacheVC::openReadStartEarliest(int,
Event*)+0x7d0)[0x6d9120]
/z/bin/traffic_server(CacheVC::handleReadDone(int, Event*)+0x27b)[0x6b79cb]
/z/bin/traffic_server(AIOCallbackInternal::io_complete(int,
void*)+0x3d)[0x662e5d]
/z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x736121]
/z/bin/traffic_server(EThread::execute()+0x6a3)[0x736d83]
/z/bin/traffic_server[0x7353ba]
/lib64/libpthread.so.0(+0x7034)[0x2b0205370034]
/lib64/libc.so.6(clone+0x6d)[0x2b02060b475d]
{noformat}
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14108410#comment-14108410
]
Zhao Yongming commented on TS-3032:
---
I don't know who have any sucess story on BIG memory system, I'd like to hear
if any.
for the problem you have, please attach some more data such as:
1. /proc/meminfo
2. the traffic_server process status: /proc//status
3. more system log related to alloc and memory, such as dmesg syslog
and, please tell us your configure options when building the binary too. hopes
that will help us inspect the problem.
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105713#comment-14105713
]
Brian Geffon commented on TS-3032:
--
Hmm 155k headers seem pretty large, do all of your previous stack traces have
such large values? Also, are such header set sizes normal for you guys?
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
void*)+0x22b)[0x59270b]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server[0x714a60]
/z/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x1ed)[0x7077cd]
/z/bin/traffic_server(EThread::process_event(Event*,
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106114#comment-14106114
]
Nikolai Gorchilov commented on TS-3032:
---
This system is running on forward proxy mode, so we're not in a control of
header sizes.
Here's a backtrace from another location:
{noformat}
FATAL: ats_malloc: couldn't allocate 108896 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b1605203837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b1605206c50]
/z/bin/traffic_server(ProcessManager::signalManager(int, char const*,
int)+0x3a)[0x64b91a]
/z/bin/traffic_server(RecMessageSend(RecMessageHdr*)+0x48)[0x72bda8]
/z/bin/traffic_server(send_push_message()+0xdd)[0x7272ed]
/z/bin/traffic_server(sync_cont::sync(int, Event*)+0xd)[0x72e67d]
/z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x736111]
/z/bin/traffic_server(EThread::execute()+0x2fb)[0x7369cb]
/z/bin/traffic_server[0x7353aa]
/lib64/libpthread.so.0(+0x7034)[0x2b1606809034]
/lib64/libc.so.6(clone+0x6d)[0x2b160754d75d]
{noformat}
HTH
FATAL: ats_malloc: couldn't allocate XX bytes
-
Key: TS-3032
URL: https://issues.apache.org/jira/browse/TS-3032
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
Labels: crash
Fix For: 5.2.0
ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
to memory allocation issue. Happens once or twice a week.
Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
suspicious in dmesg.
{noformat}
FATAL: ats_malloc: couldn't allocate 155648 bytes
/z/bin/traffic_server - STACK TRACE:
/z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
/z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
/z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
HdrHeap*)+0x8f)[0x62a54f]
/z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
/z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
/z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
/z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
(*)(HttpTransact::State*))+0x66)[0x58e356]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
/z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
void*)+0x236)[0x2b626342b508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int,
void*)+0x180)[0x59b070]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
void*)+0x173)[0x57bbb3]
/z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
/z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
/z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
/z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
/z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
