On Tue, Sep 01, 2020 at 01:04:14PM +0200, Ruediger Pluem wrote: > > > On 9/1/20 10:18 AM, Joe Orton wrote: > > On Mon, Aug 31, 2020 at 08:37:55AM +0200, Ruediger Pluem wrote: > >> On 8/30/20 7:22 AM, Christophe JAILLET wrote: > >>> Thread 66 (Thread 0x7f13a6525700 (LWP 7436)): > >>> > >>> #0 0x00007f13dd61a187 in kill () from /lib/x86_64-linux-gnu/libc.so.6 > >>> #1 <signal handler called> > >>> #2 0x00007f13dd619e97 in raise () from /lib/x86_64-linux-gnu/libc.so.6 > >>> #3 0x00007f13dd61b801 in abort () from /lib/x86_64-linux-gnu/libc.so.6 > >>> #4 0x00007f13ddc05f62 in apr_table_mergen (t=0x560cba82d790, > >>> key=key@entry=0x560cb949006a "Connection", > >>> val=val@entry=0x560cb9488f86 "close") at tables/apr_tables.c:750 > >>> #5 0x0000560cb9434fcf in ap_set_keepalive (r=r@entry=0x560cba7c9c60) at > >>> http_protocol.c:309 > >>> #6 0x0000560cb943a475 in ap_http_header_filter (f=0x560cba74f8e0, > >>> b=0x560cba65db80) at http_filters.c:1369 > >>> #7 0x0000560cb9458091 in ap_content_length_filter (f=0x560cba1d95c0, > >>> b=0x560cba65db80) at protocol.c:2040 > >>> > >> > >> My only guess is some sort of pool corruption. The key is a literal > >> and as such its memory address shouldn't be in the scope of any memory > >> managed by a pool. > > > > I agree. This is triggered by running TLSv1.2 tests with memcache > > configured as the session cache (a relatively new test). The memcache > > socache provider does *not* use AP_SOCACHE_FLAG_NOTMPSAFE so is used w/o > > locks held across multiple threads so that might be a good place to > > investigate. > > > > A pool is used for the retrieval of sessions, but it is the pool of the > associated connection (conn_rec) which seems uncritical as > we should not have two threads working with the same connection at the same > time. > > OTOH the pconf pool is used to init the session cache and the memcache apr > code creates subpools of it and uses these for allocations. > > I guess ms_find_conn in apr code should be inspected. It is called by > apr_memcache_getp and it uses these subpools (tp in this > case) for the below: > > balloc = apr_bucket_alloc_create((*conn)->tp); > (*conn)->bb = apr_brigade_create((*conn)->tp, balloc); > (*conn)->tb = apr_brigade_create((*conn)->tp, balloc); > > I think that tp can be used by multiple threads at the same time.
The subpools here are specific to the apr_memcache_conn_t, access to which is managed via apr_reslist_* in ms_find_conn() - so *should* be thread-safe I think? I think the subpool creation within ms_conn_construct() is also safe but may be missing the bigger picture. > BTW: How often does this test fail? I don't remember seeing it fail before. This test combination has only been running since r1879594 (July 7th). We could try adding an ab run or something to see if it's easier to trigger, it'd be generally a good idea for session cache testing anyway. Regards, Joe
