On Oct 7, 2014, at 1:26 AM, Matthieu Bienvenüe <[email protected]> wrote:
> OK so here is what I get in traffic.out when setting :
>
> CONFIG proxy.config.ssl.number.threads INT -1
> CONFIG proxy.config.diags.debug.enabled INT 1
> CONFIG proxy.config.diags.debug.tags STRING ssl
Could it be a 32-bit issue? I don’t have any 32-bit boxes any more, and we
don’t have any CI for it as well. I believe the decision was that as of 5.0, we
would only officially support 64-bit.
Can you test on a 64-bit box as well ? Also, did you send any request to get
this to trigger, or does it segfault right out the gate on startup?
— leif
>
>
>
> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl)
> [SSLNextProtocolAccept:mainEvent] event 202 netvc 0x438d0000
> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 16 ret: 1
> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl)
> ssl_servername_callback ssl=0xa5e0ee0 ad=112 lookup=0xa54f9c0 server=XXXX
> handshake_complete=0
> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl)
> ssl_servername_callback found SSL context 0xa552960 for requested name 'XXXX'
> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) advertising protocol
> http/1.1
> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) advertising protocol
> http/1.0
> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.527] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.527] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.527] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.528] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8194 ret: -1
> [Oct 7 09:24:57.528] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8194 ret: -1
> [Oct 7 09:24:57.528] Server {0x40709730} DEBUG: <SSLNetVConnection.cc:558
> (sslServerHandShakeEvent)> (ssl) SSL handshake error: SSL_ERROR_WANT_READ
> (2), errno=11
> [Oct 7 09:24:57.589] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.589] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.589] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl:
> 0xa5e0ee0 where: 32 ret: 1
> NOTE: Traffic Server received Sig 11: Segmentation fault
> /usr/bin/traffic_server - STACK TRACE:
> [0x4001e500]
> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup_entry+0x12)[0x4003c0f2]
> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup+0x24)[0x4003c3b4]
> /usr/bin/traffic_server[0x8308185]
> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(+0x12844)[0x40067844]
> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(SSL_accept+0x2a)[0x4008c73a]
> /usr/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x19)[0x8303d89]
> /usr/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x2b)[0x830446b]
> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0xb30)[0x8305270]
> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
> /usr/bin/traffic_server(main+0xf40)[0x80d4e30]
> /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x405c9e36]
> /usr/bin/traffic_server[0x80da229]
> [TrafficServer] using root directory '/usr'
> [Oct 7 09:24:59.682] Server {0x40709730} DEBUG: (ssl) setting SNI callbacks
> with for ctx 0x8a6b8b8
> [Oct 7 09:24:59.682] Server {0x40709730} DEBUG: (ssl) importing SNI names
> from /etc/trafficserver/ssl/new2014/100.pem
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) mapping 'XXXX' to
> certificate /etc/trafficserver/ssl/new2014/100.pem
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) indexed 'XXXX' with
> SSL_CTX 0x8a6b8b8
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) mapping 'XXXX' to
> certificate /etc/trafficserver/ssl/new2014/100.pem
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) indexed 'XXXX' with
> SSL_CTX 0x8a6b8b8
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) mapping 'YYYYYY' to
> certificate /etc/trafficserver/ssl/new2014/100.pem
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) indexed 'YYYYYY' with
> SSL_CTX 0x8a6b8b8
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) setting SNI callbacks
> with for ctx 0x8a76b30
> [Oct 7 09:24:59.684] Server {0x40709730} DEBUG: (ssl) indexed '*' with
> SSL_CTX 0x8a76b30
> [Oct 7 09:24:59.684] Server {0x40709730} DEBUG: (ssl) importing SNI names
> from /etc/trafficserver/ssl
>
>
>
>
>
> Le 03/10/2014 18:43, James Peach a écrit :
>> On Oct 3, 2014, at 3:32 AM, Matthieu Bienvenüe <[email protected]> wrote:
>>
>>> Any idea to solve this isssu ?
>> I did a quick test of setting proxy.config.ssl.number.threads to -1, and it
>> didn't crash for me. Can you enable ssl diagnostics and try again?
>>
>> CONFIG proxy.config.diags.debug.enabled INT 1
>> CONFIG proxy.config.diags.debug.tags STRING ssl
>>
>>> Matthieu
>>>
>>>
>>> Le 01/10/2014 09:50, Matthieu Bienvenüe a écrit :
>>>> Le 30/09/2014 17:47, Leif Hedstrom a écrit :
>>>>> On Sep 30, 2014, at 9:00 AM, Matthieu Bienvenüe <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Is that possible to do it on config instead of recompiling ATS ?
>>>>>
>>>>> What version are you using? I’m not 100% certain, but I’d expect Geffon’s
>>>>> additions to not have dedicated SSL threads would avoid the need for that
>>>>> patch as well? Brian? If I recall, with a recent version of ATS, you’d
>>>>> simply set proxy.config.ssl.number.threads to -1.
>>>> When I set this settings SSL don't work and I've the following stack trace
>>>> :
>>>>
>>>> NOTE: Traffic Server received Sig 11: Segmentation fault
>>>> /usr/bin/traffic_server - STACK TRACE:
>>>> [0x4001e500]
>>>> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup_entry+0x12)[0x4003c0f2]
>>>> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup+0x24)[0x4003c3b4]
>>>> /usr/bin/traffic_server[0x8308185]
>>>> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(+0x12844)[0x40067844]
>>>> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(SSL_accept+0x2a)[0x4008c73a]
>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x19)[0x8303d89]
>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x2b)[0x830446b]
>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0xb30)[0x8305270]
>>>> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
>>>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
>>>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
>>>> /usr/bin/traffic_server(main+0xf40)[0x80d4e30]
>>>> /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x405c9e36]
>>>> /usr/bin/traffic_server[0x80da229]
>>>> [TrafficServer] using root directory '/usr'
>>>>
>>>>> In either case, why is that patch not committed? Is there a Jira for it?
>>>>>
>>>>> — Leif
>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Matt
>>>>>> Le 30/09/2014 16:49, 英才 a écrit :
>>>>>>> disable AIO or patch https://github.com/phonehold/with-aio-ssl-init
>>>>>>> may solve your problem
>>>>>>>
>>>>>>> 在 2014年9月30日，下午10:41，Matthieu Bienvenüe <[email protected]> 写道：
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>>
>>>>>>>> SSL works fine with my certs, but it crashes only after a certain
>>>>>>>> amount of time/requests.
>>>>>>>>
>>>>>>>> Here is the stack trace from traffic.out:
>>>>>>>>
>>>>>>>> NOTE: Traffic Server received Sig 11: Segmentation fault
>>>>>>>> /usr/bin/traffic_server - STACK TRACE:
>>>>>>>> [0x4001e500]
>>>>>>>> /usr/bin/traffic_server(_Z12ink_aio_readP11AIOCallbacki+0x2a)[0x830056a]
>>>>>>>> /usr/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x282)[0x82c4402]
>>>>>>>> /usr/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x5be)[0x82df68e]
>>>>>>>> /usr/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLbP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0xdc)[0x82c2b4c]
>>>>>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM18do_cache_open_readEv+0x63)[0x81ab6f3]
>>>>>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x4c)[0x81aba0c]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x115)[0x81bd105]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x6af)[0x81ce7bf]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x7eb)[0x81ce8fb]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x78)[0x81cc398]
>>>>>>>> /usr/bin/traffic_server(TSHttpTxnReenable+0x1f0)[0x810ef50]
>>>>>>>> /usr/lib/trafficserver/modules/stats_over_http.so(+0x102e)[0x4095f02e]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xd8)[0x81c9718]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x1e8)[0x81c5738]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x7e)[0x81ca93e]
>>>>>>>> /usr/bin/traffic_server(_ZN18UnixNetVConnection19readSignalAndUpdateEi+0x45)[0x83166a5]
>>>>>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x10b0)[0x83057f0]
>>>>>>>> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
>>>>>>>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
>>>>>>>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
>>>>>>>> /usr/bin/traffic_server[0x8338ebb]
>>>>>>>> /lib/i386-linux-gnu/libpthread.so.0(+0x5954)[0x4046b954]
>>>>>>>> /lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0x40688cbe]
>>>>>>>> [E. Mgmt] log ==> [TrafficManager] using root directory '/usr'
>>>>>>>> [TrafficServer] using root directory '/usr'
>>>>>>>>
>>>>>>>> Here is my record.config for SSL parameters:
>>>>>>>>
>>>>>>>> CONFIG proxy.config.http.server_ports STRING 8080 4443:ssl
>>>>>>>>
>>>>>>>> CONFIG proxy.config.ssl.enabled INT 1
>>>>>>>> CONFIG proxy.config.ssl.server.cert.path STRING /etc/trafficserver/ssl/
>>>>>>>> CONFIG proxy.config.ssl.server.private_key.path STRING
>>>>>>>> /etc/trafficserver/ssl/
>>>>>>>>
>>>>>>>> And for ssl_multicert.config:
>>>>>>>>
>>>>>>>> ssl_cert_name=new2014/100.pem ssl_key_name=new2014/100.key
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le 30/09/2014 15:54, Susan Hinrichs a écrit :
>>>>>>>>> Matt,
>>>>>>>>>
>>>>>>>>> Is there a basic stack trace in traffic.out? What is your SSL
>>>>>>>>> configuration? Do you have certs set up in ssl_multicert.config? Or
>>>>>>>>> are you doing a blind tunnel on the SSL traffic?
>>>>>>>>>
>>>>>>>>> Susan
>>>>>>>>>
>>>>>>>>> On 9/30/2014 2:14 AM, Matthieu Bienvenüe wrote:
>>>>>>>>>> Hello !
>>>>>>>>>>
>>>>>>>>>> I'm configuring ATS as a reverse proxy and I need SSL support.
>>>>>>>>>>
>>>>>>>>>> ATS runs on OpenVZ on Debian. It's the version 5.0.1 installed from
>>>>>>>>>> backport packages.
>>>>>>>>>>
>>>>>>>>>> ATS works fine, SSL too. But after a while SSL makes ATS crash.
>>>>>>>>>>
>>>>>>>>>> In manager.log I found that there is a segmentation fault:
>>>>>>>>>>
>>>>>>>>>> [Sep 29 16:08:33.020] Manager {0xb6fb76d0} ERROR:
>>>>>>>>>> [LocalManager::pollMgmtProcessServer] Server Process terminated due
>>>>>>>>>> to Sig 11: Segmentation fault
>>>>>>>>>> [Sep 29 16:08:33.021] Manager {0xb6fb76d0} ERROR:
>>>>>>>>>> [Alarms::signalAlarm] Server Process was reset
>>>>>>>>>> [Sep 29 16:08:34.041] Manager {0xb6fb76d0} NOTE:
>>>>>>>>>> [LocalManager::startProxy] Launching ts process
>>>>>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE:
>>>>>>>>>> [LocalManager::pollMgmtProcessServer] New process connecting fd '16'
>>>>>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE:
>>>>>>>>>> [Alarms::signalAlarm] Server Process born
>>>>>>>>>>
>>>>>>>>>> Here is a dump of the syslog when crashing:
>>>>>>>>>>
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} FATAL:
>>>>>>>>>> [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR:
>>>>>>>>>> [LocalManager::sendMgmtMsgToProcesses] Error writing message
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR: (last
>>>>>>>>>> system error 32: Broken pipe)
>>>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: cop received child status
>>>>>>>>>> signal [5471 256]
>>>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: traffic_manager not running,
>>>>>>>>>> making sure traffic_server is dead
>>>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: spawning traffic_manager
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: --- Manager
>>>>>>>>>> Starting ---
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Manager Version:
>>>>>>>>>> Apache Traffic Server - traffic_manager - 5.0.1 - (build # 7259 on
>>>>>>>>>> Aug 25 2014 at 09:26:11)
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Unable to set
>>>>>>>>>> RLIMIT_NOFILE(7):cur(1475961),max(1475961)
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE:
>>>>>>>>>> RLIMIT_NOFILE(7):cur(30000),max(30000)
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: ERROR ==> [runAsUser]
>>>>>>>>>> Error: Failed to restore capabilities after switch to user
>>>>>>>>>> trafficserver.
>>>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: --- traffic_server
>>>>>>>>>> Starting ---
>>>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: traffic_server
>>>>>>>>>> Version: Apache Traffic Server - traffic_server - 5.0.1 - (build #
>>>>>>>>>> 7259 on Aug 25 2014 at 09:27:18)
>>>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: Unable to set
>>>>>>>>>> RLIMIT_NOFILE(7):cur(-611778560),max(-611778560)
>>>>>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR:
>>>>>>>>>> [LocalManager::pollMgmtProcessServer] Server Process terminated due
>>>>>>>>>> to Sig 11: Segmentation fault
>>>>>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR:
>>>>>>>>>> [Alarms::signalAlarm] Server Process was reset
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Any idea where to look for to solve this problem ?
>>>>>>>>>>
>>>>>>>>>> Thanks a lot !
>>>>>>>>>>
>>>>>>>>>> Matt
>>>>>>>>
>
