[
https://issues.apache.org/jira/browse/MESOS-10186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17201743#comment-17201743
]
Benjamin Mahler commented on MESOS-10186:
-----------------------------------------
There seems to be something unique to your environment that's not causing this
to be reported more broadly (this is the only report of this crash as far as
I'm aware). Can you give us more information to try to understand why this is
happening only for you? When did you first encounter this? Which versions of
libraries were working before you upgraded and started seeing this crash?
For what it's worth, here's an example of a working master from one of our
clusters:
{noformat}
$ ldd
/opt/mesosphere/packages/mesos--694f19b8312fb86ff1d095c71497cd7f538e4ba1/bin/mesos-master
linux-vdso.so.1 => (0x00007ffed2d7f000)
libmesos-1.8.2.so => /opt/mesosphere/lib/libmesos-1.8.2.so
(0x00007f3a263e5000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f3a261db000)
libevent_core-2.0.so.5 => /opt/mesosphere/lib/libevent_core-2.0.so.5
(0x00007f3a25fa8000)
librt.so.1 => /lib64/librt.so.1 (0x00007f3a25da0000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f3a25a97000)
libm.so.6 => /lib64/libm.so.6 (0x00007f3a25795000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f3a2557f000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3a25362000)
libc.so.6 => /lib64/libc.so.6 (0x00007f3a24f94000)
/lib64/ld-linux-x86-64.so.2 (0x000055f766980000)
libevent-2.0.so.5 =>
/opt/mesosphere/packages/libevent--a158693c2ee9f62e39d67980959f0952494b4313/lib/libevent-2.0.so.5
(0x00007f3a24d3d000)
libseccomp.so.2 =>
/opt/mesosphere/packages/libseccomp--e22c027075351745483adc9c10058ec376c65d3f/lib/libseccomp.so.2
(0x00007f3a24aec000)
libz.so.1 => /lib64/libz.so.1 (0x00007f3a248d6000)
libsvn_delta-1.so.1 => /opt/mesosphere/lib/libsvn_delta-1.so.1
(0x00007f3a246c2000)
libsvn_subr-1.so.1 => /opt/mesosphere/lib/libsvn_subr-1.so.1
(0x00007f3a2439e000)
libevent_openssl-2.0.so.5 =>
/opt/mesosphere/packages/libevent--a158693c2ee9f62e39d67980959f0952494b4313/lib/libevent_openssl-2.0.so.5
(0x00007f3a24197000)
libssl.so.1.0.0 => /opt/mesosphere/lib/libssl.so.1.0.0
(0x00007f3a23f26000)
libcrypto.so.1.0.0 => /opt/mesosphere/lib/libcrypto.so.1.0.0
(0x00007f3a23ab4000)
libsasl2.so.2 => /opt/mesosphere/lib/libsasl2.so.2 (0x00007f3a23899000)
libevent_pthreads-2.0.so.5 =>
/opt/mesosphere/packages/libevent--a158693c2ee9f62e39d67980959f0952494b4313/lib/libevent_pthreads-2.0.so.5
(0x00007f3a23695000)
libcurl.so.4 =>
/opt/mesosphere/packages/curl--e1ad33f11031799e5da6452429494e02e3198a45/lib/libcurl.so.4
(0x00007f3a23427000)
libapr-1.so.0 => /opt/mesosphere/lib/libapr-1.so.0 (0x00007f3a231f5000)
libaprutil-1.so.0 => /opt/mesosphere/lib/libaprutil-1.so.0
(0x00007f3a22fcd000)
libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f3a22da3000)
libsqlite3.so.0 => /lib64/libsqlite3.so.0 (0x00007f3a22aed000)
libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f3a228e8000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f3a226b0000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007f3a224ad000)
$ sudo xargs -0 -L1 -a /proc/17934/environ | grep LIBPROCESS
LIBPROCESS_SSL_CA_FILE=/run/dcos/pki/CA/ca-bundle.crt
LIBPROCESS_NUM_WORKER_THREADS=16
LIBPROCESS_SSL_VERIFY_CERT=false
LIBPROCESS_SSL_KEY_FILE=/run/dcos/pki/tls/private/mesos-master.key
LIBPROCESS_SSL_REQUIRE_CERT=false
LIBPROCESS_SSL_CIPHERS=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:AES128-SHA:AES256-SHA
LIBPROCESS_SSL_ENABLED=true
LIBPROCESS_SSL_CERT_FILE=/run/dcos/pki/tls/certs/mesos-master.crt
LIBPROCESS_SSL_SUPPORT_DOWNGRADE=false
{noformat}
> Segmentation fault while running mesos in SSL mode
> --------------------------------------------------
>
> Key: MESOS-10186
> URL: https://issues.apache.org/jira/browse/MESOS-10186
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 1.7.0, 1.9.0
> Reporter: Jaya Prasad Reddy G
> Priority: Blocker
>
> Hello,
> I've been runnning into segmentation faults while running mesos in SSL mode.
> After backtracing the coredump, this is what I found:
> {code:java}
> #0 0x00007f93060b7592 in free () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007f9303ebe8cd in CRYPTO_free () from
> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
> #2 0x00007f9303f7bfde in EVP_CIPHER_CTX_cleanup () from
> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
> #3 0x00007f93042e4275 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
> #4 0x00007f93042e59ca in SSL_set_accept_state () from
> /lib/x86_64-linux-gnu/libssl.so.1.0.0
> #5 0x00007f93059a5c72 in ?? () from
> /usr/lib/x86_64-linux-gnu/libevent_openssl-2.0.so.5
> #6 0x00007f930a801a74 in
> process::network::internal::LibeventSSLSocketImpl::accept_SSL_callback
> (request=request@entry=0x7f92f8000a80) at
> src/posix/libevent/libevent_ssl_socket.cpp:1172
> #7 0x00007f930a8021ea in
> process::network::internal::LibeventSSLSocketImpl::accept_callback
> (this=this@entry=0x55569161b430, request=request@entry=0x7f92f8000a80) at
> src/posix/libevent/libevent_ssl_socket.cpp:1124
> #8 0x00007f930a8027bc in
> process::network::internal::LibeventSSLSocketImpl::<lambda(evconnlistener*,
> int, sockaddr*, int, void*)>::operator()(evconnlistener *, int, sockaddr *,
> void *, int) (listener=0x555691667980, socket=24,
> addr=<optimized out>, arg=0x55569161be30, addr_length=<optimized out>,
> __closure=<optimized out>) at src/posix/libevent/libevent_ssl_socket.cpp:988
> #9 0x00007f9305e0829c in ?? () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #10 0x00007f9305dfa639 in event_base_loop () from
> /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> #11 0x00007f930a81c41d in process::EventLoop::run () at
> src/posix/libevent/libevent.cpp:98
> #12 0x00007f93066cbd00 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #13 0x00007f930699c6ba in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #14 0x00007f930613a60d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> {code}
> The environment variables exported before running mesos are:
> {code:java}
> MESOS_SSL_CIPHERS=ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA
>
> LIBPROCESS_SSL_ENABLED=true
> LIBPROCESS_SSL_SUPPORT_DOWNGRADE=false
> LIBPROCESS_SSL_CA_DIR=/opt/mesos/secrets/certs/
> LIBPROCESS_SSL_KEY_FILE=/opt/mesos/secrets/certs/mesos.private_key
> LIBPROCESS_SSL_CERT_FILE=/opt/mesos/secrets/certs/mesos.certificate
> LIBPROCESS_SSL_VERIFY_CERT=false
> LIBPROCESS_SSL_REQUIRE_CERT=false
> LIBPROCESS_SSL_VERIFY_IPADD=false
> LIBPROCESS_SSL_CA_FILE=/etc/ssl_ca/ca_list.pem
> LIBPROCESS_SSL_CIPHERS=ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA
>
> LIBPROCESS_SSL_ENABLE_SSL_V3=false
> LIBPROCESS_SSL_ENABLE_TLS_V1_0=false
> LIBPROCESS_SSL_ENABLE_TLS_V1_1=false
> LIBPROCESS_SSL_ENABLE_TLS_V1_2=true
> ZOOKEEPER_SSL_ENABLED=0
> ZOOKEEPER_SSL_VERIFY_CERT=false
> ZOOKEEPER_SSL_REQUIRE_CERT=false
> ZOOKEEPER_SSL_CA_FILE=/etc/ssl_ca/ca_list.pem
> ZOOKEEPER_SSL_KEY_FILE=/opt/mesos/secrets/certs/mesos.private_key
> ZOOKEEPER_SSL_CERT_FILE=/opt/mesos/secrets/certs/mesos.certificate
> {code}
> And the command used to run mesos:
> {code:java}
> mesos_master --ip=172.24.51.99 --advertise_ip=172.24.51.99
> --hostname_lookup=false --acls=file:///opt/mesos/etc/acl.json
> --modules=file:///opt/mesos/etc/modules.json --port=5050 --quorum=3
> --work_dir=/ghostcache2/mesos/registry
> --zk=zk://mycluster1.random.cluster.com:2181,mycluster2.random.cluster.com:2181,mycluster3.random.cluster.com:2181,mycluster4.random.cluster.com:2181,mycluster5.random.cluster.com:2181/mesos.mycluster
> --cluster=mycluster --hostname=mycluster2.random.cluster.com
> --offer_timeout=30secs --webui_dir=/opt/mesos/share/mesos/webui
> --whitelist=file:///opt/mesos/etc/whitelist --authenticate_agents=true
> --authenticators=crammd5 --authorizers=local
> --credentials=file:///opt/mesos/secrets/all_credentials.json
> --authenticate_frameworks=true{code}
> lddtree of the mesos_master
> {code:java}
> ~# lddtree /a/sbin/mesos_master
> mesos_master => /a/sbin/mesos_master (interpreter =>
> /lib64/ld-linux-x86-64.so.2)
> libmesos-1.9.0.so => /usr/local/myproject/opt/mesos/lib/libmesos-1.9.0.so
> libevent-2.0.so.5 => /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
> libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1
> libevent_openssl-2.0.so.5 =>
> /usr/lib/x86_64-linux-gnu/libevent_openssl-2.0.so.5
> libevent_core-2.0.so.5 =>
> /usr/lib/x86_64-linux-gnu/libevent_core-2.0.so.5
> libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0
> libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
> libevent_pthreads-2.0.so.5 =>
> /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.0.so.5
> libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1
> libcrypto.so.111.0.0 =>
> /usr/local/myproject/lib/openssl/x86_64-linux-gnu/libcrypto.so.111.0.0
> libssl.so.111.0.0 =>
> /usr/local/myproject/lib/openssl/x86_64-linux-gnu/libssl.so.111.0.0
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
> ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6{code}
>
> I tried updating mesos(from 1.7.0 to 1.9.0), libevent(2.0.21 to 2.0.22, 2.1.8
> and 2.1.12) to latest versions but still no luck. Not sure what I'm missing.
> Any help is appreciated.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
