Interesting thing to note: I switched to uwsgi 2.0.17.1 and encountered a slightly longer segmentation fault stack trace:
— !!! uWSGI process 32449 got Segmentation Fault !!! *** backtrace of 32449 *** /SCRUBBED/BINARY/PATH/bin/uwsgi(uwsgi_backtrace+0x2e) [0x46b54e] /SCRUBBED/BINARY/PATH/bin/uwsgi(uwsgi_segfault+0x21) [0x46b911] /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0) [0x7f5cc7db0cb0] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Malloc+0x248) [0x7f5cc8551928] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(_PyObject_GC_Malloc+0x19) [0x7f5cc84dce99] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(_PyObject_GC_New+0xd) [0x7f5cc84dcfed] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyDict_New+0x74) [0x7f5cc84e4444] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(_PyObject_GenericSetAttrWithDict+0x243) [0x7f5cc8454683] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_SetAttr+0x8f) [0x7f5cc84c30df] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1cda) [0x7f5cc85062ca] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7f5cc850a63d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1c5010) [0x7f5cc8541010] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f5cc84b66f3] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1b575d) [0x7f5cc853175d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x13d273) [0x7f5cc84b9273] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f5cc84b66f3] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2316) [0x7f5cc8506906] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1c4a93) [0x7f5cc8540a93] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x9f196) [0x7f5cc841b196] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x88732) [0x7f5cc8404732] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x13d2bf) [0x7f5cc84b92bf] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f5cc84b66f3] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2316) [0x7f5cc8506906] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) [0x7f5cc8509149] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) [0x7f5cc8509149] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) [0x7f5cc8509149] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) [0x7f5cc8509149] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7f5cc850a63d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1c5010) [0x7f5cc8541010] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f5cc84b66f3] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xb96cd) [0x7f5cc84356cd] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f5cc84b66f3] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7f5cc8530bc7] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x66882) [0x7f5cc83e2882] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f5cc9ab0184] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5cc7e7803d] *** end of backtrace *** — However, after moving from kernel version 4.4.0-134-generic over to 3.13.0-164-generic, the segfaults stopped. I’m wondering if there’s a memory management bug somewhere in the mix. Anyone have any ideas on how I might be able to track this back and provide a definitive bug report? — James Goin | Software Engineer, Liberator Squad > On Jan 10, 2019, at 9:57 AM, Thomas-James Goin <james.g...@oracle.com> wrote: > > Yeah, with the environment I’m operating in I don’t exactly have a choice on > that (yet). We’re evaluating putting the application in-question into a > Docker container and running it that way, but we haven’t quite gotten there > yet. > > I’m going to try upgrading to uwsgi 2.0.17.1 on a test host and see if that > resolves the segfaults. I’ll write back once I know more. Thanks for the > input! > — > > <PastedGraphic-2.tiff> > > James Goin | Software Engineer, Liberator Squad > >> On Jan 10, 2019, at 9:55 AM, Avraham Serour <tovm...@gmail.com >> <mailto:tovm...@gmail.com>> wrote: >> >> I meant in general, the version of the stuff you posted is different, no >> surprise you have different behavior >> maybe it is the python minor version, my gut blames libc, honestly I would >> pull my son ear if I caught him using ubuntu that old >> >> On Thu, Jan 10, 2019 at 3:20 PM Thomas-James Goin <james.g...@oracle.com >> <mailto:james.g...@oracle.com>> wrote: >> Which old version do you mean? The libc6 library, the uwsgi version, or the >> python2.7 library version(s)? >> — >> >> <PastedGraphic-2.tiff> >> >> James Goin | Software Engineer, Liberator Squad >> >>> On Jan 10, 2019, at 4:15 AM, Avraham Serour <tovm...@gmail.com >>> <mailto:tovm...@gmail.com>> wrote: >>> >>> it seems you already answered your own question, the old version have a bug >>> that causes segfault, consider upgrading >>> >>> On Wed, Jan 9, 2019 at 11:08 PM Thomas-James Goin <james.g...@oracle.com >>> <mailto:james.g...@oracle.com>> wrote: >>> I was referred to this mailing list by a coworker to ask for some help on a >>> segfaulting issue I’m seeing with uwsgi. Currently I’m overseeing a fleet >>> of about a dozen machines, some old and some new. The newer ones are >>> running Ubuntu 14.04.5 LTS whereas the old ones are running 14.04.4 LTS. >>> When we stood them up recently with our bootstrap process, we immediately >>> noticed that uwsgi was throwing segfaults into the log. I’ve posted the >>> segfault below and a comparison of the libc and python2.7 packages >>> installed on each for comparison. For the record: uwsgi is running in a >>> Python virtualenv with flask and pip manages the package itself from within >>> the virtualenv. >>> >>> SEGFAULT: >>> — >>> !!! uWSGI process 5015 got Segmentation Fault !!! >>> *** backtrace of 5016 *** >>> /SCRUBBED/BINARY/PATH/bin/uwsgi(uwsgi_backtrace+0x2e) [0x46b54e] >>> /SCRUBBED/BINARY/PATH/bin/uwsgi(uwsgi_segfault+0x21) [0x46b911] >>> /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0) [0x7f5e3acbecb0] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Malloc+0x248) >>> [0x7f5e3b45f928] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(_PyObject_GC_Malloc+0x19) >>> [0x7f5e3b3eae99] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyType_GenericAlloc+0x3b) >>> [0x7f5e3b3f43bb] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x13d273) [0x7f5e3b3c7273] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) >>> [0x7f5e3b3c46f3] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2316) >>> [0x7f5e3b414906] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) >>> [0x7f5e3b417149] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) >>> [0x7f5e3b417149] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1c4a93) [0x7f5e3b44ea93] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x9f196) [0x7f5e3b329196] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x88732) [0x7f5e3b312732] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x13d2bf) [0x7f5e3b3c72bf] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) >>> [0x7f5e3b3c46f3] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2316) >>> [0x7f5e3b414906] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) >>> [0x7f5e3b417149] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) >>> [0x7f5e3b417149] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) >>> [0x7f5e3b417149] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) >>> [0x7f5e3b417149] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) >>> [0x7f5e3b41863d] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1c5010) [0x7f5e3b44f010] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) >>> [0x7f5e3b3c46f3] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xb96cd) [0x7f5e3b3436cd] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) >>> [0x7f5e3b3c46f3] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) >>> [0x7f5e3b43ebc7] >>> /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x66882) [0x7f5e3b2f0882] >>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f5e3c9be184] >>> /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5e3ad8603d] >>> *** end of backtrace *** >>> --- >>> >>> OLD: >>> — >>> # cat /etc/lsb-release >>> DISTRIB_ID=Ubuntu >>> DISTRIB_RELEASE=14.04 >>> DISTRIB_CODENAME=trusty >>> DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS” >>> >>> # dpkg --list | egrep 'libc6|python2\.7' >>> ii libc6:amd64 2.19-0ubuntu6.11 >>> amd64 Embedded GNU C Library: Shared libraries >>> ii libc6-dbg:amd64 2.19-0ubuntu6.11 >>> amd64 Embedded GNU C Library: detached debugging >>> symbols >>> ii libc6-dev:amd64 2.19-0ubuntu6.11 >>> amd64 Embedded GNU C Library: Development Libraries >>> and Header Files >>> ii libpython2.7:amd64 2.7.6-8ubuntu0.3 >>> amd64 Shared Python runtime library (version 2.7) >>> ii libpython2.7-dbg:amd64 2.7.6-8ubuntu0.3 >>> amd64 Debug Build of the Python Interpreter >>> (version 2.7) >>> ii libpython2.7-dev:amd64 2.7.6-8ubuntu0.3 >>> amd64 Header files and a static library for Python >>> (v2.7) >>> ii libpython2.7-minimal:amd64 2.7.6-8ubuntu0.3 >>> amd64 Minimal subset of the Python language >>> (version 2.7) >>> ii libpython2.7-stdlib:amd64 2.7.6-8ubuntu0.3 >>> amd64 Interactive high-level object-oriented >>> language (standard library, version 2.7) >>> ii python2.7 2.7.6-8ubuntu0.3 >>> amd64 Interactive high-level object-oriented >>> language (version 2.7) >>> ii python2.7-dbg 2.7.6-8ubuntu0.3 >>> amd64 Debug Build of the Python Interpreter >>> (version 2.7) >>> ii python2.7-dev 2.7.6-8ubuntu0.3 >>> amd64 Header files and a static library for Python >>> (v2.7) >>> ii python2.7-minimal 2.7.6-8ubuntu0.3 >>> amd64 Minimal subset of the Python language >>> (version 2.7) >>> --- >>> >>> NEW: >>> — >>> # cat /etc/lsb-release >>> DISTRIB_ID=Ubuntu >>> DISTRIB_RELEASE=14.04 >>> DISTRIB_CODENAME=trusty >>> DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS” >>> >>> # dpkg --list | egrep 'libc6|python2\.7' >>> ii libc6:amd64 2.19-0ubuntu6.14 >>> amd64 Embedded GNU C Library: Shared libraries >>> ii libc6-dev:amd64 2.19-0ubuntu6.14 >>> amd64 Embedded GNU C Library: Development Libraries and >>> Header Files >>> ii libpython2.7:amd64 2.7.6-8ubuntu0.5 >>> amd64 Shared Python runtime library (version 2.7) >>> ii libpython2.7-dev:amd64 2.7.6-8ubuntu0.5 >>> amd64 Header files and a static library for Python (v2.7) >>> ii libpython2.7-minimal:amd64 2.7.6-8ubuntu0.5 >>> amd64 Minimal subset of the Python language (version 2.7) >>> ii libpython2.7-stdlib:amd64 2.7.6-8ubuntu0.5 >>> amd64 Interactive high-level object-oriented language >>> (standard library, version 2.7) >>> ii python2.7 2.7.6-8ubuntu0.5 >>> amd64 Interactive high-level object-oriented language >>> (version 2.7) >>> ii python2.7-dev 2.7.6-8ubuntu0.5 >>> amd64 Header files and a static library for Python (v2.7) >>> ii python2.7-minimal 2.7.6-8ubuntu0.5 >>> amd64 Minimal subset of the Python language (version 2.7) >>> — >>> >>> — >>> >>> <PastedGraphic-2.tiff> >>> >>> James Goin | Software Engineer, Liberator Squad >>> >>> _______________________________________________ >>> uWSGI mailing list >>> uWSGI@lists.unbit.it <mailto:uWSGI@lists.unbit.it> >>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.unbit.it_cgi-2Dbin_mailman_listinfo_uwsgi&d=DwMFaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=-TrLS3TpQre-21b9Hcu4pBUeXmQyQEOnP5OVIhuR78s&m=EYBxsLib4MGFWPL2rxUEGga3A4r7yucTCIhEqIqvueA&s=quUk9drWVYFZ4emeZGRAqEYzo1ZtLKOOxFj1fMz6aFc&e=> >>> _______________________________________________ >>> uWSGI mailing list >>> uWSGI@lists.unbit.it <mailto:uWSGI@lists.unbit.it> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.unbit.it_cgi-2Dbin_mailman_listinfo_uwsgi&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=-TrLS3TpQre-21b9Hcu4pBUeXmQyQEOnP5OVIhuR78s&m=EYBxsLib4MGFWPL2rxUEGga3A4r7yucTCIhEqIqvueA&s=quUk9drWVYFZ4emeZGRAqEYzo1ZtLKOOxFj1fMz6aFc&e= >>> >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.unbit.it_cgi-2Dbin_mailman_listinfo_uwsgi&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=-TrLS3TpQre-21b9Hcu4pBUeXmQyQEOnP5OVIhuR78s&m=EYBxsLib4MGFWPL2rxUEGga3A4r7yucTCIhEqIqvueA&s=quUk9drWVYFZ4emeZGRAqEYzo1ZtLKOOxFj1fMz6aFc&e=> >> _______________________________________________ >> uWSGI mailing list >> uWSGI@lists.unbit.it <mailto:uWSGI@lists.unbit.it> >> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.unbit.it_cgi-2Dbin_mailman_listinfo_uwsgi&d=DwMFaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=-TrLS3TpQre-21b9Hcu4pBUeXmQyQEOnP5OVIhuR78s&m=e4GWG2pE_ncxfFfnnz_Kf0hA-ct6xDHnSMqWmHsNf8M&s=BoPSSuVwfNbjp71nGwDWW5W-AY61XmY4oaaD900OS2Y&e=> >> _______________________________________________ >> uWSGI mailing list >> uWSGI@lists.unbit.it <mailto:uWSGI@lists.unbit.it> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.unbit.it_cgi-2Dbin_mailman_listinfo_uwsgi&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=-TrLS3TpQre-21b9Hcu4pBUeXmQyQEOnP5OVIhuR78s&m=e4GWG2pE_ncxfFfnnz_Kf0hA-ct6xDHnSMqWmHsNf8M&s=BoPSSuVwfNbjp71nGwDWW5W-AY61XmY4oaaD900OS2Y&e= > > _______________________________________________ > uWSGI mailing list > uWSGI@lists.unbit.it > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.unbit.it_cgi-2Dbin_mailman_listinfo_uwsgi&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=-TrLS3TpQre-21b9Hcu4pBUeXmQyQEOnP5OVIhuR78s&m=HhEYk5Qfg3_8iJDO3sN9M-D6gKOkSbm8frb8CEGHGyA&s=7w_wbQtqaIlDL9acRG-DQsOUfT-Xby_qVESmG1dcMY8&e=
_______________________________________________ uWSGI mailing list uWSGI@lists.unbit.it http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
