[
https://issues.apache.org/jira/browse/MESOS-36?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Hindman closed MESOS-36.
---------------------------------
> race condition on MesosExecutorDriver destruction for Python
> ------------------------------------------------------------
>
> Key: MESOS-36
> URL: https://issues.apache.org/jira/browse/MESOS-36
> Project: Mesos
> Issue Type: Bug
> Environment: OS X 10.6.8, Python 2.6.7
> Reporter: brian wickman
> Assignee: Benjamin Hindman
>
> There's a race condition between driver.stop/abort and the destruction of the
> Python driver object.
> For example:
> def main(args, options):
> thermos_executor = ThermosExecutor(options)
> mesos.MesosExecutorDriver(thermos_executor).run()
> will routinely cause the segfault as attached at the end of the thread.
> If this is changed to
> def main(args, options):
> thermos_executor = ThermosExecutor(options)
> drv = mesos.MesosExecutorDriver(thermos_executor)
> drv.run()
> The code works, which indicates an issue (as you can see in the stack trace
> below) in the implicit reference counting on the MesosExecutorDriver object.
> The launchTask in my example does some work, sends a framework message when
> the task is finished, then issues a driver.stop().
> Process: Python [64054]
> Path:
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python
> Identifier: Python
> Version: ??? (???)
> Code Type: X86-64 (Native)
> Parent Process: java [64016]
> Date/Time: 2011-10-07 14:30:01.103 -0700
> OS Version: Mac OS X 10.6.8 (10K549)
> Report Version: 6
> Interval Since Last Report: 944086 sec
> Crashes Since Last Report: 18
> Per-App Crashes Since Last Report: 12
> Anonymous UUID: 3A043B60-6C64-4A96-BA3A-C04C21BA960E
> Exception Type: EXC_CRASH (SIGABRT)
> Exception Codes: 0x0000000000000000, 0x0000000000000000
> Crashed Thread: 1
> Application Specific Information:
> abort() called
> Thread 0: Dispatch queue: com.apple.main-thread
> 0 libSystem.B.dylib 0x00007fff869bba6a __semwait_signal + 10
> 1 libSystem.B.dylib 0x00007fff869bf881 _pthread_cond_wait +
> 1286
> 2 org.python.python 0x00000001000ef554
> PyThread_acquire_lock + 116
> 3 org.python.python 0x00000001000b499a PyEval_RestoreThread
> + 58
> 4 org.python.python 0x00000001000f3f20
> lock_PyThread_acquire_lock + 80
> 5 org.python.python 0x00000001000bc558 PyEval_EvalFrameEx +
> 28696
> 6 org.python.python 0x00000001000bd305 PyEval_EvalCodeEx +
> 2197
> 7 org.python.python 0x00000001000bb29d PyEval_EvalFrameEx +
> 23901
> 8 org.python.python 0x00000001000bb6fa PyEval_EvalFrameEx +
> 25018
> 9 org.python.python 0x00000001000bb6fa PyEval_EvalFrameEx +
> 25018
> 10 org.python.python 0x00000001000bd305 PyEval_EvalCodeEx +
> 2197
> 11 org.python.python 0x000000010003b50d function_call + 429
> 12 org.python.python 0x000000010000bb92 PyObject_Call + 98
> 13 org.python.python 0x00000001000b76d2 PyEval_EvalFrameEx +
> 8594
> 14 org.python.python 0x00000001000bd305 PyEval_EvalCodeEx +
> 2197
> 15 org.python.python 0x000000010003b405 function_call + 165
> 16 org.python.python 0x000000010000bb92 PyObject_Call + 98
> 17 org.python.python 0x00000001000b43c7
> PyEval_CallObjectWithKeywords + 87
> 18 org.python.python 0x00000001000e202a Py_Finalize + 186
> 19 org.python.python 0x00000001000e1b46 handle_system_exit +
> 246
> 20 org.python.python 0x00000001000e1d95 PyErr_PrintEx + 437
> 21 org.python.python 0x00000001000f16a4 RunModule + 404
> 22 org.python.python 0x00000001000f2109 Py_Main + 2505
> 23 org.python.python 0x0000000100000e22 0x100000000 + 3618
> 24 org.python.python 0x0000000100000d41 0x100000000 + 3393
> Thread 1 Crashed:
> 0 libSystem.B.dylib 0x00007fff869f39ce
> __semwait_signal_nocancel + 10
> 1 libSystem.B.dylib 0x00007fff869f38d0 nanosleep$NOCANCEL +
> 129
> 2 libSystem.B.dylib 0x00007fff86a503ce usleep$NOCANCEL + 57
> 3 libSystem.B.dylib 0x00007fff86a6fa00 abort + 93
> 4 _mesos.so 0x000000010190495c
> google::LogSink::~LogSink() + 0
> 5 _mesos.so 0x000000010190473b
> google::LogMessage::Fail() + 13
> 6 _mesos.so 0x0000000101909cae
> google::LogMessage::SendToLog() + 1212
> 7 _mesos.so 0x00000001019066c6
> google::LogMessage::Flush() + 418
> 8 _mesos.so 0x0000000101907e5e
> google::LogMessageFatal::~LogMessageFatal() + 22
> 9 _mesos.so 0x0000000101915e4c
> process::ProcessManager::wait(process::ProcessBase*, process::UPID const&) +
> 670
> 10 _mesos.so 0x0000000101920eb9
> process::wait(process::UPID const&, double) + 183
> 11 _mesos.so 0x000000010186c359
> process::wait(process::ProcessBase const*, double) + 45
> 12 _mesos.so 0x000000010185d2c6
> mesos::MesosExecutorDriver::~MesosExecutorDriver() + 98
> 13 _mesos.so 0x000000010171791a
> mesos::python::MesosExecutorDriverImpl_dealloc(mesos::python::MesosExecutorDriverImpl*)
> + 42 (mesos_executor_driver_impl.cpp:159)
> 14 org.python.python 0x0000000100069521 tupledealloc + 129
> 15 org.python.python 0x0000000100010a0a PyObject_CallMethod
> + 474
> 16 _mesos.so 0x000000010171b477
> mesos::python::ProxyExecutor::launchTask(mesos::ExecutorDriver*,
> mesos::TaskDescription const&) + 103 (proxy_executor.cpp:68)
> 17 _mesos.so 0x0000000101860641
> boost::_mfi::mf2<void, mesos::Executor, mesos::ExecutorDriver*,
> mesos::TaskDescription const&>::operator()(mesos::Executor*,
> mesos::ExecutorDriver*, mesos::TaskDescription const&) const + 113
> 18 _mesos.so 0x0000000101861010 void
> boost::_bi::list3<boost::_bi::value<mesos::Executor*>,
> boost::_bi::value<mesos::MesosExecutorDriver*>,
> boost::reference_wrapper<mesos::TaskDescription const>
> >::operator()<boost::_mfi::mf2<void, mesos::Executor, mesos::ExecutorDriver*,
> mesos::TaskDescription const&>, boost::_bi::list0>(boost::_bi::type<void>,
> boost::_mfi::mf2<void, mesos::Executor, mesos::ExecutorDriver*,
> mesos::TaskDescription const&>&, boost::_bi::list0&, int) + 118
> 19 _mesos.so 0x0000000101861052
> boost::_bi::bind_t<void, boost::_mfi::mf2<void, mesos::Executor,
> mesos::ExecutorDriver*, mesos::TaskDescription const&>,
> boost::_bi::list3<boost::_bi::value<mesos::Executor*>,
> boost::_bi::value<mesos::MesosExecutorDriver*>,
> boost::reference_wrapper<mesos::TaskDescription const> > >::operator()() + 54
> 20 ??? 0x0000000401861071 0 + 17205432433
> 21 libSystem.B.dylib 0x00007fff86a62dc9 setcontext + 25
> 22 libSystem.B.dylib 0x00007fff869b9fd6 _pthread_start + 331
> 23 libSystem.B.dylib 0x00007fff869b9e89 thread_start + 13
> Thread 2:
> 0 libSystem.B.dylib 0x00007fff869c4932 select$DARWIN_EXTSN
> + 10
> 1 _mesos.so 0x0000000101956518 select_poll + 168
> 2 _mesos.so 0x0000000101957327 ev_loop + 631
> 3 _mesos.so 0x00000001019195ff
> process::serve(void*) + 26
> 4 libSystem.B.dylib 0x00007fff869b9fd6 _pthread_start + 331
> 5 libSystem.B.dylib 0x00007fff869b9e89 thread_start + 13
> Thread 1 crashed with X86 Thread State (64-bit):
> rax: 0x000000000000003c rbx: 0x00000001023e3340 rcx: 0x00000001023e32f8
> rdx: 0x0000000000000001
> rdi: 0x000000000000030f rsi: 0x0000000000000000 rbp: 0x00000001023e3330
> rsp: 0x00000001023e32f8
> r8: 0x0000000000000000 r9: 0x0000000000989680 r10: 0x0000000000000001
> r11: 0x0000000000000246
> r12: 0x0000000000000000 r13: 0x0000000102264cf8 r14: 0x0000000000000002
> r15: 0x0000000100179110
> rip: 0x00007fff869f39ce rfl: 0x0000000000000247 cr2: 0x000000010158cc00
> Binary Images:
> 0x100000000 - 0x100000ff7 +org.python.python 2.6.7 (2.6.7)
> <DE73C8D7-8FE7-91E3-E747-F9410238C08F>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python
> 0x100003000 - 0x100153ff7 +org.python.python 2.6.7, (c)
> 2004-2008 Python Software Foundation. (2.6.7)
> <F55830D0-BF78-CD2D-D7C3-8B06D61B89A2>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/Python
> 0x1002fb000 - 0x1002fcfff +_json.so ??? (???)
> <E7E8F1AE-788B-4C9C-1F32-4563FDBB9B32>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_json.so
> 0x100440000 - 0x100443ff7 +zlib.so ??? (???)
> <7F0DCA3D-EE00-0742-4F3F-F858148BD0F8>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/zlib.so
> 0x100488000 - 0x10048bfff +math.so ??? (???)
> <B8550672-AC88-C793-81A8-2C9B8F568541>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/math.so
> 0x100491000 - 0x100492ff7 +time.so ??? (???)
> <5B4EB63E-F442-0C01-3EFA-85E1578C71BF>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/time.so
> 0x100497000 - 0x10049aff7 +select.so ??? (???)
> <5F113BD9-E2E6-32FA-F5EC-8D1CB714D39D>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/select.so
> 0x10049f000 - 0x1004a0ff7 +fcntl.so ??? (???)
> <0C585E11-AA6A-C307-43FD-52E1195AC9E6>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/fcntl.so
> 0x1004e3000 - 0x1004e8ff7 +_struct.so ??? (???)
> <DE726454-5661-6198-09A3-914217769C47>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_struct.so
> 0x1004ef000 - 0x1004f1fe7 +binascii.so ??? (???)
> <F377A449-B4B9-3A96-3EC7-7D1DB272A0C2>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/binascii.so
> 0x1004f5000 - 0x1004f6fff +cStringIO.so ??? (???)
> <F4762F1D-BC48-5D5F-8747-1FECBE43DFAB>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/cStringIO.so
> 0x10053b000 - 0x10053cff7 +_hashlib.so ??? (???)
> <25BE55BB-FA03-671D-1697-6181AB32DB9B>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_hashlib.so
> 0x100540000 - 0x100541fff +termios.so ??? (???)
> <D2D1DE14-3799-B958-3EEB-70C834D1EEE0>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/termios.so
> 0x100546000 - 0x100554fe7 +datetime.so ??? (???)
> <5975895D-4A18-EBFB-FFB9-B3C815D0116D>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/datetime.so
> 0x100560000 - 0x100564fff +_collections.so ??? (???)
> <7C600BC4-8E88-B5D7-4C3F-C47BE74460BE>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_collections.so
> 0x10056a000 - 0x10056efff +operator.so ??? (???)
> <8B28D217-1C9C-19AA-3952-C1FF8FFA248A>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/operator.so
> 0x100575000 - 0x100576fff +_random.so ??? (???)
> <574D30E0-2727-337F-F39C-1701CAA0F85A>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_random.so
> 0x1007e6000 - 0x1007e9ff7 +strop.so ??? (???)
> <15BD9A78-D5BE-6E04-5595-CD0132F5C32C>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/strop.so
> 0x1007ee000 - 0x1007efff7 +_functools.so ??? (???)
> <8947EAB7-5F3E-412A-2EF6-81DE9BFDF702>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_functools.so
> 0x1007f2000 - 0x1007f4ff7 +_locale.so ??? (???)
> <3674FAA6-719B-8284-AABF-6381A8B97577>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_locale.so
> 0x1007fb000 - 0x1007fbfff +_weakref.so ??? (???)
> <89494552-9A1C-C2B3-5ADE-2F8A8DD6F977>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_weakref.so
> 0x101430000 - 0x101464fff +pyexpat.so ??? (???)
> <E028F381-F4B5-8148-84CF-9099B43A3A8B>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/pyexpat.so
> 0x1014f6000 - 0x1014faff7 +_ssl.so ??? (???)
> <01812E71-E811-0EB0-70A6-7729BA0BDFEF>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_ssl.so
> 0x101700000 - 0x101b40fff +_mesos.so ??? (???)
> <5D8D6FDF-58AC-990A-AE20-EA1BFA36B739>
> /Users/wickman/.python-eggs/mesos-68-py2.6-macosx-10.4-x86_64.egg-tmp/_mesos.so
> 0x102312000 - 0x102319fff +_socket.so ??? (???)
> <DD52D015-A984-CB02-BE51-EC8F9C0B2559>
> /Users/wickman/Local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_socket.so
> 0x7fff5fc00000 - 0x7fff5fc3bdef dyld 132.1 (???)
> <DB8B8AB0-0C97-B51C-BE8B-B79895735A33> /usr/lib/dyld
> 0x7fff808ed000 - 0x7fff809a3ff7 libobjc.A.dylib 227.0.0
> (compatibility 1.0.0) <03140531-3B2D-1EBA-DA7F-E12CC8F63969>
> /usr/lib/libobjc.A.dylib
> 0x7fff8203b000 - 0x7fff82078ff7 libssl.0.9.8.dylib 0.9.8
> (compatibility 0.9.8) <F743389F-F25A-A77D-4FCA-D6B01AF2EE6D>
> /usr/lib/libssl.0.9.8.dylib
> 0x7fff83494000 - 0x7fff835b3fe7 libcrypto.0.9.8.dylib 0.9.8
> (compatibility 0.9.8) <14115D29-432B-CF02-6B24-A60CC533A09E>
> /usr/lib/libcrypto.0.9.8.dylib
> 0x7fff83a5d000 - 0x7fff83bd4fe7 com.apple.CoreFoundation 6.6.5
> (550.43) <31A1C118-AD96-0A11-8BDF-BD55B9940EDC>
> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
> 0x7fff84ea2000 - 0x7fff84ea3ff7 com.apple.TrustEvaluationAgent 1.1
> (1) <A91CE5B9-3C63-5F8C-5052-95CCAB866F72>
> /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent
> 0x7fff86980000 - 0x7fff86b41fef libSystem.B.dylib 125.2.11
> (compatibility 1.0.0) <9AB4F1D1-89DC-0E8A-DC8E-A4FE4D69DB69>
> /usr/lib/libSystem.B.dylib
> 0x7fff86c32000 - 0x7fff86df0fff libicucore.A.dylib 40.0.0
> (compatibility 1.0.0) <4274FC73-A257-3A56-4293-5968F3428854>
> /usr/lib/libicucore.A.dylib
> 0x7fff87c06000 - 0x7fff87c0aff7 libmathCommon.A.dylib 315.0.0
> (compatibility 1.0.0) <95718673-FEEE-B6ED-B127-BCDBDB60D4E5>
> /usr/lib/system/libmathCommon.A.dylib
> 0x7fff88a86000 - 0x7fff88ad2fff libauto.dylib ??? (???)
> <F7221B46-DC4F-3153-CE61-7F52C8C293CF> /usr/lib/libauto.dylib
> 0x7fff895bd000 - 0x7fff895ceff7 libz.1.dylib 1.2.3 (compatibility
> 1.0.0) <5BAFAE5C-2307-C27B-464D-582A10A6990B> /usr/lib/libz.1.dylib
> 0x7fff89786000 - 0x7fff89803fef libstdc++.6.dylib 7.9.0
> (compatibility 7.0.0) <35ECA411-2C08-FD7D-11B1-1B7A04921A5C>
> /usr/lib/libstdc++.6.dylib
> 0x7fffffe00000 - 0x7fffffe01fff libSystem.B.dylib ??? (???)
> <9AB4F1D1-89DC-0E8A-DC8E-A4FE4D69DB69> /usr/lib/libSystem.B.dylib
> Model: MacBookAir3,1, BootROM MBA31.0061.B01, 2 processors, Intel Core 2 Duo,
> 1.6 GHz, 4 GB, SMC 1.67f4
> Graphics: NVIDIA GeForce 320M, NVIDIA GeForce 320M, PCI, 256 MB
> Memory Module: global_name
> AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0xD1),
> Broadcom BCM43xx 1.0 (5.10.131.42.4)
> Bluetooth: Version 2.4.5f3, 2 service, 12 devices, 1 incoming serial ports
> Network Service: AirPort, AirPort, en0
> Serial ATA Device: APPLE SSD SM128C, 113 GB
> USB Device: FaceTime Camera (Built-in), 0x05ac (Apple Inc.), 0x850a,
> 0x24600000 / 2
> USB Device: BRCM2070 Hub, 0x0a5c (Broadcom Corp.), 0x4500, 0x04500000 / 3
> USB Device: Bluetooth USB Host Controller, 0x05ac (Apple Inc.), 0x821b,
> 0x04530000 / 6
> USB Device: Apple Internal Keyboard / Trackpad, 0x05ac (Apple Inc.), 0x0242,
> 0x04300000 / 2
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
