Re: [lldb-dev] Weird results running lldb under Valgrind
Glad we know why at least! Thanks for bringing this to our attention. Greg > On Sep 30, 2020, at 2:53 AM, Dmitry Antipov wrote: > > On 9/29/20 11:40 PM, Greg Clayton wrote: > >> How could LLDB even function then? We are using the standard std::mutex + >> std::condition workflow here. Not sure how LLDB could even function if it >> locking was nor working as expected. > > Well, obviously this is an issue (and probably the same one) with debugging > tools. > >> Doing a quick web search, this seems to be due to a mismatched libc++ and >> libstdc++: >> https://github.com/google/sanitizers/issues/1259 > > Nice. So if your libstdc++ is new enough to use pthread_cond_clockwait(), > both TSan and > valgrind produces weird results just because they can handle > pthread_cond_timedwait() only. > > Dmitry > ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] Weird results running lldb under Valgrind
On 9/29/20 11:40 PM, Greg Clayton wrote: How could LLDB even function then? We are using the standard std::mutex + std::condition workflow here. Not sure how LLDB could even function if it locking was nor working as expected. Well, obviously this is an issue (and probably the same one) with debugging tools. Doing a quick web search, this seems to be due to a mismatched libc++ and libstdc++: https://github.com/google/sanitizers/issues/1259 Nice. So if your libstdc++ is new enough to use pthread_cond_clockwait(), both TSan and valgrind produces weird results just because they can handle pthread_cond_timedwait() only. Dmitry ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] Weird results running lldb under Valgrind
How could LLDB even function then? We are using the standard std::mutex +
std::condition workflow here. Not sure how LLDB could even function if it
locking was nor working as expected.
Doing a quick web search, this seems to be due to a mismatched libc++ and
libstdc++:
https://github.com/google/sanitizers/issues/1259
Greg
> On Sep 29, 2020, at 9:46 AM, Dmitry Antipov wrote:
>
> On 9/25/20 5:53 PM, Dmitry Antipov wrote:
>
>> On 9/24/20 9:14 PM, Greg Clayton wrote:
>>> This must be a valgrind issue, there would be major problems if the OS
>>> isn't able to lock mutex objects correctly ("mutex is locked simultaneously
>>> by two threads"). It is getting confused by a recursive mutex? LLDB uses
>>> recursive mutexes.
>> LLDB's Predicate.h uses plain std::mutex, which is not recursive, and
>> std::lock_guard/std::unique_lock
>> on top of it.
>> This needs more digging in because the latest Valgrind snapshot reports the
>> same "impossible" condition.
>
> To whom it may be interesting, thread sanitizer reports nearly the same:
>
> WARNING: ThreadSanitizer: double lock of a mutex (pid=2049545)
>#0 pthread_mutex_lock (libtsan.so.0+0x528ac)
>#1 __gthread_mutex_lock
> /usr/include/c++/10/x86_64-redhat-linux/bits/gthr-default.h:749
> (liblldb.so.12git+0xd725c0)
>#2 std::mutex::lock() /usr/include/c++/10/bits/std_mutex.h:100
> (liblldb.so.12git+0xd725c0)
>#3 std::lock_guard::lock_guard(std::mutex&)
> /usr/include/c++/10/bits/std_mutex.h:159 (liblldb.so.12git+0xd725c0)
>#4 lldb_private::Predicate::SetValue(bool,
> lldb_private::PredicateBroadcastType)
> /home/antipov/llvm/source/lldb/include/lldb/Utility/Predicate.h:91
> (liblldb.so.12git+0xd725c0)
>#5 lldb_private::EventDataReceipt::DoOnRemoval(lldb_private::Event*)
> /home/antipov/llvm/source/lldb/include/lldb/Utility/Event.h:121
> (liblldb.so.12git+0xd725c0)
>#6 lldb_private::Event::DoOnRemoval()
> /home/antipov/llvm/source/lldb/source/Utility/Event.cpp:82
> (liblldb.so.12git+0xedb7da)
>#7
> lldb_private::Listener::FindNextEventInternal(std::unique_lock&,
> lldb_private::Broadcaster*, lldb_private::ConstString const*, unsigned int,
> unsigned int, std::shared_ptr&, bool)
> /home/antipov/llvm/source/lldb/source/Utility/Listener.cpp:309
> (liblldb.so.12git+0xee6099)
>#8
> lldb_private::Listener::GetEventInternal(lldb_private::Timeout 100l> > const&, lldb_private::Broadcaster*, lldb_private::ConstString
> const*, unsigned int, unsigned int, std::shared_ptr&)
> /home/antipov/llvm/source/lldb/source/Utility/Listener.cpp:357
> (liblldb.so.12git+0xee6b63)
>#9
> lldb_private::Listener::GetEventForBroadcaster(lldb_private::Broadcaster*,
> std::shared_ptr&, lldb_private::Timeout 100l> > const&)
> /home/antipov/llvm/source/lldb/source/Utility/Listener.cpp:395
> (liblldb.so.12git+0xee6dea)
>#10
> lldb_private::Process::GetEventsPrivate(std::shared_ptr&,
> lldb_private::Timeout > const&, bool)
> /home/antipov/llvm/source/lldb/source/Target/Process.cpp:1139
> (liblldb.so.12git+0xd7931d)
>#11 lldb_private::Process::RunPrivateStateThread(bool)
> /home/antipov/llvm/source/lldb/source/Target/Process.cpp:3872
> (liblldb.so.12git+0xda3648)
>#12 lldb_private::Process::PrivateStateThread(void*)
> /home/antipov/llvm/source/lldb/source/Target/Process.cpp:3857
> (liblldb.so.12git+0xda3f87)
>#13 lldb_private::HostNativeThreadBase::ThreadCreateTrampoline(void*)
> /home/antipov/llvm/source/lldb/source/Host/common/HostNativeThreadBase.cpp:68
> (liblldb.so.12git+0xc2c0ea)
>#14 (libtsan.so.0+0x2d33f)
>
> Again, lldb_private::Predicate uses plain std::mutex, which is not recursive.
>
> Dmitry
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] Weird results running lldb under Valgrind
On 9/25/20 5:53 PM, Dmitry Antipov wrote:
On 9/24/20 9:14 PM, Greg Clayton wrote:
This must be a valgrind issue, there would be major problems if the OS isn't able to lock mutex objects correctly ("mutex is locked simultaneously by two threads"). It is getting confused by a
recursive mutex? LLDB uses recursive mutexes.
LLDB's Predicate.h uses plain std::mutex, which is not recursive, and
std::lock_guard/std::unique_lock
on top of it.
This needs more digging in because the latest Valgrind snapshot reports the same
"impossible" condition.
To whom it may be interesting, thread sanitizer reports nearly the same:
WARNING: ThreadSanitizer: double lock of a mutex (pid=2049545)
#0 pthread_mutex_lock (libtsan.so.0+0x528ac)
#1 __gthread_mutex_lock
/usr/include/c++/10/x86_64-redhat-linux/bits/gthr-default.h:749
(liblldb.so.12git+0xd725c0)
#2 std::mutex::lock() /usr/include/c++/10/bits/std_mutex.h:100
(liblldb.so.12git+0xd725c0)
#3 std::lock_guard::lock_guard(std::mutex&)
/usr/include/c++/10/bits/std_mutex.h:159 (liblldb.so.12git+0xd725c0)
#4 lldb_private::Predicate::SetValue(bool,
lldb_private::PredicateBroadcastType)
/home/antipov/llvm/source/lldb/include/lldb/Utility/Predicate.h:91
(liblldb.so.12git+0xd725c0)
#5 lldb_private::EventDataReceipt::DoOnRemoval(lldb_private::Event*)
/home/antipov/llvm/source/lldb/include/lldb/Utility/Event.h:121
(liblldb.so.12git+0xd725c0)
#6 lldb_private::Event::DoOnRemoval()
/home/antipov/llvm/source/lldb/source/Utility/Event.cpp:82
(liblldb.so.12git+0xedb7da)
#7 lldb_private::Listener::FindNextEventInternal(std::unique_lock&, lldb_private::Broadcaster*, lldb_private::ConstString const*, unsigned int, unsigned int,
std::shared_ptr&, bool) /home/antipov/llvm/source/lldb/source/Utility/Listener.cpp:309 (liblldb.so.12git+0xee6099)
#8 lldb_private::Listener::GetEventInternal(lldb_private::Timeout > const&, lldb_private::Broadcaster*, lldb_private::ConstString const*, unsigned int, unsigned int,
std::shared_ptr&) /home/antipov/llvm/source/lldb/source/Utility/Listener.cpp:357 (liblldb.so.12git+0xee6b63)
#9 lldb_private::Listener::GetEventForBroadcaster(lldb_private::Broadcaster*, std::shared_ptr&, lldb_private::Timeout > const&)
/home/antipov/llvm/source/lldb/source/Utility/Listener.cpp:395 (liblldb.so.12git+0xee6dea)
#10 lldb_private::Process::GetEventsPrivate(std::shared_ptr&, lldb_private::Timeout > const&, bool)
/home/antipov/llvm/source/lldb/source/Target/Process.cpp:1139 (liblldb.so.12git+0xd7931d)
#11 lldb_private::Process::RunPrivateStateThread(bool)
/home/antipov/llvm/source/lldb/source/Target/Process.cpp:3872
(liblldb.so.12git+0xda3648)
#12 lldb_private::Process::PrivateStateThread(void*)
/home/antipov/llvm/source/lldb/source/Target/Process.cpp:3857
(liblldb.so.12git+0xda3f87)
#13 lldb_private::HostNativeThreadBase::ThreadCreateTrampoline(void*)
/home/antipov/llvm/source/lldb/source/Host/common/HostNativeThreadBase.cpp:68
(liblldb.so.12git+0xc2c0ea)
#14 (libtsan.so.0+0x2d33f)
Again, lldb_private::Predicate uses plain std::mutex, which is not recursive.
Dmitry
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] Weird results running lldb under Valgrind
On 9/24/20 9:14 PM, Greg Clayton wrote:
This must be a valgrind issue, there would be major problems if the OS isn't able to lock
mutex objects correctly ("mutex is locked simultaneously by two threads"). It
is getting confused by a recursive mutex? LLDB uses recursive mutexes.W
LLDB's Predicate.h uses plain std::mutex, which is not recursive, and
std::lock_guard/std::unique_lock
on top of it.
This needs more digging in because the latest Valgrind snapshot reports the same
"impossible" condition.
Dmitry
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] Weird results running lldb under Valgrind
This must be a valgrind issue, there would be major problems if the OS isn't
able to lock mutex objects correctly ("mutex is locked simultaneously by two
threads"). It is getting confused by a recursive mutex? LLDB uses recursive
mutexes.
> On Sep 24, 2020, at 1:55 AM, Dmitry Antipov via lldb-dev
> wrote:
>
> Does anyone has an explanation of this weird run of 'valgrind --tool=drd':
>
> ==2715== drd, a thread error detector
> ==2715== Copyright (C) 2006-2017, and GNU GPL'd, by Bart Van Assche.
> ==2715== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
> ==2715== Command: /home/antipov/.local/llvm-12.0.0/bin/lldb
> ==2715== Parent PID: 1702
>
> In LLDB, do 'process attach --pid [PID of running Firefox]', then:
>
> ==2715== Thread 5:
> ==2715== The impossible happened: mutex is locked simultaneously by two
> threads: mutex 0xe907d10, recursion count 1, owner 1.
> ==2715==at 0x4841015: pthread_mutex_lock_intercept
> (drd_pthread_intercepts.c:893)
> ==2715==by 0x4841015: pthread_mutex_lock (drd_pthread_intercepts.c:903)
> ==2715==by 0x504FBEE: __gthread_mutex_lock (gthr-default.h:749)
> ==2715==by 0x504FBEE: lock (std_mutex.h:100)
> ==2715==by 0x504FBEE: lock_guard (std_mutex.h:159)
> ==2715==by 0x504FBEE: SetValue (Predicate.h:91)
> ==2715==by 0x504FBEE:
> lldb_private::EventDataReceipt::DoOnRemoval(lldb_private::Event*)
> (Event.h:121)
> ==2715==by 0x5113644:
> lldb_private::Listener::FindNextEventInternal(std::unique_lock&,
> lldb_private::Broadcaster*, lldb_private::ConstString const*, unsigned int,
> unsigned int, std::shared_ptr&, bool) (Listener.cpp:309)
> ==2715==by 0x5113DD1:
> lldb_private::Listener::GetEventInternal(lldb_private::Timeout 100l> > const&, lldb_private::Broadcaster*, lldb_private::ConstString
> const*, unsigned int, unsigned int, std::shared_ptr&)
> (Listener.cpp:357)
> ==2715==by 0x5113F4A:
> lldb_private::Listener::GetEventForBroadcaster(lldb_private::Broadcaster*,
> std::shared_ptr&, lldb_private::Timeout 100l> > const&) (Listener.cpp:395)
> ==2715==by 0x506ADD4: lldb_private::Process::RunPrivateStateThread(bool)
> (Process.cpp:3872)
> ==2715==by 0x506B3F5: lldb_private::Process::PrivateStateThread(void*)
> (Process.cpp:3857)
> ==2715==by 0x483DB9A: vgDrd_thread_wrapper (drd_pthread_intercepts.c:449)
> ==2715==by 0x488B3F8: start_thread (in /usr/lib64/libpthread-2.32.so)
> ==2715==by 0xDFCEA92: clone (in /usr/lib64/libc-2.32.so)
> ==2715== mutex 0xe907d10 was first observed at:
> ==2715==at 0x4840F55: pthread_mutex_lock_intercept
> (drd_pthread_intercepts.c:890)
> ==2715==by 0x4840F55: pthread_mutex_lock (drd_pthread_intercepts.c:903)
> ==2715==by 0x5058502: __gthread_mutex_lock (gthr-default.h:749)
> ==2715==by 0x5058502: lock (std_mutex.h:100)
> ==2715==by 0x5058502: lock (unique_lock.h:138)
> ==2715==by 0x5058502: unique_lock (unique_lock.h:68)
> ==2715==by 0x5058502:
> WaitFor::WaitForValueEqualTo::
> > (Predicate.h:123)
> ==2715==by 0x5058502: WaitForValueEqualTo (Predicate.h:157)
> ==2715==by 0x5058502: WaitForEventReceived (Event.h:114)
> ==2715==by 0x5058502:
> lldb_private::Process::ControlPrivateStateThread(unsigned int)
> (Process.cpp:3698)
> ==2715==by 0x505BC61:
> lldb_private::Process::StartPrivateStateThread(bool) (Process.cpp:3647)
> ==2715==by 0x5065B96:
> lldb_private::Process::Attach(lldb_private::ProcessAttachInfo&)
> (Process.cpp:2961)
> ==2715==by 0x544DBB8:
> PlatformPOSIX::Attach(lldb_private::ProcessAttachInfo&,
> lldb_private::Debugger&, lldb_private::Target*, lldb_private::Status&)
> (PlatformPOSIX.cpp:401)
> ==2715==by 0x509F531:
> lldb_private::Target::Attach(lldb_private::ProcessAttachInfo&,
> lldb_private::Stream*) (Target.cpp:3008)
> ==2715==by 0x54C3F17:
> CommandObjectProcessAttach::DoExecute(lldb_private::Args&,
> lldb_private::CommandReturnObject&) (CommandObjectProcess.cpp:386)
> ==2715==by 0x4FC0ACD: lldb_private::CommandObjectParsed::Execute(char
> const*, lldb_private::CommandReturnObject&) (CommandObject.cpp:993)
> ==2715==by 0x4FBCBD7:
> lldb_private::CommandInterpreter::HandleCommand(char const*,
> lldb_private::LazyBool, lldb_private::CommandReturnObject&,
> lldb_private::ExecutionContext*, bool, bool) (CommandInterpreter.cpp:1803)
> ==2715==by 0x4FBDB96:
> lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&,
> std::__cxx11::basic_string,
> std::allocator >&) (CommandInterpreter.cpp:2838)
> ==2715==by 0x4EF21C0: lldb_private::IOHandlerEditline::Run()
> (IOHandler.cpp:579)
> ==2715==by 0x4ED02B0: lldb_private::Debugger::RunIOHandlers()
> (Debugger.cpp:861)
>
> Hopefully this is an issue with valgrind and not lldb. But still curious
> whether someone else can reproduce something similar.
>
> Dmitry
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
[lldb-dev] Weird results running lldb under Valgrind
Does anyone has an explanation of this weird run of 'valgrind --tool=drd': ==2715== drd, a thread error detector ==2715== Copyright (C) 2006-2017, and GNU GPL'd, by Bart Van Assche. ==2715== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info ==2715== Command: /home/antipov/.local/llvm-12.0.0/bin/lldb ==2715== Parent PID: 1702 In LLDB, do 'process attach --pid [PID of running Firefox]', then: ==2715== Thread 5: ==2715== The impossible happened: mutex is locked simultaneously by two threads: mutex 0xe907d10, recursion count 1, owner 1. ==2715==at 0x4841015: pthread_mutex_lock_intercept (drd_pthread_intercepts.c:893) ==2715==by 0x4841015: pthread_mutex_lock (drd_pthread_intercepts.c:903) ==2715==by 0x504FBEE: __gthread_mutex_lock (gthr-default.h:749) ==2715==by 0x504FBEE: lock (std_mutex.h:100) ==2715==by 0x504FBEE: lock_guard (std_mutex.h:159) ==2715==by 0x504FBEE: SetValue (Predicate.h:91) ==2715==by 0x504FBEE: lldb_private::EventDataReceipt::DoOnRemoval(lldb_private::Event*) (Event.h:121) ==2715==by 0x5113644: lldb_private::Listener::FindNextEventInternal(std::unique_lock&, lldb_private::Broadcaster*, lldb_private::ConstString const*, unsigned int, unsigned int, std::shared_ptr&, bool) (Listener.cpp:309) ==2715==by 0x5113DD1: lldb_private::Listener::GetEventInternal(lldb_private::Timeout > const&, lldb_private::Broadcaster*, lldb_private::ConstString const*, unsigned int, unsigned int, std::shared_ptr&) (Listener.cpp:357) ==2715==by 0x5113F4A: lldb_private::Listener::GetEventForBroadcaster(lldb_private::Broadcaster*, std::shared_ptr&, lldb_private::Timeout > const&) (Listener.cpp:395) ==2715==by 0x506ADD4: lldb_private::Process::RunPrivateStateThread(bool) (Process.cpp:3872) ==2715==by 0x506B3F5: lldb_private::Process::PrivateStateThread(void*) (Process.cpp:3857) ==2715==by 0x483DB9A: vgDrd_thread_wrapper (drd_pthread_intercepts.c:449) ==2715==by 0x488B3F8: start_thread (in /usr/lib64/libpthread-2.32.so) ==2715==by 0xDFCEA92: clone (in /usr/lib64/libc-2.32.so) ==2715== mutex 0xe907d10 was first observed at: ==2715==at 0x4840F55: pthread_mutex_lock_intercept (drd_pthread_intercepts.c:890) ==2715==by 0x4840F55: pthread_mutex_lock (drd_pthread_intercepts.c:903) ==2715==by 0x5058502: __gthread_mutex_lock (gthr-default.h:749) ==2715==by 0x5058502: lock (std_mutex.h:100) ==2715==by 0x5058502: lock (unique_lock.h:138) ==2715==by 0x5058502: unique_lock (unique_lock.h:68) ==2715==by 0x5058502: WaitFor::WaitForValueEqualTo:: > (Predicate.h:123) ==2715==by 0x5058502: WaitForValueEqualTo (Predicate.h:157) ==2715==by 0x5058502: WaitForEventReceived (Event.h:114) ==2715==by 0x5058502: lldb_private::Process::ControlPrivateStateThread(unsigned int) (Process.cpp:3698) ==2715==by 0x505BC61: lldb_private::Process::StartPrivateStateThread(bool) (Process.cpp:3647) ==2715==by 0x5065B96: lldb_private::Process::Attach(lldb_private::ProcessAttachInfo&) (Process.cpp:2961) ==2715==by 0x544DBB8: PlatformPOSIX::Attach(lldb_private::ProcessAttachInfo&, lldb_private::Debugger&, lldb_private::Target*, lldb_private::Status&) (PlatformPOSIX.cpp:401) ==2715==by 0x509F531: lldb_private::Target::Attach(lldb_private::ProcessAttachInfo&, lldb_private::Stream*) (Target.cpp:3008) ==2715==by 0x54C3F17: CommandObjectProcessAttach::DoExecute(lldb_private::Args&, lldb_private::CommandReturnObject&) (CommandObjectProcess.cpp:386) ==2715==by 0x4FC0ACD: lldb_private::CommandObjectParsed::Execute(char const*, lldb_private::CommandReturnObject&) (CommandObject.cpp:993) ==2715==by 0x4FBCBD7: lldb_private::CommandInterpreter::HandleCommand(char const*, lldb_private::LazyBool, lldb_private::CommandReturnObject&, lldb_private::ExecutionContext*, bool, bool) (CommandInterpreter.cpp:1803) ==2715==by 0x4FBDB96: lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__cxx11::basic_string, std::allocator >&) (CommandInterpreter.cpp:2838) ==2715==by 0x4EF21C0: lldb_private::IOHandlerEditline::Run() (IOHandler.cpp:579) ==2715==by 0x4ED02B0: lldb_private::Debugger::RunIOHandlers() (Debugger.cpp:861) Hopefully this is an issue with valgrind and not lldb. But still curious whether someone else can reproduce something similar. Dmitry ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
