Re: [lldb-dev] Lldb-server spins forever in ptrace with 100% CPU on Linux Ubuntu 16.04
> could you get a backtrace of lldb-server when it is in the "stuck"
state (just attach with lldb/gdb after it hangs and do "bt")?
You wish [☹] The lldb-server does not react to any signals including SIGSTOP,
so gdb just hangs forever.
> If you can get me reasonably detailed repro steps, I can try to investigate
Unfortunately I do not have repro myself. It happens semi-randomly on some
machines and I need to borrow the machine with the problem. Here are some
details from my records:
* It is pretty big piece of RX memory, /proc//maps shows this:
409701000-40b49c000 r-xp 00:00 0
* Writing into some locations within that VMA works
* When it repros, it is pretty consistent, but changing in the target may
shift it - i.e. make no repro or fail at different address.
> are you able to still reproduce the bug with logging enabled?
Yes. Here are a few last lines from the log:
1481139040.768961000 0x7fff253c9780 Communication::Write (src = 0x7fff253c8f48,
src_len = 7) connection = 0x24a6bd0
1481139040.768963000 0x24a6bd0 ConnectionFileDescriptor::Write (src =
0x7fff253c8f48, src_len = 7)
1481139040.768973000 0x24a6cc0 Socket::Write() (socket = 6, src =
0x7fff253c8f48, src_len = 7, flags = 0) => 7 (error = (null))
1481139040.768976000 0x24a6bd0 ConnectionFileDescriptor::Write(fd = 6, src =
0x7fff253c8f48, src_len = 7) => 7 (error = (null))
1481139040.768979000 0x7fff253c9780 Communication::Read (dst = 0x7fff253c7140,
dst_len = 8192, timeout = 0 usec) connection = 0x24a6bd0
1481139040.768982000 0x24a6bd0 ConnectionFileDescriptor::BytesAvailable
(timeout_usec = 0)
1481139040.768984000 0x24a6bd0 ConnectionFileDescriptor::BytesAvailable()
::select (nfds=7, fds={6, 4}, NULL, NULL, timeout=0x7fff253c6d80)...
1481139040.768986000 0x24a6bd0 ConnectionFileDescriptor::BytesAvailable()
::select (nfds=7, fds={6, 4}, NULL, NULL, timeout=0x7fff253c6d80) => 0, error =
(null)
1481139090.788317000 0x7fff253c9780 Communication::Read (dst = 0x7fff253c7140,
dst_len = 8192, timeout = 0 usec) connection = 0x24a6bd0
1481139090.788356000 0x24a6bd0 ConnectionFileDescriptor::BytesAvailable
(timeout_usec = 0)
1481139090.788364000 0x24a6bd0 ConnectionFileDescriptor::BytesAvailable()
::select (nfds=7, fds={6, 4}, NULL, NULL, timeout=0x7fff253c6d80)...
1481139090.788368000 0x24a6bd0 ConnectionFileDescriptor::BytesAvailable()
::select (nfds=7, fds={6, 4}, NULL, NULL, timeout=0x7fff253c6d80) => 1, error =
(null)
1481139090.788378000 0x24a6cc0 Socket::Read() (socket = 6, src =
0x7fff253c7140, src_len = 25, flags = 0) => 25 (error = (null))
1481139090.788382000 0x24a6bd0 ConnectionFileDescriptor::Read() fd = 6, dst =
0x7fff253c7140, dst_len = 8192) => 25, error = (null)
1481139090.788395000 NativeProcessLinux::WriteMemory(0x409d5a7d0, 0x25271d0, 4)
1481139090.788409000 NativeProcessLinux::ReadMemory using process_vm_readv to
read 8 bytes from inferior address 0x409d5a7d0: Success
1481139090.788414000 PTRACE_POKEDATA [1][0][0][0][57][41][54][41]
Thanks,
Eugene
Sent from Outlook
From: Pavel Labath
Sent: Wednesday, December 7, 2016 2:34 AM
To: Eugene Birukov
Cc: LLDB
Subject: Re: [lldb-dev] Lldb-server spins forever in ptrace with 100% CPU on
Linux Ubuntu 16.04
Hello Eugene,
this sounds troubling, and I'd like to get to the bottom of this. If
you can get me a bit more information about this, I believe we can
figure it out:
- could you get a backtrace of lldb-server when it is in the "stuck"
state (just attach with lldb/gdb after it hangs and do "bt")? I want
to see the where is it spinning, as I don't see any obvious infinite
loop there.
- are you able to still reproduce the bug with logging enabled? If so,
I'd like to see the log file to understand this better. (You can
enable logging by starting lldb-server with: --log-file XXX.log
--log-channels "lldb all:linux all". If you're starting it via lldb
client you can set the LLDB_DEBUGSERVER_LOG_FILE and
LLDB_SERVER_LOG_CHANNELS environment vars to achieve this)
- If you can get me reasonably detailed repro steps, I can try to
investigate (I am fine with the first step being "install ubuntu 16.04
in virtualbox")
On 6 December 2016 at 23:41, Eugene Birukov via lldb-dev
wrote:
> Hi,
> 1. I believe that lldb-server spins inside ptrace. I put breakpoint on the
> highlighted line, and it does not hit. If I put breakpoint on line before,
> it hits but lldb-server hangs.
Do you mean actually inside the ptrace(2) syscall? Your description
would certainly fit that, but that sounds scary, as it would mean a
kernel bug. If that's the case, then we have to start looking in the
kernel. I have some experience with that, but If we can boil this down
to a simple use case, we can also ask the kernel ptrace folks for
help.
> 2. It seems that hang is caused by the client trying to read response too
> fast. I mean, if I step through the client code it works - i.e. there is
> significant del
[lldb-dev] [Bug 31308] New: po sometimes prints blank instead of object
https://llvm.org/bugs/show_bug.cgi?id=31308
Bug ID: 31308
Summary: po sometimes prints blank instead of object
Product: lldb
Version: 3.6
Hardware: Macintosh
OS: MacOS X
Status: NEW
Severity: normal
Priority: P
Component: All Bugs
Assignee: lldb-dev@lists.llvm.org
Reporter: m...@xmission.com
CC: llvm-b...@lists.llvm.org
Classification: Unclassified
My debug environment is Xcode 8.1 stopped at a breakpoint while debugging an
iOS application (usually 9.3.5 or 10.1) written in Objective C. Sometimes when
I use po to print an object, all it prints is a blank line. Printing with
different syntax will sometimes make it work. In this particular example the
object was a property of the argument to a simple method:
- (void)handleDataModelChange: (NSNotification *)nt
{
NSSet *updatedObjects = [[nt userInfo] objectForKey: NSUpdatedObjectsKey];
...
}
The breakpoint was set near the end of the method. Here is the output of some
po commands I tried. The last po command prints the object as it should be
printed by the first two po commands. Instead they just give blank lines.
(lldb) po nt.object
(lldb) po [nt object]
(lldb) po [nt.object class]
NSManagedObjectContext
(lldb) po (NSManagedObject *)nt.object
: defaultMainQueueContext
[lldb --version prints lldb-360.1.65]
--
You are receiving this mail because:
You are the assignee for the bug.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] logging in lldb
Sorry, I have been derailed today by trying to make format member
detection work more reasonably. I plan to come up with the next
iteration of the proposal soon, but I am not sure if it will be today,
as it is quite late here already. I do think this needs to be done in
incremental steps, but I'd like to make the iterations slightly bigger
to avoid code churn.
pl
On 7 December 2016 at 16:56, Zachary Turner wrote:
> Pavel: You might start this effort to improve logging by just adding a
> Formatv member function to log to replace printf.
>
> I suspect many call sites would be made much less verbose this way. I think
> we'll still need to do the rest too, but this seems like the least amount of
> work to make incremental progress
> On Tue, Dec 6, 2016 at 11:13 AM Zachary Turner wrote:
>>
>> On Tue, Dec 6, 2016 at 10:31 AM Greg Clayton wrote:
>>>
>>>
>>>
>>>
>>> It should be a formatter option for collections to allow you to print out
>>> different ranges. If I have a target and a target knows how to print the
>>> process and the process knows how to print the threads and the threads know
>>> how to print the stack frames, it doesn't mean I always want to see
>>> everything. Sounds like we can easily control this in the llvm::formatv case
>>> with extra formatting options:
>>>
>>> This might print just a summary of the target:
>>>
>>> llvm::formatv("{0}", target);
>>>
>>> This might print the target and info about the process:
>>>
>>> llvm::formatv("{0;process}", target);
>>>
>>> The question becomes, can we then forward arguments into the "Process"
>>> logger to control if the threads are displayed?
>>>
>>> llvm::formatv("{0;process{threads{frames}}}", target);
>>>
>>> This would allow recursive formatters to be called with specified
>>> arguments. Just a thought, but if we are going to delve into recursive
>>> formatters, we might need to think about that.
>>
>>
>> A formatter can literally do anything with the style string. So for
>> example, suppose you have formatters defined for Process and Thread. each
>> of these ultimately boils down to a function:
>>
>> void format(llvm::raw_ostream &Out, const Process &P, StringRef Style);
>> void format(llvm::raw_ostream &Out, const Thread &P, StringRef Style);
>>
>> We implement each of these, so Process can define its own Style language,
>> as can Target, as can Thread. We can come up with something better than
>> this, but say for the sake of illustration Thread's style string worked like
>> this: If empty, print thread id, otherwise treat each alphabetic character
>> as representing a "code" of what to print. P = process, T = thread id, N =
>> thread name. So "P - T - N" would print the process id, then the thread id,
>> then the thread name, separated by dashes.
>>
>> Now suppose you want to print a process. If style string is empty, it
>> prints only the process id, otherwise if it equals T then it prints the
>> first thread, or if it equals T* it prints all threads. T or T* can
>> optionally be followed by arguments to forward to the thread in square
>> brackets. So you could have:
>>
>> formatv("Process {0;P - T*[N]}", process);
>>
>> There's obviously a balance between flexibility and conciseness, but I
>> guess that's no different than C++. It's as flexible as you could possibly
>> need, sometimes to the point of allowing you to make bad design decisions :)
>>
>> In the beginning probably best to just define some simple formatters that
>> don't do anything recursive, and then some obvious patterns might start to
>> fall out and we can refine the styles we support for all of the various LLDB
>> types.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] logging in lldb
Pavel: You might start this effort to improve logging by just adding a
Formatv member function to log to replace printf.
I suspect many call sites would be made much less verbose this way. I think
we'll still need to do the rest too, but this seems like the least amount
of work to make incremental progress
On Tue, Dec 6, 2016 at 11:13 AM Zachary Turner wrote:
> On Tue, Dec 6, 2016 at 10:31 AM Greg Clayton wrote:
>
>
>
>
> It should be a formatter option for collections to allow you to print out
> different ranges. If I have a target and a target knows how to print the
> process and the process knows how to print the threads and the threads know
> how to print the stack frames, it doesn't mean I always want to see
> everything. Sounds like we can easily control this in the llvm::formatv
> case with extra formatting options:
>
> This might print just a summary of the target:
>
> llvm::formatv("{0}", target);
>
> This might print the target and info about the process:
>
> llvm::formatv("{0;process}", target);
>
> The question becomes, can we then forward arguments into the "Process"
> logger to control if the threads are displayed?
>
> llvm::formatv("{0;process{threads{frames}}}", target);
>
> This would allow recursive formatters to be called with specified
> arguments. Just a thought, but if we are going to delve into recursive
> formatters, we might need to think about that.
>
>
> A formatter can literally do anything with the style string. So for
> example, suppose you have formatters defined for Process and Thread. each
> of these ultimately boils down to a function:
>
> void format(llvm::raw_ostream &Out, const Process &P, StringRef Style);
> void format(llvm::raw_ostream &Out, const Thread &P, StringRef Style);
>
> We implement each of these, so Process can define its own Style language,
> as can Target, as can Thread. We can come up with something better than
> this, but say for the sake of illustration Thread's style string worked
> like this: If empty, print thread id, otherwise treat each alphabetic
> character as representing a "code" of what to print. P = process, T =
> thread id, N = thread name. So "P - T - N" would print the process id,
> then the thread id, then the thread name, separated by dashes.
>
> Now suppose you want to print a process. If style string is empty, it
> prints only the process id, otherwise if it equals T then it prints the
> first thread, or if it equals T* it prints all threads. T or T* can
> optionally be followed by arguments to forward to the thread in square
> brackets. So you could have:
>
> formatv("Process {0;P - T*[N]}", process);
>
> There's obviously a balance between flexibility and conciseness, but I
> guess that's no different than C++. It's as flexible as you could possibly
> need, sometimes to the point of allowing you to make bad design decisions :)
>
> In the beginning probably best to just define some simple formatters that
> don't do anything recursive, and then some obvious patterns might start to
> fall out and we can refine the styles we support for all of the various
> LLDB types.
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] Lldb-server spins forever in ptrace with 100% CPU on Linux Ubuntu 16.04
Hello Eugene, this sounds troubling, and I'd like to get to the bottom of this. If you can get me a bit more information about this, I believe we can figure it out: - could you get a backtrace of lldb-server when it is in the "stuck" state (just attach with lldb/gdb after it hangs and do "bt")? I want to see the where is it spinning, as I don't see any obvious infinite loop there. - are you able to still reproduce the bug with logging enabled? If so, I'd like to see the log file to understand this better. (You can enable logging by starting lldb-server with: --log-file XXX.log --log-channels "lldb all:linux all". If you're starting it via lldb client you can set the LLDB_DEBUGSERVER_LOG_FILE and LLDB_SERVER_LOG_CHANNELS environment vars to achieve this) - If you can get me reasonably detailed repro steps, I can try to investigate (I am fine with the first step being "install ubuntu 16.04 in virtualbox") On 6 December 2016 at 23:41, Eugene Birukov via lldb-dev wrote: > Hi, > 1. I believe that lldb-server spins inside ptrace. I put breakpoint on the > highlighted line, and it does not hit. If I put breakpoint on line before, > it hits but lldb-server hangs. Do you mean actually inside the ptrace(2) syscall? Your description would certainly fit that, but that sounds scary, as it would mean a kernel bug. If that's the case, then we have to start looking in the kernel. I have some experience with that, but If we can boil this down to a simple use case, we can also ask the kernel ptrace folks for help. > 2. It seems that hang is caused by the client trying to read response too > fast. I mean, if I step through the client code it works - i.e. there is > significant delay between client writing into pipe and issuing ::select to > wait for response. I am not sure how this fits in with the item above. I find it hard to believe that the presence of select(2) in one process would affect the outcome of ptrace() in another. Unless we are actually encountering a kernel scheduler bug, which I find unlikely. Hopefully we can get more insight here with additional logging information. > Any advice how to deal with the situation except putting random sleeps in > random places? Inserting sleeps in various places is a valid (albeit very slow) strategy for debugging races. If you can push the sleep down, you will eventually reach the place where it will be obvious what is racing (or, at least, which component is to blame). Hopefully we can do something smarter though. If you are suspecting a kernel bug, I'd recommend recreating it in a simple standalone application (fork, trace the child, write its memory), as then it is easy to ask for help pl ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
