Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Pavel Labath via lldb-commits]

On 25/01/2021 05:30, David Blaikie wrote:



On Fri, Jan 22, 2021 at 5:37 AM Pavel Labath > wrote:


On 19/01/2021 23:23, David Blaikie wrote:
 > On Tue, Jan 19, 2021 at 1:12 AM Pavel Labath mailto:pa...@labath.sk>> wrote:
 > Yeah - I have mixed feelings about debugger testing here - it is nice
 > to have end-to-end tests which makes for handy debugger testing
 > (though I guess in theory, debuginfo-tests is the place for
 > intentional end-to-end testing), though also being able to test
 > different features (DWARF version, split DWARF, dsym V object
 > debugging, etc) when they're written as end-to-end tests.

Yeah, it would be nice if there was a clearer separation between the
two
categories. The current setup has evolved organically, as the
end-to-end
API tests used to be the only kinds of tests.


 >
 > Can we write non-end-to-end API tests, then?

Kind of. There is no fundamental reason why one couldn't run llvm-mc or
whatever as a part of an API test. The main issue is that we don't have
the infrastructure for that set up right now. I think the reason for
that is that once you start dealing with "incomplete" executables which
cannot be run on the host platform, the usefulness of interactivity
goes
down sharply. It is hard for such a test to do something other than
load
up some executable and query its state. This is a perfect use case
for a
shell test.

There are exceptions though. For example we have a collection of "API"
tests which test the gdb-remote communication layer, by mocking one end
of the connection. Such tests are necessarily interactive, which is why
they ended up in the API category, but they are definitely not
end-to-end tests, and they either don't use any executables, or just
use
a static yaml2objed executable. This is why our API tests have the
ability to run yaml2obj and one could add other llvm tools in a similar
fashion.


Though then you get the drawback that appears to be Jim's original/early 
motivation: Avoiding checking the textual output of the debugger as much 
as possible (so that updating that output doesn't involve changing every 
single test case - which can cause stagnation due to it being too 
expensive to update all the tests, so improvements to the output format 
are not made). Which I have some sympathy for - though LLVM and clang 
seem to get by without this to some degree - we do/have significantly 
changed the LLVM IR format, for instance, and done regex updates to the 
FileCheck/lit tests to account for that - Clang, well, if we changed the 
format of error messages we might have a bad time, except that we rarely 
test the literal warning/error text, instead using clang's testing 
functionality with the // expected-error, etc, syntax.


Yeah, I am aware of Jim's objections to this. :) I haven't been around 
here as long as he has, and I definitely don't, and definitely haven't 
been around to witness the gdb format change attempt, but with the shell 
tests we have, I haven't seen any indication that they inhibit the 
velocity of lldb development (we haven't done any major format recently 
though -- but if we did, I don't think the shell tests would be a big 
problem). So I tend to not put so much emphasis on this aspect, and 
choose to prioritise other aspects of tests (e.g. ease of writing).


With different priorities it's not surprising (though still unfortunate) 
that we reach different conclusions.




I guess lldb doesn't have a machine readable form, like gdb's machine 
interface that might make for a more robust thing to test against most 
of the time (& then leaving a limited number of tests that test the 
user-textual output)? Instead the python API is the machine interface?


Fun fact: We used to have an lldb-mi tool in the repository, which 
implemented the gdb-mi interface on top of lldb. Although the 
implementation of the tool itself was bad, the main reason why the tool 
was removed was that its tests were horribly flaky and hard to 
understand





Another aspect of end-to-endness is being able to test a specific
component of lldb, instead of just the debugger as a whole. Here the
API
tests cannot help because the "API" is the lldb public API.


Not sure I followed here - you mean the API tests aren't more narrowly 
targeted than the Shell tests, because the API is the public API, so 
it's mostly/pretty close to what you can interact with from the Shell 
anyway - doesn't give you lower-level access akin to unit testing? Fair 
enough.


What I wanted to say here is that the Shell tests can (and some, do) go 
lower level than the lldb public API, though the "lldb-test" tool. It 
sits below the public api, and exposes some useful features of the 
underlying apis. The idea was for it to be the compromise solution for 
the "have a stringly matchable interface but 

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via lldb-commits]

On Mon, Jan 25, 2021 at 10:39 AM Jim Ingham  wrote:

> lldb creates a single line table out of the debug_line input and the
> inlined_subroutine info from the debug_info.  For instance, if you have a
> nested set of inlines that all have the same start address, they cannot be
> represented in the debug_line which requires monotonically increasing
> addresses.  But when you are "stepping" into the set of nested inlines,
> lldb would like to represent the step as first stopping before the call in
> the outer function, then stepping one by one into the nested inlines.  I
> don't know why exactly DWARF line tables need to be monotonically
> increasing in address, but we couldn't see any good reason why lldb should
> also have to do this little two-step every time we tried to figure out
> where you are, we decided to merge the info up front.
>
> You could still write Unit Tests accessing the lldb line table
> construction rather than the DWARF debug_line ingestion.  But IIRC the
> layer between the LineEntries in lldb_private and the classes in the SB API
> is pretty thin, so there's less motivation to test that way.
>

This patch/test isn't about inlined subroutines - it's about
DW_TAG_subprograms with DW_AT_ranges - specifically the patch
changes DWARFDebugInfoEntry::BuildFunctionAddressRangeTable - so it sounds
like it probably wouldn't be unit testable via the line table alone (or
perhaps the line table at all). Given the novel (to lldb) functionality,
and partial but untested implementation already in lldb prior to these
recent patches (there was at least one place with a comment explaining why
it was doing something a particular way to support discontiguous functions)
I think the end-to-end testing in some form (SB API or Shell, rather than
the narrower testing of a unit test) is probably valuable to give some more
test coverage to this whole situation, not only the one or two places I've
found specific bugs/fixes.

I'm still mystified why I couldn't reproduce the behavior using the SB API,
but I could using a shell test/using lldb interactively myself. But for now
it seems the Shell test is about the best thing I can do.

- Dave


>
> Jim
>
>
> > On Jan 24, 2021, at 8:32 PM, David Blaikie  wrote:
> >
> >
> >
> > On Fri, Jan 22, 2021 at 9:42 AM Jim Ingham  wrote:
> > If you are just loading an object file and the looking at the line table
> or something like that, would a UnitTest be more suitable?
> >
> > Maybe, though for what it's worth, this isn't an issue with the line
> table (well, not the .debug_line contents) as such - this is testing a
> change to .debug_info - specifically, the use of DW_AT_ranges on a
> DW_TAG_subprogram. But it manifested as breakpoints not having source
> information. I don't know a great deal about how the line table ended up
> interacting with the address ranges specified on the DW_TAG_subprogram, but
> it did in some way.
> >
> > Not sure if that is or isn't especially amenable to unit testing given
> the LLDB architecture of these components.
> >
> > - Dave
> >
> >
> > Jim
> >
> >
> > > On Jan 22, 2021, at 5:37 AM, Pavel Labath  wrote:
> > >
> > > On 19/01/2021 23:23, David Blaikie wrote:
> > >> On Tue, Jan 19, 2021 at 1:12 AM Pavel Labath  wrote:
> > >> Yeah - I have mixed feelings about debugger testing here - it is nice
> > >> to have end-to-end tests which makes for handy debugger testing
> > >> (though I guess in theory, debuginfo-tests is the place for
> > >> intentional end-to-end testing), though also being able to test
> > >> different features (DWARF version, split DWARF, dsym V object
> > >> debugging, etc) when they're written as end-to-end tests.
> > >
> > > Yeah, it would be nice if there was a clearer separation between the
> two categories. The current setup has evolved organically, as the
> end-to-end API tests used to be the only kinds of tests.
> > >
> > >
> > >> Can we write non-end-to-end API tests, then?
> > >
> > > Kind of. There is no fundamental reason why one couldn't run llvm-mc
> or whatever as a part of an API test. The main issue is that we don't have
> the infrastructure for that set up right now. I think the reason for that
> is that once you start dealing with "incomplete" executables which cannot
> be run on the host platform, the usefulness of interactivity goes down
> sharply. It is hard for such a test to do something other than load up some
> executable and query its state. This is a perfect use case for a shell test.
> > >
> > > There are exceptions though. For example we have a collection of "API"
> tests which test the gdb-remote communication layer, by mocking one end of
> the connection. Such tests are necessarily interactive, which is why they
> ended up in the API category, but they are definitely not end-to-end tests,
> and they either don't use any executables, or just use a static yaml2objed
> executable. This is why our API tests have the ability to run yaml2obj and
> one could add other llvm tools in a similar fashion.
> 

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Jim Ingham via lldb-commits]

> On Jan 24, 2021, at 8:30 PM, David Blaikie  wrote:
> 
> I guess lldb doesn't have a machine readable form, like gdb's machine 
> interface that might make for a more robust thing to test against most of the 
> time (& then leaving a limited number of tests that test the user-textual 
> output)? Instead the python API is the machine interface? 

That was the original idea, except that we also tried to make it easy to test 
command line output in the same tests that test the API's.  Really the only 
part of the system the API tests aren't great at is stuff that depends on 
mid-command interactivity (prompts, line-editing interactions, the REPL, stuff 
like that).  The difference between the two to my mind is pretty much that the 
Shell tests don't require you to know Python, or learn the SB API's to write 
tests.

>  
> Another aspect of end-to-endness is being able to test a specific 
> component of lldb, instead of just the debugger as a whole. Here the API 
> tests cannot help because the "API" is the lldb public API.
> 
> Not sure I followed here - you mean the API tests aren't more narrowly 
> targeted than the Shell tests, because the API is the public API, so it's 
> mostly/pretty close to what you can interact with from the Shell anyway - 
> doesn't give you lower-level access akin to unit testing? Fair enough.
>  

Yes, this is particularly true for lldb's plugins.  There isn't any really 
compelling reason for an lldb user to know the details of how DWARF was 
ingested.  At that level, lldb should present the data from the generic Symbol 
interface, not how it came from DWARF.  So if you want to test the DWARF parser 
specifically then UnitTests are the way to go.  And for testing utility classes 
which don't require lots of other subsystems to construct themselves, the 
UnitTests are much easier to write.  These also are easy to migrate as code 
changes because we know how to do "I added a parameter to an API, go fix the 
calling code" pretty well.

My take is that API tests will always be more precise because each time you get 
a bit of data that you want to test against you get only that data, you don't 
get it intermixed with other text that you have to decide how to either capture 
(and maybe end up testing things you didn't intend to test) or run the risk 
that the patterns you were using to grab the data you thought you were testing 
actually grabbed a different part of the command text.  The solution to that if 
you wanted to do more rigorous tests from the command line is to write special 
purpose commands that present output in a form that's not good for users but is 
better structured for testing.  But that seems like wasted effort and support 
burden when the API's to get what you want are right there in front of you...

Jim


___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Jim Ingham via lldb-commits]

lldb creates a single line table out of the debug_line input and the 
inlined_subroutine info from the debug_info.  For instance, if you have a 
nested set of inlines that all have the same start address, they cannot be 
represented in the debug_line which requires monotonically increasing 
addresses.  But when you are "stepping" into the set of nested inlines, lldb 
would like to represent the step as first stopping before the call in the outer 
function, then stepping one by one into the nested inlines.  I don't know why 
exactly DWARF line tables need to be monotonically increasing in address, but 
we couldn't see any good reason why lldb should also have to do this little 
two-step every time we tried to figure out where you are, we decided to merge 
the info up front.

You could still write Unit Tests accessing the lldb line table construction 
rather than the DWARF debug_line ingestion.  But IIRC the layer between the 
LineEntries in lldb_private and the classes in the SB API is pretty thin, so 
there's less motivation to test that way.

Jim


> On Jan 24, 2021, at 8:32 PM, David Blaikie  wrote:
> 
> 
> 
> On Fri, Jan 22, 2021 at 9:42 AM Jim Ingham  wrote:
> If you are just loading an object file and the looking at the line table or 
> something like that, would a UnitTest be more suitable?
> 
> Maybe, though for what it's worth, this isn't an issue with the line table 
> (well, not the .debug_line contents) as such - this is testing a change to 
> .debug_info - specifically, the use of DW_AT_ranges on a DW_TAG_subprogram. 
> But it manifested as breakpoints not having source information. I don't know 
> a great deal about how the line table ended up interacting with the address 
> ranges specified on the DW_TAG_subprogram, but it did in some way.
> 
> Not sure if that is or isn't especially amenable to unit testing given the 
> LLDB architecture of these components.
> 
> - Dave
>  
> 
> Jim
> 
> 
> > On Jan 22, 2021, at 5:37 AM, Pavel Labath  wrote:
> > 
> > On 19/01/2021 23:23, David Blaikie wrote:
> >> On Tue, Jan 19, 2021 at 1:12 AM Pavel Labath  wrote:
> >> Yeah - I have mixed feelings about debugger testing here - it is nice
> >> to have end-to-end tests which makes for handy debugger testing
> >> (though I guess in theory, debuginfo-tests is the place for
> >> intentional end-to-end testing), though also being able to test
> >> different features (DWARF version, split DWARF, dsym V object
> >> debugging, etc) when they're written as end-to-end tests.
> > 
> > Yeah, it would be nice if there was a clearer separation between the two 
> > categories. The current setup has evolved organically, as the end-to-end 
> > API tests used to be the only kinds of tests.
> > 
> > 
> >> Can we write non-end-to-end API tests, then?
> > 
> > Kind of. There is no fundamental reason why one couldn't run llvm-mc or 
> > whatever as a part of an API test. The main issue is that we don't have the 
> > infrastructure for that set up right now. I think the reason for that is 
> > that once you start dealing with "incomplete" executables which cannot be 
> > run on the host platform, the usefulness of interactivity goes down 
> > sharply. It is hard for such a test to do something other than load up some 
> > executable and query its state. This is a perfect use case for a shell test.
> > 
> > There are exceptions though. For example we have a collection of "API" 
> > tests which test the gdb-remote communication layer, by mocking one end of 
> > the connection. Such tests are necessarily interactive, which is why they 
> > ended up in the API category, but they are definitely not end-to-end tests, 
> > and they either don't use any executables, or just use a static yaml2objed 
> > executable. This is why our API tests have the ability to run yaml2obj and 
> > one could add other llvm tools in a similar fashion.
> > 
> > Another aspect of end-to-endness is being able to test a specific component 
> > of lldb, instead of just the debugger as a whole. Here the API tests cannot 
> > help because the "API" is the lldb public API. However, there are also 
> > various tricks you can do by using the low-level (debugging) commands (like 
> > the "image lookup" thing I mentioned) to interact with the lower debugger 
> > layers in some manner.
> > 
> > 
> > pl
> 

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via lldb-commits]

On Fri, Jan 22, 2021 at 9:42 AM Jim Ingham  wrote:

> If you are just loading an object file and the looking at the line table
> or something like that, would a UnitTest be more suitable?
>

Maybe, though for what it's worth, this isn't an issue with the line table
(well, not the .debug_line contents) as such - this is testing a change to
.debug_info - specifically, the use of DW_AT_ranges on a DW_TAG_subprogram.
But it manifested as breakpoints not having source information. I don't
know a great deal about how the line table ended up interacting with the
address ranges specified on the DW_TAG_subprogram, but it did in some way.

Not sure if that is or isn't especially amenable to unit testing given the
LLDB architecture of these components.

- Dave


>
> Jim
>
>
> > On Jan 22, 2021, at 5:37 AM, Pavel Labath  wrote:
> >
> > On 19/01/2021 23:23, David Blaikie wrote:
> >> On Tue, Jan 19, 2021 at 1:12 AM Pavel Labath  wrote:
> >> Yeah - I have mixed feelings about debugger testing here - it is nice
> >> to have end-to-end tests which makes for handy debugger testing
> >> (though I guess in theory, debuginfo-tests is the place for
> >> intentional end-to-end testing), though also being able to test
> >> different features (DWARF version, split DWARF, dsym V object
> >> debugging, etc) when they're written as end-to-end tests.
> >
> > Yeah, it would be nice if there was a clearer separation between the two
> categories. The current setup has evolved organically, as the end-to-end
> API tests used to be the only kinds of tests.
> >
> >
> >> Can we write non-end-to-end API tests, then?
> >
> > Kind of. There is no fundamental reason why one couldn't run llvm-mc or
> whatever as a part of an API test. The main issue is that we don't have the
> infrastructure for that set up right now. I think the reason for that is
> that once you start dealing with "incomplete" executables which cannot be
> run on the host platform, the usefulness of interactivity goes down
> sharply. It is hard for such a test to do something other than load up some
> executable and query its state. This is a perfect use case for a shell test.
> >
> > There are exceptions though. For example we have a collection of "API"
> tests which test the gdb-remote communication layer, by mocking one end of
> the connection. Such tests are necessarily interactive, which is why they
> ended up in the API category, but they are definitely not end-to-end tests,
> and they either don't use any executables, or just use a static yaml2objed
> executable. This is why our API tests have the ability to run yaml2obj and
> one could add other llvm tools in a similar fashion.
> >
> > Another aspect of end-to-endness is being able to test a specific
> component of lldb, instead of just the debugger as a whole. Here the API
> tests cannot help because the "API" is the lldb public API. However, there
> are also various tricks you can do by using the low-level (debugging)
> commands (like the "image lookup" thing I mentioned) to interact with the
> lower debugger layers in some manner.
> >
> >
> > pl
>
>
___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via lldb-commits]

On Fri, Jan 22, 2021 at 5:37 AM Pavel Labath  wrote:

> On 19/01/2021 23:23, David Blaikie wrote:
> > On Tue, Jan 19, 2021 at 1:12 AM Pavel Labath  wrote:
> > Yeah - I have mixed feelings about debugger testing here - it is nice
> > to have end-to-end tests which makes for handy debugger testing
> > (though I guess in theory, debuginfo-tests is the place for
> > intentional end-to-end testing), though also being able to test
> > different features (DWARF version, split DWARF, dsym V object
> > debugging, etc) when they're written as end-to-end tests.
>
> Yeah, it would be nice if there was a clearer separation between the two
> categories. The current setup has evolved organically, as the end-to-end
> API tests used to be the only kinds of tests.
>
>
> >
> > Can we write non-end-to-end API tests, then?
>
> Kind of. There is no fundamental reason why one couldn't run llvm-mc or
> whatever as a part of an API test. The main issue is that we don't have
> the infrastructure for that set up right now. I think the reason for
> that is that once you start dealing with "incomplete" executables which
> cannot be run on the host platform, the usefulness of interactivity goes
> down sharply. It is hard for such a test to do something other than load
> up some executable and query its state. This is a perfect use case for a
> shell test.
>
> There are exceptions though. For example we have a collection of "API"
> tests which test the gdb-remote communication layer, by mocking one end
> of the connection. Such tests are necessarily interactive, which is why
> they ended up in the API category, but they are definitely not
> end-to-end tests, and they either don't use any executables, or just use
> a static yaml2objed executable. This is why our API tests have the
> ability to run yaml2obj and one could add other llvm tools in a similar
> fashion.
>

Though then you get the drawback that appears to be Jim's original/early
motivation: Avoiding checking the textual output of the debugger as much as
possible (so that updating that output doesn't involve changing every
single test case - which can cause stagnation due to it being too expensive
to update all the tests, so improvements to the output format are not
made). Which I have some sympathy for - though LLVM and clang seem to get
by without this to some degree - we do/have significantly changed the LLVM
IR format, for instance, and done regex updates to the FileCheck/lit tests
to account for that - Clang, well, if we changed the format of error
messages we might have a bad time, except that we rarely test the literal
warning/error text, instead using clang's testing functionality with the //
expected-error, etc, syntax.

I guess lldb doesn't have a machine readable form, like gdb's machine
interface that might make for a more robust thing to test against most of
the time (& then leaving a limited number of tests that test the
user-textual output)? Instead the python API is the machine interface?


> Another aspect of end-to-endness is being able to test a specific
> component of lldb, instead of just the debugger as a whole. Here the API
> tests cannot help because the "API" is the lldb public API.


Not sure I followed here - you mean the API tests aren't more narrowly
targeted than the Shell tests, because the API is the public API, so it's
mostly/pretty close to what you can interact with from the Shell anyway -
doesn't give you lower-level access akin to unit testing? Fair enough.


> However,
> there are also various tricks you can do by using the low-level
> (debugging) commands (like the "image lookup" thing I mentioned) to
> interact with the lower debugger layers in some manner.
>
>
> pl
>
___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Pavel Labath via lldb-commits]

On 22/01/2021 18:42, Jim Ingham wrote:

If you are just loading an object file and the looking at the line table or 
something like that, would a UnitTest be more suitable?

Jim



It depends on who you're talking to. :) I, for one, think we should be 
writing more unit tests, however, this is not a universally accepted 
position (quite the opposite -- some folks will go to enormous lengths 
to avoid writing one -- creating single-purpose utilities, etc.).


A major stumbling block for writing unit tests is the ability to 
generate test inputs (and this is both a consequence of and the source 
of the aversion to unit tests). We now have the ability to generate 
binaries from yaml programatically, but for a long time, we didn't have 
even that (and our unit tests were shelling out to the yaml2obj binary), 
but there's nothing similar for other llvm tools. All of the relevant 
code is already present in llvm, so it wouldn't be impossible to create 
something like that, but it is work...


But even I have to admit that the "lit" style tests have their 
advantages, if used reasonably (no proliferation of one-shot utilities 
and flags, etc.). The tricky thing about unit tests is ensuring they 
produce good (actionable) error messages when that fails. That usually 
means writing to_string methods for all objects being compared. If those 
to_string methods are good, then there shouldn't be much difference 
between comparing their outputs vs. comparing the objects themselves. 
And this is pretty much what FileCheck does. Plus, this has some other 
advantages, like being able to run the tool on random inputs to check 
its output, not needing to recompile when you modify the input, etc.


pl
___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Jim Ingham via lldb-commits]

If you are just loading an object file and the looking at the line table or 
something like that, would a UnitTest be more suitable?

Jim


> On Jan 22, 2021, at 5:37 AM, Pavel Labath  wrote:
> 
> On 19/01/2021 23:23, David Blaikie wrote:
>> On Tue, Jan 19, 2021 at 1:12 AM Pavel Labath  wrote:
>> Yeah - I have mixed feelings about debugger testing here - it is nice
>> to have end-to-end tests which makes for handy debugger testing
>> (though I guess in theory, debuginfo-tests is the place for
>> intentional end-to-end testing), though also being able to test
>> different features (DWARF version, split DWARF, dsym V object
>> debugging, etc) when they're written as end-to-end tests.
> 
> Yeah, it would be nice if there was a clearer separation between the two 
> categories. The current setup has evolved organically, as the end-to-end API 
> tests used to be the only kinds of tests.
> 
> 
>> Can we write non-end-to-end API tests, then?
> 
> Kind of. There is no fundamental reason why one couldn't run llvm-mc or 
> whatever as a part of an API test. The main issue is that we don't have the 
> infrastructure for that set up right now. I think the reason for that is that 
> once you start dealing with "incomplete" executables which cannot be run on 
> the host platform, the usefulness of interactivity goes down sharply. It is 
> hard for such a test to do something other than load up some executable and 
> query its state. This is a perfect use case for a shell test.
> 
> There are exceptions though. For example we have a collection of "API" tests 
> which test the gdb-remote communication layer, by mocking one end of the 
> connection. Such tests are necessarily interactive, which is why they ended 
> up in the API category, but they are definitely not end-to-end tests, and 
> they either don't use any executables, or just use a static yaml2objed 
> executable. This is why our API tests have the ability to run yaml2obj and 
> one could add other llvm tools in a similar fashion.
> 
> Another aspect of end-to-endness is being able to test a specific component 
> of lldb, instead of just the debugger as a whole. Here the API tests cannot 
> help because the "API" is the lldb public API. However, there are also 
> various tricks you can do by using the low-level (debugging) commands (like 
> the "image lookup" thing I mentioned) to interact with the lower debugger 
> layers in some manner.
> 
> 
> pl

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Pavel Labath via lldb-commits]

On 19/01/2021 23:23, David Blaikie wrote:

On Tue, Jan 19, 2021 at 1:12 AM Pavel Labath  wrote:
Yeah - I have mixed feelings about debugger testing here - it is nice
to have end-to-end tests which makes for handy debugger testing
(though I guess in theory, debuginfo-tests is the place for
intentional end-to-end testing), though also being able to test
different features (DWARF version, split DWARF, dsym V object
debugging, etc) when they're written as end-to-end tests.


Yeah, it would be nice if there was a clearer separation between the two 
categories. The current setup has evolved organically, as the end-to-end 
API tests used to be the only kinds of tests.





Can we write non-end-to-end API tests, then?


Kind of. There is no fundamental reason why one couldn't run llvm-mc or 
whatever as a part of an API test. The main issue is that we don't have 
the infrastructure for that set up right now. I think the reason for 
that is that once you start dealing with "incomplete" executables which 
cannot be run on the host platform, the usefulness of interactivity goes 
down sharply. It is hard for such a test to do something other than load 
up some executable and query its state. This is a perfect use case for a 
shell test.


There are exceptions though. For example we have a collection of "API" 
tests which test the gdb-remote communication layer, by mocking one end 
of the connection. Such tests are necessarily interactive, which is why 
they ended up in the API category, but they are definitely not 
end-to-end tests, and they either don't use any executables, or just use 
a static yaml2objed executable. This is why our API tests have the 
ability to run yaml2obj and one could add other llvm tools in a similar 
fashion.


Another aspect of end-to-endness is being able to test a specific 
component of lldb, instead of just the debugger as a whole. Here the API 
tests cannot help because the "API" is the lldb public API. However, 
there are also various tricks you can do by using the low-level 
(debugging) commands (like the "image lookup" thing I mentioned) to 
interact with the lower debugger layers in some manner.



pl
___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via lldb-commits]

On Tue, Jan 19, 2021 at 1:12 AM Pavel Labath  wrote:
>
> (sorry for inactivity -- I mostly tuned out of this discussion as I
> couldn't keep up with its pace)

No worries - sorry for the noise/thanks for taking the time!

> On 08/01/2021 01:55, Jim Ingham via lldb-commits wrote:
>  > 3) However, we try to push as many tests as possible all the way
>  > through the compiler, since the lldb test suite is also one of the
>  > significant test harnesses for compiler debug output.  .s files are
>  > exposed to much less of the compiler, and so don't catch new compiler
>  > debug output bugs as well.  So unless you have a good reason to use a
>  > .s file, you shouldn't.
>
> On 11/01/2021 02:37, David Blaikie via lldb-commits wrote:
> > Thanks for all the context - so sounds like mostly based on (3) the
> > recommendation would be for this to be an API test
>
> Well... as Jim said, there are different takes on this :), and I would
> not be so quick to convert this to an API test. While that does make
> sure that the compiler part is tested, it also introduces the opposite
> problem -- symmetric bugs in clang & lldb cancelling each other out,
> which can lead to lldb only being able to debug binaries built by a
> matching compiler.

Oh, if Clang's output regressed/changed in a way that lldb didn't
notice, then lldb could regress without failing the test.

Yeah - I have mixed feelings about debugger testing here - it is nice
to have end-to-end tests which makes for handy debugger testing
(though I guess in theory, debuginfo-tests is the place for
intentional end-to-end testing), though also being able to test
different features (DWARF version, split DWARF, dsym V object
debugging, etc) when they're written as end-to-end tests.

Can we write non-end-to-end API tests, then? So the API/Shell
differentiation aren't conflated with end-to-end/assembly-to-debugging
differentiation?

> While this can be mitigated by running the tests
> against an older compiler (API tests can do that, shell tests cannot),
> that requires that:
> a) someone actually runs the tests this way
> b) those tests work with the older compiler -- unstable/experimental
> flags, or relying on very specific details of the compiler output make
> that tricky

Yep, the internal flag nature here is a bit limiting for sure.

> This is why I was saying (way back) that, if anything, we should have
> two tests, one that would be very explicit about what input it is
> testing, and a second one, which tests the integration with the compiler.

Fair enough - thanks for explaining. For now I'm pretty much ready to
give up on the API testing given my inability to reproduce the
behavior problems in an API test.

> I am not saying that every patch should have two kinds of test (most of
> the time, I am happy that a patch has _any_ test). However, I am not
> sure that the second test is really useful (except as a learning
> exercise) in this case -- that's because the -mllvm switch is
> (deliberately) very targeted, so it's hard to see what else would be
> tested by it (the codegen is presumably already tested by an llvm test).
> Also, now that you already have written the assembly test, I'd be sad to
> see it thrown away.
>
> What would make the second test really interesting (to me) is if could
> be phrased in a way that's as independent as possible from specific
> clang flags, and ideally even DWARF. Like, if it was phrased as "test
> that we can (break, step, view variables) in discontinuous functions",
> then we could run the same tests against gcc or msvc (pdb) and check
> that we do something reasonable.

Yeah, that's much harder - especially for the feature I'm working on
("ranges everywhere"), it's a very DWARF-specific feature.
Discontiguous ranges is a bit more of an explicit feature, but seem to
be used pretty differently (different compiler flags at least - not
sure if there's common source code that could be compiled with
different flags).

> (Note that the msvc part is just a pipe dream, as we currently don't
> have any infrastructure to compile test inferiors with msvc, and even if
> we did, I expect most of them would fail.)
>
> Also note that, I am not saying that we should never use -mllvm options
> in API tests. However, doing that lowers their value in my mind. The
> reason why it's not possible to make a clear cut (-mllvm does not go
> into API tests, only Shell ones), is that there are also other competing
> criteria at play here -- some things just cannot be tested with a Shell
> test due to their non-interactivity, even though they are testing a very
> specific aspect of debug info (or they are not even testing debug info
> at all).
>
>
> cheers,
> pl
>
> PS: I don't know if you've seen this, but we also have an "official"
> description of the various test types here:
> .
___
lldb-commits mailing list
lldb-commits@lists.llvm.org

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Pavel Labath via lldb-commits]

(sorry for inactivity -- I mostly tuned out of this discussion as I 
couldn't keep up with its pace)


On 08/01/2021 01:55, Jim Ingham via lldb-commits wrote:
> 3) However, we try to push as many tests as possible all the way
> through the compiler, since the lldb test suite is also one of the
> significant test harnesses for compiler debug output.  .s files are
> exposed to much less of the compiler, and so don't catch new compiler
> debug output bugs as well.  So unless you have a good reason to use a
> .s file, you shouldn't.

On 11/01/2021 02:37, David Blaikie via lldb-commits wrote:
Thanks for all the context - so sounds like mostly based on (3) the 
recommendation would be for this to be an API test


Well... as Jim said, there are different takes on this :), and I would 
not be so quick to convert this to an API test. While that does make 
sure that the compiler part is tested, it also introduces the opposite 
problem -- symmetric bugs in clang & lldb cancelling each other out, 
which can lead to lldb only being able to debug binaries built by a 
matching compiler. While this can be mitigated by running the tests 
against an older compiler (API tests can do that, shell tests cannot), 
that requires that:

a) someone actually runs the tests this way
b) those tests work with the older compiler -- unstable/experimental 
flags, or relying on very specific details of the compiler output make 
that tricky


This is why I was saying (way back) that, if anything, we should have 
two tests, one that would be very explicit about what input it is 
testing, and a second one, which tests the integration with the compiler.


I am not saying that every patch should have two kinds of test (most of 
the time, I am happy that a patch has _any_ test). However, I am not 
sure that the second test is really useful (except as a learning 
exercise) in this case -- that's because the -mllvm switch is 
(deliberately) very targeted, so it's hard to see what else would be 
tested by it (the codegen is presumably already tested by an llvm test). 
Also, now that you already have written the assembly test, I'd be sad to 
see it thrown away.


What would make the second test really interesting (to me) is if could 
be phrased in a way that's as independent as possible from specific 
clang flags, and ideally even DWARF. Like, if it was phrased as "test 
that we can (break, step, view variables) in discontinuous functions", 
then we could run the same tests against gcc or msvc (pdb) and check 
that we do something reasonable.


(Note that the msvc part is just a pipe dream, as we currently don't 
have any infrastructure to compile test inferiors with msvc, and even if 
we did, I expect most of them would fail.)


Also note that, I am not saying that we should never use -mllvm options 
in API tests. However, doing that lowers their value in my mind. The 
reason why it's not possible to make a clear cut (-mllvm does not go 
into API tests, only Shell ones), is that there are also other competing 
criteria at play here -- some things just cannot be tested with a Shell 
test due to their non-interactivity, even though they are testing a very 
specific aspect of debug info (or they are not even testing debug info 
at all).



cheers,
pl

PS: I don't know if you've seen this, but we also have an "official" 
description of the various test types here: 
.

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via lldb-commits]

On Thu, Jan 14, 2021 at 2:56 PM Jim Ingham  wrote:
>
> How were you setting the breakpoint in the case where it was failing from the 
> command line?  The breakpoint you are examining in the test in your patches 
> is a "source regular expression" breakpoint.  That would be "break set -p".  
> If you were also doing that in the command line and got a different result 
> that would be very odd, since neither interface (CLI or SB) does much work 
> before handing the request off to the common routine.  If (as I suspect) you 
> were doing "b main.c:40" in the command line, then you should make a 
> breakpoint with SBTarget.BreakpointCreateByLocation and see if that shows the 
> same problem.  That should do the same job as "break set --file --line".

Ah, thanks for the explanation (didn't realize lldb natively supported
setting a breakpoint by regex - figured that was just in the test
helpers and mapped down to setting a breakpoint by line).

I tried setting the breakpoint by regex to more closely match the API
test & still can't seem to reproduce the behavior inside the API test
outside of it:

$ 
LD_LIBRARY_PATH=/usr/local/google/home/blaikie/dev/llvm/build/default/lib:/usr/local/google/home/blaikie/install/lib:/usr/local/google/home/blaikie/install/lib64
./bin/lldb 
./lldb-test-build.noindex/lang/c/stepping/TestStepAndBreakpoints.test_and_python_api/a.out


 (lldb) target create
"./lldb-test-build.noindex/lang/c/stepping/TestStepAndBreakpoints.test_and_python_api/a.out"
Current executable set to
'/usr/local/google/home/blaikie/dev/llvm/build/default/lldb-test-build.noindex/lang/c/stepping/TestStepAndBreakpoints.test_and_python_api/a.out'
(x86_64).
(lldb) break set -p "// frame select 2, thread step-out while stopped at .c.1.."
Breakpoint 1: where = a.out`main + 22, address = 0x004011c6

(still missing the file/line information, despite the (see patch
attached to the previous for the test changes) test seeming to test
the same behavior & appears to print the file/line correctly:

   
self.assertEqual(get_description(break_1_in_main.locations[0].GetAddress().line_entry),
"narf")
AssertionError:
'/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/lang/c/stepping/main.c:40:14'
!= 'narf'
- 
/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/lang/c/stepping/main.c:40:14+
narf
)

I'm wondering if maybe something is picking up some other artifact in
my environment - the same issue with me needing to set the
LD_LIBRARY_PATH - hmm, maybe not (I uninstalled all other lldb things
I can think of, so it's not picking up some system/user installed copy
and running different code, so far as I can think)

> Note, you can also run the command line breakpoint command in API tests.  
> That's sometimes required, particularly if there's something about a 
> combination of command flags that you want to test.  Just use the 
> lldbutil.run_break_set_by_file_and_line utility.  That centralizes running 
> and checking the basic results of the break set command, so if we do change 
> the break command we only have to fix one place...  In your case the checks 
> that run_break_set_by_file_and_line do should suffice, but the command 
> returns the created breakpoint number, so if you want to poke at it further, 
> you can use  target.FindBreakpointByID to get the SBBreakpoint you just made. 
>  There are also wrappers for the other lldb breakpoint types you can make 
> from the command line, BTW.
>
> Jim
>
>
>
> > On Jan 14, 2021, at 1:38 PM, David Blaikie  wrote:
> >
> > Thanks for the pointers!
> >
> > So I tried writing an API-type test for this line table/breakpoint
> > behavior (it doesn't seem like it's entirely/just the line table (for
> > one thing, the DWARF change is purely in the subprogram DIE, not in
> > the line table)) and I can't seem to figure it out.
> >
> > The attached patch modifies lldb to undo this patch (to show the
> > breakage) and modifies an existing API test to build with the clang
> > flag to enable ranges on subprograms - I've confirmed the DWARF built
> > by this test does have subprogram ranges.
> >
> > But my best attempt at setting the breakpoint and examining it in the
> > API test seems to produce the desired filename and line number (ie:
> > does not exhibit the bug), where interacting with lldb on the command
> > line does not render the file/line (ie: the bug).
> >
> > Jim, Pavel, anyone: Ideas on how I can/should test this behavior?
> >
> > $ ./bin/llvm-lit -v
> > tools/lldb/test/API/lang/c/stepping/TestStepAndBreakpoints.py
> > -- Testing: 1 tests, 1 workers --
> > FAIL: lldb-api :: lang/c/stepping/TestStepAndBreakpoints.py (1 of 1)
> >  TEST 'lldb-api ::
> > lang/c/stepping/TestStepAndBreakpoints.py' FAILED 
> > Script:
> > --
> > /usr/bin/python3.8
> > /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u
> > CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env
> > OBJCOPY=/usr/bin/objcopy 

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Jim Ingham via lldb-commits]

How were you setting the breakpoint in the case where it was failing from the 
command line?  The breakpoint you are examining in the test in your patches is 
a "source regular expression" breakpoint.  That would be "break set -p".  If 
you were also doing that in the command line and got a different result that 
would be very odd, since neither interface (CLI or SB) does much work before 
handing the request off to the common routine.  If (as I suspect) you were 
doing "b main.c:40" in the command line, then you should make a breakpoint with 
SBTarget.BreakpointCreateByLocation and see if that shows the same problem.  
That should do the same job as "break set --file --line".

Note, you can also run the command line breakpoint command in API tests.  
That's sometimes required, particularly if there's something about a 
combination of command flags that you want to test.  Just use the 
lldbutil.run_break_set_by_file_and_line utility.  That centralizes running and 
checking the basic results of the break set command, so if we do change the 
break command we only have to fix one place...  In your case the checks that 
run_break_set_by_file_and_line do should suffice, but the command returns the 
created breakpoint number, so if you want to poke at it further, you can use  
target.FindBreakpointByID to get the SBBreakpoint you just made.  There are 
also wrappers for the other lldb breakpoint types you can make from the command 
line, BTW.

Jim



> On Jan 14, 2021, at 1:38 PM, David Blaikie  wrote:
> 
> Thanks for the pointers!
> 
> So I tried writing an API-type test for this line table/breakpoint
> behavior (it doesn't seem like it's entirely/just the line table (for
> one thing, the DWARF change is purely in the subprogram DIE, not in
> the line table)) and I can't seem to figure it out.
> 
> The attached patch modifies lldb to undo this patch (to show the
> breakage) and modifies an existing API test to build with the clang
> flag to enable ranges on subprograms - I've confirmed the DWARF built
> by this test does have subprogram ranges.
> 
> But my best attempt at setting the breakpoint and examining it in the
> API test seems to produce the desired filename and line number (ie:
> does not exhibit the bug), where interacting with lldb on the command
> line does not render the file/line (ie: the bug).
> 
> Jim, Pavel, anyone: Ideas on how I can/should test this behavior?
> 
> $ ./bin/llvm-lit -v
> tools/lldb/test/API/lang/c/stepping/TestStepAndBreakpoints.py
> -- Testing: 1 tests, 1 workers --
> FAIL: lldb-api :: lang/c/stepping/TestStepAndBreakpoints.py (1 of 1)
>  TEST 'lldb-api ::
> lang/c/stepping/TestStepAndBreakpoints.py' FAILED 
> Script:
> --
> /usr/bin/python3.8
> /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u
> CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env
> OBJCOPY=/usr/bin/objcopy --env
> LLVM_LIBS_DIR=/usr/local/google/home/blaikie/dev/llvm/build/default/./lib
> --arch x86_64 --build-dir
> /usr/local/google/home/blaikie/dev/llvm/build/default/lldb-test-build.noindex
> --lldb-module-cache-dir
> /usr/local/google/home/blaikie/dev/llvm/build/default/lldb-test-build.noindex/module-cache-lldb/lldb-api
> --clang-module-cache-dir
> /usr/local/google/home/blaikie/dev/llvm/build/default/lldb-test-build.noindex/module-cache-clang/lldb-api
> --executable /usr/local/google/home/blaikie/dev/llvm/build/default/./bin/lldb
> --compiler /usr/local/google/home/blaikie/dev/llvm/build/default/./bin/clang
> --dsymutil 
> /usr/local/google/home/blaikie/dev/llvm/build/default/./bin/dsymutil
> --filecheck 
> /usr/local/google/home/blaikie/dev/llvm/build/default/./bin/FileCheck
> --yaml2obj 
> /usr/local/google/home/blaikie/dev/llvm/build/default/./bin/yaml2obj
> --lldb-libs-dir
> /usr/local/google/home/blaikie/dev/llvm/build/default/./lib
> /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/lang/c/stepping
> -p TestStepAndBreakpoints.py
> --
> Exit Code: 1
> 
> Command Output (stdout):
> --
> lldb version 12.0.0 (g...@github.com:llvm/llvm-project.git revision
> d49974f9c98ebce5a679eced9f27add138b881fa)
>  clang revision d49974f9c98ebce5a679eced9f27add138b881fa
>  llvm revision d49974f9c98ebce5a679eced9f27add138b881fa
> Libc++ tests will not be run because: Compiling with -stdlib=libc++
> fails with the error: :1:10: fatal error: 'algorithm' file not
> found
> #include 
> ^~~
> 1 error generated.
> 
> Skipping following debug info categories: ['dsym', 'gmodules']
> objc tests will be skipped because of unsupported platform
> 
> --
> Command Output (stderr):
> --
> FAIL: LLDB 
> (/usr/local/google/home/blaikie/dev/llvm/build/default/bin/clang-x86_64)
> :: test_and_python_api (TestStepAndBreakpoints.TestCStepping)
> ==
> FAIL: test_and_python_api (TestStepAndBreakpoints.TestCStepping)
>   Test stepping over vrs. hitting breakpoints & subsequent stepping
> in 

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via lldb-commits]

Thanks for the pointers!

So I tried writing an API-type test for this line table/breakpoint
behavior (it doesn't seem like it's entirely/just the line table (for
one thing, the DWARF change is purely in the subprogram DIE, not in
the line table)) and I can't seem to figure it out.

The attached patch modifies lldb to undo this patch (to show the
breakage) and modifies an existing API test to build with the clang
flag to enable ranges on subprograms - I've confirmed the DWARF built
by this test does have subprogram ranges.

But my best attempt at setting the breakpoint and examining it in the
API test seems to produce the desired filename and line number (ie:
does not exhibit the bug), where interacting with lldb on the command
line does not render the file/line (ie: the bug).

Jim, Pavel, anyone: Ideas on how I can/should test this behavior?

$ ./bin/llvm-lit -v
tools/lldb/test/API/lang/c/stepping/TestStepAndBreakpoints.py
-- Testing: 1 tests, 1 workers --
FAIL: lldb-api :: lang/c/stepping/TestStepAndBreakpoints.py (1 of 1)
 TEST 'lldb-api ::
lang/c/stepping/TestStepAndBreakpoints.py' FAILED 
Script:
--
/usr/bin/python3.8
/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u
CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env
OBJCOPY=/usr/bin/objcopy --env
LLVM_LIBS_DIR=/usr/local/google/home/blaikie/dev/llvm/build/default/./lib
--arch x86_64 --build-dir
/usr/local/google/home/blaikie/dev/llvm/build/default/lldb-test-build.noindex
--lldb-module-cache-dir
/usr/local/google/home/blaikie/dev/llvm/build/default/lldb-test-build.noindex/module-cache-lldb/lldb-api
--clang-module-cache-dir
/usr/local/google/home/blaikie/dev/llvm/build/default/lldb-test-build.noindex/module-cache-clang/lldb-api
--executable /usr/local/google/home/blaikie/dev/llvm/build/default/./bin/lldb
--compiler /usr/local/google/home/blaikie/dev/llvm/build/default/./bin/clang
--dsymutil /usr/local/google/home/blaikie/dev/llvm/build/default/./bin/dsymutil
--filecheck 
/usr/local/google/home/blaikie/dev/llvm/build/default/./bin/FileCheck
--yaml2obj /usr/local/google/home/blaikie/dev/llvm/build/default/./bin/yaml2obj
--lldb-libs-dir
/usr/local/google/home/blaikie/dev/llvm/build/default/./lib
/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/lang/c/stepping
-p TestStepAndBreakpoints.py
--
Exit Code: 1

Command Output (stdout):
--
lldb version 12.0.0 (g...@github.com:llvm/llvm-project.git revision
d49974f9c98ebce5a679eced9f27add138b881fa)
  clang revision d49974f9c98ebce5a679eced9f27add138b881fa
  llvm revision d49974f9c98ebce5a679eced9f27add138b881fa
Libc++ tests will not be run because: Compiling with -stdlib=libc++
fails with the error: :1:10: fatal error: 'algorithm' file not
found
#include 
 ^~~
1 error generated.

Skipping following debug info categories: ['dsym', 'gmodules']
objc tests will be skipped because of unsupported platform

--
Command Output (stderr):
--
FAIL: LLDB 
(/usr/local/google/home/blaikie/dev/llvm/build/default/bin/clang-x86_64)
:: test_and_python_api (TestStepAndBreakpoints.TestCStepping)
==
FAIL: test_and_python_api (TestStepAndBreakpoints.TestCStepping)
   Test stepping over vrs. hitting breakpoints & subsequent stepping
in various forms.
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/decorators.py",
line 345, in wrapper
return func(self, *args, **kwargs)
  File 
"/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/decorators.py",
line 135, in wrapper
return func(*args, **kwargs)
  File 
"/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/decorators.py",
line 135, in wrapper
return func(*args, **kwargs)
  File 
"/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/decorators.py",
line 105, in wrapper
func(*args, **kwargs)
  File 
"/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/decorators.py",
line 105, in wrapper
func(*args, **kwargs)
  File 
"/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/decorators.py",
line 105, in wrapper
func(*args, **kwargs)
  [Previous line repeated 1 more time]
  File 
"/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/lang/c/stepping/TestStepAndBreakpoints.py",
line 48, in test_and_python_api

self.assertEqual(get_description(break_1_in_main.locations[0].GetAddress().line_entry),
"narf")
AssertionError:
'/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/lang/c/stepping/main.c:40:14'
!= 'narf'
- 
/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/lang/c/stepping/main.c:40:14+
narf
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/default/bin/clang

[Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via Phabricator via lldb-commits]

dblaikie added a comment.

In D94063#2492450 , @DavidSpickett 
wrote:

> Do you have python enabled in the build? 
> (https://lldb.llvm.org/resources/build.html#preliminaries)
>
> The cmake option LLDB_ENABLE_PYTHON defaults to Auto which has tripped me up 
> in the past. If you set it to YES then you'll get a build error if it fails 
> to find a suitable Python instead of silently disabling it.

Thanks for the hint! Yep, seems I didn't have the python3-dev package installed 
(took a while to stare at the error messages to guess/understand that that's 
what I was missing)

That got a few of these tests running, but still a significant chunk are 
"unsupported" due to (well, at least some are due to this - I'm guessing quite 
a few are due to this):

  lldb version 12.0.0 (g...@github.com:llvm/llvm-project.git revision 
d49974f9c98ebce5a679eced9f27add138b881fa)
clang revision d49974f9c98ebce5a679eced9f27add138b881fa
llvm revision d49974f9c98ebce5a679eced9f27add138b881fa
  Libc++ tests will not be run because: Compiling with -stdlib=libc++ fails 
with the error: :1:10: fatal error: 'algorithm' file not found
  #include 
   ^~~
  1 error generated.
  
  Skipping following debug info categories: ['dsym', 'gmodules']

But I do have libc++ and libc++abi listed as projects to build in my cmake 
config - but I guess because the just-built clang and just-built libc++ aren't 
installed, clang can't find libc++ so these tests fail/get marked as 
unsupported.

But how do these tests work for other people? Do they ever run in a plain build 
like this? It seems like they should/that would be important.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D94063/new/

https://reviews.llvm.org/D94063

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

[Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Spickett via Phabricator via lldb-commits]

DavidSpickett added a comment.

Do you have python enabled in the build? 
(https://lldb.llvm.org/resources/build.html#preliminaries)

The cmake option LLDB_ENABLE_PYTHON defaults to Auto which has tripped me up in 
the past. If you set it to YES then you'll get a build error if it fails to 
find a suitable Python instead of silently disabling it.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D94063/new/

https://reviews.llvm.org/D94063

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Jim Ingham via lldb-commits]

> On Jan 10, 2021, at 5:37 PM, David Blaikie  wrote:
> 
> Thanks for all the context - so sounds like mostly based on (3) the 
> recommendation would be for this to be an API test (is there a way to test 
> the line table directly? good place for reference on the SB API options - I 
> looked at a few tests and they seemed quite different ( 
> lldb/test/API/functionalities/breakpoint/move_nearest/TestMoveNearest.py and 
> lldb/test/API/commands/breakpoint/set/func-regex/TestBreakpointRegexError.py 
> ) in the way they're written, so not sure what the norms are/how they work).
> 

You can get directly at the line table from the SB API's.  There's an example 
of doing just that on the SBCompileUnit page:

https://lldb.llvm.org/python_reference/lldb.SBCompileUnit-class.html

There's a small sample test in API/sample_test which is a good place to start.  
That one assumes you are running the target, however, which you may or may not 
need to do.  We've accumulated lots of convenience methods over time to make 
testing easier, but we haven't back ported them to old tests.  If you want 
other models than the sample_test, look for ones made more recently.

There's some description of the API tests here:

https://lldb.llvm.org/resources/test.html

and some test writing info in the sources in:

llvm-project/lldb/docs/testsuite/best-practices.txt


> But more fundamentally, seems all the API tests are "unsupported" on my 
> system, and I can't seem to figure out what makes them unsupported according 
> to lit. Any ideas?

I don't know much about how the lit runner works, somebody else will have to 
answer that.

Jim

> 
> On Thu, Jan 7, 2021 at 4:55 PM Jim Ingham  wrote:
> 
> 
> > On Jan 7, 2021, at 3:57 PM, David Blaikie  wrote:
> > 
> > On Thu, Jan 7, 2021 at 3:37 PM Jim Ingham via lldb-commits
> >  wrote:
> >> 
> >> 
> >> 
> >>> On Jan 7, 2021, at 2:29 PM, David Blaikie via Phabricator via 
> >>> lldb-commits  wrote:
> >>> 
> >>> dblaikie added a comment.
> >>> 
> >>> In D94063#2485271 , @labath 
> >>> wrote:
> >>> 
>  In D94063#2483546 , @dblaikie 
>  wrote:
>  
> > If it's better to write it using C++ source and custom clang flags I 
> > can do that instead (it'll be an -mllvm flag - looks like there's one 
> > other test that does that: 
> > `lldb/test/API/lang/objc/forward-decl/TestForwardDecl.py:
> > dict(CFLAGS_EXTRAS="-dwarf-version=5 -mllvm -accel-tables=Dwarf"))`) - 
> > means the test will be a bit more convoluted to tickle the subprogram 
> > ranges, but not much - just takes two functions+function-sections.
>  
>  I certainly wouldn't want to drop the existing test.
> >>> 
> >>> Ah, what's the tradeoff between the different test types here?
> >> 
> >> This is my take (this has been a contentious issue so I'm sure there are 
> >> other takes...):
> >> 
> >> The "Shell" tests use pattern matching against the lldb command line 
> >> output.  They are useful for testing the details of the command 
> >> interaction. You can also do that using pexpect in the API tests, but the 
> >> Python 2.7 version of pexpect seemed really flakey so we switched to shell 
> >> tests for this sort of thing.
> >> 
> >> Because you are matching against text output that isn't API, they are less 
> >> stable.  For instance if we changed anything in the "break set" output, 
> >> your test would fail(*).  And because you are picking details out of that 
> >> text, the tests are less precise.  You either have to match more of the 
> >> command line than you are actually testing for, which isn't a good 
> >> practice, or you run the risk of finding the text you were looking for in 
> >> a directory path or other unrelated part of the output.  Also they are 
> >> harder to debug if you can't reproduce the failure locally, since it isn't 
> >> easy to add internal checks/output to the test to try hypotheses.  
> >> Whenever I have run into failures of this sort the first thing I do is 
> >> convert the test to an API test...
> >> 
> >> But the main benefit of the "Shell" tests is that you can write tests 
> >> without having to know Python or learn the lldb Python API's.  And if you 
> >> are coming from clang you already know how FileCheck tests work, so that's 
> >> a bonus.  I think it's legit to require that folks actually working on 
> >> lldb learn the SB API's.  But we were persuaded that it wasn't fair to 
> >> impose that on people not working on lldb, and yet such folks do sometimes 
> >> need to write tests for lldb...  So for simple tests, the Shell tests are 
> >> an okay option.  But really, there's nothing you can do in a Shell test 
> >> that you can't do in an API test.
> >> 
> >> The "API" tests use the Python SB API's - though they also have the 
> >> ability to run commands and do expect type checks on the output so for 
> >> single commands they work much as the shell 

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via lldb-commits]

Thanks for all the context - so sounds like mostly based on (3) the
recommendation would be for this to be an API test (is there a way to test
the line table directly? good place for reference on the SB API options - I
looked at a few tests and they seemed quite different
( lldb/test/API/functionalities/breakpoint/move_nearest/TestMoveNearest.py
and lldb/test/API/commands/breakpoint/set/func-regex/TestBreakpointRegexError.py
) in the way they're written, so not sure what the norms are/how they work).

But more fundamentally, seems all the API tests are "unsupported" on my
system, and I can't seem to figure out what makes them unsupported
according to lit. Any ideas?

On Thu, Jan 7, 2021 at 4:55 PM Jim Ingham  wrote:

>
>
> > On Jan 7, 2021, at 3:57 PM, David Blaikie  wrote:
> >
> > On Thu, Jan 7, 2021 at 3:37 PM Jim Ingham via lldb-commits
> >  wrote:
> >>
> >>
> >>
> >>> On Jan 7, 2021, at 2:29 PM, David Blaikie via Phabricator via
> lldb-commits  wrote:
> >>>
> >>> dblaikie added a comment.
> >>>
> >>> In D94063#2485271 , @labath
> wrote:
> >>>
>  In D94063#2483546 ,
> @dblaikie wrote:
> 
> > If it's better to write it using C++ source and custom clang flags I
> can do that instead (it'll be an -mllvm flag - looks like there's one other
> test that does that:
> `lldb/test/API/lang/objc/forward-decl/TestForwardDecl.py:
> dict(CFLAGS_EXTRAS="-dwarf-version=5 -mllvm -accel-tables=Dwarf"))`) -
> means the test will be a bit more convoluted to tickle the subprogram
> ranges, but not much - just takes two functions+function-sections.
> 
>  I certainly wouldn't want to drop the existing test.
> >>>
> >>> Ah, what's the tradeoff between the different test types here?
> >>
> >> This is my take (this has been a contentious issue so I'm sure there
> are other takes...):
> >>
> >> The "Shell" tests use pattern matching against the lldb command line
> output.  They are useful for testing the details of the command
> interaction. You can also do that using pexpect in the API tests, but the
> Python 2.7 version of pexpect seemed really flakey so we switched to shell
> tests for this sort of thing.
> >>
> >> Because you are matching against text output that isn't API, they are
> less stable.  For instance if we changed anything in the "break set"
> output, your test would fail(*).  And because you are picking details out
> of that text, the tests are less precise.  You either have to match more of
> the command line than you are actually testing for, which isn't a good
> practice, or you run the risk of finding the text you were looking for in a
> directory path or other unrelated part of the output.  Also they are harder
> to debug if you can't reproduce the failure locally, since it isn't easy to
> add internal checks/output to the test to try hypotheses.  Whenever I have
> run into failures of this sort the first thing I do is convert the test to
> an API test...
> >>
> >> But the main benefit of the "Shell" tests is that you can write tests
> without having to know Python or learn the lldb Python API's.  And if you
> are coming from clang you already know how FileCheck tests work, so that's
> a bonus.  I think it's legit to require that folks actually working on lldb
> learn the SB API's.  But we were persuaded that it wasn't fair to impose
> that on people not working on lldb, and yet such folks do sometimes need to
> write tests for lldb...  So for simple tests, the Shell tests are an okay
> option.  But really, there's nothing you can do in a Shell test that you
> can't do in an API test.
> >>
> >> The "API" tests use the Python SB API's - though they also have the
> ability to run commands and do expect type checks on the output so for
> single commands they work much as the shell tests do (there's even a
> FileCheck style assert IIRC).  They are a little more verbose than shell
> tests (though we've reduced the boilerplate significantly over the years).
> And of course you have to know the SB API's.  But for instance, if you
> wanted to know that a breakpoint was set on line 5 of foo.c, you can set
> the breakpoint, then ask the resultant SBBreakpoint object what it's file &
> line numbers were directly.  So once you've gotten familiar with the setup,
> IMO you can write much higher quality tests with the API tests.
> >>
> >>
> >> Jim
> >>
> >> (*) I am personally not at all in favor of the Shell tests, but that's
> in part because back in the day I was asked to make a simple and useful
> change to the output of the gdb "break" command but then righting the gdb
> testsuite - which is all based on expecting the results of various gdb
> commands - was so tedious that we ended up dropping the change instead.  I
> don't want to get to that place with lldb, but the hope is that as long as
> we mostly write API tests, we can avoid encumbering the current command
> outputs too heavily...
> >
> >
> > Thanks for the 

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Jim Ingham via lldb-commits]

> On Jan 7, 2021, at 3:57 PM, David Blaikie  wrote:
> 
> On Thu, Jan 7, 2021 at 3:37 PM Jim Ingham via lldb-commits
>  wrote:
>> 
>> 
>> 
>>> On Jan 7, 2021, at 2:29 PM, David Blaikie via Phabricator via lldb-commits 
>>>  wrote:
>>> 
>>> dblaikie added a comment.
>>> 
>>> In D94063#2485271 , @labath wrote:
>>> 
 In D94063#2483546 , @dblaikie 
 wrote:
 
> If it's better to write it using C++ source and custom clang flags I can 
> do that instead (it'll be an -mllvm flag - looks like there's one other 
> test that does that: 
> `lldb/test/API/lang/objc/forward-decl/TestForwardDecl.py:
> dict(CFLAGS_EXTRAS="-dwarf-version=5 -mllvm -accel-tables=Dwarf"))`) - 
> means the test will be a bit more convoluted to tickle the subprogram 
> ranges, but not much - just takes two functions+function-sections.
 
 I certainly wouldn't want to drop the existing test.
>>> 
>>> Ah, what's the tradeoff between the different test types here?
>> 
>> This is my take (this has been a contentious issue so I'm sure there are 
>> other takes...):
>> 
>> The "Shell" tests use pattern matching against the lldb command line output. 
>>  They are useful for testing the details of the command interaction. You can 
>> also do that using pexpect in the API tests, but the Python 2.7 version of 
>> pexpect seemed really flakey so we switched to shell tests for this sort of 
>> thing.
>> 
>> Because you are matching against text output that isn't API, they are less 
>> stable.  For instance if we changed anything in the "break set" output, your 
>> test would fail(*).  And because you are picking details out of that text, 
>> the tests are less precise.  You either have to match more of the command 
>> line than you are actually testing for, which isn't a good practice, or you 
>> run the risk of finding the text you were looking for in a directory path or 
>> other unrelated part of the output.  Also they are harder to debug if you 
>> can't reproduce the failure locally, since it isn't easy to add internal 
>> checks/output to the test to try hypotheses.  Whenever I have run into 
>> failures of this sort the first thing I do is convert the test to an API 
>> test...
>> 
>> But the main benefit of the "Shell" tests is that you can write tests 
>> without having to know Python or learn the lldb Python API's.  And if you 
>> are coming from clang you already know how FileCheck tests work, so that's a 
>> bonus.  I think it's legit to require that folks actually working on lldb 
>> learn the SB API's.  But we were persuaded that it wasn't fair to impose 
>> that on people not working on lldb, and yet such folks do sometimes need to 
>> write tests for lldb...  So for simple tests, the Shell tests are an okay 
>> option.  But really, there's nothing you can do in a Shell test that you 
>> can't do in an API test.
>> 
>> The "API" tests use the Python SB API's - though they also have the ability 
>> to run commands and do expect type checks on the output so for single 
>> commands they work much as the shell tests do (there's even a FileCheck 
>> style assert IIRC).  They are a little more verbose than shell tests (though 
>> we've reduced the boilerplate significantly over the years).  And of course 
>> you have to know the SB API's.  But for instance, if you wanted to know that 
>> a breakpoint was set on line 5 of foo.c, you can set the breakpoint, then 
>> ask the resultant SBBreakpoint object what it's file & line numbers were 
>> directly.  So once you've gotten familiar with the setup, IMO you can write 
>> much higher quality tests with the API tests.
>> 
>> 
>> Jim
>> 
>> (*) I am personally not at all in favor of the Shell tests, but that's in 
>> part because back in the day I was asked to make a simple and useful change 
>> to the output of the gdb "break" command but then righting the gdb testsuite 
>> - which is all based on expecting the results of various gdb commands - was 
>> so tedious that we ended up dropping the change instead.  I don't want to 
>> get to that place with lldb, but the hope is that as long as we mostly write 
>> API tests, we can avoid encumbering the current command outputs too 
>> heavily...
> 
> 
> Thanks for the context, I really appreciate it.
> 
> The aspect I was especially curious about here was less in regards to
> the mechanics of how the behavior is examined/tested (between shell
> and SB API) but more in regards to source code versus assembly - where
> the assembly can more explicitly target some DWARF feature, but isn't
> especially portable - whereas the source code could be portable to
> test on different architectures, but might require either more
> contortions to reliably produce the desired DWARF, or possibly extra
> compiler flags (that was especialyl of interest since Pavel mentioned
> these tests could be portable across compilers, so how would 

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via lldb-commits]

On Thu, Jan 7, 2021 at 3:37 PM Jim Ingham via lldb-commits
 wrote:
>
>
>
> > On Jan 7, 2021, at 2:29 PM, David Blaikie via Phabricator via lldb-commits 
> >  wrote:
> >
> > dblaikie added a comment.
> >
> > In D94063#2485271 , @labath wrote:
> >
> >> In D94063#2483546 , @dblaikie 
> >> wrote:
> >>
> >>> If it's better to write it using C++ source and custom clang flags I can 
> >>> do that instead (it'll be an -mllvm flag - looks like there's one other 
> >>> test that does that: 
> >>> `lldb/test/API/lang/objc/forward-decl/TestForwardDecl.py:
> >>> dict(CFLAGS_EXTRAS="-dwarf-version=5 -mllvm -accel-tables=Dwarf"))`) - 
> >>> means the test will be a bit more convoluted to tickle the subprogram 
> >>> ranges, but not much - just takes two functions+function-sections.
> >>
> >> I certainly wouldn't want to drop the existing test.
> >
> > Ah, what's the tradeoff between the different test types here?
>
> This is my take (this has been a contentious issue so I'm sure there are 
> other takes...):
>
> The "Shell" tests use pattern matching against the lldb command line output.  
> They are useful for testing the details of the command interaction.  You can 
> also do that using pexpect in the API tests, but the Python 2.7 version of 
> pexpect seemed really flakey so we switched to shell tests for this sort of 
> thing.
>
> Because you are matching against text output that isn't API, they are less 
> stable.  For instance if we changed anything in the "break set" output, your 
> test would fail(*).  And because you are picking details out of that text, 
> the tests are less precise.  You either have to match more of the command 
> line than you are actually testing for, which isn't a good practice, or you 
> run the risk of finding the text you were looking for in a directory path or 
> other unrelated part of the output.  Also they are harder to debug if you 
> can't reproduce the failure locally, since it isn't easy to add internal 
> checks/output to the test to try hypotheses.  Whenever I have run into 
> failures of this sort the first thing I do is convert the test to an API 
> test...
>
> But the main benefit of the "Shell" tests is that you can write tests without 
> having to know Python or learn the lldb Python API's.  And if you are coming 
> from clang you already know how FileCheck tests work, so that's a bonus.  I 
> think it's legit to require that folks actually working on lldb learn the SB 
> API's.  But we were persuaded that it wasn't fair to impose that on people 
> not working on lldb, and yet such folks do sometimes need to write tests for 
> lldb...  So for simple tests, the Shell tests are an okay option.  But 
> really, there's nothing you can do in a Shell test that you can't do in an 
> API test.
>
> The "API" tests use the Python SB API's - though they also have the ability 
> to run commands and do expect type checks on the output so for single 
> commands they work much as the shell tests do (there's even a FileCheck style 
> assert IIRC).  They are a little more verbose than shell tests (though we've 
> reduced the boilerplate significantly over the years).  And of course you 
> have to know the SB API's.  But for instance, if you wanted to know that a 
> breakpoint was set on line 5 of foo.c, you can set the breakpoint, then ask 
> the resultant SBBreakpoint object what it's file & line numbers were 
> directly.  So once you've gotten familiar with the setup, IMO you can write 
> much higher quality tests with the API tests.
>
>
> Jim
>
> (*) I am personally not at all in favor of the Shell tests, but that's in 
> part because back in the day I was asked to make a simple and useful change 
> to the output of the gdb "break" command but then righting the gdb testsuite 
> - which is all based on expecting the results of various gdb commands - was 
> so tedious that we ended up dropping the change instead.  I don't want to get 
> to that place with lldb, but the hope is that as long as we mostly write API 
> tests, we can avoid encumbering the current command outputs too heavily...


Thanks for the context, I really appreciate it.

The aspect I was especially curious about here was less in regards to
the mechanics of how the behavior is examined/tested (between shell
and SB API) but more in regards to source code versus assembly - where
the assembly can more explicitly target some DWARF feature, but isn't
especially portable - whereas the source code could be portable to
test on different architectures, but might require either more
contortions to reliably produce the desired DWARF, or possibly extra
compiler flags (that was especialyl of interest since Pavel mentioned
these tests could be portable across compilers, so how would I specify
any necessary custom flags to get clang to produce the desired DWARF
(& the tests would be fairly useless for other compilers without those

Re: [Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Jim Ingham via lldb-commits]

> On Jan 7, 2021, at 2:29 PM, David Blaikie via Phabricator via lldb-commits 
>  wrote:
> 
> dblaikie added a comment.
> 
> In D94063#2485271 , @labath wrote:
> 
>> In D94063#2483546 , @dblaikie wrote:
>> 
>>> If it's better to write it using C++ source and custom clang flags I can do 
>>> that instead (it'll be an -mllvm flag - looks like there's one other test 
>>> that does that: `lldb/test/API/lang/objc/forward-decl/TestForwardDecl.py:   
>>>  dict(CFLAGS_EXTRAS="-dwarf-version=5 -mllvm 
>>> -accel-tables=Dwarf"))`) - means the test will be a bit more convoluted to 
>>> tickle the subprogram ranges, but not much - just takes two 
>>> functions+function-sections.
>> 
>> I certainly wouldn't want to drop the existing test.
> 
> Ah, what's the tradeoff between the different test types here?

This is my take (this has been a contentious issue so I'm sure there are other 
takes...):

The "Shell" tests use pattern matching against the lldb command line output.  
They are useful for testing the details of the command interaction.  You can 
also do that using pexpect in the API tests, but the Python 2.7 version of 
pexpect seemed really flakey so we switched to shell tests for this sort of 
thing.

Because you are matching against text output that isn't API, they are less 
stable.  For instance if we changed anything in the "break set" output, your 
test would fail(*).  And because you are picking details out of that text, the 
tests are less precise.  You either have to match more of the command line than 
you are actually testing for, which isn't a good practice, or you run the risk 
of finding the text you were looking for in a directory path or other unrelated 
part of the output.  Also they are harder to debug if you can't reproduce the 
failure locally, since it isn't easy to add internal checks/output to the test 
to try hypotheses.  Whenever I have run into failures of this sort the first 
thing I do is convert the test to an API test...

But the main benefit of the "Shell" tests is that you can write tests without 
having to know Python or learn the lldb Python API's.  And if you are coming 
from clang you already know how FileCheck tests work, so that's a bonus.  I 
think it's legit to require that folks actually working on lldb learn the SB 
API's.  But we were persuaded that it wasn't fair to impose that on people not 
working on lldb, and yet such folks do sometimes need to write tests for 
lldb...  So for simple tests, the Shell tests are an okay option.  But really, 
there's nothing you can do in a Shell test that you can't do in an API test.

The "API" tests use the Python SB API's - though they also have the ability to 
run commands and do expect type checks on the output so for single commands 
they work much as the shell tests do (there's even a FileCheck style assert 
IIRC).  They are a little more verbose than shell tests (though we've reduced 
the boilerplate significantly over the years).  And of course you have to know 
the SB API's.  But for instance, if you wanted to know that a breakpoint was 
set on line 5 of foo.c, you can set the breakpoint, then ask the resultant 
SBBreakpoint object what it's file & line numbers were directly.  So once 
you've gotten familiar with the setup, IMO you can write much higher quality 
tests with the API tests.


Jim

(*) I am personally not at all in favor of the Shell tests, but that's in part 
because back in the day I was asked to make a simple and useful change to the 
output of the gdb "break" command but then righting the gdb testsuite - which 
is all based on expecting the results of various gdb commands - was so tedious 
that we ended up dropping the change instead.  I don't want to get to that 
place with lldb, but the hope is that as long as we mostly write API tests, we 
can avoid encumbering the current command outputs too heavily...

Jim

> 
>> However, it could be useful to have c++ test too. This one could feature a 
>> more complicated executable, and be more open-ended/exploratory and test 
>> end-to-end functionality (including compiler integration), instead of a 
>> targeted "did we parse DW_AT_ranges correctly" regression test. Then this 
>> test could go into the `API` test category, as we have the ability to run 
>> those kinds of tests against different compilers.
> 
> Does this include support for custom compiler flags (it'd currently take a 
> non-official/internal-only llvm flag to create the DW_AT_ranges on a 
> subprogram that I have in mind, for instance)?
> 
>> However, all of that is strictly optional.
> 
> I'll consider it for a separate commit.
> 
> 
> Repository:
>  rG LLVM Github Monorepo
> 
> CHANGES SINCE LAST ACTION
>  https://reviews.llvm.org/D94063/new/
> 
> https://reviews.llvm.org/D94063
> 
> ___
> lldb-commits mailing list
> lldb-commits@lists.llvm.org
> 

[Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via Phabricator via lldb-commits]

dblaikie added a comment.

In D94063#2485271 , @labath wrote:

> In D94063#2483546 , @dblaikie wrote:
>
>> If it's better to write it using C++ source and custom clang flags I can do 
>> that instead (it'll be an -mllvm flag - looks like there's one other test 
>> that does that: `lldb/test/API/lang/objc/forward-decl/TestForwardDecl.py:
>> dict(CFLAGS_EXTRAS="-dwarf-version=5 -mllvm -accel-tables=Dwarf"))`) 
>> - means the test will be a bit more convoluted to tickle the subprogram 
>> ranges, but not much - just takes two functions+function-sections.
>
> I certainly wouldn't want to drop the existing test.

Ah, what's the tradeoff between the different test types here?

> However, it could be useful to have c++ test too. This one could feature a 
> more complicated executable, and be more open-ended/exploratory and test 
> end-to-end functionality (including compiler integration), instead of a 
> targeted "did we parse DW_AT_ranges correctly" regression test. Then this 
> test could go into the `API` test category, as we have the ability to run 
> those kinds of tests against different compilers.

Does this include support for custom compiler flags (it'd currently take a 
non-official/internal-only llvm flag to create the DW_AT_ranges on a subprogram 
that I have in mind, for instance)?

> However, all of that is strictly optional.

I'll consider it for a separate commit.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D94063/new/

https://reviews.llvm.org/D94063

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

[Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via Phabricator via lldb-commits]

This revision was landed with ongoing or failed builds.
This revision was automatically updated to reflect the committed changes.
Closed by commit rG274afac9a17f: lldb: Add support for DW_AT_ranges on 
DW_TAG_subprograms (authored by dblaikie).

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D94063/new/

https://reviews.llvm.org/D94063

Files:
  lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.cpp
  lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.h
  lldb/test/Shell/SymbolFile/DWARF/Inputs/subprogram_ranges.s
  lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test

Index: lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test
===
--- /dev/null
+++ lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test
@@ -0,0 +1,17 @@
+# REQUIRES: x86
+# RUN: %clang -target x86_64-pc-linux -g -O0 %S/Inputs/subprogram_ranges.s -o %t.out
+# RUN: %lldb -b -s %s %t.out 2>&1 | FileCheck %s
+
+# Test breaking on symbols and printing variables when a DW_TAG_subprogram uses
+# DW_AT_ranges instead of DW_AT_low_pc/DW_AT_high_pc.  While the assembly here
+# is a bit unrealistic - it's a single-entry range using DWARFv4 which isn't
+# useful for anything (a single-entry range with DWARFv5 can reduce address
+# relocations, and multi-entry ranges can be used for function sections), but
+# it's the simplest thing to test. If anyone's updating this test at some
+# point, feel free to replace it with another equivalent test if it's
+# especially useful, but don't dismiss it as pointless just because it's a bit
+# weird.
+
+b main
+# CHECK: (lldb) b main
+# CHECK-NEXT: Breakpoint 1: where = subprogram_ranges.test.tmp.out`main + 6 at main.c:2:7
Index: lldb/test/Shell/SymbolFile/DWARF/Inputs/subprogram_ranges.s
===
--- /dev/null
+++ lldb/test/Shell/SymbolFile/DWARF/Inputs/subprogram_ranges.s
@@ -0,0 +1,159 @@
+	.text
+	.file	"main.c"
+	.globl	main# -- Begin function main
+	.p2align	4, 0x90
+	.type	main,@function
+main:   # @main
+.Lfunc_begin0:
+	.file	1 "/usr/local/google/home/blaikie/dev/scratch" "main.c"
+	.loc	1 1 0   # main.c:1:0
+	.cfi_startproc
+# %bb.0:# %entry
+	pushq	%rbp
+	.cfi_def_cfa_offset 16
+	.cfi_offset %rbp, -16
+	movq	%rsp, %rbp
+	.cfi_def_cfa_register %rbp
+	xorl	%eax, %eax
+.Ltmp0:
+	.loc	1 2 7 prologue_end  # main.c:2:7
+	movl	$3, -4(%rbp)
+	.loc	1 3 1   # main.c:3:1
+	popq	%rbp
+	.cfi_def_cfa %rsp, 8
+	retq
+.Ltmp1:
+.Lfunc_end0:
+	.size	main, .Lfunc_end0-main
+	.cfi_endproc
+# -- End function
+	.section	.debug_abbrev,"",@progbits
+	.byte	1   # Abbreviation Code
+	.byte	17  # DW_TAG_compile_unit
+	.byte	1   # DW_CHILDREN_yes
+	.byte	37  # DW_AT_producer
+	.byte	14  # DW_FORM_strp
+	.byte	19  # DW_AT_language
+	.byte	5   # DW_FORM_data2
+	.byte	3   # DW_AT_name
+	.byte	14  # DW_FORM_strp
+	.byte	16  # DW_AT_stmt_list
+	.byte	23  # DW_FORM_sec_offset
+	.byte	27  # DW_AT_comp_dir
+	.byte	14  # DW_FORM_strp
+	.byte	17  # DW_AT_low_pc
+	.byte	1   # DW_FORM_addr
+	.byte	18  # DW_AT_high_pc
+	.byte	6   # DW_FORM_data4
+	.byte	0   # EOM(1)
+	.byte	0   # EOM(2)
+	.byte	2   # Abbreviation Code
+	.byte	46  # DW_TAG_subprogram
+	.byte	1   # DW_CHILDREN_yes
+	.byte	85  # DW_AT_ranges
+	.byte	23  # DW_FORM_sec_offset
+	.byte	64  # DW_AT_frame_base
+	.byte	24  # DW_FORM_exprloc
+	.byte	3   # DW_AT_name
+	.byte	14  # DW_FORM_strp
+	.byte	58  # DW_AT_decl_file
+	.byte	11  # DW_FORM_data1
+	.byte	59  # DW_AT_decl_line
+	.byte	11  # DW_FORM_data1
+	.byte	73  # DW_AT_type
+	.byte	19  # DW_FORM_ref4
+	.byte	63  # DW_AT_external
+	.byte	25  # DW_FORM_flag_present
+	.byte	0   # EOM(1)
+	.byte	0   # EOM(2)
+	.byte	3

[Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Pavel Labath via Phabricator via lldb-commits]

labath accepted this revision.
labath added a comment.
This revision is now accepted and ready to land.

In D94063#2483546 , @dblaikie wrote:

> If it's better to write it using C++ source and custom clang flags I can do 
> that instead (it'll be an -mllvm flag - looks like there's one other test 
> that does that: `lldb/test/API/lang/objc/forward-decl/TestForwardDecl.py: 
>dict(CFLAGS_EXTRAS="-dwarf-version=5 -mllvm -accel-tables=Dwarf"))`) - 
> means the test will be a bit more convoluted to tickle the subprogram ranges, 
> but not much - just takes two functions+function-sections.

I certainly wouldn't want to drop the existing test. However, it could be 
useful to have c++ test too. This one could feature a more complicated 
executable, and be more open-ended/exploratory and test end-to-end 
functionality (including compiler integration), instead of a targeted "did we 
parse DW_AT_ranges correctly" regression test. Then this test could go into the 
`API` test category, as we have the ability to run those kinds of tests against 
different compilers.

However, all of that is strictly optional.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D94063/new/

https://reviews.llvm.org/D94063

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

[Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via Phabricator via lldb-commits]

dblaikie added a comment.

In D94063#2481867 , @labath wrote:

> The fix is good, but the test could be improved.

Yeah - haven't written lldb patches before so totally open to suggestions on 
the testing front for sure. Thanks!

> Combining assembly input with running the inferior effectively limits the 
> test to a single platform (assembly is not portable, and running requires asm 
> to match the host).

If it's better to write it using C++ source and custom clang flags I can do 
that instead (it'll be an -mllvm flag - looks like there's one other test that 
does that: `lldb/test/API/lang/objc/forward-decl/TestForwardDecl.py:
dict(CFLAGS_EXTRAS="-dwarf-version=5 -mllvm -accel-tables=Dwarf"))`) - means 
the test will be a bit more convoluted to tickle the subprogram ranges, but not 
much - just takes two functions+function-sections.

> AFAICT, we don't actually need to run the binary to test this fix -- checking 
> just the "breakpoint set" command should suffice (if you want to be more 
> explicit in what is being checked, you can also run "breakpoint list -v" and 
> test that). Then you'd just need to replace `%clang_host` with `%clang 
> -target x86_64-pc-linux` (or something) and `UNSUPPORTED: system-windows` 
> with `REQUIRES: x86`.

Yeah, I was hoping to setup the test for testing both these things in a uniform 
way, but if best practice is to test them with different mechanisms I'm OK with 
that.

> I'll write about variable printing in the other review.

Thanks!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D94063/new/

https://reviews.llvm.org/D94063

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

[Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via Phabricator via lldb-commits]

dblaikie updated this revision to Diff 315035.
dblaikie added a comment.

Update test to avoid running the program


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D94063/new/

https://reviews.llvm.org/D94063

Files:
  lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.cpp
  lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.h
  lldb/test/Shell/SymbolFile/DWARF/Inputs/subprogram_ranges.s
  lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test

Index: lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test
===
--- /dev/null
+++ lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test
@@ -0,0 +1,17 @@
+# REQUIRES: x86
+# RUN: %clang -target x86_64-pc-linux -g -O0 %S/Inputs/subprogram_ranges.s -o %t.out
+# RUN: %lldb -b -s %s %t.out 2>&1 | FileCheck %s
+
+# Test breaking on symbols and printing variables when a DW_TAG_subprogram uses
+# DW_AT_ranges instead of DW_AT_low_pc/DW_AT_high_pc.  While the assembly here
+# is a bit unrealistic - it's a single-entry range using DWARFv4 which isn't
+# useful for anything (a single-entry range with DWARFv5 can reduce address
+# relocations, and multi-entry ranges can be used for function sections), but
+# it's the simplest thing to test. If anyone's updating this test at some
+# point, feel free to replace it with another equivalent test if it's
+# especially useful, but don't dismiss it as pointless just because it's a bit
+# weird.
+
+b main
+# CHECK: (lldb) b main
+# CHECK-NEXT: Breakpoint 1: where = subprogram_ranges.test.tmp.out`main + 6 at main.c:2:7
Index: lldb/test/Shell/SymbolFile/DWARF/Inputs/subprogram_ranges.s
===
--- /dev/null
+++ lldb/test/Shell/SymbolFile/DWARF/Inputs/subprogram_ranges.s
@@ -0,0 +1,159 @@
+	.text
+	.file	"main.c"
+	.globl	main# -- Begin function main
+	.p2align	4, 0x90
+	.type	main,@function
+main:   # @main
+.Lfunc_begin0:
+	.file	1 "/usr/local/google/home/blaikie/dev/scratch" "main.c"
+	.loc	1 1 0   # main.c:1:0
+	.cfi_startproc
+# %bb.0:# %entry
+	pushq	%rbp
+	.cfi_def_cfa_offset 16
+	.cfi_offset %rbp, -16
+	movq	%rsp, %rbp
+	.cfi_def_cfa_register %rbp
+	xorl	%eax, %eax
+.Ltmp0:
+	.loc	1 2 7 prologue_end  # main.c:2:7
+	movl	$3, -4(%rbp)
+	.loc	1 3 1   # main.c:3:1
+	popq	%rbp
+	.cfi_def_cfa %rsp, 8
+	retq
+.Ltmp1:
+.Lfunc_end0:
+	.size	main, .Lfunc_end0-main
+	.cfi_endproc
+# -- End function
+	.section	.debug_abbrev,"",@progbits
+	.byte	1   # Abbreviation Code
+	.byte	17  # DW_TAG_compile_unit
+	.byte	1   # DW_CHILDREN_yes
+	.byte	37  # DW_AT_producer
+	.byte	14  # DW_FORM_strp
+	.byte	19  # DW_AT_language
+	.byte	5   # DW_FORM_data2
+	.byte	3   # DW_AT_name
+	.byte	14  # DW_FORM_strp
+	.byte	16  # DW_AT_stmt_list
+	.byte	23  # DW_FORM_sec_offset
+	.byte	27  # DW_AT_comp_dir
+	.byte	14  # DW_FORM_strp
+	.byte	17  # DW_AT_low_pc
+	.byte	1   # DW_FORM_addr
+	.byte	18  # DW_AT_high_pc
+	.byte	6   # DW_FORM_data4
+	.byte	0   # EOM(1)
+	.byte	0   # EOM(2)
+	.byte	2   # Abbreviation Code
+	.byte	46  # DW_TAG_subprogram
+	.byte	1   # DW_CHILDREN_yes
+	.byte	85  # DW_AT_ranges
+	.byte	23  # DW_FORM_sec_offset
+	.byte	64  # DW_AT_frame_base
+	.byte	24  # DW_FORM_exprloc
+	.byte	3   # DW_AT_name
+	.byte	14  # DW_FORM_strp
+	.byte	58  # DW_AT_decl_file
+	.byte	11  # DW_FORM_data1
+	.byte	59  # DW_AT_decl_line
+	.byte	11  # DW_FORM_data1
+	.byte	73  # DW_AT_type
+	.byte	19  # DW_FORM_ref4
+	.byte	63  # DW_AT_external
+	.byte	25  # DW_FORM_flag_present
+	.byte	0   # EOM(1)
+	.byte	0   # EOM(2)
+	.byte	3   # Abbreviation Code
+	.byte	52  # DW_TAG_variable
+	.byte	0   

[Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[Pavel Labath via Phabricator via lldb-commits]

labath added a comment.

The fix is good, but the test could be improved. Combining assembly input with 
running the inferior effectively limits the test to a single platform (assembly 
is not portable, and running requires asm to match the host). AFAICT, we don't 
actually need to run the binary to test this fix -- checking just the 
"breakpoint set" command should suffice (if you want to be more explicit in 
what is being checked, you can also run "breakpoint list -v" and test that). 
Then you'd just need to replace `%clang_host` with `%clang -target 
x86_64-pc-linux` (or something) and `UNSUPPORTED: system-windows` with 
`REQUIRES: x86`.

I'll write about variable printing in the other review.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D94063/new/

https://reviews.llvm.org/D94063

___
lldb-commits mailing list
lldb-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits

[Lldb-commits] [PATCH] D94063: lldb: Add support for DW_AT_ranges on DW_TAG_subprograms

[David Blaikie via Phabricator via lldb-commits]

dblaikie created this revision.
dblaikie added a reviewer: labath.
dblaikie requested review of this revision.
Herald added a reviewer: jdoerfert.
Herald added subscribers: lldb-commits, sstefan1.
Herald added a project: LLDB.

gcc already produces debug info with this form
-freorder-block-and-partition
clang produces this sort of thing with -fbasic-block-sections and with a
coming-soon tweak to use ranges in DWARFv5 where they can allow greater
reuse of debug_addr than the low/high_pc forms.

This fixes the case of breaking on a function name, but leaves broken
printing a variable - a follow-up commit will add that and improve the
test case to match.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D94063

Files:
  lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.cpp
  lldb/source/Plugins/SymbolFile/DWARF/DWARFDebugInfoEntry.h
  lldb/test/Shell/SymbolFile/DWARF/Inputs/subprogram_ranges.s
  lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test

Index: lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test
===
--- /dev/null
+++ lldb/test/Shell/SymbolFile/DWARF/subprogram_ranges.test
@@ -0,0 +1,29 @@
+# UNSUPPORTED: system-windows
+# RUN: %clang_host -g -O0 %S/Inputs/subprogram_ranges.s -o %t.out
+# RUN: not %lldb -b -s %s %t.out 2>&1 | FileCheck %s
+
+# Test breaking on symbols and printing variables when a DW_TAG_subprogram uses
+# DW_AT_ranges instead of DW_AT_low_pc/DW_AT_high_pc.  While the assembly here
+# is a bit unrealistic - it's a single-entry range using DWARFv4 which isn't
+# useful for anything (a single-entry range with DWARFv5 can reduce address
+# relocations, and multi-entry ranges can be used for function sections), but
+# it's the simplest thing to test. If anyone's updating this test at some
+# point, feel free to replace it with another equivalent test if it's
+# especially useful, but don't dismiss it as pointless just because it's a bit
+# weird.
+
+b main
+# CHECK: (lldb) b main
+# CHECK-NEXT: Breakpoint 1: where = subprogram_ranges.test.tmp.out`main + 6 at main.c:2:7
+
+run
+# Stopping inside of the stop hook range
+# CHECK: (lldb) run
+
+print var
+# Check that local variable names can be looked up
+# FIXME: This should be: (int) $0 = {{.*}}
+# CHECK: (lldb) print var
+# CHECK-NEXT: error: :1:1: use of undeclared identifier 'var'
+
+q
Index: lldb/test/Shell/SymbolFile/DWARF/Inputs/subprogram_ranges.s
===
--- /dev/null
+++ lldb/test/Shell/SymbolFile/DWARF/Inputs/subprogram_ranges.s
@@ -0,0 +1,159 @@
+	.text
+	.file	"main.c"
+	.globl	main# -- Begin function main
+	.p2align	4, 0x90
+	.type	main,@function
+main:   # @main
+.Lfunc_begin0:
+	.file	1 "/usr/local/google/home/blaikie/dev/scratch" "main.c"
+	.loc	1 1 0   # main.c:1:0
+	.cfi_startproc
+# %bb.0:# %entry
+	pushq	%rbp
+	.cfi_def_cfa_offset 16
+	.cfi_offset %rbp, -16
+	movq	%rsp, %rbp
+	.cfi_def_cfa_register %rbp
+	xorl	%eax, %eax
+.Ltmp0:
+	.loc	1 2 7 prologue_end  # main.c:2:7
+	movl	$3, -4(%rbp)
+	.loc	1 3 1   # main.c:3:1
+	popq	%rbp
+	.cfi_def_cfa %rsp, 8
+	retq
+.Ltmp1:
+.Lfunc_end0:
+	.size	main, .Lfunc_end0-main
+	.cfi_endproc
+# -- End function
+	.section	.debug_abbrev,"",@progbits
+	.byte	1   # Abbreviation Code
+	.byte	17  # DW_TAG_compile_unit
+	.byte	1   # DW_CHILDREN_yes
+	.byte	37  # DW_AT_producer
+	.byte	14  # DW_FORM_strp
+	.byte	19  # DW_AT_language
+	.byte	5   # DW_FORM_data2
+	.byte	3   # DW_AT_name
+	.byte	14  # DW_FORM_strp
+	.byte	16  # DW_AT_stmt_list
+	.byte	23  # DW_FORM_sec_offset
+	.byte	27  # DW_AT_comp_dir
+	.byte	14  # DW_FORM_strp
+	.byte	17  # DW_AT_low_pc
+	.byte	1   # DW_FORM_addr
+	.byte	18  # DW_AT_high_pc
+	.byte	6   # DW_FORM_data4
+	.byte	0   # EOM(1)
+	.byte	0   # EOM(2)
+	.byte	2   # Abbreviation Code
+	.byte	46  # DW_TAG_subprogram
+	.byte	1   # DW_CHILDREN_yes
+	.byte	85  # DW_AT_ranges
+	.byte	23  # DW_FORM_sec_offset
+	.byte	64  # DW_AT_frame_base
+	.byte	24  # DW_FORM_exprloc
+	.byte	3   #