09.08.2016, 18:42, "Zachary Turner via lldb-dev" <lldb-dev@lists.llvm.org>:
> There are a lot of reasons for the lack of tests. Off the top of my head,
> two of the biggest ones are:
>
> 1) Some areas of LLDB have been historically hard to test. The unwinder and
> core dumps come to mind. You can't really just check in an 800MB core dump
> into the repo.
Side note: core dumps are often higly compressible, so they may have quite
reasonable size e.g. in .xz format
> 2) Tests are very heavyweight. You have to write a Makefile. You have to
> write a Python script that uses the SB API. you have to write a C program.
> Then you have to figure out the right incantation of dotest.py to run your
> test. Once you've done this many times it becomes easier, but the barrier to
> entry is high.
>
> #1 can be improved through the use of unit tests and fuzzing. Sure, it's
> hard to generate a full blown executable that contains every edge case the
> unwinder might ever experience and then have it try to unwind through there.
> Especially considering that the unwinder behaves differently on every
> platform. But it's much more manageable to write a unit test that constructs
> a particular sequence of bytes in memory, passes it to the unwinder, and
> checks the return value of some function that is supposed to handle that.
> It's still not entirely simple, since the unwinder is partially heuristic,
> but at least it's more manageable.
>
> There are also ways we can write IR by hand and have llc generate some byte
> code for us and then pass that to those same functions. Again, it's not like
> we can just start doing this overnight, but there are ways. The question is
> just how serious of an effort are we prepared to make and how much time are
> we prepared to put into making this testable versus implementing new
> features, fixing bugs, etc.
>
> #2 could potentially be improved by lit style tests. As I mentioned in my
> last post, think lldbinline style tests. Not appropriate for everything, but
> certainly for a lot of things. If you only had to have one file which is a C
> program with some annotations in the file, that is a much lower barrier to
> entry.
>
> Again, the real question is just how much effort are we actually prepared to
> put into this? I'd love it if there were entire days or weeks that were just
> testing weeks, where all we did was add new tests (or refactor code to make
> it more testable) and people didn't work on anything else. I've been
> inactive for a while because I've had to prioritize work on some things in
> LLVM, but I could make time for something like that.
>
> On Mon, Aug 8, 2016 at 6:12 PM Vedant Kumar via lldb-dev
> <lldb-dev@lists.llvm.org> wrote:
>> FWIW, as a happy lldb user, it's exciting to read about these planned
>> developments.
>>
>> I have two follow-up questions about the section on 'Testing Strategy
>> Evaluation': (1) what is lldb's policy on including test updates with bug fix
>> commits and functional changes, and (2) how is this policy enforced?
>>
>> AFAICT, it seems that lldb is not as strict about its test policy as other
>> llvm
>> sub-projects. That could be to its detriment. Here are some very rough
>> numbers
>> on the number of commits which include test updates [1]:
>>
>> - lldb: 287 of the past 1000 commits
>> - llvm: 511 of the past 1000 commits
>> - clang: 622 of the past 1000 commits
>> - compiler-rt: 543 of the past 1000 commits
>>
>> NFC commits make these numbers a bit noisy. But, unless lldb has a much
>> higher
>> ratio of NFC commits to functional changes as compared to other llvm
>> sub-projects, this is a concerning statistic.
>>
>> best,
>> vedant
>>
>> [1] Based on ToT = r278069.
>>
>> HAS_TEST=0
>> TOTAL=1000
>> for HASH in $(git log --oneline -n$TOTAL | cut -d' ' -f1); do
>> git show --stat $HASH | grep "|" | grep -q test && HAS_TEST=$((HAS_TEST+1))
>> done
>> echo $HAS_TEST "/" $TOTAL
>>
>>> On Aug 8, 2016, at 2:57 PM, Zachary Turner via lldb-dev
>>> <lldb-dev@lists.llvm.org> wrote:
>>>
>>>
>>>
>>> On Mon, Aug 8, 2016 at 2:40 PM Kate Stone via lldb-dev
>>> <lldb-dev@lists.llvm.org> wrote:
>>> LLDB has come a long way since the project was first announced. As a
>>> robust debugger for C-family languages and Swift, LLDB is constantly in use
>>> by millions of developers. It has also become a foundation for bringing up
>>> debugger support for other languages like Go and RenderScript. In addition
>>> to the original macOS implementation the Linux LLDB port is in active use
>>> and Windows support has made significant strides. IDE and editor
>>> integration via both SB APIs and MI have made LLDB available to even more
>>> users. It’s definitely a project every contributor can be proud of and I’d
>>> like to take a moment to thank everyone who has been involved in one way or
>>> another.
>>>
>>> It’s also a project that shows some signs of strain due to its rapid
>>> growth. We’ve accumulated some technical debt that must be paid off, and
>>> in general it seems like a good time to reflect on where we'll be headed
>>> next. We’ve outlined a few goals for discussion below as well as one more
>>> short-term action. Discussion is very much encouraged.
>>>
>>> Forward-Looking Goals
>>>
>>> 1. Testing Strategy Evaluation
>>>
>>> Keeping our code base healthy is next to impossible without a robust
>>> testing strategy. Our existing suite of tests is straightforward to run
>>> locally, and serves as a foundation for continuous integration. That said,
>>> it is definitely not exhaustive. Obvious priorities for improvement
>>> include gathering coverage information, investing in more conventional unit
>>> tests in addition to the suite of end-to-end tests, and introducing tests
>>> in code bases where we depend on debugger-specific behavior (e.g.: for
>>> expression evaluation.)
>>> I know this is going to be controversial, but I think we should at least do
>>> a serious evaluation of whether using the lit infrastructure would work for
>>> LLDB. Conventional wisdom is that it won't work because LLDB tests are
>>> fundamentally different than LLVM tests. I actually completely agree with
>>> the latter part. They are fundamentally different.
>>>
>>> However, we've seen some effort to move towards lldb inline tests, and in a
>>> sense that's conceptually exactly what lit tests are. My concern is that
>>> nobody with experience working on LLDB has a sufficient understanding of
>>> what lit is capable of to really figure this out.
>>>
>>> I know when I mentioned this some months ago Jonathan Roelofs chimed in and
>>> said that he believes lit is extensible enough to support LLDB's use case.
>>> The argument -- if I remember it correctly -- is that the traditional view
>>> of what a lit test (i.e. a sequence of commands that checks the output of a
>>> program against expected output) is one particular implementation of a
>>> lit-style test. But that you can make your own which do whatever you want.
>>>
>>> This would not just be busy work either. I think everyone involved with
>>> LLDB has experienced flakiness in the test suite. Sometimes it's flakiness
>>> in LLDB itself, but sometimes it is flakiness in the test infrastructure.
>>> It would be nice to completely eliminate one source of flakiness.
>>>
>>> I think it would be worth having some LLDB experts sit down in person with
>>> some lit experts and brainstorm ways to make LLDB use lit.
>>>
>>> Certainly it's worth a serious look, even if nothing comes of it.
>>>
>>>
>>> 4. Good Citizenship in the LLVM Community
>>>
>>> Last, but definitely not least, LLDB should endeavor to be a good citizen
>>> of the LLVM community. We should encourage developers to think of the
>>> technology stack as a coherent effort, where common code should be
>>> introduced at an appropriate level in the stack. Opportunities to factor
>>> reusable aspects of the LLDB code base up the stack into LLVM will be
>>> pursued.
>>>
>>> One arbitrary source of inconsistency at present is LLDB’s coding standard.
>>> That brings us to…
>>>
>>> Near-Term Goal: Standardizing on LLVM-style clang-format Rules
>>>
>>> We’ve heard from several in the community that would prefer to have a
>>> single code formatting style to further unify the two code bases. Using
>>> clang-format with the default LLVM conventions would simplify code
>>> migration, editor configuration, and coding habits for developers who work
>>> in multiple LLVM projects. There are non-trivial implications to
>>> reformatting a code base with this much history. It can obfuscate history
>>> and impact downstream projects by complicating merges. Ideally, it should
>>> be done once with as much advance notice as is practical. Here’s the
>>> timeline we’re proposing:
>>>
>>> Today - mechanical reformatting proposed, comment period begins
>>>
>>> To get a preview of what straightforward reformatting of the code looks
>>> like, just follow these steps to get a clean copy of the repository and
>>> reformat it:
>>> • Check out a clean copy of the existing repository
>>> • Edit .clang-format in the root of the tree, remove all but the line
>>>“BasedOnStyle: LLVM”
>>> • Change your current working directory to the root of the tree to
>>>reformat
>>> • Double-check to make sure you did step 3 ;-)
>>> • Run the following shell command: "find . -name "*.[c,cpp,h] -exec
>>>clang-format -i {} +"
>>> Very excited about this one, personally. While I have my share of qualms
>>> with LLVM's style, the benefit of having consistency is hard to overstate.
>>> It greatly reduces the effort to switch between codebases, a direct
>>> consequence of which is that it encourages people with LLVM expertise to
>>> jump into the LLDB codebase, which hopefully can help to tear down the
>>> invisible wall between the two.
>>>
>>> As a personal aside, this allows me to go back to my normal workflow of
>>> having 3 edit source files opened simultaneously and tiled horizontally,
>>> which is very nice.
>>>
>>> _______________________________________________
>>> lldb-dev mailing list
>>> lldb-dev@lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> ,
>
> _______________________________________________
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
--
Regards,
Konstantin
_______________________________________________
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
