Re: [lldb-dev] Testing through api vs. commands

2015-10-07 Thread Todd Fiala via lldb-dev
Hey Zachary, > What are the chances of someone attempting to get the existing unit test runner working in the Xcode build? Or at least attempting to and seeing if there's any major blockers that prevent it from working? I fully intend to do that. I need to do a few more pieces of infrastructura

Re: [lldb-dev] Testing through api vs. commands

2015-10-07 Thread Zachary Turner via lldb-dev
What are the chances of someone attempting to get the existing unit test runner working in the Xcode build? Or at least attempting to and seeing if there's any major blockers that prevent it from working? On Wed, Oct 7, 2015 at 11:54 AM Jim Ingham wrote: > > > On Oct 7, 2015, at 11:40 AM, Zacha

Re: [lldb-dev] Testing through api vs. commands

2015-10-07 Thread Jim Ingham via lldb-dev
> On Oct 7, 2015, at 11:40 AM, Zachary Turner wrote: > > > > On Wed, Oct 7, 2015 at 11:26 AM Jim Ingham wrote: > > > On Oct 7, 2015, at 11:16 AM, Jim Ingham wrote: > > > >> > >> On Oct 7, 2015, at 10:37 AM, Zachary Turner via lldb-dev > >> wrote: > >> > >> > >> > >> On Wed, Oct 7, 2015 at

Re: [lldb-dev] Testing through api vs. commands

2015-10-07 Thread Zachary Turner via lldb-dev
On Wed, Oct 7, 2015 at 11:26 AM Jim Ingham wrote: > > > On Oct 7, 2015, at 11:16 AM, Jim Ingham wrote: > > > >> > >> On Oct 7, 2015, at 10:37 AM, Zachary Turner via lldb-dev < > lldb-dev@lists.llvm.org> wrote: > >> > >> > >> > >> On Wed, Oct 7, 2015 at 10:17 AM Greg Clayton > wrote: > >> > >> >

Re: [lldb-dev] Testing through api vs. commands

2015-10-07 Thread Jim Ingham via lldb-dev
> On Oct 7, 2015, at 11:16 AM, Jim Ingham wrote: > >> >> On Oct 7, 2015, at 10:37 AM, Zachary Turner via lldb-dev >> wrote: >> >> >> >> On Wed, Oct 7, 2015 at 10:17 AM Greg Clayton wrote: >> >> >>> On Oct 7, 2015, at 10:05 AM, Zachary Turner via lldb-dev >>> wrote: >>> >>> Jim, Greg,

Re: [lldb-dev] Testing through api vs. commands

2015-10-07 Thread Jim Ingham via lldb-dev
> On Oct 7, 2015, at 10:37 AM, Zachary Turner via lldb-dev > wrote: > > > > On Wed, Oct 7, 2015 at 10:17 AM Greg Clayton wrote: > > > > On Oct 7, 2015, at 10:05 AM, Zachary Turner via lldb-dev > > wrote: > > > > Jim, Greg, > > > > Can I get some feedback on this? I would like to start e

Re: [lldb-dev] Testing through api vs. commands

2015-10-07 Thread Greg Clayton via lldb-dev
> > So in summary, it sounds like we agree on the following guidelines: > > 1) If you're committing a CL and it is possible to test it through the SB > API, you should only submit an SB API test, and not a HandleCommand test. agreed > 2) If you're committing a CL and it's not possible to test

Re: [lldb-dev] Testing through api vs. commands

2015-10-07 Thread Zachary Turner via lldb-dev
On Wed, Oct 7, 2015 at 10:37 AM Zachary Turner wrote: > One more question: I mentioned earlier that we should enforce the > distinction between HandleCommand tests and python api tests at an > organizational level. In other words, all HandleCommand tests go in > lldb/test/command-api, and all ne

Re: [lldb-dev] Testing through api vs. commands

2015-10-07 Thread Zachary Turner via lldb-dev
On Wed, Oct 7, 2015 at 10:17 AM Greg Clayton wrote: > > > > On Oct 7, 2015, at 10:05 AM, Zachary Turner via lldb-dev < > lldb-dev@lists.llvm.org> wrote: > > > > Jim, Greg, > > > > Can I get some feedback on this? I would like to start enforcing this > moving forward. I want to make sure we're i

Re: [lldb-dev] Testing through api vs. commands

2015-10-07 Thread Greg Clayton via lldb-dev
> On Oct 7, 2015, at 10:05 AM, Zachary Turner via lldb-dev > wrote: > > Jim, Greg, > > Can I get some feedback on this? I would like to start enforcing this moving > forward. I want to make sure we're in agreement. > > On Mon, Oct 5, 2015 at 12:30 PM Todd Fiala wrote: > IMHO that all sou

Re: [lldb-dev] Testing through api vs. commands

2015-10-07 Thread Zachary Turner via lldb-dev
Jim, Greg, Can I get some feedback on this? I would like to start enforcing this moving forward. I want to make sure we're in agreement. On Mon, Oct 5, 2015 at 12:30 PM Todd Fiala wrote: > IMHO that all sounds reasonable. > > FWIW - I wrote some tests for the test system changes I put in (for

Re: [lldb-dev] Testing through api vs. commands

2015-10-05 Thread Todd Fiala via lldb-dev
> I didn't follow those threads, can you summarize what the race condition was? Is it likely to crop up in the normal test suite? i.e. not the test suite that tests the test suite. Sure. At least on OS X and Linux, the impl of subprocess.Popen.wait() and subprocess.Popen.poll() are written such

Re: [lldb-dev] Testing through api vs. commands

2015-10-05 Thread Zachary Turner via lldb-dev
I didn't follow those threads, can you summarize what the race condition was? Is it likely to crop up in the normal test suite? i.e. not the test suite that tests the test suite. On Mon, Oct 5, 2015 at 12:30 PM Todd Fiala wrote: > IMHO that all sounds reasonable. > > FWIW - I wrote some tests

Re: [lldb-dev] Testing through api vs. commands

2015-10-05 Thread Todd Fiala via lldb-dev
IMHO that all sounds reasonable. FWIW - I wrote some tests for the test system changes I put in (for the pure-python impl of timeout support), and in the process, I discovered a race condition in using a python facility that there really is no way I would have found anywhere near as reasonably wit

Re: [lldb-dev] Testing through api vs. commands

2015-10-05 Thread Zachary Turner via lldb-dev
On Fri, Sep 11, 2015 at 11:42 AM Jim Ingham wrote: > I have held from the beginning that the only tests that should be written > using HandleCommand are those that explicitly test command behavior, and if > it is possible to write a test using the SB API you should always do it > that way for the

Re: [lldb-dev] Testing through api vs. commands

2015-09-17 Thread Zachary Turner via lldb-dev
On Thu, Sep 17, 2015 at 10:39 AM wrote: > On Thu, Sep 17, 2015 at 05:20:14PM +, Zachary Turner wrote: > > > > a) they should be explicitly marked as interface tests. > > > What's the decorator for this? > > > > There's not one currently. > > Will there be? > Whenever someone adds one :) If y

Re: [lldb-dev] Testing through api vs. commands

2015-09-17 Thread via lldb-dev
On Thu, Sep 17, 2015 at 05:20:14PM +, Zachary Turner wrote: > > > a) they should be explicitly marked as interface tests. > > What's the decorator for this? > > There's not one currently. Will there be? > > > d) Results of these interface tests should also not be *verified* by the > > > use

Re: [lldb-dev] Testing through api vs. commands

2015-09-17 Thread Zachary Turner via lldb-dev
On Thu, Sep 17, 2015 at 10:31 AM wrote: > On Tue, Sep 15, 2015 at 04:46:41PM -0700, Jim Ingham wrote: > > > In any case, there's a lot I never could figure out how to do in the SB > > > API that I could only do via commands. For example, how do you test > > > that a trailing space at the end of

Re: [lldb-dev] Testing through api vs. commands

2015-09-17 Thread via lldb-dev
On Tue, Sep 15, 2015 at 04:46:41PM -0700, Jim Ingham wrote: > > In any case, there's a lot I never could figure out how to do in the SB > > API that I could only do via commands. For example, how do you test > > that a trailing space at the end of the expr --language option's argument > > is trimm

Re: [lldb-dev] Testing through api vs. commands

2015-09-17 Thread Zachary Turner via lldb-dev
On Thu, Sep 17, 2015 at 10:08 AM wrote: > On Tue, Sep 15, 2015 at 11:36:00PM +, Zachary Turner wrote: > > a) they should be explicitly marked as interface tests. > > What's the decorator for this? > There's not one currently. > > > d) Results of these interface tests should also not be *ver

Re: [lldb-dev] Testing through api vs. commands

2015-09-17 Thread via lldb-dev
On Tue, Sep 15, 2015 at 11:36:00PM +, Zachary Turner wrote: > a) they should be explicitly marked as interface tests. What's the decorator for this? > d) Results of these interface tests should also not be *verified* by the > use of self.expect, but itself through the API. (Relying on the te

Re: [lldb-dev] Testing through api vs. commands

2015-09-15 Thread Jim Ingham via lldb-dev
> On Sep 15, 2015, at 4:23 PM, d...@burble.org wrote: > > I do still think we need some tests that verify commands run, but I think > those tests should focus not on doing complicated interactions with the > debugger, and instead just verifying that things parse correctly and the >

Re: [lldb-dev] Testing through api vs. commands

2015-09-15 Thread Zachary Turner via lldb-dev
I agree that we should test the command interface, but a) they should be explicitly marked as interface tests. b) There should be MUCH fewer. c) It should only verify that typing a particular command maps to the right core sequence of public / private API calls. Not that the debugger functionalit

Re: [lldb-dev] Testing through api vs. commands

2015-09-15 Thread via lldb-dev
> > > > I do still think we need some tests that verify commands run, but I > > > > think those tests should focus not on doing complicated interactions > > > > with the debugger, and instead just verifying that things parse > > > > correctly and the command is configured correctly, with the und

Re: [lldb-dev] Testing through api vs. commands

2015-09-11 Thread Jim Ingham via lldb-dev
> On Sep 11, 2015, at 12:49 PM, Zachary Turner wrote: > > Sounds good. I think perhaps one reason a lot of tests are written using the > commands is because not all functionality available through commands is > available through the SB API. Adrian is working on creating some tests for > cor

Re: [lldb-dev] Testing through api vs. commands

2015-09-11 Thread Zachary Turner via lldb-dev
Sounds good. I think perhaps one reason a lot of tests are written using the commands is because not all functionality available through commands is available through the SB API. Adrian is working on creating some tests for core dump debugging on Windows right now, and there's no way to access th

Re: [lldb-dev] Testing through api vs. commands

2015-09-11 Thread Jim Ingham via lldb-dev
> On Sep 11, 2015, at 11:47 AM, Zachary Turner wrote: > > We'll probably rewrite tests that we find are failing specifically as a > result of issues like this, but I agree it's not worth re-writing everything > else except on an as-needed basis. > > To make the distinction explicit and enforc

Re: [lldb-dev] Testing through api vs. commands

2015-09-11 Thread Jim Ingham via lldb-dev
> On Sep 11, 2015, at 11:47 AM, Zachary Turner wrote: > > We'll probably rewrite tests that we find are failing specifically as a > result of issues like this, but I agree it's not worth re-writing everything > else except on an as-needed basis. > > To make the distinction explicit and enforc

Re: [lldb-dev] Testing through api vs. commands

2015-09-11 Thread Zachary Turner via lldb-dev
We'll probably rewrite tests that we find are failing specifically as a result of issues like this, but I agree it's not worth re-writing everything else except on an as-needed basis. To make the distinction explicit and enforce it kind of at an organnizational level, would it be worth creating fo

Re: [lldb-dev] Testing through api vs. commands

2015-09-11 Thread Jim Ingham via lldb-dev
I have held from the beginning that the only tests that should be written using HandleCommand are those that explicitly test command behavior, and if it is possible to write a test using the SB API you should always do it that way for the very reasons you cite. Not everybody agreed with me at f

[lldb-dev] Testing through api vs. commands

2015-09-11 Thread Zachary Turner via lldb-dev
The past few weeks I've spent a lot of time xfailing the rest of the failing tests on windows so we can enable tests to run on the bots. One thing I ran into more frequently than I would have liked is that the tests were failing not because the functionality is broken, but because the substrings b