https://github.com/SLTozer updated https://github.com/llvm/llvm-project/pull/204365
>From 85af6cd7cc83bcdfc66d32b7b891e6c55156d48a Mon Sep 17 00:00:00 2001 From: Stephen Tozer <[email protected]> Date: Tue, 16 Jun 2026 17:16:21 +0100 Subject: [PATCH 1/2] [Dexter] Document the structured script model This patch adds documentation for the script model to the Dexter README, shunting heuristic-mode information into a separate doc, creating a new doc for script-mode, and linking to both (with a brief summary of the differences) from the base README. --- .../debuginfo-tests/dexter/Heuristic.md | 231 +++++++++++++++++ .../debuginfo-tests/dexter/README.md | 238 +----------------- .../debuginfo-tests/dexter/Script.md | 213 ++++++++++++++++ 3 files changed, 450 insertions(+), 232 deletions(-) create mode 100644 cross-project-tests/debuginfo-tests/dexter/Heuristic.md create mode 100644 cross-project-tests/debuginfo-tests/dexter/Script.md diff --git a/cross-project-tests/debuginfo-tests/dexter/Heuristic.md b/cross-project-tests/debuginfo-tests/dexter/Heuristic.md new file mode 100644 index 0000000000000..a78c433747fb8 --- /dev/null +++ b/cross-project-tests/debuginfo-tests/dexter/Heuristic.md @@ -0,0 +1,231 @@ +# Dexter Heuristic Testing + +The old test model for Dexter is based on a debug experience "heuristic", combining a variety of different measures produced by "commands" into a single score, ranging from 0.0 (the lowest score) to 1.0 (a perfect score). The contribution from each command and each kind of failure towards this overall score is determine by a set of weights, which are assigned default values but can be given explicit overrides via command line. The set of commands and their behaviour are documented in [Commands.md](./Commands.md). + +## Running a test case + +The following commands build fibonacci.cpp from the tests/nostdlib directory and run it in LLDB, reporting the debug experience heuristic. The first pair of commands build with no optimizations (-O0) and score 1.0000. The second pair of commands build with optimizations (-O2) and score 0.2832 which suggests a worse debugging experience. + + clang -O0 -g tests/nostdlib/fibonacci.cpp -o tests/nostdlib/fibonacci/test + dexter.py test --binary tests/nostdlib/fibonacci/test --debugger lldb -- tests/nostdlib/fibonacci/test.cpp + test.cpp = (1.0000) + + clang -O2 -g tests/nostdlib/fibonacci/test.cpp -o tests/nostdlib/fibonacci/test + dexter.py test --binary tests/nostdlib/fibonacci/test --debugger lldb -- tests/nostdlib/fibonacci/test.cpp + test.cpp = (0.2832) + +## An example test case + +The sample test case (tests/nostdlib/fibonacci) looks like this: + + 1. #ifdef _MSC_VER + 2. # define DEX_NOINLINE __declspec(noinline) + 3. #else + 4. # define DEX_NOINLINE __attribute__((__noinline__)) + 5. #endif + 6. + 7. DEX_NOINLINE + 8. void Fibonacci(int terms, int& total) + 9. { + 0. int first = 0; + 11. int second = 1; + 12. for (int i = 0; i < terms; ++i) + 13. { + 14. int next = first + second; // DexLabel('start') + 15. total += first; + 16. first = second; + 17. second = next; // DexLabel('end') + 18. } + 19. } + 20. + 21. int main() + 22. { + 23. int total = 0; + 24. Fibonacci(5, total); + 25. return total; + 26. } + 27. + 28. /* + 29. DexExpectWatchValue('i', '0', '1', '2', '3', '4', + 30. from_line='start', to_line='end') + 31. DexExpectWatchValue('first', '0', '1', '2', '3', '5', + 32. from_line='start', to_line='end') + 33. DexExpectWatchValue('second', '1', '2', '3', '5', + 34 from_line='start', to_line='end') + 35. DexExpectWatchValue('total', '0', '1', '2', '4', '7', + 36. from_line='start', to_line='end') + 37. DexExpectWatchValue('next', '1', '2', '3', '5', '8', + 38. from_line='start', to_line='end') + 39. DexExpectWatchValue('total', '7', on_line=25) + 40. DexExpectStepKind('FUNC_EXTERNAL', 0) + 41. */ + +[DexLabel][1] is used to give a name to a line number. + +The [DexExpectWatchValue][2] command states that an expression, e.g. `i`, should +have particular values, `'0', '1', '2', '3','4'`, sequentially over the program +lifetime on particular lines. You can refer to a named line or simply the line +number (See line 39). + +At the end of the test is the following line: + + DexExpectStepKind('FUNC_EXTERNAL', 0) + +This [DexExpectStepKind][3] command indicates that we do not expect the debugger +to step into a file outside of the test directory. + +[1]: Commands.md#DexLabel +[2]: Commands.md#DexExpectWatchValue +[3]: Commands.md#DexExpectStepKind + +## Detailed DExTer reports + +Running the command below launches the tests/nostdlib/fibonacci test case in DExTer, using LLDB as the debugger and producing a detailed report: + + $ dexter.py test --vs-solution clang-cl_vs2015 --debugger vs2017 --cflags="/Ox /Zi" --ldflags="/Zi" -v -- tests/nostdlib/fibonacci + +The detailed report is enabled by `-v` and shows a breakdown of the information from each debugger step. For example: + + fibonacci = (0.2832) + + ## BEGIN ## + [1, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 23, 1, "BREAKPOINT", "FUNC", {}] + [2, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 24, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}] + [3, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 25, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}] + . [4, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "FUNC", {}] + . [5, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] + . [6, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] + . [7, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] + . [8, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] + . [9, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "0"}] + . [10, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] + . [11, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] + . [12, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "1"}] + . [13, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] + . [14, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] + . [15, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "2", "total": "0", "first": "1"}] + . [16, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] + . [17, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] + . [18, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "3", "total": "0", "first": "2"}] + . [19, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] + . [20, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] + . [21, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "5", "total": "0", "first": "3"}] + . [22, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] + . [23, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] + . [24, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 20, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}] + [25, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 26, 1, "BREAKPOINT", "FUNC", {"total": "7"}] + ## END (25 steps) ## + + + step kind differences [0/1] + FUNC_EXTERNAL: + 0 + + test.cpp:15-18 [first] [9/21] + expected encountered values: + 0 + 1 + 2 + 3 + + missing values: + 5 [-6] + + result optimized away: + step 5 (Variable is optimized away and not available.) [-3] + step 7 (Variable is optimized away and not available.) + step 8 (Variable is optimized away and not available.) + step 11 (Variable is optimized away and not available.) + step 14 (Variable is optimized away and not available.) + step 17 (Variable is optimized away and not available.) + step 20 (Variable is optimized away and not available.) + step 23 (Variable is optimized away and not available.) + + test.cpp:15-18 [i] [15/21] + result optimized away: + step 5 (Variable is optimized away and not available.) [-3] + step 7 (Variable is optimized away and not available.) [-3] + step 8 (Variable is optimized away and not available.) [-3] + step 9 (Variable is optimized away and not available.) [-3] + step 11 (Variable is optimized away and not available.) [-3] + step 12 (Variable is optimized away and not available.) + step 14 (Variable is optimized away and not available.) + step 15 (Variable is optimized away and not available.) + step 17 (Variable is optimized away and not available.) + step 18 (Variable is optimized away and not available.) + step 20 (Variable is optimized away and not available.) + step 21 (Variable is optimized away and not available.) + step 23 (Variable is optimized away and not available.) + + test.cpp:15-18 [second] [21/21] + expected encountered values: + 1 + 2 + 3 + 5 + + result optimized away: + step 5 (Variable is optimized away and not available.) [-3] + step 7 (Variable is optimized away and not available.) [-3] + step 8 (Variable is optimized away and not available.) [-3] + step 11 (Variable is optimized away and not available.) [-3] + step 14 (Variable is optimized away and not available.) [-3] + step 17 (Variable is optimized away and not available.) [-3] + step 20 (Variable is optimized away and not available.) [-3] + step 23 (Variable is optimized away and not available.) + + test.cpp:15-18 [total] [21/21] + expected encountered values: + 0 + + missing values: + 1 [-6] + 2 [-6] + 4 [-6] + 7 [-3] + + test.cpp:16-18 [next] [15/21] + result optimized away: + step 5 (Variable is optimized away and not available.) [-3] + step 8 (Variable is optimized away and not available.) [-3] + step 11 (Variable is optimized away and not available.) [-3] + step 14 (Variable is optimized away and not available.) [-3] + step 17 (Variable is optimized away and not available.) [-3] + step 20 (Variable is optimized away and not available.) + step 23 (Variable is optimized away and not available.) + + test.cpp:26 [total] [0/7] + expected encountered values: + 7 + +The first line + + fibonacci = (0.2832) + +shows a score of 0.2832 suggesting that unexpected behavior has been seen. This score is on scale of 0.0000 to 1.000, with 0.000 being the worst score possible and 1.000 being the best score possible. The verbose output shows the reason for any scoring. For example: + + test.cpp:15-18 [first] [9/21] + expected encountered values: + 0 + 1 + 2 + 3 + + missing values: + 5 [-6] + + result optimized away: + step 5 (Variable is optimized away and not available.) [-3] + step 7 (Variable is optimized away and not available.) + step 8 (Variable is optimized away and not available.) + step 11 (Variable is optimized away and not available.) + step 14 (Variable is optimized away and not available.) + step 17 (Variable is optimized away and not available.) + step 20 (Variable is optimized away and not available.) + step 23 (Variable is optimized away and not available.) + +shows that for `first` the expected values 0, 1, 2 and 3 were seen, 5 was not. On some steps the variable was reported as being optimized away. + +## Writing new test cases + +Each test can be either embedded within the source file using comments or included as a separate file with the .dex extension. Dexter does not include support for building test cases, although if a Visual Studio Solution (.sln) is used as the test file, VS will build the program as part of launching a debugger session if it has not already been built. diff --git a/cross-project-tests/debuginfo-tests/dexter/README.md b/cross-project-tests/debuginfo-tests/dexter/README.md index 99aa6fb601da7..fed3efe6db4dc 100644 --- a/cross-project-tests/debuginfo-tests/dexter/README.md +++ b/cross-project-tests/debuginfo-tests/dexter/README.md @@ -6,7 +6,7 @@ DExTer is a suite of tools used to evaluate the "User Debugging Experience". DEx ## Supported Debuggers -DExTer currently supports the Visual Studio 2015 and Visual Studio 2017 debuggers via the [DTE interface](https://docs.microsoft.com/en-us/dotnet/api/envdte.dte), and LLDB via its [Python interface](https://lldb.llvm.org/python-reference.html). GDB is not currently supported. +DExTer currently supports the Visual Studio 2015 and Visual Studio 2017 debuggers via the [DTE interface](https://docs.microsoft.com/en-us/dotnet/api/envdte.dte), and LLDB via its [Python interface](https://lldb.llvm.org/python-reference.html) and its [DAP interface](https://lldb.llvm.org/use/lldbdap.html). GDB is not currently supported. The following command evaluates your environment, listing the available and compatible debuggers: @@ -26,236 +26,10 @@ This is required to access the DTE interface for the Visual Studio debuggers. <python-executable> -m pip install pywin32 -### clang +## Usage -DExTer is current compatible with 'clang' and 'clang-cl' compiler drivers. The compiler must be available for DExTer, for example the following command should successfully build a runnable executable. +Dexter has two distinct usage modes: "heuristic" mode and "script" mode. The heuristic mode is the old/original mode for Dexter, using a set of declarative commands to control the debug session, fetch information, and produce a single debug experience heuristic score as the final output. The script mode is the new/current mode used by Dexter, which uses a structured YAML script to control the debug session and fetch information, and produces as its output a list of metrics which collectively describe the debug experience. - <compiler-executable> tests/nostdlib/fibonacci/test.cpp - -## Running a test case - -The following commands build fibonacci.cpp from the tests/nostdlib directory and run it in LLDB, reporting the debug experience heuristic. The first pair of commands build with no optimizations (-O0) and score 1.0000. The second pair of commands build with optimizations (-O2) and score 0.2832 which suggests a worse debugging experience. - - clang -O0 -g tests/nostdlib/fibonacci.cpp -o tests/nostdlib/fibonacci/test - dexter.py test --binary tests/nostdlib/fibonacci/test --debugger lldb -- tests/nostdlib/fibonacci/test.cpp - test.cpp = (1.0000) - - clang -O2 -g tests/nostdlib/fibonacci/test.cpp -o tests/nostdlib/fibonacci/test - dexter.py test --binary tests/nostdlib/fibonacci/test --debugger lldb -- tests/nostdlib/fibonacci/test.cpp - test.cpp = (0.2832) - -## An example test case - -The sample test case (tests/nostdlib/fibonacci) looks like this: - - 1. #ifdef _MSC_VER - 2. # define DEX_NOINLINE __declspec(noinline) - 3. #else - 4. # define DEX_NOINLINE __attribute__((__noinline__)) - 5. #endif - 6. - 7. DEX_NOINLINE - 8. void Fibonacci(int terms, int& total) - 9. { - 0. int first = 0; - 11. int second = 1; - 12. for (int i = 0; i < terms; ++i) - 13. { - 14. int next = first + second; // DexLabel('start') - 15. total += first; - 16. first = second; - 17. second = next; // DexLabel('end') - 18. } - 19. } - 20. - 21. int main() - 22. { - 23. int total = 0; - 24. Fibonacci(5, total); - 25. return total; - 26. } - 27. - 28. /* - 29. DexExpectWatchValue('i', '0', '1', '2', '3', '4', - 30. from_line='start', to_line='end') - 31. DexExpectWatchValue('first', '0', '1', '2', '3', '5', - 32. from_line='start', to_line='end') - 33. DexExpectWatchValue('second', '1', '2', '3', '5', - 34 from_line='start', to_line='end') - 35. DexExpectWatchValue('total', '0', '1', '2', '4', '7', - 36. from_line='start', to_line='end') - 37. DexExpectWatchValue('next', '1', '2', '3', '5', '8', - 38. from_line='start', to_line='end') - 39. DexExpectWatchValue('total', '7', on_line=25) - 40. DexExpectStepKind('FUNC_EXTERNAL', 0) - 41. */ - -[DexLabel][1] is used to give a name to a line number. - -The [DexExpectWatchValue][2] command states that an expression, e.g. `i`, should -have particular values, `'0', '1', '2', '3','4'`, sequentially over the program -lifetime on particular lines. You can refer to a named line or simply the line -number (See line 39). - -At the end of the test is the following line: - - DexExpectStepKind('FUNC_EXTERNAL', 0) - -This [DexExpectStepKind][3] command indicates that we do not expect the debugger -to step into a file outside of the test directory. - -[1]: Commands.md#DexLabel -[2]: Commands.md#DexExpectWatchValue -[3]: Commands.md#DexExpectStepKind - -## Detailed DExTer reports - -Running the command below launches the tests/nostdlib/fibonacci test case in DExTer, using LLDB as the debugger and producing a detailed report: - - $ dexter.py test --vs-solution clang-cl_vs2015 --debugger vs2017 --cflags="/Ox /Zi" --ldflags="/Zi" -v -- tests/nostdlib/fibonacci - -The detailed report is enabled by `-v` and shows a breakdown of the information from each debugger step. For example: - - fibonacci = (0.2832) - - ## BEGIN ## - [1, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 23, 1, "BREAKPOINT", "FUNC", {}] - [2, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 24, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}] - [3, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 25, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}] - . [4, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "FUNC", {}] - . [5, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] - . [6, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] - . [7, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] - . [8, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] - . [9, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "0"}] - . [10, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] - . [11, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] - . [12, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "1"}] - . [13, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] - . [14, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] - . [15, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "2", "total": "0", "first": "1"}] - . [16, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] - . [17, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] - . [18, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "3", "total": "0", "first": "2"}] - . [19, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] - . [20, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] - . [21, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "5", "total": "0", "first": "3"}] - . [22, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}] - . [23, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}] - . [24, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 20, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}] - [25, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 26, 1, "BREAKPOINT", "FUNC", {"total": "7"}] - ## END (25 steps) ## - - - step kind differences [0/1] - FUNC_EXTERNAL: - 0 - - test.cpp:15-18 [first] [9/21] - expected encountered values: - 0 - 1 - 2 - 3 - - missing values: - 5 [-6] - - result optimized away: - step 5 (Variable is optimized away and not available.) [-3] - step 7 (Variable is optimized away and not available.) - step 8 (Variable is optimized away and not available.) - step 11 (Variable is optimized away and not available.) - step 14 (Variable is optimized away and not available.) - step 17 (Variable is optimized away and not available.) - step 20 (Variable is optimized away and not available.) - step 23 (Variable is optimized away and not available.) - - test.cpp:15-18 [i] [15/21] - result optimized away: - step 5 (Variable is optimized away and not available.) [-3] - step 7 (Variable is optimized away and not available.) [-3] - step 8 (Variable is optimized away and not available.) [-3] - step 9 (Variable is optimized away and not available.) [-3] - step 11 (Variable is optimized away and not available.) [-3] - step 12 (Variable is optimized away and not available.) - step 14 (Variable is optimized away and not available.) - step 15 (Variable is optimized away and not available.) - step 17 (Variable is optimized away and not available.) - step 18 (Variable is optimized away and not available.) - step 20 (Variable is optimized away and not available.) - step 21 (Variable is optimized away and not available.) - step 23 (Variable is optimized away and not available.) - - test.cpp:15-18 [second] [21/21] - expected encountered values: - 1 - 2 - 3 - 5 - - result optimized away: - step 5 (Variable is optimized away and not available.) [-3] - step 7 (Variable is optimized away and not available.) [-3] - step 8 (Variable is optimized away and not available.) [-3] - step 11 (Variable is optimized away and not available.) [-3] - step 14 (Variable is optimized away and not available.) [-3] - step 17 (Variable is optimized away and not available.) [-3] - step 20 (Variable is optimized away and not available.) [-3] - step 23 (Variable is optimized away and not available.) - - test.cpp:15-18 [total] [21/21] - expected encountered values: - 0 - - missing values: - 1 [-6] - 2 [-6] - 4 [-6] - 7 [-3] - - test.cpp:16-18 [next] [15/21] - result optimized away: - step 5 (Variable is optimized away and not available.) [-3] - step 8 (Variable is optimized away and not available.) [-3] - step 11 (Variable is optimized away and not available.) [-3] - step 14 (Variable is optimized away and not available.) [-3] - step 17 (Variable is optimized away and not available.) [-3] - step 20 (Variable is optimized away and not available.) - step 23 (Variable is optimized away and not available.) - - test.cpp:26 [total] [0/7] - expected encountered values: - 7 - -The first line - - fibonacci = (0.2832) - -shows a score of 0.2832 suggesting that unexpected behavior has been seen. This score is on scale of 0.0000 to 1.000, with 0.000 being the worst score possible and 1.000 being the best score possible. The verbose output shows the reason for any scoring. For example: - - test.cpp:15-18 [first] [9/21] - expected encountered values: - 0 - 1 - 2 - 3 - - missing values: - 5 [-6] - - result optimized away: - step 5 (Variable is optimized away and not available.) [-3] - step 7 (Variable is optimized away and not available.) - step 8 (Variable is optimized away and not available.) - step 11 (Variable is optimized away and not available.) - step 14 (Variable is optimized away and not available.) - step 17 (Variable is optimized away and not available.) - step 20 (Variable is optimized away and not available.) - step 23 (Variable is optimized away and not available.) - -shows that for `first` the expected values 0, 1, 2 and 3 were seen, 5 was not. On some steps the variable was reported as being optimized away. - -## Writing new test cases - -Each test can be either embedded within the source file using comments or included as a separate file with the .dex extension. Dexter does not include support for building test cases, although if a Visual Studio Solution (.sln) is used as the test file, VS will build the program as part of launching a debugger session if it has not already been built. +For more information, see: +### [Scripts.md](./Script.md) +### [Heuristic.md](./Heuristic.md) diff --git a/cross-project-tests/debuginfo-tests/dexter/Script.md b/cross-project-tests/debuginfo-tests/dexter/Script.md new file mode 100644 index 0000000000000..05ad986d495eb --- /dev/null +++ b/cross-project-tests/debuginfo-tests/dexter/Script.md @@ -0,0 +1,213 @@ +# Dexter Script Testing + +Dexter's script mode can be accessed by using the `--use-script` flag. + +Dexter scripts are represented by YAML documents, which contain various "nodes" instructing Dexter how to step through the debuggee program, what information to collect and store from the debugger, and how to evaluate the result. A simple Dexter script looks something like this: + +```yaml +--- +!where {function: foo}: + !value arg: 5 + !type arg: int + !and {lines: !range [10, 14]}: + !value local: ['a', 'b', 'c'] +!where {function: bar}: + !where {function: baz}: + !step exactly: [20, 21, 22, 23, 24] +... +``` + +This Dexter test checks that: +- When the debugger steps into `foo`, the type and value of `arg` is always `(int) 5`. +- While the debugger is in `foo` and the current line is between 10 and 14 (inclusive), the value of `local` is `'a'`, `'b'`, or `'c'`. +- While the debugger is in the function `baz`, which was called directly from the function `bar`, the lines that the debugger steps through exactly the lines 20-24 in order. + +The Dexter test follows a structure based on the nodes - in the example above, each line starts with a node. Some of the basic types of node are: + +- `!where` describes a single stack frame using either a function name or a filename + line range; these will be used by the debugger to set breakpoints. We consider a `!where` node to be "active" when the current stack frame matches the `!where` node. A `!where` node can either appear at the "root" of the script, or it can appear as the child of another `!where`, in which case it will only be active when its parent `!where` matches the frame above it. For example, the `!where {function: baz}` node is only active when the next frame up is `bar`, matching its parent. +- `!and` is similar to `!where`, but it can only match the same stack frame as its parent `!where` (and cannot appear at the root of the script). For example, the `!and {lines: !range [10, 14]}` node is only active when the current line number is in the range [10-14] *and* the current function is `foo`, because the `!and` is a child of `!where {function: foo}`. `!where` and `!and` nodes are collectively referred to as "state" nodes. +- `!value` and `!type` are "expect" nodes, meaning they describe testable output from the debugger. These nodes must appear a children of a state node (`!where` or `!and`), and are active whenever their parent is active. The form these nodes take is `!(value|type) <variable-name>: <expected-values>`, and their function is to collect information for `<variable-name>` while the debugger is running and the node is active, and compare that to `<expected-values>` during the evaluation step to produce the final test results. +- `!step` is another kind of expect node, which tests the line numbers seen while stepping through the program, and its expected value is a list of line numbers that we expect to see (or not see in some cases - see more detailed documentation below). +- `!range` isn't a "script node" as the others above are, but a "utility node", meaning it is used by other nodes to represent some data. `!range [<start>, <stop>]` represents an inclusive range from `start` to `stop`, and is used by state nodes. + +All these nodes are arranged in a nested map structure, where each state node maps to its children. A YAML document containing this structure is embedded in the input test file: the file may either be a YAML file, where the whole file is a single document, or else the first valid YAML document contained in the file which is also a valid Dexter test script will be used. Generally, this requires one line that is just `---` to start the document, and another which is just `...` to end the document. + +# Script Nodes + +## State Nodes + +State nodes are matched against stack frames when the debugger is stopped, and are used to control how Dexter controls the debugger (e.g. what step/continue actions to take after each stop) and determine the scope where other nodes are evaluated. State nodes have child nodes, declare some form of state that can be compared against a particular stack frame to produce either a match or non-match; when they match against the current frame, their children are evaluated. State nodes have the following rules: + +- **Root** state nodes are top-level nodes in the script. Each time the debugger stops, Dexter attempts to match each root node to each stack frame, searching from the *root-most* to the *leaf-most* stack frame, and stopping at the first matching frame (if any). Root nodes must always be `!where` nodes. +- **Nested** state nodes refer to any non-root state nodes. There are two kinds of nested state node possible: `!where` and `!and`. A nested `!where` node can only match the stack frame called from the frame that matches its parent state node. A nested `!and` node can only match the same stack frame as its parent state node. +- Each state node can match only one frame per-step, e.g. if you have a recursive function `fib` and a root node `!where {function: fib}`, the node will only match the outermost call, not any of the recursive calls. Conversely however, it is possible for a single frame to be matched by different state nodes. +- Both `!where` and `!and` nodes have the format: `!<type> { <args>, ... }`, supporting the following arguments: + - `function: <function-name>` - Declares the name of a function to match on the current frame. This must be an exact match according to the debugger's presented function name, which may including namespace qualifiers. Mutually exclusive with `lines` or `file`. + - `lines: <line> | <range>` - Declares one or more line numbers that the frame should match. Mutually exclusive with `function`; may be passed along with `file`, and if `file` is omitted it defaults to the script filepath. This argument takes either a single line number, or an inclusive range of line numbers in the form `!range [<start>, <end>]`. Labels may be provided instead of literal numbers (see below). + - `file: <file-name>` - Declares the file that the frame's source location should match. Mutually exclusive with `function`, can only be passed if `line` is also passed. + - `for_hit_count: <count>` - Means that the state node can only become active `count` times. A state node only "becomes" active when it was previously inactive, meaning we don't increment the hit count for a state node if was also active in the previous step. + - `after_hit_count: <count>` - Means that the state node can only become active after it is reached `count` times. As with `for_hit_count`, the hit count is not incremented if the state node would have also been active in the previous step. + - `conditions: <cond>` - Means that the state node is only active when the condition given by `cond` is true, which will be evaluated every step that the state node would be active. If a state node with a condition also has a child `!where` node, `cond` will *not* be re-evaluated while in the called frame - the condition is assumed to remain true until we return to the frame that contains it. +- Additionally, `!and` supports one more argument: + - `at_frame_idx: <frame-index>` - Means that instead of the `!and` node matching the same frame as its parent state node, it matches the frame at index `frame-index`, where the leaf frame has index 0. All other conditions of the `!and` and all of its children will be evaluated against that frame, providing a way to explicitly test values in frames other than the current frame. An `!and` node with `at_frame_idx` cannot contain any nested `!where` nodes. + +As a convenience, Dexter supports use of "labels" instead of literal line numbers. These labels have the form `!label <name>`, and will be substituted by a line number that the label represents. Line numbers for each label come from the program source files: when Dexter encounters a `!label`, it will search the source file where that line should appear. When it encounters the string `!dex_label <name>` - regardless of surrounding context - it maps that label name to the line on which that string appears within the file. The source file that Dexter searches is the `file` argument given by the state node or its nearest parent state node, up to the nearest `!where`. If no file is given, the test file is assumed as a default. Since the file should generally be a relative path, not absolute, we treat it as being relative to the `--source-root-dir` argument if it is given, or the test file's directory otherwise. + +```cpp +int main() { + for (int i = 0; i < 5; ++i) + (void)0; // !dex_label loop +} + +/* +--- +!where {lines: !label loop}: + !value i: [0, 1, 2, 3, 4] +... +*/ +``` + +## Expect Nodes + +Expect nodes describe some expected debugger output, which will be compared against the actual debugger output and be scored in Dexter's output. There are two kinds of expect: variable expects and step expects. + +### Variable Expects + +Variable expects test the debugger output for a specific variable. This takes the form of either `!type <variable>: <expected-types>` or `!value <variable>: <expected-values>`. While debugging, Dexter will fetch variable information for every variable with an active expect, and all collected values will be compared to all expected values; Dexter will report when any expected values are not seen in the output, or when any unexpected values are seen in the output. There are different forms of expected value that can be declared: + +```yaml +# A single expected value may be declared for a variable, meaning Dexter expects +# the variable to have only that value while the expect is active. +!value x: 0 +!type x: int + +# A list of expected values means that Dexter expects the variable to have all +# of those values at least once while the expect is active; order is not +# checked, and repetitions are ignored. +!value c: ['a', 'b', 'c'] +!type c: [char] + +# An expected value for an aggregate, or other decomposable variable, may give +# expected values for individual members of that variable. Not all members need +# to have expected values declared. +!value tuple: + first: 1 + second: 'z' +!type tuple: + first: int + second: char + +# Aggregate expected values and lists of expected values can be combined: a +# variable may have a list of aggregate values, and a member expected value may +# have a list of expected values. When determining which expected value an +# aggregate matches, if there are no exact matches Dexter will select the +# closest match, and give a "partial match" score in the final output. +!value point: + - x: 2 + y: 4 + - x: 4 + y: [8, 10] + +# When testing pointers, the `!address` node may be used to specify an abstract +# label for an address and optional offset. The first time Dexter matches a +# valid pointer against an !address, it will assume a valid match and assign +# that address to the !address label. In all future matches, that !address label +# will be considered equal to the assigned address, allowing pointer variables +# that are always the same, or at a fixed offset, to be compared. +!value iterator: [!address base, !address base + 8, !address base + 16] + +# Float values can be tested using the `!float` node, which performs a float +# equality check between the expected and actual debugger output rather than a +# string comparison. An optional range may be provided which causes the `!float` +# to match any value within that range. +!value f: [!float 10, !float 5.2 +- 0.01] +``` + +### Step Expects + +Step expects test the stepping behaviour of the debugger. A step node can only have, as its expected value, a list of non-negative integers. During evaluation, Dexter will compare all of the line numbers stepped on while the step expect was active to the expected steps, according to one of 3 rules: + +- `!step exactly`: while this !expect is active, we expect see exactly the expected lines in-order as many times as they appear in the expected lines list. +- `!step order`: while this !expect is active, we expect to see each of the expected lines in-order at least as many times as they appear in the expected list, ignoring excess lines and lines not in the expected lines list. +- `!step never`: while this !expect is active, we expect to not see any of the lines in the expected lines list. + +### Rewriting Expects + +Dexter has an additional feature, "script rewriting", which can be used to generate expected values for a script from debugger output. This is used whenever an expect is declared with no expected value; this can be represented as either `!value var: null`, or `? !value var`, using the YAML syntax for a mapping key with an absent value. If at least one expect has no expected value, and a `--results-directory` is passed to Dexter, then Dexter will write a copy of the script file into the results directory; this copy will have the same name as the original file and identical contents, except that the Dexter script is replaced with a copy that has all missing expected values filled in. For example, for the following program with an embedded dexter script: + +```cpp +int main() { + for (int i = 0; i < 10; ++i) + (void)0; // !dex_label loop + return 0; +} + +/* +--- +!where {lines: !label loop}: + ? !value i +... +*/ +``` + +Building this program, and running Dexter using the source file as the test will result in Dexter generating a copy in the results directory that looks like: + +```cpp +void foo(int); +int main() { + for (int i = 0; i < 10; ++i) + bar(i) // !dex_label call + return 0; +} + +/* +--- +!where {lines: !label call}: + !value i: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] +... +*/ +``` + +This works for `!value`, `!type`, and `!step` nodes, and when using script rewriting there are two additional types of expect nodes that can be used: `!value/all <scope>` and `!type/all <scope>`. These nodes fetch debugger output for *all* variables within a particular debugger scope, as defined by the DAP specification; see: https://microsoft.github.io/debug-adapter-protocol/specification#Requests_Scopes. These nodes are not directly evaluated; they must have no expected values, and when Dexter rewrites the original script, they will be replaced with `!value`/`!type` nodes for each variable that was seen in the expect's scope while it was active, inserted under !and nodes that cover that variable's live range(s). For example: + +```cpp +int main() { + int factorial = 1; + for (int i = 1; i < 10; ++i) + factorial *= i; + char *string = getString(factorial); + return 0; +} + +/* +--- +!where {function: main}: + ? !value/all Locals + ? !type/all Locals +... +*/ +``` + +The script above, when run with Dexter using `lldb` (which defines a "Locals" scope), will produce the following script as output (this will be embedded in the output file as usual): + +```yaml +!where {function: main}: + !and {lines: !range [3, 6]}: + !value factorial: [1, 2, 6, 24, 120, 720, 5040, 40320, 362880] + !type factorial: int + !and {lines: 160}: + !value i: [1, 2, 3, 4, 5, 6, 7, 8, 9] + !type i: int + !and {lines: 162}: + !value string: "362880" + !type string: "char*" +``` + +These nodes can be used to quickly generate debug info benchmark scripts for large programs by running them against an `-O0 -g` built program to capture a snapshot of what "perfect" debugging should look like in the program, which can then be compared against various build configurations, e.g. comparing optimization levels or compiler versions. These generated scripts may be less useful for comparing debug info quality across different compilers or debuggers however, since any differences in how different compilers or debuggers render variable information (e.g. different derived type rendering) will result in failed expects. + +## Execution Nodes + +Execution nodes are used to directly perform debugger actions within the debugging session. The only execution node is `!then <action>`, which may appear as the sole child of a state node, e.g. `!where {lines: 10}: !then finish`. Then action requested by the `!then` node is performed after the parent state node becomes active, so in the prior example, when the debugger reaches line 10, instead of stepping off of the line as it normally would, Dexter will submit a "finish" command. There are two actions supported by `!then`: +- `!then step_out` - Performs a "step out", exiting the current function. This will disable all breakpoints for any `!where` nodes below the currently active `!where` node, meaning this command should always succeed in exiting the current frame. +- `!then finish` - Ends the debugger session. This command allows Dexter to test programs that run for a long time, or never exit. + >From 3e360ca856a6f0b8c40574d5122dbe85e450a9dd Mon Sep 17 00:00:00 2001 From: Stephen Tozer <[email protected]> Date: Wed, 24 Jun 2026 14:50:04 +0100 Subject: [PATCH 2/2] Address various review comments --- .../debuginfo-tests/dexter/Script.md | 41 ++++++++++--------- 1 file changed, 21 insertions(+), 20 deletions(-) diff --git a/cross-project-tests/debuginfo-tests/dexter/Script.md b/cross-project-tests/debuginfo-tests/dexter/Script.md index 05ad986d495eb..f9710dcb0b6b4 100644 --- a/cross-project-tests/debuginfo-tests/dexter/Script.md +++ b/cross-project-tests/debuginfo-tests/dexter/Script.md @@ -19,12 +19,12 @@ Dexter scripts are represented by YAML documents, which contain various "nodes" This Dexter test checks that: - When the debugger steps into `foo`, the type and value of `arg` is always `(int) 5`. -- While the debugger is in `foo` and the current line is between 10 and 14 (inclusive), the value of `local` is `'a'`, `'b'`, or `'c'`. +- While the debugger is in `foo` and the current line is between 10 and 14 (inclusive), the value of `local` is each of `'a'`, `'b'`, and `'c'` at least once. - While the debugger is in the function `baz`, which was called directly from the function `bar`, the lines that the debugger steps through exactly the lines 20-24 in order. The Dexter test follows a structure based on the nodes - in the example above, each line starts with a node. Some of the basic types of node are: -- `!where` describes a single stack frame using either a function name or a filename + line range; these will be used by the debugger to set breakpoints. We consider a `!where` node to be "active" when the current stack frame matches the `!where` node. A `!where` node can either appear at the "root" of the script, or it can appear as the child of another `!where`, in which case it will only be active when its parent `!where` matches the frame above it. For example, the `!where {function: baz}` node is only active when the next frame up is `bar`, matching its parent. +- `!where` describes a single stack frame using either a function name or a filename + line range; these will be used by the debugger to set breakpoints. We consider a `!where` node to be "active" when the current stack frame matches the `!where` node. A `!where` node can either appear at the "root" of the script, or it can appear as the child of another `!where`, in which case it will only be active when its parent `!where` matches the frame above it. For example, the `!where {function: baz}` node is only active when the next frame up is `bar`, matching its parent. Child nodes are represented in YAML as mapping entries under the parent node, which is determined using the indent level. - `!and` is similar to `!where`, but it can only match the same stack frame as its parent `!where` (and cannot appear at the root of the script). For example, the `!and {lines: !range [10, 14]}` node is only active when the current line number is in the range [10-14] *and* the current function is `foo`, because the `!and` is a child of `!where {function: foo}`. `!where` and `!and` nodes are collectively referred to as "state" nodes. - `!value` and `!type` are "expect" nodes, meaning they describe testable output from the debugger. These nodes must appear a children of a state node (`!where` or `!and`), and are active whenever their parent is active. The form these nodes take is `!(value|type) <variable-name>: <expected-values>`, and their function is to collect information for `<variable-name>` while the debugger is running and the node is active, and compare that to `<expected-values>` during the evaluation step to produce the final test results. - `!step` is another kind of expect node, which tests the line numbers seen while stepping through the program, and its expected value is a list of line numbers that we expect to see (or not see in some cases - see more detailed documentation below). @@ -40,18 +40,18 @@ State nodes are matched against stack frames when the debugger is stopped, and a - **Root** state nodes are top-level nodes in the script. Each time the debugger stops, Dexter attempts to match each root node to each stack frame, searching from the *root-most* to the *leaf-most* stack frame, and stopping at the first matching frame (if any). Root nodes must always be `!where` nodes. - **Nested** state nodes refer to any non-root state nodes. There are two kinds of nested state node possible: `!where` and `!and`. A nested `!where` node can only match the stack frame called from the frame that matches its parent state node. A nested `!and` node can only match the same stack frame as its parent state node. -- Each state node can match only one frame per-step, e.g. if you have a recursive function `fib` and a root node `!where {function: fib}`, the node will only match the outermost call, not any of the recursive calls. Conversely however, it is possible for a single frame to be matched by different state nodes. +- Each state node can match only one frame per-step, e.g. if you have a recursive function `fib` and a root node `!where {function: fib}`, the node will only match the first/rootmost call, not any of the recursive calls. Conversely however, it is possible for a single frame to be matched by different state nodes. - Both `!where` and `!and` nodes have the format: `!<type> { <args>, ... }`, supporting the following arguments: - `function: <function-name>` - Declares the name of a function to match on the current frame. This must be an exact match according to the debugger's presented function name, which may including namespace qualifiers. Mutually exclusive with `lines` or `file`. - - `lines: <line> | <range>` - Declares one or more line numbers that the frame should match. Mutually exclusive with `function`; may be passed along with `file`, and if `file` is omitted it defaults to the script filepath. This argument takes either a single line number, or an inclusive range of line numbers in the form `!range [<start>, <end>]`. Labels may be provided instead of literal numbers (see below). - - `file: <file-name>` - Declares the file that the frame's source location should match. Mutually exclusive with `function`, can only be passed if `line` is also passed. + - `lines: <line> | <range>` - Declares one or more line numbers that the frame should match. Mutually exclusive with `function`; may be passed along with `file`, and if `file` is omitted for a `!where` it defaults to the script filepath, while for an `!and` node no explicit `file` is assumed, i.e. the `!and` will only match the line number and not the current file. This argument takes either a single line number, or an inclusive range of line numbers in the form `!range [<start>, <end>]`. Labels may be provided instead of literal numbers (see below). + - `file: <file-name>` - Declares the file that the frame's source location should match. Mutually exclusive with `function`, can only be passed if `line` is also passed. If `--debugger-use-relative-paths` is passed to Dexter, then the filepath will be treated as a relative path from `--source-root-dir` when setting breakpoints; otherwise, the filepath will be passed to the debugger verbatim when setting breakpoints (which generally works for local builds). - `for_hit_count: <count>` - Means that the state node can only become active `count` times. A state node only "becomes" active when it was previously inactive, meaning we don't increment the hit count for a state node if was also active in the previous step. - `after_hit_count: <count>` - Means that the state node can only become active after it is reached `count` times. As with `for_hit_count`, the hit count is not incremented if the state node would have also been active in the previous step. - - `conditions: <cond>` - Means that the state node is only active when the condition given by `cond` is true, which will be evaluated every step that the state node would be active. If a state node with a condition also has a child `!where` node, `cond` will *not* be re-evaluated while in the called frame - the condition is assumed to remain true until we return to the frame that contains it. + - `conditions: <cond>` - Means that the state node is only active when the condition given by `cond` is true, which will be evaluated every step that the state node would be active. If a state node with a condition also has a child `!where` node, `cond` will *not* be re-evaluated while in the called frame - the condition is assumed to remain true until we return to the frame that contains it. The `condition` is checked before determining hit counts, i.e. the hit count for a node with `for_hit_count` or `after_hit_count` will only be incremented if `cond` is true. - Additionally, `!and` supports one more argument: - `at_frame_idx: <frame-index>` - Means that instead of the `!and` node matching the same frame as its parent state node, it matches the frame at index `frame-index`, where the leaf frame has index 0. All other conditions of the `!and` and all of its children will be evaluated against that frame, providing a way to explicitly test values in frames other than the current frame. An `!and` node with `at_frame_idx` cannot contain any nested `!where` nodes. -As a convenience, Dexter supports use of "labels" instead of literal line numbers. These labels have the form `!label <name>`, and will be substituted by a line number that the label represents. Line numbers for each label come from the program source files: when Dexter encounters a `!label`, it will search the source file where that line should appear. When it encounters the string `!dex_label <name>` - regardless of surrounding context - it maps that label name to the line on which that string appears within the file. The source file that Dexter searches is the `file` argument given by the state node or its nearest parent state node, up to the nearest `!where`. If no file is given, the test file is assumed as a default. Since the file should generally be a relative path, not absolute, we treat it as being relative to the `--source-root-dir` argument if it is given, or the test file's directory otherwise. +As a convenience, Dexter supports use of "labels" instead of literal line numbers. These labels have the form `!label <name>`, and will be substituted by a line number that the label represents. Line numbers for each label come from the program source files: when Dexter encounters a `!label`, it will search the source file where that line should appear. When it encounters the string `!dex_label <name>` - regardless of surrounding context - it maps that label name to the line on which that string appears within the file. The source file that Dexter searches for labels is determined depending on the state node: for a `!where` node, either its explicit `file` argument is used, or the test file is used as the default; for an `!and` node, the label file is either the `file` argument given by the `!and` node or its nearest parent node up to the nearest `!where`, or if no explicit `file` is found then the filepath of the current frame is used as the default. Since the file should generally be a relative path, not absolute, we treat it as being relative to the `--source-root-dir` argument if it is given, or the test file's directory otherwise. ```cpp int main() { @@ -69,11 +69,11 @@ int main() { ## Expect Nodes -Expect nodes describe some expected debugger output, which will be compared against the actual debugger output and be scored in Dexter's output. There are two kinds of expect: variable expects and step expects. +Expect nodes describe some expected debugger output, which will be compared against the actual debugger output and be scored in Dexter's output. There are two kinds of expect node: variable expect nodes and step expect nodes. -### Variable Expects +### Variable Expect Nodes -Variable expects test the debugger output for a specific variable. This takes the form of either `!type <variable>: <expected-types>` or `!value <variable>: <expected-values>`. While debugging, Dexter will fetch variable information for every variable with an active expect, and all collected values will be compared to all expected values; Dexter will report when any expected values are not seen in the output, or when any unexpected values are seen in the output. There are different forms of expected value that can be declared: +Variable expect nodes test the debugger output for a specific variable. This takes the form of either `!type <variable>: <expected-types>` or `!value <variable>: <expected-values>`. While debugging, Dexter will fetch variable information for every variable with an active expect, and all collected values will be compared to all expected values; Dexter will report when any expected values are not seen in the output, or when any unexpected values are seen in the output. There are different forms of expected value that can be declared: ```yaml # A single expected value may be declared for a variable, meaning Dexter expects @@ -109,11 +109,12 @@ Variable expects test the debugger output for a specific variable. This takes th y: [8, 10] # When testing pointers, the `!address` node may be used to specify an abstract -# label for an address and optional offset. The first time Dexter matches a -# valid pointer against an !address, it will assume a valid match and assign -# that address to the !address label. In all future matches, that !address label -# will be considered equal to the assigned address, allowing pointer variables -# that are always the same, or at a fixed offset, to be compared. +# label for an address and optional offset. The first time while evaluating a +# trace that Dexter matches a valid pointer against an !address, it will assume +# a valid match and assign that address to the !address label. In all future +# matches during the current evaluatoin, that !address label will be considered +# equal to the assigned address, allowing pointer variables that are always the +# same, or at a fixed offset, to be compared. !value iterator: [!address base, !address base + 8, !address base + 16] # Float values can be tested using the `!float` node, which performs a float @@ -123,15 +124,15 @@ Variable expects test the debugger output for a specific variable. This takes th !value f: [!float 10, !float 5.2 +- 0.01] ``` -### Step Expects +### Step Expect Nodes -Step expects test the stepping behaviour of the debugger. A step node can only have, as its expected value, a list of non-negative integers. During evaluation, Dexter will compare all of the line numbers stepped on while the step expect was active to the expected steps, according to one of 3 rules: +Step expect nodes test the stepping behaviour of the debugger. A step node can only have, as its expected value, a list of non-negative integers. During evaluation, Dexter will compare all of the line numbers stepped on while the step expect was active to the expected steps, according to one of 3 rules: - `!step exactly`: while this !expect is active, we expect see exactly the expected lines in-order as many times as they appear in the expected lines list. - `!step order`: while this !expect is active, we expect to see each of the expected lines in-order at least as many times as they appear in the expected list, ignoring excess lines and lines not in the expected lines list. - `!step never`: while this !expect is active, we expect to not see any of the lines in the expected lines list. -### Rewriting Expects +### Rewriting Expect Nodes Dexter has an additional feature, "script rewriting", which can be used to generate expected values for a script from debugger output. This is used whenever an expect is declared with no expected value; this can be represented as either `!value var: null`, or `? !value var`, using the YAML syntax for a mapping key with an absent value. If at least one expect has no expected value, and a `--results-directory` is passed to Dexter, then Dexter will write a copy of the script file into the results directory; this copy will have the same name as the original file and identical contents, except that the Dexter script is replaced with a copy that has all missing expected values filled in. For example, for the following program with an embedded dexter script: @@ -203,11 +204,11 @@ The script above, when run with Dexter using `lldb` (which defines a "Locals" sc !type string: "char*" ``` -These nodes can be used to quickly generate debug info benchmark scripts for large programs by running them against an `-O0 -g` built program to capture a snapshot of what "perfect" debugging should look like in the program, which can then be compared against various build configurations, e.g. comparing optimization levels or compiler versions. These generated scripts may be less useful for comparing debug info quality across different compilers or debuggers however, since any differences in how different compilers or debuggers render variable information (e.g. different derived type rendering) will result in failed expects. +These nodes can be used to quickly generate debug info benchmark scripts for large programs by running them against an `-O0 -g` built program to capture a snapshot of what "perfect" debugging should look like in the program, which can then be compared against various build configurations, e.g. comparing optimization levels or compiler versions. These generated scripts may be less useful for comparing debug info quality across different compilers or debuggers however, since any differences in how different compilers or debuggers render variable information (e.g. different derived type rendering) will result in failed expectations. ## Execution Nodes Execution nodes are used to directly perform debugger actions within the debugging session. The only execution node is `!then <action>`, which may appear as the sole child of a state node, e.g. `!where {lines: 10}: !then finish`. Then action requested by the `!then` node is performed after the parent state node becomes active, so in the prior example, when the debugger reaches line 10, instead of stepping off of the line as it normally would, Dexter will submit a "finish" command. There are two actions supported by `!then`: -- `!then step_out` - Performs a "step out", exiting the current function. This will disable all breakpoints for any `!where` nodes below the currently active `!where` node, meaning this command should always succeed in exiting the current frame. +- `!then step_out` - Performs a "step out", exiting the current function. This will disable all breakpoints for any child `!where` nodes of the currently active `!where` node, meaning this command should always succeed in exiting the current frame. - `!then finish` - Ends the debugger session. This command allows Dexter to test programs that run for a long time, or never exit. _______________________________________________ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
