[Bug binutils/28602] binutils/testsuite/lib/binutils-common.exp: Support free-form shell commands and check patterns

[hp at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28602

--- Comment #2 from Hans-Peter Nilsson  ---
(In reply to Fangrui Song from comment #0)
> 3) Can we allow free-form shell commands instead of specialized directives?
> 
> It is mythical what can and what can't be used in `#foo:`.
> OK, a user can figure out this with trial and error.

"Users" are here programmers, which are expected to be able to find the "myth"
in binutils-common.exp and read the head comment, maybe find a flaw or extend
it.  The method to accomplish this isn't really different to most other
programming-related problems.  The difference is mostly that the testsuite for
the tools isn't as documented as the tools themselves, just as can be expected.

> If we allow free-form shell commands and use line prefixes to differentiate
> the two output streams (objdump -d output and ld -Map output):
> 
>## Test local ifunc are dumped in the link map.
>#RUN: ld: -shared -Map tmpdir/ifunc-1-x86.map --hash-style=sysv %s -o %t
>#RUN: objdump -dw %t | check  # by default, CHECK is the prefix
>#RUN: cat tmpdir/ifunc-1-x86.map | check MAP
> 
>#CHECK: [ \t0-9a-f]+:[ \t0-9a-f]+call[
> \t0-9a-fq]+<\*ABS\*(\+0x[0-9a-f]+|)@plt>
>#CHECK-N: if there is a next line
>#CHECK-N: likewise.
>#CHECK:   skip arbitrary lines
>#CHECK-N: if there is a next line
> 
>#MAP: Local IFUNC function ...

Maybe this suggestion can fit run_dump_test but with the splitting-thing it
evolves into a whole different kind of beast.  That sounds like this should be
a different "proc" (with a different suffix).

Something reading from a descriptive file in a manner *similar* to
run_dump_test may be useful.  I just don't believe bolting stuff onto
run_dump_test is the preferred method.

People needing new features for new tests not simply matched by run_dump_test,
or need to run random shell commands, usually write inline dejagnu-tcl in an
existing or new .exp, usually for the purpose of a single test or array of
similar tests.  There's no myth or magic here - at least not of a form
different to the kind of programming performed on the actual tools!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug binutils/28602] binutils/testsuite/lib/binutils-common.exp: Support free-form shell commands and check patterns

[i at maskray dot me]

https://sourceware.org/bugzilla/show_bug.cgi?id=28602

--- Comment #1 from Fangrui Song  ---
>From https://maskray.me/blog/2021-08-08-toolchain-testing#split-file

In Aug 2020, I added `split-file` to llvm-project which allows you to place
multiple extra files in one file. It'd be nice if binutils has something
similar.

Use case A (organizing input of different formats (e.g. linker
script+assembly) in one file).

```text
# RUN: split-file %s %t
# RUN: llvm-mc %t/asm -o %t.o
# RUN: ld.lld -T %t/lds %t.o -o %t
This is sometimes better than the %S/Inputs/ approach because the user
can see the auxiliary files immediately and don't have to open another file.

# asm
...
# lds
...
```

Use case B (for utilities which don't have built-in input splitting feature):

```text
// RUN: split-file %s %t
// RUN: llc < %t/1.ll | FileCheck %s --check-prefix=CASE1
// RUN: llc < %t/2.ll | FileCheck %s --check-prefix=CASE2
Combing tests prudently can improve readability.
For example, when testing parsing errors if the recovery mechanism isn't
possible,
grouping the tests in one file can more readily see test coverage/strategy.

//--- 1.ll
...
//--- 2.ll
...
```

-- 
You are receiving this mail because:
You are on the CC list for the bug.