> Jose E. Marchesi wrote:
>>> Jose E. Marchesi wrote:
>>>
>>>>> [...]
>>>>>
>>>>>> I agree that distcheck is good but not a cure all. Any static
>>>>>> system can be attacked when there is motive, and unit tests are
>>>>>> easily gamed.
>>>>>>
>>>>> The issue seems to be releases containing binary data for unit tests,
>>>>> instead of source or scripts to generate that data. In this case,
>>>>> that binary data was used to smuggle in heavily obfuscated object
>>>>> code.
>>>>>
>>>> As a side note, GNU poke (https://jemarch.net/poke) is good for
>>>> generating arbitrarily complex binary data from clear textual
>>>> descriptions.
>>>>
>>> While it is suitable for that use, at last check poke is itself very
>>> complex, complete with its own JIT-capable VM. This is good for
>>> interactive use, but I get nervous about complexity in testsuites,
>>> where simplicity can greatly aid debugging, and it /might/ be possible
>>> to hide a backdoor similarly in a poke pickle. (This seems to be a
>>> general problem with powerful interactive editors.)
>>>
>>
>> Yes, I agree simplicity it is very desirable, in testsuites and actually
>> everywhere else. I also am not fond of dragging in dependencies.
>>
>
> Exactly---I am sure that poke is great for interactive use, but a
> self-contained solution is probably better for a testsuite.
>
>> But I suppose we also agree in that it is not possible to assembly
>> non-trivial binary data structures in a simple way, without somehow
>> moving the complexity of the encoding into some sort of generator, which
>> will not be simple. The GDB testsuite, for example, ships with a DWARF
>> assembler written in around 3000 lines of Tcl. Sure, it is simpler than
>> poke and doesn't drag in additional dependencies. But it has to be
>> carefully maintained and kept up to date, and the complexity is there.
>>
>
> The problem for a compression tool testsuite is that compression
> formats are (I believe) defined as byte-streams or bit-streams.
> Further, the generator(s) must be able to produce /incorrect/ output
> as well, in order to test error handling.
>>> Further, GNU poke defines its own specialized programming language for
>>> manipulating binary data. Supplying generator programs in C (or C++)
>>> for binary test data in a package that itself uses C (or C++) ensures
>>> that every developer with the skills to improve or debug the package
>>> can also understand the testcase generators.
>>>
>>
>> Here we will have to disagree.
>>
>> IMO it is precisely the many and tricky details on properly marshaling
>> binary data in general-purpose programming languages that would have
>> greater odds to lead to difficult to understand, difficult to maintain
>> and possibly buggy or malicious encoders. The domain specific language
>> is here an advantage, not a liability.
>>
>> This you need to do in C to encode and generate test data for a single
>> signed 32-bit NUMBER in an output file in a _more or less_ portable way:
>>
>> void generate_testdata (off_t offset, int endian, int number)
>> {
>> int bin_flag = 0, fd;
>>
>> #ifdef _WIN32
>> int bin_flag = O_BINARY;
>> #endif
>> fd = open ("testdata.bin", bin_flag, S_IWUSR);
>> if (fd == -1)
>> fatal ("error generating data.");
>> if (endian == BIG)
>> {
>> b[0] = (number >> 24) & 0xff;
>> b[1] = (number >> 16) & 0xff;
>> b[2] = (number >> 8) & 0xff;
>> b[3] = number & 0xff;
>> }
>> else
>> {
>> b[3] = (number >> 24) & 0xff;
>> b[2] = (number >> 16) & 0xff;
>> b[1] = (number >> 8) & 0xff;
>> b[0] = number & 0xff;
>> }
>>
>> lseek (fd, offset, SEEK_SET);
>> for (i = 0; i < 4; ++i)
>> write (fd, &b[i], 1);
>> close (fd);
>> }
>>
>
> While that is a nice general solution, (aside from neglecting the
> declaration "uint8_t b[4];"; with "int b[4];", the code would only
> work on a little-endian processor; with no declaration, the compiler
> will reject it) a compression format would be expected to define the
> endianess of stored values, so the major branch in that function would
> collapse to just one of its alternatives. Compression formats are
> generally defined as streams, so a different decomposition of the
> problem would likely make more sense: (example untested)
>
> void emit_int32le (FILE * out, int value)
> {
> unsigned int R, i;
>
> for (R = (unsigned int)value, i = 0; i < 4; R = R >> 8, i++)
> if (fputc(R & 0xff, out) == EOF)
> fatal("error writing int32le");
> }
> Other code handles opening OUT, or OUT is actually stdout and we
> are writing down a pipe or the shell handled opening the file.
> (The main function can easily check that stdout is not a terminal
> and bail out if it is.) Remember that I am suggesting test
> generator programs, which do not need to be as general as
> ordinary code, nor do they need the same level of
> user-friendliness, since they are expected to be run from scripts
> that encode the precise knowledge of how to call them. (That
> this version is also probably more efficient by avoiding a
> syscall for every byte written is irrelevant for its intended
> use.)
>> This is the Poke equivalent:
>>
>> fun generate_testdata = (offset<uint<64>,B> off, int<32> endian, int<32>
>> number) void:
>> {
>> var fd = open ("testdata.bin");
>> set_endian (endian);
>> int<32> @ fd : off = number;
>> close (fd);
>> }
>>
>> And thanks to the DSL, this scales nicely to more complex structures,
>> such as an ELF64 relocation instead of a signed 32-bit integer:
>>
>> fun generate_testdata = (offset<uint<64>,B> off, int<32> endian, int<32>
>> number) void:
>> {
>> type Elf64_RelInfo =
>> struct Elf64_Xword
>> {
>> uint<32> r_sym;
>> uint<32> r_type;
>> };
>> type Elf64_Rela =
>> struct
>> {
>> offset<uint<64>,B> r_offset;
>> Elf64_RelInfo r_info;
>> offset<int<64>,B> r_addend;
>> };
>> var fd = open ("got32reloc.bin");
>> set_endian (endian);
>> Elf64_Rela @ 0#B
>> = Elf64_Rela { r_info = Elf64_RelInfo { r_sym = 0xff00, r_type = 3 } }
>> close (fd);
>> }
>>
>
> Indeed Poke's language is excellent for concisely describing complex
> structures accurately, closely resembling C's syntax, but if I
> understand correctly, compression tool test data has few complex
> structures and is mostly byte-streams.
> Also, how did poke know to send the structure to "got32reloc.bin"?
> Was it simply the most-recently-opened file (or perhaps the top entry
> on a stack of recently-opened files)? Were the OFF and NUMBER
> parameters intentionally unused, accidentally unused, or somehow
> implicitly used in the second example?
The map in the second example ougth to be:
Elf64_Rela @ fd : 0#B
= Elf64_Rela { r_info = Elf64_RelInfo { r_sym = 0xff00, r_type = 3 } }
>
> -- Jacob
