wombatu-kun opened a new issue, #19266:
URL: https://github.com/apache/hudi/issues/19266
`scripts/release/validate_source_binary_files.sh` is meant to keep binary
files out of the Apache source release. It has never fired on Linux, which is
every CI runner and most release managers' machines.
### The flag
The check pipes every file through `file -I`:
numBinaryFiles=`find . -iname '*' | xargs -I {} file -I {} | grep -va
directory | ... | wc -l`
`-I` is the BSD/macOS spelling of the MIME option. GNU `file` rejects it:
$ file -I pom.xml
file: invalid option -- 'I'
Usage: file [-bcCdEhikLlNnprsSvzZ0] [--apple] [--extension]
[--mime-encoding] ...
$ echo $?
1
It prints usage to stderr and nothing to stdout, so `wc -l` counts zero and
`numBinaryFiles` is always `0`. `set -o errexit` does not catch it: the
pipeline sits inside a command substitution in an assignment, and without
`pipefail` the assignment takes the exit status of the last stage (`sed`),
which is 0.
Confirmed by running the current script, unmodified, against a clean
checkout of master, whose tree contains 105 image files outside `src/test/`: it
exits 0 and reports `No Binary Files in the source files? - [OK]`.
### The directory
Fixing only the flag turns CI red, because the check is also invoked in the
wrong place. Its contract is "run me from the root of a source release tree" -
that is how `validate_staged_release.sh` calls it, after `cd`-ing into the
extracted tarball. But the `validate-source` job in `bot.yml` runs it against
the raw git checkout, which still contains `rfc/`, `docker/images/`,
`hudi-notebooks/notebooks/common/images/` and `.idea/` - all of them
deliberately excluded from the source release by `create_source_directory.sh`.
A working check reports:
| Where it runs | Files flagged on current master |
| --- | --- |
| Raw git checkout (what `bot.yml` does today) | 105 |
| Source release directory (what the script is written for) | 0 |
### Consequence
Nothing has stopped a binary from entering the source release. #19265 adds
`hudi-agent-gateway/src/hudi_agent_gateway/ui/hudi-logo.png`, which
`create_source_directory.sh` does not exclude, and no check went red. Found
while reviewing that PR:
https://github.com/apache/hudi/pull/19265#discussion_r3567769689
### Fix
Ask libmagic for the charset rather than the MIME type (`file --mime`, the
long option that both GNU and BSD accept; `charset=binary` is a direct answer
that does not depend on libmagic's growing type vocabulary), run the check
inside the source release directory in `bot.yml`, and add `set -o pipefail` so
a broken `file` invocation can never again read as "no binary files found".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
