morningman opened a new pull request, #66983:
URL: https://github.com/apache/doris/pull/66983
## Proposed changes
Downstream projects that want to run integration tests against Doris need one
container they can start and connect to, not a compose file with an FE, a BE
and a wait loop. This adds that image for the **4.1 release line** under
`docker/runtime/all-in-one/4.1/`.
It also removes the old `docker/runtime/all-in-one/` recipe, which no longer
worked: it pinned JDK 8 (so 3.0+ would not start), carried a literal
`apache-doris-x.x.x-bin-` placeholder that had to be hand-edited, and laid
its
`resource/` tree out incompatibly with what `docker-build.sh` produces. Its
entrypoint also called `stop_fe.sh` / `stop_be.sh`, which cannot work under
`--console` (neither leaves a usable pid file).
### Assembly, not repackaging
`fe/` and `be/` are copied straight out of the official `apache/doris:fe-<v>`
and `:be-<v>` images, so a new Doris release needs no work here. buildx
resolves those per target platform, which is what lets one tag be multi-arch
without staging per-arch directories by hand. A locally built `./output` and
an
extracted release tarball are accepted too; all three normalise into the same
artifact stage.
### Two tags
| tag | covers | size |
|---|---|---|
| `all-in-one-<v>` | internal tables, Hive, Iceberg (incl. system tables),
Paimon, JDBC catalogs, external-table writeback, Java UDF | 2.46 GB |
| `all-in-one-<v>-full` | the above plus Hudi, Trino connector, MaxCompute |
2.99 GB |
against 4.9 GB for the same payload untouched.
The largest single saving is not a connector. `apache/doris:be-4.1.3` ships
`doris_be` with debug info; `strip --strip-debug` takes it from **2213 MB to
430 MB** while keeping `.symtab`, so crash backtraces still resolve function
names and only lose file:line. Build with `--strip none` to keep the binary
byte-identical to the official image.
Both the strip and the pruning happen in the artifact stage. Doing either
after
the final `COPY` would only add a whiteout layer and the bytes would still
ship.
### What must never be pruned
`resource/prune.txt` is the only file that needs to track a release, and it
records the three directories that look like external-table extras but are
the
JNI baseline every such read goes through:
- `be/lib/java_extensions/preload-extensions` — no Java sources at all; it is
the parquet-hadoop-bundle / hadoop-common / libthrift / arrow / AWS SDK
dependency bundle. Preloaded by `bin/start_be.sh`, and `DORIS_PRELOAD_JAR`
must stay first on the classpath.
- `be/lib/java_extensions/java-udf` — preloaded alongside it.
- `be/lib/hadoop_hdfs` — the JVM side of libhdfs (157 jars). libhdfs is
linked
statically into `doris_be` in 4.1, hence the empty `native/`, but it is
still
a JNI wrapper and cannot open an HDFS file without them.
FE is left alone: branch-4.1 has no FE plugin split, so all 634 jars in
`fe/lib` are on the startup classpath.
### Startup is a sequence, not a sleep
FE → poll until ready → BE → register the backend → poll until FE reports it
alive → flag ready. FE readiness is a metadata query, deliberately **not**
`select 1`: Nereids picks a backend as the scan node for that, which cannot
succeed before the BE the entrypoint has not started yet.
`HEALTHCHECK` asks FE's `/api/health`, whose `online_backend_num` answers
both
"is FE ready" and "is the BE alive" in one request, so downstream can use
`depends_on: condition: service_healthy` instead of sleeping.
If either process exits, the container exits non-zero — a dead FE becomes a
failed job rather than a job timeout. Shutdown signals process groups, since
`--console` leaves no usable pid file for the stop scripts.
### Usage
```yaml
services:
doris:
image: apache/doris:all-in-one-4.1.3
ports: ["9030:9030", "8030:8030", "8040:8040"]
integration-test:
image: my-project-tests:latest
depends_on:
doris:
condition: service_healthy
```
## Testing
`resource/smoke-test.sh <tag> [base|full]` starts the image, waits for
`healthy`, asserts the JNI baseline and the flavor payload, then runs
create / insert / stream load / aggregate / schema change. Everything goes
through `docker exec`, so the host needs only docker.
Both tags come up healthy in ~17 s and pass on **arm64**. Multi-arch tags
were
built and verified locally (one index, `linux/amd64` + `linux/arm64`).
**Not yet verified, and worth a reviewer's attention:**
1. The **amd64** half is verified structurally only — correct ELF, JDK path
and
payload, and `doris_be --version` byte-identical to the official binary —
but
never actually run. Doris BE does not survive emulation on an Apple
Silicon
host, and neither does the stock `apache/doris:be-4.1.3` amd64 image, so
this
needs a run on real x86 hardware.
2. **Reading a real external table** has not been exercised. The base tag
rests
on Hive/Iceberg data reads needing no format-specific JNI scanner — the
set
of scanner classes BE can construct is closed and neither appears in it —
but
that is still an argument from code, not a test. It needs the fixtures
under
`docker/thirdparties`.
## Notes
- No CI wiring in this PR; `build.sh` is meant to be run by hand for now.
- The directory targets 4.1 only. Other release lines differ enough in
payload
layout to deserve their own directory rather than a version switch.
- Naming: this uses `all-in-one-<version>` to line up with `fe-<version>` /
`be-<version>`. The repo already has one `doris-all-in-one-2.1.0` tag under
the older convention; happy to switch if maintainers prefer it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01QhGHox19pMRy2cvs7A6xsq
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
