Andrey Yarovoy created HDDS-17333:
-------------------------------------
Summary: Add a secure Ozone + YARN compose environment that can
run YARN jobs
Key: HDDS-17333
URL: https://issues.apache.org/jira/browse/HDDS-17333
Project: Apache Ozone
Issue Type: Bug
Components: docker
Reporter: Andrey Yarovoy
*Problem*
There's no docker images that can be used to reproduce MapReduce-on-Ozone
behavior in dev environment. The
existing `ozonesecure` + `common/hadoop-secure.yaml` setup runs MapReduce smoke
tests (PI, WordCount), but
differs from a production NodeManager in ways that hide real costs:
- It uses `DefaultContainerExecutor`, which localizes job files inside the
long-lived NodeManager JVM. A secure
NodeManager uses `LinuxContainerExecutor`, which starts a separate
`ContainerLocalizer` JVM, with a
fresh Ozone client, for each container. In our tests that cost about 2 s for
the AppMaster and 2–3 s for each map
container on another node. With `DefaultContainerExecutor` it was 0 s.
- It has a single NodeManager, so maps never run on a node that still has to
localize the job's files.
- Its 4 GB map and reduce containers, Recon, S3 Gateway and HttpFS don't fit
next to YARN in a devs Docker VM.
Also, a pipeline limit of 1 leaves no RATIS/THREE pipeline when one datanode
goes stale.
- The `apache/hadoop` image ships `container-executor` without setuid, and on
3.4.3 (Ubuntu 22.04) it is linked
against OpenSSL 1.1, which the image doesn't have. So `LinuxContainerExecutor`
can't run as shipped.
### Proposed change
A new compose environment, eg. `compose/ozonesecure-mr`, reusing `ozonesecure`
and `common/hadoop-secure.yaml`:
- **NodeManagers:** N of them (`nm`, `nm2`, …) with their own hostnames and the
shared `nm/nm` principal. Each
offers 2 GB of container memory, so a job's containers spread across nodes.
- **`LinuxContainerExecutor`:** a NodeManager start script installs
`container-executor` setuid root under a
root-owned prefix, with its own `container-executor.cfg`. Where needed it adds
`libssl1.1`. A small derived
NodeManager image, or a fixed `apache/hadoop` image, would be cleaner than
installing at start time.
- **Small cluster:** only `kdc kms scm om datanode rm jhs nm*`, with
`ozone.datanode.pipeline.limit=2` and
512–768 MB MapReduce containers.
- **`test-dfsio.sh`:** runs TestDFSIO write/read on `ofs://` with the MR
staging dir on Ozone. It collects the
per-container logs and the NodeManager logs and summarizes per-job phases:
AppMaster and map localization from
the NodeManager's container state changes, AppMaster and task start-up from the
container logs, and time outside
TestDFSIO's own timer.
- **Not part of the default acceptance run,** because it's resource-heavy. A
1-NodeManager, 1-job variant could
serve as a smoke test.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
