[
https://issues.apache.org/jira/browse/HDDS-17333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-17333:
----------------------------------
Description:
*Problem*
There's no docker images that can be used to reproduce MapReduce-on-Ozone
behavior in dev environment. The
existing `ozonesecure` + `common/hadoop-secure.yaml` setup runs MapReduce smoke
tests (PI, WordCount), but
differs from a production NodeManager in ways that hide real costs:
- It uses `DefaultContainerExecutor`, which localizes job files inside the
long-lived NodeManager JVM. A secure
NodeManager uses `LinuxContainerExecutor`, which starts a separate
`ContainerLocalizer` JVM, with a
fresh Ozone client, for each container. In our tests that cost about 2 s for
the AppMaster and 2–3 s for each map
container on another node. With `DefaultContainerExecutor` it was 0 s.
- It has a single NodeManager, so maps never run on a node that still has to
localize the job's files.
- Its 4 GB map and reduce containers, Recon, S3 Gateway and HttpFS don't fit
next to YARN in a devs Docker VM.
Also, a pipeline limit of 1 leaves no RATIS/THREE pipeline when one datanode
goes stale.
- The `apache/hadoop` image ships `container-executor` without setuid, and on
3.4.3 (Ubuntu 22.04) it is linked
against OpenSSL 1.1, which the image doesn't have. So `LinuxContainerExecutor`
can't run as shipped.
*Proposed change*
A new compose environment, eg. `compose/ozonesecure-mr`, reusing `ozonesecure`
and `common/hadoop-secure.yaml`:
- *{*}NodeManagers:{*}* N of them (`nm`, `nm2`, …) with their own hostnames
and the shared `nm/nm` principal. Each
offers 2 GB of container memory, so a job's containers spread across nodes.
- *{*}`LinuxContainerExecutor`:{*}* a NodeManager start script installs
`container-executor` setuid root under a
root-owned prefix, with its own `container-executor.cfg`. Where needed it adds
`libssl1.1`. A small derived
NodeManager image, or a fixed `apache/hadoop` image, would be cleaner than
installing at start time.
- *{*}Small cluster:{*}* only `kdc kms scm om datanode rm jhs nm*`, with
`ozone.datanode.pipeline.limit=2` and
512–768 MB MapReduce containers.
- *{*}`test-dfsio.sh`:{*}* runs TestDFSIO write/read on `ofs://` with the MR
staging dir on Ozone. It collects the
per-container logs and the NodeManager logs and summarizes per-job phases:
AppMaster and map localization from
the NodeManager's container state changes, AppMaster and task start-up from the
container logs, and time outside
TestDFSIO's own timer.
- *{*}Not part of the default acceptance run,{*}* because it's resource-heavy.
A 1-NodeManager, 1-job variant could
serve as a smoke test.
was:
*Problem*
There's no docker images that can be used to reproduce MapReduce-on-Ozone
behavior in dev environment. The
existing `ozonesecure` + `common/hadoop-secure.yaml` setup runs MapReduce smoke
tests (PI, WordCount), but
differs from a production NodeManager in ways that hide real costs:
- It uses `DefaultContainerExecutor`, which localizes job files inside the
long-lived NodeManager JVM. A secure
NodeManager uses `LinuxContainerExecutor`, which starts a separate
`ContainerLocalizer` JVM, with a
fresh Ozone client, for each container. In our tests that cost about 2 s for
the AppMaster and 2–3 s for each map
container on another node. With `DefaultContainerExecutor` it was 0 s.
- It has a single NodeManager, so maps never run on a node that still has to
localize the job's files.
- Its 4 GB map and reduce containers, Recon, S3 Gateway and HttpFS don't fit
next to YARN in a devs Docker VM.
Also, a pipeline limit of 1 leaves no RATIS/THREE pipeline when one datanode
goes stale.
- The `apache/hadoop` image ships `container-executor` without setuid, and on
3.4.3 (Ubuntu 22.04) it is linked
against OpenSSL 1.1, which the image doesn't have. So `LinuxContainerExecutor`
can't run as shipped.
### Proposed change
A new compose environment, eg. `compose/ozonesecure-mr`, reusing `ozonesecure`
and `common/hadoop-secure.yaml`:
- **NodeManagers:** N of them (`nm`, `nm2`, …) with their own hostnames and the
shared `nm/nm` principal. Each
offers 2 GB of container memory, so a job's containers spread across nodes.
- **`LinuxContainerExecutor`:** a NodeManager start script installs
`container-executor` setuid root under a
root-owned prefix, with its own `container-executor.cfg`. Where needed it adds
`libssl1.1`. A small derived
NodeManager image, or a fixed `apache/hadoop` image, would be cleaner than
installing at start time.
- **Small cluster:** only `kdc kms scm om datanode rm jhs nm*`, with
`ozone.datanode.pipeline.limit=2` and
512–768 MB MapReduce containers.
- **`test-dfsio.sh`:** runs TestDFSIO write/read on `ofs://` with the MR
staging dir on Ozone. It collects the
per-container logs and the NodeManager logs and summarizes per-job phases:
AppMaster and map localization from
the NodeManager's container state changes, AppMaster and task start-up from the
container logs, and time outside
TestDFSIO's own timer.
- **Not part of the default acceptance run,** because it's resource-heavy. A
1-NodeManager, 1-job variant could
serve as a smoke test.
> Add a secure Ozone + YARN compose environment that can run YARN jobs
> ---------------------------------------------------------------------
>
> Key: HDDS-17333
> URL: https://issues.apache.org/jira/browse/HDDS-17333
> Project: Apache Ozone
> Issue Type: Bug
> Components: docker
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> *Problem*
>
> There's no docker images that can be used to reproduce MapReduce-on-Ozone
> behavior in dev environment. The
> existing `ozonesecure` + `common/hadoop-secure.yaml` setup runs MapReduce
> smoke tests (PI, WordCount), but
> differs from a production NodeManager in ways that hide real costs:
> - It uses `DefaultContainerExecutor`, which localizes job files inside the
> long-lived NodeManager JVM. A secure
> NodeManager uses `LinuxContainerExecutor`, which starts a separate
> `ContainerLocalizer` JVM, with a
> fresh Ozone client, for each container. In our tests that cost about 2 s for
> the AppMaster and 2–3 s for each map
> container on another node. With `DefaultContainerExecutor` it was 0 s.
> - It has a single NodeManager, so maps never run on a node that still has to
> localize the job's files.
> - Its 4 GB map and reduce containers, Recon, S3 Gateway and HttpFS don't fit
> next to YARN in a devs Docker VM.
> Also, a pipeline limit of 1 leaves no RATIS/THREE pipeline when one datanode
> goes stale.
> - The `apache/hadoop` image ships `container-executor` without setuid, and
> on 3.4.3 (Ubuntu 22.04) it is linked
> against OpenSSL 1.1, which the image doesn't have. So
> `LinuxContainerExecutor` can't run as shipped.
> *Proposed change*
>
> A new compose environment, eg. `compose/ozonesecure-mr`, reusing
> `ozonesecure` and `common/hadoop-secure.yaml`:
> - *{*}NodeManagers:{*}* N of them (`nm`, `nm2`, …) with their own hostnames
> and the shared `nm/nm` principal. Each
> offers 2 GB of container memory, so a job's containers spread across nodes.
> - *{*}`LinuxContainerExecutor`:{*}* a NodeManager start script installs
> `container-executor` setuid root under a
> root-owned prefix, with its own `container-executor.cfg`. Where needed it
> adds `libssl1.1`. A small derived
> NodeManager image, or a fixed `apache/hadoop` image, would be cleaner than
> installing at start time.
> - *{*}Small cluster:{*}* only `kdc kms scm om datanode rm jhs nm*`, with
> `ozone.datanode.pipeline.limit=2` and
> 512–768 MB MapReduce containers.
> - *{*}`test-dfsio.sh`:{*}* runs TestDFSIO write/read on `ofs://` with the MR
> staging dir on Ozone. It collects the
> per-container logs and the NodeManager logs and summarizes per-job phases:
> AppMaster and map localization from
> the NodeManager's container state changes, AppMaster and task start-up from
> the container logs, and time outside
> TestDFSIO's own timer.
> - *{*}Not part of the default acceptance run,{*}* because it's
> resource-heavy. A 1-NodeManager, 1-job variant could
> serve as a smoke test.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
