errose28 commented on code in PR #101:
URL: https://github.com/apache/ozone-site/pull/101#discussion_r1830234822
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to
wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other
pre-built image).
- Changing configurations in Docker Compose (`docker-config` file,
transformation.py, etc).
+
Review Comment:
We can delete the TODO above this section since we are filling in the page
in this PR.
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to
wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other
pre-built image).
- Changing configurations in Docker Compose (`docker-config` file,
transformation.py, etc).
+
+This guide explains how to run Apache Ozone using Docker Compose, either with
locally built sources or pre-built images.
+
+## Prerequisites
+
+- Docker Engine 20.10.0 or higher
+- Docker Compose V2
+- Built Ozone distribution (if running from local build)
+
+## Running Ozone
+
+You can run Ozone either using your locally built version or using official
pre-built Docker images.
Review Comment:
I think we should organize this a little differently. We can have one
section on running docker-compose from a pre-build Ozone tarball, regardless of
where the tarball came from (built from source or Ozone release). Then we can
mention the pre-built docker images for each release separately if people want
to use those for other things.
I don't see why someone would pull just the compose file from the tarball
and manually pair it with a docker image built from the same version.
##########
package.json:
##########
@@ -1,5 +1,5 @@
{
- "packageManager": "[email protected]",
+ "packageManager": "[email protected]",
Review Comment:
Can you clarify why we need the pnpm version bump? Is mermaid incompatible
with older versions?
##########
cspell.yaml:
##########
@@ -48,6 +48,7 @@ words:
- ratis
- OM
- SCM
+- DN
Review Comment:
Unlike OM and SCM we don't usually use `DN` as an abbreviation for datanode
when talking about Ozone in an official context, so it was deliberately omitted
here. It might be better to use an inline ignore to skip it in the mermaid code
block.
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to
wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other
pre-built image).
- Changing configurations in Docker Compose (`docker-config` file,
transformation.py, etc).
+
+This guide explains how to run Apache Ozone using Docker Compose, either with
locally built sources or pre-built images.
+
+## Prerequisites
+
+- Docker Engine 20.10.0 or higher
+- Docker Compose V2
+- Built Ozone distribution (if running from local build)
+
+## Running Ozone
+
+You can run Ozone either using your locally built version or using official
pre-built Docker images.
+
+### Option 1: Running from Local Build
+
+If you've built Ozone from source, follow these steps:
+
+1. Navigate to the compose directory in your build output:
+
+ ```bash
+ cd hadoop-ozone/dist/target/ozone-*-SNAPSHOT/compose/ozone
+ ```
+
+2. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+The local build uses the `ozone-docker-runner` image, which is automatically
created during the build process to wrap your compiled Ozone binaries.
+
+### Option 2: Running from Pre-built Images
+
+If you haven't built Ozone locally, you can quickly start a cluster using
official pre-built images from Docker Hub:
+
+1. Create a directory for your Ozone deployment:
+
+ ```bash
+ mkdir ozone && cd ozone
+ ```
+
+2. Download the example compose file:
+
+ ```bash
+ curl \
+
https://raw.githubusercontent.com/apache/ozone/master/hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml
\
+ -o docker-compose.yaml
+ ```
+
+3. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+This will pull the official Apache Ozone images from Docker Hub.
+
+## Architecture Diagram
+
+This image shows the containers that will be created when running the
`docker-compose up -d` command.
+
+```mermaid
+graph TB
+ subgraph "Apache Ozone Architecture"
+ scm["SCM<br/>Storage Container Manager<br/>(Port: 9876, 9860)"]
+ om["OM<br/>Ozone Manager<br/>(Port: 9874, 9862)"]
+ s3g["S3G<br/>S3 Gateway<br/>(Port: 9878)"]
+ recon["Recon<br/>Monitoring Service<br/>(Port: 9888)"]
+ httpfs["HttpFS<br/>HTTP FileSystem<br/>(Port: 14000)"]
+
+ subgraph "Data Nodes"
+ dn1["DataNode 1<br/>(Port: 19864, 9882)"]
+ dnN["DataNode N<br/>(Port: 19864, 9882)"]
+ end
+
+ %% Connections
+ scm --> dn1
+ scm --> dnN
+ om --> dn1
+ om --> dnN
+ s3g --> om
+ s3g --> scm
+ httpfs --> om
+ httpfs --> scm
+ recon --> scm
+ recon --> om
+ end
+
+ classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px;
+ classDef datanode fill:#e1f5fe,stroke:#0288d1,stroke-width:2px;
+ classDef manager fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ class dn1,dnN datanode;
+ class scm,om manager;
+```
+
+## Cluster Configuration
+
+### Default Services
+
+The default Docker Compose configuration includes:
+
+- Storage Container Manager (SCM)
+- Ozone Manager (OM)
+- S3 Gateway
+- Recon (Monitoring Service)
+- Datanodes
+- HttpFS
+
+### Port Mappings
+
+The default setup exposes the following ports:
Review Comment:
I'm not sure the RPC connections will work with outside clients. the
configuration and hostnames would need to be configured manually. HTTP
endpoints should work out of the box though.
##########
package.json:
##########
@@ -16,22 +16,23 @@
},
"dependencies": {
"@docusaurus/core": "3.3.2",
- "@docusaurus/preset-classic": "3.3.2",
"@docusaurus/plugin-pwa": "3.3.2",
+ "@docusaurus/preset-classic": "3.3.2",
+ "@docusaurus/theme-mermaid": "3.3.2",
Review Comment:
I checked out this change and ran `docker compse up --build` to build an
image with the new dependencies but got this error. Not sure exactly what the
cause is:
```
ozone-site-site-1 |
ozone-site-site-1 | > [email protected] start /ozone-site
ozone-site-site-1 | > docusaurus start --port 3001 "--host" "0.0.0.0"
ozone-site-site-1 |
ozone-site-site-1 | [INFO] Starting the development server...
ozone-site-site-1 |
ozone-site-site-1 | [ERROR] Error: Docusaurus was unable to resolve the
"@docusaurus/theme-mermaid" theme. Make sure one of the following packages are
installed:
ozone-site-site-1 | - @docusaurus/theme-mermaid
ozone-site-site-1 | - @docusaurus/docusaurus-theme-theme-mermaid
ozone-site-site-1 | at resolveModuleName
(/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/plugins/moduleShorthand.js:41:15)
ozone-site-site-1 | at normalizeShorthand
(/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/plugins/configs.js:76:60)
ozone-site-site-1 | at
/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/plugins/configs.js:90:63
ozone-site-site-1 | at Array.map (<anonymous>)
ozone-site-site-1 | at loadPluginConfigs
(/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/plugins/configs.js:90:48)
ozone-site-site-1 | at async initPlugins
(/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/plugins/init.js:42:27)
ozone-site-site-1 | at async
/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/plugins/plugins.js:136:36
ozone-site-site-1 | at async loadSite
(/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/server/site.js:127:45)
ozone-site-site-1 | at async createReloadableSite
(/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/commands/start/utils.js:46:16)
ozone-site-site-1 | at async Command.start
(/ozone-site/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]/node_modules/@docusaurus/core/lib/commands/start/start.js:22:28)
ozone-site-site-1 | [INFO] Docusaurus version: 3.3.2
ozone-site-site-1 | Node version: v20.18.0
ozone-site-site-1 | ELIFECYCLE Command failed with exit code 1.
ozone-site-site-1 exited with code 1
```
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to
wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other
pre-built image).
- Changing configurations in Docker Compose (`docker-config` file,
transformation.py, etc).
+
+This guide explains how to run Apache Ozone using Docker Compose, either with
locally built sources or pre-built images.
+
+## Prerequisites
+
+- Docker Engine 20.10.0 or higher
+- Docker Compose V2
Review Comment:
Let's link to the relevant pages on the docker website for installing these
components.
##########
docs/06-troubleshooting/15-docker.md:
##########
@@ -0,0 +1,7 @@
+---
+sidebar_label: Docker
+---
+
+# Troubleshooting Docker
Review Comment:
```suggestion
# Troubleshooting Ozone in Docker
```
Or some similar title to indicate we aren't troubleshooting docker in
general.
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to
wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other
pre-built image).
- Changing configurations in Docker Compose (`docker-config` file,
transformation.py, etc).
+
+This guide explains how to run Apache Ozone using Docker Compose, either with
locally built sources or pre-built images.
+
+## Prerequisites
+
+- Docker Engine 20.10.0 or higher
+- Docker Compose V2
+- Built Ozone distribution (if running from local build)
+
+## Running Ozone
+
+You can run Ozone either using your locally built version or using official
pre-built Docker images.
+
+### Option 1: Running from Local Build
+
+If you've built Ozone from source, follow these steps:
+
+1. Navigate to the compose directory in your build output:
+
+ ```bash
+ cd hadoop-ozone/dist/target/ozone-*-SNAPSHOT/compose/ozone
+ ```
+
+2. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+The local build uses the `ozone-docker-runner` image, which is automatically
created during the build process to wrap your compiled Ozone binaries.
+
+### Option 2: Running from Pre-built Images
+
+If you haven't built Ozone locally, you can quickly start a cluster using
official pre-built images from Docker Hub:
+
+1. Create a directory for your Ozone deployment:
+
+ ```bash
+ mkdir ozone && cd ozone
+ ```
+
+2. Download the example compose file:
+
+ ```bash
+ curl \
+
https://raw.githubusercontent.com/apache/ozone/master/hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml
\
+ -o docker-compose.yaml
+ ```
+
+3. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+This will pull the official Apache Ozone images from Docker Hub.
+
+## Architecture Diagram
+
+This image shows the containers that will be created when running the
`docker-compose up -d` command.
+
+```mermaid
+graph TB
+ subgraph "Apache Ozone Architecture"
+ scm["SCM<br/>Storage Container Manager<br/>(Port: 9876, 9860)"]
+ om["OM<br/>Ozone Manager<br/>(Port: 9874, 9862)"]
+ s3g["S3G<br/>S3 Gateway<br/>(Port: 9878)"]
+ recon["Recon<br/>Monitoring Service<br/>(Port: 9888)"]
+ httpfs["HttpFS<br/>HTTP FileSystem<br/>(Port: 14000)"]
+
+ subgraph "Data Nodes"
+ dn1["DataNode 1<br/>(Port: 19864, 9882)"]
+ dnN["DataNode N<br/>(Port: 19864, 9882)"]
+ end
+
+ %% Connections
+ scm --> dn1
+ scm --> dnN
+ om --> dn1
+ om --> dnN
+ s3g --> om
+ s3g --> scm
+ httpfs --> om
+ httpfs --> scm
+ recon --> scm
+ recon --> om
+ end
+
+ classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px;
+ classDef datanode fill:#e1f5fe,stroke:#0288d1,stroke-width:2px;
+ classDef manager fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ class dn1,dnN datanode;
+ class scm,om manager;
+```
+
+## Cluster Configuration
+
+### Default Services
+
+The default Docker Compose configuration includes:
+
+- Storage Container Manager (SCM)
+- Ozone Manager (OM)
+- S3 Gateway
+- Recon (Monitoring Service)
+- Datanodes
+- HttpFS
+
+### Port Mappings
+
+The default setup exposes the following ports:
+
+- SCM: 9876 (RPC), 9860 (Client)
+- OM: 9874 (RPC), 9862 (Client)
+- S3 Gateway: 9878
+- Recon: 9888
+- HttpFS: 14000
+- Datanodes: 19864 (Container), 9882 (Client)
+
+## Cluster Management
+
+Common Docker Compose commands:
+
+```bash
+# Start the cluster
+docker-compose up -d
+
+# Stop the cluster
+docker-compose down
+
+# View service logs
+docker-compose logs -f [service_name]
+
+# Scale data nodes
+docker-compose up -d --scale datanode=3
Review Comment:
This only works on clusters that provide a service called `datanode`. Some
of the compose clusters like `topology` and `upgrade` that need to use
individual datanodes differently give each datanode a different service.
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to
wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other
pre-built image).
- Changing configurations in Docker Compose (`docker-config` file,
transformation.py, etc).
+
+This guide explains how to run Apache Ozone using Docker Compose, either with
locally built sources or pre-built images.
+
+## Prerequisites
+
+- Docker Engine 20.10.0 or higher
+- Docker Compose V2
+- Built Ozone distribution (if running from local build)
+
+## Running Ozone
+
+You can run Ozone either using your locally built version or using official
pre-built Docker images.
+
+### Option 1: Running from Local Build
+
+If you've built Ozone from source, follow these steps:
+
+1. Navigate to the compose directory in your build output:
+
+ ```bash
+ cd hadoop-ozone/dist/target/ozone-*-SNAPSHOT/compose/ozone
+ ```
+
+2. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+The local build uses the `ozone-docker-runner` image, which is automatically
created during the build process to wrap your compiled Ozone binaries.
+
+### Option 2: Running from Pre-built Images
+
+If you haven't built Ozone locally, you can quickly start a cluster using
official pre-built images from Docker Hub:
+
+1. Create a directory for your Ozone deployment:
+
+ ```bash
+ mkdir ozone && cd ozone
+ ```
+
+2. Download the example compose file:
+
+ ```bash
+ curl \
+
https://raw.githubusercontent.com/apache/ozone/master/hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml
\
+ -o docker-compose.yaml
+ ```
+
+3. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+This will pull the official Apache Ozone images from Docker Hub.
+
+## Architecture Diagram
+
+This image shows the containers that will be created when running the
`docker-compose up -d` command.
+
+```mermaid
+graph TB
+ subgraph "Apache Ozone Architecture"
+ scm["SCM<br/>Storage Container Manager<br/>(Port: 9876, 9860)"]
+ om["OM<br/>Ozone Manager<br/>(Port: 9874, 9862)"]
+ s3g["S3G<br/>S3 Gateway<br/>(Port: 9878)"]
+ recon["Recon<br/>Monitoring Service<br/>(Port: 9888)"]
+ httpfs["HttpFS<br/>HTTP FileSystem<br/>(Port: 14000)"]
+
+ subgraph "Data Nodes"
+ dn1["DataNode 1<br/>(Port: 19864, 9882)"]
+ dnN["DataNode N<br/>(Port: 19864, 9882)"]
+ end
+
+ %% Connections
+ scm --> dn1
+ scm --> dnN
+ om --> dn1
+ om --> dnN
+ s3g --> om
+ s3g --> scm
+ httpfs --> om
+ httpfs --> scm
+ recon --> scm
+ recon --> om
+ end
+
+ classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px;
+ classDef datanode fill:#e1f5fe,stroke:#0288d1,stroke-width:2px;
+ classDef manager fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ class dn1,dnN datanode;
+ class scm,om manager;
+```
+
+## Cluster Configuration
+
+### Default Services
+
+The default Docker Compose configuration includes:
+
+- Storage Container Manager (SCM)
+- Ozone Manager (OM)
+- S3 Gateway
+- Recon (Monitoring Service)
+- Datanodes
+- HttpFS
+
+### Port Mappings
+
+The default setup exposes the following ports:
+
+- SCM: 9876 (RPC), 9860 (Client)
+- OM: 9874 (RPC), 9862 (Client)
+- S3 Gateway: 9878
+- Recon: 9888
+- HttpFS: 14000
+- Datanodes: 19864 (Container), 9882 (Client)
+
+## Cluster Management
+
+Common Docker Compose commands:
+
+```bash
+# Start the cluster
+docker-compose up -d
Review Comment:
Lets use `docker compose` since v2 is the standard going forward.
##########
docs/08-developer-guide/02-run/02-docker-compose.md:
##########
@@ -10,3 +10,143 @@ sidebar_label: Docker Compose
- Define the Ozone runner image at `ozone-docker-runner` and its purpose to
wrap Ozone binaries.
- How to define which image gets used in the cluster (runner or other
pre-built image).
- Changing configurations in Docker Compose (`docker-config` file,
transformation.py, etc).
+
+This guide explains how to run Apache Ozone using Docker Compose, either with
locally built sources or pre-built images.
+
+## Prerequisites
+
+- Docker Engine 20.10.0 or higher
+- Docker Compose V2
+- Built Ozone distribution (if running from local build)
+
+## Running Ozone
+
+You can run Ozone either using your locally built version or using official
pre-built Docker images.
+
+### Option 1: Running from Local Build
+
+If you've built Ozone from source, follow these steps:
+
+1. Navigate to the compose directory in your build output:
+
+ ```bash
+ cd hadoop-ozone/dist/target/ozone-*-SNAPSHOT/compose/ozone
+ ```
+
+2. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+The local build uses the `ozone-docker-runner` image, which is automatically
created during the build process to wrap your compiled Ozone binaries.
+
+### Option 2: Running from Pre-built Images
+
+If you haven't built Ozone locally, you can quickly start a cluster using
official pre-built images from Docker Hub:
+
+1. Create a directory for your Ozone deployment:
+
+ ```bash
+ mkdir ozone && cd ozone
+ ```
+
+2. Download the example compose file:
+
+ ```bash
+ curl \
+
https://raw.githubusercontent.com/apache/ozone/master/hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml
\
+ -o docker-compose.yaml
+ ```
+
+3. Start the cluster:
+
+ ```bash
+ docker-compose up -d
+ ```
+
+This will pull the official Apache Ozone images from Docker Hub.
+
+## Architecture Diagram
Review Comment:
To avoid duplicate content I think we should leave this information for the
page dedicated to Ozone architecture. Default ports will be covered in
`docs/administrator-guide/configuration/basic/network/`. This may mean we don't
need to add mermaid in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
