This is an automated email from the ASF dual-hosted git repository.
aw pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/yetus.git
The following commit(s) were added to refs/heads/main by this push:
new 8136270 YETUS-1063. Enhance documentaton for docker-cleanup and
docker in gen… (#188)
8136270 is described below
commit 8136270c5d32d3e973fb54221b00b1c406c3daab
Author: Allen Wittenauer <[email protected]>
AuthorDate: Mon Nov 2 06:18:16 2020 -0800
YETUS-1063. Enhance documentaton for docker-cleanup and docker in gen…
(#188)
---
.../in-progress/precommit/docker-cleanup.html.md | 14 ++++-
.../in-progress/precommit/docker.html.md | 64 ++++++++++++++++++++++
.../in-progress/precommit/robots/index.html.md | 4 +-
3 files changed, 80 insertions(+), 2 deletions(-)
diff --git
a/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md
b/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md
index c16af5c..59a5416 100644
---
a/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md
+++
b/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md
@@ -21,15 +21,25 @@
<!-- MarkdownTOC levels="1,2" autolink="true" indent=" " bullets="*"
bracket="round" -->
+* [Description](#description)
* [Usage](#usage)
+ * [Default Mode](#default-mode)
+ * [Sentinel Mode](#sentinel-mode)
<!-- /MarkdownTOC -->
+# Description
+
`docker-cleanup` is a command to perform precommit's Docker cleanup
functionality outside of patch and build testing.
+It is designed to be run as a regularly automated job on CI systems that do
not have their own Docker cleanup
+facilities. It is built from the perspective that nothing on the system is
permanent. It supports many of the same
+options as `test-patch`'s [docker](../docker)-mode. Please see that page for
more details.
# Usage
-Executing `docker-cleanup` will perform the same Docker image as the `--robot`
option.
+## Default Mode
+
+Executing `docker-cleanup` with no additional options will perform the same
work as the `--robot` option.
```bash
$ docker-cleanup
@@ -45,6 +55,8 @@ Deleted:
sha256:1a96c79a0a9ab538c6c7765dc908eca3e689270b778d6ae2add558e89792a7d8
```
+## Sentinel Mode
+
`docker-cleanup` also supports the `--sentinel` mode to kill and remove stale
running containers:
```bash
diff --git
a/asf-site-src/source/documentation/in-progress/precommit/docker.html.md
b/asf-site-src/source/documentation/in-progress/precommit/docker.html.md
index 592181c..30b454d 100644
--- a/asf-site-src/source/documentation/in-progress/precommit/docker.html.md
+++ b/asf-site-src/source/documentation/in-progress/precommit/docker.html.md
@@ -28,9 +28,15 @@
* [Pulling a Docker tag](#pulling-a-docker-tag)
* [Using a cache](#using-a-cache)
* [Platforms](#platforms)
+ * [Container Directory](#container-directory)
* [Resource Controls](#resource-controls)
* [Privileged Mode](#privileged-mode)
* [Docker in Docker](#docker-in-docker)
+* [Cleaning the Docker Environment](#cleaning-the-docker-environment)
+ * [Images](#images)
+ * [Containers](#containers)
+ * [Dry Running Cleaning](#dry-running-cleaning)
+ * [Standalone Facility](#standalone-facility)
<!-- /MarkdownTOC -->
@@ -68,6 +74,12 @@ With the `--docker-cache-from` parameter, other images may
be utilized to provid
When either building or pull an image, test-patch supports the
`--docker-platform` flag to pass in the Docker `--platform` flag. This allows
you full control over what kind of image the software either creates or fetches.
+## Container Directory
+
+By default, precommit will use `/precommit` as the directory where it will
store any necessary components that are
+not provided by other flags in system (such as `--basedir` or `--patch-dir`).
If that directory conflicts with some other
+need, then the `--docker-work-dir` option may be provided to set a different
path.
+
# Resource Controls
Docker's `--memory` flag is supported via the `--dockermemlimit` option. This
enables the container's memory size to be limited. This may be important to
set to prevent things like broken unit tests bringing down the entire build
server. See [the Docker
documentation](https://docs.docker.com/engine/admin/resource_constraints/) for
more details. Apache Yetus also sets the `--oom-score-adj` to 500 in order to
offer itself as the first processes to be killed if memory is low.
@@ -83,3 +95,55 @@ In some cases, the work being performed inside the Docker
container requires ext
With the usage of the `--dockerind` flag, test-patch will mount the
`/var/run/docker.sock` UNIX socket into the container to enable
Docker-in-Docker mode. Additionally, the `--docker-socket` option will let one
set the socket to mount in situations where that isn't the location of the
socket, such as a dockerd proxy providing authentication.
NOTE: Using --dockerind requires the availability of the `stat` command
that supports either -c '%g' (GNU form) or -f '%g' (BSD form).
+
+# Cleaning the Docker Environment
+
+With the growing use of Docker for CI purposes, so is the growing need to help
manage the Docker environment. Stale, large
+caches and stuck containers are a common problem when tests go haywire. In
order to help combat that, Apache Yetus includes
+facilities as part of precommit to help keep these things under control. By
default, none of these cleaning actions occur.
+However if the system is in [Robot](../robots)-mode or `--sentinel` is used,
carefully controlled cleaning will happen.
+
+ NOTE: Docker volumes are not managed by Apache Yetus.
+
+## Images
+
+For container images, there are three modes. Each mode increases the amount
of coverage that Apache Yetus will use to
+remove images.
+
+| Mode | Images Types | Time Frame |
+|:----:|:------------:|:----------:|
+| default | all | NA |
+| `--robot` | dangling, label=org.apache.yetus | 1 week |
+| `--sentinel` | all | 1 week |
+
+Before it begins, Apache Yetus will show the contents of `docker images` so
that there is a history of what was in
+the cache. This also helps determine what was deleted later. Next, it finds
images that fall within the time frame of
+the given mode. Then it will untag those images and followed by trying to
remove them by sha. If an
+image is in use, an error will be reported but ignored by Apache Yetus. It is
an information message to help give the user
+an idea of what is happening under the hood.
+
+## Containers
+
+For containers there are two modes:
+
+| Mode | Images Types | Time Frame |
+|:----:|:------------:|:----------:|
+| default | all | NA |
+| `--sentinel` | all | 24 hours |
+
+Under just plain `--robot`, containers are left alone. Under `--sentinel`,
containers (regardless of state) are
+killed after 24 hours.
+
+Just as with images, Apache Yetus will report all the running containers on
the system. Then it will, if
+necessary, send a `docker kill` followed by a `docker rm` remove any
containers that are over the runtime
+limit.
+
+## Dry Running Cleaning
+
+To see what the system would have done, the `--dockerdelrep` option is
provided to report, but not act, on deletions.
+
+## Standalone Facility
+
+By popular demand, Apache Yetus provides the
[docker-cleanup](../docker-cleanup) program to perform these functions
+outside of running jobs. This program is useful for CI systems that do not
regularly run Apache Yetus or for
+simply consolidating the maintenance into one location.
diff --git
a/asf-site-src/source/documentation/in-progress/precommit/robots/index.html.md
b/asf-site-src/source/documentation/in-progress/precommit/robots/index.html.md
index 6f63bca..41431ae 100644
---
a/asf-site-src/source/documentation/in-progress/precommit/robots/index.html.md
+++
b/asf-site-src/source/documentation/in-progress/precommit/robots/index.html.md
@@ -76,7 +76,9 @@ If `--robot` is specified without an instance, a random
number is generated and
# Sentinel Mode
-If stuck Docker containers are a problem, a more aggressive robot may be
enabled with the `--sentinel` option. This option enables killing containers
that have been running for over 24 hours as well.
+If stuck Docker containers are a problem, a more aggressive robot may be
enabled with the `--sentinel` option. This option
+enables killing containers that have been running for over 24 hours as well.
See the documentation on [docker](../docker)
+for more information.
# Continuous Improvement
