This is an automated email from the ASF dual-hosted git repository. aw pushed a commit to branch YETUS-1034-release in repository https://gitbox.apache.org/repos/asf/yetus.git
commit 6539d2378945cb2468312adfaa6c28a0206f6bd5 Author: Allen Wittenauer <[email protected]> AuthorDate: Mon Nov 2 06:18:16 2020 -0800 YETUS-1063. Enhance documentaton for docker-cleanup and docker in gen… (#188) --- .../in-progress/precommit/docker-cleanup.html.md | 14 ++++- .../in-progress/precommit/docker.html.md | 64 ++++++++++++++++++++++ .../in-progress/precommit/robots/index.html.md | 4 +- 3 files changed, 80 insertions(+), 2 deletions(-) diff --git a/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md b/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md index c16af5c..59a5416 100644 --- a/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md +++ b/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md @@ -21,15 +21,25 @@ <!-- MarkdownTOC levels="1,2" autolink="true" indent=" " bullets="*" bracket="round" --> +* [Description](#description) * [Usage](#usage) + * [Default Mode](#default-mode) + * [Sentinel Mode](#sentinel-mode) <!-- /MarkdownTOC --> +# Description + `docker-cleanup` is a command to perform precommit's Docker cleanup functionality outside of patch and build testing. +It is designed to be run as a regularly automated job on CI systems that do not have their own Docker cleanup +facilities. It is built from the perspective that nothing on the system is permanent. It supports many of the same +options as `test-patch`'s [docker](../docker)-mode. Please see that page for more details. # Usage -Executing `docker-cleanup` will perform the same Docker image as the `--robot` option. +## Default Mode + +Executing `docker-cleanup` with no additional options will perform the same work as the `--robot` option. ```bash $ docker-cleanup @@ -45,6 +55,8 @@ Deleted: sha256:1a96c79a0a9ab538c6c7765dc908eca3e689270b778d6ae2add558e89792a7d8 ``` +## Sentinel Mode + `docker-cleanup` also supports the `--sentinel` mode to kill and remove stale running containers: ```bash diff --git a/asf-site-src/source/documentation/in-progress/precommit/docker.html.md b/asf-site-src/source/documentation/in-progress/precommit/docker.html.md index 592181c..30b454d 100644 --- a/asf-site-src/source/documentation/in-progress/precommit/docker.html.md +++ b/asf-site-src/source/documentation/in-progress/precommit/docker.html.md @@ -28,9 +28,15 @@ * [Pulling a Docker tag](#pulling-a-docker-tag) * [Using a cache](#using-a-cache) * [Platforms](#platforms) + * [Container Directory](#container-directory) * [Resource Controls](#resource-controls) * [Privileged Mode](#privileged-mode) * [Docker in Docker](#docker-in-docker) +* [Cleaning the Docker Environment](#cleaning-the-docker-environment) + * [Images](#images) + * [Containers](#containers) + * [Dry Running Cleaning](#dry-running-cleaning) + * [Standalone Facility](#standalone-facility) <!-- /MarkdownTOC --> @@ -68,6 +74,12 @@ With the `--docker-cache-from` parameter, other images may be utilized to provid When either building or pull an image, test-patch supports the `--docker-platform` flag to pass in the Docker `--platform` flag. This allows you full control over what kind of image the software either creates or fetches. +## Container Directory + +By default, precommit will use `/precommit` as the directory where it will store any necessary components that are +not provided by other flags in system (such as `--basedir` or `--patch-dir`). If that directory conflicts with some other +need, then the `--docker-work-dir` option may be provided to set a different path. + # Resource Controls Docker's `--memory` flag is supported via the `--dockermemlimit` option. This enables the container's memory size to be limited. This may be important to set to prevent things like broken unit tests bringing down the entire build server. See [the Docker documentation](https://docs.docker.com/engine/admin/resource_constraints/) for more details. Apache Yetus also sets the `--oom-score-adj` to 500 in order to offer itself as the first processes to be killed if memory is low. @@ -83,3 +95,55 @@ In some cases, the work being performed inside the Docker container requires ext With the usage of the `--dockerind` flag, test-patch will mount the `/var/run/docker.sock` UNIX socket into the container to enable Docker-in-Docker mode. Additionally, the `--docker-socket` option will let one set the socket to mount in situations where that isn't the location of the socket, such as a dockerd proxy providing authentication. NOTE: Using --dockerind requires the availability of the `stat` command that supports either -c '%g' (GNU form) or -f '%g' (BSD form). + +# Cleaning the Docker Environment + +With the growing use of Docker for CI purposes, so is the growing need to help manage the Docker environment. Stale, large +caches and stuck containers are a common problem when tests go haywire. In order to help combat that, Apache Yetus includes +facilities as part of precommit to help keep these things under control. By default, none of these cleaning actions occur. +However if the system is in [Robot](../robots)-mode or `--sentinel` is used, carefully controlled cleaning will happen. + + NOTE: Docker volumes are not managed by Apache Yetus. + +## Images + +For container images, there are three modes. Each mode increases the amount of coverage that Apache Yetus will use to +remove images. + +| Mode | Images Types | Time Frame | +|:----:|:------------:|:----------:| +| default | all | NA | +| `--robot` | dangling, label=org.apache.yetus | 1 week | +| `--sentinel` | all | 1 week | + +Before it begins, Apache Yetus will show the contents of `docker images` so that there is a history of what was in +the cache. This also helps determine what was deleted later. Next, it finds images that fall within the time frame of +the given mode. Then it will untag those images and followed by trying to remove them by sha. If an +image is in use, an error will be reported but ignored by Apache Yetus. It is an information message to help give the user +an idea of what is happening under the hood. + +## Containers + +For containers there are two modes: + +| Mode | Images Types | Time Frame | +|:----:|:------------:|:----------:| +| default | all | NA | +| `--sentinel` | all | 24 hours | + +Under just plain `--robot`, containers are left alone. Under `--sentinel`, containers (regardless of state) are +killed after 24 hours. + +Just as with images, Apache Yetus will report all the running containers on the system. Then it will, if +necessary, send a `docker kill` followed by a `docker rm` remove any containers that are over the runtime +limit. + +## Dry Running Cleaning + +To see what the system would have done, the `--dockerdelrep` option is provided to report, but not act, on deletions. + +## Standalone Facility + +By popular demand, Apache Yetus provides the [docker-cleanup](../docker-cleanup) program to perform these functions +outside of running jobs. This program is useful for CI systems that do not regularly run Apache Yetus or for +simply consolidating the maintenance into one location. diff --git a/asf-site-src/source/documentation/in-progress/precommit/robots/index.html.md b/asf-site-src/source/documentation/in-progress/precommit/robots/index.html.md index 6f63bca..41431ae 100644 --- a/asf-site-src/source/documentation/in-progress/precommit/robots/index.html.md +++ b/asf-site-src/source/documentation/in-progress/precommit/robots/index.html.md @@ -76,7 +76,9 @@ If `--robot` is specified without an instance, a random number is generated and # Sentinel Mode -If stuck Docker containers are a problem, a more aggressive robot may be enabled with the `--sentinel` option. This option enables killing containers that have been running for over 24 hours as well. +If stuck Docker containers are a problem, a more aggressive robot may be enabled with the `--sentinel` option. This option +enables killing containers that have been running for over 24 hours as well. See the documentation on [docker](../docker) +for more information. # Continuous Improvement
