This is an automated email from the ASF dual-hosted git repository.
bneradt pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/trafficserver-ci.git
The following commit(s) were added to refs/heads/main by this push:
new 0152c98 Add controller GitHub mirror tooling (#435)
0152c98 is described below
commit 0152c98d9836b56631a2ad1a7eac9bdbf2b28560
Author: Brian Neradt <[email protected]>
AuthorDate: Wed Jun 10 15:45:13 2026 -0500
Add controller GitHub mirror tooling (#435)
CI jobs currently clone ATS and trafficserver-ci directly from GitHub, which
makes every docker agent depend on external fetches and leaves PR checkout
speed at the mercy of GitHub and VM networking.
This adds a self-contained github-mirror package with controller install,
mirror update, webhook, cron fallback, and verification tooling. It also
moves Jenkins checkout URLs to the controller mirrors and adds PR mirror
readiness checks before GitHub build fanout.
Co-authored-by: bneradt <[email protected]>
---
github-mirror/README.md | 449 +++++++++++++++++++++
github-mirror/ats/remap-snippet.config | 6 +
github-mirror/bin/check-docker-access.sh | 68 ++++
github-mirror/bin/check-mirror.sh | 83 ++++
github-mirror/bin/github-mirror-webhook.py | 223 ++++++++++
github-mirror/bin/init-mirrors.sh | 140 +++++++
github-mirror/bin/install-controller.sh | 144 +++++++
github-mirror/bin/update-mirror.sh | 218 ++++++++++
github-mirror/cron/github-mirror | 15 +
.../env/github-mirror-webhook.env.example | 14 +
github-mirror/git-daemon/git-daemon.default | 7 +
github-mirror/httpd/mirror.conf | 14 +
.../systemd/github-mirror-fallback.service | 14 +
github-mirror/systemd/github-mirror-fallback.timer | 12 +
.../systemd/github-mirror-webhook.service | 24 ++
jenkins/branch/autest.pipeline | 2 +-
jenkins/branch/clang_analyzer.pipeline | 2 +-
jenkins/branch/cmake.pipeline | 2 +-
jenkins/branch/coverage.pipeline | 2 +-
jenkins/branch/coverity.pipeline | 4 +-
jenkins/branch/docs.pipeline | 2 +-
jenkins/branch/format.pipeline | 2 +-
jenkins/branch/freebsd.pipeline | 2 +-
jenkins/branch/in_tree.pipeline | 2 +-
jenkins/branch/os_build.pipeline | 2 +-
jenkins/branch/osx-m1.pipeline | 2 +-
jenkins/branch/osx.pipeline | 2 +-
jenkins/branch/out_of_tree.pipeline | 2 +-
jenkins/branch/quiche.pipeline | 2 +-
jenkins/branch/rat.pipeline | 2 +-
jenkins/github/github_polling.pipeline | 2 +-
jenkins/github/toplevel.pipeline | 2 +-
32 files changed, 1449 insertions(+), 18 deletions(-)
diff --git a/github-mirror/README.md b/github-mirror/README.md
new file mode 100644
index 0000000..ec09a39
--- /dev/null
+++ b/github-mirror/README.md
@@ -0,0 +1,449 @@
+# GitHub Mirror For ATS CI
+
+This directory is the complete source of truth for the GitHub mirror used by
+the Apache Traffic Server Jenkins controller.
+
+The goal is to avoid every Jenkins docker agent cloning directly from GitHub.
+GitHub sends webhook deliveries to the controller, the controller updates local
+bare mirrors under `/home/mirror`, and Jenkins jobs clone from:
+
+```text
+https://ci.trafficserver.apache.org/mirror/trafficserver.git
+https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git
+```
+
+The package is intentionally self-contained. If the controller is lost, this
+README plus the files in this directory are enough to rebuild the mirror.
+
+## Architecture
+
+```text
+GitHub
+ |
+ | push and pull_request webhooks
+ v
+https://ci.trafficserver.apache.org/github-mirror-webhook
+ |
+ | ATS remap
+ v
+127.0.0.1:9419/github-mirror-webhook
+ |
+ | signed webhook receiver, running as gitdaemon
+ v
+/home/mirror/trafficserver.git
+/home/mirror/trafficserver-ci.git
+ |
+ | httpd export under /mirror/ plus git-daemon on 9418
+ v
+Jenkins controller and docker agents
+```
+
+The webhook service only accepts signed GitHub payloads for:
+
+- `apache/trafficserver`
+- `apache/trafficserver-ci`
+
+`apache/trafficserver` mirrors heads, tags, and pull request refs. The PR refs
+are required because Jenkins PR jobs receive `GITHUB_PR_HEAD_SHA` and then run
a
+`GitSCM` checkout from the mirror.
+
+`apache/trafficserver-ci` mirrors heads and tags.
+
+## ASF Infra Request
+
+Ask ASF Infra to add the following GitHub webhooks.
+
+```text
+Hello ASF Infra,
+
+The Apache Traffic Server project would like GitHub webhooks added for our
+Jenkins Git mirror on ci.trafficserver.apache.org.
+
+Payload URL:
+ https://ci.trafficserver.apache.org/github-mirror-webhook
+
+Content type:
+ application/json
+
+Secret:
+ Please generate a webhook secret or coordinate it with us out of band.
+ We will install it only on the Jenkins controller in:
+ /etc/trafficserver-github-mirror/github-mirror-webhook.env
+
+Repositories and events:
+ apache/trafficserver:
+ - ping
+ - push
+ - pull_request
+
+ apache/trafficserver-ci:
+ - ping
+ - push
+
+Rationale:
+ Our Jenkins jobs run on a fleet of docker hosts behind the controller. The
+ jobs currently clone repeatedly from GitHub. We are moving those checkouts to
+ a local read-only mirror on the controller. The webhook keeps branch and pull
+ request refs current before Jenkins fans work out to the docker hosts.
+
+Thanks.
+```
+
+## Fresh Controller Install
+
+These steps assume Ubuntu and a controller that will serve
+`ci.trafficserver.apache.org` through ATS.
+
+1. Clone or copy `trafficserver-ci` onto the controller.
+
+ ```bash
+ git clone https://github.com/apache/trafficserver-ci.git
/tmp/trafficserver-ci
+ cd /tmp/trafficserver-ci
+ ```
+
+2. Install the mirror package.
+
+ ```bash
+ sudo github-mirror/bin/install-controller.sh
+ ```
+
+ The installer:
+
+ - installs `git`, `git-daemon-sysvinit`, `python3`, and `util-linux`;
+ - installs this package to `/opt/trafficserver-ci/github-mirror`;
+ - creates/configures `/home/mirror`;
+ - creates `/home/mirror/trafficserver.git`;
+ - creates `/home/mirror/trafficserver-ci.git`;
+ - installs systemd units;
+ - installs `/etc/default/git-daemon`;
+ - enables the fallback refresh timer.
+
+3. Install the GitHub webhook secret.
+
+ ```bash
+ sudo install -d -m 0700 /etc/trafficserver-github-mirror
+ sudo editor /etc/trafficserver-github-mirror/github-mirror-webhook.env
+ ```
+
+ Set:
+
+ ```text
+ GITHUB_WEBHOOK_SECRET=<secret from ASF Infra>
+ ```
+
+ Keep the file root-owned and private:
+
+ ```bash
+ sudo chown root:root
/etc/trafficserver-github-mirror/github-mirror-webhook.env
+ sudo chmod 0600 /etc/trafficserver-github-mirror/github-mirror-webhook.env
+ ```
+
+4. Configure the HTTPS webhook endpoint in ATS.
+
+ Add `github-mirror/ats/remap-snippet.config` before the generic
+ `ci.trafficserver.apache.org` Jenkins remap in:
+
+ ```text
+ /opt/ats/etc/trafficserver/remap.config
+ ```
+
+ Reload ATS:
+
+ ```bash
+ sudo /opt/ats/bin/traffic_ctl config reload
+ ```
+
+5. Export `/home/mirror` as `/mirror/`.
+
+ If the controller already has httpd serving `/mirror/`, keep that setup.
+ For a fresh controller, use `github-mirror/httpd/mirror.conf` as the
+ reference config for the httpd instance behind ATS. The updater runs
+ `git update-server-info`, so a static HTTP export is sufficient.
+
+6. Start the webhook receiver.
+
+ ```bash
+ sudo systemctl restart github-mirror-webhook.service
+ sudo systemctl status github-mirror-webhook.service
+ ```
+
+7. Confirm the timer and git-daemon are active.
+
+ ```bash
+ systemctl list-timers github-mirror-fallback.timer
+ sudo service git-daemon status
+ ```
+
+## Interim Cron Rollout
+
+Use this section while ASF Infra is still setting up the GitHub webhooks. The
+cron updater keeps the mirrors fresh enough for Jenkins by fetching heads,
tags,
+and ATS pull request refs every five minutes.
+
+1. Install the current `trafficserver-ci` branch on `controller`.
+
+ ```bash
+ ssh controller
+ git clone https://github.com/apache/trafficserver-ci.git
/tmp/trafficserver-ci
+ cd /tmp/trafficserver-ci
+ ```
+
+ If testing a PR branch before merge, fetch that branch into this checkout
+ before running the installer.
+
+2. Install the mirror package but do not start the webhook service or systemd
+ fallback timer yet.
+
+ ```bash
+ sudo START_WEBHOOK=0 START_FALLBACK_TIMER=0 \
+ github-mirror/bin/install-controller.sh
+ ```
+
+ This initializes `/home/mirror/trafficserver.git` and
+ `/home/mirror/trafficserver-ci.git`, installs the scripts under
+ `/opt/trafficserver-ci/github-mirror`, and starts `git-daemon`.
+
+3. Install the temporary cron file.
+
+ ```bash
+ sudo install -o root -g root -m 0644 \
+ /opt/trafficserver-ci/github-mirror/cron/github-mirror \
+ /etc/cron.d/github-mirror
+ sudo systemctl restart cron
+ ```
+
+4. Run one manual refresh and verify refs.
+
+ ```bash
+ sudo -u gitdaemon \
+ /opt/trafficserver-ci/github-mirror/bin/update-mirror.sh --all
+
+ /opt/trafficserver-ci/github-mirror/bin/check-mirror.sh
+ ```
+
+ If you have a current open ATS PR number, verify PR refs too:
+
+ ```bash
+ /opt/trafficserver-ci/github-mirror/bin/check-mirror.sh --pr <pr-number>
+ ```
+
+5. Verify from at least one docker host.
+
+ From any checkout of this repo on a host that can SSH through `controller`:
+
+ ```bash
+ github-mirror/bin/check-docker-access.sh docker12
+ ```
+
+ From `controller` itself:
+
+ ```bash
+ CONTROLLER=- \
+ /opt/trafficserver-ci/github-mirror/bin/check-docker-access.sh docker12
+ ```
+
+6. Update Jenkins job configuration so the PR and branch top-level jobs pass
+ the ATS mirror URL as `GITHUB_URL`. While the temporary one-minute cron is
+ active, set the GitHub PR top-level job quiet period to at least 90 seconds
+ so the mirror has time to fetch new PR refs before child jobs start.
+
+ ```text
+ GITHUB_URL=https://ci.trafficserver.apache.org/mirror/trafficserver.git
+ quietPeriod=90
+ ```
+
+ Then run a small PR job such as docs or RAT before starting the full build
+ fanout.
+
+7. Watch the cron updater and Jenkins checkouts.
+
+ ```bash
+ grep github-mirror /var/log/syslog
+ git ls-remote https://ci.trafficserver.apache.org/mirror/trafficserver.git \
+ refs/heads/master
+ ```
+
+8. When ASF webhooks are available, install the secret, start the webhook, send
+ a GitHub ping delivery, then remove the temporary cron file.
+
+ ```bash
+ sudo systemctl restart github-mirror-webhook.service
+ sudo rm -f /etc/cron.d/github-mirror
+ sudo systemctl restart cron
+ ```
+
+## Mirror Operations
+
+Initialize or reconfigure the mirrors:
+
+```bash
+sudo /opt/trafficserver-ci/github-mirror/bin/init-mirrors.sh
+```
+
+Recreate mirrors from scratch:
+
+```bash
+sudo /opt/trafficserver-ci/github-mirror/bin/init-mirrors.sh --force
+```
+
+Refresh both mirrors manually:
+
+```bash
+sudo -u gitdaemon \
+ /opt/trafficserver-ci/github-mirror/bin/update-mirror.sh --all
+```
+
+Refresh one ATS PR:
+
+```bash
+sudo -u gitdaemon \
+ /opt/trafficserver-ci/github-mirror/bin/update-mirror.sh trafficserver --pr
12345
+```
+
+Check local and public refs:
+
+```bash
+/opt/trafficserver-ci/github-mirror/bin/check-mirror.sh --pr 12345
+```
+
+Check from docker agents:
+
+```bash
+/opt/trafficserver-ci/github-mirror/bin/check-docker-access.sh --pr 12345
docker1 docker12
+```
+
+When running that command directly on the controller, use:
+
+```bash
+CONTROLLER=- /opt/trafficserver-ci/github-mirror/bin/check-docker-access.sh
docker12
+```
+
+## Webhook Testing
+
+After ASF Infra adds the webhook, use the GitHub UI to send a `ping` delivery.
+The response should be HTTP 200 and the service logs should show `pong`.
+
+View logs:
+
+```bash
+journalctl -u github-mirror-webhook.service -f
+```
+
+A bad secret or unsigned payload should return HTTP 401 and must not update any
+repository.
+
+## Jenkins Integration
+
+Jenkins should clone from these URLs:
+
+```text
+https://ci.trafficserver.apache.org/mirror/trafficserver.git
+https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git
+```
+
+For GitHub PR jobs, configure the top-level job's `GITHUB_URL` parameter to:
+
+```text
+https://ci.trafficserver.apache.org/mirror/trafficserver.git
+```
+
+The top-level pipeline passes that value to child jobs. During the temporary
+cron rollout, set the top-level PR job quiet period to at least 90 seconds.
Once
+the webhook is live and verified, the quiet period can be removed or reduced.
+
+For branch jobs, configure the top-level branch jobs' `GITHUB_URL` parameter to
+the same ATS mirror URL. Child jobs will receive that value from the fanout
job.
+
+## Migrating An Existing Controller
+
+The historical controller may have older cron jobs such as:
+
+```text
+root crontab: */5 * * * * /admin/bin/update-mirrors.sh
+/etc/cron.d/mirror: * * * * * mirror /home/mirror/bin/gh-mirror.sh ...
+```
+
+During rollout, leave them in place until the webhook has processed real
+deliveries and Jenkins has completed at least one PR build from the mirror.
+
+After validation, disable the old jobs and keep only:
+
+```text
+github-mirror-fallback.timer
+```
+
+Suggested cleanup:
+
+```bash
+sudo crontab -e
+sudo rm -f /etc/cron.d/mirror
+sudo systemctl restart cron
+```
+
+Do not delete the old scripts until the new path has run for a few days.
+
+## Rollback
+
+1. Stop webhook updates.
+
+ ```bash
+ sudo systemctl stop github-mirror-webhook.service
+ sudo systemctl stop github-mirror-fallback.timer
+ ```
+
+2. Point Jenkins job parameters back at GitHub:
+
+ ```text
+ https://github.com/apache/trafficserver.git
+ https://github.com/apache/trafficserver-ci.git
+ ```
+
+3. If needed, re-enable the previous cron updater.
+
+Rollback does not require deleting `/home/mirror`.
+
+## Troubleshooting
+
+Missing PR ref:
+
+```bash
+sudo -u gitdaemon \
+ /opt/trafficserver-ci/github-mirror/bin/update-mirror.sh trafficserver --pr
<number>
+git --git-dir=/home/mirror/trafficserver.git show-ref refs/pull/<number>/head
+```
+
+Webhook returns 401:
+
+- Confirm ASF Infra and the controller have the same secret.
+- Confirm the env file is readable by systemd and not world-readable:
+
+ ```bash
+ sudo systemctl cat github-mirror-webhook.service
+ sudo ls -l /etc/trafficserver-github-mirror/github-mirror-webhook.env
+ ```
+
+Jenkins cannot clone from HTTPS:
+
+- Verify ATS remap order.
+- Verify the httpd `/mirror/` export.
+- Verify the public URL:
+
+ ```bash
+ git ls-remote https://ci.trafficserver.apache.org/mirror/trafficserver.git
refs/heads/master
+ ```
+
+Docker hosts cannot reach the mirror:
+
+```bash
+/opt/trafficserver-ci/github-mirror/bin/check-docker-access.sh docker12
+```
+
+Webhook service will not start:
+
+```bash
+journalctl -u github-mirror-webhook.service -n 100 --no-pager
+sudo systemctl status github-mirror-webhook.service
+```
+
+The service intentionally refuses to start when `GITHUB_WEBHOOK_SECRET` is
+unset or still set to `CHANGE_ME`.
diff --git a/github-mirror/ats/remap-snippet.config
b/github-mirror/ats/remap-snippet.config
new file mode 100644
index 0000000..753e443
--- /dev/null
+++ b/github-mirror/ats/remap-snippet.config
@@ -0,0 +1,6 @@
+# Put this before the generic Jenkins/ci.trafficserver.apache.org catch-all
remap.
+#
+# GitHub will POST webhook deliveries here. ATS forwards them to the local
+# webhook receiver, which validates X-Hub-Signature-256 before doing any work.
+map https://ci.trafficserver.apache.org/github-mirror-webhook
http://127.0.0.1:9419/github-mirror-webhook \
+ @action=allow @method=POST
diff --git a/github-mirror/bin/check-docker-access.sh
b/github-mirror/bin/check-docker-access.sh
new file mode 100755
index 0000000..6156f03
--- /dev/null
+++ b/github-mirror/bin/check-docker-access.sh
@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+#
+# Verify mirror access from Jenkins docker hosts.
+
+set -euo pipefail
+
+CONTROLLER=${CONTROLLER:-controller}
+PUBLIC_BASE=${PUBLIC_BASE:-https://ci.trafficserver.apache.org/mirror}
+PR_NUMBER=${PR_NUMBER:-}
+
+usage() {
+ cat <<'EOF'
+Usage:
+ check-docker-access.sh [--pr NUMBER] docker1 [docker2 ...]
+
+Runs git ls-remote checks on each docker host through the controller SSH hop.
+
+Environment:
+ CONTROLLER SSH ProxyJump host used to reach docker hosts. Default:
controller
+ Set to empty or "-" when running directly on the controller.
+ PUBLIC_BASE Public HTTPS mirror base URL. Default:
https://ci.trafficserver.apache.org/mirror
+ PR_NUMBER Optional PR number to verify.
+EOF
+}
+
+die() {
+ printf 'error: %s\n' "$*" >&2
+ exit 1
+}
+
+while [ $# -gt 0 ]; do
+ case "$1" in
+ --pr)
+ shift
+ [ $# -gt 0 ] || die "--pr requires a number"
+ PR_NUMBER=$1
+ ;;
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ *)
+ break
+ ;;
+ esac
+ shift
+done
+
+[ $# -gt 0 ] || die "provide at least one docker host"
+
+for docker_host in "$@"; do
+ printf 'checking %s via %s\n' "${docker_host}" "${CONTROLLER}" >&2
+ ssh_args=(-o BatchMode=yes)
+ if [ -n "${CONTROLLER}" ] && [ "${CONTROLLER}" != "-" ]; then
+ ssh_args+=(-J "${CONTROLLER}")
+ fi
+ ssh "${ssh_args[@]}" "${docker_host}" \
+ "PUBLIC_BASE=$(printf '%q' "${PUBLIC_BASE}") PR_NUMBER=$(printf '%q'
"${PR_NUMBER}") bash -s" <<'REMOTE_CHECK'
+set -e
+git ls-remote "$PUBLIC_BASE/trafficserver.git" refs/heads/master >/dev/null
+git ls-remote "$PUBLIC_BASE/trafficserver-ci.git" refs/heads/main >/dev/null
+if [ -n "$PR_NUMBER" ]; then
+ git ls-remote "$PUBLIC_BASE/trafficserver.git" "refs/pull/${PR_NUMBER}/head"
>/dev/null
+fi
+REMOTE_CHECK
+done
+
+printf 'docker mirror access checks passed\n' >&2
diff --git a/github-mirror/bin/check-mirror.sh
b/github-mirror/bin/check-mirror.sh
new file mode 100755
index 0000000..2078689
--- /dev/null
+++ b/github-mirror/bin/check-mirror.sh
@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+#
+# Smoke-check the local and public mirror endpoints.
+
+set -euo pipefail
+
+MIRROR_ROOT=${MIRROR_ROOT:-/home/mirror}
+PUBLIC_BASE=${PUBLIC_BASE:-https://ci.trafficserver.apache.org/mirror}
+GIT=${GIT:-git}
+PR_NUMBER=${PR_NUMBER:-}
+
+usage() {
+ cat <<'EOF'
+Usage:
+ check-mirror.sh [--pr NUMBER]
+
+Environment:
+ MIRROR_ROOT Local mirror root. Default: /home/mirror
+ PUBLIC_BASE Public HTTPS mirror base URL. Default:
https://ci.trafficserver.apache.org/mirror
+ GIT Git executable. Default: git
+ PR_NUMBER Optional PR number to verify.
+EOF
+}
+
+log() {
+ printf '%s\n' "$*" >&2
+}
+
+die() {
+ log "error: $*"
+ exit 1
+}
+
+while [ $# -gt 0 ]; do
+ case "$1" in
+ --pr)
+ shift
+ [ $# -gt 0 ] || die "--pr requires a number"
+ PR_NUMBER=$1
+ ;;
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ *)
+ die "unknown argument: $1"
+ ;;
+ esac
+ shift
+done
+
+check_local_ref() {
+ local repo=$1
+ local ref=$2
+ local repo_dir="${MIRROR_ROOT}/${repo}.git"
+ [ -d "${repo_dir}" ] || die "missing local mirror: ${repo_dir}"
+ "${GIT}" --git-dir="${repo_dir}" show-ref --verify --quiet "${ref}" ||
+ die "missing local ref ${ref} in ${repo}"
+ log "local ${repo} has ${ref}"
+}
+
+check_remote_ref() {
+ local repo=$1
+ local ref=$2
+ local url="${PUBLIC_BASE}/${repo}.git"
+ local output
+ output=$("${GIT}" ls-remote "${url}" "${ref}")
+ [ -n "${output}" ] || die "missing public ref ${ref} at ${url}"
+ log "public ${url} has ${ref}"
+}
+
+check_local_ref trafficserver refs/heads/master
+check_local_ref trafficserver-ci refs/heads/main
+check_remote_ref trafficserver refs/heads/master
+check_remote_ref trafficserver-ci refs/heads/main
+
+if [ -n "${PR_NUMBER}" ]; then
+ [[ "${PR_NUMBER}" =~ ^[0-9]+$ ]] || die "invalid PR number: ${PR_NUMBER}"
+ check_local_ref trafficserver "refs/pull/${PR_NUMBER}/head"
+ check_remote_ref trafficserver "refs/pull/${PR_NUMBER}/head"
+fi
+
+log "mirror checks passed"
diff --git a/github-mirror/bin/github-mirror-webhook.py
b/github-mirror/bin/github-mirror-webhook.py
new file mode 100755
index 0000000..b0593b4
--- /dev/null
+++ b/github-mirror/bin/github-mirror-webhook.py
@@ -0,0 +1,223 @@
+#!/usr/bin/env python3
+"""GitHub webhook receiver for the ATS CI Git mirrors."""
+
+from __future__ import annotations
+
+import hashlib
+import hmac
+import json
+import os
+import subprocess
+import sys
+import time
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from typing import Any
+from urllib.parse import urlsplit
+
+
+EXPECTED_REPOS = {
+ "apache/trafficserver": "trafficserver",
+ "apache/trafficserver-ci": "trafficserver-ci",
+}
+
+
+def getenv_int(name: str, default: int) -> int:
+ value = os.environ.get(name)
+ if value is None:
+ return default
+ try:
+ return int(value)
+ except ValueError:
+ raise SystemExit(f"{name} must be an integer")
+
+
+WEBHOOK_HOST = os.environ.get("WEBHOOK_HOST", "127.0.0.1")
+WEBHOOK_PORT = getenv_int("WEBHOOK_PORT", 9419)
+WEBHOOK_PATH = os.environ.get("WEBHOOK_PATH", "/github-mirror-webhook")
+WEBHOOK_SECRET = os.environ.get("GITHUB_WEBHOOK_SECRET")
+UPDATE_MIRROR = os.environ.get(
+ "UPDATE_MIRROR",
+ "/opt/trafficserver-ci/github-mirror/bin/update-mirror.sh",
+)
+MIRROR_ROOT = os.environ.get("MIRROR_ROOT", "/home/mirror")
+MAX_BODY_BYTES = getenv_int("MAX_BODY_BYTES", 1024 * 1024)
+UPDATE_TIMEOUT_SECONDS = getenv_int("UPDATE_TIMEOUT_SECONDS", 600)
+
+
+def log(message: str) -> None:
+ print(f"github-mirror-webhook: {message}", file=sys.stderr, flush=True)
+
+
+def verify_signature(body: bytes, header_value: str | None) -> bool:
+ if not WEBHOOK_SECRET or WEBHOOK_SECRET == "CHANGE_ME":
+ log("GITHUB_WEBHOOK_SECRET is not configured")
+ return False
+ if not header_value or not header_value.startswith("sha256="):
+ return False
+ expected = "sha256=" + hmac.new(
+ WEBHOOK_SECRET.encode("utf-8"),
+ body,
+ hashlib.sha256,
+ ).hexdigest()
+ return hmac.compare_digest(expected, header_value)
+
+
+def run_update(*args: str) -> str:
+ command = [UPDATE_MIRROR, *args]
+ env = os.environ.copy()
+ env["MIRROR_ROOT"] = MIRROR_ROOT
+
+ start = time.monotonic()
+ completed = subprocess.run(
+ command,
+ env=env,
+ text=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ timeout=UPDATE_TIMEOUT_SECONDS,
+ check=False,
+ )
+ elapsed = time.monotonic() - start
+ output = completed.stdout.strip()
+ if output:
+ for line in output.splitlines():
+ log(line)
+
+ if completed.returncode != 0:
+ raise RuntimeError(
+ f"{' '.join(command)} failed with exit {completed.returncode}"
+ )
+
+ return f"updated via {' '.join(command)} in {elapsed:.1f}s"
+
+
+def repo_name(payload: dict[str, Any]) -> str:
+ repository = payload.get("repository")
+ if not isinstance(repository, dict):
+ raise ValueError("payload missing repository object")
+
+ full_name = repository.get("full_name")
+ if not isinstance(full_name, str):
+ raise ValueError("payload missing repository.full_name")
+ if full_name not in EXPECTED_REPOS:
+ raise PermissionError(f"unexpected repository: {full_name}")
+ return full_name
+
+
+def handle_event(event: str, payload: dict[str, Any]) -> tuple[int, str]:
+ full_name = repo_name(payload)
+ mirror_repo = EXPECTED_REPOS[full_name]
+
+ if event == "ping":
+ return 200, f"pong for {full_name}\n"
+
+ if event == "push":
+ result = run_update(mirror_repo, "--heads-tags")
+ return 200, result + "\n"
+
+ if event == "pull_request":
+ if mirror_repo != "trafficserver":
+ raise ValueError("pull_request events are only accepted for
apache/trafficserver")
+
+ number = payload.get("number")
+ if not isinstance(number, int):
+ raise ValueError("payload missing integer pull request number")
+
+ action = payload.get("action")
+ if action == "closed":
+ result = run_update("trafficserver", "--delete-pr", str(number),
"--heads-tags")
+ else:
+ result = run_update("trafficserver", "--pr", str(number))
+ return 200, result + "\n"
+
+ raise ValueError(f"unsupported event: {event}")
+
+
+class WebhookHandler(BaseHTTPRequestHandler):
+ protocol_version = "HTTP/1.1"
+
+ def send_text(self, status: int, body: str) -> None:
+ encoded = body.encode("utf-8")
+ self.send_response(status)
+ self.send_header("Content-Type", "text/plain; charset=utf-8")
+ self.send_header("Content-Length", str(len(encoded)))
+ self.end_headers()
+ self.wfile.write(encoded)
+
+ def do_POST(self) -> None: # noqa: N802 - BaseHTTPRequestHandler API
+ path = urlsplit(self.path).path
+ if path != WEBHOOK_PATH:
+ self.send_text(404, "not found\n")
+ return
+
+ event = self.headers.get("X-GitHub-Event", "")
+ delivery = self.headers.get("X-GitHub-Delivery", "unknown-delivery")
+
+ try:
+ length = int(self.headers.get("Content-Length", "0"))
+ except ValueError:
+ self.send_text(400, "invalid content length\n")
+ return
+
+ if length <= 0:
+ self.send_text(400, "empty request body\n")
+ return
+ if length > MAX_BODY_BYTES:
+ self.send_text(413, "request body too large\n")
+ return
+
+ body = self.rfile.read(length)
+ if not verify_signature(body, self.headers.get("X-Hub-Signature-256")):
+ log(f"rejected delivery={delivery} event={event}: bad signature")
+ self.send_text(401, "bad signature\n")
+ return
+
+ try:
+ payload = json.loads(body.decode("utf-8"))
+ if not isinstance(payload, dict):
+ raise ValueError("payload root is not an object")
+ status, response = handle_event(event, payload)
+ log(f"accepted delivery={delivery} event={event}:
{response.strip()}")
+ self.send_text(status, response)
+ except PermissionError as exc:
+ log(f"rejected delivery={delivery} event={event}: {exc}")
+ self.send_text(403, f"{exc}\n")
+ except (ValueError, json.JSONDecodeError) as exc:
+ log(f"bad delivery={delivery} event={event}: {exc}")
+ self.send_text(400, f"{exc}\n")
+ except subprocess.TimeoutExpired:
+ log(f"timed out delivery={delivery} event={event}")
+ self.send_text(504, "mirror update timed out\n")
+ except Exception as exc: # Keep GitHub delivery diagnostics explicit.
+ log(f"failed delivery={delivery} event={event}: {exc}")
+ self.send_text(500, f"{exc}\n")
+
+ def do_GET(self) -> None: # noqa: N802 - BaseHTTPRequestHandler API
+ path = urlsplit(self.path).path
+ if path == "/healthz":
+ self.send_text(200, "ok\n")
+ else:
+ self.send_text(405, "method not allowed\n")
+
+ def log_message(self, fmt: str, *args: Any) -> None:
+ log(fmt % args)
+
+
+def main() -> int:
+ if not WEBHOOK_SECRET or WEBHOOK_SECRET == "CHANGE_ME":
+ log("refusing to start without GITHUB_WEBHOOK_SECRET")
+ return 1
+
+ server = ThreadingHTTPServer((WEBHOOK_HOST, WEBHOOK_PORT), WebhookHandler)
+ log(f"listening on http://{WEBHOOK_HOST}:{WEBHOOK_PORT}{WEBHOOK_PATH}";)
+ try:
+ server.serve_forever()
+ except KeyboardInterrupt:
+ log("shutting down")
+ finally:
+ server.server_close()
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/github-mirror/bin/init-mirrors.sh
b/github-mirror/bin/init-mirrors.sh
new file mode 100755
index 0000000..85243d7
--- /dev/null
+++ b/github-mirror/bin/init-mirrors.sh
@@ -0,0 +1,140 @@
+#!/usr/bin/env bash
+#
+# Create or refresh the bare mirrors used by the ATS Jenkins controller.
+
+set -euo pipefail
+
+SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+
+MIRROR_ROOT=${MIRROR_ROOT:-/home/mirror}
+MIRROR_USER=${MIRROR_USER:-gitdaemon}
+MIRROR_GROUP=${MIRROR_GROUP:-nogroup}
+GIT=${GIT:-git}
+FORCE=0
+FETCH_AFTER_INIT=1
+
+usage() {
+ cat <<'EOF'
+Usage:
+ init-mirrors.sh [--force] [--no-fetch]
+
+Creates/configures:
+ /home/mirror/trafficserver.git
+ /home/mirror/trafficserver-ci.git
+
+Options:
+ --force Remove existing mirror directories before reinitializing.
+ --no-fetch Configure repositories but do not fetch data.
+
+Environment:
+ MIRROR_ROOT Mirror root. Default: /home/mirror
+ MIRROR_USER Owner for mirror files. Default: gitdaemon
+ MIRROR_GROUP Group for mirror files. Default: nogroup
+ GIT Git executable. Default: git
+EOF
+}
+
+log() {
+ printf '%s\n' "$*" >&2
+}
+
+die() {
+ log "error: $*"
+ exit 1
+}
+
+run_as_mirror_user() {
+ if [ "$(id -u)" -eq 0 ]; then
+ runuser -u "${MIRROR_USER}" -- "$@"
+ else
+ "$@"
+ fi
+}
+
+configure_remote() {
+ local repo_dir=$1
+ local remote_url=$2
+ shift 2
+
+ "${GIT}" --git-dir="${repo_dir}" config remote.origin.url "${remote_url}"
+ "${GIT}" --git-dir="${repo_dir}" config --unset-all remote.origin.fetch
>/dev/null 2>&1 || true
+
+ local refspec
+ for refspec in "$@"; do
+ "${GIT}" --git-dir="${repo_dir}" config --add remote.origin.fetch
"${refspec}"
+ done
+}
+
+init_repo() {
+ local name=$1
+ local remote_url=$2
+ shift 2
+
+ local repo_dir="${MIRROR_ROOT}/${name}.git"
+
+ if [ "${FORCE}" -eq 1 ] && [ -e "${repo_dir}" ]; then
+ log "removing existing ${repo_dir}"
+ rm -rf "${repo_dir}"
+ fi
+
+ if [ ! -d "${repo_dir}" ]; then
+ log "creating ${repo_dir}"
+ run_as_mirror_user "${GIT}" init --bare "${repo_dir}"
+ fi
+
+ configure_remote "${repo_dir}" "${remote_url}" "$@"
+ touch "${repo_dir}/git-daemon-export-ok"
+
+ if [ "$(id -u)" -eq 0 ]; then
+ chown -R "${MIRROR_USER}:${MIRROR_GROUP}" "${repo_dir}"
+ fi
+
+ log "configured ${name} mirror"
+}
+
+while [ $# -gt 0 ]; do
+ case "$1" in
+ --force)
+ FORCE=1
+ ;;
+ --no-fetch)
+ FETCH_AFTER_INIT=0
+ ;;
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ *)
+ die "unknown argument: $1"
+ ;;
+ esac
+ shift
+done
+
+if ! id "${MIRROR_USER}" >/dev/null 2>&1; then
+ die "user ${MIRROR_USER} does not exist; run install-controller.sh first"
+fi
+
+install -d -o "${MIRROR_USER}" -g "${MIRROR_GROUP}" -m 0755 "${MIRROR_ROOT}"
+install -d -o "${MIRROR_USER}" -g "${MIRROR_GROUP}" -m 0755
"${MIRROR_ROOT}/.locks"
+
+init_repo trafficserver https://github.com/apache/trafficserver.git \
+ '+refs/heads/*:refs/heads/*' \
+ '+refs/tags/*:refs/tags/*' \
+ '+refs/pull/*:refs/pull/*'
+
+init_repo trafficserver-ci https://github.com/apache/trafficserver-ci.git \
+ '+refs/heads/*:refs/heads/*' \
+ '+refs/tags/*:refs/tags/*'
+
+if [ "${FETCH_AFTER_INIT}" -eq 1 ]; then
+ log "fetching initial mirror contents"
+ if [ "$(id -u)" -eq 0 ]; then
+ run_as_mirror_user env MIRROR_ROOT="${MIRROR_ROOT}" GIT="${GIT}" \
+ "${SCRIPT_DIR}/update-mirror.sh" --all
+ else
+ MIRROR_ROOT="${MIRROR_ROOT}" GIT="${GIT}" "${SCRIPT_DIR}/update-mirror.sh"
--all
+ fi
+fi
+
+log "mirror initialization complete"
diff --git a/github-mirror/bin/install-controller.sh
b/github-mirror/bin/install-controller.sh
new file mode 100755
index 0000000..88be0ca
--- /dev/null
+++ b/github-mirror/bin/install-controller.sh
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+#
+# Install the GitHub mirror package on the Jenkins controller.
+
+set -euo pipefail
+
+SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+PACKAGE_ROOT=$(cd "${SCRIPT_DIR}/.." && pwd)
+
+INSTALL_ROOT=${INSTALL_ROOT:-/opt/trafficserver-ci/github-mirror}
+MIRROR_ROOT=${MIRROR_ROOT:-/home/mirror}
+MIRROR_USER=${MIRROR_USER:-gitdaemon}
+MIRROR_GROUP=${MIRROR_GROUP:-nogroup}
+ENV_DIR=${ENV_DIR:-/etc/trafficserver-github-mirror}
+ENV_FILE=${ENV_FILE:-${ENV_DIR}/github-mirror-webhook.env}
+APT_INSTALL=${APT_INSTALL:-1}
+START_WEBHOOK=${START_WEBHOOK:-auto}
+START_FALLBACK_TIMER=${START_FALLBACK_TIMER:-1}
+INIT_MIRRORS=${INIT_MIRRORS:-1}
+
+usage() {
+ cat <<'EOF'
+Usage:
+ sudo github-mirror/bin/install-controller.sh
+
+Environment:
+ INSTALL_ROOT Installed package path. Default:
/opt/trafficserver-ci/github-mirror
+ MIRROR_ROOT Mirror root. Default: /home/mirror
+ MIRROR_USER Mirror owner/service user. Default: gitdaemon
+ MIRROR_GROUP Mirror group. Default: nogroup
+ ENV_DIR Secret/env directory. Default:
/etc/trafficserver-github-mirror
+ APT_INSTALL Install required apt packages when set to 1. Default: 1
+ INIT_MIRRORS Run init-mirrors.sh after install when set to 1. Default: 1
+ START_WEBHOOK auto, 1, or 0. Default: auto
+ START_FALLBACK_TIMER
+ Enable/start the systemd fallback timer when set to 1.
Default: 1
+EOF
+}
+
+log() {
+ printf '%s\n' "$*" >&2
+}
+
+die() {
+ log "error: $*"
+ exit 1
+}
+
+render_template() {
+ local src=$1
+ local dst=$2
+ sed \
+ -e "s#@INSTALL_ROOT@#${INSTALL_ROOT}#g" \
+ -e "s#@MIRROR_ROOT@#${MIRROR_ROOT}#g" \
+ -e "s#@MIRROR_USER@#${MIRROR_USER}#g" \
+ -e "s#@MIRROR_GROUP@#${MIRROR_GROUP}#g" \
+ "${src}" > "${dst}"
+}
+
+if [ $# -gt 0 ]; then
+ case "$1" in
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ *)
+ die "unknown argument: $1"
+ ;;
+ esac
+fi
+
+[ "$(id -u)" -eq 0 ] || die "run as root"
+
+if [ "${APT_INSTALL}" = "1" ]; then
+ apt-get update
+ DEBIAN_FRONTEND=noninteractive apt-get install -y \
+ git \
+ git-daemon-sysvinit \
+ python3 \
+ util-linux
+fi
+
+if ! id "${MIRROR_USER}" >/dev/null 2>&1; then
+ useradd --system --home-dir /home/gitdaemon --shell /usr/sbin/nologin
"${MIRROR_USER}"
+fi
+
+install -d -o root -g root -m 0755 "$(dirname "${INSTALL_ROOT}")"
+rm -rf "${INSTALL_ROOT}.new"
+install -d -o root -g root -m 0755 "${INSTALL_ROOT}.new"
+cp -a "${PACKAGE_ROOT}/." "${INSTALL_ROOT}.new/"
+find "${INSTALL_ROOT}.new/bin" -type f -name '*.sh' -exec chmod 0755 {} +
+find "${INSTALL_ROOT}.new/bin" -type f -name '*.py' -exec chmod 0755 {} +
+rm -rf "${INSTALL_ROOT}"
+mv "${INSTALL_ROOT}.new" "${INSTALL_ROOT}"
+chown -R root:root "${INSTALL_ROOT}"
+
+install -d -o "${MIRROR_USER}" -g "${MIRROR_GROUP}" -m 0755 "${MIRROR_ROOT}"
+install -d -o root -g root -m 0700 "${ENV_DIR}"
+if [ ! -f "${ENV_FILE}" ]; then
+ install -o root -g root -m 0600 \
+ "${INSTALL_ROOT}/env/github-mirror-webhook.env.example" \
+ "${ENV_FILE}"
+ log "created ${ENV_FILE}; set GITHUB_WEBHOOK_SECRET before starting webhook
deliveries"
+fi
+
+install -o root -g root -m 0644 \
+ "${INSTALL_ROOT}/git-daemon/git-daemon.default" \
+ /etc/default/git-daemon
+
+tmp_unit=$(mktemp)
+render_template "${INSTALL_ROOT}/systemd/github-mirror-webhook.service"
"${tmp_unit}"
+install -o root -g root -m 0644 "${tmp_unit}"
/etc/systemd/system/github-mirror-webhook.service
+render_template "${INSTALL_ROOT}/systemd/github-mirror-fallback.service"
"${tmp_unit}"
+install -o root -g root -m 0644 "${tmp_unit}"
/etc/systemd/system/github-mirror-fallback.service
+render_template "${INSTALL_ROOT}/systemd/github-mirror-fallback.timer"
"${tmp_unit}"
+install -o root -g root -m 0644 "${tmp_unit}"
/etc/systemd/system/github-mirror-fallback.timer
+rm -f "${tmp_unit}"
+
+systemctl daemon-reload
+systemctl enable git-daemon.service >/dev/null 2>&1 || true
+systemctl restart git-daemon.service >/dev/null 2>&1 || service git-daemon
restart
+
+if [ "${INIT_MIRRORS}" = "1" ]; then
+ MIRROR_ROOT="${MIRROR_ROOT}" MIRROR_USER="${MIRROR_USER}"
MIRROR_GROUP="${MIRROR_GROUP}" \
+ "${INSTALL_ROOT}/bin/init-mirrors.sh"
+fi
+
+systemctl enable github-mirror-webhook.service
+if [ "${START_FALLBACK_TIMER}" = "1" ]; then
+ systemctl enable --now github-mirror-fallback.timer
+else
+ systemctl disable --now github-mirror-fallback.timer >/dev/null 2>&1 || true
+fi
+
+if [ "${START_WEBHOOK}" = "1" ] ||
+ { [ "${START_WEBHOOK}" = "auto" ] && grep -q '^GITHUB_WEBHOOK_SECRET='
"${ENV_FILE}" &&
+ ! grep -q '^GITHUB_WEBHOOK_SECRET=CHANGE_ME' "${ENV_FILE}"; }; then
+ systemctl restart github-mirror-webhook.service
+else
+ log "webhook service installed but not started; configure ${ENV_FILE}, then
run:"
+ log " sudo systemctl restart github-mirror-webhook.service"
+fi
+
+log "install complete"
diff --git a/github-mirror/bin/update-mirror.sh
b/github-mirror/bin/update-mirror.sh
new file mode 100755
index 0000000..6dc2e24
--- /dev/null
+++ b/github-mirror/bin/update-mirror.sh
@@ -0,0 +1,218 @@
+#!/usr/bin/env bash
+#
+# Update one or more GitHub mirror repositories under /home/mirror.
+
+set -euo pipefail
+
+MIRROR_ROOT=${MIRROR_ROOT:-/home/mirror}
+LOCK_ROOT=${LOCK_ROOT:-${MIRROR_ROOT}/.locks}
+LOCK_WAIT=${LOCK_WAIT:-300}
+GIT=${GIT:-git}
+LOCK_DIR_TO_REMOVE=""
+
+usage() {
+ cat <<'EOF'
+Usage:
+ update-mirror.sh --all
+ update-mirror.sh trafficserver [--all|--heads-tags|--pr NUMBER|--delete-pr
NUMBER]
+ update-mirror.sh trafficserver-ci [--all|--heads-tags]
+
+Environment:
+ MIRROR_ROOT Directory containing *.git mirrors. Default: /home/mirror
+ LOCK_ROOT Directory for flock lock files. Default: $MIRROR_ROOT/.locks
+ LOCK_WAIT Seconds to wait for a repo lock. Default: 300
+ GIT Git executable. Default: git
+EOF
+}
+
+log() {
+ printf '%s\n' "$*" >&2
+}
+
+die() {
+ log "error: $*"
+ exit 1
+}
+
+repo_dir_for() {
+ case "$1" in
+ trafficserver|trafficserver.git)
+ printf '%s/trafficserver.git\n' "${MIRROR_ROOT}"
+ ;;
+ trafficserver-ci|trafficserver-ci.git)
+ printf '%s/trafficserver-ci.git\n' "${MIRROR_ROOT}"
+ ;;
+ *)
+ return 1
+ ;;
+ esac
+}
+
+repo_name_for() {
+ case "$1" in
+ trafficserver|trafficserver.git)
+ printf 'trafficserver\n'
+ ;;
+ trafficserver-ci|trafficserver-ci.git)
+ printf 'trafficserver-ci\n'
+ ;;
+ *)
+ return 1
+ ;;
+ esac
+}
+
+validate_pr_number() {
+ [[ "$1" =~ ^[0-9]+$ ]] || die "invalid pull request number: $1"
+}
+
+acquire_repo_lock() {
+ local repo=$1
+ mkdir -p "${LOCK_ROOT}"
+
+ if command -v flock >/dev/null 2>&1; then
+ exec 9>"${LOCK_ROOT}/${repo}.lock"
+ flock -w "${LOCK_WAIT}" 9 || die "timed out waiting for ${repo} lock"
+ return
+ fi
+
+ local lock_dir="${LOCK_ROOT}/${repo}.lockdir"
+ local deadline=$((SECONDS + LOCK_WAIT))
+ while ! mkdir "${lock_dir}" >/dev/null 2>&1; do
+ if [ "${SECONDS}" -ge "${deadline}" ]; then
+ die "timed out waiting for ${repo} lock"
+ fi
+ sleep 1
+ done
+ LOCK_DIR_TO_REMOVE=${lock_dir}
+ trap 'if [ -n "${LOCK_DIR_TO_REMOVE:-}" ]; then rm -rf
"${LOCK_DIR_TO_REMOVE}"; fi' EXIT
+}
+
+fetch_required() {
+ local repo_dir=$1
+ shift
+ log "fetching required refs: $*"
+ "${GIT}" --git-dir="${repo_dir}" fetch --prune origin "$@"
+}
+
+fetch_optional() {
+ local repo_dir=$1
+ shift
+ log "fetching optional refs: $*"
+ if ! "${GIT}" --git-dir="${repo_dir}" fetch origin "$@"; then
+ log "optional fetch failed; continuing: $*"
+ fi
+}
+
+delete_ref() {
+ local repo_dir=$1
+ local ref=$2
+ if "${GIT}" --git-dir="${repo_dir}" show-ref --verify --quiet "${ref}"; then
+ log "deleting ${ref}"
+ "${GIT}" --git-dir="${repo_dir}" update-ref -d "${ref}"
+ else
+ log "ref not present, nothing to delete: ${ref}"
+ fi
+}
+
+update_repo() {
+ local requested_repo=$1
+ shift
+
+ local repo
+ repo=$(repo_name_for "${requested_repo}") || die "unknown repo:
${requested_repo}"
+
+ local repo_dir
+ repo_dir=$(repo_dir_for "${repo}") || die "unknown repo: ${repo}"
+ [ -d "${repo_dir}" ] || die "mirror does not exist: ${repo_dir}; run
init-mirrors.sh first"
+
+ acquire_repo_lock "${repo}"
+
+ local selectors=("$@")
+ if [ ${#selectors[@]} -eq 0 ]; then
+ selectors=(--all)
+ fi
+
+ local did_work=0
+ local selector
+ while [ ${#selectors[@]} -gt 0 ]; do
+ selector=${selectors[0]}
+ selectors=("${selectors[@]:1}")
+
+ case "${selector}" in
+ --all)
+ if [ "${repo}" = "trafficserver" ]; then
+ fetch_required "${repo_dir}" \
+ '+refs/heads/*:refs/heads/*' \
+ '+refs/tags/*:refs/tags/*' \
+ '+refs/pull/*:refs/pull/*'
+ else
+ fetch_required "${repo_dir}" \
+ '+refs/heads/*:refs/heads/*' \
+ '+refs/tags/*:refs/tags/*'
+ fi
+ did_work=1
+ ;;
+ --heads-tags)
+ fetch_required "${repo_dir}" \
+ '+refs/heads/*:refs/heads/*' \
+ '+refs/tags/*:refs/tags/*'
+ did_work=1
+ ;;
+ --pr)
+ [ "${repo}" = "trafficserver" ] || die "--pr is only valid for
trafficserver"
+ [ ${#selectors[@]} -gt 0 ] || die "--pr requires a pull request number"
+ local pr_number=${selectors[0]}
+ selectors=("${selectors[@]:1}")
+ validate_pr_number "${pr_number}"
+ fetch_required "${repo_dir}" \
+ "+refs/pull/${pr_number}/head:refs/pull/${pr_number}/head"
+ fetch_optional "${repo_dir}" \
+ "+refs/pull/${pr_number}/merge:refs/pull/${pr_number}/merge"
+ did_work=1
+ ;;
+ --delete-pr)
+ [ "${repo}" = "trafficserver" ] || die "--delete-pr is only valid for
trafficserver"
+ [ ${#selectors[@]} -gt 0 ] || die "--delete-pr requires a pull request
number"
+ local pr_number=${selectors[0]}
+ selectors=("${selectors[@]:1}")
+ validate_pr_number "${pr_number}"
+ delete_ref "${repo_dir}" "refs/pull/${pr_number}/head"
+ delete_ref "${repo_dir}" "refs/pull/${pr_number}/merge"
+ did_work=1
+ ;;
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ *)
+ die "unknown selector: ${selector}"
+ ;;
+ esac
+ done
+
+ [ "${did_work}" -eq 1 ] || die "no update work was requested"
+ "${GIT}" --git-dir="${repo_dir}" update-server-info
+ log "updated ${repo} mirror at ${repo_dir}"
+}
+
+if [ $# -eq 0 ]; then
+ usage
+ exit 2
+fi
+
+case "$1" in
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ --all)
+ update_repo trafficserver --all
+ update_repo trafficserver-ci --all
+ ;;
+ *)
+ repo_arg=$1
+ shift
+ update_repo "${repo_arg}" "$@"
+ ;;
+esac
diff --git a/github-mirror/cron/github-mirror b/github-mirror/cron/github-mirror
new file mode 100644
index 0000000..c14d77a
--- /dev/null
+++ b/github-mirror/cron/github-mirror
@@ -0,0 +1,15 @@
+# Temporary cron fallback for GitHub mirror updates before ASF webhooks are
live.
+#
+# Install on the controller as:
+# sudo install -o root -g root -m 0644 \
+# /opt/trafficserver-ci/github-mirror/cron/github-mirror \
+# /etc/cron.d/github-mirror
+#
+# Remove this file after the GitHub webhook has been validated and the
+# github-mirror-fallback.timer is the only periodic fallback updater.
+
+SHELL=/bin/bash
+PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+MIRROR_ROOT=/home/mirror
+
+* * * * * gitdaemon /opt/trafficserver-ci/github-mirror/bin/update-mirror.sh
--all 2>&1 | /usr/bin/logger -t github-mirror-cron
diff --git a/github-mirror/env/github-mirror-webhook.env.example
b/github-mirror/env/github-mirror-webhook.env.example
new file mode 100644
index 0000000..17564f8
--- /dev/null
+++ b/github-mirror/env/github-mirror-webhook.env.example
@@ -0,0 +1,14 @@
+# Copy to /etc/trafficserver-github-mirror/github-mirror-webhook.env.
+# Keep the installed copy root-owned and mode 0600.
+
+GITHUB_WEBHOOK_SECRET=CHANGE_ME
+
+WEBHOOK_HOST=127.0.0.1
+WEBHOOK_PORT=9419
+WEBHOOK_PATH=/github-mirror-webhook
+
+MIRROR_ROOT=/home/mirror
+UPDATE_MIRROR=/opt/trafficserver-ci/github-mirror/bin/update-mirror.sh
+
+MAX_BODY_BYTES=1048576
+UPDATE_TIMEOUT_SECONDS=600
diff --git a/github-mirror/git-daemon/git-daemon.default
b/github-mirror/git-daemon/git-daemon.default
new file mode 100644
index 0000000..8d820d4
--- /dev/null
+++ b/github-mirror/git-daemon/git-daemon.default
@@ -0,0 +1,7 @@
+# Defaults for git-daemon initscript.
+
+GIT_DAEMON_ENABLE=true
+GIT_DAEMON_USER=gitdaemon
+GIT_DAEMON_BASE_PATH=/home/mirror
+GIT_DAEMON_DIRECTORY=/home/mirror
+GIT_DAEMON_OPTIONS="--export-all --reuseaddr --port=9418"
diff --git a/github-mirror/httpd/mirror.conf b/github-mirror/httpd/mirror.conf
new file mode 100644
index 0000000..abfce7a
--- /dev/null
+++ b/github-mirror/httpd/mirror.conf
@@ -0,0 +1,14 @@
+# Apache httpd snippet for exporting /home/mirror as /mirror/.
+#
+# This simple static export works with `git update-server-info`, which the
+# mirror updater runs after every fetch. If the controller already has an httpd
+# container or vhost serving /mirror/, keep that and use this as the rebuild
+# reference.
+
+Alias /mirror/ /home/mirror/
+
+<Directory /home/mirror/>
+ Options Indexes FollowSymLinks
+ AllowOverride None
+ Require all granted
+</Directory>
diff --git a/github-mirror/systemd/github-mirror-fallback.service
b/github-mirror/systemd/github-mirror-fallback.service
new file mode 100644
index 0000000..36f21e5
--- /dev/null
+++ b/github-mirror/systemd/github-mirror-fallback.service
@@ -0,0 +1,14 @@
+[Unit]
+Description=ATS CI GitHub mirror fallback refresh
+Documentation=file:@INSTALL_ROOT@/README.md
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+User=@MIRROR_USER@
+Group=@MIRROR_GROUP@
+Environment=MIRROR_ROOT=@MIRROR_ROOT@
+ExecStart=@INSTALL_ROOT@/bin/update-mirror.sh --all
+TimeoutStartSec=30min
+Nice=5
diff --git a/github-mirror/systemd/github-mirror-fallback.timer
b/github-mirror/systemd/github-mirror-fallback.timer
new file mode 100644
index 0000000..f3fe411
--- /dev/null
+++ b/github-mirror/systemd/github-mirror-fallback.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Run ATS CI GitHub mirror fallback refresh
+Documentation=file:@INSTALL_ROOT@/README.md
+
+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=5min
+AccuracySec=30s
+Unit=github-mirror-fallback.service
+
+[Install]
+WantedBy=timers.target
diff --git a/github-mirror/systemd/github-mirror-webhook.service
b/github-mirror/systemd/github-mirror-webhook.service
new file mode 100644
index 0000000..b93141b
--- /dev/null
+++ b/github-mirror/systemd/github-mirror-webhook.service
@@ -0,0 +1,24 @@
+[Unit]
+Description=ATS CI GitHub mirror webhook receiver
+Documentation=file:@INSTALL_ROOT@/README.md
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=@MIRROR_USER@
+Group=@MIRROR_GROUP@
+EnvironmentFile=/etc/trafficserver-github-mirror/github-mirror-webhook.env
+WorkingDirectory=@MIRROR_ROOT@
+ExecStart=@INSTALL_ROOT@/bin/github-mirror-webhook.py
+Restart=on-failure
+RestartSec=5s
+RestartPreventExitStatus=1
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=full
+ProtectHome=false
+ReadWritePaths=@MIRROR_ROOT@
+
+[Install]
+WantedBy=multi-user.target
diff --git a/jenkins/branch/autest.pipeline b/jenkins/branch/autest.pipeline
index f988d81..d710ae8 100644
--- a/jenkins/branch/autest.pipeline
+++ b/jenkins/branch/autest.pipeline
@@ -47,7 +47,7 @@ pipeline {
stage('Clone') {
steps {
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
diff --git a/jenkins/branch/clang_analyzer.pipeline
b/jenkins/branch/clang_analyzer.pipeline
index bbe973d..fe2f4ef 100644
--- a/jenkins/branch/clang_analyzer.pipeline
+++ b/jenkins/branch/clang_analyzer.pipeline
@@ -32,7 +32,7 @@ pipeline {
stage('Clone') {
steps {
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
diff --git a/jenkins/branch/cmake.pipeline b/jenkins/branch/cmake.pipeline
index 478c022..804c3b0 100644
--- a/jenkins/branch/cmake.pipeline
+++ b/jenkins/branch/cmake.pipeline
@@ -35,7 +35,7 @@ pipeline {
stage('Clone') {
steps {
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
diff --git a/jenkins/branch/coverage.pipeline b/jenkins/branch/coverage.pipeline
index 971d03d..250d21b 100644
--- a/jenkins/branch/coverage.pipeline
+++ b/jenkins/branch/coverage.pipeline
@@ -34,7 +34,7 @@ pipeline {
stage('Clone') {
steps {
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
diff --git a/jenkins/branch/coverity.pipeline b/jenkins/branch/coverity.pipeline
index 6276a8a..433809d 100644
--- a/jenkins/branch/coverity.pipeline
+++ b/jenkins/branch/coverity.pipeline
@@ -16,7 +16,7 @@ pipeline {
stage('Initialization') {
steps {
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
@@ -25,7 +25,7 @@ pipeline {
rm -rf *
set -x
'''
- git
'https://github.com/apache/trafficserver.git'
+ git
'https://ci.trafficserver.apache.org/mirror/trafficserver.git'
sh '''#!/bin/bash
set +x
source
/opt/rh/gcc-toolset-11/enable
diff --git a/jenkins/branch/docs.pipeline b/jenkins/branch/docs.pipeline
index 6ada3b5..d910edd 100644
--- a/jenkins/branch/docs.pipeline
+++ b/jenkins/branch/docs.pipeline
@@ -28,7 +28,7 @@ pipeline {
}
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
diff --git a/jenkins/branch/format.pipeline b/jenkins/branch/format.pipeline
index 9963684..b6c0a01 100644
--- a/jenkins/branch/format.pipeline
+++ b/jenkins/branch/format.pipeline
@@ -30,7 +30,7 @@ pipeline {
stage('Clone') {
steps {
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
diff --git a/jenkins/branch/freebsd.pipeline b/jenkins/branch/freebsd.pipeline
index af5d39c..a451c1b 100644
--- a/jenkins/branch/freebsd.pipeline
+++ b/jenkins/branch/freebsd.pipeline
@@ -5,7 +5,7 @@ pipeline {
steps {
echo 'Starting Clone'
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
diff --git a/jenkins/branch/in_tree.pipeline b/jenkins/branch/in_tree.pipeline
index 86cad5a..a9cdcc6 100644
--- a/jenkins/branch/in_tree.pipeline
+++ b/jenkins/branch/in_tree.pipeline
@@ -35,7 +35,7 @@ pipeline {
stage('Clone') {
steps {
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
diff --git a/jenkins/branch/os_build.pipeline b/jenkins/branch/os_build.pipeline
index 1929534..0690951 100644
--- a/jenkins/branch/os_build.pipeline
+++ b/jenkins/branch/os_build.pipeline
@@ -27,7 +27,7 @@ pipeline {
steps {
echo 'Starting Clone'
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
diff --git a/jenkins/branch/osx-m1.pipeline b/jenkins/branch/osx-m1.pipeline
index 9991b7b..6270f72 100644
--- a/jenkins/branch/osx-m1.pipeline
+++ b/jenkins/branch/osx-m1.pipeline
@@ -5,7 +5,7 @@ pipeline {
steps {
echo 'Starting Clone'
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
diff --git a/jenkins/branch/osx.pipeline b/jenkins/branch/osx.pipeline
index 25c5bb2..b5fbf86 100644
--- a/jenkins/branch/osx.pipeline
+++ b/jenkins/branch/osx.pipeline
@@ -5,7 +5,7 @@ pipeline {
steps {
echo 'Starting Clone'
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
diff --git a/jenkins/branch/out_of_tree.pipeline
b/jenkins/branch/out_of_tree.pipeline
index 14bb8a6..9cfa039 100644
--- a/jenkins/branch/out_of_tree.pipeline
+++ b/jenkins/branch/out_of_tree.pipeline
@@ -35,7 +35,7 @@ pipeline {
stage('Clone') {
steps {
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
diff --git a/jenkins/branch/quiche.pipeline b/jenkins/branch/quiche.pipeline
index bd52e8c..9d95358 100644
--- a/jenkins/branch/quiche.pipeline
+++ b/jenkins/branch/quiche.pipeline
@@ -35,7 +35,7 @@ pipeline {
stage('Clone') {
steps {
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
diff --git a/jenkins/branch/rat.pipeline b/jenkins/branch/rat.pipeline
index 7f520fd..15c686e 100644
--- a/jenkins/branch/rat.pipeline
+++ b/jenkins/branch/rat.pipeline
@@ -32,7 +32,7 @@ pipeline {
stage('Clone') {
steps {
dir('ci') {
- git url:
'https://github.com/apache/trafficserver-ci',
+ git url:
'https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git',
branch: 'main'
}
dir('src') {
diff --git a/jenkins/github/github_polling.pipeline
b/jenkins/github/github_polling.pipeline
index 5d99e65..9035ae7 100644
--- a/jenkins/github/github_polling.pipeline
+++ b/jenkins/github/github_polling.pipeline
@@ -7,7 +7,7 @@ String buildJob(String ghcontext, String jobName, String
shard='') {
currentBuild.description = "Builds:<br>"
}
currentBuild.displayName = "PR: #${GITHUB_PR_NUMBER} - Build:
#${BUILD_NUMBER}"
- https_github_url = GITHUB_REPO_GIT_URL.replace("git://", "https://";)
+ https_github_url = env.GITHUB_URL ?:
GITHUB_REPO_GIT_URL.replace("git://", "https://";)
def parms = [
string(name: 'SHA1', value: GITHUB_PR_HEAD_SHA),
diff --git a/jenkins/github/toplevel.pipeline b/jenkins/github/toplevel.pipeline
index e3e5fc7..e1e7012 100644
--- a/jenkins/github/toplevel.pipeline
+++ b/jenkins/github/toplevel.pipeline
@@ -7,7 +7,7 @@ String buildJob(String ghcontext, String jobName, String
shard='') {
currentBuild.description = "Builds:<br>"
}
currentBuild.displayName = "PR: #${GITHUB_PR_NUMBER} - Build:
#${BUILD_NUMBER}"
- https_github_url = GITHUB_REPO_GIT_URL.replace("git://", "https://";)
+ https_github_url = env.GITHUB_URL ?:
GITHUB_REPO_GIT_URL.replace("git://", "https://";)
def parms = [
string(name: 'SHA1', value: GITHUB_PR_HEAD_SHA),
