This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 91eed64d3c8 Check the AIP and ADIP wiki trees daily and report
problems to Slack (#76845)
91eed64d3c8 is described below
commit 91eed64d3c8479bed4201316ce5fa4f587437a51
Author: Jarek Potiuk <[email protected]>
AuthorDate: Sun Nov 15 06:53:48 2026 +0100
Check the AIP and ADIP wiki trees daily and report problems to Slack
(#76845)
* Check the AIP and ADIP wiki trees daily and report problems to Slack
The AIP and ADIP index pages are now generated: each list is a Page
Properties
Report over one folder, so an AIP sits in the folder for its state and
carries
a matching State badge and state label. A page in the wrong folder, a State
or
label that does not match it, or a stale count in a folded section silently
puts an AIP in the wrong list or drops it.
scripts/ci/verify_improvement_proposals_wiki.py reads the wiki anonymously
and
reports each such page; a scheduled workflow runs it daily and posts the
list
to #internal-airflow-ci-cd when anything is out of line, or a short notice
when
the check itself could not run.
Generated-by: Claude Opus 5
* Check the ADIP tree with the same state folders as the AIP tree
ADIPs are now grouped into the same state folders as AIPs (WIP, Under
discussion, Accepted, Completed, Abandoned or replaced), with the same State
badges, adip/<state> labels and folded Completed and Abandoned sections. The
check runs the folder, badge, label and count checks on both trees, and the
ADIP title, number and Formerly checks on the ADIP pages.
Generated-by: Claude Opus 5
* Allow new ADIPs without a Formerly row
Only the ADIPs that started as AIPs have a former AIP number. New ADIPs,
created from the ADIP template in the 'ADIPs under discussion' folder, have
none, so the check now only validates the Formerly row when it is there.
Generated-by: Claude Opus 5
* Read Page Properties macro parameters in linear time
A details macro with many parameters and no rich-text body made the
parameter pattern backtrack exponentially, because a parameter's value
could run past its own closing tag. A page edited that way would hang
the daily wiki check instead of reporting it. CodeQL flagged it as
py/redos.
A self-closing parameter before the body also hid the properties table,
so a correct State row would have been reported as missing.
Generated-by: Claude Opus 5
* Read Page Properties keys whose colon sits outside the bold text
Turning tags into spaces leaves a key like `<strong>State</strong>:`
as "State :", so the State row was not found and a correctly filed
proposal would be reported as missing its state every day.
Generated-by: Claude Opus 5
* Read Page Properties around nested tables and search only the Airflow
space
A table nested in a property value ended the properties table at its own
closing tag, so the rows after it, such as State, were lost and a
correctly filed proposal would be reported as missing its state.
The label search covered every cwiki space, so a page with the same
label in another space would be reported as outside its folder.
Generated-by: Claude Opus 5
---
...scheduled-verify-improvement-proposals-wiki.yml | 64 +++
dev/breeze/doc/ci/04_selective_checks.md | 2 +-
.../src/airflow_breeze/utils/selective_checks.py | 3 +-
dev/breeze/tests/test_selective_checks.py | 5 +
scripts/ci/verify_improvement_proposals_wiki.py | 472 +++++++++++++++++++
.../ci/test_verify_improvement_proposals_wiki.py | 505 +++++++++++++++++++++
6 files changed, 1049 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/scheduled-verify-improvement-proposals-wiki.yml
b/.github/workflows/scheduled-verify-improvement-proposals-wiki.yml
new file mode 100644
index 00000000000..ed7a79b6c6a
--- /dev/null
+++ b/.github/workflows/scheduled-verify-improvement-proposals-wiki.yml
@@ -0,0 +1,64 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Scheduled verify improvement proposals wiki"
+on: # yamllint disable-line rule:truthy
+ schedule:
+ # Daily at 06:30 UTC
+ - cron: '30 6 * * *'
+ workflow_dispatch:
+permissions:
+ contents: read
+jobs:
+ verify-improvement-proposals-wiki:
+ name: "Verify AIP and ADIP wiki trees"
+ if: github.event_name != 'schedule' || github.repository ==
'apache/airflow'
+ runs-on: ["ubuntu-slim"]
+ timeout-minutes: 15
+ steps:
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
+ with:
+ persist-credentials: false
+ - name: "Verify AIP and ADIP wiki trees"
+ id: verify
+ env:
+ RUN_URL: "${{ github.server_url }}/${{ github.repository
}}/actions/runs/${{ github.run_id }}"
+ run: >
+ python3 scripts/ci/verify_improvement_proposals_wiki.py
+ --slack-payload slack-message.json
+ --run-url "${RUN_URL}"
+ - name: "Notify Slack about the problems"
+ if: failure() && hashFiles('slack-message.json') != ''
+ uses:
slackapi/slack-github-action@dcb1066f776dd043e64d0e8ba94ca15cc7e1875d # v4.0.0
+ with:
+ method: chat.postMessage
+ token: ${{ secrets.SLACK_BOT_TOKEN }}
+ payload-file-path: "slack-message.json"
+ # yamllint disable rule:line-length
+ - name: "Notify Slack that the check could not run"
+ if: failure() && hashFiles('slack-message.json') == ''
+ uses:
slackapi/slack-github-action@dcb1066f776dd043e64d0e8ba94ca15cc7e1875d # v4.0.0
+ with:
+ method: chat.postMessage
+ token: ${{ secrets.SLACK_BOT_TOKEN }}
+ payload: |
+ channel: "internal-airflow-ci-cd"
+ text: >-
+ :warning: The AIP/ADIP wiki consistency check could not run.
+ See: ${{ github.server_url }}/${{ github.repository
}}/actions/runs/${{ github.run_id }}
diff --git a/dev/breeze/doc/ci/04_selective_checks.md
b/dev/breeze/doc/ci/04_selective_checks.md
index 52c1db2c98e..717282567ae 100644
--- a/dev/breeze/doc/ci/04_selective_checks.md
+++ b/dev/breeze/doc/ci/04_selective_checks.md
@@ -422,7 +422,7 @@ We have the following Groups of files for CI that determine
which tests are run:
drive prek static checks rather than the test matrix, so they only trigger
the CI image build (via the
`Prek files` group) — `mypy-scripts` and the image-based static checks still
run, but the full test
matrix does not; and **non-test workflows** (`.github/workflows/` for
security scans, doc publishing,
- notifications, backporting, calendar/stale bots, …) cannot change test
outcomes. Workflows that run or
+ notifications, backporting, calendar/wiki/stale bots, …) cannot change test
outcomes. Workflows that run or
configure the test suite (`ci-amd`, `ci-arm`, `run-unit-tests`, `k8s-tests`,
`helm-tests`, image builds,
…) still force the full matrix.
* `Prek files` - `scripts/ci/prek/...` static-check hooks. They do not force
full tests, but they trigger
diff --git a/dev/breeze/src/airflow_breeze/utils/selective_checks.py
b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
index f41e1493b82..b0cb135ac97 100644
--- a/dev/breeze/src/airflow_breeze/utils/selective_checks.py
+++ b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
@@ -262,7 +262,8 @@ CI_FILE_GROUP_MATCHES: HashableDict[FileGroupForCi] =
HashableDict(
r"^\.github/workflows/(?!("
r"asf-allowlist-check|automatic-backport|backport-cli|ci-duration-monitor|ci-notification|"
r"codeql-analysis|e2e-flaky-tests-report|milestone-tag-assistant|notify-uv-lock-conflicts|"
-
r"publish-docs-to-s3|recheck-old-bug-report|scheduled-verify-release-calendar|stale"
+
r"publish-docs-to-s3|recheck-old-bug-report|scheduled-verify-improvement-proposals-wiki|"
+ r"scheduled-verify-release-calendar|stale"
r")\.yml$)",
r"^dev/breeze/src",
r"^dev/breeze/pyproject\.toml",
diff --git a/dev/breeze/tests/test_selective_checks.py
b/dev/breeze/tests/test_selective_checks.py
index 50fac6f29c7..90c9abee495 100644
--- a/dev/breeze/tests/test_selective_checks.py
+++ b/dev/breeze/tests/test_selective_checks.py
@@ -2815,6 +2815,11 @@ def
test_ui_compile_hooks_not_skipped_when_ui_openapi_spec_changes(files: tuple[
{"full-tests-needed": "false"},
id="Non-test workflow (ci-notification) does not force full tests",
),
+ pytest.param(
+
(".github/workflows/scheduled-verify-improvement-proposals-wiki.yml",),
+ {"full-tests-needed": "false"},
+ id="Non-test workflow (improvement proposals wiki check) does not
force full tests",
+ ),
],
)
def test_non_test_workflows_do_not_force_full_tests(files: tuple[str, ...],
expected_outputs: dict[str, str]):
diff --git a/scripts/ci/verify_improvement_proposals_wiki.py
b/scripts/ci/verify_improvement_proposals_wiki.py
new file mode 100644
index 00000000000..b4bab3eb479
--- /dev/null
+++ b/scripts/ci/verify_improvement_proposals_wiki.py
@@ -0,0 +1,472 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# /// script
+# requires-python = ">=3.11"
+# ///
+"""
+Check that the AIP and ADIP pages on the Airflow wiki are laid out the way
their index pages expect.
+
+The index pages are generated: each list is a Page Properties Report over the
pages of one
+folder, so a page in the wrong place, or with a State that does not match
where it sits, shows
+up in the wrong list or not at all. This script reads the wiki anonymously and
reports every
+such page.
+
+The AIP tree
(https://cwiki.apache.org/confluence/display/AIRFLOW/AIP+-+Airflow+Improvement+Proposals)
+and the ADIP tree are laid out the same way, and for each of them:
+
+ * the index page has only the state folders (and, for AIPs, the folder of
ADIP stubs) as
+ children;
+ * every page in a state folder has one Page Properties block whose State
badge matches the
+ folder, the tree's base label, and the state label of that folder only;
+ * a page carrying a state label sits in that state's folder;
+ * the counts in the titles of the folded sections on the index page match
the folders.
+
+Every ADIP is also titled ``ADIP-<n> <title>`` with a unique number; an ADIP
that started as an AIP
+has its old number in a Formerly row.
+
+Exits 1 when anything is out of line. With ``--slack-payload <file>`` it then
also writes a
+``chat.postMessage`` payload listing the problems, for the workflow to post.
+"""
+
+from __future__ import annotations
+
+import argparse
+import gzip
+import http.client
+import json
+import re
+import sys
+import time
+import urllib.parse
+import urllib.request
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from pathlib import Path
+
+WIKI = "https://cwiki.apache.org/confluence"
+WIKI_SPACE = "AIRFLOW"
+SLACK_CHANNEL = "internal-airflow-ci-cd"
+MAX_PROBLEMS_IN_SLACK = 25
+MAX_TITLE_LENGTH = 100
+MAX_PROBLEM_LENGTH = 300
+# Slack rejects a section block whose text is longer than 3000 characters.
+MAX_SECTION_LENGTH = 2900
+ATTEMPTS = 4
+
+
+@dataclass(frozen=True)
+class StateFolder:
+ page_id: str
+ state: str
+ badge: str
+ # How the folded section of this folder on the index page names its pages,
e.g.
+ # "completed" for "8 completed ADIPs (click to expand)"; None when it is
not folded.
+ folded: str | None = None
+
+
+@dataclass(frozen=True)
+class ProposalTree:
+ name: str
+ index_id: str
+ base_label: str
+ folders: tuple[StateFolder, ...]
+ other_children: frozenset[str] = frozenset()
+
+ def get_label(self, folder: StateFolder) -> str:
+ return f"{self.base_label}/{folder.state}"
+
+ def get_state_labels(self) -> set[str]:
+ return {self.get_label(folder) for folder in self.folders}
+
+ def get_folded_title(self, folder: StateFolder, count: int) -> str:
+ return f"{count} {folder.folded} {self.name}s (click to expand)"
+
+
+def build_state_folders(wip: str, draft: str, accepted: str, completed: str,
abandoned: str):
+ return (
+ StateFolder(wip, "wip", "WIP"),
+ StateFolder(draft, "draft", "Under discussion"),
+ StateFolder(accepted, "accepted", "Accepted"),
+ StateFolder(completed, "completed", "Completed", "completed"),
+ StateFolder(abandoned, "abandoned", "Abandoned", "abandoned or
replaced"),
+ )
+
+
+AIP_TREE = ProposalTree(
+ "AIP",
+ "89066602",
+ "airflow-improvement-proposal",
+ build_state_folders("462816682", "462816536", "462816537", "173087665",
"173087786"),
+ # Pages under the old titles of the AIPs that became ADIPs.
+ other_children=frozenset({"462816661"}),
+)
+ADIP_TREE = ProposalTree(
+ "ADIP",
+ "462816431",
+ "adip",
+ build_state_folders("462816703", "462816705", "462816707", "462816709",
"462816711"),
+)
+TREES = (AIP_TREE, ADIP_TREE)
+
+# Pages in a state folder that the people who keep the wiki in order cannot
edit, with the
+# reason. Their State badge and labels are not checked; their place in the
tree still is.
+RESTRICTED_PAGES = {
+ "247827224": "AIP-52 has edit restrictions",
+}
+
+
+@dataclass
+class Page:
+ page_id: str
+ title: str
+ body: str = ""
+ labels: list[str] = field(default_factory=list)
+
+ @property
+ def url(self) -> str:
+ return f"{WIKI}/pages/viewpage.action?pageId={self.page_id}"
+
+
+def get_json(path: str, params: dict[str, str], sleep: Callable[[float], None]
= time.sleep) -> dict:
+ url = f"{WIKI}/rest/api/{path}?{urllib.parse.urlencode(params)}"
+ # Uncompressed responses of more than a few tens of kilobytes get cut off
mid-transfer.
+ request = urllib.request.Request(
+ url,
+ headers={
+ "Accept": "application/json",
+ "Accept-Encoding": "gzip",
+ "User-Agent": "apache-airflow-wiki-consistency-check",
+ },
+ )
+ for attempt in range(1, ATTEMPTS):
+ try:
+ return read_json(request)
+ except (http.client.HTTPException, OSError):
+ sleep(5 * attempt)
+ return read_json(request)
+
+
+def read_json(request: urllib.request.Request) -> dict:
+ with urllib.request.urlopen(request, timeout=60) as response:
+ raw = response.read()
+ if response.headers.get("Content-Encoding") == "gzip":
+ raw = gzip.decompress(raw)
+ return json.loads(raw)
+
+
+def build_page(raw: dict) -> Page:
+ return Page(
+ page_id=raw["id"],
+ title=raw["title"],
+ body=raw.get("body", {}).get("storage", {}).get("value", ""),
+ labels=[label["name"] for label in raw.get("metadata",
{}).get("labels", {}).get("results", [])],
+ )
+
+
+def fetch_all(path: str, params: dict[str, str], fetch: Callable[[str,
dict[str, str]], dict]) -> list[Page]:
+ pages: list[Page] = []
+ start = 0
+ while True:
+ data = fetch(path, {**params, "limit": "50", "start": str(start)})
+ pages.extend(build_page(raw) for raw in data["results"])
+ if not data["results"] or "next" not in data.get("_links", {}):
+ return pages
+ start += len(data["results"])
+
+
+def fetch_page(page_id: str, fetch: Callable[[str, dict[str, str]], dict] =
get_json) -> Page:
+ return build_page(fetch(f"content/{page_id}", {"expand":
"body.storage,metadata.labels"}))
+
+
+def fetch_children(page_id: str, fetch: Callable[[str, dict[str, str]], dict]
= get_json) -> list[Page]:
+ # Bodies are fetched one page at a time: a response carrying several AIP
bodies is large
+ # enough to be cut off mid-transfer.
+ listed = fetch_all(f"content/{page_id}/child/page", {}, fetch)
+ return [fetch_page(page.page_id, fetch) for page in listed]
+
+
+def fetch_labelled(label: str, fetch: Callable[[str, dict[str, str]], dict] =
get_json) -> list[Page]:
+ return fetch_all(
+ "content/search", {"cql": f'label = "{label}" and type = page and
space = {WIKI_SPACE}'}, fetch
+ )
+
+
+def get_text(html: str) -> str:
+ return re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", html).replace(" ",
" ")).strip()
+
+
+def find_macro_body_end(body: str, start: int) -> int:
+ """Return where the rich-text body opened after ``start`` closes, skipping
nested macro bodies."""
+ depth = 0
+ for tag in re.finditer(r"<(/?)ac:rich-text-body>", body[start:]):
+ depth += -1 if tag.group(1) else 1
+ if depth == 0:
+ return start + tag.start()
+ return len(body)
+
+
+def find_macro_body_start(body: str, start: int) -> int | None:
+ """
+ Return where the rich-text body of the macro whose opening tag continues
at ``start`` begins.
+
+ ``None`` when the macro is self-closing or its parameters are followed by
anything but its
+ own body. The parameters are walked one at a time rather than with one
repeated pattern:
+ a repeated group whose items can each span markup backtracks exponentially
when no body
+ follows (CodeQL ``py/redos``).
+ """
+ opening = re.compile(r"[^>]*>").match(body, start)
+ if not opening or opening.group().endswith("/>"):
+ return None
+ pos = opening.end()
+ while parameter := re.compile(r"\s*<ac:parameter\b[^>]*>").match(body,
pos):
+ if parameter.group().endswith("/>"):
+ pos = parameter.end()
+ continue
+ closing = body.find("</ac:parameter>", parameter.end())
+ if closing == -1:
+ return None
+ pos = closing + len("</ac:parameter>")
+ rich_text = re.compile(r"\s*<ac:rich-text-body>").match(body, pos)
+ return rich_text.end() - len("<ac:rich-text-body>") if rich_text else None
+
+
+def find_properties(body: str) -> list[dict[str, str]]:
+ """Return the rows of each Page Properties block as {key: value storage
HTML}."""
+ blocks = []
+ for match in re.finditer(r'<ac:structured-macro ac:name="details"', body):
+ # Only a table inside the macro's own rich-text body is read by the
index report; a
+ # self-closing macro, or one with no body, has no properties.
+ body_start = find_macro_body_start(body, match.end())
+ if body_start is None:
+ macro_body = ""
+ else:
+ macro_body = body[body_start : find_macro_body_end(body,
body_start)]
+ rows: dict[str, str] = {}
+ for cells in find_table_rows(macro_body):
+ if len(cells) >= 2:
+ rows.setdefault(get_text(cells[0]).rstrip(": "), cells[1])
+ blocks.append(rows)
+ return blocks
+
+
+def find_table_rows(html: str) -> list[list[str]]:
+ """
+ Return the cells of each row of the first table in ``html``, as storage
HTML.
+
+ Rows and cells of a table nested in a cell stay inside that cell, so a
nested table cannot end
+ the outer table early or add rows to it.
+ """
+ rows: list[list[str]] = []
+ row: list[str] | None = None
+ cell_start: int | None = None
+ depth = 0
+ for tag in re.finditer(r"<(/?)(table|tr|td|th)(?=[\s/>])[^>]*>", html):
+ closing, name = tag.group(1) == "/", tag.group(2)
+ if name == "table":
+ if not closing:
+ depth += 1
+ elif depth > 0:
+ depth -= 1
+ if depth == 0:
+ break
+ elif depth != 1:
+ continue
+ elif name == "tr":
+ if closing and row is not None:
+ rows.append(row)
+ row = None if closing else []
+ elif closing:
+ if row is not None and cell_start is not None:
+ row.append(html[cell_start : tag.start()])
+ cell_start = None
+ else:
+ cell_start = tag.end()
+ return rows
+
+
+def find_badge(cell: str) -> str | None:
+ match = re.search(
+ r'<ac:structured-macro ac:name="status".*?<ac:parameter
ac:name="title">([^<]*)</ac:parameter>',
+ cell,
+ re.S,
+ )
+ return match.group(1) if match else None
+
+
+def check_properties(page: Page, allowed_badges: set[str]) -> tuple[list[str],
dict[str, str]]:
+ blocks = find_properties(page.body)
+ if len(blocks) != 1:
+ return [f"has {len(blocks)} Page Properties blocks, expected 1"],
blocks[0] if blocks else {}
+ properties = blocks[0]
+ if "State" not in properties:
+ return ["has no State row in its Page Properties block"], properties
+ badge = find_badge(properties["State"])
+ if badge not in allowed_badges:
+ return [f"State badge is {badge!r}, expected one of
{sorted(allowed_badges)}"], properties
+ return [], properties
+
+
+def escape_for_slack(text: str) -> str:
+ # Wiki titles are free text: unescaped, "<!channel>" would notify the
channel and a "<" or
+ # "|" would change the link.
+ return text.replace("&", "&").replace("<", "<").replace(">",
">").replace("|", "|")
+
+
+def shorten(title: str) -> str:
+ return title if len(title) <= MAX_TITLE_LENGTH else title[:
MAX_TITLE_LENGTH - 1] + "…"
+
+
+def describe(page: Page, problem: str) -> str:
+ # Problems quote wiki text, which can be arbitrarily long; Slack rejects
oversized sections.
+ if len(problem) > MAX_PROBLEM_LENGTH:
+ problem = problem[: MAX_PROBLEM_LENGTH - 1] + "…"
+ return f"<{page.url}|{escape_for_slack(shorten(page.title))}>:
{escape_for_slack(problem)}"
+
+
+def check_tree(
+ tree: ProposalTree,
+ index: Page,
+ index_children: list[Page],
+ folders: dict[str, list[Page]],
+ labelled: dict[str, list[Page]],
+) -> list[str]:
+ problems = []
+ allowed_children = {folder.page_id for folder in tree.folders} |
tree.other_children
+ for page in index_children:
+ if page.page_id not in allowed_children:
+ problems.append(
+ describe(page, f"sits directly under the {tree.name} index;
move it into a state folder")
+ )
+ for missing in sorted(allowed_children - {page.page_id for page in
index_children}):
+ problems.append(
+ describe(
+ Page(missing, f"page {missing}"), f"folder is no longer
directly under the {tree.name} index"
+ )
+ )
+ state_labels = tree.get_state_labels()
+ for folder in tree.folders:
+ label = tree.get_label(folder)
+ pages = folders[folder.page_id]
+ for page in pages:
+ if page.page_id in RESTRICTED_PAGES:
+ continue
+ page_problems, _ = check_properties(page, {folder.badge})
+ page_state_labels = state_labels.intersection(page.labels)
+ if page_state_labels != {label}:
+ page_problems.append(f"state labels are
{sorted(page_state_labels)}, expected ['{label}']")
+ if tree.base_label not in page.labels:
+ page_problems.append(f"lacks the {tree.base_label!r} label")
+ problems.extend(describe(page, problem) for problem in
page_problems)
+ in_folder = {page.page_id for page in pages}
+ for page in labelled[label]:
+ if page.page_id not in in_folder:
+ problems.append(describe(page, f"has the {label!r} label but
is not in that state's folder"))
+ if folder.folded:
+ expected = tree.get_folded_title(folder, len(pages))
+ if f'<ac:parameter ac:name="title">{expected}</ac:parameter>' not
in index.body:
+ problems.append(describe(index, f"the folded section should be
titled {expected!r}"))
+ return problems
+
+
+def check_adip_pages(pages: list[Page]) -> list[str]:
+ problems = []
+ numbers: dict[int, Page] = {}
+ for page in pages:
+ match = re.match(r"ADIP-(\d+) \S", page.title)
+ if not match:
+ problems.append(describe(page, "title does not start with 'ADIP-'
and its number"))
+ elif (number := int(match.group(1))) in numbers:
+ problems.append(describe(page, f"reuses the number of
{shorten(numbers[number].title)!r}"))
+ else:
+ numbers[number] = page
+ if "(formerly" in page.title:
+ problems.append(
+ describe(page, "title repeats the former AIP number; it is the
Formerly property")
+ )
+ # ADIPs that started as AIPs keep their old number in a Formerly row;
new ones have none.
+ blocks = find_properties(page.body)
+ formerly = get_text(blocks[0].get("Formerly", "")) if blocks else ""
+ if formerly and not re.fullmatch(r"AIP-\d+", formerly):
+ problems.append(describe(page, f"Formerly is {formerly!r},
expected 'AIP-' and its number"))
+ return problems
+
+
+def collect_problems(fetch: Callable[[str, dict[str, str]], dict] = get_json)
-> list[str]:
+ problems = []
+ for tree in TREES:
+ folders = {folder.page_id: fetch_children(folder.page_id, fetch) for
folder in tree.folders}
+ labelled = {
+ tree.get_label(folder): fetch_labelled(tree.get_label(folder),
fetch) for folder in tree.folders
+ }
+ index = fetch_page(tree.index_id, fetch)
+ problems += check_tree(
+ tree, index, fetch_all(f"content/{tree.index_id}/child/page", {},
fetch), folders, labelled
+ )
+ if tree is ADIP_TREE:
+ problems += check_adip_pages([page for pages in folders.values()
for page in pages])
+ return problems
+
+
+def build_slack_payload(problems: list[str], run_url: str | None) -> dict:
+ shown = problems[:MAX_PROBLEMS_IN_SLACK]
+ lines = [
+ f":warning: *AIP/ADIP wiki consistency check found {len(problems)}
problem(s)*",
+ "Move each AIP page to the folder for its state and keep its State
badge and labels in line "
+ "with that folder.",
+ *(f"• {problem}" for problem in shown),
+ ]
+ if len(problems) > len(shown):
+ lines.append(f"… and {len(problems) - len(shown)} more")
+ if run_url:
+ lines.append(f"<{run_url}|View the run>")
+ sections: list[str] = []
+ for line in lines:
+ if sections and len(sections[-1]) + 1 + len(line) <=
MAX_SECTION_LENGTH:
+ sections[-1] += "\n" + line
+ else:
+ sections.append(line)
+ return {
+ "channel": SLACK_CHANNEL,
+ "text": f":warning: AIP/ADIP wiki consistency check found
{len(problems)} problem(s)",
+ "blocks": [{"type": "section", "text": {"type": "mrkdwn", "text":
text}} for text in sections],
+ }
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(
+ description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter
+ )
+ parser.add_argument(
+ "--slack-payload", type=Path, help="write a Slack payload here when
problems are found"
+ )
+ parser.add_argument("--run-url", help="link to the CI run, added to the
Slack message")
+ args = parser.parse_args(argv)
+ problems = collect_problems()
+ if not problems:
+ print("The AIP and ADIP wiki trees are consistent.")
+ return 0
+ print(f"Found {len(problems)} problem(s):")
+ for problem in problems:
+ print(f" - {problem}")
+ if args.slack_payload:
+ args.slack_payload.write_text(json.dumps(build_slack_payload(problems,
args.run_url), indent=2))
+ return 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/tests/ci/test_verify_improvement_proposals_wiki.py
b/scripts/tests/ci/test_verify_improvement_proposals_wiki.py
new file mode 100644
index 00000000000..525ae75eea1
--- /dev/null
+++ b/scripts/tests/ci/test_verify_improvement_proposals_wiki.py
@@ -0,0 +1,505 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import gzip
+import http.client
+import importlib.util
+import json
+import sys
+import time
+from pathlib import Path
+from unittest import mock
+
+import pytest
+
+MODULE_PATH = Path(__file__).resolve().parents[3] / "scripts" / "ci" /
"verify_improvement_proposals_wiki.py"
+
+
[email protected]
+def wiki():
+ module_name = "test_verify_improvement_proposals_wiki_module"
+ sys.modules.pop(module_name, None)
+ spec = importlib.util.spec_from_file_location(module_name, MODULE_PATH)
+ assert spec is not None
+ assert spec.loader is not None
+ module = importlib.util.module_from_spec(spec)
+ sys.modules[module_name] = module
+ spec.loader.exec_module(module)
+ return module
+
+
+def badge(title, colour="Grey"):
+ return (
+ '<ac:structured-macro ac:name="status" ac:schema-version="1"
ac:macro-id="x">'
+ f'<ac:parameter ac:name="colour">{colour}</ac:parameter>'
+ f'<ac:parameter
ac:name="title">{title}</ac:parameter></ac:structured-macro>'
+ )
+
+
+def properties(rows):
+ cells = "".join(
+ f"<tr><td><strong>{key}</strong></td><td>{value}</td></tr>" for key,
value in rows.items()
+ )
+ return (
+ '<ac:structured-macro ac:name="details" ac:schema-version="1"
ac:macro-id="y">'
+
f"<ac:rich-text-body><table><tbody>{cells}</tbody></table></ac:rich-text-body></ac:structured-macro>"
+ )
+
+
+def folded(count, kind, name="AIP"):
+ return (
+ '<ac:structured-macro ac:name="expand"
ac:schema-version="1"><ac:parameter ac:name="title">'
+ f"{count} {kind} {name}s (click to
expand)</ac:parameter></ac:structured-macro>"
+ )
+
+
+def aip(wiki, folder, page_id="1", title="AIP-1 Something", state=None,
labels=None, tree=None):
+ tree = tree or wiki.AIP_TREE
+ return wiki.Page(
+ page_id,
+ title,
+ properties({"State": badge(state or folder.badge) + " extra detail"}),
+ [tree.base_label, tree.get_label(folder)] if labels is None else
labels,
+ )
+
+
+def build_tree(wiki, tree=None, pages=None, index_children=None,
labelled=None, index_body=None):
+ """A consistent tree with one proposal in each state folder, with
overrides."""
+ tree = tree or wiki.AIP_TREE
+ folders = {f.page_id: [aip(wiki, f, page_id=f"p{f.page_id}", tree=tree)]
for f in tree.folders}
+ folders.update(pages or {})
+ children = [wiki.Page(f.page_id, f.badge) for f in tree.folders] + [
+ wiki.Page(page_id, "other") for page_id in tree.other_children
+ ]
+ by_label = {tree.get_label(f): folders[f.page_id] for f in tree.folders}
+ by_label.update(labelled or {})
+ body = (
+ index_body
+ if index_body is not None
+ else folded(1, "completed", tree.name) + folded(1, "abandoned or
replaced", tree.name)
+ )
+ return (
+ tree,
+ wiki.Page(tree.index_id, f"{tree.name} index", body),
+ children + (index_children or []),
+ folders,
+ by_label,
+ )
+
+
+def folder_of(wiki, badge_title, tree=None):
+ return next(f for f in (tree or wiki.AIP_TREE).folders if f.badge ==
badge_title)
+
+
[email protected]("tree_name", ["AIP_TREE", "ADIP_TREE"])
+def test_consistent_trees_have_no_problems(wiki, tree_name):
+ assert wiki.check_tree(*build_tree(wiki, tree=getattr(wiki, tree_name)))
== []
+
+
+def test_page_directly_under_the_aip_index_is_reported(wiki):
+ stray = wiki.Page("99", "AIP-200 New")
+ problems = wiki.check_tree(*build_tree(wiki, index_children=[stray]))
+ assert problems == [
+ f"<{stray.url}|AIP-200 New>: sits directly under the AIP index; move
it into a state folder"
+ ]
+
+
[email protected](
+ ("body", "expected"),
+ [
+ pytest.param("", "has 0 Page Properties blocks, expected 1",
id="no-block"),
+ pytest.param(
+ properties({"State": badge("Completed")}) * 2,
+ "has 2 Page Properties blocks, expected 1",
+ id="two-blocks",
+ ),
+ pytest.param(
+ properties({"Discussion Thread": "x"}),
+ "has no State row in its Page Properties block",
+ id="no-state",
+ ),
+ pytest.param(
+ properties({"State": "Completed"}),
+ "State badge is None, expected one of ['Completed']",
+ id="plain-text-state",
+ ),
+ pytest.param(
+ properties({"State": badge("Accepted")}),
+ "State badge is 'Accepted', expected one of ['Completed']",
+ id="badge-of-another-state",
+ ),
+ ],
+)
+def test_aip_properties_must_match_the_folder(wiki, body, expected):
+ completed = folder_of(wiki, "Completed")
+ page = wiki.Page("5", "AIP-5 X", body, [wiki.AIP_TREE.base_label,
wiki.AIP_TREE.get_label(completed)])
+ problems = wiki.check_tree(*build_tree(wiki, pages={completed.page_id:
[page]}))
+ assert problems == [f"<{page.url}|AIP-5 X>: {expected}"]
+
+
[email protected](
+ ("labels", "expected"),
+ [
+ pytest.param(
+ ["airflow-improvement-proposal"],
+ ["state labels are [], expected
['airflow-improvement-proposal/completed']"],
+ id="no-state-label",
+ ),
+ pytest.param(
+ [
+ "airflow-improvement-proposal",
+ "airflow-improvement-proposal/completed",
+ "airflow-improvement-proposal/draft",
+ ],
+ [
+ "state labels are ['airflow-improvement-proposal/completed',
'airflow-improvement-proposal/draft'],"
+ " expected ['airflow-improvement-proposal/completed']"
+ ],
+ id="two-state-labels",
+ ),
+ pytest.param(
+ ["airflow-improvement-proposal/completed"],
+ ["lacks the 'airflow-improvement-proposal' label"],
+ id="no-base-label",
+ ),
+ ],
+)
+def test_aip_labels_must_match_the_folder(wiki, labels, expected):
+ completed = folder_of(wiki, "Completed")
+ page = aip(wiki, completed, page_id="5", title="AIP-5 X", labels=labels)
+ problems = wiki.check_tree(*build_tree(wiki, pages={completed.page_id:
[page]}))
+ assert problems == [f"<{page.url}|AIP-5 X>: {problem}" for problem in
expected]
+
+
+def test_restricted_page_is_not_checked_for_badge_or_labels(wiki):
+ completed = folder_of(wiki, "Completed")
+ page = wiki.Page(next(iter(wiki.RESTRICTED_PAGES)), "AIP-52 X",
properties({"State": "Completed"}), [])
+ assert wiki.check_tree(*build_tree(wiki, pages={completed.page_id:
[page]})) == []
+
+
+def test_titles_and_problems_are_escaped_and_shortened_for_slack(wiki):
+ page = wiki.Page("9", "AIP-9 <!channel> & a|b")
+ assert (
+ wiki.describe(page, "see <x>") == f"<{page.url}|AIP-9 <!channel>
& a|b>: see <x>"
+ )
+ long_page = wiki.Page("10", "A" * 300)
+ assert wiki.describe(long_page, "x") == f"<{long_page.url}|{'A' * 99}…>: x"
+
+
[email protected](
+ "macro",
+ [
+ pytest.param('<ac:structured-macro ac:name="details"
ac:schema-version="1" />', id="self-closing"),
+ pytest.param(
+ '<ac:structured-macro ac:name="details"
ac:schema-version="1"></ac:structured-macro>',
+ id="no-body",
+ ),
+ pytest.param(
+ '<ac:structured-macro ac:name="details"><ac:parameter
ac:name="id">x</ac:parameter>'
+ '</ac:structured-macro><ac:structured-macro ac:name="expand">'
+ '<ac:parameter
ac:name="title">T</ac:parameter><ac:rich-text-body>',
+ id="parameter-then-another-macro-body",
+ ),
+ ],
+)
+def test_macro_without_a_body_has_no_properties_even_before_a_table(wiki,
macro):
+ body = macro +
f"<table><tbody><tr><td>State</td><td>{badge('Completed')}</td></tr></tbody></table>"
+ assert wiki.find_properties(body) == [{}]
+
+
[email protected](
+ "parameters",
+ [
+ pytest.param('<ac:parameter ac:name="id" />', id="self-closing"),
+ pytest.param('<ac:parameter ac:name="id">x</ac:parameter>',
id="closed"),
+ pytest.param(
+ '<ac:parameter ac:name="id">x</ac:parameter> <ac:parameter
ac:name="hidden" />', id="both"
+ ),
+ ],
+)
+def test_properties_after_macro_parameters_are_read(wiki, parameters):
+ body = properties({"State": badge("Completed")}).replace(
+ "<ac:rich-text-body>", parameters + "<ac:rich-text-body>", 1
+ )
+ assert wiki.find_badge(wiki.find_properties(body)[0]["State"]) ==
"Completed"
+
+
[email protected](
+ "key",
+ [
+ pytest.param("<strong>State</strong>:", id="colon-after-bold"),
+ pytest.param("<strong>State:</strong>", id="colon-in-bold"),
+ pytest.param("<p><strong>State</strong> :</p>",
id="space-before-colon"),
+ ],
+)
+def test_property_key_with_a_formatted_colon_is_read(wiki, key):
+ body = (
+ '<ac:structured-macro
ac:name="details"><ac:rich-text-body><table><tbody>'
+ f"<tr><td>{key}</td><td>{badge('Completed')}</td></tr>"
+ "</tbody></table></ac:rich-text-body></ac:structured-macro>"
+ )
+ assert wiki.find_badge(wiki.find_properties(body)[0]["State"]) ==
"Completed"
+
+
+def test_macro_with_many_parameters_and_no_body_is_read_in_linear_time(wiki):
+ body = '<ac:structured-macro ac:name="details"><ac:parameter>' +
"</ac:parameter><ac:parameter>" * 24
+ start = time.monotonic()
+ assert wiki.find_properties(body) == [{}]
+ assert time.monotonic() - start < 1
+
+
[email protected]("tree_name", ["AIP_TREE", "ADIP_TREE"])
+def test_folder_moved_out_of_the_index_is_reported(wiki, tree_name):
+ tree, index, children, folders, labelled = build_tree(wiki,
tree=getattr(wiki, tree_name))
+ moved = children[0]
+ problems = wiki.check_tree(tree, index, children[1:], folders, labelled)
+ assert problems == [
+ f"<{moved.url}|page {moved.page_id}>: folder is no longer directly
under the {tree.name} index"
+ ]
+
+
+def test_properties_with_a_nested_macro_in_a_cell_are_read(wiki):
+ nested = (
+ '<ac:structured-macro
ac:name="info"><ac:rich-text-body><p>note</p></ac:rich-text-body>'
+ "</ac:structured-macro>"
+ )
+ body = properties({"Discussion Thread": nested, "State":
badge("Completed")})
+ assert wiki.find_badge(wiki.find_properties(body)[0]["State"]) ==
"Completed"
+
+
[email protected](
+ "nested",
+ [
+
pytest.param("<table><tbody><tr><td>a</td><td>b</td></tr></tbody></table>",
id="nested-table"),
+ pytest.param(
+
"<table><tbody><tr><th>State</th><td>WRONG</td></tr></tbody></table>",
id="nested-state-row"
+ ),
+ ],
+)
+def test_properties_after_a_value_with_a_nested_table_are_read(wiki, nested):
+ body = properties({"Discussion Thread": nested, "State":
badge("Completed")})
+ rows = wiki.find_properties(body)[0]
+ assert list(rows) == ["Discussion Thread", "State"]
+ assert rows["Discussion Thread"] == nested
+ assert wiki.find_badge(rows["State"]) == "Completed"
+
+
+def test_properties_table_outside_the_macro_is_not_read(wiki):
+ body = (
+ '<ac:structured-macro ac:name="details"
ac:schema-version="1"><ac:rich-text-body><p></p>'
+ "</ac:rich-text-body></ac:structured-macro>"
+
f"<table><tbody><tr><td>State</td><td>{badge('Completed')}</td></tr></tbody></table>"
+ )
+ assert wiki.find_properties(body) == [{}]
+
+
+def test_state_label_outside_its_folder_is_reported(wiki):
+ draft = folder_of(wiki, "Under discussion")
+ label = wiki.AIP_TREE.get_label(draft)
+ elsewhere = wiki.Page("77", "AIP-101: Appendix", labels=[label])
+ tree = build_tree(wiki)
+ tree[4][label] = [*tree[3][draft.page_id], elsewhere]
+ problems = wiki.check_tree(*tree)
+ assert problems == [
+ f"<{elsewhere.url}|AIP-101: Appendix>: has the '{label}' label but is
not in that state's folder"
+ ]
+
+
[email protected]("tree_name", ["AIP_TREE", "ADIP_TREE"])
+def test_stale_count_in_a_folded_section_is_reported(wiki, tree_name):
+ name = getattr(wiki, tree_name).name
+ tree = build_tree(
+ wiki,
+ tree=getattr(wiki, tree_name),
+ index_body=folded(7, "completed", name) + folded(1, "abandoned or
replaced", name),
+ )
+ problems = wiki.check_tree(*tree)
+ assert problems == [
+ f"<{tree[1].url}|{name} index>: the folded section should be titled "
+ f"'1 completed {name}s (click to expand)'"
+ ]
+
+
+def adip(wiki, number=1, title=None, formerly="AIP-4"):
+ rows = {"State": badge("Completed", "Green")}
+ if formerly is not None:
+ rows["Formerly"] = f'<span style="color:
rgb(151,160,175);">{formerly}</span>'
+ return wiki.Page(
+ str(number), title or f"ADIP-{number} Something", properties(rows),
["adip", "adip/completed"]
+ )
+
+
+def test_consistent_adip_pages_have_no_problems(wiki):
+ assert wiki.check_adip_pages([adip(wiki, 1), adip(wiki, 2,
formerly="AIP-6")]) == []
+
+
[email protected](
+ ("page_kwargs", "expected"),
+ [
+ pytest.param(
+ {"title": "Automation of tests"}, ["title does not start with
'ADIP-' and its number"], id="title"
+ ),
+ pytest.param(
+ {"title": "ADIP-2 Something (formerly AIP-6)"},
+ ["title repeats the former AIP number; it is the Formerly
property"],
+ id="formerly-in-title",
+ ),
+ pytest.param({"formerly": None}, [], id="new-adip-without-formerly"),
+ pytest.param(
+ {"formerly": "AIP 6"}, ["Formerly is 'AIP 6', expected 'AIP-' and
its number"], id="bad-formerly"
+ ),
+ ],
+)
+def test_adip_page_problems(wiki, page_kwargs, expected):
+ page = adip(wiki, 2, **page_kwargs)
+ assert wiki.check_adip_pages([adip(wiki, 1), page]) == [
+ f"<{page.url}|{page.title}>: {p}" for p in expected
+ ]
+
+
+def test_duplicate_adip_number_is_reported_with_the_other_title_escaped(wiki):
+ first, second = adip(wiki, 3, title="ADIP-3 <!channel>"), adip(wiki, 4,
title="ADIP-3 Two")
+ assert wiki.check_adip_pages([first, second]) == [
+ f"<{second.url}|ADIP-3 Two>: reuses the number of 'ADIP-3
<!channel>'"
+ ]
+
+
+def test_fetch_children_pages_through_the_listing_and_reads_each_body(wiki):
+ calls = []
+
+ def fetch(path, params):
+ calls.append((path, params.get("start")))
+ if path.endswith("/child/page"):
+ if params["start"] == "0":
+ return {"results": [{"id": "1", "title": "A"}], "_links":
{"next": "more"}}
+ return {"results": [{"id": "2", "title": "B"}], "_links": {}}
+ page_id = path.rsplit("/", 1)[1]
+ return {
+ "id": page_id,
+ "title": f"T{page_id}",
+ "body": {"storage": {"value": f"<p>{page_id}</p>"}},
+ "metadata": {"labels": {"results": [{"name": "adip"}]}},
+ }
+
+ pages = wiki.fetch_children("10", fetch)
+ assert [(p.page_id, p.title, p.body, p.labels) for p in pages] == [
+ ("1", "T1", "<p>1</p>", ["adip"]),
+ ("2", "T2", "<p>2</p>", ["adip"]),
+ ]
+ assert calls == [
+ ("content/10/child/page", "0"),
+ ("content/10/child/page", "1"),
+ ("content/1", None),
+ ("content/2", None),
+ ]
+
+
+def test_fetch_labelled_searches_by_label(wiki):
+ fetch = mock.create_autospec(
+ wiki.get_json, return_value={"results": [{"id": "3", "title":
"AIP-3"}], "_links": {}}
+ )
+ assert [p.page_id for p in
wiki.fetch_labelled("airflow-improvement-proposal/wip", fetch)] == ["3"]
+ assert (
+ fetch.call_args.args[1]["cql"]
+ == 'label = "airflow-improvement-proposal/wip" and type = page and
space = AIRFLOW'
+ )
+
+
[email protected]("urllib.request.urlopen")
+def test_get_json_retries_and_decompresses_gzip(urlopen, wiki):
+ response = mock.MagicMock(spec=http.client.HTTPResponse)
+ response.__enter__.return_value = response
+ response.read.return_value = gzip.compress(json.dumps({"ok":
True}).encode())
+ response.headers = {"Content-Encoding": "gzip"}
+ urlopen.side_effect = [http.client.RemoteDisconnected("gone"), response]
+ sleep = mock.create_autospec(time.sleep)
+ assert wiki.get_json("content/1", {}, sleep=sleep) == {"ok": True}
+ sleep.assert_called_once_with(5)
+ assert urlopen.call_args.args[0].get_header("Accept-encoding") == "gzip"
+
+
[email protected]("urllib.request.urlopen",
side_effect=http.client.IncompleteRead(b""))
+def test_get_json_gives_up_after_the_last_attempt(urlopen, wiki):
+ with pytest.raises(http.client.IncompleteRead):
+ wiki.get_json("content/1", {}, sleep=mock.create_autospec(time.sleep))
+ assert urlopen.call_count == wiki.ATTEMPTS
+
+
+def test_slack_payload_lists_problems_and_truncates(wiki):
+ problems = [f"problem {i}" for i in range(wiki.MAX_PROBLEMS_IN_SLACK + 2)]
+ payload = wiki.build_slack_payload(problems, "https://run")
+ text = payload["blocks"][0]["text"]["text"]
+ assert payload["channel"] == wiki.SLACK_CHANNEL
+ assert f"found {len(problems)} problem(s)" in payload["text"]
+ assert "• problem 0" in text
+ assert f"• problem {wiki.MAX_PROBLEMS_IN_SLACK}" not in text
+ assert "… and 2 more" in text
+ assert text.endswith("<https://run|View the run>")
+
+
+def test_main_exits_0_and_writes_no_payload_when_consistent(wiki, tmp_path):
+ payload = tmp_path / "slack.json"
+ with mock.patch.object(wiki, "collect_problems", return_value=[]):
+ assert wiki.main(["--slack-payload", str(payload)]) == 0
+ assert not payload.exists()
+
+
+def test_main_exits_1_and_writes_the_payload_on_problems(wiki, tmp_path):
+ payload = tmp_path / "slack.json"
+ with mock.patch.object(wiki, "collect_problems", return_value=["bad
page"]):
+ assert wiki.main(["--slack-payload", str(payload), "--run-url",
"https://run"]) == 1
+ assert "• bad page" in
json.loads(payload.read_text())["blocks"][0]["text"]["text"]
+
+
+def test_slack_payload_splits_long_lists_into_sections_within_the_limit(wiki):
+ problems = [f"problem {i} " + "x" * 400 for i in
range(wiki.MAX_PROBLEMS_IN_SLACK)]
+ sections = [block["text"]["text"] for block in
wiki.build_slack_payload(problems, None)["blocks"]]
+ assert len(sections) > 1
+ assert all(len(text) <= wiki.MAX_SECTION_LENGTH for text in sections)
+ assert "\n".join(sections).count("• problem") == wiki.MAX_PROBLEMS_IN_SLACK
+
+
+def test_collect_problems_checks_both_trees_and_the_adip_pages(wiki):
+ def fetch_children(page_id, fetch):
+ return [wiki.Page(f"in-{page_id}", "x")]
+
+ with (
+ mock.patch.object(wiki, "fetch_children", side_effect=fetch_children),
+ mock.patch.object(wiki, "fetch_labelled", return_value=[]),
+ mock.patch.object(wiki, "fetch_page", side_effect=lambda page_id,
fetch: wiki.Page(page_id, "index")),
+ mock.patch.object(wiki, "fetch_all", return_value=[]),
+ mock.patch.object(wiki, "check_tree", side_effect=lambda tree, *args:
[f"tree {tree.name}"]),
+ mock.patch.object(wiki, "check_adip_pages", return_value=["adip
pages"]) as check_adip_pages,
+ ):
+ assert
wiki.collect_problems(fetch=mock.create_autospec(wiki.get_json)) == [
+ "tree AIP",
+ "tree ADIP",
+ "adip pages",
+ ]
+ assert [page.page_id for page in check_adip_pages.call_args.args[0]] == [
+ f"in-{folder.page_id}" for folder in wiki.ADIP_TREE.folders
+ ]
+
+
+def
test_long_wiki_text_in_a_problem_keeps_the_slack_payload_within_limits(wiki):
+ page = adip(wiki, 2, formerly="x" * 5000)
+ problems = wiki.check_adip_pages([page])
+ assert len(problems) == 1
+ assert problems[0].endswith("…")
+ sections = [block["text"]["text"] for block in
wiki.build_slack_payload(problems, None)["blocks"]]
+ assert all(len(text) <= wiki.MAX_SECTION_LENGTH for text in sections)