This is an automated email from the ASF dual-hosted git repository.
kaxil pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 548181679d7 Match providers to Explore categories by integration name,
not just id (#70498)
548181679d7 is described below
commit 548181679d7c553c5ce6b6bc569fa1b73f6cae42
Author: Wei Lee <[email protected]>
AuthorDate: Mon Aug 3 22:59:22 2026 +0800
Match providers to Explore categories by integration name, not just id
(#70498)
* Match providers to Explore categories by integration name, not just id
The Explore page and each provider's category chips only matched a
provider to a category by substring-matching the category's keywords
against the provider's id. Providers whose id doesn't happen to
contain a relevant keyword never showed up, even when their
provider.yaml declares integrations that obviously belong there. For
example, common-ai (wraps pydantic-ai) ships hooks named "Pydantic AI",
"MCP Server", "LangChain", and "LlamaIndex", each tagged ai, but none of
those words are substrings of the id "common-ai", so it never appeared
under AI & Machine Learning or in a search for "LangChain"/"MCP".
Match against each provider's declared integration names (already
extracted as provider.categories[].name) in addition to its id, and
add the ai-ml category keywords needed for common-ai's integrations.
Also scan connection_types[].external_integrations when present so a
future field populating that data benefits automatically.
* Add more provider explore-category keywords found via integration-name
audit
Auditing all provider.yaml integration names/tags against each Explore
category's keyword list turned up further clear misses of the same
id-substring bug fixed for common-ai: MSSQL, OpenSearch, IBM Cloudant,
and ArangoDB never showed up under Databases; Exasol and ClickHouse
never showed up under Data Warehouses; DingTalk and Apprise never
showed up under Messaging & Notifications (same tags: [service] class
as the Slack/Discord/Telegram/PagerDuty/Opsgenie entries already
there); and gRPC never showed up under Workflow & Orchestration despite
sharing the same tags: [protocol] as ssh/http/ftp already listed.
Deliberately left out: Apache Kafka and IBM MQ, which also lack any
matching keyword. Both are enterprise message-broker/queueing
middleware (tags: [apache]/[software]) rather than the
notification/alerting channels the Messaging & Notifications category
is themed around and its description names (Slack, email, SMS,
alerting services), so adding them is a judgment call left for a
maintainer rather than an obvious fix.
* Registry: Tighten Explore category keyword matching
The keyword matcher compared in both directions, which was safe while the
left-hand side was always a provider slug. Now that it also runs against
free-text integration names, a long keyword could swallow short ones the
tree already has ("Tpt", "Ttu", "YDB", "Bteq"). Matching forward only
removes that hazard without changing any provider's category membership.
Keywords and integration names also spell separators differently, which
forced duplicate entries such as 'pydantic ai' alongside 'pydantic-ai'.
Collapsing separator runs to a single space lets one keyword cover both
spellings. Collapsing rather than stripping matters: stripping would turn
"Microsoft Power BI" into a string containing "ftp".
The connection-type branch read a field that cannot exist, since the
contract that validates providers.json forbids unknown keys and does not
declare it.
* Registry: Let the providers search box find integration names
Matching providers to Explore categories by integration name fixed the
category dropdown but not the search box beside it, which still compared
the provider's display name alone. Typing the name of an integration a
provider declares — "langchain", "opensearch" — found nothing, which is
the case people actually reach for the search box to solve.
* Registry: Include integration names in the search index
The site has two independent searches: the filter box on /providers/, and
the global modal reached from the header and the home page hero. The modal
is backed by Pagefind, whose provider records are built from custom
content rather than page HTML, so making integration names available as a
data attribute did nothing for it.
That left the more prominent of the two searches unable to find a provider
by an integration it declares — "langchain" returned nothing even though
Common AI lists LangChain.
* Registry: Match the providers search box on separator variants
The build-time category matcher collapses runs of "-_ " to a single space
before comparing, but the client-side filter box compared raw strings, so
the two halves of the same feature disagreed: "pydantic-ai", "pydantic_ai"
and "mcp-server" found nothing, and "speech to text" missed Google because
the integration is named "Google Cloud Speech-to-Text".
* Registry: Build the provider card's integration list inline
The loop that accumulated integration names relied on a push side effect
inside a comma expression, which reads as if it were working around
Nunjucks scoping when a single inline expression renders the same output.
* Registry: Rank Explore category badges by monthly downloads
Each Explore card shows six provider badges taken with `| slice(0, 6)` off
an unsorted list, so the row was whatever providers.json happened to list
first. Widening a category's membership then pushed recognisable providers
out of the row: Messaging & Notifications lost Slack and SMTP while the
card's own copy still read "Slack, email, SMS, and alerting services",
Databases lost MySQL and PostgreSQL, Workflow lost HTTP and SSH.
Membership is unchanged; only which six are shown. The Top Providers and
Incubating Providers rows on the same page already rank before slicing.
* Registry: Correct the docs for what category keywords match
The keyword matcher now compares against a provider's declared integration
names as well as its id, and normalises separators before comparing, but the
prose a contributor reads before adding a keyword still described the
id-only substring rule.
The comment on the shared matcher also credited the per-provider page's
category chips; providerCategoryMap's only consumer is the category
dropdown on /providers/.
* Registry: Let provider search find providers by id
The build-time matcher searches a provider's id alongside its integration
names, but neither search entry point did. On /providers/ the filter box saw
only the display name and the integration names, so `clickhousedb`,
`cncf-kubernetes`, `microsoft-mssql`, `edge3`, `common-dataquality`,
`microsoft-psrp` and `microsoft-winrm` were unreachable from their own id,
and the distribution name the card prints under the title matched nothing at
all. The Pagefind index had the same gap.
The id is indexed on its own rather than as part of the distribution name,
and the shared `apache-airflow-providers-` prefix is dropped off the query
instead: indexing the full name would make `apache`, `airflow` and
`providers` match every provider.
---
registry/README.md | 8 +++-
registry/scripts/build-pagefind-index.mjs | 10 ++++-
registry/src/_data/exploreCategories.js | 10 ++---
registry/src/_data/exploreCategoryProviders.js | 13 ++++--
registry/src/_data/providerCategoryMap.js | 3 +-
registry/src/_data/providerKeywordMatch.js | 58 ++++++++++++++++++++++++++
registry/src/js/provider-filters.js | 26 +++++++++++-
registry/src/providers.njk | 4 +-
8 files changed, 117 insertions(+), 15 deletions(-)
diff --git a/registry/README.md b/registry/README.md
index 4459c65da00..803156c1c4d 100644
--- a/registry/README.md
+++ b/registry/README.md
@@ -256,11 +256,15 @@ The Explore page and provider filtering use categories
defined in
- `id` — URL-safe identifier (e.g., `cloud`, `databases`, `ai-ml`)
- `name` — Display name
-- `keywords` — List of substrings matched against `provider.id`
+- `keywords` — List of substrings matched against `provider.id` and the
provider's
+ declared integration names (`provider.categories[].name`)
- `icon`, `color`, `description` — Visual properties
Providers are assigned to categories at build time by checking if any keyword
in a
-category matches (substring) the provider's ID. A provider can belong to
multiple
+category is a substring of the provider's ID or of one of its declared
integration
+names, compared case-insensitively after collapsing runs of `-`, `_` and
whitespace to
+a single space (so `pydantic-ai` matches the integration named "Pydantic AI").
The
+matcher lives in `src/_data/providerKeywordMatch.js`. A provider can belong to
multiple
categories.
### Homepage "New Providers" Section
diff --git a/registry/scripts/build-pagefind-index.mjs
b/registry/scripts/build-pagefind-index.mjs
index 7811717bdb9..027226ef70b 100644
--- a/registry/scripts/build-pagefind-index.mjs
+++ b/registry/scripts/build-pagefind-index.mjs
@@ -40,9 +40,17 @@ async function buildPagefindIndex() {
let modulesAdded = 0;
for (const provider of providers.providers) {
+ const integrations = (provider.categories || [])
+ .map((category) => category.name)
+ .filter(Boolean)
+ .join(' ');
+
await index.addCustomRecord({
url: `/providers/${provider.id}/${provider.version}/`,
- content: `${provider.name} ${provider.description}`,
+ // The id is indexed on its own, not as part of the distribution name:
+ // the `apache-airflow-providers-` prefix is shared by every provider and
+ // would make 'apache' or 'airflow' match all of them.
+ content: `${provider.name} ${provider.id} ${provider.description}
${integrations}`,
language: 'en',
meta: {
type: 'provider',
diff --git a/registry/src/_data/exploreCategories.js
b/registry/src/_data/exploreCategories.js
index f407377ecf4..04a3de1e042 100644
--- a/registry/src/_data/exploreCategories.js
+++ b/registry/src/_data/exploreCategories.js
@@ -31,7 +31,7 @@ module.exports = [
name: 'Databases',
icon: '<svg fill="none" stroke="currentColor" viewBox="0 0 24 24"><path
stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 7v10c0
2.21 3.582 4 8 4s8-1.79 8-4V7M4 7c0 2.21 3.582 4 8 4s8-1.79 8-4M4 7c0-2.21
3.582-4 8-4s8 1.79 8 4" /></svg>',
color: 'blue',
- keywords: ['postgres', 'mysql', 'mongo', 'redis', 'neo4j',
'elasticsearch', 'cassandra', 'couchbase', 'influxdb', 'sqlite', 'odbc',
'jdbc', 'common-sql'],
+ keywords: ['postgres', 'mysql', 'mongo', 'redis', 'neo4j',
'elasticsearch', 'cassandra', 'couchbase', 'influxdb', 'sqlite', 'odbc',
'jdbc', 'common-sql', 'mssql', 'opensearch', 'cloudant', 'arangodb'],
description: 'SQL, NoSQL, and time-series databases',
},
{
@@ -39,7 +39,7 @@ module.exports = [
name: 'Data Warehouses',
icon: '<svg fill="none" stroke="currentColor" viewBox="0 0 24 24"><path
stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M19 11H5m14
0a2 2 0 012 2v6a2 2 0 01-2 2H5a2 2 0 01-2-2v-6a2 2 0 012-2m14 0V9a2 2 0
00-2-2M5 11V9a2 2 0 012-2m0 0V5a2 2 0 012-2h6a2 2 0 012 2v2M7 7h10" /></svg>',
color: 'purple',
- keywords: ['snowflake', 'databricks', 'teradata', 'vertica', 'trino',
'presto', 'dbt'],
+ keywords: ['snowflake', 'databricks', 'teradata', 'vertica', 'trino',
'presto', 'dbt', 'exasol', 'clickhouse'],
description: 'Snowflake, Databricks, and analytics platforms',
},
{
@@ -47,7 +47,7 @@ module.exports = [
name: 'Messaging & Notifications',
icon: '<svg fill="none" stroke="currentColor" viewBox="0 0 24 24"><path
stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8 12h.01M12
12h.01M16 12h.01M21 12c0 4.418-4.03 8-9 8a9.863 9.863 0 01-4.255-.949L3
20l1.395-3.72C3.512 15.042 3 13.574 3 12c0-4.418 4.03-8 9-8s9 3.582 9 8z"
/></svg>',
color: 'green',
- keywords: ['slack', 'sendgrid', 'smtp', 'discord', 'telegram',
'pagerduty', 'opsgenie', 'twilio'],
+ keywords: ['slack', 'sendgrid', 'smtp', 'discord', 'telegram',
'pagerduty', 'opsgenie', 'twilio', 'dingtalk', 'apprise'],
description: 'Slack, email, SMS, and alerting services',
},
{
@@ -55,7 +55,7 @@ module.exports = [
name: 'AI & Machine Learning',
icon: '<svg fill="none" stroke="currentColor" viewBox="0 0 24 24"><path
stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9.663
17h4.673M12 3v1m6.364 1.636l-.707.707M21 12h-1M4
12H3m3.343-5.657l-.707-.707m2.828 9.9a5 5 0 117.072 0l-.548.547A3.374 3.374 0
0014 18.469V19a2 2 0 11-4 0v-.531c0-.895-.356-1.754-.988-2.386l-.548-.547z"
/></svg>',
color: 'amber',
- keywords: ['openai', 'cohere', 'anthropic', 'huggingface', 'mlflow',
'pinecone', 'qdrant', 'weaviate', 'pgvector'],
+ keywords: ['openai', 'cohere', 'anthropic', 'huggingface', 'mlflow',
'pinecone', 'qdrant', 'weaviate', 'pgvector', 'langchain', 'llamaindex', 'mcp',
'pydantic ai'],
description: 'OpenAI, vector DBs, and ML platforms',
},
{
@@ -71,7 +71,7 @@ module.exports = [
name: 'Workflow & Orchestration',
icon: '<svg fill="none" stroke="currentColor" viewBox="0 0 24 24"><path
stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4
4v5h.582m15.356 2A8.001 8.001 0 004.582 9m0 0H9m11 11v-5h-.581m0 0a8.003 8.003
0 01-15.357-2m15.357 2H15" /></svg>',
color: 'rose',
- keywords: ['standard', 'celery', 'docker', 'kubernetes', 'ssh', 'http',
'ftp', 'sftp'],
+ keywords: ['standard', 'celery', 'docker', 'kubernetes', 'ssh', 'http',
'ftp', 'sftp', 'grpc'],
description: 'Core operators, executors, and connectivity',
},
];
diff --git a/registry/src/_data/exploreCategoryProviders.js
b/registry/src/_data/exploreCategoryProviders.js
index 8cd35ab915a..0b0365733a5 100644
--- a/registry/src/_data/exploreCategoryProviders.js
+++ b/registry/src/_data/exploreCategoryProviders.js
@@ -19,6 +19,7 @@
const providersData = require("./providers.json");
const exploreCategories = require("./exploreCategories");
+const { providerMatchesKeyword } = require("./providerKeywordMatch");
module.exports = function () {
const map = {};
@@ -26,15 +27,19 @@ module.exports = function () {
const matched = [];
for (const provider of providersData.providers) {
for (const keyword of category.keywords) {
- if (
- provider.id.includes(keyword) ||
- keyword.includes(provider.id)
- ) {
+ if (providerMatchesKeyword(provider, keyword)) {
matched.push(provider);
break;
}
}
}
+ // explore.njk shows only the first six as badges, so rank before slicing
+ // (as its Top/Incubating rows already do) — otherwise the row is whatever
+ // providers.json happened to list first, and widening a category's
+ // membership silently pushes the well-known names off it.
+ matched.sort(
+ (a, b) => (b.pypi_downloads?.monthly || 0) - (a.pypi_downloads?.monthly
|| 0),
+ );
map[category.id] = matched;
}
return map;
diff --git a/registry/src/_data/providerCategoryMap.js
b/registry/src/_data/providerCategoryMap.js
index eaf41c67194..8daab272058 100644
--- a/registry/src/_data/providerCategoryMap.js
+++ b/registry/src/_data/providerCategoryMap.js
@@ -19,6 +19,7 @@
const providersData = require("./providers.json");
const exploreCategories = require("./exploreCategories");
+const { providerMatchesKeyword } = require("./providerKeywordMatch");
module.exports = function () {
const map = {};
@@ -26,7 +27,7 @@ module.exports = function () {
const cats = [];
for (const category of exploreCategories) {
for (const keyword of category.keywords) {
- if (provider.id.includes(keyword) || keyword.includes(provider.id)) {
+ if (providerMatchesKeyword(provider, keyword)) {
cats.push(category.id);
break;
}
diff --git a/registry/src/_data/providerKeywordMatch.js
b/registry/src/_data/providerKeywordMatch.js
new file mode 100644
index 00000000000..8f3bfd4f5bf
--- /dev/null
+++ b/registry/src/_data/providerKeywordMatch.js
@@ -0,0 +1,58 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// Shared by providerCategoryMap.js and exploreCategoryProviders.js so the
+// two directions of the same provider <-> explore-category matching (used
+// respectively by the category dropdown on /providers/ and the Explore landing
+// page's per-category provider listing) can't drift apart.
+
+// A value "matches" a keyword if the value contains the keyword,
+// case-insensitively, after collapsing runs of "-_\s" to a single space
+// (so 'pydantic-ai' matches "Pydantic AI"). Collapse, don't strip:
+// stripping would turn "Microsoft Power BI" into "microsoftpowerbi",
+// which contains "ftp" and would falsely match the orchestration category.
+function normalize(text) {
+ return text.toLowerCase().replace(/[-_\s]+/g, ' ');
+}
+
+function fuzzyIncludes(value, keyword) {
+ if (!value) {
+ return false;
+ }
+ return normalize(value).includes(normalize(keyword));
+}
+
+// Every string a provider is searchable by: its id/slug and its declared
+// integration names (provider.categories[].name — e.g. "LangChain",
+// "Pydantic AI").
+function collectSearchableValues(provider) {
+ const values = [provider.id];
+ for (const category of provider.categories || []) {
+ if (category.name) {
+ values.push(category.name);
+ }
+ }
+ return values;
+}
+
+function providerMatchesKeyword(provider, keyword) {
+ return collectSearchableValues(provider).some((value) =>
fuzzyIncludes(value, keyword));
+}
+
+module.exports = { providerMatchesKeyword };
diff --git a/registry/src/js/provider-filters.js
b/registry/src/js/provider-filters.js
index 721594659c9..a143003acf1 100644
--- a/registry/src/js/provider-filters.js
+++ b/registry/src/js/provider-filters.js
@@ -33,16 +33,40 @@
let currentSearch = '';
let debounceTimer;
+ // Kept in sync by hand with normalize() in
src/_data/providerKeywordMatch.js:
+ // that module runs at build time under CommonJS and this file is a browser
+ // IIFE, so the two can't share it. If the two drift, a query like
+ // 'pydantic-ai' stops finding the integration named "Pydantic AI".
+ function normalize(text) {
+ return text.toLowerCase().replace(/[-_\s]+/g, ' ');
+ }
+
+ // The card prints the full distribution name under the title, so that string
+ // has to be a working query. Matching against it directly would make
+ // 'apache', 'airflow' and 'providers' match every card, so drop the shared
+ // prefix off the query instead and match the id alone.
+ const PACKAGE_PREFIX = normalize('apache-airflow-providers-');
+
+ function normalizeSearch(text) {
+ const search = normalize(text);
+ return search.startsWith(PACKAGE_PREFIX) ?
search.slice(PACKAGE_PREFIX.length) : search;
+ }
+
function filterProviders() {
let visibleCount = 0;
+ const search = normalizeSearch(currentSearch);
providerItems.forEach(item => {
const lifecycle = item.dataset.lifecycle;
const name = item.dataset.name || '';
+ const id = item.dataset.id || '';
const categories = item.dataset.categories || '';
+ const integrations = item.dataset.integrations || '';
const matchesLifecycle = currentLifecycle === 'all' || lifecycle ===
currentLifecycle;
- const matchesSearch = name.includes(currentSearch.toLowerCase());
+ const matchesSearch = normalize(name).includes(search) ||
+ normalize(id).includes(search) ||
+ normalize(integrations).includes(search);
const matchesCategory = currentCategory === 'all' ||
categories.split(',').includes(currentCategory);
diff --git a/registry/src/providers.njk b/registry/src/providers.njk
index f5d1b524c84..5668cf98e2c 100644
--- a/registry/src/providers.njk
+++ b/registry/src/providers.njk
@@ -66,9 +66,11 @@ mainClass: providers-page
<li class="provider-item"
data-lifecycle="{{ lcDisplay }}"
data-name="{{ provider.name | lower }}"
+ data-id="{{ provider.id | lower }}"
data-downloads="{{ provider.pypi_downloads.monthly }}"
data-updated="{{ provider.last_updated or '' }}"
- data-categories="{{ cats | join(',') }}">
+ data-categories="{{ cats | join(',') }}"
+ data-integrations="{{ (provider.categories or []) |
selectattr('name') | join(',', 'name') | lower }}">
{% include "provider-card.njk" %}
</li>
{% endfor %}
