aminghadersohi commented on PR #45149:
URL: https://github.com/apache/superset/pull/45149#issuecomment-6088594847
## Acceptance test at head `23b892033b96edeabc9c67c1ee2546e289786142`
Setup: local `superset mcp run --host 127.0.0.1 --port 5899` from this
branch (SQLite metadata DB, `MCP_DEV_USERNAME=admin`, default
`MCP_TOOL_SEARCH_CONFIG`: bm25, `max_results: 5`, `always_visible:
[health_check, get_instance_info]`), driven over streamable HTTP with a
`fastmcp.Client` calling `search_tools`. Baseline: the same server run from the
merge base `1b35374fec` on port 5898.
| # | Criterion | Result | Evidence |
|---|---|---|---|
| 1 | For `get dataset info execute sql`, the tool named by the query
(`get_dataset_info`) ranks above unrelated tools, including the pinned
always-visible ones | PASS | HEAD: `['get_dataset_info', 'execute_sql',
'query_dataset', 'get_chart_sql', 'update_dataset']`<br>Base:
`['query_dataset', 'execute_sql', 'get_chart_sql', 'update_dataset',
'get_dataset_info']` (last slot, at risk of falling off with a slightly
different catalog)<br>`health_check` / `get_instance_info` stay pinned in
`list_tools` and do not take search slots |
| 2 | Existing top-k contract kept (≤ `max_results`; no promise that every
multi-intent tool is returned) | PASS | Every response above has exactly 5
tools. `list charts and get dashboard info` → `['get_dashboard_info',
'list_charts', 'get_dashboard_layout', 'get_chart_info', 'get_report_info']`
(base lacked `list_charts`). Unit tests `test_tool_search_transform.py` cover
the limit holding when more tools qualify |
| 3 | Exact-name search still works | PASS | `get_dataset_info` →
`get_dataset_info` first (same as base); `get dataset info` → same |
| 4 | No spurious promotion from scattered or partial name words | PASS |
`info dataset get` (out of order) → `get_dataset_info` not promoted, same
output as base; `execute sql` → unchanged from base |
| 5 | `execute_sql` safety metadata unchanged | PASS | `annotations:
{readOnlyHint: False, destructiveHint: True, ...}`, `tags: ['mutate']` |
| 6 | Unit tests | PASS | `pytest
tests/unit_tests/mcp_service/test_tool_search_transform.py
tests/unit_tests/mcp_service/test_mcp_server.py
tests/unit_tests/mcp_service/test_mcp_config.py` → `176 passed`<br>`pytest
tests/unit_tests/mcp_service/` → `8897 passed, 11 skipped` |
Not exercised live: the downstream gateway's own `limit` and merging
(outside this repo). The local `search_tools` takes only `query`, so the
`limit: 10` variant does not apply here. No datasets were needed, since search
ranks the tool catalog only.
Verdict: **PASS**. Local servers and temporary venv/DB have been torn down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
