adityamparikh opened a new pull request, #210:
URL: https://github.com/apache/solr-mcp/pull/210
Implements #208: an `index-url` tool that indexes a JSON, CSV, XML or
Markdown document set from an http(s) URL into a Solr collection without the
payload passing through the model, registered in **both** the STDIO and HTTP
transports.
Design record with every decision and its reason:
`docs/superpowers/specs/2026-09-15-url-ingestion-design.md` (in this PR).
## Why
The four inline indexing tools take the payload as a tool-call argument, so
the model emits every byte. Measured when #197 was closed: 61 records took over
two minutes, of which Solr took under a second. The server's differentiator is
the onboarding loop (look at data, design schema, index, verify, search) and it
broke at the indexing step, with no workaround in chat-only clients such as
Claude Desktop. A URL argument is ~20 tokens regardless of payload size.
## What
```
index-url(collection, url, format?)
```
- **Allow-listed hosts**, checked on the requested URL and on every redirect
hop. `SOLR_INDEX_URL_ALLOWED_HOSTS` takes exact hosts, `*.suffix` patterns or
`*`. Default: `raw.githubusercontent.com,*.githubusercontent.com,github.com`,
so the tutorial URL works with no configuration and an HTTP deployment is not
an SSRF primitive on day one.
- **Link-local and cloud-metadata addresses refused** on every hop even with
`*`.
- **No credentials or caller headers, ever**; embedded `user:pass@` is
rejected. Only `Accept` and `User-Agent` are sent.
- **Redirects:** five followed, the sixth is an error, `https`→`http`
refused.
- **Non-2xx is an error before any body is read** (a GitHub 404 body would
otherwise index as a CSV row).
- **Size cap** `SOLR_INDEX_URL_MAX_BYTES`, default 10 MB, checked from
`Content-Length` and while reading. Over the cap, the error points at `bin/solr
post` / the `/update` handler.
- **Format:** explicit `format` > requested URL extension (query string
ignored) > final URL extension > `Content-Type`; `text/plain` and `text/html`
never resolve.
- **Body read in memory** and handed to the existing document creators
through a new shared `IndexingService.indexPayload`, which the four inline
tools now also use (`IndexingServiceTest` passes unmodified). Nothing is
indexed unless the whole document parses, so every fetch or parse error
truthfully says so.
- **HTTP client:** Spring `RestClient` over `HttpURLConnection` (already on
the classpath), whose read timeout covers the body; redirect following disabled
at the connection so the policy sees each hop. `SOLR_INDEX_URL_CONNECT_TIMEOUT`
10s, `SOLR_INDEX_URL_READ_TIMEOUT` 30s.
- **Guidance:** the `index-data` prompt and the server instructions route
URLs under the cap to `index-url`, larger datasets to `bin/solr post` or `curl`
against `/update` (run where the file is, which also covers Claude Desktop
local files), and pasted data to the inline tools.
## Relationship to earlier decisions
- **#194** closed a STDIO-only `index-file` for giving the transports
different surfaces. This PR adds no transport-specific tool and reverses
nothing recorded there; the local-file case is served by `bin/solr post`
guidance.
- **#197** kept per-format inline tools for token shape. `index-url` carries
no payload, so one tool with an optional `format` suffices.
- **#205** (forward CSV/XML to Solr's handlers): no conflict. This PR adds
no parser code and touches the four inline tool methods only to extract
`indexPayload`. After #205 merges, a follow-up can proxy the fetched body
straight to Solr's `/update` for JSON/CSV/XML and drop the cap for those
formats with the same tool contract.
- A first version that streamed through server-side parsers with a
hand-written watchdog, kept `index-file`, and allowed any host by default is
preserved at `wip/index-url-streaming` and is not proposed.
## Threat model
`THREAT_MODEL.md` §8.5 said the client "cannot inject a target URL" at
critical severity; this tool exercises that deliberately behind an allow-list.
The PR narrows §8.5 to the Solr backend and its credentials, adds the
allow-listed fetch and the DNS-rebinding window as a §9 bounded property, adds
the `*`-on-an-internal-network misuse to §11, the `index-url` non-finding to
§11a, a dated entry to §12, qualifies §13, adds the three knobs to §5a,
corrects the tool counts in §1/§5a, and updates `docs/security/stdio.md` and
`http.md`.
**Two defaults are the author's choice and need maintainer confirmation:**
the allow-list contents and the 10 MB cap. Both are single constants.
## Tests
- New, all running natively: `UrlTargetPolicyTest` (21), `IndexFormatsTest`
(14), `UrlFetcherTest` (17, real JDK `HttpServer`),
`UrlIndexingIntegrationTest` (5, Testcontainers Solr).
- `UrlIndexingServiceTest` (14, Mockito, the only class skipped natively).
- Both MCP client transports assert `index-url` and its hints through the
shared base and add a URL round trip plus a refused metadata address.
- `IndexingServiceTest` unmodified.
## Verification
```
./gradlew build BUILD SUCCESSFUL — 478 tests, 0 failures, 7 skipped
(all 7 in OtlpExportIntegrationTest, @Disabled on
main since 375a710)
./gradlew nativeTest -Pnative BUILD SUCCESSFUL — 317 successful, 0 failed,
158 skipped
(skipped = the pre-existing @DisabledInNativeImage
Mockito classes and the
@Disabled OTLP class, plus UrlIndexingServiceTest,
the one new Mockito class)
```
Manual STDIO run of the fat jar against a local Solr, indexing the raw
GitHub `shows.json`:
```
[tools/list] index-url present
[index-url] isError=False :: Successfully indexed 61 of 61 documents into
collection 'shows-url-manual'. Indexed field names ...
[search] isError=False :: {"numFound":61,...}
[index-url http://example.invalid/x.json] isError=True :: The URL's host is
not on this server's allow-list. Allowed by default: raw.githubusercontent.com,
*.githubusercontent.com, github.com. ...
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
