morningman opened a new pull request, #70232: URL: https://github.com/apache/doris/pull/70232
### Background The `http()` table-valued function reads CSV, JSON, Parquet, and ORC files from HTTP(S) URLs; for an `hf://` URI the FE expands the HuggingFace path into download URLs. `external_table_p0/tvf/test_http_tvf` (group `p0`, so every P0 run executes it) serves most of its fixtures from an HTTP server it starts inside the regression runner, and builds their URLs from `httpTvfHost`, else `externalEnvIp`, which is `127.0.0.1` in every pipeline config. Its last seven queries read public HuggingFace datasets through `hf://`. ### What problem does this PR solve? Issue Number: None Related PR: #65877, #66013, #66027 (branch-4.1 fixes by @hello-stephen, ported here), #65299 (added the in-runner HTTP server and its host lookup), #60915 (moved the suite into P0 and relaxed sql15 / sql16 to `count > 0`) PR Type: Test Problem Summary: On master the suite still depends on huggingface.co and on HuggingFace's mutable `main` branches, and it advertises the runner as `127.0.0.1`, which a BE on another host resolves to itself. This PR ports the three branch-4.1 fixes: a loopback or wildcard `externalEnvIp` falls back to the runner's host address (an explicit `httpTvfHost` is still used as given), the seven `hf://` reads use byte-identical mirrors in the regression OSS bucket pinned to a dataset revision, and the mirrored `prompts.csv` skips its header line. With the mirror pinned, sql15 / sql16 assert the exact count 204 again. ### Details - **Failure:** None on master CI today: P0 passed 66 of 66 runs on 2026-11-04, though the HuggingFace queries took about 146 of the suite's 149 seconds (build 1092587). On branch-4.1 the suite failed with `Connection refused` on `http://127.0.0.1:<port>` with remote BEs (#65877), with `java.net.SocketTimeoutException: Connect timed out` to HuggingFace (#66013), and then with sql15 getting 205 instead of 204 on the mirror (#66027). - **Cause:** The suite assumed that every BE runs on the runner's host, true in P0 only, and that huggingface.co is reachable with fixed `main` branches: `prompts.csv` on `fka/awesome-chatgpt-prompts` `main` now has five columns and is over six times the 204-record file the golden came from. The mirrored CSV keeps a header line that `hf://` hid, hence `skip_lines = 1`. - **Stability evidence:** No query reaches huggingface.co any more. The only remote host left is `doris-regression-hk.oss-cn-hongkong.aliyuncs.com` (public reads), which P0 already uses in `query_p0/test_parquet_dict` (28 of 28 recent runs passed). Since master's P0 does not fail today, there is no before-and-after failure rate. ### Testing - `-forceGenOut` run of the suite on a local 1 FE + 1 BE cluster from master `4530f761fa1` (the FE carries local patches that don't touch the HTTP TVF), with P0's `externalEnvIp = 127.0.0.1` and no `httpTvfHost`: the runner advertised `http://192.168.1.164:<port>`, and the 22 blocks generated for sql01 to sql18 and both `replaced_*` checks are byte-identical to master's `.out`, with sql15 / sql16 now after `replaced_after`, where they run. sql19 then hit the 900 s query timeout: this machine downloads the 20 MB Parquet mirror at about 10 KB/s. - The suite without sql19 to sql21, in compare mode against the committed `.out`: passed. With `skip_lines` removed from sql15 it fails with `Expect cell is: 204 But real is: 205`. - sql19 to sql21 did not run locally. Each mirrored Parquet file has the same size as on HuggingFace and a byte-identical footer (schema, row groups, statistics), and each footer's minimum `text` is the first row the `.out` expects. This PR's P0 is their first run against the mirror on master. ### Notes for Reviewers - Apart from a one-line comment on the loopback check, the suite now matches branch-4.2's except for master's own parts: the `p0` label (#60915) and the `mutable/` cases (#68847). - `hf://` loses its only end-to-end coverage (real HuggingFace URLs, tree-API glob expansion, `@revision`); `HFUtilsTest` still covers parsing, URL conversion, and glob detection. - sql16, sql20, and the second sql21 now repeat sql15 / sql19, since plain HTTP has no globbing; they stay so that the goldens and branch-4.x line up. - In P0 the runner now advertises its host address instead of `127.0.0.1`; both work there, but branch-4.x labels the suite `p2`, so P0 runs this fallback for the first time. - `ai_p0/test_ai_profile` copied the old host lookup and has the same latent loopback problem; it is not changed here. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: sql19 to sql21 not run locally (mirrors checked against HuggingFace by size and Parquet footer); this PR's P0 runs them ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: the port of the three branch-4.1 changes onto master's suite, the `.out` update (generated by the framework), and this description - Human review: not stated - Prompts: The author asked to port branch-4.1 #65877, #66013, and #66027 to master as part of moving branch-4.2's changes back to master, and, after seeing that the mirrors drop the only `hf://` end-to-end coverage, chose to port all three as on branch-4.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
