morningman opened a new pull request, #68617:
URL: https://github.com/apache/doris/pull/68617
### What problem does this PR solve?
Issue Number: N/A
Related PR: the follow-up PR (the second half: BE switches to these packages
and the submodules go away), #66511 (moved datasketches-cpp into the BE CMake
tree), #49644 / #53945 (an earlier OpenBLAS + faiss third-party attempt and its
revert)
Problem Summary:
**Context.** The ANN index is built on faiss, and faiss needs a BLAS/LAPACK
implementation; the `datasketches_hll_union_agg` functions need the
datasketches-cpp headers. Both are git submodules under `contrib/`:
`contrib/openblas` points at the `openblas` branch of apache/doris-thirdparty
and `contrib/datasketches-cpp` at apache/datasketches-cpp. Before every BE
build, `build.sh --be` and `run-be-ut.sh` run `git submodule update` on them,
and `be/src/storage/index/ann/cmake-protect` `add_subdirectory()`s OpenBLAS, so
it is compiled inside every BE build directory (datasketches-cpp is
`add_subdirectory()`ed too, header-only). Every other dependency comes from
`thirdparty/`: `build-thirdparty.sh` builds it once into
`thirdparty/installed`, and the apache/doris-thirdparty automation publishes
that directory as the prebuilt archives that CI and most developers download.
**1. The problem, and what it cost**
- Fetching. On a fresh checkout `git submodule update` clones the whole
apache/doris-thirdparty repository (about 650 MB on GitHub) for
`contrib/openblas`, or downloads a 24.7 MB tarball when the clone fails.
- Compiling. Every cold BE build compiles OpenBLAS: 6,725 build steps on
macOS arm64, where the rest of the BE is 740 unity steps, and 12,577 on Linux
x86_64, where `DYNAMIC_ARCH` builds kernels for every CPU generation. In ASAN
builds its configure step also runs an ASAN-instrumented `getarch` probe, which
is where macOS ASAN builds hung before #68595.
- Doris does not change either library. The doris-thirdparty openblas branch
is the OpenBLAS 0.3.30 release plus two build-only edits (it forces `-O3
-Wno-everything` so that in-tree Debug and ASAN builds compile, and comments
out a CMake message); every other file is identical (`diff -r`).
datasketches-cpp is upstream at a pinned commit.
**2. What this PR does, and why it helps**
This PR is the first half: it adds both libraries to the third-party build
and changes nothing that consumes them. The second half, the follow-up PR,
builds BE against them and removes the submodules. It can only land after the
prebuilt archives from apache/doris-thirdparty contain the two libraries,
because BE stops at link time without them.
- `openblas`: OpenBLAS 0.3.30 from the upstream release, configured the way
cmake-protect configured it: static, LAPACK built from its C translation plus
ReLAPACK, no CBLAS, no Fortran, OpenMP threading, no AVX-512. It installs
`lib64/libopenblas.a` and its headers under `include/openblas`.
- Three settings that the in-tree build took from the machine running it are
pinned, because a prebuilt archive is built on a CI runner and then runs
everywhere:
- `NUM_THREADS=128`. It defaults to the build host's core count and caps
the threads one call can use at run time, so a 4-vCPU runner would cap it at 4.
In BE, BLAS runs multi-threaded only while building an index, and that is
bounded by the index build's OpenMP budget (`ScopedOmpThreadBudget`).
- `DYNAMIC_ARCH` on Linux, now also on aarch64, so the kernels are chosen
on the CPU that runs BE. On aarch64 the in-tree build compiled for the build
host only. The arm64 prebuilt is built on GitHub's Cobalt 100 (Neoverse N2)
runners, and when OpenBLAS detects that core it compiles the whole library with
`-march=armv8.5-a+sve+sve2`, which faults on Graviton2, Ampere Altra or Kunpeng
920.
- `TARGET`, the CPU the code outside the kernels is compiled for, which
also follows the build host unless set (OpenBLAS's README recommends setting it
together with `DYNAMIC_ARCH`): `HASWELL`, `NEHALEM` plus `NO_AVX`/`NO_AVX2`
when `USE_AVX2=0` as cmake-protect did, and `ARMV8` on aarch64, the baselines
BE itself is compiled for.
- The OpenMP runtime needs care in two places:
- macOS: Homebrew's LLVM ships no OpenMP runtime, so the build points
FindOpenMP at Homebrew's libomp, the runtime BE already links on macOS, and
stops with an explicit message when it is not installed. The macOS jobs of
`build-thirdparty.yml` now install it.
- gcc: gcc in the LDB toolchain v0.25, which the Linux job of
`build-thirdparty.yml` uses, is configured for offloading
(`--enable-offload-targets=nvptx-none`) but ships no `crtoffloadbegin.o` /
`crtoffloadend.o`, so every link with `-fopenmp` fails, and FindOpenMP's probe
with it. The build names the flag and libgomp itself; a static library is never
linked. The gcc-built library calls the `GOMP_*` entry points, which LLVM's
libomp provides when a clang-built BE links it.
- `datasketches`: datasketches-cpp at 46025e9, the commit `contrib/` pins,
because no release carries its HLL union fix yet. It is header-only and
installs into `include/DataSketches`, where BE included it from before #66511.
What it buys, once the follow-up PR lands:
- No submodule fetch for these two libraries, and OpenBLAS is compiled once
per third-party build instead of once per BE build directory.
- ASAN builds no longer run OpenBLAS's `getarch` probe.
- The prebuilt OpenBLAS supports the CPUs BE supports and uses as many
threads as BE asks for, whatever runner built it.
**3. How the pieces fit**
```
thirdparty/vars.sh OPENBLAS_* / DATASKETCHES_*: URL, md5
thirdparty/build-thirdparty.sh
|- build_openblas -> installed/lib64/libopenblas.a,
installed/include/openblas/
'- build_datasketches -> installed/include/DataSketches/
|
apache/doris-thirdparty automation ->
doris-thirdparty-prebuilt-<os>-<arch>.tar.xz -> CI, build-env image, developers
|
(the follow-up PR) be/cmake/thirdparty.cmake: add_thirdparty(openblas LIB64
NOTADD) -> imported target `openblas`
contrib/faiss/faiss/CMakeLists.txt: elseif(TARGET openblas) ->
links it (faiss unchanged)
ann_index -> faiss -> libopenblas.a -> doris_be
BE sources: #include <DataSketches/hll.hpp> through
installed/include
```
Not touched here: BE and its build. Nothing uses the new packages until the
follow-up PR.
**Before this merges**: apache/doris-thirdparty's `build-target.yml`
installs its own macOS packages and needs `libomp` added in the same way, or
the macOS legs of the automation fail at this package.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- macOS arm64, clang 20: `build-thirdparty.sh openblas datasketches`
with no libomp flags in the environment, as on CI. With a stubbed `brew` that
has no libomp, the build stops with the explicit message.
- Linux x86_64 in `apache/doris:build-env-ldb-toolchain-latest` (LDB
v0.25): `DORIS_TOOLCHAIN=clang build-thirdparty.sh openblas datasketches`, the
configuration of the doris-thirdparty automation. faiss built against the
result passes the smoke test below; at run time `DYNAMIC_ARCH` picks the
Nehalem kernels on the (Rosetta-emulated) CPU and the library reports
`MAX_THREADS=128`.
- The same image with `DORIS_TOOLCHAIN=gcc`, the configuration of
the Linux job of `build-thirdparty.yml`: without the explicit OpenMP settings
configure fails with `Could NOT find OpenMP_C`; with them both packages build,
and faiss built with clang against the gcc-built library passes the smoke test
with LDB's libomp supplying the `GOMP_*` symbols.
- aarch64 `DYNAMIC_ARCH` with `TARGET=ARMV8` and clang 20, built on
the macOS host because no aarch64 Linux image with the toolchain was at hand:
all 14 cores compile, the SVE and SME ones included, and on a CPU without SVE
the library selects `neoversen1` and passes the smoke test. OpenBLAS 0.3.30's
CMake SME probe always fails (it compiles assembly text as C), so `NO_SME` is
always defined and the ARMV9SME table is never referenced.
- Smoke test: faiss linked against the prebuilt `libopenblas.a` runs
`IndexFlatL2` search (`sgemm`), PCA training (`ssyrk` + `dsyev`; the
eigenvalues add up to the trace), OPQ training (`sgesvd`; the rotation is
orthonormal to 1e-6) and IVF-PQ training and search under OpenMP, all correct.
- `thirdparty/test`: the download fallback, md5, juicefs mirror,
azure retry, adbc JNI configuration and paimon build tests pass.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No. Nothing uses the new packages yet.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
