Re: [PR] [fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef [doris]
morningman merged PR #67451: URL: https://github.com/apache/doris/pull/67451 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef [doris]
hello-stephen commented on PR #67451: URL: https://github.com/apache/doris/pull/67451#issuecomment-5515547203 # BE Regression && UT Coverage Report Increment line coverage `100% (0/0)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67451_cdf92420da1cf2e11a4cff4efd00e7daba03dc66_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67451_cdf92420da1cf2e11a4cff4efd00e7daba03dc66_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 65.05% (29601/45503) | | Line Coverage | 49.95% (319092/638809) | | Region Coverage | 45.52% (260602/572516) | | Branch Coverage | 46.77% (120489/257639) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef [doris]
hello-stephen commented on PR #67451: URL: https://github.com/apache/doris/pull/67451#issuecomment-5511793728 # BE UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/cdf92420da1cf2e11a4cff4efd00e7daba03dc66_cdf92420da1cf2e11a4cff4efd00e7daba03dc66/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/cdf92420da1cf2e11a4cff4efd00e7daba03dc66_cdf92420da1cf2e11a4cff4efd00e7daba03dc66/report/index.html) | Category | Coverage | |---|| | Function Coverage | 62.95% (29573/46980) | | Line Coverage | 47.99% (310073/646180) | | Region Coverage | 43.56% (250351/574707) | | Branch Coverage | 45.09% (116501/258378) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef [doris]
morningman commented on PR #67451: URL: https://github.com/apache/doris/pull/67451#issuecomment-5510093564 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef [doris]
hello-stephen commented on PR #67451: URL: https://github.com/apache/doris/pull/67451#issuecomment-5513004658 ClickBench: Total hot run time: 14.72 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit cdf92420da1cf2e11a4cff4efd00e7daba03dc66, data reload: false query1 0.000.000.00 query2 0.080.040.04 query3 0.260.110.11 query4 1.600.090.10 query5 0.170.170.16 query6 1.230.680.70 query7 0.030.010.00 query8 0.050.040.03 query9 0.300.210.21 query10 0.340.330.34 query11 0.160.110.12 query12 0.150.130.12 query13 0.310.310.31 query14 0.460.450.44 query15 0.360.350.35 query16 0.240.260.22 query17 0.720.700.70 query18 0.190.180.17 query19 1.181.171.15 query20 0.020.000.00 query21 15.45 0.160.11 query22 5.070.040.04 query23 16.20 0.250.11 query24 3.080.350.28 query25 0.120.040.03 query26 0.800.160.12 query27 0.040.030.02 query28 3.580.580.27 query29 12.42 3.232.59 query30 0.260.110.11 query31 2.760.350.17 query32 3.510.320.23 query33 1.381.411.55 query34 15.39 2.231.78 query35 1.761.721.75 query36 0.480.290.28 query37 0.080.040.04 query38 0.040.030.02 query39 0.030.020.03 query40 0.120.080.08 query41 0.090.030.02 query42 0.030.020.02 query43 0.030.030.03 Total cold run time: 90.57 s Total hot run time: 14.72 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef [doris]
hello-stephen commented on PR #67451: URL: https://github.com/apache/doris/pull/67451#issuecomment-5512948586 TPC-DS: Total hot run time: 82497 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit cdf92420da1cf2e11a4cff4efd00e7daba03dc66, data reload: false query5 4252431 348 348 query6 393 145 122 122 query7 4921386 231 231 query8 316 127 115 115 query9 8680287929112879 query10 394 217 181 181 query11 53861044912 912 query12 138 71 78 71 query13 1206445 333 333 query14 5992222820812081 query14_11978197219711971 query15 172 121 112 112 query16 931 366 359 359 query17 849 464 371 371 query18 2345336 239 239 query19 190 144 112 112 query20 79 70 71 70 query21 206 101 88 88 query22 5544543553675367 query23 6717641459695969 query23_16139619660336033 query24 73591123782 782 query24_1782 785 811 785 query25 449 304 256 256 query26 1243237 140 140 query27 2785412 264 264 query28 4693153115361531 query29 903 423 326 326 query30 262 155 131 131 query31 822 394 333 333 query32 159 83 72 72 query33 473 221 165 165 query34 1055830 467 467 query35 401 395 348 348 query36 571 571 533 533 query37 117 81 69 69 query38 1020842 830 830 query39 493 475 466 466 query39_1453 483 454 454 query40 204 92 82 82 query41 53 52 53 52 query42 72 71 73 71 query43 246 243 217 217 query44 1078531 551 531 query45 107 105 98 98 query46 788 822 535 535 query47 774 730 715 715 query48 302 296 232 232 query49 534 236 193 193 query50 721 273 198 198 query51 8142815883138158 query52 69 67 62 62 query53 198 195 162 162 query54 320 213 152 152 query55 82 58 60 58 query56 236 156 167 156 query57 645 627 667 627 query58 207 182 164 164 query59 1220122611181118 query60 242 180 177 177 query61 120 117 121 117 query62 424 195 181 181 query63 170 135 139 135 query64 2647698 685 685 query65 1659159715621562 query66 1756244 193 193 query67 9895975896169616 query68 27491134737 737 query69 348 228 181 181 query70 675 620 610 610 query71 250 180 166 166 query72 2363171615971597 query73 641 572 333 333 query74 1574121611291129 query75 11681101964 964 query76 2296719 551 551 query77 256 257 206 206 query78 3977369432993299 query79 1369812 593 593 query80 1222342 300 300 query81 495 154 143 143 query82 932 131 103 103 query83 338 218 198 198 query84 296 120 97 97 query85 888 427 375 375 query86 510 170 173 170 query87 1011978 903 903 query88 2822208621002086 query89 293 193 173 173 query90 1914132 124 124 query91 131 121 102 102 query92 82 73 73 73 query93 14241133738 738 query94 745 242 233 233 query95 514 251 305 251 query96 847 570 265 265 query97 1060108210301030 query98 142 155 129 129 query99 469 344 307 307 Total cold run time: 177023 ms Total hot run time: 82497 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
Re: [PR] [fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef [doris]
hello-stephen commented on PR #67451: URL: https://github.com/apache/doris/pull/67451#issuecomment-5512863588 TPC-H: Total hot run time: 16922 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit cdf92420da1cf2e11a4cff4efd00e7daba03dc66, data reload: false -- Round 1 -- q1 17565 310130733073 q2 2112251 221 221 q3 10227 868 525 525 q4 4680243 203 203 q5 7668555 382 382 q6 138 115 92 92 q7 525 502 387 387 q8 9251912 946 912 q9 3438240123842384 q10 6519876 726 726 q11 396 202 187 187 q12 614 260 197 197 q13 18132 154011481148 q14 161 153 134 134 q15 q16 433 400 362 362 q17 1405901 838 838 q18 3111230222552255 q19 1272877 785 785 q20 372 286 204 204 q21 5597167218031672 q22 353 259 235 235 Total cold run time: 93969 ms Total hot run time: 16922 ms - Round 2, with runtime_filter_mode=off - q1 3463341133753375 q2 507 392 371 371 q3 2187229621482148 q4 11941170900 900 q5 2196214421682144 q6 168 121 86 86 q7 1026939 890 890 q8 1620143114211421 q9 3173314631163116 q10 1861177216691669 q11 363 269 254 254 q12 457 430 349 349 q13 1483152711671167 q14 176 182 155 155 q15 q16 389 395 371 371 q17 3675325231893189 q18 4845445347674453 q19 951 880 863 863 q20 1009962 810 810 q21 3856326032513251 q22 405 353 320 320 Total cold run time: 35004 ms Total hot run time: 31302 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef [doris]
hello-stephen commented on PR #67451: URL: https://github.com/apache/doris/pull/67451#issuecomment-5510062648 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[PR] [fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef [doris]
morningman opened a new pull request, #67451:
URL: https://github.com/apache/doris/pull/67451
### What problem does this PR solve?
Issue Number: close #67448, close #67445
Related PR: #66615
Problem Summary:
Two BE build failures on arm64, both found while building on Apple Silicon.
**1. `kuromoji_build_dict` fails to link on macOS arm64 (#67448)**
`sh build.sh --be` fails while linking `bin/kuromoji_build_dict`:
```
ld: fixup error (kind=arm64_b26) at
'__ZN8tcmallocL14memalign_pagesEmmbb'+0x1DC from
libtcmalloc.a[2](libtcmalloc_la-tcmalloc.o),
B/BL out of range (displacement=-135695688, max is +/-128MB), from
0x10816FF98 to 0x17250 ('___clang_call_terminate')
__TEXT addr=0x1, size=0x00831
google_mallocaddr=0x10816eac0, size=0x014fc
malloc_hook addr=0x10816ffbc, size=0x001dc
```
The offline dictionary converter only calls the kuromoji builder/parser, but
those return `Status`, and `Status` reaches `config.cpp` (-> `ExecEnv`),
`status.cpp` (-> thrift/protobuf/`BackendOptions`) and `stack_util.cpp`, so the
tool's link closure is effectively the whole BE and its `__TEXT` exceeds
arm64's +/-128 MB direct-branch reach. Apple's linker lays tcmalloc's custom
`google_malloc` / `malloc_hook` sections out after `__text` and cannot insert
branch islands there, so the branch from tcmalloc back to
`___clang_call_terminate` at the start of `__TEXT` cannot be relaxed. The
failed link leaves the four dictionary files ungenerated, the install-time
guard fires, and no `output/` is produced. The triage comment on #67448 reports
the same failure on the `macos-15` runner of the `BE UT (macOS)` workflow.
Trimming the tool's link line (the issue's first suggestion) is not possible
without refactoring `Status`, so this applies the #66615 treatment per target:
on macOS arm64 the tool links against the system allocator, and
`gperftools_stubs.cpp` supplies the few gperftools symbols that are still
referenced unconditionally. `doris_be` keeps tcmalloc, Linux link lines are
unchanged, `kuromoji_dict` stays in `ALL` and the install-time file check stays.
**2. `SNII_CRC32C_X86` is undefined on non-x86 targets (#67445)**
`be/src/storage/index/snii/encoding/crc32c.cpp` tests `SNII_CRC32C_X86` with
`#if` but only defines it on x86_64. The BE compiles with `-Wundef -Werror`, so
on aarch64 the four `#if` sites fail:
```
crc32c.cpp:105:5: error: 'SNII_CRC32C_X86' is not defined, evaluates to 0
[-Werror,-Wundef]
```
The file is `BE_TEST`-only, so `run-be-ut.sh` cannot build on Apple Silicon
or Linux aarch64, while CI never sees it (the only aarch64 workflow builds with
`MAKE_TEST=OFF`). Define the flag as `0` on the other branch: the x86 branch is
unchanged, and the non-x86 build keeps only the portable slice-by-8 reference
path, which the `hw_*` seams already fall back to.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
macOS 26.5 arm64, Homebrew clang 20.1.8, on master `72071af801d`:
- Relinking `kuromoji_build_dict` in the same build directory with the
pristine `be/CMakeLists.txt` reproduces the `arm64_b26` fixup error above
(`__TEXT` size `0x0831`, tcmalloc five times on the link line). With this
change `ninja -v kuromoji_build_dict` links: no `tcmalloc` on the link line,
`gperftools_stubs.cpp.o` present, `__TEXT` `0x082fc000` with no `google_malloc`
/ `malloc_hook` sections.
- `ninja kuromoji_dict` regenerates `system.bin` / `matrix.bin` /
`chardef.bin` / `unkdict.bin` (325871 surfaces, 392126 lexicon rows, matrix
1316x1316), identical in size to the previous output.
- A full `DISABLE_BUILD_UI=ON bash build.sh --be` ends with
`Successfully build Doris` and `output/be/dict/kuromoji/` contains all four
non-empty files.
- `crc32c.cpp`: `clang++ -std=c++20 -DBE_TEST -Wundef -Werror
-fsyntax-only` on arm64 reproduces the four errors before this change and
passes after it; `clang-format` is clean. A standalone arm64 program compiled
with `-DBE_TEST` compares `crc32c_slice8_extend` / `crc32c_hw_serial_extend` /
`crc32c_hw3_extend` against the bundled Google crc32c over 15 sizes x 6
alignments plus split-extend cases: 168 comparisons identical,
`crc32c_has_hw()` reports false, `"123456789"` -> `0xE3069283`.
- Linux is covered by the regular CI builds; the CMake change is guarded
by `OS_MACOSX AND ARCH_ARM`, so Linux link lines are byte-identical.
