This is an automated email from the ASF dual-hosted git repository.
jacktengg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c309a6aeb67 [refactor](be) Replace handwritten UTF-8 validators with
simdutf (#68406)
c309a6aeb67 is described below
commit c309a6aeb678e72f79e8f6d2a754fc8301ecc6e0
Author: HappenLee <[email protected]>
AuthorDate: Wed Sep 23 21:52:12 2026 +0800
[refactor](be) Replace handwritten UTF-8 validators with simdutf (#68406)
### What problem does this PR solve?
Issue Number: N/A
Related PR: #68369
Problem Summary:
Doris maintains separate handwritten SSE, NEON and scalar UTF-8
validators. Replace them with the simdutf dependency added in #68369,
keeping the existing boolean validation entry point and file-scan
validation setting. SIMD selection and short-input handling use
simdutf's runtime dispatch; no CPU-specific or length-specific dispatch
is added in Doris.
Remove the obsolete implementation and scalar-only API, update its
remaining test caller, and drop the old source/distribution license and
tooling exclusions. Bundle simdutf 9.2.0's complete BSD-3-Clause
ISA-detection notice in the binary distribution and reference it from
the license manifest. Fix the existing UTF-8 tests, whose
`sizeof(vector)` loops only exercised the first test case, and extend
coverage for Unicode boundaries, malformed and truncated sequences,
embedded NUL, empty/null buffers, SIMD boundaries, unaligned exact-sized
buffers, long inputs, independent row validation and the file-scan
bypass.
### Release note
None
### Check List (For Author)
- Test: Unit tests and static checks
- License follow-up: verified the bundled BSD notice byte-for-byte
against simdutf v9.2.0, checked both license-tool exclusion removals,
and executed the existing `build.sh` packaging function to confirm the
manifest and complete notice are included intact.
- License-Eye v0.8.0 passed with the repository configuration on the
four follow-up files and the newly unexcluded `utf8_check.cpp` (1 valid
source, 0 invalid, 4 ignored license/configuration files). The full
local worktree scan stopped on the existing `doris-skills` submodule
directory (`read doris-skills: is a directory`); the full repository
license check is left to CI.
- Passed all 9 UTF-8 utility tests in a standalone harness compiling the
actual modified implementation, test source and generated Thrift types.
Both the implementation and simdutf 9.2.0 were built with ASAN/UBSAN.
Passed with automatic dispatch and the supported icelake, haswell,
westmere and fallback implementations.
- Passed repository formatting and build-hygiene checks; clang-tidy
passed for the UTF-8 implementation and tests.
- Attempted the standard BE ASAN test runner with UTF-8, ColumnString,
SQL-function and Variant filters. The build stopped in unchanged
`common/cpp/aws_common.cpp` because the local AWS dependency lacks
`aws/core/auth/GeneralHTTPCredentialsProvider.h`; these integrated tests
did not run.
- clang-tidy for the one-line Variant test-helper update is blocked by
an existing unmatched `NOLINTEND` in unchanged
`be/src/core/types.h:576`.
- Behavior changed: No; validation semantics and the file-scan bypass
are preserved.
- Does this need documentation: No
---
.clang-format-ignore | 1 -
.licenserc.yaml | 1 -
.rat-excludes | 1 -
LICENSE.txt | 26 --
be/src/util/simdutf8check.h | 313 -----------------------
be/src/util/utf8_check.cpp | 340 ++-----------------------
be/src/util/utf8_check.h | 5 +-
be/test/util/utf8_check_test.cpp | 170 ++++++++++---
be/test/util/variant/variant_test_utils.cpp | 2 +-
build-support/run-clang-tidy.sh | 1 -
dist/LICENSE-dist.txt | 28 +-
dist/licenses/LICENSE-simdutf-isadetection.txt | 46 ++++
12 files changed, 200 insertions(+), 734 deletions(-)
diff --git a/.clang-format-ignore b/.clang-format-ignore
index 675b0803161..bdecd10100c 100644
--- a/.clang-format-ignore
+++ b/.clang-format-ignore
@@ -8,6 +8,5 @@ be/src/util/sse2neo.h
be/src/util/sse2neon.h
be/src/util/mustache/mustache.h
be/src/util/mustache/mustache.cc
-be/src/util/utf8_check.cpp
be/src/storage/index/inverted/analyzer/kuromoji/dict/darts.h
cloud/src/common/defer.h
diff --git a/.licenserc.yaml b/.licenserc.yaml
index 92608eadef1..67cc48afe08 100644
--- a/.licenserc.yaml
+++ b/.licenserc.yaml
@@ -126,7 +126,6 @@ header:
- "be/src/util/mustache/mustache.h"
- "be/src/util/sse2neo.h"
- "be/src/util/sse2neon.h"
- - "be/src/util/utf8_check.cpp"
- "be/src/storage/index/inverted/analyzer/kuromoji/dict/darts.h"
- "be/src/pch/*"
- "be/test/data"
diff --git a/.rat-excludes b/.rat-excludes
index 451e8c52963..d9987c215d1 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -35,6 +35,5 @@ string_search.hpp
coding.*
condition_variable.*
murmur_hash3.*
-utf8_check.cpp
.markdownlintignore
sse2neon.h
diff --git a/LICENSE.txt b/LICENSE.txt
index eeb1709f444..4210475bbe4 100644
--- a/LICENSE.txt
+++ b/LICENSE.txt
@@ -412,32 +412,6 @@ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-------------------------------------------------------------------------------
-be/src/util/utf8_check.cpp: MIT license
-
- MIT License
-
- Copyright (c) 2019 Yibo Cai
-
- Permission is hereby granted, free of charge, to any person obtaining a copy
- of this software and associated documentation files (the "Software"), to deal
- in the Software without restriction, including without limitation the rights
- to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- copies of the Software, and to permit persons to whom the Software is
- furnished to do so, subject to the following conditions:
-
- The above copyright notice and this permission notice shall be included in
all
- copies or substantial portions of the Software.
-
- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- SOFTWARE.
-
---------------------------------------------------------------------------------
-
be/src/util/condition_variable* : BSD-style license
Copyright (c) 2011 The Chromium Authors. All rights reserved.
diff --git a/be/src/util/simdutf8check.h b/be/src/util/simdutf8check.h
deleted file mode 100644
index 87e5b1aaa46..00000000000
--- a/be/src/util/simdutf8check.h
+++ /dev/null
@@ -1,313 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#pragma once
-
-#include <stdbool.h>
-#include <stddef.h>
-#include <stdint.h>
-#include <string.h>
-#include <x86intrin.h>
-
-/*
- * These functions are used for validating utf8 string.
- * Details can be seen here: https://github.com/lemire/fastvalidate-utf-8
- */
-
-/*
- * legal utf-8 byte sequence
- * http://www.unicode.org/versions/Unicode6.0.0/ch03.pdf - page 94
- *
- * Code Points 1st 2s 3s 4s
- * U+0000..U+007F 00..7F
- * U+0080..U+07FF C2..DF 80..BF
- * U+0800..U+0FFF E0 A0..BF 80..BF
- * U+1000..U+CFFF E1..EC 80..BF 80..BF
- * U+D000..U+D7FF ED 80..9F 80..BF
- * U+E000..U+FFFF EE..EF 80..BF 80..BF
- * U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
- * U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
- * U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
- *
- */
-
-// all byte values must be no larger than 0xF4
-static inline void checkSmallerThan0xF4(__m128i current_bytes, __m128i*
has_error) {
- // unsigned, saturates to 0 below max
- *has_error = _mm_or_si128(*has_error, _mm_subs_epu8(current_bytes,
_mm_set1_epi8(0xF4)));
-}
-
-static inline __m128i continuationLengths(__m128i high_nibbles) {
- return _mm_shuffle_epi8(_mm_setr_epi8(1, 1, 1, 1, 1, 1, 1, 1, // 0xxx
(ASCII)
- 0, 0, 0, 0, // 10xx
(continuation)
- 2, 2, // 110x
- 3, // 1110
- 4), // 1111, next should be 0 (not
checked here)
- high_nibbles);
-}
-
-static inline __m128i carryContinuations(__m128i initial_lengths, __m128i
previous_carries) {
- __m128i right1 = _mm_subs_epu8(_mm_alignr_epi8(initial_lengths,
previous_carries, 16 - 1),
- _mm_set1_epi8(1));
- __m128i sum = _mm_add_epi8(initial_lengths, right1);
-
- __m128i right2 =
- _mm_subs_epu8(_mm_alignr_epi8(sum, previous_carries, 16 - 2),
_mm_set1_epi8(2));
- return _mm_add_epi8(sum, right2);
-}
-
-static inline void checkContinuations(__m128i initial_lengths, __m128i carries,
- __m128i* has_error) {
- // overlap || underlap
- // carry > length && length > 0 || !(carry > length) && !(length > 0)
- // (carries > length) == (lengths > 0)
- __m128i overunder = _mm_cmpeq_epi8(_mm_cmpgt_epi8(carries,
initial_lengths),
- _mm_cmpgt_epi8(initial_lengths,
_mm_setzero_si128()));
-
- *has_error = _mm_or_si128(*has_error, overunder);
-}
-
-// when 0xED is found, next byte must be no larger than 0x9F
-// when 0xF4 is found, next byte must be no larger than 0x8F
-// next byte must be continuation, ie sign bit is set, so signed < is ok
-static inline void checkFirstContinuationMax(__m128i current_bytes, __m128i
off1_current_bytes,
- __m128i* has_error) {
- __m128i maskED = _mm_cmpeq_epi8(off1_current_bytes, _mm_set1_epi8(0xED));
- __m128i maskF4 = _mm_cmpeq_epi8(off1_current_bytes, _mm_set1_epi8(0xF4));
-
- __m128i badfollowED = _mm_and_si128(_mm_cmpgt_epi8(current_bytes,
_mm_set1_epi8(0x9F)), maskED);
- __m128i badfollowF4 = _mm_and_si128(_mm_cmpgt_epi8(current_bytes,
_mm_set1_epi8(0x8F)), maskF4);
-
- *has_error = _mm_or_si128(*has_error, _mm_or_si128(badfollowED,
badfollowF4));
-}
-
-// map off1_hibits => error condition
-// hibits off1 cur
-// C => < C2 && true
-// E => < E1 && < A0
-// F => < F1 && < 90
-// else false && false
-static inline void checkOverlong(__m128i current_bytes, __m128i
off1_current_bytes, __m128i hibits,
- __m128i previous_hibits, __m128i* has_error) {
- __m128i off1_hibits = _mm_alignr_epi8(hibits, previous_hibits, 16 - 1);
- __m128i initial_mins =
- _mm_shuffle_epi8(_mm_setr_epi8(-128, -128, -128, -128, -128, -128,
-128, -128, -128,
- -128, -128, -128, // 10xx => false
- 0xC2, -128, // 110x
- 0xE1, // 1110
- 0xF1),
- off1_hibits);
-
- __m128i initial_under = _mm_cmpgt_epi8(initial_mins, off1_current_bytes);
-
- __m128i second_mins =
- _mm_shuffle_epi8(_mm_setr_epi8(-128, -128, -128, -128, -128, -128,
-128, -128, -128,
- -128, -128, -128, // 10xx => false
- 127, 127, // 110x => true
- 0xA0, // 1110
- 0x90),
- off1_hibits);
- __m128i second_under = _mm_cmpgt_epi8(second_mins, current_bytes);
- *has_error = _mm_or_si128(*has_error, _mm_and_si128(initial_under,
second_under));
-}
-
-struct processed_utf_bytes {
- __m128i rawbytes;
- __m128i high_nibbles;
- __m128i carried_continuations;
-};
-
-static inline void count_nibbles(__m128i bytes, struct processed_utf_bytes*
answer) {
- answer->rawbytes = bytes;
- answer->high_nibbles = _mm_and_si128(_mm_srli_epi16(bytes, 4),
_mm_set1_epi8(0x0F));
-}
-
-// check whether the current bytes are valid UTF-8
-// at the end of the function, previous gets updated
-static struct processed_utf_bytes checkUTF8Bytes(__m128i current_bytes,
- struct processed_utf_bytes*
previous,
- __m128i* has_error) {
- struct processed_utf_bytes pb;
- count_nibbles(current_bytes, &pb);
-
- checkSmallerThan0xF4(current_bytes, has_error);
-
- __m128i initial_lengths = continuationLengths(pb.high_nibbles);
-
- pb.carried_continuations = carryContinuations(initial_lengths,
previous->carried_continuations);
-
- checkContinuations(initial_lengths, pb.carried_continuations, has_error);
-
- __m128i off1_current_bytes = _mm_alignr_epi8(pb.rawbytes,
previous->rawbytes, 16 - 1);
- checkFirstContinuationMax(current_bytes, off1_current_bytes, has_error);
-
- checkOverlong(current_bytes, off1_current_bytes, pb.high_nibbles,
previous->high_nibbles,
- has_error);
- return pb;
-}
-
-static bool validate_utf8_fast(const char* src, size_t len) {
- size_t i = 0;
- __m128i has_error = _mm_setzero_si128();
- struct processed_utf_bytes previous = {.rawbytes = _mm_setzero_si128(),
- .high_nibbles = _mm_setzero_si128(),
- .carried_continuations =
_mm_setzero_si128()};
- if (len >= 16) {
- for (; i <= len - 16; i += 16) {
- __m128i current_bytes = _mm_loadu_si128((const __m128i*)(src + i));
- previous = checkUTF8Bytes(current_bytes, &previous, &has_error);
- }
- }
-
- // last part
- if (i < len) {
- char buffer[16];
- memset(buffer, 0, 16);
- memcpy(buffer, src + i, len - i);
- __m128i current_bytes = _mm_loadu_si128((const __m128i*)(buffer));
- previous = checkUTF8Bytes(current_bytes, &previous, &has_error);
- } else {
- has_error = _mm_or_si128(
- _mm_cmpgt_epi8(previous.carried_continuations,
- _mm_setr_epi8(9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 1)),
- has_error);
- }
-
- return _mm_testz_si128(has_error, has_error);
-}
-
-#ifdef __AVX2__
-
-/*****************************/
-static inline __m256i push_last_byte_of_a_to_b(__m256i a, __m256i b) {
- return _mm256_alignr_epi8(b, _mm256_permute2x128_si256(a, b, 0x21), 15);
-}
-
-static inline __m256i push_last_2bytes_of_a_to_b(__m256i a, __m256i b) {
- return _mm256_alignr_epi8(b, _mm256_permute2x128_si256(a, b, 0x21), 14);
-}
-
-// all byte values must be no larger than 0xF4
-static inline void avxcheckSmallerThan0xF4(__m256i current_bytes, __m256i*
has_error) {
- // unsigned, saturates to 0 below max
- *has_error =
- _mm256_or_si256(*has_error, _mm256_subs_epu8(current_bytes,
_mm256_set1_epi8(0xF4)));
-}
-
-static inline __m256i avxcontinuationLengths(__m256i high_nibbles) {
- return _mm256_shuffle_epi8(_mm256_setr_epi8(1, 1, 1, 1, 1, 1, 1, 1, //
0xxx (ASCII)
- 0, 0, 0, 0, //
10xx (continuation)
- 2, 2, // 110x
- 3, // 1110
- 4, // 1111, next should be 0
(not checked here)
- 1, 1, 1, 1, 1, 1, 1, 1, //
0xxx (ASCII)
- 0, 0, 0, 0, //
10xx (continuation)
- 2, 2, // 110x
- 3, // 1110
- 4 // 1111, next should be 0
(not checked here)
- ),
- high_nibbles);
-}
-
-static inline __m256i avxcarryContinuations(__m256i initial_lengths, __m256i
previous_carries) {
- __m256i right1 =
_mm256_subs_epu8(push_last_byte_of_a_to_b(previous_carries, initial_lengths),
- _mm256_set1_epi8(1));
- __m256i sum = _mm256_add_epi8(initial_lengths, right1);
-
- __m256i right2 =
_mm256_subs_epu8(push_last_2bytes_of_a_to_b(previous_carries, sum),
- _mm256_set1_epi8(2));
- return _mm256_add_epi8(sum, right2);
-}
-
-static inline void avxcheckContinuations(__m256i initial_lengths, __m256i
carries,
- __m256i* has_error) {
- // overlap || underlap
- // carry > length && length > 0 || !(carry > length) && !(length > 0)
- // (carries > length) == (lengths > 0)
- __m256i overunder =
- _mm256_cmpeq_epi8(_mm256_cmpgt_epi8(carries, initial_lengths),
- _mm256_cmpgt_epi8(initial_lengths,
_mm256_setzero_si256()));
-
- *has_error = _mm256_or_si256(*has_error, overunder);
-}
-
-// when 0xED is found, next byte must be no larger than 0x9F
-// when 0xF4 is found, next byte must be no larger than 0x8F
-// next byte must be continuation, ie sign bit is set, so signed < is ok
-static inline void avxcheckFirstContinuationMax(__m256i current_bytes, __m256i
off1_current_bytes,
- __m256i* has_error) {
- __m256i maskED = _mm256_cmpeq_epi8(off1_current_bytes,
_mm256_set1_epi8(0xED));
- __m256i maskF4 = _mm256_cmpeq_epi8(off1_current_bytes,
_mm256_set1_epi8(0xF4));
-
- __m256i badfollowED =
- _mm256_and_si256(_mm256_cmpgt_epi8(current_bytes,
_mm256_set1_epi8(0x9F)), maskED);
- __m256i badfollowF4 =
- _mm256_and_si256(_mm256_cmpgt_epi8(current_bytes,
_mm256_set1_epi8(0x8F)), maskF4);
-
- *has_error = _mm256_or_si256(*has_error, _mm256_or_si256(badfollowED,
badfollowF4));
-}
-
-// map off1_hibits => error condition
-// hibits off1 cur
-// C => < C2 && true
-// E => < E1 && < A0
-// F => < F1 && < 90
-// else false && false
-static inline void avxcheckOverlong(__m256i current_bytes, __m256i
off1_current_bytes,
- __m256i hibits, __m256i previous_hibits,
__m256i* has_error) {
- __m256i off1_hibits = push_last_byte_of_a_to_b(previous_hibits, hibits);
- __m256i initial_mins =
- _mm256_shuffle_epi8(_mm256_setr_epi8(-128, -128, -128, -128, -128,
-128, -128, -128,
- -128, -128, -128, -128, //
10xx => false
- 0xC2, -128, //
110x
- 0xE1, //
1110
- 0xF1, -128, -128, -128, -128,
-128, -128, -128,
- -128, -128, -128, -128, -128,
// 10xx => false
- 0xC2, -128,
// 110x
- 0xE1,
// 1110
- 0xF1),
- off1_hibits);
-
- __m256i initial_under = _mm256_cmpgt_epi8(initial_mins,
off1_current_bytes);
-
- __m256i second_mins =
- _mm256_shuffle_epi8(_mm256_setr_epi8(-128, -128, -128, -128, -128,
-128, -128, -128,
- -128, -128, -128, -128, //
10xx => false
- 127, 127, //
110x => true
- 0xA0, //
1110
- 0x90, -128, -128, -128, -128,
-128, -128, -128,
- -128, -128, -128, -128, -128,
// 10xx => false
- 127, 127,
// 110x => true
- 0xA0,
// 1110
- 0x90),
- off1_hibits);
- __m256i second_under = _mm256_cmpgt_epi8(second_mins, current_bytes);
- *has_error = _mm256_or_si256(*has_error, _mm256_and_si256(initial_under,
second_under));
-}
-
-struct avx_processed_utf_bytes {
- __m256i rawbytes;
- __m256i high_nibbles;
- __m256i carried_continuations;
-};
-
-static inline void avx_count_nibbles(__m256i bytes, struct
avx_processed_utf_bytes* answer) {
- answer->rawbytes = bytes;
- answer->high_nibbles = _mm256_and_si256(_mm256_srli_epi16(bytes, 4),
_mm256_set1_epi8(0x0F));
-}
-
-#endif // __AVX2__
diff --git a/be/src/util/utf8_check.cpp b/be/src/util/utf8_check.cpp
index f90c27e5e91..1d507930747 100644
--- a/be/src/util/utf8_check.cpp
+++ b/be/src/util/utf8_check.cpp
@@ -1,332 +1,29 @@
-// Copyright (c) cyb70289(https://github.com/cyb70289). All rights reserved.
-// Use of this source code is governed by a MIT license that can be
-// found in the LICENSE file.
-
-/*
- * These functions are used for validating utf8 string.
- * Details can be seen here: https://github.com/cyb70289/utf8/
- */
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
#include "util/utf8_check.h"
-#if defined(__i386) || defined(__x86_64__)
-#include "util/simdutf8check.h"
-#elif defined(__aarch64__)
-#include <arm_neon.h>
-#endif
+#include <simdutf.h>
-/*
- * http://www.unicode.org/versions/Unicode6.0.0/ch03.pdf - page 94
- *
- * Table 3-7. Well-Formed UTF-8 Byte Sequences
- *
- * +--------------------+------------+-------------+------------+-------------+
- * | Code Points | First Byte | Second Byte | Third Byte | Fourth Byte |
- * +--------------------+------------+-------------+------------+-------------+
- * | U+0000..U+007F | 00..7F | | | |
- * +--------------------+------------+-------------+------------+-------------+
- * | U+0080..U+07FF | C2..DF | 80..BF | | |
- * +--------------------+------------+-------------+------------+-------------+
- * | U+0800..U+0FFF | E0 | A0..BF | 80..BF | |
- * +--------------------+------------+-------------+------------+-------------+
- * | U+1000..U+CFFF | E1..EC | 80..BF | 80..BF | |
- * +--------------------+------------+-------------+------------+-------------+
- * | U+D000..U+D7FF | ED | 80..9F | 80..BF | |
- * +--------------------+------------+-------------+------------+-------------+
- * | U+E000..U+FFFF | EE..EF | 80..BF | 80..BF | |
- * +--------------------+------------+-------------+------------+-------------+
- * | U+10000..U+3FFFF | F0 | 90..BF | 80..BF | 80..BF |
- * +--------------------+------------+-------------+------------+-------------+
- * | U+40000..U+FFFFF | F1..F3 | 80..BF | 80..BF | 80..BF |
- * +--------------------+------------+-------------+------------+-------------+
- * | U+100000..U+10FFFF | F4 | 80..8F | 80..BF | 80..BF |
- * +--------------------+------------+-------------+------------+-------------+
- */
namespace doris {
-bool validate_utf8_naive(const char* data, size_t len) {
- while (len) {
- int bytes;
- const unsigned char byte1 = data[0];
-
- /* 00..7F */
- if (byte1 <= 0x7F) {
- bytes = 1;
- /* C2..DF, 80..BF */
- } else if (len >= 2 && byte1 >= 0xC2 && byte1 <= 0xDF &&
- (signed char)data[1] <= (signed char)0xBF) {
- bytes = 2;
- } else if (len >= 3) {
- const unsigned char byte2 = data[1];
-
- /* Is byte2, byte3 between 0x80 ~ 0xBF */
- const int byte2_ok = (signed char)byte2 <= (signed char)0xBF;
- const int byte3_ok = (signed char)data[2] <= (signed char)0xBF;
-
- if (byte2_ok && byte3_ok &&
- /* E0, A0..BF, 80..BF */
- ((byte1 == 0xE0 && byte2 >= 0xA0) ||
- /* E1..EC, 80..BF, 80..BF */
- (byte1 >= 0xE1 && byte1 <= 0xEC) ||
- /* ED, 80..9F, 80..BF */
- (byte1 == 0xED && byte2 <= 0x9F) ||
- /* EE..EF, 80..BF, 80..BF */
- (byte1 >= 0xEE && byte1 <= 0xEF))) {
- bytes = 3;
- } else if (len >= 4) {
- /* Is byte4 between 0x80 ~ 0xBF */
- const int byte4_ok = (signed char)data[3] <= (signed char)0xBF;
-
- if (byte2_ok && byte3_ok && byte4_ok &&
- /* F0, 90..BF, 80..BF, 80..BF */
- ((byte1 == 0xF0 && byte2 >= 0x90) ||
- /* F1..F3, 80..BF, 80..BF, 80..BF */
- (byte1 >= 0xF1 && byte1 <= 0xF3) ||
- /* F4, 80..8F, 80..BF, 80..BF */
- (byte1 == 0xF4 && byte2 <= 0x8F))) {
- bytes = 4;
- } else {
- return false;
- }
- } else {
- return false;
- }
- } else {
- return false;
- }
-
- len -= bytes;
- data += bytes;
- }
-
- return true;
-}
-
-#if defined(__i386) || defined(__x86_64__)
-bool validate_utf8(const char* src, size_t len) {
- return validate_utf8_fast(src, len);
-}
-#elif defined(__aarch64__)
-/*
- * Map high nibble of "First Byte" to legal character length minus 1
- * 0x00 ~ 0xBF --> 0
- * 0xC0 ~ 0xDF --> 1
- * 0xE0 ~ 0xEF --> 2
- * 0xF0 ~ 0xFF --> 3
- */
-const uint8_t _first_len_tbl[] = {
- 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 2, 3,
-};
-
-/* Map "First Byte" to 8-th item of range table (0xC2 ~ 0xF4) */
-static const uint8_t _first_range_tbl[] = {
- 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 8,
-};
-
-/*
- * Range table, map range index to min and max values
- * Index 0 : 00 ~ 7F (First Byte, ascii)
- * Index 1,2,3: 80 ~ BF (Second, Third, Fourth Byte)
- * Index 4 : A0 ~ BF (Second Byte after E0)
- * Index 5 : 80 ~ 9F (Second Byte after ED)
- * Index 6 : 90 ~ BF (Second Byte after F0)
- * Index 7 : 80 ~ 8F (Second Byte after F4)
- * Index 8 : C2 ~ F4 (First Byte, non ascii)
- * Index 9~15 : illegal: u >= 255 && u <= 0
- */
-static const uint8_t _range_min_tbl[] = {
- 0x00, 0x80, 0x80, 0x80, 0xA0, 0x80, 0x90, 0x80,
- 0xC2, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
-};
-static const uint8_t _range_max_tbl[] = {
- 0x7F, 0xBF, 0xBF, 0xBF, 0xBF, 0x9F, 0xBF, 0x8F,
- 0xF4, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-};
-
-/*
- * This table is for fast handling four special First Bytes(E0,ED,F0,F4), after
- * which the Second Byte are not 80~BF. It contains "range index adjustment".
- * - The idea is to minus byte with E0, use the result(0~31) as the index to
- * lookup the "range index adjustment". Then add the adjustment to original
- * range index to get the correct range.
- * - Range index adjustment
- * +------------+---------------+------------------+----------------+
- * | First Byte | original range| range adjustment | adjusted range |
- * +------------+---------------+------------------+----------------+
- * | E0 | 2 | 2 | 4 |
- * +------------+---------------+------------------+----------------+
- * | ED | 2 | 3 | 5 |
- * +------------+---------------+------------------+----------------+
- * | F0 | 3 | 3 | 6 |
- * +------------+---------------+------------------+----------------+
- * | F4 | 4 | 4 | 8 |
- * +------------+---------------+------------------+----------------+
- * - Below is a uint8x16x2 table, data is interleaved in NEON register. So I'm
- * putting it vertically. 1st column is for E0~EF, 2nd column for F0~FF.
- */
-static const uint8_t _range_adjust_tbl[] = {
- /* index -> 0~15 16~31 <- index */
- /* E0 -> */ 2,
- 3, /* <- F0 */
- 0,
- 0,
- 0,
- 0,
- 0,
- 0,
- 0,
- 4, /* <- F4 */
- 0,
- 0,
- 0,
- 0,
- 0,
- 0,
- 0,
- 0,
- 0,
- 0,
- 0,
- 0,
- 0,
- 0,
- 0,
- 0,
- /* ED -> */ 3,
- 0,
- 0,
- 0,
- 0,
- 0,
-};
-
-/* 2x ~ 4x faster than naive method */
-/* Return true on success, false on error */
-bool utf8_range(const char* data, size_t len) {
- if (len >= 16) {
- uint8x16_t prev_input = vdupq_n_u8(0);
- uint8x16_t prev_first_len = vdupq_n_u8(0);
-
- /* Cached tables */
- const uint8x16_t first_len_tbl = vld1q_u8(_first_len_tbl);
- const uint8x16_t first_range_tbl = vld1q_u8(_first_range_tbl);
- const uint8x16_t range_min_tbl = vld1q_u8(_range_min_tbl);
- const uint8x16_t range_max_tbl = vld1q_u8(_range_max_tbl);
- const uint8x16x2_t range_adjust_tbl = vld2q_u8(_range_adjust_tbl);
-
- /* Cached values */
- const uint8x16_t const_1 = vdupq_n_u8(1);
- const uint8x16_t const_2 = vdupq_n_u8(2);
- const uint8x16_t const_e0 = vdupq_n_u8(0xE0);
-
- uint8x16_t error = vdupq_n_u8(0);
-
- while (len >= 16) {
- const uint8x16_t input = vld1q_u8((const uint8_t*)data);
-
- /* high_nibbles = input >> 4 */
- const uint8x16_t high_nibbles = vshrq_n_u8(input, 4);
-
- /* first_len = legal character length minus 1 */
- /* 0 for 00~7F, 1 for C0~DF, 2 for E0~EF, 3 for F0~FF */
- /* first_len = first_len_tbl[high_nibbles] */
- const uint8x16_t first_len = vqtbl1q_u8(first_len_tbl,
high_nibbles);
-
- /* First Byte: set range index to 8 for bytes within 0xC0 ~ 0xFF */
- /* range = first_range_tbl[high_nibbles] */
- uint8x16_t range = vqtbl1q_u8(first_range_tbl, high_nibbles);
-
- /* Second Byte: set range index to first_len */
- /* 0 for 00~7F, 1 for C0~DF, 2 for E0~EF, 3 for F0~FF */
- /* range |= (first_len, prev_first_len) << 1 byte */
- range = vorrq_u8(range, vextq_u8(prev_first_len, first_len, 15));
-
- /* Third Byte: set range index to saturate_sub(first_len, 1) */
- /* 0 for 00~7F, 0 for C0~DF, 1 for E0~EF, 2 for F0~FF */
- uint8x16_t tmp1, tmp2;
- /* tmp1 = saturate_sub(first_len, 1) */
- tmp1 = vqsubq_u8(first_len, const_1);
- /* tmp2 = saturate_sub(prev_first_len, 1) */
- tmp2 = vqsubq_u8(prev_first_len, const_1);
- /* range |= (tmp1, tmp2) << 2 bytes */
- range = vorrq_u8(range, vextq_u8(tmp2, tmp1, 14));
-
- /* Fourth Byte: set range index to saturate_sub(first_len, 2) */
- /* 0 for 00~7F, 0 for C0~DF, 0 for E0~EF, 1 for F0~FF */
- /* tmp1 = saturate_sub(first_len, 2) */
- tmp1 = vqsubq_u8(first_len, const_2);
- /* tmp2 = saturate_sub(prev_first_len, 2) */
- tmp2 = vqsubq_u8(prev_first_len, const_2);
- /* range |= (tmp1, tmp2) << 3 bytes */
- range = vorrq_u8(range, vextq_u8(tmp2, tmp1, 13));
-
- /*
- * Now we have below range indices caluclated
- * Correct cases:
- * - 8 for C0~FF
- * - 3 for 1st byte after F0~FF
- * - 2 for 1st byte after E0~EF or 2nd byte after F0~FF
- * - 1 for 1st byte after C0~DF or 2nd byte after E0~EF or
- * 3rd byte after F0~FF
- * - 0 for others
- * Error cases:
- * 9,10,11 if non ascii First Byte overlaps
- * E.g., F1 80 C2 90 --> 8 3 10 2, where 10 indicates error
- */
-
- /* Adjust Second Byte range for special First Bytes(E0,ED,F0,F4) */
- /* See _range_adjust_tbl[] definition for details */
- /* Overlaps lead to index 9~15, which are illegal in range table */
- uint8x16_t shift1 = vextq_u8(prev_input, input, 15);
- uint8x16_t pos = vsubq_u8(shift1, const_e0);
- range = vaddq_u8(range, vqtbl2q_u8(range_adjust_tbl, pos));
-
- /* Load min and max values per calculated range index */
- uint8x16_t minv = vqtbl1q_u8(range_min_tbl, range);
- uint8x16_t maxv = vqtbl1q_u8(range_max_tbl, range);
-
- /* Check value range */
- error = vorrq_u8(error, vcltq_u8(input, minv));
- error = vorrq_u8(error, vcgtq_u8(input, maxv));
-
- prev_input = input;
- prev_first_len = first_len;
-
- data += 16;
- len -= 16;
- }
-
- /* Delay error check till loop ends */
- if (vmaxvq_u8(error)) return false;
-
- /* Find previous token (not 80~BF) */
- uint32_t token4;
- vst1q_lane_u32(&token4, vreinterpretq_u32_u8(prev_input), 3);
-
- const int8_t* token = (const int8_t*)&token4;
- int lookahead = 0;
- if (token[3] > (int8_t)0xBF)
- lookahead = 1;
- else if (token[2] > (int8_t)0xBF)
- lookahead = 2;
- else if (token[1] > (int8_t)0xBF)
- lookahead = 3;
-
- data -= lookahead;
- len += lookahead;
- }
-
- /* Check remaining bytes with naive method */
- return validate_utf8_naive(data, len);
-}
bool validate_utf8(const char* src, size_t len) {
- return utf8_range(src, len);
+ return simdutf::validate_utf8(src, len);
}
-#else
-bool validate_utf8(const char* src, size_t len) {
- return validate_utf8_naive(src, len);
-}
-#endif
bool validate_utf8(const TFileScanRangeParams& params, const char* src, size_t
len) {
if (params.__isset.file_attributes &&
!params.file_attributes.enable_text_validate_utf8) {
@@ -334,4 +31,5 @@ bool validate_utf8(const TFileScanRangeParams& params, const
char* src, size_t l
}
return validate_utf8(src, len);
}
+
} // namespace doris
diff --git a/be/src/util/utf8_check.h b/be/src/util/utf8_check.h
index 7e9b7a2a9de..1388834596c 100644
--- a/be/src/util/utf8_check.h
+++ b/be/src/util/utf8_check.h
@@ -22,11 +22,8 @@
#include <cstddef>
namespace doris {
-// check utf8 code using simd instructions
-// Return true - success, false fail
+// Validate UTF-8 with simdutf runtime CPU dispatch. Empty input is valid.
bool validate_utf8(const char* src, size_t len);
-// check utf8 use naive c++
-bool validate_utf8_naive(const char* data, size_t len);
bool validate_utf8(const TFileScanRangeParams& params, const char* src, size_t
len);
} // namespace doris
diff --git a/be/test/util/utf8_check_test.cpp b/be/test/util/utf8_check_test.cpp
index 4e8a0a78ad0..b685542edda 100644
--- a/be/test/util/utf8_check_test.cpp
+++ b/be/test/util/utf8_check_test.cpp
@@ -17,45 +17,58 @@
#include "util/utf8_check.h"
-#include <gtest/gtest-message.h>
-#include <gtest/gtest-test-part.h>
+#include <gtest/gtest.h>
+#include <cstring>
+#include <memory>
+#include <string>
+#include <string_view>
#include <vector>
-#include "gtest/gtest_pred_impl.h"
-
namespace doris {
-struct test {
- const char* data;
- int len;
-};
-
class Utf8CheckTest : public testing::Test {
-public:
- Utf8CheckTest() {}
- virtual ~Utf8CheckTest() {}
+protected:
+ static void check(std::string_view input, bool expected) {
+ // Exact-sized buffers let an instrumented validator detect reads past
the input.
+ auto data = std::make_unique<char[]>(input.size());
+ std::memcpy(data.get(), input.data(), input.size());
+ EXPECT_EQ(validate_utf8(data.get(), input.size()), expected);
+ }
-private:
/* positive tests */
- std::vector<test> pos = {{"", 0},
- {"\x00", 1},
- {"\x66", 1},
- {"\x7F", 1},
- {"\x00\x7F", 2},
- {"\x7F\x00", 2},
- {"\xC2\x80", 2},
- {"\xDF\xBF", 2},
- {"\xE0\xA0\x80", 3},
- {"\xE0\xA0\xBF", 3},
- {"\xED\x9F\x80", 3},
- {"\xEF\x80\xBF", 3},
- {"\xF0\x90\xBF\x80", 4},
- {"\xF2\x81\xBE\x99", 4},
- {"\xF4\x8F\x88\xAA", 4}};
+ std::vector<std::string_view> pos = {{"", 0},
+ {"\x00", 1},
+ {"f", 1},
+ {"\x7F", 1},
+ {"\x00\x7F", 2},
+ {"\x7F\x00", 2},
+ {"\xC2\x80", 2},
+ {"\xDF\xBF", 2},
+ {"\xE0\xA0\x80", 3},
+ {"\xE0\xA0\xBF", 3},
+ {"\xED\x9F\x80", 3},
+ {"\xEF\x80\xBF", 3},
+ {"\xF0\x90\xBF\x80", 4},
+ {"\xF2\x81\xBE\x99", 4},
+ {"\xF4\x8F\x88\xAA", 4},
+ {"\xED\x9F\xBF", 3},
+ {"\xEE\x80\x80", 3},
+ {"\xEF\xBF\xBF", 3},
+ {"\xF0\x90\x80\x80", 4},
+ {"\xF4\x8F\xBF\xBF", 4}};
/* negative tests */
- std::vector<test> neg = {
+ std::vector<std::string_view> neg = {
+ {"\xC2", 1},
+ {"\xE0", 1},
+ {"\xE0\xA0", 2},
+ {"\xF0", 1},
+ {"\xF0\x90", 2},
+ {"\xF0\x90\x80", 3},
+ {"\xF5\x80\x80\x80", 4},
+ {"\xFE", 1},
+ {"\xFF", 1},
{"\x80", 1},
{"\xBF", 1},
{"\xC0\x80", 2},
@@ -96,28 +109,107 @@ private:
35}};
};
TEST_F(Utf8CheckTest, empty) {
- EXPECT_TRUE(validate_utf8(pos[0].data, pos[0].len));
+ EXPECT_TRUE(validate_utf8(nullptr, 0));
+ check("", true);
}
TEST_F(Utf8CheckTest, normal) {
- for (int i = 0; i < sizeof(pos) / sizeof(pos[0]); ++i) {
- EXPECT_TRUE(validate_utf8(pos[i].data, pos[i].len));
+ for (const auto& value : pos) {
+ check(value, true);
}
}
TEST_F(Utf8CheckTest, abnormal) {
- for (int i = 0; i < sizeof(neg) / sizeof(neg[0]); ++i) {
- EXPECT_FALSE(validate_utf8(neg[i].data, neg[i].len));
+ for (const auto& value : neg) {
+ check(value, false);
+ }
+}
+
+TEST_F(Utf8CheckTest, embedded_nul) {
+ check(std::string_view("a\0\xE4\xB8\xAD\0z", 7), true);
+ check(std::string_view("a\0\xFF", 3), false);
+ check(std::string_view("a\0\xE4\xB8", 4), false);
+}
+
+TEST_F(Utf8CheckTest, block_boundaries) {
+ // Exercise every sequence across 16-, 32- and 64-byte SIMD boundaries,
+ // both at the end of the input and followed by another complete block.
+ for (size_t prefix_size = 0; prefix_size < 130; ++prefix_size) {
+ SCOPED_TRACE(prefix_size);
+ for (size_t suffix_size : {0, 1, 16, 32, 64}) {
+ SCOPED_TRACE(suffix_size);
+ const std::string prefix(prefix_size, 'a');
+ const std::string suffix(suffix_size, 'b');
+ for (const auto& value : pos) {
+ check(prefix + std::string(value) + suffix, true);
+ }
+ for (const auto& value : neg) {
+ check(prefix + std::string(value) + suffix, false);
+ }
+ }
}
}
-TEST_F(Utf8CheckTest, naive) {
- for (int i = 0; i < sizeof(pos) / sizeof(pos[0]); ++i) {
- EXPECT_TRUE(validate_utf8_naive(pos[i].data, pos[i].len));
+TEST_F(Utf8CheckTest, unaligned_inputs) {
+ for (size_t offset = 0; offset < 64; ++offset) {
+ SCOPED_TRACE(offset);
+ for (size_t size : {1, 15, 16, 17, 31, 32, 33, 63, 64, 65, 127, 128,
129, 1024}) {
+ SCOPED_TRACE(size);
+ auto data = std::make_unique<char[]>(offset + size);
+ std::memset(data.get(), 'a', offset + size);
+ EXPECT_TRUE(validate_utf8(data.get() + offset, size));
+ data[offset + size - 1] = '\xFF';
+ EXPECT_FALSE(validate_utf8(data.get() + offset, size));
+ }
}
- for (int i = 0; i < sizeof(neg) / sizeof(neg[0]); ++i) {
- EXPECT_FALSE(validate_utf8_naive(neg[i].data, neg[i].len));
+}
+
+TEST_F(Utf8CheckTest, long_inputs) {
+ for (size_t size : {1024, 4096, 65536}) {
+ SCOPED_TRACE(size);
+ std::string data(size, 'a');
+ check(data, true);
+ for (size_t offset : {size_t(0), size / 2, size - 1}) {
+ data[offset] = '\xFF';
+ check(data, false);
+ data[offset] = 'a';
+ }
+ data.clear();
+ for (size_t i = 0; i < size; ++i) {
+ data += "a\xE4\xB8\xAD\xF0\x9F\x98\x80";
+ }
+ check(data, true);
+ data.pop_back();
+ check(data, false);
}
}
+TEST_F(Utf8CheckTest, independent_inputs) {
+ // A truncated sequence in one row must not consume the next row's bytes.
+ const std::string data = "\xE4\xB8\xAD";
+ EXPECT_FALSE(validate_utf8(data.data(), 2));
+ EXPECT_FALSE(validate_utf8(data.data() + 2, 1));
+ EXPECT_TRUE(validate_utf8(data.data(), data.size()));
+}
+
+TEST_F(Utf8CheckTest, file_scan_validation_setting) {
+ TFileScanRangeParams params;
+ const std::string invalid = "\xFF";
+ EXPECT_FALSE(validate_utf8(params, invalid.data(), invalid.size()));
+
+ TFileAttributes attributes;
+ params.__set_file_attributes(attributes);
+ EXPECT_FALSE(validate_utf8(params, invalid.data(), invalid.size()));
+
+ attributes.__set_enable_text_validate_utf8(false);
+ params.__set_file_attributes(attributes);
+ EXPECT_TRUE(validate_utf8(params, invalid.data(), invalid.size()));
+
+ attributes.__set_enable_text_validate_utf8(true);
+ params.__set_file_attributes(attributes);
+ EXPECT_FALSE(validate_utf8(params, invalid.data(), invalid.size()));
+ EXPECT_TRUE(validate_utf8(params, "valid", 5));
+ EXPECT_TRUE(validate_utf8(params, nullptr, 0));
+}
+
} // namespace doris
diff --git a/be/test/util/variant/variant_test_utils.cpp
b/be/test/util/variant/variant_test_utils.cpp
index e4bd16a657d..5d26b42b3f4 100644
--- a/be/test/util/variant/variant_test_utils.cpp
+++ b/be/test/util/variant/variant_test_utils.cpp
@@ -108,7 +108,7 @@ uint8_t minimum_decimal_width(__int128 value) {
}
void require_valid_utf8(StringRef value, const char* description) {
- if (value.size != 0 && !validate_utf8_naive(value.data, value.size)) {
+ if (value.size != 0 && !validate_utf8(value.data, value.size)) {
fail(description);
}
}
diff --git a/build-support/run-clang-tidy.sh b/build-support/run-clang-tidy.sh
index be16b03ee69..b97f2055b7f 100755
--- a/build-support/run-clang-tidy.sh
+++ b/build-support/run-clang-tidy.sh
@@ -50,7 +50,6 @@ EXCLUDED_PATTERNS=(
"be/src/util/mustache/"
"be/src/util/sse2neo.h"
"be/src/util/sse2neon.h"
- "be/src/util/utf8_check.cpp"
"cloud/src/common/defer.h"
"contrib/"
)
diff --git a/dist/LICENSE-dist.txt b/dist/LICENSE-dist.txt
index 0c74fbd9fa5..82e932999bc 100644
--- a/dist/LICENSE-dist.txt
+++ b/dist/LICENSE-dist.txt
@@ -397,32 +397,6 @@ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-------------------------------------------------------------------------------
-be/src/util/utf8_check.cpp: MIT license
-
- MIT License
-
- Copyright (c) 2019 Yibo Cai
-
- Permission is hereby granted, free of charge, to any person obtaining a copy
- of this software and associated documentation files (the "Software"), to deal
- in the Software without restriction, including without limitation the rights
- to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- copies of the Software, and to permit persons to whom the Software is
- furnished to do so, subject to the following conditions:
-
- The above copyright notice and this permission notice shall be included in
all
- copies or substantial portions of the Software.
-
- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- SOFTWARE.
-
---------------------------------------------------------------------------------
-
be/src/util/condition_variable* : BSD-style license
Copyright (c) 2011 The Chromium Authors. All rights reserved.
@@ -1482,6 +1456,8 @@ The Apache Software License, Version 2.0
* benchmark: 1.8.0
* simdjson: 3.0.1
* simdutf: 9.2.0
+ - includes BSD-3-Clause code: simdutf/internal/isadetection.h
+ - additional license: licenses/LICENSE-simdutf-isadetection.txt
* libhdfs3: 2.3.8
* opentelemetry-proto: 0.18.0
* opentelemetry-cpp: 1.4.0
diff --git a/dist/licenses/LICENSE-simdutf-isadetection.txt
b/dist/licenses/LICENSE-simdutf-isadetection.txt
new file mode 100644
index 00000000000..48eec069118
--- /dev/null
+++ b/dist/licenses/LICENSE-simdutf-isadetection.txt
@@ -0,0 +1,46 @@
+simdutf 9.2.0: simdutf/internal/isadetection.h (BSD-3-Clause)
+Source:
https://github.com/simdutf/simdutf/blob/v9.2.0/include/simdutf/internal/isadetection.h
+
+From
+https://github.com/endorno/pytorch/blob/master/torch/lib/TH/generic/simd/simd.h
+Highly modified.
+
+Copyright (c) 2016- Facebook, Inc (Adam Paszke)
+Copyright (c) 2014- Facebook, Inc (Soumith Chintala)
+Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
+Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
+Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
+Copyright (c) 2011-2013 NYU (Clement Farabet)
+Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou,
+Iain Melvin, Jason Weston) Copyright (c) 2006 Idiap Research Institute
+(Samy Bengio) Copyright (c) 2001-2004 Idiap Research Institute (Ronan
Collobert,
+Samy Bengio, Johnny Mariethoz)
+
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in the
+ documentation and/or other materials provided with the distribution.
+
+3. Neither the names of Facebook, Deepmind Technologies, NYU, NEC Laboratories
+America and IDIAP Research Institute nor the names of its contributors may be
+ used to endorse or promote products derived from this software without
+ specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
