This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new ad2d991276e Xiaomi vectorization blog updates (#171)
ad2d991276e is described below
commit ad2d991276e24ce981f0eb4c35d2672ab252f9c1
Author: zihaohuang0429 <[email protected]>
AuthorDate: Thu Dec 8 13:34:37 2022 +0800
Xiaomi vectorization blog updates (#171)
* Create xiaomi_vectorization
---
blog/xiaomi_vector.md | 182 +++++++++++++++++++++
.../xiaomi_vector.md | 165 +++++++++++++++++++
static/images/xiaomi/en/kv.png | Bin 0 -> 371079 bytes
static/images/xiaomi/en/page_1.jpg | Bin 0 -> 77380 bytes
static/images/xiaomi/en/page_10.jpg | Bin 0 -> 102576 bytes
static/images/xiaomi/en/page_11.jpg | Bin 0 -> 78811 bytes
static/images/xiaomi/en/page_12.jpg | Bin 0 -> 110644 bytes
static/images/xiaomi/en/page_2.jpg | Bin 0 -> 86623 bytes
static/images/xiaomi/en/page_3.jpg | Bin 0 -> 48102 bytes
static/images/xiaomi/en/page_4.jpg | Bin 0 -> 167092 bytes
static/images/xiaomi/en/page_5.jpg | Bin 0 -> 101582 bytes
static/images/xiaomi/en/page_6.jpg | Bin 0 -> 63859 bytes
static/images/xiaomi/en/page_7.jpg | Bin 0 -> 74977 bytes
static/images/xiaomi/en/page_8.jpg | Bin 0 -> 79881 bytes
static/images/xiaomi/en/page_9.jpg | Bin 0 -> 80394 bytes
static/images/xiaomi/en/test | 1 +
static/images/xiaomi/zh/1.png | Bin 0 -> 60173 bytes
static/images/xiaomi/zh/10.png | Bin 0 -> 92207 bytes
static/images/xiaomi/zh/11.png | Bin 0 -> 59243 bytes
static/images/xiaomi/zh/12.png | Bin 0 -> 94960 bytes
static/images/xiaomi/zh/2.png | Bin 0 -> 87314 bytes
static/images/xiaomi/zh/3.png | Bin 0 -> 46955 bytes
static/images/xiaomi/zh/4.png | Bin 0 -> 56330 bytes
static/images/xiaomi/zh/5.png | Bin 0 -> 79103 bytes
static/images/xiaomi/zh/6.png | Bin 0 -> 50031 bytes
static/images/xiaomi/zh/7.png | Bin 0 -> 57061 bytes
static/images/xiaomi/zh/8.png | Bin 0 -> 61894 bytes
static/images/xiaomi/zh/9.png | Bin 0 -> 62881 bytes
static/images/xiaomi/zh/kv.png | Bin 0 -> 371079 bytes
static/images/xiaomi/zh/test | 1 +
30 files changed, 349 insertions(+)
diff --git a/blog/xiaomi_vector.md b/blog/xiaomi_vector.md
new file mode 100644
index 00000000000..3609d7f2fb7
--- /dev/null
+++ b/blog/xiaomi_vector.md
@@ -0,0 +1,182 @@
+---
+{
+ "title": "Practice and Optimization of Apache Doris in Xiaomi",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+> Guide: Xiaomi Group introduced Apache Doris in 2019. At present, Apache
Doris has been widely used in dozens of business departments within Xiaomi. A
set of data ecology with Apache Doris has been formed. This article is
transcribed from an online meetup speech of the Doris community, aiming to
share the practice of Apache Doris in Xiaomi.
+
+> Author: ZuoWei, OLAP Engineer, Xiaomi
+
+
+
+# About Xiaomi
+[Xiaomi Corporation](https://www.mi.com/global) (“Xiaomi” or the “Group”;
HKG:1810), a consumer electronics and smart manufacturing company with
smartphones and smart hardware connected by an Internet of Things (IoT)
platform. In 2021, Xiaomi's total revenue amounted to RMB328.3
billion(USD472,231,316,200), an increase of 33.5% year-over-year; Adjusted net
profit was RMB22.0 billion(USD3,164,510,800), an increase of 69.5%
year-over-year.
+
+Due to the growing need of data analysis, Xiaomi Group introduced Apache Doris
in 2019. As one of the earliest users of Apache Doris, Xiaomi Group has been
deeply involved in the open-source community. After three years of development,
Apache Doris has been widely used in dozens of business departments within
Xiaomi, such as Advertising, New Retail, Growth Analysis, Dashboards,
UserPortraits, [AISTAR](https://airstar.com/home), [Xiaomi
Youpin](https://www.xiaomiyoupin.com). Within Xiaomi [...]
+
+
+
+At present, Apache Doris already has dozens of clusters in Xiaomi, with an
overall scale of hundreds of virtual machines . Among them, the largest single
cluster reaches nearly 100 nodes, with dozens of real-time data synchronization
tasks. And the largest daily increment of a single table rocket to 12 billion,
supporting PB-level storage. And a single cluster can support more than 20,000
multi-dimensional analysis queries per day.
+
+# Architecture Evolution
+The original intention of Xiaomi to introduce Apache Doris is to solve the
problems encountered in user behavior analysis. With the development of
Xiaomi's Internet business, the demand for growth analysis using user behavior
data is becoming stronger and stronger. If each business branch builds its own
growth analysis system, it will not only be costly, but also inefficient.
Therefore, if there is a product that can help them stop worrying about
underlying complex technical details, it [...]
+
+## Previous Architecture
+The growth analysis platform project was established in mid-2018. At that
time, based on the consideration of development time and cost, Xiaomi reused
various existing big data basic components (HDFS, Kudu, SparkSQL, etc.) to
build a growth analysis query system based on Lambda architecture. The
architecture of the first version of the GA system is shown in the figure
below, including the following aspects:
+
+- Data Source: The data source is the front-end embedded data and user
behavior data.
+- Data Access: The event tracking data is uniformly cleaned and ingested into
Xiaomi's internal self-developed message queue, and the data is imported into
Kudu through Spark Streaming.
+- Storage: Separate hot and cold data in the storage layer. Hot data is stored
in Kudu, and cold data is stored in HDFS. At the same time, partitioning is
carried out in the storage layer. When the partition unit is day, part of the
data will be cooled and stored on HDFS every night.
+- Compute and Query: In the query layer, use SparkSQL to perform federated
queries on the data on Kudu and HDFS, and finally display the query results on
the front-end page.
+
+
+
+At that time, the first version of the growth analysis platform helped us
solve a series of problems in the user operation process, but there were also
two problems:
+
+### Problem No.1: Scattered components
+Since the historical architecture is based on the combination of SparkSQL +
Kudu + HDFS, too many dependent components lead to high operation and
maintenance costs. The original design is that each component uses the
resources of the public cluster, but in practice, it is found that during the
execution of the query job, the query performance is easily affected by other
jobs in the public cluster, and query jitter is prone to occur, especially when
reading data from the HDFS public clust [...]
+
+### Problem No.2: High resource consumption
+When querying through SparkSQL, the latency is relatively high. SparkSQL is a
query engine designed based on a batch processing system. In the process of
exchanging data shuffle between each stage, it still needs to be placed on the
disk, and the delay in completing the SQL query is relatively high. In order to
ensure that SQL queries are not affected by resources, we ensure query
performance by adding machines. However, in practice, we find that there is
limited room for performance imp [...]
+
+In response to the above two problems, our goal is to seek an MPP database
that integrates computing and storage to replace our current storage and
computing layer components. After technical selection, we finally decided to
use Apache Doris to replace the older generation of historical architecture.
+
+## New Choice
+Popular MPP-based query engines such as Impala and Presto, can efficiently
support SQL queries, but they still need to rely on Kudu, HDFS, Hive Metastore
and other storage system, which increase the operation and maintenance costs.
At the same time, due to the separation of storage and compute, the query
engine cannot easily find the data changes in the storage layer, resulting in
bad performance in detailed query optimization. If you want to cache at the SQL
layer, you cannot guarantee [...]
+
+Apache Doris is a top-level project of the Apache Foundation. It is mainly
positioned as a high-performance, real-time analytical database, and is mainly
used to solve reports and multi-dimensional analysis. It integrates Google Mesa
and Cloudera Impala technologies. We conducted an in-depth performance tests on
Doris and communicated with the community many times. And finally, we
determined to replace the previous computing and storage components with Doris.
+
+## New Architecture Based on Apache Doris
+The new architecture obtains event tracking data from the data source. Then
data is ingested into Apache Doris. Query results can be directly displayed in
the applications. In this way, Doris has truly realized the unification of
computing, storage, and resource management tools.
+
+
+
+We chose Doris because:
+- Doris has excellent query performance and can meet our business needs.
+- Doris supports standard SQL, and the learning cost is low.
+- Doris does not depend on other external components and is easy to operate
and maintain.
+- The Apache Doris community is very active and friendly, crowded with
contributors. It is easier for further versions upgrades and convenient for
maintenance.
+
+## Query Performance Comparision between Apache Doris & Spark SQL
+Note: The comparison is based on Apache Doris V0.13
+
+
+
+We selected a business model with an average daily data volume of about 1
billion, and conducted performance tests on Doris in different scenarios,
including 6 event analysis scenarios, 3 retention analysis scenarios, and 3
funnel analysis scenarios. After comparing it with the previous
architecture(SparkSQL+Kudu+HDFS), we found out:
+- In the event analysis scenario, the average query time was reduced by 85%.
+- In the scenarios of retention analysis and funnel analysis, the average
query time was reduced by 50%.
+
+# Real Practice
+Below we will introduce our experience of data import, data query, A/B test in
the business application of Apache Doris.
+
+## Data Import
+
+
+
+Xiaomi writes data into Doris mainly through Stream Load, Broker Load and a
small amount of data by Insert. Usually data is generally ingested into the
message queue first, which is divided into real-time and offline data.
+
+### How to write real-time data into Apache Doris:
+After part of real-time data processed by Flink, they will be ingested into
Doris through Flink-Doris-Connector provided by Apache Doris. The rest of the
data is ingested through Spark Streaming. The bottom layer of these two writing
approaches both rely on the Stream Load provided by Apache Doris.
+
+### How to write offline data into Apache Doris:
+After offline data is partially ingested into Hive, they will be ingested into
Doris through Xiaomi's data import tool. Users can directly submit Broker Load
tasks to the Xiaomi's data import tool and import data directly into Doris, or
import data through Spark SQL, which relies on the Spark-Doris-Connector
provided by Apache Doris. Spark Doris Connector is actually the encapsulation
of Stream Load.
+
+## Data Qurey
+
+
+
+Users can query after data import is done. Inside Xiaomi, we query through our
data platform. Users can perform visual queries on Doris through Xiaomi's data
platform, and conduct user behavior analysis and user portrait analysis. In
order to help our teams conduct event analysis, retention analysis, funnel
analysis, path analysis and other behavioral analysis, we have added
corresponding UDF (User Defined Function) and UDAF (User Defined Aggregate
Function) to Doris.
+
+In the upcoming version 1.2, Apache Doris adds the function of synchronizing
metadata through external table, such as Hive/Hudi/Iceberg and Multi Catalog
tool. External table query improves performance, and the ability to access
external tables greatly increases ease of use. In the future, we will consider
querying Hive and Iceberg data directly through Doris, which builds an
architecture of datalake.
+
+## A/B Test
+In real business, the A/B test is a method of comparing two versions of
strategies against each other to determine which one performs better. A/B test
is essentially an experiment where two or more variants of a page are shown to
users at random, and statistical analysis. It is popular approach used to
determine which variation performs better for a given conversion goal. Xiaomi's
A/B test platform is an operation tool product that conducts the A/B test with
experimental grouping, traffi [...]
+
+Apache Doris also provides services to Xiaomi's A/B test platform. Everyday,
Xiaomi's A/B test platform needs to process a temendous amount of data with
billions of queries. That's why Xiaomi's A/B test platform is eager to improve
the query performance.
+
+Apache Doris V1.1 released just in time and has fully supported vectorization
in the processing and storage. Compared with the non-vectorized version, the
query performance has been significantly improved. It is time to update
Xiaomi's Doris cluster to the latest version. That's why we first launched the
latest vectorized version of Doris on Xiaomi's A/B test platform.
+
+## Test before Launch
+Note: The following tests are based on Apache Doris V1.1.2
+
+We built a test cluster for Apache Doris V1.1.2, which is as big as that of
the Xiaomi online Apache Doris V0.13 version, to test before the vectorization
version goes online. The test is divided into two aspects: single SQL parrellel
query test and batch SQL concurrent query test.
+
+The configurations of the two clusters are exactly the same, and the specific
configuration information is as follows:
+- Scale: 3 FEs + 89 virtual machines
+- CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz 16 cores 32 threads × 2
+- Memory: 256GB
+- Disk: 7.3TB × 12 HDD
+
+### Single SQL Parrellel Query Test
+We choose 7 classic queries in the Xiaomi A/B test. For each query, we limited
the time range to 1 day, 7 days, and 20 days for testing, where the daily
partition data size is about 3.1 billion (the data volume is about 2 TB). The
test results are shown in the figures:
+
+
+
+
+
+
+
+The Apache Doris V1.1.2 has at least 3~5 times performance improvement
compared to the Xiaomi online Doris V0.13, which is remarkable.
+
+## Optimization
+Note: The following tests are based on Apache Doris V1.1.2
+
+Based on Xiaomi's A/B test business data, we tuned Apache Doris V1.1.2 and
conducted concurrent query tests on the tuned Doris V1.1.2 and Xiaomi's online
Doris V0.13. The test results are as follows.
+
+### Optimization in Test 1
+We choose user deduplication, index summation, and covariance calculation
query(the total number of SQL is 3245) in the A/B test to conduct concurrent
query tests on the two versions. The single-day partition data of the table is
about 3.1 billion (the amount of data is about 2 TB) and the query will be
based on the latest week's data. The test results are shown in the figures:
+
+
+
+Compared with Apache Doris V0.13, the overall average latency of Doris V1.1.2
is reduced by about 48%, and the P95 latency is reduced by about 49%. In this
test, the query performance of Doris V1.1.2 was nearly doubled compared to
Doris V0.13.
+
+### Optimization in Test 2
+We choose 7 A/B test reports to test the two versions. Each A/B test report is
corresponded to two modules in Xiaomi A/B test platform and each module
represents thousands of SQL query. Each report submits query tasks to the
cluster where the two versions reside at the same concurrency. The test results
are shown in the figure:
+
+
+
+Compared with Doris V0.13, Doris V1.1.2 reduces the overall average latency by
around 52%. In the test, the query performance of Doris V1.1.2 version was more
than 1 time higher than that of Doris V0.13.
+
+### Optimization in Test 3
+To verify the performance of the tuned Apache Doris V1.1.2 in other cases, we
choose the Xiaomi user behavior analysis to conduct concurrent query
performance tests of Doris V1.1.2 and Doris V0.13. We choose behavior analysis
query for 4 days on October 24, 25, 26 and 27, 2022. The test results are shown
in the figures:
+
+
+
+Compared with Doris V0.13, the overall average latency of Doris V1.1.2 has
been reduced by about 77%, and the P95 latency has been reduced by about 83%.
In this test, the query performance of Doris V1.1.2 version is 4~6 times higher
than that of Doris V0.13.
+
+# Conclusion
+Since we adopted Apache Doris in 2019, Apache Doris has currently served
dozens of businesses and sub-brands within Xiaomi, with dozens of clusters and
hundreds of nodes. It completes more than 10,000 user online analysis queries
every day and is responsible for most of the online analysis in Xiaomi.
+
+After performance test and tuning, Apache Doris V1.1.2 has met the launch
requirements of the Xiaomi A/B test platform and does well in query performance
and stability. In some cases, it even exceeds our expectations, such as the
overall average latency being reduced by about 77% in our tuned version.
+
+Meanwhile, some functions have in the above been released in Apache Doris V1.0
or V1.1, some PRs have been merged into the community Master Fork and should
be released soon. Recently the activity of the community has been greatly
enhanceed. We are glad to see that Apache Doris has become more and more
mature, and stepped forward to an integrated datalake. We truly believe that in
the future, more data analysis will be explored and realized within Apache
Doris.
+
+
+# Contact Us
+Apache Doris Website：http://doris.apache.org
+
+Github Homepage：https://github.com/apache/doris
+
+Email to DEV：[email protected]
+
diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/xiaomi_vector.md
b/i18n/zh-CN/docusaurus-plugin-content-blog/xiaomi_vector.md
new file mode 100644
index 00000000000..381ed0e2f9f
--- /dev/null
+++ b/i18n/zh-CN/docusaurus-plugin-content-blog/xiaomi_vector.md
@@ -0,0 +1,165 @@
+---
+{
+ "title": "最佳实践: Apache Doris 在小米数据场景的应用实践与优化",
+ "language": "zh"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+>导读：小米集团于 2019 年首次引入了 Apache Doris ，目前 Apache Doris
已经在小米内部数十个业务中得到广泛应用，并且在小米内部已经形成一套以 Apache Doris 为核心的数据生态。本篇文章转录自 Doris 社区线上
Meetup 主题演讲，旨在分享 Apache Doris 在小米数据场景的落地实践与优化实践。
+
+>作者｜魏祚 小米 OLAP 引擎研发工程师
+
+
+
+# 关于小米
+小米公司（“小米”或“集团”；HKG：1810），一家消费电子和智能制造公司，其智能手机和智能硬件通过物联网 (IoT) 平台连接。
2021年，小米总收入达到人民币3283亿元（4722.3131.62亿美元），同比增长33.5%；调整后净利润为人民币 220 亿元（316,451.08
万美元），同比增长 69.5%。
+
+因分析业务的增长，小米集团于 2019 年首次引入了 Apache Doris 。经过三年时间的发展，目前 Apache Doris
已经在广告投放、新零售、增长分析、数据看板、用户画像、天星数科、小米有品、等小米内部数十个业务和品牌中得到广泛应用，并且在小米内部已经围绕 Apache
Doris 为核心建设了数据生态。
+
+
+
+当前 Apache Doris 在小米内部已经具有数十个集群、总体达到数百台 BE
节点的规模，其中单集群最大规模达到近百台节点，拥有数十个实时数据同步任务，每日单表最大增量 120 亿、支持 PB 级别存储，单集群每天可以支持 2W
次以上的多维分析查询。
+
+# 架构演进
+小米引入 Apache Doris
的初衷是为了解决内部的用户行为分析中所遇到的问题。随着小米互联网业务的发展，利用用户行为数据进行增长分析的需求越来越强烈。如果每个业务产品线都自己搭建一套增长分析系统，不仅成本高昂，效率也不高。因此如果能有一款产品能够帮助他们不用关心底层的复杂技术细节，让相关业务人员能够专注于自己的技术工作，可以极大提高工作效率。所以，小米大数据和云平台联合开发了增长分析系统
Growing Analytics（下文中简称 GA)，旨在提供一个灵活的多维实时查询和分析平台，可以统一管理数据接入和查询方案，帮助业务线做好精细化运营。
+
+## 历史架构
+增长分析平台立项于 2018 年年中，当时基于开发时间和成本，技术栈等因素的考虑，小米复用了现有各种大数据基础组件（HDFS, Kudu, SparkSQL
等），搭建了一套基于 Lamda 架构的增长分析查询系统。GA 系统初代版本的架构如下图所示，包含了以下几个方面：
+- 数据源：数据源是前端的埋点数据以及用户行为数据。
+- 数据接入层：对埋点数据进行统一的清洗后写入小米内部自研的消息队列中，并通过 Spark Streaming 将数据导入Kudu 中。
+- 存储层：在存储层中进行冷热数据分离。热数据存放在 Kudu 中，冷数据则会存放在 HDFS
上。同时在存储层中进行分区，当分区单位为天时，每晚会将一部分数据转冷并存储到 HDFS 上。
+- 计算层/查询层：在查询层中，使用 SparkSQL 对 Kudu 与 HDFS 上数据进行联邦查询，最终把查询结果显示在前端页面。
+
+
+
+在当时的历史背景下，初代版本的增长分析平台帮助我们解决了一系列用户运营过程中的问题，但同时在历史架构中也存在了两个问题：
+
+### 第一个问题：组件分散
+由于历史架构是基于 SparkSQL + Kudu + HDFS
的组合，依赖的组件过多导致运维成本较高。原本的设计是各个组件都使用公共集群的资源，但是实践过程中发现执行查询作业的过程中，查询性能容易受到公共集群其他作业的影响，容易发生查询抖动，尤其在读取
HDFS 公共集群的数据时，有时较为缓慢。
+
+### 第二个问题：资源占用高
+通过 SparkSQL 进行查询时，延迟相对较高。SparkSQL 是基于批处理系统设计的查询引擎，在每个 Stage 之间交换数据 Shuffle
的过程中依然需要进行落盘，完成 SQL 查询的时延较高。为了保证 SQL
查询不受资源的影响，我们通过添加机器来保证查询性能，但是实践过程中发现，性能提升的空间有限，这套解决方案并不能充分地利用机器资源来达到高效查询的目的，存在一定的资源浪费。
+
+针对上述两个问题，我们的目标是寻求一款计算、存储一体的 MPP 数据库来替代我们目前的存储计算层的组件，在通过技术选型后，最终我们决定使用 Apache
Doris 替换老一代历史架构。
+
+## 重新选型
+
+MPP架构的查询引擎，如Impala,Presto等能够高效地支持SQL查询，但是仍然需要依赖Kudu, HDFS, Hive Metastore等组件,
运维成本依然比较高。同时，由于计算存储分离，查询引擎不能很好地及时感知存储层的数据变化，就无法做更细致的查询优化。如想在SQL层做缓存就无法保证查询的结果是最新的。
+
+Doris是Apache基金会顶级项目，主要定位是高性能的、支持实时的分析型数据库， 主要用于解决报表和多维分析。它主要集成了 Google Mesa 和
Cloudera Impala
技术。我们对Doris进行了内部的性能测试并多次和社区沟通交流，确定了用Doris替换原来的计算存储组件的解决方案。我们新的架构如下图所示：
+
+## 基于 Apache Doris 的新版架构
+新版架构从数据源获取埋点数据后，数据接入后写入 Apache Doris
后可以直接查询结果并在前端进行显示。真正实现了通过Doris统一了计算、存储，和资源管理yarn相关工具。
+
+
+
+我们选择 Doris 原因：
+- Doris 具有优秀的查询性能，能够满足业务需求。
+- Doris 支持标准 SQL ，用户使用与学习成本较低。
+- Doris 不依赖于其他的外部系统，运维简单。
+- Doris 社区拥有很高活跃度，版本迭代快。开发者规模大，有利于后续系统的维护升级。
+
+## 新旧架构性能对比
+
+
+
+我们选取了日均数据量大约 10 亿的业务，分别在不同场景下对Doris进行了性能测试，其中包含 6 个事件分析场景，3 个留存分析场景以及 3
个漏斗分析场景。经过与【SparkSQL+Kudu+HDFS】的旧方案对比后，我们发现：
+- 在事件分析的场景下，平均查询所耗时间降低了 85%。
+- 在留存分析和漏斗分析场景下，平均查询所耗时间降低了 50%。
+
+# 应用实践
+下面将介绍我们在Apache Doris应用中数据导入、数据查询、A/B测试的经验。
+
+## 数据导入
+
+
+
+小米内部主要通过 Stream Load 与 Broker Load 以及少量 Insert
方式来导入数据到Doris。数据一般会先写入到消息队列中，分为实时数据和离线数据两个部分。
+实时数据如何写入到Apache Doris 中：一部分实时数据通过 Flink数据处理 后， 并通过 Doris 社区提供的 Flink Doris
Connector 组件写入到 Doris 中。另一部分数据通过 Spark Streaming 组件写入。这两种写入方式的底层都依赖的是社区提供的
Stream Load。
+离线数据如何写入到Apache Doris 中：离线数据部分写入 Hive 后，通过小米的数据工场将数据导入到 Doris 中。用户可以直接在数据工场提交
Broker Load 任务并将数据直接导入 Doris 中，也可以通过 Spark SQL 将数据导入 。Spark SQL 方式则是依赖了 Doris
社区提供的 Spark Doris Connector 组件，其底层为 Doris 的 Stream Load 的封装。
+
+## 数据查询
+
+
+
+用户通过数据工场将数据导入至 Doris 后即可进行查询。在小米内部，可以通过自研的数鲸平台进行查询的。用户可以通过数鲸平台对 Doris
进行可视化的查询，并展开用户行为分析和用户画像分析。其中，为帮助业务进行事件分析、留存分析、漏斗分析、路径分析等行为分析，我们为 Doris 添加了相应的
UDF （User Defined Function）和 UDAF (User Defined Aggregate Function)。
+在即将发布的 1.2 版本中，Doris添加了外表元数据同步的功能，支持 Hive/Hudi/Iceberg 外表并增加了 Multi
Catalog。查询外部表提升了性能，接入外表大幅增加了易用性。在未来，我们考虑直接通过 Doris 查询 Hive 与 Iceberg
数据，构建湖仓一体的架构。
+
+## A/B测试
+小米的 A/B 实验平台对 Apache Doris 查询性能的提升有着迫切的需求，因此我们选择优先在小米的 A/B 实验平台上线 Apache Doris
向量化版本，也就是 1.1.2 版本。
+
+小米的 A/B 实验平台是一款通过 A/B
测试的方式，借助实验分组、流量拆分与科学评估等手段来辅助完成科学的业务决策，最终实现业务增长的一款运营工具产品。在实际业务中，为了验证一个新策略的效果，通常需要准备原策略
A 和新策略 B 两种方案。随后在总体用户中取出一小部分，将这部分用户完全随机地分在两个组中，使两组用户在统计角度无差别。将原策略 A 和新策略 B
分别展示给不同的用户组，一段时间后，结合统计方法分析数据，得到两种策略生效后指标的变化结果，并以此来判断新策略 B 是否符合预期。
+
+小米的 A/B 实验平台有几类典型的查询应用：用户去重、指标求和、实验协方差计算等，查询类型会涉及较多的
Count(distinct)、Bitmap计算、Like语句等。
+
+### 上线前验证
+我们基于 Apache Doris 1.1.2 版本搭建了一个和小米线上 Apache Doris 0.13
版本在机器配置和机器规模上完全相同的测试集群，用于向量化版本上线前的验证。验证测试分为两个方面：单 SQL 串行查询测试和批量 SQL
并发查询测试。在这两种测试中，我们在保证两个集群数据完全相同的条件下，分别在 Doris 1.1.2 测试集群和小米线上 Doris 0.13
集群执行相同的查询 SQL 来做性能对比。我们的目标是，Doris 1.1.2 版本在小米线上 Doris 0.13 版本的基础上有 1 倍的查询性能提升。
+两个集群配置完全相同，具体配置信息如下：
+- 集群规模：3 FE + 89 BE
+- BE节点CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz 16核 32线程 × 2
+- BE节点内存：256GB
+- BE节点磁盘：7.3TB × 12 HDD
+
+#### 单 SQL 串行查询测试
+在该测试场景中，我们选取了小米A/B 实验场景中 7 个典型的查询 Case，针对每一个查询 Case，我们将扫描的数据时间范围分别限制为 1 天、7 天和
20 天进行查询测试，其中单日分区数据量级大约为 31 亿（数据量大约 2 TB），测试结果如图所示：
+
+
+
+
+
+
+
+根据以上小米 A/B 实验场景下的单 SQL 串行查询测试结果所示，Doris 1.1.2 版本相比小米线上 Doris 0.13 版本至少有 3~5
倍的性能提升，效果显著。
+
+### 调优测试结果
+我们基于小米的 A/B实验场景对 Apache Doris 1.1.2 版本进行了一系列调优，并将调优后的 Doris 1.1.2 版本与小米线上
Doris 0.13 版本分别进行了并发查询测试。测试情况如下：
+
+#### 测试 1
+我们选择了 A/B 实验场景中一批典型的用户去重、指标求和以及协方差计算的查询 Case（SQL 总数量为
3245）对两个版本进行并发查询测试，测试表的单日分区数据大约为 31 亿（数据量大约 2
TB），查询的数据范围会覆盖最近一周的分区。测试结果如图所示，Doris 1.1.2 版本相比 Doris0.13版本，总体的平均延迟降低了大约
48%，P95 延迟降低了大约 49%。在该测试中，Doris 1.1.2 版本相比 Doris0.13 版本的查询性能提升了接近 1 倍。
+
+
+
+#### 测试 2
+我们选择了 A/B实验场景下的 7 份 A/B 实验报告对两个版本进行测试，每份 A/B 实验报告对应小米
A/B实验平台页面的两个模块，每个模块对应数百或数千条查询 SQL。每一份实验报告都以相同的并发向两个版本所在的集群提交查询任务。测试结果如图所示，Doris
1.1.2 版本相比 Doris 0.13 版本，总体的平均延迟降低了大约 52%。在该测试中，Doris 1.1.2 版本相比 Doris 0.13
版本的查询性能提升了超过 1 倍。
+
+
+
+#### 测试 3
+为了验证调优后的 Apache Doris 1.1.2 版本在小米 A/B 实验场景之外的性能表现，我们选取了小米用户行为分析场景进行了 Doris
1.1.2 版本和 Doris 0.13 版本的并发查询性能测试。我们选取了 2022年10月24日、25日、26日和 27日这 4
天的小米线上真实的行为分析查询 Case 进行对比查询，测试结果如图所示，Doris 1.1.2 版本相比 Doris 0.13
版本，总体的平均延迟降低了大约7 7%，P95 延迟降低了大约 83%。在该测试中，Doris 1.1.2 版本相比 Doris 0.13 版本的查询性能有
4~6 倍的提升。
+
+
+
+# 总结
+自从 Apache Doris 从 2019 年上线第一个业务至今，目前 Apache Doris
已经在小米内部服务了数十个业务及子品牌、集群数量达到数十个、节点规模达到数百台。每天完成数万次用户在线分析查询，承担了包括增长分析和报表查询等绝大多数在线分析的需求。
+
+经过一个多月的性能调优和测试，Apache Doris 1.1.2 版本在查询性能和稳定性方面已经达到了小米
A/B实验平台的上线要求，在某些场景下的查询性能甚至超过了我们的预期，希望本次分享可以给有需要的朋友一些可借鉴的经验参考。
+
+与此同时，在以上小米的实践中，已有部分功能在 Apache Doris 1.0 或 1.1 版本中发布，部分 PR 已经合入社区
Master，将在不久后发布的 1.2 新版本中与大家见面。随着社区的快速发展，有越来越多小伙伴参与到社区建设中，社区活跃度有了极大的提升。Apache
Doris 已经变得越来越成熟，并开始从单一计算存储一体的分析型 MPP 数据库走向湖仓一体的道路，相信在未来，还会有更多的数据分析场景被探索和实现。
+
+
+# 联系我们
+官网：http://doris.apache.org
+
+Github：https://github.com/apache/doris
+
+dev邮件组：[email protected]
+
diff --git a/static/images/xiaomi/en/kv.png b/static/images/xiaomi/en/kv.png
new file mode 100644
index 00000000000..e6775369347
Binary files /dev/null and b/static/images/xiaomi/en/kv.png differ
diff --git a/static/images/xiaomi/en/page_1.jpg
b/static/images/xiaomi/en/page_1.jpg
new file mode 100644
index 00000000000..49d5d75bd71
Binary files /dev/null and b/static/images/xiaomi/en/page_1.jpg differ
diff --git a/static/images/xiaomi/en/page_10.jpg
b/static/images/xiaomi/en/page_10.jpg
new file mode 100644
index 00000000000..988327264b3
Binary files /dev/null and b/static/images/xiaomi/en/page_10.jpg differ
diff --git a/static/images/xiaomi/en/page_11.jpg
b/static/images/xiaomi/en/page_11.jpg
new file mode 100644
index 00000000000..f4f5256b120
Binary files /dev/null and b/static/images/xiaomi/en/page_11.jpg differ
diff --git a/static/images/xiaomi/en/page_12.jpg
b/static/images/xiaomi/en/page_12.jpg
new file mode 100644
index 00000000000..97381a45262
Binary files /dev/null and b/static/images/xiaomi/en/page_12.jpg differ
diff --git a/static/images/xiaomi/en/page_2.jpg
b/static/images/xiaomi/en/page_2.jpg
new file mode 100644
index 00000000000..1a545f45383
Binary files /dev/null and b/static/images/xiaomi/en/page_2.jpg differ
diff --git a/static/images/xiaomi/en/page_3.jpg
b/static/images/xiaomi/en/page_3.jpg
new file mode 100644
index 00000000000..b5927449226
Binary files /dev/null and b/static/images/xiaomi/en/page_3.jpg differ
diff --git a/static/images/xiaomi/en/page_4.jpg
b/static/images/xiaomi/en/page_4.jpg
new file mode 100644
index 00000000000..15d3f267c58
Binary files /dev/null and b/static/images/xiaomi/en/page_4.jpg differ
diff --git a/static/images/xiaomi/en/page_5.jpg
b/static/images/xiaomi/en/page_5.jpg
new file mode 100644
index 00000000000..2807028871c
Binary files /dev/null and b/static/images/xiaomi/en/page_5.jpg differ
diff --git a/static/images/xiaomi/en/page_6.jpg
b/static/images/xiaomi/en/page_6.jpg
new file mode 100644
index 00000000000..aee12017b0a
Binary files /dev/null and b/static/images/xiaomi/en/page_6.jpg differ
diff --git a/static/images/xiaomi/en/page_7.jpg
b/static/images/xiaomi/en/page_7.jpg
new file mode 100644
index 00000000000..4585bea1c1a
Binary files /dev/null and b/static/images/xiaomi/en/page_7.jpg differ
diff --git a/static/images/xiaomi/en/page_8.jpg
b/static/images/xiaomi/en/page_8.jpg
new file mode 100644
index 00000000000..f886c98d996
Binary files /dev/null and b/static/images/xiaomi/en/page_8.jpg differ
diff --git a/static/images/xiaomi/en/page_9.jpg
b/static/images/xiaomi/en/page_9.jpg
new file mode 100644
index 00000000000..6434b6b8f5b
Binary files /dev/null and b/static/images/xiaomi/en/page_9.jpg differ
diff --git a/static/images/xiaomi/en/test b/static/images/xiaomi/en/test
new file mode 100644
index 00000000000..8b137891791
--- /dev/null
+++ b/static/images/xiaomi/en/test
@@ -0,0 +1 @@
+
diff --git a/static/images/xiaomi/zh/1.png b/static/images/xiaomi/zh/1.png
new file mode 100644
index 00000000000..46296055a8f
Binary files /dev/null and b/static/images/xiaomi/zh/1.png differ
diff --git a/static/images/xiaomi/zh/10.png b/static/images/xiaomi/zh/10.png
new file mode 100644
index 00000000000..4844293d331
Binary files /dev/null and b/static/images/xiaomi/zh/10.png differ
diff --git a/static/images/xiaomi/zh/11.png b/static/images/xiaomi/zh/11.png
new file mode 100644
index 00000000000..6ec1f3cdf70
Binary files /dev/null and b/static/images/xiaomi/zh/11.png differ
diff --git a/static/images/xiaomi/zh/12.png b/static/images/xiaomi/zh/12.png
new file mode 100644
index 00000000000..16c9b5edb3c
Binary files /dev/null and b/static/images/xiaomi/zh/12.png differ
diff --git a/static/images/xiaomi/zh/2.png b/static/images/xiaomi/zh/2.png
new file mode 100644
index 00000000000..eaa2123583e
Binary files /dev/null and b/static/images/xiaomi/zh/2.png differ
diff --git a/static/images/xiaomi/zh/3.png b/static/images/xiaomi/zh/3.png
new file mode 100644
index 00000000000..6cd9e232bd1
Binary files /dev/null and b/static/images/xiaomi/zh/3.png differ
diff --git a/static/images/xiaomi/zh/4.png b/static/images/xiaomi/zh/4.png
new file mode 100644
index 00000000000..42f0642f4e5
Binary files /dev/null and b/static/images/xiaomi/zh/4.png differ
diff --git a/static/images/xiaomi/zh/5.png b/static/images/xiaomi/zh/5.png
new file mode 100644
index 00000000000..d0eb413a5a4
Binary files /dev/null and b/static/images/xiaomi/zh/5.png differ
diff --git a/static/images/xiaomi/zh/6.png b/static/images/xiaomi/zh/6.png
new file mode 100644
index 00000000000..6ccea54e2c8
Binary files /dev/null and b/static/images/xiaomi/zh/6.png differ
diff --git a/static/images/xiaomi/zh/7.png b/static/images/xiaomi/zh/7.png
new file mode 100644
index 00000000000..5eeb11db8aa
Binary files /dev/null and b/static/images/xiaomi/zh/7.png differ
diff --git a/static/images/xiaomi/zh/8.png b/static/images/xiaomi/zh/8.png
new file mode 100644
index 00000000000..f46aad2a333
Binary files /dev/null and b/static/images/xiaomi/zh/8.png differ
diff --git a/static/images/xiaomi/zh/9.png b/static/images/xiaomi/zh/9.png
new file mode 100644
index 00000000000..18c41151914
Binary files /dev/null and b/static/images/xiaomi/zh/9.png differ
diff --git a/static/images/xiaomi/zh/kv.png b/static/images/xiaomi/zh/kv.png
new file mode 100644
index 00000000000..e6775369347
Binary files /dev/null and b/static/images/xiaomi/zh/kv.png differ
diff --git a/static/images/xiaomi/zh/test b/static/images/xiaomi/zh/test
new file mode 100644
index 00000000000..8b137891791
--- /dev/null
+++ b/static/images/xiaomi/zh/test
@@ -0,0 +1 @@
+
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
