This is an automated email from the ASF dual-hosted git repository.
juzhiyuan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/apisix-website.git
The following commit(s) were added to refs/heads/master by this push:
new 6280d80 docs: add APISIX integrate SkyWalking plugins blog (#793)
6280d80 is described below
commit 6280d8090b838e4c8e5dc9ca1225ae4bfea9dbbb
Author: Sylvia <39793568+sylviab...@users.noreply.github.com>
AuthorDate: Tue Dec 7 13:59:03 2021 +0800
docs: add APISIX integrate SkyWalking plugins blog (#793)
---
.../12/07/apisix-integrate-skywalking-plugin.md | 198 +++++++++++++++++++++
.../12/07/apisix-integrate-skywalking-plugin.md | 196 ++++++++++++++++++++
2 files changed, 394 insertions(+)
diff --git a/website/blog/2021/12/07/apisix-integrate-skywalking-plugin.md
b/website/blog/2021/12/07/apisix-integrate-skywalking-plugin.md
new file mode 100644
index 0000000..a0a01d1
--- /dev/null
+++ b/website/blog/2021/12/07/apisix-integrate-skywalking-plugin.md
@@ -0,0 +1,198 @@
+---
+title: "Apache APISIX Integrates with SkyWalking to Create a Full Fange of Log
Processing"
+author: "Haochao Zhuang"
+authorURL: "https://github.com/dmsolr";
+authorImageURL: "https://avatars.githubusercontent.com/u/29735230?v=4";
+keywords:
+- Apache APISIX
+- Apache SkyWalking
+- Log Process
+- Plugin
+description: This paper mainly introduces two Apache APISIX integrated
SkyWalking log plug-ins to provide a more convenient operation and environment
for log processing in Apache APISIX.
+tags: [Technology]
+---
+
+> This paper mainly introduces two Apache APISIX integrated SkyWalking log
plug-ins to provide a more convenient operation and environment for log
processing in Apache APISIX.
+
+<!--truncate-->
+
+In the field of observability, the three main directions of data collection
and analysis, Metrics, Logger and Tracing, are usually used to achieve insight
into the operational status of applications.
+
+Apache APISIX has integrated Apache SkyWaling Tracing capabilities as early as
version 1.4, with features such as error logging and access log collection
added in subsequent versions. Now with Apache SkyWalking's support for Metrics,
it enables Apache APISIX to implement a one-stop observable solution in
integrated mode, covering both logging, metrics and call tracing.
+
+## Feature Development Background
+
+Those of you who are familiar with Apache APISIX should know that Apache
APISIX produces two types of logs during operation, namely the access log and
the error log.
+
+Access logs record detailed information about each request and are logs
generated within the scope of the request, so they can be directly associated
with Tracing. Error logs, on the other hand, are Apache APISIX runtime output
log messages, which are application-wide logs, but cannot be 100% associated
with requests.
+
+At present, Apache APISIX provides very rich log processing plug-ins,
including TCP/HTTP/Kafka and other collection and reporting plug-ins, but they
are weakly associated with Tracing. Take Apache SkyWalking as an example. We
extract the SkyWalking Tracing Conetxt Header from the log records of Apache
APISIX and export it to the file system, and then use the log processing
framework (fluentbit) to convert the logs into a log format acceptable to
SkyWalking. The Tracing Context is then pa [...]
+
+Obviously, the above way of handling the process is tedious and complicated,
and requires additional conversion of log formats. For this reason, in
[PR#5500](https://github.com/apache/apisix/pull/5550) we have implemented the
Apache SkyWalking access log into the Apache APISIX plug-in ecosystem to make
it easier for users to collect and process logs using Apache SkyWalking in
Apache APISIX.
+
+## Introduction of the New Plugins
+
+### SkyWalking Logger Pulgin
+
+The SkyWalking Logger plugin parses the SkyWalking Tracing Context Header and
prints the relevant Tracing Context information to the log, thus enabling the
log to be associated with the call chain.
+
+By using this plug-in, Apache APISIX can get the SkyWalking Tracing Context
and associate it with Tracing even if the SkyWalking Tracing plug-in is not
turned on, if Apache SkyWalking is already integrated downstream.
+
+
+
+The above Content is the log content, where the Apache APISIX metadata
configuration is used to collect request-related information. You can later
modify the Log Format to customize the log content by Plugin Metadata, please
refer to the [official
documentation.](https://apisix.apache.org/docs/apisix/plugins/skywalking-logger#metadata)
+
+#### How to Use
+
+When using this plugin, since the SkyWalking plugin is "not enabled" by
default, you need to manually modify the `plugins` section in the
`conf/default-apisix.yaml` file to enable the plugin.
+
+```yaml
+plugins:
+ ...
+ - error-log-logger
+ ...
+```
+
+Then you can use the SkyWalking Tracing plug-in to get the tracing data
directly, so you can verify that the Logging plug-in-related features are
enabled and working properly.
+
+##### Step 1: Create a route
+
+Next, create a route and bind the SkyWalking Tracing plugin and the SkyWalking
Logging plugin. More details of the plugin configuration can be found in the
[official Apache APISIX
documentation](https://apisix.apache.org/docs/apisix/plugins/skywalking-logger).
+
+```shell
+curl -X PUT 'http://192.168.0.108:9080/apisix/admin/routes/1001' \
+-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
+-H 'Content-Type: application/json' \
+-d '{
+ "uri": "/get",
+ "plugins": {
+ "skywalking": {
+ "sample_ratio": 1
+ },
+ "skywalking-logger": {
+ "endpoint_addr": "http://127.0.0.1:12800";
+ }
+ },
+ "upstream": {
+ "type": "roundrobin",
+ "nodes": {
+ "httpbin.org:80": 1
+ }
+ }
+}'
+```
+
+##### Step 2: Log Processing
+
+On the Apache SkyWalking side, you can use LAL (Logger Analysis Language)
scripts for log processing, such as Tag extraction, SkyWalking metadata
correction, and so on.
+
+The main purpose of Tag extraction here is to facilitate subsequent retrieval
and to add dependencies to the Metrics statistics. The following code can be
used to configure the SkyWalking LAL script to complete the Tag extraction. For
more information on how to use the SkyWalking LAL script, please refer to the
[official Apache SkyWalking
documentation](https://skywalking.apache.org/docs/main/latest/en/concepts-and-designs/lal/).
+
+```yaml
+# The default LAL script to save all logs, behaving like the versions before
8.5.0.
+rules:
+ - name: default
+ dsl: |
+ filter {
+ json {
+ abortOnFailure false
+ }
+
+ extractor {
+ tag routeId: parsed.route_id
+ tag upstream: parsed.upstream
+ tag clientIp: parsed.client_ip
+ tag latency: parsed.latency
+ }
+
+ sink {
+ }
+ }
+```
+
+After configuring the above LAL script in SkyWalking OAP Server the following
log will be displayed.
+
+
+
+Details of the expanded log are as follows.
+
+
+
+As you can see from the above, displaying `routeId`, `upstream` and `clientIp`
as key-value pairs is much easier than searching directly in the log body. This
is because the Tag format not only supports log display format and search, but
also generates information such as Metrics using MAL statistics.
+
+### SkyWalking Error Logger Plugin
+
+The error-log-logger plug-in now supports the SkyWalking log format, and you
can now use the http-error-log plug-in to quickly connect Apache APISIX error
logs to Apache SkyWalking. Currently, error logs do not have access to
SkyWalking Tracing Context information, and therefore cannot be directly
associated with SkyWalking Tracing.
+
+The main reason for the error log to be integrated into SkyWalking is to
centralize the Apache APISIX log data and to make it easier to view all
observable data within SkyWalking.
+
+
+
+#### How to Use
+
+Since the error-log-logger plugin is "not enabled" by default, you still need
to enable the plugin in the way mentioned above.
+
+```yaml
+plugins:
+ ...
+ - error-log-logger
+ ...
+```
+
+##### Step 1: Bind the route
+
+After enabling, you need to bind the plugin to routes or global rules. Here we
take "bind routes" as an example.
+
+```shell
+curl -X PUT
'http://192.168.0.108:9080/apisix/admin/plugin_metadata/error-log-logger' \
+-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
+-H 'Content-Type: application/json' \
+-d '{
+ "inactive_timeout": 10,
+ "level": "ERROR",
+ "skywalking": {
+ "endpoint_addr": "http://127.0.0.1:12800/v3/logs";
+ }
+}'
+```
+
+>Note that the `endpoint_addr` is the SkyWalking OAP Server address and needs
to have the URI (i.e. `/v3/logs`).
+
+##### Step 2: LAL Processing
+
+In much the same way as the Access Log processing, the logs are also processed
by LAL when they reach SkyWalking OAP Server. Therefore, we can still use the
SkyWalking LAL script to analyze and process the log messages.
+
+It is important to note that the Error Log message body is in text format. If
you are extracting tags, you will need to use regular expressions to do this.
Unlike Access Log, which handles the message body in a slightly different way,
Acces Log uses JSON format and can directly reference the fields of the JSON
object using JSON parsing, but the rest of the process is largely the same.
+
+Tags can also be used to optimize the display and retrieval for subsequent
metrics calculations using SkyWalking MAL.
+
+```json
+rules:
+ - name: apisix-errlog
+ dsl: |
+ filter {
+ text {
+ regexp "(?<datetime>\\d{4}/\\d{2}/\\d{2} \\d{2}:\\d{2}:\\d{2})
\\[(?<level>\\w+)\\] \\d+\\#\\d+:( \\*\\d+ \\[(?<module>\\w+)\\]
(?<position>.*\\.lua:\\d+): (?<function>\\w+\\(\\)):)* (?<msg>.+)"
+ }
+ extractor {
+ tag level: parsed.level
+ if (parsed?.module) {
+ tag module: parsed.module
+ tag position: parsed.position
+ tag function: parsed.function
+ }
+ }
+ sink {
+ }
+ }
+```
+
+After the LAL script used by SkyWalking OAP Server, some of the Tags will be
extracted from the logs, as shown below.
+
+
+
+## Summary
+
+This article introduces two logging plug-ins for Apache APISIX that integrate
with SkyWalking to provide a more convenient operation and environment for
logging in Apache APISIX afterwards.
+
+We hope that through this article, you will have a fuller understanding of the
new features and be able to use Apache APISIX for centralized management of
observable data more conveniently in the future.
diff --git
a/website/i18n/zh/docusaurus-plugin-content-blog/2021/12/07/apisix-integrate-skywalking-plugin.md
b/website/i18n/zh/docusaurus-plugin-content-blog/2021/12/07/apisix-integrate-skywalking-plugin.md
new file mode 100644
index 0000000..3bf5374
--- /dev/null
+++
b/website/i18n/zh/docusaurus-plugin-content-blog/2021/12/07/apisix-integrate-skywalking-plugin.md
@@ -0,0 +1,196 @@
+---
+title: "强强联合！APISIX 集成 SkyWalking 打造全方位日志处理"
+author: "庄浩潮"
+authorURL: "https://github.com/dmsolr";
+authorImageURL: "https://avatars.githubusercontent.com/u/29735230?v=4";
+keywords:
+- Apache APISIX
+- Apache SkyWalking
+- 日志处理
+- 插件集成
+description: 本文主要介绍了两款 Apache APISIX 集成 SkyWalking 的日志插件，为之后大家在 Apache APISIX
中进行日志处理提供更方便的操作与环境。
+tags: [Technology]
+---
+
+> 本文主要介绍了两款 Apache APISIX 集成 SkyWalking 的日志插件，为之后大家在 Apache APISIX
中进行日志处理提供更方便的操作与环境。
+
+<!--truncate-->
+
+在可观测性领域，通常采用 Metrics、Logger 与 Tracing 三大方向的数据收集与分析，以达到洞察应用运行状态的目的，Apache
SkyWalking 的日志处理正好具备了以上三方面。
+
+Apache APISIX 早在 1.4 版本就已经集成 Apache SkyWaling Tracing
能力，并在后续版本中加入了错误日志和访问日志收集等功能。如今随着 Apache SkyWalking 对 Metrics 的支持，能够帮助 Apache
APISIX 在集成模式下实现一站式可观测方案，同时覆盖到日志、度量和调用追踪。
+
+## 功能开发背景
+
+熟悉 Apache APISIX 的小伙伴应该知道，Apache APISIX 在运行中会产出两种日志，即访问日志和错误日志。
+
+访问日志记录着每个请求的详细信息，属于**请求范围**内产生的日志，因此可以直接与 Tracing 关联。而错误日志则是 Apache APISIX
**运行时**产出日志信息，是整个应用级别日志，但无法确保能百分百关联到请求上。
+
+目前 Apache APISIX 提供了非常丰富的日志处理插件，包括 TCP/HTTP/Kafka 等收集上报插件，但它们与 Tracing
关联都比较弱。以 Apache SkyWalking 为例，提取 Apache APISIX 端日志记录中的 SkyWalking Tracing
Conetxt Header 并输出到文件系统，之后利用日志处理框架（fluentbit）将日志转成 SkyWalking 可接受的日志格式。后续从中解析提取
Tracing Context，从而获得 Tracing ID 进而与 Trace 建立联系。
+
+显然，上述方式处理流程比较繁琐复杂，还需要额外转换日志格式。为此，在
[PR#5500](https://github.com/apache/apisix/pull/5550) 中我们实现了将 Apache SkyWalking
访问日志接入 Apache APISIX 插件生态，方便用户在使用 Apache APISIX 中更方便地利用 Apache SkyWalking
进行收集和处理相关日志。
+
+## 全新日志插件介绍
+
+### SkyWalking Logger 插件
+
+SkyWalking Logger 插件能够解析 SkyWalking Tracing Context Header，并将相关 Tracing
Context 信息打印到日志中，从而实现日志与调用链建立关联。
+
+通过使用此插件，可以实现在下游已经集成 Apache SkyWalking 的情况下，Apache APISIX 即便没有打开 SkyWalking
Tracing 插件也能获取到 SkyWalking Tracing Context 并实现与 Tracing 关联。
+
+
+
+>上图 Content 为日志内容，这里使用了 Apache APISIX metadata 配置来收集 Request 相关信息。后续可通过 Plugin
Metadata 修改 Log Format
定制日志内容，具体可以参考[官方文档](https://apisix.apache.org/docs/apisix/plugins/skywalking-logger#metadata)。
+
+#### 使用方法
+
+在使用该插件时，由于 SkyWalking 插件默认为「不开启」，所以需要手动修改 `conf/default-apisix.yaml` 文件中
`plugins` 章节来解开SkyWalking 注释从而启用插件。
+
+```yaml
+plugins:
+ ...
+ - skywalking
+ ...
+```
+
+之后便可使用 SkyWalking Tracing 插件直接得到 Tracing 数据，以便接下来验证 Logging 插件相关功能是否能够正常开启与工作。
+
+##### 步骤一：创建路由
+
+接下来创建一个路由，并绑定 SkyWalking Tracing 插件和 SkyWalking Logging 插件。关于插件具体配置细节可以参考
[Apache APISIX
官方文档](https://apisix.apache.org/docs/apisix/plugins/skywalking-logger)，这里不再赘述。
+
+```shell
+curl -X PUT 'http://192.168.0.108:9080/apisix/admin/routes/1001' \
+-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
+-H 'Content-Type: application/json' \
+-d '{
+ "uri": "/get",
+ "plugins": {
+ "skywalking": {
+ "sample_ratio": 1
+ },
+ "skywalking-logger": {
+ "endpoint_addr": "http://127.0.0.1:12800";
+ }
+ },
+ "upstream": {
+ "type": "roundrobin",
+ "nodes": {
+ "httpbin.org:80": 1
+ }
+ }
+}'
+```
+
+##### 步骤二：日志处理
+
+在 Apache SkyWalking 侧，可以使用 LAL（Logger Analysis Language）脚本进行日志处理，比如 Tag
提取、SkyWalking 元数据修正等。
+
+这里进行 Tag 提取主要是为了后续检索方便，以及在 Metrics 统计时添加相关依赖。可使用如下代码进行 SkyWalking LAL 脚本配置来完成
Tag 提取，更多关于 SkyWalking LAL 脚本使用方法可以参考 [Apache SkyWalking
官方文档](https://skywalking.apache.org/docs/main/latest/en/concepts-and-designs/lal/)。
+
+```yaml
+# The default LAL script to save all logs, behaving like the versions before
8.5.0.
+rules:
+ - name: default
+ dsl: |
+ filter {
+ json {
+ abortOnFailure false
+ }
+
+ extractor {
+ tag routeId: parsed.route_id
+ tag upstream: parsed.upstream
+ tag clientIp: parsed.client_ip
+ tag latency: parsed.latency
+ }
+
+ sink {
+ }
+ }
+```
+
+在 SkyWalking OAP Server 配置上述 LAL 脚本后将显示如下日志：
+
+
+
+展开日志详情如下：
+
+
+
+通过上述操作可以看到，将 `routeId`，`upstream` 和 `clientIp` 以键值对形式展示，比在日志正文中直接查找要方便许多，还能用于
MAL 统计生成相关的度量指标。
+
+### SkyWalking Error Logger 插件
+
+目前 error-log-logger 插件已支持 SkyWalking 日志格式，现在可以使用 error-log-logger 插件将 Apache
APISIX 错误日志快捷接入到 Apache SkyWalking 中。目前，错误日志还无法获取 SkyWalking Tracing Context
信息，因此无法直接与 SkyWalking Tracing 产生关联。
+
+关于 Error Log 接入 SkyWalking 主要是为了能集中管理 Apache APISIX 的日志数据，同时也方便在 SkyWalking
内查看所有可观测性数据。
+
+
+
+#### 使用方法
+
+由于 error-log-logger 插件默认「不开启」，所以仍需要上文中提到的方法进行插件开启。
+
+```yaml
+plugins:
+ ...
+ - error-log-logger
+ ...
+```
+
+##### 步骤一：绑定路由
+
+启用之后，需要将插件绑定到路由或者 `global rules` 上，这里我们以「绑定路由」为例。
+
+```shell
+curl -X PUT
'http://192.168.0.108:9080/apisix/admin/plugin_metadata/error-log-logger' \
+-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
+-H 'Content-Type: application/json' \
+-d '{
+ "inactive_timeout": 10,
+ "level": "ERROR",
+ "skywalking": {
+ "endpoint_addr": "http://127.0.0.1:12800/v3/logs";
+ }
+}'
+```
+
+>注意这里 `endpoint_addr` 是 SkyWalking OAP Server 地址，需要带上 URI（即 `/v3/logs`)。
+
+##### 步骤二：LAL 处理
+
+与 Access Log 处理方式基本一样，日志在送达 SkyWalking OAP Server 时，同样会经过 LAL 处理。因此，我们依然可以使用
SkyWalking LAL 脚本来分析处理日志信息。
+
+需要注意的是，Error Log 日志消息体使用文本格式。如果进行 Tags 提取，则需要使用正则表达式来完成。与 Access Log
处理消息正文的方式略有不同，Acces Log 使用 JSON 格式，可以直接使用 JSON 解析后引用 JSON 对象的字段，其他处理流程则大体一致。
+
+同时也可以利用 Tags 来优化显示效果与检索，方便后续使用 SkyWalking MAL 进行 Metrics 计算。
+
+```json
+rules:
+ - name: apisix-errlog
+ dsl: |
+ filter {
+ text {
+ regexp "(?<datetime>\\d{4}/\\d{2}/\\d{2} \\d{2}:\\d{2}:\\d{2})
\\[(?<level>\\w+)\\] \\d+\\#\\d+:( \\*\\d+ \\[(?<module>\\w+)\\]
(?<position>.*\\.lua:\\d+): (?<function>\\w+\\(\\)):)* (?<msg>.+)"
+ }
+ extractor {
+ tag level: parsed.level
+ if (parsed?.module) {
+ tag module: parsed.module
+ tag position: parsed.position
+ tag function: parsed.function
+ }
+ }
+ sink {
+ }
+ }
+```
+
+在 SkyWalking OAP Server 使用的 LAL 脚本之后，将会在日志中提取部分 Tags，效果如下图。
+
+
+
+## 总结
+
+本文主要介绍了两款 Apache APISIX 集成 SkyWalking 的日志插件，为之后大家在 Apache APISIX
中进行日志处理提供更方便的操作与环境。希望通过本篇内容，大家可以对新功能有了更充分的理解，后续可以更方便地利用 Apache APISIX
进行可观测数据的集中管理。
