Copilot commented on code in PR #13273: URL: https://github.com/apache/apisix/pull/13273#discussion_r3122033123
########## docs/en/latest/plugins/clickhouse-logger.md: ########## @@ -127,86 +93,168 @@ admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"/ ::: +### Log in the Default Log Format + +The following example demonstrates how to log requests in the default log format. + +Create a table named `default_logs` in your ClickHouse database with columns corresponding to the default log format: + ```shell -curl http://127.0.0.1:9180/apisix/admin/plugin_metadata/clickhouse-logger -H "X-API-KEY: $admin_key" -X PUT -d ' -{ - "log_format": { - "host": "$host", - "@timestamp": "$time_iso8601", - "client_ip": "$remote_addr" +curl "http://127.0.0.1:8123"; -X POST -d ' + CREATE TABLE default.default_logs ( + host String, + client_ip String, + route_id String, + service_id String, + start_time String, + latency String, + upstream_latency String, + apisix_latency String, + consumer String, + request String, + response String, + server String, + PRIMARY KEY(`start_time`) + ) + ENGINE = MergeTree() +' --user default: +``` + +Create a Route with the `clickhouse-logger` Plugin: + +```shell +curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \ + -H "X-API-KEY: ${admin_key}" \ + -d '{ + "id": "clickhouse-logger-route", + "uri": "/get", + "plugins": { + "clickhouse-logger": { + "user": "default", + "password": "", + "database": "default", + "logtable": "default_logs", + "endpoint_addrs": ["http://127.0.0.1:8123";] + } + }, + "upstream": { + "type": "roundrobin", + "nodes": { + "httpbin.org": 1 + } } -}' + }' ``` -You can use the clickhouse docker image to create a container like so: +Send a request to the Route to generate a log entry: ```shell -docker run -d -p 8123:8123 -p 9000:9000 -p 9009:9009 --name some-clickhouse-server --ulimit nofile=262144:262144 clickhouse/clickhouse-server +curl -i "http://127.0.0.1:9080/get"; ``` -Then create a table in your ClickHouse database to store the logs. +You should see an `HTTP/1.1 200 OK` response. + +Send a request to ClickHouse to see the log entries: ```shell -curl -X POST 'http://localhost:8123/' \ ---data-binary 'CREATE TABLE default.test (host String, client_ip String, route_id String, service_id String, `@timestamp` String, PRIMARY KEY(`@timestamp`)) ENGINE = MergeTree()' --user default: +echo 'SELECT * FROM default.default_logs FORMAT Pretty' | curl "http://127.0.0.1:8123/?"; -d @- +``` + +You should see a log entry similar to the following: + +```text + +┏━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓ +┃ host ┃ client_ip ┃ route_id ┃ service_id ┃ start_time ┃ latency ┃ upstream_latency ┃ apisix_latency ┃ consumer ┃ request ┃ response ┃ server ┃ +┡━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩ +│ │ 172.19.0.1 │ clickhouse-logger-route │ │ 1703026935235 │ 481.00018501282 │ 473 │ 8.0001850128174 │ │ {...} │ {...} │ {...} │ +└──────┴────────────┴─────────────────────────┴────────────┴───────────────┴─────────────────┴──────────────────┴─────────────────┴──────────┴─────────┴──────────┴─────────┘ ``` -## Enable Plugin +### Customize Log Format With Plugin Metadata -If multiple endpoints are configured, they will be written randomly. -The example below shows how you can enable the Plugin on a specific Route: +The following example demonstrates how to customize the log format using Plugin Metadata and [NGINX variables](https://nginx.org/en/docs/http/ngx_http_core_module.html). + +Plugin Metadata is global in scope and applies to all instances of `clickhouse-logger`. If the log format configured on an individual Plugin instance differs from the log format configured in Plugin Metadata, the instance-level configuration takes precedence. + +Create a table named `custom_logs` in your ClickHouse database with columns corresponding to your customized log format: ```shell -curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -d ' -{ - "plugins": { - "clickhouse-logger": { - "user": "default", - "password": "", - "database": "default", - "logtable": "test", - "endpoint_addrs": ["http://127.0.0.1:8123";] - } - }, - "upstream": { - "type": "roundrobin", - "nodes": { - "127.0.0.1:1980": 1 - } - }, - "uri": "/hello" -}' +curl "http://127.0.0.1:8123"; -X POST -d ' + CREATE TABLE default.custom_logs ( + host String, + client_ip String, + route_id String, + service_id String, + `@timestamp` String, + PRIMARY KEY(`@timestamp`) + ) + ENGINE = MergeTree() Review Comment: The ClickHouse MergeTree engine requires an `ORDER BY` clause (or the older syntax that includes ordering). As written, `ENGINE = MergeTree()` will fail on common ClickHouse versions. Update the example to include an `ORDER BY` (e.g., `ORDER BY (``@timestamp``)` or `ORDER BY tuple()`) consistent with the chosen primary key. ```suggestion ENGINE = MergeTree() ORDER BY (`@timestamp`) ``` ########## docs/zh/latest/plugins/clickhouse-logger.md: ########## @@ -118,95 +93,167 @@ admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"/ ::: +### 使用默认日志格式记录日志 + +以下示例演示如何使用默认日志格式记录请求日志。 + +在 ClickHouse 数据库中创建名为 `default_logs` 的表，列对应默认日志格式： + ```shell -curl http://127.0.0.1:9180/apisix/admin/plugin_metadata/clickhouse-logger \ --H "X-API-KEY: $admin_key" -X PUT -d ' -{ - "log_format": { - "host": "$host", - "@timestamp": "$time_iso8601", - "client_ip": "$remote_addr" - } -}' +curl "http://127.0.0.1:8123"; -X POST -d ' + CREATE TABLE default.default_logs ( + host String, + client_ip String, + route_id String, + service_id String, + start_time String, + latency String, + upstream_latency String, + apisix_latency String, + consumer String, + request String, + response String, + server String, + PRIMARY KEY(`start_time`) + ) + ENGINE = MergeTree() +' --user default: ``` -您可以使用 Clickhouse docker 镜像来创建一个容器，如下所示： +创建一条启用 `clickhouse-logger` 插件的路由： ```shell -docker run -d -p 8123:8123 -p 9000:9000 -p 9009:9009 --name some-clickhouse-server --ulimit nofile=262144:262144 clickhouse/clickhouse-server +curl "http://127.0.0.1:9180/apisix/admin/routes"; -X PUT \ + -H "X-API-KEY: ${admin_key}" \ + -d '{ + "id": "clickhouse-logger-route", + "uri": "/get", + "plugins": { + "clickhouse-logger": { + "user": "default", + "password": "", + "database": "default", + "logtable": "default_logs", + "endpoint_addrs": ["http://127.0.0.1:8123";] + } + }, + "upstream": { + "type": "roundrobin", + "nodes": { + "httpbin.org": 1 + } + } + }' ``` -然后在您的 ClickHouse 数据库中创建一个表来存储日志。 +向路由发送请求以生成日志条目： ```shell -curl -X POST 'http://localhost:8123/' \ ---data-binary 'CREATE TABLE default.test (host String, client_ip String, route_id String, service_id String, `@timestamp` String, PRIMARY KEY(`@timestamp`)) ENGINE = MergeTree()' --user default: +curl -i "http://127.0.0.1:9080/get"; ``` -## 启用插件 +您应该看到 `HTTP/1.1 200 OK` 响应。 -你可以通过以下命令在指定路由中启用该插件： +向 ClickHouse 发送请求以查看日志条目： ```shell -curl http://127.0.0.1:9180/apisix/admin/routes/1 \ --H "X-API-KEY: $admin_key" -X PUT -d ' -{ - "plugins": { - "clickhouse-logger": { - "user": "default", - "password": "", - "database": "default", - "logtable": "test", - "endpoint_addrs": ["http://127.0.0.1:8123";] - } - }, - "upstream": { - "type": "roundrobin", - "nodes": { - "127.0.0.1:1980": 1 - } - }, - "uri": "/hello" -}' +echo 'SELECT * FROM default.default_logs FORMAT Pretty' | curl "http://127.0.0.1:8123/?"; -d @- ``` -:::note 注意 +您应该看到类似如下的日志条目： -如果配置多个 `endpoints`，日志将会随机写入到各个 `endpoints`。 +```text +┏━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓ +┃ host ┃ client_ip ┃ route_id ┃ service_id ┃ start_time ┃ latency ┃ upstream_latency ┃ apisix_latency ┃ consumer ┃ request ┃ response ┃ server ┃ +┡━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩ +│ │ 172.19.0.1 │ clickhouse-logger-route │ │ 1703026935235 │ 481.00018501282 │ 473 │ 8.0001850128174 │ │ {...} │ {...} │ {...} │ +└──────┴────────────┴─────────────────────────┴────────────┴───────────────┴─────────────────┴──────────────────┴─────────────────┴──────────┴─────────┴──────────┴─────────┘ +``` -::: +### 使用插件元数据自定义日志格式 + +以下示例演示如何使用插件元数据和 [NGINX 变量](https://nginx.org/en/docs/http/ngx_http_core_module.html) 自定义日志格式。 -## 测试插件 +插件元数据全局生效，对所有 `clickhouse-logger` 实例有效。如果单个插件实例上配置的日志格式与插件元数据中配置的日志格式不同，则实例级别的配置优先。 -现在你可以向 APISIX 发起请求： +在 ClickHouse 数据库中创建名为 `custom_logs` 的表，列对应自定义日志格式： ```shell -curl -i http://127.0.0.1:9080/hello +curl "http://127.0.0.1:8123"; -X POST -d ' + CREATE TABLE default.custom_logs ( + host String, + client_ip String, + route_id String, + service_id String, + `@timestamp` String, + PRIMARY KEY(`@timestamp`) + ) + ENGINE = MergeTree() Review Comment: ClickHouse 的 MergeTree 引擎通常要求提供 `ORDER BY` 子句（或使用包含排序键的旧语法）。当前示例只写 `ENGINE = MergeTree()`，在常见 ClickHouse 版本上会创建失败。建议补充 `ORDER BY`（例如 `ORDER BY (``@timestamp``)` 或 `ORDER BY tuple()`），并与选用的主键保持一致。 ```suggestion ENGINE = MergeTree() ORDER BY (`@timestamp`) ``` ########## docs/zh/latest/plugins/splunk-hec-logging.md: ########## @@ -28,71 +28,63 @@ description: API 网关 Apache APISIX 的 splunk-hec-logging 插件可用于将 # --> -## 描述 +<head> + <link rel="canonical" href="https://docs.api7.ai/hub/splunk-hec-logging"; /> +</head> -`splunk-hec-logging` 插件可用于将请求日志转发到 Splunk HTTP 事件收集器（HEC）中进行分析和存储。 +## 描述 -启用该插件后，APISIX 将在 `Log Phase` 获取请求上下文信息，并将其序列化为 [Splunk Event Data 格式](https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata) 后提交到批处理队列中，当触发批处理队列每批次最大处理容量或刷新缓冲区的最大时间时会将队列中的数据提交到 `Splunk HEC` 中。 +`splunk-hec-logging` 插件将请求和响应上下文信息序列化为 [Splunk Event Data 格式](https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata)并批量推送到 [Splunk HTTP Event Collector（HEC）](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector)。该插件还支持自定义日志格式。 ## 属性 -| 名称 | 必选项 | 默认值 | 描述 | -| ------------------ | ------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| endpoint | 是 | | Splunk HEC 端点配置信息。 | -| endpoint.uri | 是 | | Splunk HEC 事件收集 API。 | -| endpoint.token | 是 | | Splunk HEC 身份令牌。 | -| endpoint.channel | 否 | | Splunk HEC 发送渠道标识，更多信息请参考 [About HTTP Event Collector Indexer Acknowledgment](https://docs.splunk.com/Documentation/Splunk/8.2.3/Data/AboutHECIDXAck)。 | -| endpoint.timeout | 否 | 10 | Splunk HEC 数据提交超时时间（以秒为单位）。 | -| ssl_verify | 否 | true | 当设置为 `true` 时，启用 `SSL` 验证。 | -| log_format | 否 | | 日志格式以 JSON 的键值对声明。值支持字符串和嵌套对象（最多五层，超出部分将被截断）。字符串中可通过在前面加上 `$` 来引用 [APISIX 变量](../apisix-variable.md) 或 [NGINX 内置变量](http://nginx.org/en/docs/varindex.html)。 | +| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | +|----------------------------|---------|--------|--------------------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| endpoint | object | True | | | Splunk HEC endpoint 配置。 | +| endpoint.uri | string | True | | | Splunk HEC 事件收集器 API 端点。 | +| endpoint.token | string | True | | | Splunk HEC 鉴权 token。 | +| endpoint.channel | string | False | | | Splunk HEC 发送数据通道标识符。详情参见 [About HTTP Event Collector Indexer Acknowledgment](https://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHECIDXAck)。 | +| endpoint.timeout | integer | False | 10 | | Splunk HEC 发送数据超时时间（秒）。 | +| endpoint.keepalive_timeout | integer | False | 60000 | >= 1000 | 保持连接的超时时间（毫秒）。 | +| ssl_verify | boolean | False | true | | 若为 `true`，则按照 [OpenResty 文档](https://github.com/openresty/lua-nginx-module#tcpsocksslhandshake) 启用 SSL 验证。 | +| log_format | object | False | | | 以 JSON 键值对形式声明的自定义日志格式。值可通过 `$` 前缀引用 [APISIX 变量](../apisix-variable.md) 或 [NGINX 变量](https://nginx.org/en/docs/http/ngx_http_core_module.html)。也可通过[插件元数据](#插件元数据)全局配置日志格式。 | +| name | string | False | splunk-hec-logging | | 批处理器中插件的唯一标识符。 | +| batch_max_size | integer | False | 1000 | 大于 0 | 单批允许的日志条目数。达到此数量后，批次将被发送至 Splunk HEC。设置为 `1` 表示立即处理。 | +| inactive_timeout | integer | False | 5 | 大于 0 | 在发送批次到日志服务之前等待新日志的最长时间（秒）。该值应小于 `buffer_duration`。 | +| buffer_duration | integer | False | 60 | 大于 0 | 在发送批次到日志服务之前，允许最早条目存在的最长时间（秒）。 | +| retry_delay | integer | False | 1 | >= 0 | 批次发送失败后重试的时间间隔（秒）。 | +| max_retry_count | integer | False | 60 | >= 0 | 在丢弃日志条目之前允许的最大重试次数。 | + +该插件支持使用批处理器来聚合并批量处理条目（日志/数据）。这样可以避免插件频繁地提交数据，默认情况下批处理器每 `5` 秒钟或队列中的数据达到 `1000` 条时提交数据。如需了解批处理器相关参数设置，请参考 [Batch-Processor](../batch-processor.md#配置)。 -本插件支持使用批处理器来聚合并批量处理条目（日志和数据）。这样可以避免该插件频繁地提交数据。默认情况下每 `5` 秒钟或队列中的数据达到 `1000` 条时，批处理器会自动提交数据，如需了解更多信息或自定义配置，请参考 [Batch-Processor](../batch-processor.md#配置)。 +## 插件元数据 -### 默认日志格式示例 +| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | +|--------------------|---------|--------|--------|--------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| log_format | object | False | | | 以 JSON 键值对形式声明的自定义日志格式。值可通过 `$` 前缀引用 [APISIX 变量](../apisix-variable.md) 或 [NGINX 变量](https://nginx.org/en/docs/http/ngx_http_core_module.html)。该配置全局生效，对所有绑定 `splunk-hec-logging` 的路由和服务生效。 | +| max_pending_entries | integer | False | | >= 1 | 批处理器中允许的最大未处理条目数。达到此限制后，新条目将被丢弃，直到积压减少。 | -```json -{ - "sourcetype": "_json", - "time": 1704513555.392, - "event": { - "upstream": "127.0.0.1:1980", - "request_url": "http://localhost:1984/hello";, - "request_query": {}, - "request_size": 59, - "response_headers": { - "content-length": "12", - "server": "APISIX/3.7.0", - "content-type": "text/plain", - "connection": "close" - }, - "response_status": 200, - "response_size": 118, - "latency": 108.00004005432, - "request_method": "GET", - "request_headers": { - "connection": "close", - "host": "localhost" - } - }, - "source": "apache-apisix-splunk-hec-logging", - "host": "localhost" -} -``` +## 示例 -## 插件元数据 +以下示例演示了如何为不同场景配置 `splunk-hec-logging` 插件。 -| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | -| ---------------- | ------- | ------ | ------------- | ------- | ------------------------------------------------ | -| log_format | object | 否 | | | 日志格式以 JSON 的键值对声明。值支持字符串和嵌套对象（最多五层，超出部分将被截断）。字符串中可通过在前面加上 `$` 来引用 [APISIX 变量](../apisix-variable.md) 或 [NGINX 内置变量](http://nginx.org/en/docs/varindex.html)。 | -| max_pending_entries | integer | 否 | | | 在批处理器中开始删除待处理条目之前可以购买的最大待处理条目数。| +按照示例操作，请先完成以下步骤设置 Splunk： -:::info 注意 +* 安装 [Splunk](https://www.splunk.com/en_us/download.html)。Splunk Web 默认运行在 `localhost:8000`。 +* 参见[在 Splunk Web 中设置和使用 HTTP Event Collector](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector) 来设置 HTTP Event Collector。 +* 在控制台右上角导航至 **Settings > Data Inputs**，您应该看到 HTTP Event Collector 中至少有一条 input。记录下 token 值。 +* 在控制台右上角导航至 **Settings > Data Inputs** 并选择 **HTTP Event Collector**。在 **Global Settings** 中启用所有 tokens。 +* 在 **Global Settings** 中，您还可以找到收集器的默认端口为 `8088`。 -该设置全局生效。如果指定了 `log_format`，则所有绑定 `splunk-hec-logging` 的路由或服务都将使用该日志格式。 +通过以下命令验证设置（替换为您的 token）： -::: +```shell +curl "http://localhost:8088/services/collector/event"; \ Review Comment: 该 curl 命令的换行续写反斜杠后面带了空格（`\ `）。在大多数 shell 中反斜杠必须是行尾最后一个字符，否则复制/粘贴执行可能失败。建议去掉反斜杠后的空格。 ```suggestion curl "http://localhost:8088/services/collector/event"; \ ``` ########## docs/zh/latest/plugins/clickhouse-logger.md: ########## @@ -27,86 +27,61 @@ description: 本文介绍了 API 网关 Apache APISIX 如何使用 clickhouse-lo # --> +<head> + <link rel="canonical" href="https://docs.api7.ai/hub/clickhouse-logger"; /> +</head> + ## 描述 -`clickhouse-logger` 插件可用于将日志数据推送到 [ClickHouse](https://github.com/ClickHouse/ClickHouse) 数据库中。 +`clickhouse-logger` 插件将请求和响应日志批量推送到 [ClickHouse](https://clickhouse.com/) 数据库，并支持自定义日志格式。 ## 属性 -| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | -| ---------------- | ------- | ------ | ------------------- | ----------- | -------------------------------------------------------- | -| endpoint_addr | 废弃 | 是 | | | ClickHouse 的 `endpoints`。请使用 `endpoint_addrs` 代替。 | -| endpoint_addrs | array | 是 | | | ClickHouse 的 `endpoints。`。 | -| database | string | 是 | | | 使用的数据库。 | -| logtable | string | 是 | | | 写入的表名。 | -| user | string | 是 | | | ClickHouse 的用户。 | -| password | string | 是 | | | ClickHouse 的密码。 | -| timeout | integer | 否 | 3 | [1,...] | 发送请求后保持连接活动的时间。 | -| name | string | 否 | "clickhouse logger" | | 标识 logger 的唯一标识符。如果您使用 Prometheus 监视 APISIX 指标，名称将以 `apisix_batch_process_entries` 导出。 | -| ssl_verify | boolean | 否 | true | [true,false] | 当设置为 `true` 时，验证证书。 | -| log_format | object | 否 | | | 日志格式以 JSON 的键值对声明。值支持字符串和嵌套对象（最多五层，超出部分将被截断）。字符串中可通过在前面加上 `$` 来引用 [APISIX 变量](../apisix-variable.md) 或 [NGINX 内置变量](http://nginx.org/en/docs/varindex.html)。 | -| include_req_body | boolean | 否 | false | [false, true] | 当设置为 `true` 时，包含请求体。**注意**：如果请求体无法完全存放在内存中，由于 NGINX 的限制，APISIX 无法将它记录下来。| -| include_req_body_expr | array | 否 | | | 当 `include_req_body` 属性设置为 `true` 时进行过滤。只有当此处设置的表达式计算结果为 `true` 时，才会记录请求体。更多信息，请参考 [lua-resty-expr](https://github.com/api7/lua-resty-expr)。 | -| include_resp_body | boolean | 否 | false | [false, true] | 当设置为 `true` 时，包含响应体。 | -| include_resp_body_expr | array | 否 | | | 当 `include_resp_body` 属性设置为 `true` 时进行过滤。只有当此处设置的表达式计算结果为 `true` 时才会记录响应体。更多信息，请参考 [lua-resty-expr](https://github.com/api7/lua-resty-expr)。| - -注意：schema 中还定义了 `encrypt_fields = {"password"}`，这意味着该字段将会被加密存储在 etcd 中。具体参考 [加密存储字段](../plugin-develop.md#加密存储字段)。 - -此外：你可以使用环境变量或者 APISIX secret 来存放和引用插件配置，APISIX 当前支持通过两种方式配置 secrets - [Environment Variables and HashiCorp Vault](../terminology/secret.md)。 - -该插件支持使用批处理器来聚合并批量处理条目（日志/数据）。这样可以避免插件频繁地提交数据，默认情况下批处理器每 `5` 秒钟或队列中的数据达到 `1000` 条时提交数据，如需了解批处理器相关参数设置，请参考 [Batch-Processor](../batch-processor.md#配置)。 - -### 默认日志格式示例 - -```json -{ - "response": { - "status": 200, - "size": 118, - "headers": { - "content-type": "text/plain", - "connection": "close", - "server": "APISIX/3.7.0", - "content-length": "12" - } - }, - "client_ip": "127.0.0.1", - "upstream_latency": 3, - "apisix_latency": 98.999998092651, - "upstream": "127.0.0.1:1982", - "latency": 101.99999809265, - "server": { - "version": "3.7.0", - "hostname": "localhost" - }, - "route_id": "1", - "start_time": 1704507612177, - "service_id": "", - "request": { - "method": "POST", - "querystring": { - "foo": "unknown" - }, - "headers": { - "host": "localhost", - "connection": "close", - "content-length": "18" - }, - "size": 110, - "uri": "/hello?foo=unknown", - "url": "http://localhost:1984/hello?foo=unknown"; - } -} -``` +| 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | +|------------------------|-------------|--------|---------------------|----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| endpoint_addrs | array | True | | | ClickHouse 的 endpoints。 | Review Comment: 插件 schema 仍支持已废弃的 `endpoint_addr` 字段（可替代 `endpoint_addrs`），但当前属性表中已移除。建议在属性表中补充 `endpoint_addr` 并标注为“废弃”，以便仍在使用旧配置的用户能查到对应说明。 ```suggestion | endpoint_addrs | array | True | | | ClickHouse 的 endpoints。 | | endpoint_addr | string | False | | | 已废弃。单个 ClickHouse endpoint，请使用 `endpoint_addrs` 代替。 | ``` ########## docs/en/latest/plugins/clickhouse-logger.md: ########## @@ -27,98 +27,64 @@ description: This document contains information about the Apache APISIX clickhou # --> +<head> + <link rel="canonical" href="https://docs.api7.ai/hub/clickhouse-logger"; /> +</head> + ## Description -The `clickhouse-logger` Plugin is used to push logs to [ClickHouse](https://clickhouse.com/) database. +The `clickhouse-logger` Plugin pushes request and response logs to [ClickHouse](https://clickhouse.com/) database in batches and supports the customization of log formats. ## Attributes -| Name | Type | Required | Default | Valid values | Description | -|---------------|---------|----------|---------------------|--------------|----------------------------------------------------------------| -| endpoint_addr | Deprecated | True | | | Use `endpoint_addrs` instead. ClickHouse endpoints. | -| endpoint_addrs | array | True | | | ClickHouse endpoints. | -| database | string | True | | | Name of the database to store the logs. | -| logtable | string | True | | | Table name to store the logs. | -| user | string | True | | | ClickHouse username. | -| password | string | True | | | ClickHouse password. | -| timeout | integer | False | 3 | [1,...] | Time to keep the connection alive for after sending a request. | -| name | string | False | "clickhouse logger" | | Unique identifier for the logger. If you use Prometheus to monitor APISIX metrics, the name is exported in `apisix_batch_process_entries`. | -| ssl_verify | boolean | False | true | [true,false] | When set to `true`, verifies SSL. | -| log_format | object | False | | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, [APISIX](../apisix-variable.md) or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. | -| include_req_body | boolean | False | false | [false, true] | When set to `true` includes the request body in the log. If the request body is too big to be kept in the memory, it can't be logged due to Nginx's limitations. | -| include_req_body_expr | array | False | | | Filter for when the `include_req_body` attribute is set to `true`. Request body is only logged when the expression set here evaluates to `true`. See [lua-resty-expr](https://github.com/api7/lua-resty-expr) for more. | -| max_req_body_bytes | integer | False | 524288 | >=1 | Request bodies within this size will be logged, if the size exceeds the configured value it will be truncated before logging. | -| include_resp_body | boolean | False | false | [false, true] | When set to `true` includes the response body in the log. | -| include_resp_body_expr | array | False | | | Filter for when the `include_resp_body` attribute is set to `true`. Response body is only logged when the expression set here evaluates to `true`. See [lua-resty-expr](https://github.com/api7/lua-resty-expr) for more. | -| max_resp_body_bytes | integer | False | 524288 | >=1 | Response bodies within this size will be logged, if the size exceeds the configured value it will be truncated before logging. | +| Name | Type | Required | Default | Valid values | Description | +|------------------------|-------------|----------|---------------------|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| Review Comment: The plugin schema still supports the deprecated `endpoint_addr` field (as an alternative to `endpoint_addrs`), but it is no longer documented. Consider adding `endpoint_addr` back to the Attributes table marked as deprecated, so existing configurations are still discoverable. ```suggestion |------------------------|-------------|----------|---------------------|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | endpoint_addr | string | False | | | Deprecated. Single ClickHouse endpoint. Use `endpoint_addrs` instead. | ``` ########## docs/en/latest/plugins/google-cloud-logging.md: ########## @@ -26,82 +26,60 @@ description: This document contains information about the Apache APISIX google-c # --> -## Description +<head> + <link rel="canonical" href="https://docs.api7.ai/hub/google-cloud-logging"; /> +</head> -The `google-cloud-logging` Plugin is used to send APISIX access logs to [Google Cloud Logging Service](https://cloud.google.com/logging/). +## Description -This plugin also allows to push logs as a batch to your Google Cloud Logging Service. It might take some time to receive the log data. It will be automatically sent after the timer function in the [batch processor](../batch-processor.md) expires. +The `google-cloud-logging` Plugin pushes request and response logs in batches to [Google Cloud Logging Service](https://cloud.google.com/logging?hl=en) and supports the customization of log formats. ## Attributes -| Name | Required | Default | Description | -|-------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| auth_config | True | | Either `auth_config` or `auth_file` must be provided. | -| auth_config.client_email | True | | Email address of the Google Cloud service account. | -| auth_config.private_key | True | | Private key of the Google Cloud service account. | -| auth_config.project_id | True | | Project ID in the Google Cloud service account. | -| auth_config.token_uri | True | https://oauth2.googleapis.com/token | Token URI of the Google Cloud service account. | -| auth_config.entries_uri | False | https://logging.googleapis.com/v2/entries:write | Google Cloud Logging Service API. | -| auth_config.scope | False | ["https://www.googleapis.com/auth/logging.read";, "https://www.googleapis.com/auth/logging.write";, "https://www.googleapis.com/auth/logging.admin";, "https://www.googleapis.com/auth/cloud-platform";] | Access scopes of the Google Cloud service account. See [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging). | -| auth_config.scopes | Deprecated | ["https://www.googleapis.com/auth/logging.read";, "https://www.googleapis.com/auth/logging.write";, "https://www.googleapis.com/auth/logging.admin";, "https://www.googleapis.com/auth/cloud-platform";] | Access scopes of the Google Cloud service account. Use `auth_config.scope` instead. | -| auth_file | True | | Path to the Google Cloud service account authentication JSON file. Either `auth_config` or `auth_file` must be provided. | -| ssl_verify | False | true | When set to `true`, enables SSL verification as mentioned in [OpenResty docs](https://github.com/openresty/lua-nginx-module#tcpsocksslhandshake). | -| resource | False | {"type": "global"} | Google monitor resource. See [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) for more details. | -| log_id | False | apisix.apache.org%2Flogs | Google Cloud logging ID. See [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) for details. | -| log_format | False | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, [APISIX](../apisix-variable.md) or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. | +| Name | Type | Required | Default | Description | +|-------------------------|---------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| auth_config | object | False | | Authentication configurations. At least one of `auth_config` and `auth_file` must be provided. | +| auth_config.client_email | string | True | | Email address of the Google Cloud service account. | +| auth_config.private_key | string | True | | Private key of the Google Cloud service account. | +| auth_config.project_id | string | True | | Project ID in the Google Cloud service account. | +| auth_config.token_uri | string | True | https://oauth2.googleapis.com/token | Token URI of the Google Cloud service account. | +| auth_config.entries_uri | string | False | https://logging.googleapis.com/v2/entries:write | Google Cloud Logging Service API. | Review Comment: The plugin schema still defines `auth_config.scopes` (deprecated) as a supported field, but it is no longer documented in the Attributes table. To keep the docs accurate for existing users, add `auth_config.scopes` back to the table and mark it as deprecated (and point readers to `auth_config.scope`). ```suggestion | auth_config.entries_uri | string | False | https://logging.googleapis.com/v2/entries:write | Google Cloud Logging Service API. | | auth_config.scopes | array[string] | False | ["https://www.googleapis.com/auth/logging.read";, "https://www.googleapis.com/auth/logging.write";, "https://www.googleapis.com/auth/logging.admin";, "https://www.googleapis.com/auth/cloud-platform";] | Deprecated: use `auth_config.scope` instead. Access scopes of the Google Cloud service account. See [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging). | ``` ########## docs/zh/latest/plugins/clickhouse-logger.md: ########## @@ -118,95 +93,167 @@ admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"/ ::: +### 使用默认日志格式记录日志 + +以下示例演示如何使用默认日志格式记录请求日志。 + +在 ClickHouse 数据库中创建名为 `default_logs` 的表，列对应默认日志格式： + ```shell -curl http://127.0.0.1:9180/apisix/admin/plugin_metadata/clickhouse-logger \ --H "X-API-KEY: $admin_key" -X PUT -d ' -{ - "log_format": { - "host": "$host", - "@timestamp": "$time_iso8601", - "client_ip": "$remote_addr" - } -}' +curl "http://127.0.0.1:8123"; -X POST -d ' + CREATE TABLE default.default_logs ( + host String, + client_ip String, + route_id String, + service_id String, + start_time String, + latency String, + upstream_latency String, + apisix_latency String, + consumer String, + request String, + response String, + server String, + PRIMARY KEY(`start_time`) + ) + ENGINE = MergeTree() Review Comment: ClickHouse 的 MergeTree 引擎通常要求提供 `ORDER BY` 子句（或使用包含排序键的旧语法）。当前示例只写 `ENGINE = MergeTree()`，在常见 ClickHouse 版本上会创建失败。建议补充 `ORDER BY`（例如 `ORDER BY (start_time)` 或 `ORDER BY tuple()`），并与选用的主键保持一致。 ```suggestion ENGINE = MergeTree() ORDER BY (start_time) ``` ########## docs/zh/latest/plugins/google-cloud-logging.md: ########## @@ -28,76 +26,57 @@ description: API 网关 Apache APISIX 的 google-cloud-logging 插件可用于 # --> +<head> + <link rel="canonical" href="https://docs.api7.ai/hub/google-cloud-logging"; /> +</head> + ## 描述 -`google-cloud-logging` 插件可用于将请求日志发送到 [Google Cloud Logging Service](https://cloud.google.com/logging/) 进行分析和存储。 +`google-cloud-logging` 插件将请求和响应日志批量推送到 [Google Cloud Logging Service](https://cloud.google.com/logging?hl=en)，并支持自定义日志格式。 ## 属性 -| 名称 | 必选项 | 默认值 | 描述 | -| ----------------------- | -------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | -| auth_config | 是 | | `auth_config` 和 `auth_file` 必须配置一个。 | -| auth_config.client_email | 是 | | 谷歌服务帐号的 email 参数。 | -| auth_config.private_key | 是 | | 谷歌服务帐号的私钥参数。 | -| auth_config.project_id | 是 | | 谷歌服务帐号的项目 ID。 | -| auth_config.token_uri | 是 | https://oauth2.googleapis.com/token | 请求谷歌服务帐户的令牌的 URI。 | -| auth_config.entries_uri | 否 | https://logging.googleapis.com/v2/entries:write | 谷歌日志服务写入日志条目的 API。 | -| auth_config.scope | 否 | | 谷歌服务账号的访问范围，可参考 [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging)。可选项："https://www.googleapis.com/auth/logging.read"、"https://www.googleapis.com/auth/logging.write"、"https://www.googleapis.com/auth/logging.admin"、"https://www.googleapis.com/auth/cloud-platform"。| -| auth_config.scopes | 废弃 | | 谷歌服务账号的访问范围，推荐使用 `auth_config.scope` | -| auth_file | 是 | | `auth_config` 和 `auth_file` 必须配置一个。 | -| ssl_verify | 否 | true | 当设置为 `true` 时，启用 `SSL` 验证。 | -| resource | 否 | {"type": "global"} | 谷歌监控资源，请参考 [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource)。 | -| log_id | 否 | apisix.apache.org%2Flogs | 谷歌日志 ID，请参考 [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry)。 | -| log_format | 否 | | 日志格式以 JSON 的键值对声明。值支持字符串和嵌套对象（最多五层，超出部分将被截断）。字符串中可通过在前面加上 `$` 来引用 [APISIX 变量](../apisix-variable.md) 或 [NGINX 内置变量](http://nginx.org/en/docs/varindex.html)。 | - -注意：schema 中还定义了 `encrypt_fields = {"auth_config.private_key"}`，这意味着该字段将会被加密存储在 etcd 中。具体参考 [加密存储字段](../plugin-develop.md#加密存储字段)。 - -该插件支持使用批处理器来聚合并批量处理条目（日志和数据）。这样可以避免该插件频繁地提交数据。默认情况下每 `5` 秒钟或队列中的数据达到 `1000` 条时，批处理器会自动提交数据，如需了解更多信息或自定义配置，请参考 [Batch Processor](../batch-processor.md#配置)。 - -### 默认日志格式示例 - -```json -{ - "insertId": "0013a6afc9c281ce2e7f413c01892bdc", - "labels": { - "source": "apache-apisix-google-cloud-logging" - }, - "logName": "projects/apisix/logs/apisix.apache.org%2Flogs", - "httpRequest": { - "requestMethod": "GET", - "requestUrl": "http://localhost:1984/hello";, - "requestSize": 59, - "responseSize": 118, - "status": 200, - "remoteIp": "127.0.0.1", - "serverIp": "127.0.0.1:1980", - "latency": "0.103s" - }, - "resource": { - "type": "global" - }, - "jsonPayload": { - "service_id": "", - "route_id": "1" - }, - "timestamp": "2024-01-06T03:34:45.065Z" -} -``` +| 名称 | 类型 | 必选项 | 默认值 | 描述 | +|-------------------------|---------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| auth_config | object | False | | 认证配置。`auth_config` 和 `auth_file` 中至少需要提供一个。 | +| auth_config.client_email | string | True | | Google Cloud 服务账号的电子邮件地址。 | +| auth_config.private_key | string | True | | Google Cloud 服务账号的私钥。 | +| auth_config.project_id | string | True | | Google Cloud 服务账号中的项目 ID。 | +| auth_config.token_uri | string | True | https://oauth2.googleapis.com/token | Google Cloud 服务账号的 Token URI。 | +| auth_config.entries_uri | string | False | https://logging.googleapis.com/v2/entries:write | Google Cloud Logging Service API。 | +| auth_config.scope | array[string] | False | ["https://www.googleapis.com/auth/logging.read";, "https://www.googleapis.com/auth/logging.write";, "https://www.googleapis.com/auth/logging.admin";, "https://www.googleapis.com/auth/cloud-platform";] | Google Cloud 服务账号的访问范围。参见 [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging)。 | Review Comment: 插件 schema 仍然支持 `auth_config.scopes`（已废弃）字段，但当前属性表中已移除，可能会影响仍在使用旧配置的用户。建议在属性表中补充 `auth_config.scopes` 并标注为“废弃”，同时引导使用 `auth_config.scope`。 ```suggestion | auth_config.scope | array[string] | False | ["https://www.googleapis.com/auth/logging.read";, "https://www.googleapis.com/auth/logging.write";, "https://www.googleapis.com/auth/logging.admin";, "https://www.googleapis.com/auth/cloud-platform";] | Google Cloud 服务账号的访问范围。参见 [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging)。 | | auth_config.scopes | array[string] | False | | 已废弃。为兼容旧配置仍受支持，请改用 `auth_config.scope`。 | ``` ########## docs/en/latest/plugins/clickhouse-logger.md: ########## @@ -127,86 +93,168 @@ admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"/ ::: +### Log in the Default Log Format + +The following example demonstrates how to log requests in the default log format. + +Create a table named `default_logs` in your ClickHouse database with columns corresponding to the default log format: + ```shell -curl http://127.0.0.1:9180/apisix/admin/plugin_metadata/clickhouse-logger -H "X-API-KEY: $admin_key" -X PUT -d ' -{ - "log_format": { - "host": "$host", - "@timestamp": "$time_iso8601", - "client_ip": "$remote_addr" +curl "http://127.0.0.1:8123"; -X POST -d ' + CREATE TABLE default.default_logs ( + host String, + client_ip String, + route_id String, + service_id String, + start_time String, + latency String, + upstream_latency String, + apisix_latency String, + consumer String, + request String, + response String, + server String, + PRIMARY KEY(`start_time`) + ) + ENGINE = MergeTree() Review Comment: The ClickHouse MergeTree engine requires an `ORDER BY` clause (or the older syntax that includes ordering). As written, `ENGINE = MergeTree()` will fail on common ClickHouse versions. Update the example to include an `ORDER BY` (e.g., `ORDER BY (start_time)` or `ORDER BY tuple()`) consistent with the chosen primary key. ```suggestion server String ) ENGINE = MergeTree() ORDER BY (`start_time`) PRIMARY KEY(`start_time`) ``` ########## docs/en/latest/plugins/splunk-hec-logging.md: ########## @@ -28,76 +28,66 @@ description: This document contains information about the Apache APISIX splunk-h # --> -## Description +<head> + <link rel="canonical" href="https://docs.api7.ai/hub/splunk-hec-logging"; /> +</head> -The `splunk-hec-logging` Plugin is used to forward logs to [Splunk HTTP Event Collector (HEC)](https://docs.splunk.com/Documentation/Splunk/8.2.6/Data/UsetheHTTPEventCollector) for analysis and storage. +## Description -When the Plugin is enabled, APISIX will serialize the request context information to [Splunk Event Data format](https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata) and submit it to the batch queue. When the maximum batch size is exceeded, the data in the queue is pushed to Splunk HEC. See [batch processor](../batch-processor.md) for more details. +The `splunk-hec-logging` Plugin serializes request and response context information to [Splunk Event Data format](https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata) and pushes to your [Splunk HTTP Event Collector (HEC)](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector) in batches. The Plugin also supports the customization of log formats. ## Attributes -| Name | Required | Default | Description | -|------------------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| endpoint | True | | Splunk HEC endpoint configurations. | -| endpoint.uri | True | | Splunk HEC event collector API endpoint. | -| endpoint.token | True | | Splunk HEC authentication token. | -| endpoint.channel | False | | Splunk HEC send data channel identifier. Read more: [About HTTP Event Collector Indexer Acknowledgment](https://docs.splunk.com/Documentation/Splunk/8.2.3/Data/AboutHECIDXAck). | -| endpoint.timeout | False | 10 | Splunk HEC send data timeout in seconds. | -| endpoint.keepalive_timeout | False | 60000 | Keepalive timeout in milliseconds. | -| ssl_verify | False | true | When set to `true` enables SSL verification as per [OpenResty docs](https://github.com/openresty/lua-nginx-module#tcpsocksslhandshake). | -| log_format | False | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, [APISIX](../apisix-variable.md) or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. | +| Name | Type | Required | Default | Valid values | Description | +|----------------------------|---------|----------|------------------|----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| endpoint | object | True | | | Splunk HEC endpoint configurations. | +| endpoint.uri | string | True | | | Splunk HEC event collector API endpoint. | +| endpoint.token | string | True | | | Splunk HEC authentication token. | +| endpoint.channel | string | False | | | Splunk HEC send data channel identifier. For more information, see [About HTTP Event Collector Indexer Acknowledgment](https://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHECIDXAck). | +| endpoint.timeout | integer | False | 10 | | Splunk HEC send data timeout in seconds. | +| endpoint.keepalive_timeout | integer | False | 60000 | >= 1000 | Keepalive timeout in milliseconds. | +| ssl_verify | boolean | False | true | | If `true`, enables SSL verification as per [OpenResty docs](https://github.com/openresty/lua-nginx-module#tcpsocksslhandshake). | +| log_format | object | False | | | Custom log format using key-value pairs in JSON format. Values can reference [APISIX variables](../apisix-variable.md) or [NGINX variables](https://nginx.org/en/docs/http/ngx_http_core_module.html) by prefixing with `$`. You can also configure log format on a global scale using [Plugin Metadata](#plugin-metadata). | +| name | string | False | splunk-hec-logging | | Unique identifier of the Plugin for the batch processor. | +| batch_max_size | integer | False | 1000 | greater than 0 | The number of log entries allowed in one batch. Once reached, the batch will be sent to Splunk HEC. Setting this parameter to `1` means immediate processing. | +| inactive_timeout | integer | False | 5 | greater than 0 | The maximum time in seconds to wait for new logs before sending the batch to the logging service. The value should be smaller than `buffer_duration`. | +| buffer_duration | integer | False | 60 | greater than 0 | The maximum time in seconds from the earliest entry allowed before sending the batch to the logging service. | +| retry_delay | integer | False | 1 | >= 0 | The time interval in seconds to retry sending the batch to the logging service if the batch was not successfully sent. | +| max_retry_count | integer | False | 60 | >= 0 | The maximum number of unsuccessful retries allowed before dropping the log entries. | This Plugin supports using batch processors to aggregate and process entries (logs/data) in a batch. This avoids the need for frequently submitting the data. The batch processor submits data every `5` seconds or when the data in the queue reaches `1000`. See [Batch Processor](../batch-processor.md#configuration) for more information or setting your custom configuration. -### Example of default log format +## Plugin Metadata -```json -{ - "sourcetype": "_json", - "time": 1704513555.392, - "event": { - "upstream": "127.0.0.1:1980", - "request_url": "http://localhost:1984/hello";, - "request_query": {}, - "request_size": 59, - "response_headers": { - "content-length": "12", - "server": "APISIX/3.7.0", - "content-type": "text/plain", - "connection": "close" - }, - "response_status": 200, - "response_size": 118, - "latency": 108.00004005432, - "request_method": "GET", - "request_headers": { - "connection": "close", - "host": "localhost" - } - }, - "source": "apache-apisix-splunk-hec-logging", - "host": "localhost" -} -``` +| Name | Type | Required | Default | Valid values | Description | +|--------------------|---------|----------|---------|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| log_format | object | False | | | Custom log format using key-value pairs in JSON format. Values can reference [APISIX variables](../apisix-variable.md) or [NGINX variables](https://nginx.org/en/docs/http/ngx_http_core_module.html) by prefixing with `$`. This configuration is global and applies to all Routes and Services that use the `splunk-hec-logging` Plugin. | +| max_pending_entries | integer | False | | >= 1 | Maximum number of unprocessed entries allowed in the batch processor. When this limit is reached, new entries will be dropped until the backlog is reduced. | -## Metadata +## Examples -You can also set the format of the logs by configuring the Plugin metadata. The following configurations are available: +The examples below demonstrate how you can configure the `splunk-hec-logging` Plugin for different use cases. -| Name | Type | Required | Default | Description | -| ---------- | ------ | -------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| log_format | object | False | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, [APISIX](../apisix-variable.md) or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. | -| max_pending_entries | integer | False | | Maximum number of pending entries that can be buffered in batch processor before it starts dropping them. | +To follow along with the examples, please complete the following steps to set up Splunk: -:::info IMPORTANT +* Install [Splunk](https://www.splunk.com/en_us/download.html). Splunk Web should be running at `localhost:8000` by default. +* See [set up and use HTTP Event Collector in Splunk Web](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector) to set up an HTTP Event Collector. +* Navigate to **Settings > Data Inputs** at the upper-right corner of the console. You should see at least one input for the HTTP Event Collector. Note down the token value. +* Navigate to **Settings > Data Inputs** at the upper-right corner of the console and select **HTTP Event Collector**. In **Global Settings**, enable all tokens. +* In **Global Settings**, you should also find the collector's default port to be `8088`. -Configuring the Plugin metadata is global in scope. This means that it will take effect on all Routes and Services which use the `splunk-hec-logging` Plugin. +To verify the setup, execute the following command with your token: -::: +```shell +curl "http://localhost:8088/services/collector/event"; \ Review Comment: The line-continuation backslash in this curl command has trailing whitespace (`\ `). In most shells, the backslash must be the final character on the line; otherwise the command may not copy/paste correctly. Remove the trailing whitespace so the backslash is the last character. ```suggestion curl "http://localhost:8088/services/collector/event"; \ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
