kayx23 commented on code in PR #13273: URL: https://github.com/apache/apisix/pull/13273#discussion_r3136726183
########## docs/en/latest/plugins/clickhouse-logger.md: ########## @@ -27,98 +27,64 @@ description: This document contains information about the Apache APISIX clickhou # --> +<head> + <link rel="canonical" href="https://docs.api7.ai/hub/clickhouse-logger"; /> +</head> + ## Description -The `clickhouse-logger` Plugin is used to push logs to [ClickHouse](https://clickhouse.com/) database. +The `clickhouse-logger` Plugin pushes request and response logs to [ClickHouse](https://clickhouse.com/) database in batches and supports the customization of log formats. ## Attributes -| Name | Type | Required | Default | Valid values | Description | -|---------------|---------|----------|---------------------|--------------|----------------------------------------------------------------| -| endpoint_addr | Deprecated | True | | | Use `endpoint_addrs` instead. ClickHouse endpoints. | -| endpoint_addrs | array | True | | | ClickHouse endpoints. | -| database | string | True | | | Name of the database to store the logs. | -| logtable | string | True | | | Table name to store the logs. | -| user | string | True | | | ClickHouse username. | -| password | string | True | | | ClickHouse password. | -| timeout | integer | False | 3 | [1,...] | Time to keep the connection alive for after sending a request. | -| name | string | False | "clickhouse logger" | | Unique identifier for the logger. If you use Prometheus to monitor APISIX metrics, the name is exported in `apisix_batch_process_entries`. | -| ssl_verify | boolean | False | true | [true,false] | When set to `true`, verifies SSL. | -| log_format | object | False | | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, [APISIX](../apisix-variable.md) or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. | -| include_req_body | boolean | False | false | [false, true] | When set to `true` includes the request body in the log. If the request body is too big to be kept in the memory, it can't be logged due to Nginx's limitations. | -| include_req_body_expr | array | False | | | Filter for when the `include_req_body` attribute is set to `true`. Request body is only logged when the expression set here evaluates to `true`. See [lua-resty-expr](https://github.com/api7/lua-resty-expr) for more. | -| max_req_body_bytes | integer | False | 524288 | >=1 | Request bodies within this size will be logged, if the size exceeds the configured value it will be truncated before logging. | -| include_resp_body | boolean | False | false | [false, true] | When set to `true` includes the response body in the log. | -| include_resp_body_expr | array | False | | | Filter for when the `include_resp_body` attribute is set to `true`. Response body is only logged when the expression set here evaluates to `true`. See [lua-resty-expr](https://github.com/api7/lua-resty-expr) for more. | -| max_resp_body_bytes | integer | False | 524288 | >=1 | Response bodies within this size will be logged, if the size exceeds the configured value it will be truncated before logging. | +| Name | Type | Required | Default | Valid values | Description | +|------------------------|-------------|----------|---------------------|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| endpoint_addrs | array | True | | | ClickHouse endpoints. | +| database | string | True | | | Name of the database to store the logs. | +| logtable | string | True | | | Name of the table that stores the logs. | +| user | string | True | | | ClickHouse username. From APISIX 3.16.0, supports referencing values from environment variables using the `$ENV://` prefix or from a secret manager using the `$secret://` prefix. For more information, see [secrets](../terminology/secret.md). | +| password | string | True | | | ClickHouse password. From APISIX 3.16.0, supports referencing values from environment variables using the `$ENV://` prefix or from a secret manager using the `$secret://` prefix. For more information, see [secrets](../terminology/secret.md). | +| timeout | integer | False | 3 | greater than 0 | Time in seconds to keep the connection alive after sending a request. | +| ssl_verify | boolean | False | true | | If `true`, verify SSL. | +| log_format | object | False | | | Custom log format using key-value pairs in JSON format. Values can reference [APISIX variables](../apisix-variable.md) or [NGINX variables](https://nginx.org/en/docs/http/ngx_http_core_module.html) by prefixing with `$`. You can also configure log format on a global scale using [Plugin Metadata](#plugin-metadata). | +| include_req_body | boolean | False | false | | If `true`, include the request body in the log. Note that if the request body is too big to be kept in the memory, it cannot be logged due to NGINX's limitations. | +| include_req_body_expr | array | False | | | An array of one or more conditions in the form of [APISIX expressions](https://github.com/api7/lua-resty-expr). Used when `include_req_body` is `true`. Request body is only logged when the expressions evaluate to `true`. | +| include_resp_body | boolean | False | false | | If `true`, include the response body in the log. | +| include_resp_body_expr | array | False | | | An array of one or more conditions in the form of [APISIX expressions](https://github.com/api7/lua-resty-expr). Used when `include_resp_body` is `true`. Response body is only logged when the expressions evaluate to `true`. | +| max_req_body_bytes | integer | False | 524288 | >= 1 | Maximum request body size in bytes to include in the log. If the request body exceeds this value, it will be truncated. Available in APISIX from 3.16.0. | +| max_resp_body_bytes | integer | False | 524288 | >= 1 | Maximum response body size in bytes to include in the log. If the response body exceeds this value, it will be truncated. Available in APISIX from 3.16.0. | +| name | string | False | "clickhouse logger" | | Unique identifier of the Plugin for the batch processor. If you use [Prometheus](./prometheus.md) to monitor APISIX metrics, the name is exported in `apisix_batch_process_entries`. | +| batch_max_size | integer | False | 1000 | greater than 0 | The number of log entries allowed in one batch. Once reached, the batch will be sent to ClickHouse. Setting this parameter to `1` means immediate processing. | +| inactive_timeout | integer | False | 5 | greater than 0 | The maximum time in seconds to wait for new logs before sending the batch to the logging service. The value should be smaller than `buffer_duration`. | +| buffer_duration | integer | False | 60 | greater than 0 | The maximum time in seconds from the earliest entry allowed before sending the batch to the logging service. | Review Comment: The default for `max_retry_count` is documented as `60`, but the batch-processor schema (`apisix/utils/batch-processor.lua`) defines the default as `0`, and the `clickhouse-logger` schema does not override it. Please correct this to `0`. ########## docs/en/latest/plugins/splunk-hec-logging.md: ########## @@ -28,76 +28,66 @@ description: This document contains information about the Apache APISIX splunk-h # --> -## Description +<head> + <link rel="canonical" href="https://docs.api7.ai/hub/splunk-hec-logging"; /> +</head> -The `splunk-hec-logging` Plugin is used to forward logs to [Splunk HTTP Event Collector (HEC)](https://docs.splunk.com/Documentation/Splunk/8.2.6/Data/UsetheHTTPEventCollector) for analysis and storage. +## Description -When the Plugin is enabled, APISIX will serialize the request context information to [Splunk Event Data format](https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata) and submit it to the batch queue. When the maximum batch size is exceeded, the data in the queue is pushed to Splunk HEC. See [batch processor](../batch-processor.md) for more details. +The `splunk-hec-logging` Plugin serializes request and response context information to [Splunk Event Data format](https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata) and pushes to your [Splunk HTTP Event Collector (HEC)](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector) in batches. The Plugin also supports the customization of log formats. ## Attributes -| Name | Required | Default | Description | -|------------------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| endpoint | True | | Splunk HEC endpoint configurations. | -| endpoint.uri | True | | Splunk HEC event collector API endpoint. | -| endpoint.token | True | | Splunk HEC authentication token. | -| endpoint.channel | False | | Splunk HEC send data channel identifier. Read more: [About HTTP Event Collector Indexer Acknowledgment](https://docs.splunk.com/Documentation/Splunk/8.2.3/Data/AboutHECIDXAck). | -| endpoint.timeout | False | 10 | Splunk HEC send data timeout in seconds. | -| endpoint.keepalive_timeout | False | 60000 | Keepalive timeout in milliseconds. | -| ssl_verify | False | true | When set to `true` enables SSL verification as per [OpenResty docs](https://github.com/openresty/lua-nginx-module#tcpsocksslhandshake). | -| log_format | False | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, [APISIX](../apisix-variable.md) or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. | +| Name | Type | Required | Default | Valid values | Description | +|----------------------------|---------|----------|------------------|----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| endpoint | object | True | | | Splunk HEC endpoint configurations. | +| endpoint.uri | string | True | | | Splunk HEC event collector API endpoint. | +| endpoint.token | string | True | | | Splunk HEC authentication token. | +| endpoint.channel | string | False | | | Splunk HEC send data channel identifier. For more information, see [About HTTP Event Collector Indexer Acknowledgment](https://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHECIDXAck). | +| endpoint.timeout | integer | False | 10 | | Splunk HEC send data timeout in seconds. | +| endpoint.keepalive_timeout | integer | False | 60000 | >= 1000 | Keepalive timeout in milliseconds. | +| ssl_verify | boolean | False | true | | If `true`, enables SSL verification as per [OpenResty docs](https://github.com/openresty/lua-nginx-module#tcpsocksslhandshake). | +| log_format | object | False | | | Custom log format using key-value pairs in JSON format. Values can reference [APISIX variables](../apisix-variable.md) or [NGINX variables](https://nginx.org/en/docs/http/ngx_http_core_module.html) by prefixing with `$`. You can also configure log format on a global scale using [Plugin Metadata](#plugin-metadata). | +| name | string | False | splunk-hec-logging | | Unique identifier of the Plugin for the batch processor. | +| batch_max_size | integer | False | 1000 | greater than 0 | The number of log entries allowed in one batch. Once reached, the batch will be sent to Splunk HEC. Setting this parameter to `1` means immediate processing. | +| inactive_timeout | integer | False | 5 | greater than 0 | The maximum time in seconds to wait for new logs before sending the batch to the logging service. The value should be smaller than `buffer_duration`. | +| buffer_duration | integer | False | 60 | greater than 0 | The maximum time in seconds from the earliest entry allowed before sending the batch to the logging service. | +| retry_delay | integer | False | 1 | >= 0 | The time interval in seconds to retry sending the batch to the logging service if the batch was not successfully sent. | +| max_retry_count | integer | False | 60 | >= 0 | The maximum number of unsuccessful retries allowed before dropping the log entries. | This Plugin supports using batch processors to aggregate and process entries (logs/data) in a batch. This avoids the need for frequently submitting the data. The batch processor submits data every `5` seconds or when the data in the queue reaches `1000`. See [Batch Processor](../batch-processor.md#configuration) for more information or setting your custom configuration. -### Example of default log format +## Plugin Metadata Review Comment: Same issue: `max_retry_count` default is documented as `60` but the actual default (from `apisix/utils/batch-processor.lua`) is `0`. The `splunk-hec-logging` schema does not override it. Please correct this to `0`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
