Re: [I] bug: elasticsearch-logger plugin collects logs and calls log loss in certain situations [apisix]
Baoyuantop closed issue #12381: bug: elasticsearch-logger plugin collects logs and calls log loss in certain situations URL: https://github.com/apache/apisix/issues/12381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: elasticsearch-logger plugin collects logs and calls log loss in certain situations [apisix]
Baoyuantop commented on issue #12381: URL: https://github.com/apache/apisix/issues/12381#issuecomment-3252365779 If there is still a problem, please open it again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: elasticsearch-logger plugin collects logs and calls log loss in certain situations [apisix]
Baoyuantop commented on issue #12381: URL: https://github.com/apache/apisix/issues/12381#issuecomment-3116916509 Hi @shihuncl, any updates? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: elasticsearch-logger plugin collects logs and calls log loss in certain situations [apisix]
Baoyuantop commented on issue #12381: URL: https://github.com/apache/apisix/issues/12381#issuecomment-3055934918 It would be better to show the complete code or the result of git diff. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: elasticsearch-logger plugin collects logs and calls log loss in certain situations [apisix]
shihuncl commented on issue #12381:
URL: https://github.com/apache/apisix/issues/12381#issuecomment-3050744173
> If you have changed code, you should bring it up, as it helps to
troubleshoot the problem, and it is not 100% sure that they are not related.
elasticsearch-logger.lua:
local os_date = os.date
local function get_logger_entry(conf, ctx)
local index_name = conf.field.index .. "-" .. os_date("%Y-%m-%d")
local entry = log_util.get_log_entry(plugin_name, conf, ctx)
return core.json.encode({
create = {
_index = index_name,
_type = conf.field.type
}
}) .. "\n" ..
core.json.encode(entry) .. "\n"
end
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: elasticsearch-logger plugin collects logs and calls log loss in certain situations [apisix]
Baoyuantop commented on issue #12381: URL: https://github.com/apache/apisix/issues/12381#issuecomment-3027167043 If you have changed code, you should bring it up, as it helps to troubleshoot the problem, and it is not 100% sure that they are not related. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: elasticsearch-logger plugin collects logs and calls log loss in certain situations [apisix]
shihuncl commented on issue #12381: URL: https://github.com/apache/apisix/issues/12381#issuecomment-3026855055 Why is there an inconsistency? Is it because I changed the elasticsearch-logger.lua script? But the script I changed does not involve the insertion of the overall call log. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: elasticsearch-logger plugin collects logs and calls log loss in certain situations [apisix]
Baoyuantop commented on issue #12381:
URL: https://github.com/apache/apisix/issues/12381#issuecomment-3026580270
I can't reproduce this problem locally. This is my configuration file:
```
version: "3"
services:
apisix:
image: apache/apisix:${APISIX_IMAGE_TAG:-3.9.1-debian}
restart: always
volumes:
- ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro
depends_on:
- etcd
##network_mode: host
ports:
- "9180:9180/tcp"
- "9080:9080/tcp"
- "9091:9091/tcp"
- "9443:9443/tcp"
- "9092:9092/tcp"
networks:
apisix:
etcd:
image: bitnami/etcd:3.5.11
restart: always
volumes:
- etcd_data:/bitnami/etcd
environment:
ETCD_ENABLE_V2: "true"
ALLOW_NONE_AUTHENTICATION: "yes"
ETCD_ADVERTISE_CLIENT_URLS: "http://etcd:2379";
ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379";
ports:
- "2379:2379/tcp"
networks:
apisix:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.17.1
ports:
- "9200:9200"
- "9300:9300"
environment:
- ES_JAVA_OPTS=-Xms512m -Xmx512m
- discovery.type=single-node
- xpack.security.enabled=false
networks:
apisix:
kibana:
image: docker.elastic.co/kibana/kibana:7.17.1
ports:
- "5601:5601"
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
networks:
apisix:
networks:
apisix:
driver: bridge
volumes:
etcd_data:
driver: local
```
create a route and global rule
```
{
"uri": "/ip",
"upstream": {
"nodes": {
"httpbin.org:80": 1
},
"type": "roundrobin"
}
}
```
```
{
"plugins": {
"elasticsearch-logger": {
"endpoint_addrs": [
"http://elasticsearch:9200";
],
"field": {
"index": "gateway",
"type": "logs"
},
"ssl_verify": false,
"timeout": 60,
"retry_delay": 1,
"buffer_duration": 60,
"max_retry_count": 0,
"batch_max_size": 5,
"inactive_timeout": 5
}
}
}
```
send a request like
```
curl -i
"127.0.0.1:9080/ip?projectCode=P4065&beginDate=2019-11-01%2000:00:00&endDate=2019-11-01%2000:00:00"
```
and
```
curl -i "127.0.0.1:9080/ip?projectCode=P4065&beginDate&endDate"
```
I can see the logs in kibana
```
apisix_latency:33 client_ip:192.168.148.1 latency:607
request.headers.accept:*/* request.headers.host:127.0.0.1:9080
request.headers.user-agent:curl/8.7.1 request.method:GET
request.querystring.beginDate:2019-11-01 00:00:00
request.querystring.beginDate.keyword:2019-11-01 00:00:00
request.querystring.endDate:2019-11-01 00:00:00
request.querystring.endDate.keyword:2019-11-01 00:00:00
request.querystring.projectCode:P4065
request.querystring.projectCode.keyword:P4065 request.size:159
request.uri:/ip?projectCode=P4065&beginDate=2019-11-01%2000:00:00&endDate=2019-11-01%2000:00:00
request.url:http://127.0.0.1:9080/ip?projectCode=P4065&beginDate=2019-11-01%2000:00:00&endDate=2019-11-01%2000:00:00
response.headers.access-control-allow-credentials:true
response.headers.access-control-allow-origin:*
response.headers.connection:close response.headers.content-length:46
response.headers.content-type:application/json response.headers.date:Wed, 02
Jul 2025 06:03:30 GMT
```
```
apisix_latency:106 client_ip:192.168.148.1 latency:588
request.headers.accept:*/* request.headers.host:127.0.0.1:9080
request.headers.user-agent:curl/8.7.1 request.method:GET
request.querystring.beginDate:true request.querystring.beginDate.keyword:true
request.querystring.endDate:true request.querystring.endDate.keyword:true
request.querystring.projectCode:P4065
request.querystring.projectCode.keyword:P4065 request.size:115
request.uri:/ip?projectCode=P4065&beginDate&endDate
request.url:http://127.0.0.1:9080/ip?projectCode=P4065&beginDate&endDate
response.headers.access-control-allow-credentials:true
response.headers.access-control-allow-origin:*
response.headers.connection:close response.headers.content-length:46
response.headers.content-type:application/json response.headers.date:Wed, 02
Jul 2025 06:05:05 GMT response.headers.server:APISIX/3.9.1 response.size:272
response.status:200 route_id:1 server.hostname:3f67f911a73a
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: elasticsearch-logger plugin collects logs and calls log loss in certain situations [apisix]
Baoyuantop commented on issue #12381: URL: https://github.com/apache/apisix/issues/12381#issuecomment-3007184722 Hi @shihuncl, can you provide a detailed reproduction process? In addition, I did not find any configuration related to the elasticsearch-logger plugin from the information provided. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: elasticsearch-logger plugin collects logs and calls log loss in certain situations [apisix]
shihuncl commented on issue #12381:
URL: https://github.com/apache/apisix/issues/12381#issuecomment-3007224821
> Hi [@shihuncl](https://github.com/shihuncl), can you provide a detailed
reproduction process? In addition, I did not find any configuration related to
the elasticsearch-logger plugin from the information provided.
apisix service config:{
"name": "service config",
"desc": "1",
"upstream_id": "5c770b9e35144cb5bc8bc52ed1c1541c",
"plugins": {
"basic-auth": {
"_meta": {
"disable": false
},
"hide_credentials": false
},
"elasticsearch-logger": {
"_meta": {
"disable": false
},
"auth": {
"password": "",
"username": "elastic"
},
"batch_max_size": 100,
"buffer_duration": 60,
"endpoint_addr": "https://test:9200";,
"field": {
"index": "log"
},
"inactive_timeout": 5,
"include_query_args": false,
"include_req_body": false,
"include_resp_body": false,
"max_retry_count": 0,
"name": "elasticsearch-logger",
"retry_delay": 1,
"ssl_verify": false,
"timeout": 60
}
}
}
When the requested URL path parameter is irregular or the parameter value
contains spaces, the API call record will not be stored.
example:
1.https://test/mdm/api/service/datapub/rest/api/v1/pmdm/projectTeam?projectCode=P4065&beginDate=2019-11-01%2000:00:00&endDate=2019-11-01%2000:00:00
2.https://test/mdm/api/service/datapub/rest/api/v1/pmdm/projectTeam?projectCode=P4065&beginDate&endDate'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
