arturas-miles opened a new issue, #12141:
URL: https://github.com/apache/apisix/issues/12141
### Description
Hey community,
I was wondering if it is just me or my config is messy?
But I have failing etcd at least once a month and I am running 2 different
Apisix, one on GCP GKE another on Azure AKS
Both are deployed same way - ArgoCD, Helm chart with a following config
Helm chart:
```yaml
dependencies:
- name: apisix
version: 2.10.0
repository: https://charts.apiseven.com
```
etcd values
```yaml
etcd:
replicaCount: 3
auth:
token:
type: simple
```
and the error log from one of the pods
```bash
{"level":"info","ts":"2025-04-15T08:28:31.37395Z","caller":"flags/flag.go:113","msg":"recognized
and used environment
variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"}
{"level":"info","ts":"2025-04-15T08:28:31.373959Z","caller":"flags/flag.go:113","msg":"recognized
and used environment
variable","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"}
{"level":"info","ts":"2025-04-15T08:28:31.373963Z","caller":"flags/flag.go:113","msg":"recognized
and used environment
variable","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value":"new"}
{"level":"info","ts":"2025-04-15T08:28:31.373968Z","caller":"flags/flag.go:113","msg":"recognized
and used environment
variable","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":"etcd-cluster-k8s"}
{"level":"info","ts":"2025-04-15T08:28:31.373974Z","caller":"flags/flag.go:113","msg":"recognized
and used environment
variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"http://0.0.0.0:2379"}
{"level":"info","ts":"2025-04-15T08:28:31.373981Z","caller":"flags/flag.go:113","msg":"recognized
and used environment
variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"http://0.0.0.0:2380"}
{"level":"info","ts":"2025-04-15T08:28:31.373985Z","caller":"flags/flag.go:113","msg":"recognized
and used environment
variable","variable-name":"ETCD_LOG_LEVEL","variable-value":"info"}
{"level":"info","ts":"2025-04-15T08:28:31.373992Z","caller":"flags/flag.go:113","msg":"recognized
and used environment
variable","variable-name":"ETCD_NAME","variable-value":"apisix-etcd-1"}
{"level":"info","ts":"2025-04-15T08:28:31.374001Z","caller":"flags/flag.go:113","msg":"recognized
and used environment
variable","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"false"}
{"level":"warn","ts":"2025-04-15T08:28:31.374023Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_TRUSTED_CA_FILE="}
{"level":"warn","ts":"2025-04-15T08:28:31.37404Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_DISABLE_STORE_MEMBER_ID=no"}
{"level":"warn","ts":"2025-04-15T08:28:31.374051Z","caller":"flags/flag.go:93","msg":"unrecognized
environment
variable","environment-variable":"ETCD_CONF_FILE=/opt/bitnami/etcd/conf/etcd.yaml"}
{"level":"warn","ts":"2025-04-15T08:28:31.374059Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_SNAPSHOT_HISTORY_LIMIT=1"}
{"level":"warn","ts":"2025-04-15T08:28:31.374067Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_ON_K8S=yes"}
{"level":"warn","ts":"2025-04-15T08:28:31.374074Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_SNAPSHOTS_DIR=/snapshots"}
{"level":"warn","ts":"2025-04-15T08:28:31.37408Z","caller":"flags/flag.go:93","msg":"unrecognized
environment
variable","environment-variable":"ETCD_BIN_DIR=/opt/bitnami/etcd/bin"}
{"level":"warn","ts":"2025-04-15T08:28:31.374086Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_VOLUME_DIR=/bitnami/etcd"}
{"level":"warn","ts":"2025-04-15T08:28:31.374094Z","caller":"flags/flag.go:93","msg":"unrecognized
environment
variable","environment-variable":"ETCD_CLUSTER_DOMAIN=apisix-etcd-headless.ingress-apisix.svc.cluster.local"}
{"level":"warn","ts":"2025-04-15T08:28:31.3741Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_DISASTER_RECOVERY=no"}
{"level":"warn","ts":"2025-04-15T08:28:31.374108Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_KEY_FILE="}
{"level":"warn","ts":"2025-04-15T08:28:31.374114Z","caller":"flags/flag.go:93","msg":"unrecognized
environment
variable","environment-variable":"ETCD_CONF_DIR=/opt/bitnami/etcd/conf"}
{"level":"warn","ts":"2025-04-15T08:28:31.374121Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_DAEMON_GROUP=etcd"}
{"level":"warn","ts":"2025-04-15T08:28:31.374127Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_START_FROM_SNAPSHOT=no"}
{"level":"warn","ts":"2025-04-15T08:28:31.374137Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_INIT_SNAPSHOT_FILENAME="}
{"level":"warn","ts":"2025-04-15T08:28:31.37415Z","caller":"flags/flag.go:93","msg":"unrecognized
environment
variable","environment-variable":"ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot"}
{"level":"warn","ts":"2025-04-15T08:28:31.374153Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_DISABLE_PRESTOP=no"}
{"level":"warn","ts":"2025-04-15T08:28:31.374158Z","caller":"flags/flag.go:93","msg":"unrecognized
environment
variable","environment-variable":"ETCD_TMP_DIR=/opt/bitnami/etcd/tmp"}
{"level":"warn","ts":"2025-04-15T08:28:31.374167Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_BASE_DIR=/opt/bitnami/etcd"}
{"level":"warn","ts":"2025-04-15T08:28:31.374175Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_CERT_FILE="}
{"level":"warn","ts":"2025-04-15T08:28:31.37418Z","caller":"flags/flag.go:93","msg":"unrecognized
environment
variable","environment-variable":"ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs"}
{"level":"warn","ts":"2025-04-15T08:28:31.374207Z","caller":"flags/flag.go:93","msg":"unrecognized
environment variable","environment-variable":"ETCD_DAEMON_USER=etcd"}
{"level":"warn","ts":"2025-04-15T08:28:31.374271Z","caller":"embed/config.go:676","msg":"Running
http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2025-04-15T08:28:31.374292Z","caller":"etcdmain/etcd.go:73","msg":"Running:
","args":["etcd"]}
{"level":"warn","ts":"2025-04-15T08:28:31.37434Z","caller":"etcdmain/etcd.go:446","msg":"found
invalid file under data
directory","filename":"member_id","data-dir":"/bitnami/etcd/data"}
{"level":"info","ts":"2025-04-15T08:28:31.374357Z","caller":"etcdmain/etcd.go:116","msg":"server
has been already
initialized","data-dir":"/bitnami/etcd/data","dir-type":"member"}
{"level":"warn","ts":"2025-04-15T08:28:31.374373Z","caller":"embed/config.go:676","msg":"Running
http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2025-04-15T08:28:31.374381Z","caller":"embed/etcd.go:127","msg":"configuring
peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}
{"level":"info","ts":"2025-04-15T08:28:31.374641Z","caller":"embed/etcd.go:135","msg":"configuring
client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2025-04-15T08:28:31.374752Z","caller":"embed/etcd.go:309","msg":"starting
an etcd
server","etcd-version":"3.5.10","git-sha":"0223ca5","go-version":"go1.20.11","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":true,"name":"apisix-etcd-1","data-dir":"/bitnami/etcd/data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/bitnami/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379","http://apisix-etcd.ingress-apisix.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metr
ics-urls":],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2025-04-15T08:28:32.014031Z","caller":"etcdserver/backend.go:81","msg":"opened
backend db","path":"/bitnami/etcd/data/member/snap/db","took":"635.715715ms"}
{"level":"info","ts":"2025-04-15T08:28:32.610844Z","caller":"etcdserver/server.go:509","msg":"recovered
v2 store from snapshot","snapshot-index":7200072,"snapshot-size":"12 kB"}
{"level":"info","ts":"2025-04-15T08:28:32.610894Z","caller":"etcdserver/server.go:522","msg":"recovered
v3 backend from snapshot","backend-size-bytes":2145570816,"backend-size":"2.1
GB","backend-size-in-use-bytes":2145550336,"backend-size-in-use":"2.1 GB"}
{"level":"info","ts":"2025-04-15T08:28:32.632797Z","caller":"etcdserver/raft.go:530","msg":"restarting
local
member","cluster-id":"b9ae902a9398fd61","local-member-id":"47324e080c98137d","commit-index":7217105}
{"level":"info","ts":"2025-04-15T08:28:32.633276Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"47324e080c98137d
switched to configuration voters=(4171443932729916013 5130248722006414205
16309538207480531017)"}
{"level":"info","ts":"2025-04-15T08:28:32.633307Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"47324e080c98137d
became follower at term 12"}
{"level":"info","ts":"2025-04-15T08:28:32.633317Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft
47324e080c98137d [peers: [39e3f227d2d33a6d,47324e080c98137d,e2571e968b89c849],
term: 12, commit: 7217105, applied: 7200072, lastindex: 7217105, lastterm: 12]"}
{"level":"info","ts":"2025-04-15T08:28:32.63343Z","caller":"api/capability.go:75","msg":"enabled
capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2025-04-15T08:28:32.633442Z","caller":"membership/cluster.go:278","msg":"recovered/added
member from
store","cluster-id":"b9ae902a9398fd61","local-member-id":"47324e080c98137d","recovered-remote-peer-id":"39e3f227d2d33a6d","recovered-remote-peer-urls":["http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"info","ts":"2025-04-15T08:28:32.633451Z","caller":"membership/cluster.go:278","msg":"recovered/added
member from
store","cluster-id":"b9ae902a9398fd61","local-member-id":"47324e080c98137d","recovered-remote-peer-id":"47324e080c98137d","recovered-remote-peer-urls":["http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"info","ts":"2025-04-15T08:28:32.633458Z","caller":"membership/cluster.go:278","msg":"recovered/added
member from
store","cluster-id":"b9ae902a9398fd61","local-member-id":"47324e080c98137d","recovered-remote-peer-id":"e2571e968b89c849","recovered-remote-peer-urls":["http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"info","ts":"2025-04-15T08:28:32.633467Z","caller":"membership/cluster.go:287","msg":"set
cluster version from store","cluster-version":"3.5"}
{"level":"warn","ts":"2025-04-15T08:28:32.638724Z","caller":"auth/store.go:1241","msg":"simple
token is not cryptographically signed"}
{"level":"info","ts":"2025-04-15T08:28:33.735131Z","caller":"mvcc/kvstore.go:407","msg":"kvstore
restored","current-rev":1796704}
{"level":"info","ts":"2025-04-15T08:28:33.741796Z","caller":"etcdserver/quota.go:94","msg":"enabled
backend quota with default
value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1
GB"}
{"level":"info","ts":"2025-04-15T08:28:33.749119Z","caller":"rafthttp/peer.go:133","msg":"starting
remote peer","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.749147Z","caller":"rafthttp/pipeline.go:72","msg":"started
HTTP pipelining with remote
peer","local-member-id":"47324e080c98137d","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.749442Z","caller":"rafthttp/stream.go:169","msg":"started
stream writer with remote
peer","local-member-id":"47324e080c98137d","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.749547Z","caller":"rafthttp/stream.go:169","msg":"started
stream writer with remote
peer","local-member-id":"47324e080c98137d","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.749753Z","caller":"rafthttp/peer.go:137","msg":"started
remote peer","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.749788Z","caller":"rafthttp/stream.go:395","msg":"started
stream reader with remote peer","stream-reader-type":"stream MsgApp
v2","local-member-id":"47324e080c98137d","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.749814Z","caller":"rafthttp/transport.go:317","msg":"added
remote
peer","local-member-id":"47324e080c98137d","remote-peer-id":"39e3f227d2d33a6d","remote-peer-urls":["http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"info","ts":"2025-04-15T08:28:33.749789Z","caller":"rafthttp/stream.go:395","msg":"started
stream reader with remote peer","stream-reader-type":"stream
Message","local-member-id":"47324e080c98137d","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.749841Z","caller":"rafthttp/peer.go:133","msg":"starting
remote peer","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.749857Z","caller":"rafthttp/pipeline.go:72","msg":"started
HTTP pipelining with remote
peer","local-member-id":"47324e080c98137d","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.750099Z","caller":"rafthttp/stream.go:169","msg":"started
stream writer with remote
peer","local-member-id":"47324e080c98137d","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.750259Z","caller":"rafthttp/stream.go:169","msg":"started
stream writer with remote
peer","local-member-id":"47324e080c98137d","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.750452Z","caller":"rafthttp/peer.go:137","msg":"started
remote peer","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.750476Z","caller":"rafthttp/stream.go:395","msg":"started
stream reader with remote peer","stream-reader-type":"stream MsgApp
v2","local-member-id":"47324e080c98137d","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.750503Z","caller":"rafthttp/transport.go:317","msg":"added
remote
peer","local-member-id":"47324e080c98137d","remote-peer-id":"e2571e968b89c849","remote-peer-urls":["http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"info","ts":"2025-04-15T08:28:33.750549Z","caller":"etcdserver/server.go:845","msg":"starting
etcd
server","local-member-id":"47324e080c98137d","local-server-version":"3.5.10","cluster-id":"b9ae902a9398fd61","cluster-version":"3.5"}
{"level":"info","ts":"2025-04-15T08:28:33.750571Z","caller":"rafthttp/stream.go:395","msg":"started
stream reader with remote peer","stream-reader-type":"stream
Message","local-member-id":"47324e080c98137d","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.750653Z","caller":"fileutil/purge.go:44","msg":"started
to purge
file","dir":"/bitnami/etcd/data/member/snap","suffix":"snap.db","max":5,"interval":"30s"}
{"level":"info","ts":"2025-04-15T08:28:33.750721Z","caller":"fileutil/purge.go:44","msg":"started
to purge
file","dir":"/bitnami/etcd/data/member/snap","suffix":"snap","max":5,"interval":"30s"}
{"level":"info","ts":"2025-04-15T08:28:33.750774Z","caller":"fileutil/purge.go:44","msg":"started
to purge
file","dir":"/bitnami/etcd/data/member/wal","suffix":"wal","max":5,"interval":"30s"}
{"level":"info","ts":"2025-04-15T08:28:33.750726Z","caller":"etcdserver/server.go:754","msg":"starting
initial election tick advance","election-ticks":10}
{"level":"info","ts":"2025-04-15T08:28:33.752439Z","caller":"embed/etcd.go:597","msg":"serving
peer traffic","address":"[::]:2380"}
{"level":"info","ts":"2025-04-15T08:28:33.752455Z","caller":"embed/etcd.go:569","msg":"cmux::serve","address":"[::]:2380"}
{"level":"info","ts":"2025-04-15T08:28:33.75245Z","caller":"embed/etcd.go:278","msg":"now
serving
peer/client/metrics","local-member-id":"47324e080c98137d","initial-advertise-peer-urls":["http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379","http://apisix-etcd.ingress-apisix.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":]}
{"level":"warn","ts":"2025-04-15T08:28:33.766188Z","caller":"etcdserver/server.go:1127","msg":"server
error","error":"the member has been permanently removed from the cluster"}
{"level":"warn","ts":"2025-04-15T08:28:33.766235Z","caller":"etcdserver/server.go:1128","msg":"data-dir
used by this member must be removed"}
{"level":"warn","ts":"2025-04-15T08:28:33.766314Z","caller":"etcdserver/server.go:2073","msg":"stopped
publish because server is
stopped","local-member-id":"47324e080c98137d","local-member-attributes":"{Name:apisix-etcd-1
ClientURLs:[http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379
http://apisix-etcd.ingress-apisix.svc.cluster.local:2379]}","publish-timeout":"7s","error":"etcdserver:
server stopped"}
{"level":"info","ts":"2025-04-15T08:28:33.77063Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"47324e080c98137d
switched to configuration voters=(4171443932729916013 16309538207480531017)"}
{"level":"info","ts":"2025-04-15T08:28:33.770674Z","caller":"membership/cluster.go:472","msg":"removed
member","cluster-id":"b9ae902a9398fd61","local-member-id":"47324e080c98137d","removed-remote-peer-id":"47324e080c98137d","removed-remote-peer-urls":["http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"info","ts":"2025-04-15T08:28:33.770713Z","caller":"rafthttp/peer.go:330","msg":"stopping
remote peer","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.770742Z","caller":"rafthttp/stream.go:294","msg":"stopped
TCP streaming connection with remote peer","stream-writer-type":"unknown
stream","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.77077Z","caller":"rafthttp/stream.go:294","msg":"stopped
TCP streaming connection with remote peer","stream-writer-type":"unknown
stream","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.770808Z","caller":"rafthttp/pipeline.go:85","msg":"stopped
HTTP pipelining with remote
peer","local-member-id":"47324e080c98137d","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.770842Z","caller":"rafthttp/stream.go:442","msg":"stopped
stream reader with remote peer","stream-reader-type":"stream MsgApp
v2","local-member-id":"47324e080c98137d","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.770878Z","caller":"rafthttp/stream.go:442","msg":"stopped
stream reader with remote peer","stream-reader-type":"stream
Message","local-member-id":"47324e080c98137d","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.7709Z","caller":"rafthttp/peer.go:335","msg":"stopped
remote peer","remote-peer-id":"39e3f227d2d33a6d"}
{"level":"info","ts":"2025-04-15T08:28:33.770905Z","caller":"rafthttp/peer.go:330","msg":"stopping
remote peer","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.77091Z","caller":"rafthttp/stream.go:294","msg":"stopped
TCP streaming connection with remote peer","stream-writer-type":"unknown
stream","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.77093Z","caller":"rafthttp/stream.go:294","msg":"stopped
TCP streaming connection with remote peer","stream-writer-type":"unknown
stream","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.770956Z","caller":"rafthttp/pipeline.go:85","msg":"stopped
HTTP pipelining with remote
peer","local-member-id":"47324e080c98137d","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.770964Z","caller":"rafthttp/stream.go:442","msg":"stopped
stream reader with remote peer","stream-reader-type":"stream MsgApp
v2","local-member-id":"47324e080c98137d","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.77098Z","caller":"rafthttp/stream.go:442","msg":"stopped
stream reader with remote peer","stream-reader-type":"stream
Message","local-member-id":"47324e080c98137d","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.770995Z","caller":"rafthttp/peer.go:335","msg":"stopped
remote peer","remote-peer-id":"e2571e968b89c849"}
{"level":"info","ts":"2025-04-15T08:28:33.853325Z","caller":"etcdmain/main.go:44","msg":"notifying
init daemon"}
{"level":"info","ts":"2025-04-15T08:28:33.853372Z","caller":"etcdmain/main.go:50","msg":"successfully
notified init daemon"}
stream closed EOF for ingress-apisix/apisix-etcd-1 (etcd)
```
### Environment
- APISIX version (run `apisix version`): 3.11.0
- Operating system (run `uname -a`): debian
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl
http://127.0.0.1:9090/v1/server_info`): bitnami/etcd:3.5.10-debian-11-r2
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
