GitHub user tuanhoangth1603 added a comment to the discussion: Traefik return
404 page not found - Kubernetes on Cloudstack 4.17.2.0
> any logs on cloudstack management server and in the pod
> `cloud-controller-manager` ?
Thanks for your reply. Here is logs of CCM
```I0821 15:10:39.921244 1 serving.go:313] Generated self-signed cert
in-memory
W0821 15:11:05.221926 1 client_config.go:552] Neither --kubeconfig nor
--master was specified. Using the inClusterConfig. This might not work.
I0821 15:11:05.420413 1 controllermanager.go:120] Version:
v0.0.0-master+$Format:%h$
I0821 15:11:05.422112 1 configmap_cafile_content.go:202] Starting
client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0821 15:11:05.422169 1 shared_informer.go:223] Waiting for caches to
sync for
client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0821 15:11:05.422223 1 configmap_cafile_content.go:202] Starting
client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0821 15:11:05.422240 1 shared_informer.go:223] Waiting for caches to
sync for
client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0821 15:11:05.618715 1 secure_serving.go:178] Serving securely on
[::]:10258
I0821 15:11:05.618813 1 leaderelection.go:242] attempting to acquire
leader lease kube-system/cloud-controller-manager...
I0821 15:11:05.619351 1 tlsconfig.go:240] Starting
DynamicServingCertificateController
I0821 15:11:05.918307 1 shared_informer.go:230] Caches are synced for
client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0821 15:11:05.918307 1 shared_informer.go:230] Caches are synced for
client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0821 15:11:06.018794 1 event.go:278]
Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system",
Name:"cloud-controller-manager", UID:"79917607-9b73-462e-bf49-76a5cb88c24b",
APIVersion:"v1", ResourceVersion:"785", FieldPath:""}): type: 'Normal' reason:
'LeaderElection'
cloud-controller-manager-574bcb86c-fn5r5_957b1aed-2d27-4dac-b6af-418d983f7572
became leader
E0821 15:11:06.018661 1 event.go:316] Could not construct reference to:
'&v1.Lease{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""},
ObjectMeta:v1.ObjectMeta{Name:"cloud-controller-manager", GenerateName:"",
Namespace:"kube-system", SelfLink:"",
UID:"33584655-e620-4127-9ce5-e1ac08e95df8", ResourceVersion:"786",
Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0,
ext:63859849865, loc:(*time.Location)(0x30c2040)}},
DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil),
Labels:map[string]string(nil), Annotations:map[string]string(nil),
OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil),
ClusterName:"",
ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"cloudstack-ccm",
Operation:"Update", APIVersion:"coordination.k8s.io/v1",
Time:(*v1.Time)(0xc0001a6340), FieldsType:"FieldsV1",
FieldsV1:(*v1.FieldsV1)(0xc0001a6380)}}},
Spec:v1.LeaseSpec{HolderIdentity:(*string)(nil),
LeaseDurationSeconds:(*int32)(nil), AcquireTime:(*
v1.MicroTime)(nil), RenewTime:(*v1.MicroTime)(nil),
LeaseTransitions:(*int32)(nil)}}' due to: 'no kind is registered for the type
v1.Lease in scheme "pkg/runtime/scheme.go:101"'. Will not report event:
'Normal' 'LeaderElection'
'cloud-controller-manager-574bcb86c-fn5r5_957b1aed-2d27-4dac-b6af-418d983f7572
became leader'
I0821 15:11:06.118337 1 leaderelection.go:252] successfully acquired
lease kube-system/cloud-controller-manager
I0821 15:11:06.120413 1 node_controller.go:110] Sending events to api
server.
I0821 15:11:06.120516 1 controllermanager.go:247] Started "cloud-node"
I0821 15:11:06.220667 1 node_lifecycle_controller.go:78] Sending events
to api server
I0821 15:11:06.220758 1 controllermanager.go:247] Started
"cloud-node-lifecycle"
I0821 15:11:06.319991 1 controllermanager.go:247] Started "service"
I0821 15:11:06.320047 1 core.go:101] Will not configure cloud provider
routes for allocate-node-cidrs: false, configure-cloud-routes: true.
W0821 15:11:06.320064 1 controllermanager.go:244] Skipping "route"
I0821 15:11:06.418280 1 controller.go:208] Starting service controller
I0821 15:11:06.418313 1 shared_informer.go:223] Waiting for caches to
sync for service
I0821 15:11:06.918866 1 shared_informer.go:230] Caches are synced for
service
E0821 15:11:08.719857 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:11:08.721984 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:11:08.818466 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:11:08.821324 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:11:08.823046 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:11:08.920077 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:16:09.221094 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:16:09.319975 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:16:09.321029 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:16:09.323385 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:16:09.324513 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:16:09.327037 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:21:09.922209 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:21:10.019894 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:21:10.021178 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:21:10.118357 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:21:10.120241 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:21:10.122821 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
I0821 15:25:38.304728 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3023", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:25:38.519534 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:25:38.519646 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3023", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
I0821 15:25:43.520304 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3026", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:25:43.521805 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:25:43.521878 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3026", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
I0821 15:25:53.522501 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3026", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:25:53.524087 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:25:53.524171 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3026", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
E0821 15:26:10.619873 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:26:10.622562 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:26:10.718288 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:26:10.721020 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:26:10.722174 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:26:10.724422 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
I0821 15:26:11.190439 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:26:11.191841 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:26:11.191884 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
I0821 15:26:13.524803 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:26:13.526348 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:26:13.526489 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
I0821 15:27:33.527261 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:27:33.721428 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:27:33.818172 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
I0821 15:30:13.723275 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:30:13.921406 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:30:14.018213 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
E0821 15:31:10.921484 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:31:11.119953 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:31:11.121277 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:31:11.123845 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:31:11.125066 1 node_controller.go:237] error retrieving instance
ID: invalid character '<' looking for beginning of value
E0821 15:31:11.127270 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
```
GitHub link:
https://github.com/apache/cloudstack/discussions/9906#discussioncomment-11187117
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscr...@cloudstack.apache.org
