Re: Issues with cloudstack-kubernetes-provider load balancer

[Kiran manohar Chavala]

Hi David

I think the names are fine ,Please find the kubectl output and screenshot
attached

---
❯ kubectl get nodes
NAME STATUS   ROLES   AGE
VERSION
kiran-kube-cluster-control-18254bced26   Readycontrol-plane   13h
v1.24.0
kiran-kube-cluster-node-18254bf3fef  Ready  13h
v1.24.0

---
Also when I give kubectl get svc is the external-ip expected to appear?

The load balancing is working fine , if i manually give loadbalancing rule
to Nodeport ip (screenshot attached)


---
❯ kubectl get svc
NAME TYPE   CLUSTER-IP   EXTERNAL-IP   PORT(S)
 AGE
kiranLoadBalancer   10.105.243.14580:30905/TCP
12h
kubernetes   ClusterIP  10.96.0.1443/TCP
 13h


❯ kubectl describe svc kiran
Name: kiran
Namespace:default
Labels:   run=nginx
Annotations:  
Selector: run=nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families:  IPv4
IP:   10.105.243.145
IPs:  10.105.243.145
Port:   80/TCP
TargetPort:   80/TCP
NodePort:   30905/TCP
Endpoints:10.44.0.4:80,10.44.0.5:80
Session Affinity: None
External Traffic Policy:  Cluster
Events:
  Type Reason  AgeFrom
   Message
   --     
   ---
  Warning  SyncLoadBalancerFailed  35m (x140 over 12h)
 service-controller  Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: Get
http://10.102.192.220:8080/client/api?apiKey=ih7O6aKn0-LD7FL2Y6H3wfXv0B1iRXQOG7rZc5iip__eNmiRpKx_27dF5sy_6kJ-7OZt7mx7w9wZHu9Pq9bRfA=listLoadBalancerRules=afbeaa6d1594f4e1d9568064fec18b7c=true=json=aaY6yZwMlRgzPicp6y4co1rV%2Bl4%3D:
dial tcp 10.102.192.220:8080: i/o timeout
  Normal   EnsuringLoadBalancer3m27s (x146 over 12h)
 service-controller  Ensuring load balancer

---



On Mon, Aug 1, 2022 at 9:02 AM David Jumani 
wrote:

> Hi Kiran
>
> I've discussed this issue with Pierre offline.
> This happens when the names of the VMs in CloudStack do not match the node
> names in Kubernetes, likely an upper-lower case issue
> A PR to fix it has been raised :
> https://github.com/apache/cloudstack-kubernetes-provider/pull/41
> As a workaround, you can rename the VMs in CloudStack to exactly match the
> output of `kubectl get nodes`
> 
> From: Kiran manohar Chavala 
> Sent: Sunday, July 31, 2022 11:21 AM
> To: users@cloudstack.apache.org 
> Subject: Re: Issues with cloudstack-kubernetes-provider load balancer
>
> Hi Pierre Le Fevre
>
> Facing a similar kind of  issue when trying to acquire Loadbalncer Ip
>
> Any workaround for the issue
>
> ❯ k describe svc kiran
> Name: kiran
> Namespace:default
> Labels:   run=nginx
> Annotations:  
> Selector: run=nginx
> Type: LoadBalancer
> IP Family Policy: SingleStack
> IP Families:  IPv4
> IP:   10.99.211.32
> IPs:  10.99.211.32
> Port:   80/TCP
> TargetPort:   8000/TCP
> NodePort:   31588/TCP
> Endpoints:10.44.0.4:8000,10.44.0.5:8000
> Session Affinity: None
> External Traffic Policy:  Cluster
> Events:
>   Type Reason  Age  From
>  Message
>    --   
>  ---
>   Normal   EnsuringLoadBalancer61s (x2 over 2m31s)  service-controller
>  Ensuring load balancer
>   Warning  SyncLoadBalancerFailed  31s (x2 over 2m1s)   service-controller
>  Error syncing load balancer: failed to ensure load balancer: error
> retrieving load balancer rules: Get
>
> http://10.102.192.220:8080/client/api?apiKey=ih7O6aKn0-LD7FL2Y6H3wfXv0B1iRXQOG7rZc5iip__eNmiRpKx_27dF5sy_6kJ-7OZt7mx7w9wZHu9Pq9bRfA=listLoadBalancerRules=a94511f7024b24efdaae576625a6fe68=true=json=bpLHBJTA1LgLGV9DCevvKshu2Zg%3D
> :
> dial tcp 10.102.192.220:8080: i/o timeout
>
> Regards
> Kiran
>
>
>
>
>
>
>
>
> On Thu, Jun 16, 2022 at 8:33 PM Pierre Le Fevre  wrote:
>
> > Hi all,
> >
> > I have recently been experimenting with Kubernetes in CloudStack and came
> > across a quite odd problem with the communication
> > between cloudstack-kubernetes-provider and the CloudStack API.
> >
> > After some experimentation with the cloud-config api-url variable, it
> seems
> > that connection should be happening. I tried using wget from within
> > the pods with success.
> > However, whenever services try to sync the load balancer to CloudStack,
> the
> > error* Error syncing load balancer: failed to ensure load balancer: could
> > not find network *shows up.
> >
> > Taking a look at the logs of 

Re: Issues with cloudstack-kubernetes-provider load balancer

[David Jumani]

Hi Kiran

I've discussed this issue with Pierre offline.
This happens when the names of the VMs in CloudStack do not match the node 
names in Kubernetes, likely an upper-lower case issue
A PR to fix it has been raised : 
https://github.com/apache/cloudstack-kubernetes-provider/pull/41
As a workaround, you can rename the VMs in CloudStack to exactly match the 
output of `kubectl get nodes`

From: Kiran manohar Chavala 
Sent: Sunday, July 31, 2022 11:21 AM
To: users@cloudstack.apache.org 
Subject: Re: Issues with cloudstack-kubernetes-provider load balancer

Hi Pierre Le Fevre

Facing a similar kind of  issue when trying to acquire Loadbalncer Ip

Any workaround for the issue

❯ k describe svc kiran
Name: kiran
Namespace:default
Labels:   run=nginx
Annotations:  
Selector: run=nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families:  IPv4
IP:   10.99.211.32
IPs:  10.99.211.32
Port:   80/TCP
TargetPort:   8000/TCP
NodePort:   31588/TCP
Endpoints:10.44.0.4:8000,10.44.0.5:8000
Session Affinity: None
External Traffic Policy:  Cluster
Events:
  Type Reason  Age  From
 Message
   --   
 ---
  Normal   EnsuringLoadBalancer61s (x2 over 2m31s)  service-controller
 Ensuring load balancer
  Warning  SyncLoadBalancerFailed  31s (x2 over 2m1s)   service-controller
 Error syncing load balancer: failed to ensure load balancer: error
retrieving load balancer rules: Get
http://10.102.192.220:8080/client/api?apiKey=ih7O6aKn0-LD7FL2Y6H3wfXv0B1iRXQOG7rZc5iip__eNmiRpKx_27dF5sy_6kJ-7OZt7mx7w9wZHu9Pq9bRfA=listLoadBalancerRules=a94511f7024b24efdaae576625a6fe68=true=json=bpLHBJTA1LgLGV9DCevvKshu2Zg%3D:
dial tcp 10.102.192.220:8080: i/o timeout

Regards
Kiran






 

On Thu, Jun 16, 2022 at 8:33 PM Pierre Le Fevre  wrote:

> Hi all,
>
> I have recently been experimenting with Kubernetes in CloudStack and came
> across a quite odd problem with the communication
> between cloudstack-kubernetes-provider and the CloudStack API.
>
> After some experimentation with the cloud-config api-url variable, it seems
> that connection should be happening. I tried using wget from within
> the pods with success.
> However, whenever services try to sync the load balancer to CloudStack, the
> error* Error syncing load balancer: failed to ensure load balancer: could
> not find network *shows up.
>
> Taking a look at the logs of *cloud-controller-manager*, the same three
> rows come up repeatedly:
>
> I0616 14:02:26.189085   1 event.go:278]
> Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"balls",
> UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
> ResourceVersion:"1776", FieldPath:""}): type: 'Normal' reason:
> 'EnsuringLoadBalancer' Ensuring load balancer
>
> E0616 14:02:26.273801   1 controller.go:244] error processing service
> default/balls (will retry): failed to ensure load balancer: could not find
> network
>
> I0616 14:02:26.274430   1 event.go:278]
> Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"balls",
> UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
> ResourceVersion:"1776", FieldPath:""}): type: 'Warning' reason:
> 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
> balancer: could not find network
>
>
> When watching the logs of our management server, the request
> *listNetworks *appears,
> coming from the kubernetes IP. It does, however, include an empty *id
> *parameter ,
> which in turn returns the error:
>
> {
> "listnetworksresponse": {
> "uuidList": [],
> "errorcode": 431,
> "cserrorcode": ,
> "errortext": "Unable to execute API command listnetworks due to
> invalid value. Invalid parameter id value= due to incorrect long value
> format, or entity does not exist or due to incorrect parameter annotation
> for the field in api cmd class."
> }
> }
>
> The request: http://*domain*/client/api?apiKey=*key*=listNetworks;
> *id=*=json=*signature*
>
> Looking at the code of cloudstack-kubernetes-provider
> (cloudstack_loadbalancer.go:417), it seems like the error should return
> *... **could not find network [network ID]*, but in our case, no ID shows
> up (it could be nil?).
> It appears that cs.verifyHosts (cloudstack_loadbalancer.go:100) is failing
> to update lb.networkID, which cascades into this error.
>
> Seeing as it shows up even after creating different clusters, could this
> still be a configuration issue, or is it a known bug in CKS?
>
> Thanks in advance,
> Pierre Le Fevre
>

Re: Unable to login to GUI onto second management server

[Andrei Mikhailovsky]

Hi Harikrishna, 

Tried the below, but still have the same issue. 

also, after trying what you've suggested, I've started the old management 
server and I was still able to login. not sure if the host setting does 
anything login related... 

Andrei 

> From: "Harikrishna Patnala" 
> To: "Andrei Mikhailovsky" 
> Cc: "users" 
> Sent: Thursday, 28 July, 2022 09:54:39
> Subject: Re: Unable to login to GUI onto second management server

> Hi Andrei,

> Can you please also try the below steps? I'm just making sure all pointers are
> to the new management server only.

> 1. Keep only the new management server IP in the host configuration.
> 2. Stop the old management server
> 3. Restart the new management server
> Thanks,
> Harikrishna

> From: Andrei Mikhailovsky 
> Sent: Wednesday, July 27, 2022 6:45 PM
> To: Harikrishna Patnala 
> Cc: users 
> Subject: Re: Unable to login to GUI onto second management server
> Hi Harikrishna,

> I have added the new management server IP address into the host configuration
> from the gui. It now shows:

> host  The ip address of management server. This can also accept comma 
> separated
> addresses.Advanced
> 192.168.169.13,192.168.169.21

> After that I've started the new management server and unfortunately, I still
> have the same issue.

> I have also noticed that after starting the new management server, the table
> mshost has been updated to reflect the server status as Up.:

>| 4 | 115129173025114 | 1658099918669 | ais-cloudhost13.csprdc.arhont.com |
>| 98405826-0861-11ea-a1da-8003fe80 | Up | 4.16.1.0 | 127.0.0.1 | 9090 |
> | 2022-07-27 13:10:05 | NULL | 0 |
>| 5 | 165004275141402 | 1658927302926 | ais-compute1.cloud.arhont.com |
>| 0d1522a5-5d08-46af-b59c-b577aa22e9bb | Up | 4.16.1.0 | 192.168.169.21 | 9090 
>|
> | 2022-07-27 13:08:32 | NULL | 0 |

> Anything else I should try?

> Thanks

> Andrei

>> From: "Harikrishna Patnala" 
>> To: "Andrei Mikhailovsky" , "users"
>> 
>> Sent: Wednesday, 27 July, 2022 07:21:24
>> Subject: Re: Unable to login to GUI onto second management server

>> Hi Andrei,

>> If the purpose of the second management server is about migration please 
>> ignore
>> the previous reply.

>> You have the right pointer to the procedure and I hope you have followed it.

>> Please try to provide the following information.

>> 1. Is the old management server also in the 4.16.1 version?
>>2. Which database.properties file you have changed to point to the new 
>> database
>> ?
>>3. Can you check the database table "configuration", what is the value 
>> for the
>> configuration with the name "host", is it your new MS host address ?
>>4. Also, check the "mshost" table in the database if it is pointing to 
>> the new
>> management server.
>> Regards,
>> Harikrishna

>> From: Andrei Mikhailovsky 
>> Sent: Monday, July 25, 2022 7:46 PM
>> To: users 
>> Cc: Harikrishna Patnala 
>> Subject: Re: Unable to login to GUI onto second management server

>> Hi Harikrishna,

>> Having read the links that you've sent I am not sure that my issues are 
>> related.
>> Perhaps I should have explained my current set up / intensions a bit more. My
>> main reasons for adding the multiple management servers is not to provide the
>> HA / load balancing, but rather to migrate the current management server from
>> old hardware to the new one. I was referring to the post sent by Andrija 
>> Panic
>> ( [ https://www.mail-archive.com/users@cloudstack.apache.org/msg32889.html |
>> https://www.mail-archive.com/users@cloudstack.apache.org/msg32889.html ] )
>> where Andrija has suggested that one should install the second management
>> server, connect it to the database, move the database to a new server and
>> change the database properties to point the new management server to the new
>> db.

>> In my tests, I have installed the second management server without any
>> proxy/load balancing and I tried to connect and authenticate directly to the 
>> IP
>> address of the second management server. I've tried it with the primary
>> management server switched on and off, but I still have the same issues. If I
>> am connecting directly to the new management server IP, I don't see how 
>> having
>> nginx proxy settings changes would fix my issue. Also, I have not seen 
>> anything
>> in the documentation that explicitly requires having a proxy if you install 
>> the
>> second management server.

>> Why do you think my issue relates to CORS?

>> Andrei

>> - Original Message -
>> > From: "Harikrishna Patnala" 
>> > To: "users" 
>> > Sent: Wednesday, 20 July, 2022 05:10:13
>> > Subject: Re: Unable to login to GUI onto second management server

>> > Hi Andrei,

>> > This looks to me like a CORS issue.

>> > Have you set up any load balancer for these management servers. There is a
>> > section
>>> [
>>> http://docs.cloudstack.apache.org/en/4.16.1.0/adminguide/reliability.html#management-server-load-balancing
>> > |
>> 

Re: Changing the connecting hosts when using Ceph as primary

[Nazmul Parvej]

Hi Sven and Wido,

I am facing the same issue with my ACSv4.17. I added my RBD Primary Storage
with Single RADOS monitor IP, Now I want to change it to a round-robin DNS
following are my SoP please correct me if I am wrong.

*1. Change in MySQL database: *There are two tables in the cloud database
called storage_pool and storage_pool_view  I will update both of them to
my round-robin DNS.

*2. Apply the Impact to Hosts:* One by one host restart to reload Libvirt
and the CloudStack agent to the right XML loaded.

Yours sincerely,


Nazmul Parvej


On Fri, Jul 29, 2022 at 7:56 PM Wido den Hollander  wrote:

> Hi,
>
> The easiest way is to reboot your hosts one by one. This will reload
> Libvirt and the CloudStack agent and will make sure you have the right
> XML loaded.
>
> Wido
>
> On 7/27/22 10:22, Sven Barczyk wrote:
> > Hello everyone,
> >
> >
> >
> > i caught myself beeing stupid.
> > While deploying my cluster, I’ve decided to use my mon1.ceph to be the
> > main RBD Host which leads now to the problem, that a restart of mon1
> > will be hell.
> >
> > In despite, i changed the host to a round-robin dns (changed by sql ),
> > which is the suggested method of the documentation and thought all new
> > Instances will be deployed with this new host from the database.
> > But no , still uses the mon1.ceph while deploying new instances (checked
> > with virsh dumpxml on my hosts), instead of the new set host, mons.ceph
> > which round-robins through all my mons.
> >
> > Restart of Cloudstack-management did not help.
> >
> > Is there anyone who could give me an hint where this rbd hosts might be
> > hardcoded in the cloudstack-config ?
> >
> >
> >
> > Best regards,
> > Sven
> >
>

Multiple RADOS monitor IPs to add RBD Primary Storage

[Nazmul Parvej]

Hi There,

I am using ACSv4.17

I added my primary storage using a single mon IP to attach the RBD pool. My
questions are following.

1. How to add multiple RADOS monitor IPs to add RBD Primary Storage?

2. Is there any way to add multiple RADOS monitor IPs to Existingly added
RBD Primary Storage which was a single RADOS monitor IP?


Yours sincerely,


Nazmul Parvej
Deputy Manager, Product Development
IT Division

Bangladesh Export Import Company Ltd.

Level-9, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh

Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14193, Fax:  +880 2 
95757

Cell: +8801787680841, Email: nazmul.par...@bol-online.com, Web:
www.bol-online.com

Re: Can't upload ISO from local

[Vivek Kumar]

Hello,

If you want to upload from local you must offload the SSL certificate in your 
System VMs first. Because it try to upload the file over HTTPs and due to self 
sign or certificate error, it won’t let you upload. 

Vivek Kumar
Sr. Manager - Cloud & DevOps
TechOps | Indiqus Technologies

vivek.ku...@indiqus.com 
www.indiqus.com 




> On 29-Jul-2022, at 11:33 PM, Airamek  wrote:
> 
> Thanks for the idea! I got in trough ssh, and it has some interesting errors.
> 
> Jul 29 17:56:40 s-2-VM systemd[4111]: PAM _pam_load_conf_file: unable to open 
> config for /etc/pam.d/null
> Jul 29 17:56:40 s-2-VM systemd[4111]: PAM error loading (null)
> Jul 29 17:56:40 s-2-VM systemd[4111]: PAM failed: Critical error - immediate 
> abort
> Jul 29 17:56:40 s-2-VM systemd[4111]: PAM _pam_init_handlers: error reading 
> /etc/pam.d/systemd-user
> Jul 29 17:56:40 s-2-VM systemd[4111]: user@0.service: Failed to set up PAM 
> session: Operation not permitted
> Jul 29 17:56:40 s-2-VM systemd[4111]: PAM _pam_init_handlers: [Critical error 
> - immediate abort]
> Jul 29 17:56:40 s-2-VM systemd[4111]: user@0.service: Failed at step PAM 
> spawning /lib/systemd/systemd: Operation not permitted
> ░░ Subject: Process /lib/systemd/systemd could not be executed
> ░░ Defined-By: systemd
> ░░ Support: https://www.debian.org/support
> ░░
> ░░ The process /lib/systemd/systemd could not be executed and failed.
> ░░
> ░░ The error number returned by this process is ERRNO.
> Jul 29 17:56:40 s-2-VM systemd[4111]: PAM error reading PAM configuration file
> Jul 29 17:56:40 s-2-VM systemd[4111]: PAM pam_start: failed to initialize 
> handlers
> Jul 29 17:56:40 s-2-VM systemd[1]: user@0.service: Main process exited, 
> code=exited, status=224/PAM
> 
> Any idea what could cause this?
> 
> On 7/29/2022 7:42 PM, Sven Barczyk wrote:
>> Hi,
>> have you checked you SecStorageVM ?
>> 
>> kind Regards
>> Sven
>> 
>> -Ursprüngliche Nachricht-
>> Von: Airamek 
>> Gesendet: Freitag, 29. Juli 2022 19:30
>> An: users@cloudstack.apache.org
>> Betreff: Can't upload ISO from local
>> 
>> After a lot of trial and error, the system machines are running and their 
>> consoles are accessible, but I can't upload ISO files trough the web 
>> interface. I'm using version 4.17.0.1, and my setup consists of a management 
>> server with sql(MariaDB), and a hypervisor with the primary and secondary 
>> nfs shares running. Both machines are running Debian.
>> 
>> The only error-like log I could find is in the syslog of the management
>> server:
>> 
>> Jul 29 19:19:08 laena java[5707]: WARN
>> [c.c.a.d.ParamGenericValidationWorker] (qtp1648001170-17:ctx-29085af5
>> ctx-89738967) (logid:eafc7337) Received unknown parameters for command 
>> listZones. Unknown parameters : listall Jul 29 19:19:08 laena java[5707]: 
>> WARN [c.c.a.d.ParamGenericValidationWorker] (qtp1648001170-14:ctx-13826793
>> ctx-f1f86014) (logid:dff1eeea) Received unknown parameters for command 
>> listOsTypes. Unknown parameters : listall Jul 29 19:19:30 laena java[5707]: 
>> ERROR [c.c.a.AlertManagerImpl]
>> (CapacityChecker:ctx-8d290200) (logid:c5703391) Caught exception in 
>> recalculating capacity Jul 29 19:19:30 laena java[5707]: 
>> java.lang.NullPointerException Jul 29 19:19:30 laena java[5707]: #011at
>> com.cloud.configuration.ConfigurationManagerImpl.findPodAllocationState(ConfigurationManagerImpl.java:7170)
>> Jul 29 19:19:30 laena java[5707]: #011at 
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> Jul 29 19:19:30 laena java[5707]: #011at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> Jul 29 19:19:30 laena java[5707]: #011at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> Jul 29 19:19:30 laena java[5707]: #011at
>> java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> Jul 29 19:19:30 laena java[5707]: #011at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>> Jul 29 19:19:30 laena java[5707]: #011at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>> Jul 29 19:19:30 laena java[5707]: #011at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>> Jul 29 19:19:30 laena java[5707]: #011at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
>> Jul 29 19:19:30 laena java[5707]: #011at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>> Jul 29 19:19:30 laena java[5707]: #011at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
>> Jul 29 19:19:30 laena java[5707]: #011at 
>> com.sun.proxy.$Proxy122.findPodAllocationState(Unknown Source) Jul