Re: Kubernetes service has not been configured properly to provision Kubernetes clusters

[Pearl d'Silva]

Hi David,

Not being able to select the OS type for VMware hypervisor type is an expected 
behaviour - as stated in the note in the following section: 
http://docs.cloudstack.apache.org/en/latest/adminguide/templates.html#creating-a-template-from-an-existing-virtual-machine
 - which states:
"Since version 4.15, VMware templates do not allow users or administrators 
selecting an OS Type when registering a template. The OS Type is obtained from 
the template after it is registered."
However, the template registration should go through successfully. Can you 
re-try the template registration.

Thanks,
Pearl


From: David Larsen 
Sent: Thursday, April 22, 2021 9:51 PM
To: users@cloudstack.apache.org 
Subject: SV: Kubernetes service has not been configured properly to provision 
Kubernetes clusters

Thanks Pearl.

OK, I see.
I tried to register a new CoreOS ova template from URL, but when I choose 
Hypervisor "VMware",  the "OS Type" disappear and replaced with "Keyboard Type" 
 so I cannot choose OS Type=CoreOS...
Is this normal?
The template register result is "Ready=False" and Status: Did not find a guest 
OS with type otherLinux64Guest


David Larsen

-Opprinnelig melding-
Fra: Pearl d'Silva 
Sendt: torsdag 22. april 2021 16:50
Til: users@cloudstack.apache.org
Emne: Re: Kubernetes service has not been configured properly to provision 
Kubernetes clusters

Hi David,

On CS 4.15, you will still have to register the CoreOS template to setup 
Kubernetes Clusters and register it with the same name as the global setting 
'cloud.kubernetes.cluster.template.name.'.
The ShapeBlue blog that you are probably referring to is something that will be 
made available in the next LTS release.

Thanks,
Pearl

From: David Larsen 
Sent: Thursday, April 22, 2021 7:47 PM
To: users@cloudstack.apache.org 
Subject: Kubernetes service has not been configured properly to provision 
Kubernetes clusters

Hi 

I have now upgraded from CS 4.13 to CS 4.15 and I have tried to configure 
Kubernetes service from the latest Cloudstack documentation page (CloudStack 
Kubernetes Service — Apache CloudStack 4.15.0.0 
documentation)
  with no success…

As I have read from this forum and shapeblue’s pages, the CoreOS template setup 
is not needed anymore. But do I need to configure anything else from the 
GUI/cmd regarding this?

My 4.15 system vm uploaded before upgrading to 4.15 is not marked as “public”… 
is this a problem? Should I delete it and create a new systemvm for 4.15?

I have added Kubernetes versions 1.16.0 and 1.16.3 and they are marked as 
“ready”.
The global parameter in Cloudstack for Kubernetes is enabled.

When I try to create Kubernetes cluster with a test user, I get this error:
Request Failed (530) Kubernetes service has not been configured properly to 
provision Kubernetes clusters



Med vennlig hilsen

David Larsen
Senior systemkonsulent



pearl.dsi...@shapeblue.com
https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.shapeblue.com%2Fdata=04%7C01%7C%7Cd4047d507be447676ca008d9059e31e5%7C1dd023eed2894f208926463c9b991b5f%7C1%7C0%7C637546999272338838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=T%2FaE5I6Ds7PihIHfhbfTRIqPIQHFcRiSBy2a2QOfO%2BE%3Dreserved=0
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK @shapeblue




pearl.dsi...@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 

Re: Issue about Client communication error, please retry later.

[Nicolas Vazquez]

Thanks Gustavo,

I see on the agent logs that might be a problem accessing your primary storage:

2021-04-21 15:48:21,188 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-2:null) (logid:f615d126) Attempting to create storage 
pool e70944bc-e918-37ee-a890-6b66bff7f605 (NetworkFilesystem) in libvirt
2021-04-21 15:48:21,189 WARN  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-2:null) (logid:f615d126) Storage pool 
e70944bc-e918-37ee-a890-6b66bff7f605 was not found running in libvirt. Need to 
create it.
2021-04-21 15:48:21,189 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-2:null) (logid:f615d126) Didn't find an existing storage 
pool e70944bc-e918-37ee-a890-6b66bff7f605 by UUID, checking for pools with 
duplicate paths
2021-04-21 15:48:21,337 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-2:null) (logid:f615d126) Trying to fetch storage pool 
e70944bc-e918-37ee-a890-6b66bff7f605 from libvirt

What type of storage are you using, local storage? If using shared storage, can 
you check by the mount command if you are able to mount the storage pools?

Regards,
Nicolas Vazquez

From: GUSTAVO RAU DE ALMEIDA CALLOU 
Sent: Thursday, April 22, 2021 4:18 PM
To: users@cloudstack.apache.org 
Subject: Re: Issue about Client communication error, please retry later.

Hi Gabriel,

Instead of attaching, I think it might be better to consider the link to
download the logs.

The agent log is available here:
https://www.dropbox.com/s/3350wpexmjoyhd8/agent.log?dl=0

And the management-server:
https://www.dropbox.com/s/7bszzxsfaridaui/management-server.log?dl=0

Any suggestions about how to solve such issue that does not allow us to
view the console?

Best regards,
Gustavo Callou


Em qui., 22 de abr. de 2021 às 10:48, Gabriel Bräscher 
escreveu:

> Hi Gustavo,
>
> I think that the log file has not been attached, can you please
> double-check it? Thanks!
>
> Regards,
> Gabriel.
>
> Em qua., 21 de abr. de 2021 às 19:07, GUSTAVO RAU DE ALMEIDA CALLOU <
> gustavo.cal...@ufrpe.br> escreveu:
>
> > Dear Nicolas Vazquez,
> >
> > Yes, I am trying to configure the cloudstack on a minimal CentOS7 using a
> > VMware player. All (storage, hypervisor and manager) are set in just one
> > machine About the hypervisor, I adopted the KVM.
> >
> > Regarding the logs, I am sending the logs of the manager as well as from
> > the agent. The only error that I was able to find was "connection with
> > libvirtd is broken invalid connection pointer in virconnectgetversion".
> > Anyway, you can find attached to this message both logs.
> >
> > About ssh, I am only able to connect to the cloudstack system, which is
> > running the CentOS. I mean I cannot connect to the instances the I am
> able
> > to create using ssh.
> > I am able to start an instance using UbuntuServer.iso, but I cannot
> > install the operating system because once I click on the "console view",
> > the connection only works for a couple of seconds. All the time I lose
> the
> > connection.
> >
> > Let me know any suggestions about how to proceed to solve this problem.
> >
> > Please, let me know if it is possible to proceed to try to configure the
> > Cloudstack running in a VMware player for class lessons only.
> >
> > Best regards,
> > Gustavo
> >
> > Em qua., 21 de abr. de 2021 às 11:43, Nicolas Vazquez <
> > nicolas.vazq...@shapeblue.com> escreveu:
> >
> >> Hi Gustavo,
> >>
> >> Correct me if I'm wrong, I read you installed your management server on
> >> CentOS 7, what about the hypervisor side? Can you check if the
> management
> >> server logs provide more information about the error or is it a
> >> communications error between the management server and hypervisor?
> >> If you have managed to deploy some VMs you can enable the ssh ports from
> >> the network configuration tab.
> >>
> >> Regards,
> >> Nicolas Vazquez
> >> 
> >> From: GUSTAVO RAU DE ALMEIDA CALLOU 
> >> Sent: Tuesday, April 20, 2021 10:58 PM
> >> To: users@cloudstack.apache.org 
> >> Subject: Issue about Client communication error, please retry later.
> >>
> >> Dear Cloudstack support team,
> >>
> >> I have been trying to configure Cloudstack 4.13.1 (LTS) in a VMware KVM
> >> (60GB HD, 8GB of RAM, 8 processors) in the CentOS7.
> >> I got success to configure and install the whole package.
> >>
> >> However, once I have been trying to access the console view I have been
> >> receiving the following error message: "Client communication error,
> please
> >> retry later."  I mean, the OS opens and after a couple of seconds, I
> >> receive this error. I am receiving this message for both the instance
> VMs
> >> and system VM.
> >>
> >> Besides that, I tried to access, through the remote system application
> >> (eg.
> >> ssh), the VMs without success.
> >>
> >> Please let me know asap how to process to solve such issue.
> >>
> >> Best regards,
> >>
> >> Gustavo Callou
> >> and
> >> Alison 

Re: Issue about Client communication error, please retry later.

[GUSTAVO RAU DE ALMEIDA CALLOU]

Hi Gabriel,

Instead of attaching, I think it might be better to consider the link to
download the logs.

The agent log is available here:
https://www.dropbox.com/s/3350wpexmjoyhd8/agent.log?dl=0

And the management-server:
https://www.dropbox.com/s/7bszzxsfaridaui/management-server.log?dl=0

Any suggestions about how to solve such issue that does not allow us to
view the console?

Best regards,
Gustavo Callou


Em qui., 22 de abr. de 2021 às 10:48, Gabriel Bräscher 
escreveu:

> Hi Gustavo,
>
> I think that the log file has not been attached, can you please
> double-check it? Thanks!
>
> Regards,
> Gabriel.
>
> Em qua., 21 de abr. de 2021 às 19:07, GUSTAVO RAU DE ALMEIDA CALLOU <
> gustavo.cal...@ufrpe.br> escreveu:
>
> > Dear Nicolas Vazquez,
> >
> > Yes, I am trying to configure the cloudstack on a minimal CentOS7 using a
> > VMware player. All (storage, hypervisor and manager) are set in just one
> > machine About the hypervisor, I adopted the KVM.
> >
> > Regarding the logs, I am sending the logs of the manager as well as from
> > the agent. The only error that I was able to find was "connection with
> > libvirtd is broken invalid connection pointer in virconnectgetversion".
> > Anyway, you can find attached to this message both logs.
> >
> > About ssh, I am only able to connect to the cloudstack system, which is
> > running the CentOS. I mean I cannot connect to the instances the I am
> able
> > to create using ssh.
> > I am able to start an instance using UbuntuServer.iso, but I cannot
> > install the operating system because once I click on the "console view",
> > the connection only works for a couple of seconds. All the time I lose
> the
> > connection.
> >
> > Let me know any suggestions about how to proceed to solve this problem.
> >
> > Please, let me know if it is possible to proceed to try to configure the
> > Cloudstack running in a VMware player for class lessons only.
> >
> > Best regards,
> > Gustavo
> >
> > Em qua., 21 de abr. de 2021 às 11:43, Nicolas Vazquez <
> > nicolas.vazq...@shapeblue.com> escreveu:
> >
> >> Hi Gustavo,
> >>
> >> Correct me if I'm wrong, I read you installed your management server on
> >> CentOS 7, what about the hypervisor side? Can you check if the
> management
> >> server logs provide more information about the error or is it a
> >> communications error between the management server and hypervisor?
> >> If you have managed to deploy some VMs you can enable the ssh ports from
> >> the network configuration tab.
> >>
> >> Regards,
> >> Nicolas Vazquez
> >> 
> >> From: GUSTAVO RAU DE ALMEIDA CALLOU 
> >> Sent: Tuesday, April 20, 2021 10:58 PM
> >> To: users@cloudstack.apache.org 
> >> Subject: Issue about Client communication error, please retry later.
> >>
> >> Dear Cloudstack support team,
> >>
> >> I have been trying to configure Cloudstack 4.13.1 (LTS) in a VMware KVM
> >> (60GB HD, 8GB of RAM, 8 processors) in the CentOS7.
> >> I got success to configure and install the whole package.
> >>
> >> However, once I have been trying to access the console view I have been
> >> receiving the following error message: "Client communication error,
> please
> >> retry later."  I mean, the OS opens and after a couple of seconds, I
> >> receive this error. I am receiving this message for both the instance
> VMs
> >> and system VM.
> >>
> >> Besides that, I tried to access, through the remote system application
> >> (eg.
> >> ssh), the VMs without success.
> >>
> >> Please let me know asap how to process to solve such issue.
> >>
> >> Best regards,
> >>
> >> Gustavo Callou
> >> and
> >> Alison Vinicius
> >> --
> >> --
> >> Professor at DC / UFRPE, Brazil
> >> PhD in Computer Science at UFPE, Brazil
> >> http://callou.pythonanywhere.com/
> >>
> >> nicolas.vazq...@shapeblue.com
> >> www.shapeblue.com
> >> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> >> @shapeblue
> >>
> >>
> >>
> >>
> >
> > --
> > --
> > Professor at DC / UFRPE, Brazil
> > PhD in Computer Science at UFPE, Brazil
> > http://callou.pythonanywhere.com/
> >
>


-- 
-- 
Professor at DC / UFRPE, Brazil
PhD in Computer Science at UFPE, Brazil
http://callou.pythonanywhere.com/

Re: Cannot start VMs due to Libvirt exception in agent

[Fariborz Navidan]

 Hi Everybody,

I just fixed the issue :) As per mentioned in the link below, the issue was
systemd-machined daemon which tracks locally running virtual machines. I
just restarted it as well as restarted libvirtd and cloudstack-agent and it
started to work fine.

https://www.softaculous.com/board/index.php?tid=14450

Thanks

On Thu, Apr 22, 2021 at 9:35 PM Fariborz Navidan 
wrote:

> Hello,
>
> I'm running 4.15.0.0
>
> On Thu, Apr 22, 2021 at 9:05 PM Gabriel Bräscher 
> wrote:
>
>> Hi Fariborz,
>>
>> What version of CloudStack are you running?
>>
>> Em qui., 22 de abr. de 2021 às 13:14, Fariborz Navidan <
>> mdvlinqu...@gmail.com> escreveu:
>>
>> > Hello All,
>> >
>> > I am not able to start any stopped VM. Agent and Libvirtd are both
>> running.
>> > However, I see the following exception in agent.log:
>> >
>> > This is the exception I found in the agent.log:
>> >
>> > 2021-04-22 12:21:29,045 WARN  [kvm.resource.LibvirtKvmAgentHook]
>> > (agentRequest-Handler-1:null) (logid:4d875f04) Groovy scripting engine
>> is
>> > not initialized. Data transformation skipped.
>> > 2021-04-22 12:21:47,260 WARN
>> [resource.wrapper.LibvirtStartCommandWrapper]
>> > (agentRequest-Handler-4:null) (logid:1a2bdac6) LibvirtException
>> > org.libvirt.LibvirtException: Failed to activate service
>> > 'org.freedesktop.machine1': timed out
>> > at org.libvirt.ErrorHandler.processError(Unknown Source)
>> > at org.libvirt.Connect.processError(Unknown Source)
>> > at org.libvirt.Connect.processError(Unknown Source)
>> > at org.libvirt.Connect.domainCreateXML(Unknown Source)
>> > at
>> >
>> >
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611)
>> > at
>> >
>> >
>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85)
>> > at
>> >
>> >
>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:45)
>> > at
>> >
>> >
>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
>> > at
>> >
>> >
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643)
>> > at com.cloud.agent.Agent.processRequest(Agent.java:661)
>> > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
>> > at com.cloud.utils.nio.Task.call(Task.java:83)
>> > at com.cloud.utils.nio.Task.call(Task.java:29)
>> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> > at
>> >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> > at
>> >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> > at java.base/java.lang.Thread.run(Thread.java:834)
>> >
>> >
>> > Please help me.
>> >
>> > Thanks
>> >
>>
>

Re: Cannot start stopped VMs

[Fariborz Navidan]

Hi Everybody,

I just fixed the issue :) As per mentioned in the link below, the issue was
systemd-machined daemon which tracks locally running virtual machines. I
just restarted it as well as restarted libvirtd and cloudstack-agent and it
started to work fine.

https://www.softaculous.com/board/index.php?tid=14450

Thanks

On Thu, Apr 22, 2021 at 9:57 PM Fariborz Navidan 
wrote:

> Hi Nicolas,
>
> I just updated the SSL certificate for the hostname of the host with a
> valid Let's Encrypt cert but the host is still in Unsecure state. I think
> it is something to do with the Also VMs still cannot start.
>
> Thanks
>
> On Thu, Apr 22, 2021 at 9:39 PM Nicolas Vazquez <
> nicolas.vazq...@shapeblue.com> wrote:
>
>> Hi Fariborz,
>>
>> I've been taking a look at the thread - you mentioned the hosts are in
>> 'Unsecure' state. You should use the provision certificate API to secure
>> the hosts (can do it from the UI as well). Can you try if that solves your
>> issue?
>>
>> Regards,
>> Nicolas Vazquez
>> 
>> From: Fariborz Navidan 
>> Sent: Thursday, April 22, 2021 9:12 AM
>> To: users@cloudstack.apache.org 
>> Subject: Re: Cannot start stopped VMs
>>
>> Anyone can help me?
>>
>> On Wed, Apr 21, 2021 at 7:46 PM Fariborz Navidan 
>> wrote:
>>
>> > Any chance to identify the reason?
>> >
>> > Thanks
>> >
>> > On Wed, Apr 21, 2021 at 1:54 AM Fariborz Navidan > >
>> > wrote:
>> >
>> >> Hi.
>> >> \Now there is no "Connection Closed" error in agent.log nut it is still
>> >> unable to start VMs. I have replaced agent.log with new one at
>> >> https://r9host.com/log/agent.log
>> >>
>> >> Kindly please check it for me
>> >> Thanks
>> >>
>> >> On Tue, Apr 20, 2021 at 8:01 PM Fariborz Navidan <
>> mdvlinqu...@gmail.com>
>> >> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Thank you for  your assistance. Libvirtd and cloudstack-agent services
>> >>> are running and are active. "netstat -t | grep 8250" shows two
>> ESTABLISHED
>> >>> connections. Also in the infrastructure section, the host's status is
>> shown
>> >>> as "Unsecured".
>> >>>
>> >>> Thanks
>> >>>
>> >>> On Tue, Apr 20, 2021 at 9:36 AM Harikrishna Patnala <
>> >>> harikrishna.patn...@shapeblue.com> wrote:
>> >>>
>>  Hi,
>> 
>>  At first glance on the logs, I could see below reasons for the start
>> VM
>>  failures
>> 
>>  management server logs:
>>  2021-04-19 18:31:13,539 INFO  [c.c.v.VirtualMachineManagerImpl]
>>  (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3)
>>  (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to
>> Connection
>>  is closed
>> 
>>  agent logs:
>>  2021-04-19 18:31:25,017 WARN  [kvm.resource.LibvirtKvmAgentHook]
>>  (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting
>> engine is
>>  not initialized. Data transformation skipped.
>>  2021-04-19 18:31:25,686 WARN
>>  [resource.wrapper.LibvirtStartCommandWrapper]
>> (agentRequest-Handler-1:null)
>>  (logid:92f2a447) LibvirtException
>>  org.libvirt.LibvirtException: Connection is closed
>>  at org.libvirt.ErrorHandler.processError(Unknown Source)
>>  at org.libvirt.Connect.processError(Unknown Source)
>>  at org.libvirt.Connect.processError(Unknown Source)
>>  at org.libvirt.Connect.domainCreateXML(Unknown Source)
>>  at
>> 
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611)
>>  at
>> 
>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85)
>> 
>> 
>>  Can you check if libvirtd service running on your KVM host? Also, you
>>  can do a quick KVM agent restart and see if it works "systemctl
>> restart
>>  cloudstack-agent"
>> 
>>  Regards,
>>  Harikrishna
>>  
>>  From: Fariborz Navidan 
>>  Sent: Tuesday, April 20, 2021 12:22 AM
>>  To: users@cloudstack.apache.org 
>>  Subject: Cannot start stopped VMs
>> 
>>  Hello,
>> 
>>  Today, I just updated SSL certificate and it failed re-deploying
>> system
>>  VMs. Also, when I try to start a VM that is stopped, it fails with
>>  message:
>>  Unable to create deployment due to insufficient capacity. I couldn't
>>  find
>>  the reason from logs. I have uploaded  partial management server and
>>  agent
>>  logs.
>> 
>>  https://www.r9host.com/log/management-server.log
>>  https://www.r9host.com/log/agent.log
>> 
>>  It is a mission critical issue for us. I would appreciate if you guys
>>  check
>>  the logs and let us know what we can do to fix this issue.
>> 
>>  Thanks.
>> 
>>  harikrishna.patn...@shapeblue.com
>>  www.shapeblue.com
>>  3 London Bridge Street,  3rd floor, News Building, London  SE1 9SG
>>  @shapeblue

Re: Cannot start stopped VMs

[Fariborz Navidan]

Hi Nicolas,

I just updated the SSL certificate for the hostname of the host with a
valid Let's Encrypt cert but the host is still in Unsecure state. I think
it is something to do with the Also VMs still cannot start.

Thanks

On Thu, Apr 22, 2021 at 9:39 PM Nicolas Vazquez <
nicolas.vazq...@shapeblue.com> wrote:

> Hi Fariborz,
>
> I've been taking a look at the thread - you mentioned the hosts are in
> 'Unsecure' state. You should use the provision certificate API to secure
> the hosts (can do it from the UI as well). Can you try if that solves your
> issue?
>
> Regards,
> Nicolas Vazquez
> 
> From: Fariborz Navidan 
> Sent: Thursday, April 22, 2021 9:12 AM
> To: users@cloudstack.apache.org 
> Subject: Re: Cannot start stopped VMs
>
> Anyone can help me?
>
> On Wed, Apr 21, 2021 at 7:46 PM Fariborz Navidan 
> wrote:
>
> > Any chance to identify the reason?
> >
> > Thanks
> >
> > On Wed, Apr 21, 2021 at 1:54 AM Fariborz Navidan 
> > wrote:
> >
> >> Hi.
> >> \Now there is no "Connection Closed" error in agent.log nut it is still
> >> unable to start VMs. I have replaced agent.log with new one at
> >> https://r9host.com/log/agent.log
> >>
> >> Kindly please check it for me
> >> Thanks
> >>
> >> On Tue, Apr 20, 2021 at 8:01 PM Fariborz Navidan  >
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Thank you for  your assistance. Libvirtd and cloudstack-agent services
> >>> are running and are active. "netstat -t | grep 8250" shows two
> ESTABLISHED
> >>> connections. Also in the infrastructure section, the host's status is
> shown
> >>> as "Unsecured".
> >>>
> >>> Thanks
> >>>
> >>> On Tue, Apr 20, 2021 at 9:36 AM Harikrishna Patnala <
> >>> harikrishna.patn...@shapeblue.com> wrote:
> >>>
>  Hi,
> 
>  At first glance on the logs, I could see below reasons for the start
> VM
>  failures
> 
>  management server logs:
>  2021-04-19 18:31:13,539 INFO  [c.c.v.VirtualMachineManagerImpl]
>  (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3)
>  (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to
> Connection
>  is closed
> 
>  agent logs:
>  2021-04-19 18:31:25,017 WARN  [kvm.resource.LibvirtKvmAgentHook]
>  (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting
> engine is
>  not initialized. Data transformation skipped.
>  2021-04-19 18:31:25,686 WARN
>  [resource.wrapper.LibvirtStartCommandWrapper]
> (agentRequest-Handler-1:null)
>  (logid:92f2a447) LibvirtException
>  org.libvirt.LibvirtException: Connection is closed
>  at org.libvirt.ErrorHandler.processError(Unknown Source)
>  at org.libvirt.Connect.processError(Unknown Source)
>  at org.libvirt.Connect.processError(Unknown Source)
>  at org.libvirt.Connect.domainCreateXML(Unknown Source)
>  at
> 
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611)
>  at
> 
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85)
> 
> 
>  Can you check if libvirtd service running on your KVM host? Also, you
>  can do a quick KVM agent restart and see if it works "systemctl
> restart
>  cloudstack-agent"
> 
>  Regards,
>  Harikrishna
>  
>  From: Fariborz Navidan 
>  Sent: Tuesday, April 20, 2021 12:22 AM
>  To: users@cloudstack.apache.org 
>  Subject: Cannot start stopped VMs
> 
>  Hello,
> 
>  Today, I just updated SSL certificate and it failed re-deploying
> system
>  VMs. Also, when I try to start a VM that is stopped, it fails with
>  message:
>  Unable to create deployment due to insufficient capacity. I couldn't
>  find
>  the reason from logs. I have uploaded  partial management server and
>  agent
>  logs.
> 
>  https://www.r9host.com/log/management-server.log
>  https://www.r9host.com/log/agent.log
> 
>  It is a mission critical issue for us. I would appreciate if you guys
>  check
>  the logs and let us know what we can do to fix this issue.
> 
>  Thanks.
> 
>  harikrishna.patn...@shapeblue.com
>  www.shapeblue.com
>  3 London Bridge Street,  3rd floor, News Building, London  SE1 9SG
>  @shapeblue
> 
> 
> 
> 
>
> nicolas.vazq...@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> @shapeblue
>
>
>
>

Re: Cannot start stopped VMs

[Nicolas Vazquez]

Hi Fariborz,

I've been taking a look at the thread - you mentioned the hosts are in 
'Unsecure' state. You should use the provision certificate API to secure the 
hosts (can do it from the UI as well). Can you try if that solves your issue?

Regards,
Nicolas Vazquez

From: Fariborz Navidan 
Sent: Thursday, April 22, 2021 9:12 AM
To: users@cloudstack.apache.org 
Subject: Re: Cannot start stopped VMs

Anyone can help me?

On Wed, Apr 21, 2021 at 7:46 PM Fariborz Navidan 
wrote:

> Any chance to identify the reason?
>
> Thanks
>
> On Wed, Apr 21, 2021 at 1:54 AM Fariborz Navidan 
> wrote:
>
>> Hi.
>> \Now there is no "Connection Closed" error in agent.log nut it is still
>> unable to start VMs. I have replaced agent.log with new one at
>> https://r9host.com/log/agent.log
>>
>> Kindly please check it for me
>> Thanks
>>
>> On Tue, Apr 20, 2021 at 8:01 PM Fariborz Navidan 
>> wrote:
>>
>>> Hi,
>>>
>>> Thank you for  your assistance. Libvirtd and cloudstack-agent services
>>> are running and are active. "netstat -t | grep 8250" shows two ESTABLISHED
>>> connections. Also in the infrastructure section, the host's status is shown
>>> as "Unsecured".
>>>
>>> Thanks
>>>
>>> On Tue, Apr 20, 2021 at 9:36 AM Harikrishna Patnala <
>>> harikrishna.patn...@shapeblue.com> wrote:
>>>
 Hi,

 At first glance on the logs, I could see below reasons for the start VM
 failures

 management server logs:
 2021-04-19 18:31:13,539 INFO  [c.c.v.VirtualMachineManagerImpl]
 (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3)
 (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to Connection
 is closed

 agent logs:
 2021-04-19 18:31:25,017 WARN  [kvm.resource.LibvirtKvmAgentHook]
 (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting engine is
 not initialized. Data transformation skipped.
 2021-04-19 18:31:25,686 WARN
 [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-1:null)
 (logid:92f2a447) LibvirtException
 org.libvirt.LibvirtException: Connection is closed
 at org.libvirt.ErrorHandler.processError(Unknown Source)
 at org.libvirt.Connect.processError(Unknown Source)
 at org.libvirt.Connect.processError(Unknown Source)
 at org.libvirt.Connect.domainCreateXML(Unknown Source)
 at
 com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611)
 at
 com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85)


 Can you check if libvirtd service running on your KVM host? Also, you
 can do a quick KVM agent restart and see if it works "systemctl restart
 cloudstack-agent"

 Regards,
 Harikrishna
 
 From: Fariborz Navidan 
 Sent: Tuesday, April 20, 2021 12:22 AM
 To: users@cloudstack.apache.org 
 Subject: Cannot start stopped VMs

 Hello,

 Today, I just updated SSL certificate and it failed re-deploying system
 VMs. Also, when I try to start a VM that is stopped, it fails with
 message:
 Unable to create deployment due to insufficient capacity. I couldn't
 find
 the reason from logs. I have uploaded  partial management server and
 agent
 logs.

 https://www.r9host.com/log/management-server.log
 https://www.r9host.com/log/agent.log

 It is a mission critical issue for us. I would appreciate if you guys
 check
 the logs and let us know what we can do to fix this issue.

 Thanks.

 harikrishna.patn...@shapeblue.com
 www.shapeblue.com
 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SG
 @shapeblue





nicolas.vazq...@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 

Re: Cannot start VMs due to Libvirt exception in agent

[Fariborz Navidan]

Hello,

I'm running 4.15.0.0

On Thu, Apr 22, 2021 at 9:05 PM Gabriel Bräscher 
wrote:

> Hi Fariborz,
>
> What version of CloudStack are you running?
>
> Em qui., 22 de abr. de 2021 às 13:14, Fariborz Navidan <
> mdvlinqu...@gmail.com> escreveu:
>
> > Hello All,
> >
> > I am not able to start any stopped VM. Agent and Libvirtd are both
> running.
> > However, I see the following exception in agent.log:
> >
> > This is the exception I found in the agent.log:
> >
> > 2021-04-22 12:21:29,045 WARN  [kvm.resource.LibvirtKvmAgentHook]
> > (agentRequest-Handler-1:null) (logid:4d875f04) Groovy scripting engine is
> > not initialized. Data transformation skipped.
> > 2021-04-22 12:21:47,260 WARN
> [resource.wrapper.LibvirtStartCommandWrapper]
> > (agentRequest-Handler-4:null) (logid:1a2bdac6) LibvirtException
> > org.libvirt.LibvirtException: Failed to activate service
> > 'org.freedesktop.machine1': timed out
> > at org.libvirt.ErrorHandler.processError(Unknown Source)
> > at org.libvirt.Connect.processError(Unknown Source)
> > at org.libvirt.Connect.processError(Unknown Source)
> > at org.libvirt.Connect.domainCreateXML(Unknown Source)
> > at
> >
> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611)
> > at
> >
> >
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85)
> > at
> >
> >
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:45)
> > at
> >
> >
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
> > at
> >
> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643)
> > at com.cloud.agent.Agent.processRequest(Agent.java:661)
> > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
> > at com.cloud.utils.nio.Task.call(Task.java:83)
> > at com.cloud.utils.nio.Task.call(Task.java:29)
> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:834)
> >
> >
> > Please help me.
> >
> > Thanks
> >
>

Re: Cannot start VMs due to Libvirt exception in agent

[Gabriel Bräscher]

Hi Fariborz,

What version of CloudStack are you running?

Em qui., 22 de abr. de 2021 às 13:14, Fariborz Navidan <
mdvlinqu...@gmail.com> escreveu:

> Hello All,
>
> I am not able to start any stopped VM. Agent and Libvirtd are both running.
> However, I see the following exception in agent.log:
>
> This is the exception I found in the agent.log:
>
> 2021-04-22 12:21:29,045 WARN  [kvm.resource.LibvirtKvmAgentHook]
> (agentRequest-Handler-1:null) (logid:4d875f04) Groovy scripting engine is
> not initialized. Data transformation skipped.
> 2021-04-22 12:21:47,260 WARN  [resource.wrapper.LibvirtStartCommandWrapper]
> (agentRequest-Handler-4:null) (logid:1a2bdac6) LibvirtException
> org.libvirt.LibvirtException: Failed to activate service
> 'org.freedesktop.machine1': timed out
> at org.libvirt.ErrorHandler.processError(Unknown Source)
> at org.libvirt.Connect.processError(Unknown Source)
> at org.libvirt.Connect.processError(Unknown Source)
> at org.libvirt.Connect.domainCreateXML(Unknown Source)
> at
>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611)
> at
>
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85)
> at
>
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:45)
> at
>
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
> at
>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643)
> at com.cloud.agent.Agent.processRequest(Agent.java:661)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
> at com.cloud.utils.nio.Task.call(Task.java:83)
> at com.cloud.utils.nio.Task.call(Task.java:29)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
>
>
> Please help me.
>
> Thanks
>

SV: Kubernetes service has not been configured properly to provision Kubernetes clusters

[David Larsen]

Thanks Pearl.

OK, I see. 
I tried to register a new CoreOS ova template from URL, but when I choose 
Hypervisor "VMware",  the "OS Type" disappear and replaced with "Keyboard Type" 
 so I cannot choose OS Type=CoreOS... 
Is this normal?
The template register result is "Ready=False" and Status: Did not find a guest 
OS with type otherLinux64Guest


David Larsen 

-Opprinnelig melding-
Fra: Pearl d'Silva  
Sendt: torsdag 22. april 2021 16:50
Til: users@cloudstack.apache.org
Emne: Re: Kubernetes service has not been configured properly to provision 
Kubernetes clusters

Hi David,

On CS 4.15, you will still have to register the CoreOS template to setup 
Kubernetes Clusters and register it with the same name as the global setting 
'cloud.kubernetes.cluster.template.name.'.
The ShapeBlue blog that you are probably referring to is something that will be 
made available in the next LTS release.

Thanks,
Pearl

From: David Larsen 
Sent: Thursday, April 22, 2021 7:47 PM
To: users@cloudstack.apache.org 
Subject: Kubernetes service has not been configured properly to provision 
Kubernetes clusters

Hi 

I have now upgraded from CS 4.13 to CS 4.15 and I have tried to configure 
Kubernetes service from the latest Cloudstack documentation page (CloudStack 
Kubernetes Service — Apache CloudStack 4.15.0.0 
documentation)
  with no success…

As I have read from this forum and shapeblue’s pages, the CoreOS template setup 
is not needed anymore. But do I need to configure anything else from the 
GUI/cmd regarding this?

My 4.15 system vm uploaded before upgrading to 4.15 is not marked as “public”… 
is this a problem? Should I delete it and create a new systemvm for 4.15?

I have added Kubernetes versions 1.16.0 and 1.16.3 and they are marked as 
“ready”.
The global parameter in Cloudstack for Kubernetes is enabled.

When I try to create Kubernetes cluster with a test user, I get this error:
Request Failed (530) Kubernetes service has not been configured properly to 
provision Kubernetes clusters



Med vennlig hilsen

David Larsen
Senior systemkonsulent



pearl.dsi...@shapeblue.com
https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.shapeblue.com%2Fdata=04%7C01%7C%7Cd4047d507be447676ca008d9059e31e5%7C1dd023eed2894f208926463c9b991b5f%7C1%7C0%7C637546999272338838%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=T%2FaE5I6Ds7PihIHfhbfTRIqPIQHFcRiSBy2a2QOfO%2BE%3Dreserved=0
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK @shapeblue
  
 

Cannot start VMs due to Libvirt exception in agent

[Fariborz Navidan]

Hello All,

I am not able to start any stopped VM. Agent and Libvirtd are both running.
However, I see the following exception in agent.log:

This is the exception I found in the agent.log:

2021-04-22 12:21:29,045 WARN  [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-1:null) (logid:4d875f04) Groovy scripting engine is
not initialized. Data transformation skipped.
2021-04-22 12:21:47,260 WARN  [resource.wrapper.LibvirtStartCommandWrapper]
(agentRequest-Handler-4:null) (logid:1a2bdac6) LibvirtException
org.libvirt.LibvirtException: Failed to activate service
'org.freedesktop.machine1': timed out
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:45)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643)
at com.cloud.agent.Agent.processRequest(Agent.java:661)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)


Please help me.

Thanks

Re: Kubernetes service has not been configured properly to provision Kubernetes clusters

[Pearl d'Silva]

Hi David,

On CS 4.15, you will still have to register the CoreOS template to setup 
Kubernetes Clusters and register it with the same name as the global setting 
'cloud.kubernetes.cluster.template.name.'.
The ShapeBlue blog that you are probably referring to is something that will be 
made available in the next LTS release.

Thanks,
Pearl

From: David Larsen 
Sent: Thursday, April 22, 2021 7:47 PM
To: users@cloudstack.apache.org 
Subject: Kubernetes service has not been configured properly to provision 
Kubernetes clusters

Hi 

I have now upgraded from CS 4.13 to CS 4.15 and I have tried to configure 
Kubernetes service from the latest Cloudstack documentation page (CloudStack 
Kubernetes Service — Apache CloudStack 4.15.0.0 
documentation)
  with no success…

As I have read from this forum and shapeblue’s pages, the CoreOS template setup 
is not needed anymore. But do I need to configure anything else from the 
GUI/cmd regarding this?

My 4.15 system vm uploaded before upgrading to 4.15 is not marked as “public”… 
is this a problem? Should I delete it and create a new systemvm for 4.15?

I have added Kubernetes versions 1.16.0 and 1.16.3 and they are marked as 
“ready”.
The global parameter in Cloudstack for Kubernetes is enabled.

When I try to create Kubernetes cluster with a test user, I get this error:
Request Failed (530) Kubernetes service has not been configured properly to 
provision Kubernetes clusters



Med vennlig hilsen

David Larsen
Senior systemkonsulent



pearl.dsi...@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 

Kubernetes service has not been configured properly to provision Kubernetes clusters

[David Larsen]

Hi 

I have now upgraded from CS 4.13 to CS 4.15 and I have tried to configure 
Kubernetes service from the latest Cloudstack documentation page (CloudStack 
Kubernetes Service — Apache CloudStack 4.15.0.0 
documentation)
  with no success…

As I have read from this forum and shapeblue’s pages, the CoreOS template setup 
is not needed anymore. But do I need to configure anything else from the 
GUI/cmd regarding this?

My 4.15 system vm uploaded before upgrading to 4.15 is not marked as “public”… 
is this a problem? Should I delete it and create a new systemvm for 4.15?

I have added Kubernetes versions 1.16.0 and 1.16.3 and they are marked as 
“ready”.
The global parameter in Cloudstack for Kubernetes is enabled.

When I try to create Kubernetes cluster with a test user, I get this error:
Request Failed (530) Kubernetes service has not been configured properly to 
provision Kubernetes clusters



Med vennlig hilsen

David Larsen
Senior systemkonsulent

Re: Issue about Client communication error, please retry later.

[Gabriel Bräscher]

Hi Gustavo,

I think that the log file has not been attached, can you please
double-check it? Thanks!

Regards,
Gabriel.

Em qua., 21 de abr. de 2021 às 19:07, GUSTAVO RAU DE ALMEIDA CALLOU <
gustavo.cal...@ufrpe.br> escreveu:

> Dear Nicolas Vazquez,
>
> Yes, I am trying to configure the cloudstack on a minimal CentOS7 using a
> VMware player. All (storage, hypervisor and manager) are set in just one
> machine About the hypervisor, I adopted the KVM.
>
> Regarding the logs, I am sending the logs of the manager as well as from
> the agent. The only error that I was able to find was "connection with
> libvirtd is broken invalid connection pointer in virconnectgetversion".
> Anyway, you can find attached to this message both logs.
>
> About ssh, I am only able to connect to the cloudstack system, which is
> running the CentOS. I mean I cannot connect to the instances the I am able
> to create using ssh.
> I am able to start an instance using UbuntuServer.iso, but I cannot
> install the operating system because once I click on the "console view",
> the connection only works for a couple of seconds. All the time I lose the
> connection.
>
> Let me know any suggestions about how to proceed to solve this problem.
>
> Please, let me know if it is possible to proceed to try to configure the
> Cloudstack running in a VMware player for class lessons only.
>
> Best regards,
> Gustavo
>
> Em qua., 21 de abr. de 2021 às 11:43, Nicolas Vazquez <
> nicolas.vazq...@shapeblue.com> escreveu:
>
>> Hi Gustavo,
>>
>> Correct me if I'm wrong, I read you installed your management server on
>> CentOS 7, what about the hypervisor side? Can you check if the management
>> server logs provide more information about the error or is it a
>> communications error between the management server and hypervisor?
>> If you have managed to deploy some VMs you can enable the ssh ports from
>> the network configuration tab.
>>
>> Regards,
>> Nicolas Vazquez
>> 
>> From: GUSTAVO RAU DE ALMEIDA CALLOU 
>> Sent: Tuesday, April 20, 2021 10:58 PM
>> To: users@cloudstack.apache.org 
>> Subject: Issue about Client communication error, please retry later.
>>
>> Dear Cloudstack support team,
>>
>> I have been trying to configure Cloudstack 4.13.1 (LTS) in a VMware KVM
>> (60GB HD, 8GB of RAM, 8 processors) in the CentOS7.
>> I got success to configure and install the whole package.
>>
>> However, once I have been trying to access the console view I have been
>> receiving the following error message: "Client communication error, please
>> retry later."  I mean, the OS opens and after a couple of seconds, I
>> receive this error. I am receiving this message for both the instance VMs
>> and system VM.
>>
>> Besides that, I tried to access, through the remote system application
>> (eg.
>> ssh), the VMs without success.
>>
>> Please let me know asap how to process to solve such issue.
>>
>> Best regards,
>>
>> Gustavo Callou
>> and
>> Alison Vinicius
>> --
>> --
>> Professor at DC / UFRPE, Brazil
>> PhD in Computer Science at UFPE, Brazil
>> http://callou.pythonanywhere.com/
>>
>> nicolas.vazq...@shapeblue.com
>> www.shapeblue.com
>> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
>> @shapeblue
>>
>>
>>
>>
>
> --
> --
> Professor at DC / UFRPE, Brazil
> PhD in Computer Science at UFPE, Brazil
> http://callou.pythonanywhere.com/
>

Re: Cannot start stopped VMs

[Fariborz Navidan]

Anyone can help me?

On Wed, Apr 21, 2021 at 7:46 PM Fariborz Navidan 
wrote:

> Any chance to identify the reason?
>
> Thanks
>
> On Wed, Apr 21, 2021 at 1:54 AM Fariborz Navidan 
> wrote:
>
>> Hi.
>> \Now there is no "Connection Closed" error in agent.log nut it is still
>> unable to start VMs. I have replaced agent.log with new one at
>> https://r9host.com/log/agent.log
>>
>> Kindly please check it for me
>> Thanks
>>
>> On Tue, Apr 20, 2021 at 8:01 PM Fariborz Navidan 
>> wrote:
>>
>>> Hi,
>>>
>>> Thank you for  your assistance. Libvirtd and cloudstack-agent services
>>> are running and are active. "netstat -t | grep 8250" shows two ESTABLISHED
>>> connections. Also in the infrastructure section, the host's status is shown
>>> as "Unsecured".
>>>
>>> Thanks
>>>
>>> On Tue, Apr 20, 2021 at 9:36 AM Harikrishna Patnala <
>>> harikrishna.patn...@shapeblue.com> wrote:
>>>
 Hi,

 At first glance on the logs, I could see below reasons for the start VM
 failures

 management server logs:
 2021-04-19 18:31:13,539 INFO  [c.c.v.VirtualMachineManagerImpl]
 (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3)
 (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to Connection
 is closed

 agent logs:
 2021-04-19 18:31:25,017 WARN  [kvm.resource.LibvirtKvmAgentHook]
 (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting engine is
 not initialized. Data transformation skipped.
 2021-04-19 18:31:25,686 WARN
 [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-1:null)
 (logid:92f2a447) LibvirtException
 org.libvirt.LibvirtException: Connection is closed
 at org.libvirt.ErrorHandler.processError(Unknown Source)
 at org.libvirt.Connect.processError(Unknown Source)
 at org.libvirt.Connect.processError(Unknown Source)
 at org.libvirt.Connect.domainCreateXML(Unknown Source)
 at
 com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611)
 at
 com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85)


 Can you check if libvirtd service running on your KVM host? Also, you
 can do a quick KVM agent restart and see if it works "systemctl restart
 cloudstack-agent"

 Regards,
 Harikrishna
 
 From: Fariborz Navidan 
 Sent: Tuesday, April 20, 2021 12:22 AM
 To: users@cloudstack.apache.org 
 Subject: Cannot start stopped VMs

 Hello,

 Today, I just updated SSL certificate and it failed re-deploying system
 VMs. Also, when I try to start a VM that is stopped, it fails with
 message:
 Unable to create deployment due to insufficient capacity. I couldn't
 find
 the reason from logs. I have uploaded  partial management server and
 agent
 logs.

 https://www.r9host.com/log/management-server.log
 https://www.r9host.com/log/agent.log

 It is a mission critical issue for us. I would appreciate if you guys
 check
 the logs and let us know what we can do to fix this issue.

 Thanks.

 harikrishna.patn...@shapeblue.com
 www.shapeblue.com
 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SG
 @shapeblue

Re: Changing the role of the account

[Rohit Yadav]

+1 wot Wido advised - it's not supported in UI, but root admin can call the API.

Regards.

From: Wido den Hollander 
Sent: Thursday, April 22, 2021 15:38
To: d...@cloudstack.apache.org ; Rakesh Venkatesh 
; users 
Subject: Re: Changing the role of the account

That's already possible:
https://cloudstack.apache.org/api/apidocs-4.15/apis/updateAccount.html

roleid: "The UUID of the dynamic role to set for the account"

Isn't that what you are looking for?

Wido


rohit.ya...@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 

On 22/04/2021 11:55, Rakesh Venkatesh wrote:
> Hello folks
>
> I dont think there is an endpoint right now to change the role of the
> account. If I have to change that in db, that should be doable right by
> changing the id's in two or three tables. I hope that wont break anything
> else. Is it good to add a new parameter to updateAccount api to take the
> new role id?
>

Re: Cannot start stopped VMs

[Fariborz Navidan]

This is the exception I found in the agent.log:

2021-04-22 12:21:29,045 WARN  [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-1:null) (logid:4d875f04) Groovy scripting engine is
not initialized. Data transformation skipped.
2021-04-22 12:21:47,260 WARN  [resource.wrapper.LibvirtStartCommandWrapper]
(agentRequest-Handler-4:null) (logid:1a2bdac6) LibvirtException
org.libvirt.LibvirtException: Failed to activate service
'org.freedesktop.machine1': timed out
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:45)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643)
at com.cloud.agent.Agent.processRequest(Agent.java:661)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

Why is this exception raised?

Thanks

On Thu, Apr 22, 2021 at 4:42 PM Fariborz Navidan 
wrote:

> Anyone can help me?
>
> On Wed, Apr 21, 2021 at 7:46 PM Fariborz Navidan 
> wrote:
>
>> Any chance to identify the reason?
>>
>> Thanks
>>
>> On Wed, Apr 21, 2021 at 1:54 AM Fariborz Navidan 
>> wrote:
>>
>>> Hi.
>>> \Now there is no "Connection Closed" error in agent.log nut it is still
>>> unable to start VMs. I have replaced agent.log with new one at
>>> https://r9host.com/log/agent.log
>>>
>>> Kindly please check it for me
>>> Thanks
>>>
>>> On Tue, Apr 20, 2021 at 8:01 PM Fariborz Navidan 
>>> wrote:
>>>
 Hi,

 Thank you for  your assistance. Libvirtd and cloudstack-agent services
 are running and are active. "netstat -t | grep 8250" shows two ESTABLISHED
 connections. Also in the infrastructure section, the host's status is shown
 as "Unsecured".

 Thanks

 On Tue, Apr 20, 2021 at 9:36 AM Harikrishna Patnala <
 harikrishna.patn...@shapeblue.com> wrote:

> Hi,
>
> At first glance on the logs, I could see below reasons for the start
> VM failures
>
> management server logs:
> 2021-04-19 18:31:13,539 INFO  [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3)
> (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to Connection
> is closed
>
> agent logs:
> 2021-04-19 18:31:25,017 WARN  [kvm.resource.LibvirtKvmAgentHook]
> (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting engine is
> not initialized. Data transformation skipped.
> 2021-04-19 18:31:25,686 WARN
> [resource.wrapper.LibvirtStartCommandWrapper] 
> (agentRequest-Handler-1:null)
> (logid:92f2a447) LibvirtException
> org.libvirt.LibvirtException: Connection is closed
> at org.libvirt.ErrorHandler.processError(Unknown Source)
> at org.libvirt.Connect.processError(Unknown Source)
> at org.libvirt.Connect.processError(Unknown Source)
> at org.libvirt.Connect.domainCreateXML(Unknown Source)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85)
>
>
> Can you check if libvirtd service running on your KVM host? Also, you
> can do a quick KVM agent restart and see if it works "systemctl restart
> cloudstack-agent"
>
> Regards,
> Harikrishna
> 
> From: Fariborz Navidan 
> Sent: Tuesday, April 20, 2021 12:22 AM
> To: users@cloudstack.apache.org 
> Subject: Cannot start stopped VMs
>
> Hello,
>
> Today, I just updated SSL certificate and it failed re-deploying system
> VMs. Also, when I try to start a VM that is stopped, it fails with
> message:
> Unable to create deployment due to insufficient capacity. I couldn't
> find
> the reason from logs. I have uploaded  partial management server and
> agent
> logs.
>
> 

Re: S3 API in Cloudstack

[Hean Seng]

Where can I download the plug in ? or this is paid plugin?

On Thu, Apr 22, 2021 at 10:56 AM X Guest  wrote:

> You need the s3 plugin.
> http://docs.cloudstack.apache.org/en/latest/developersguide/plugins.html
>
> regards.
>
> On Thu, Apr 22, 2021, at 9:05 AM, Hean Seng wrote:
> > Hi
> >
> > I am exploring / testing S3 API in Cloudstack.  In documentation, it
> seems
> > mentioned have this features.
> >
> > However, not able to get it up and running. Anybody have clue  for get
> this
> > up. ?  or some documentation for  read and try it out
> >
> > Thank you
> >
> > --
> > Regards,
> > Hean Seng
> >
>
> --
>   https://xguest.net/
>


-- 
Regards,
Hean Seng

RE: VM console keeps pausing to the point of being unusable

[Adam Witwicki]

Paste is now working!

Thanks

Adam

-Original Message-
From: Adam Witwicki 
Sent: 22 April 2021 09:18
To: users@cloudstack.apache.org
Subject: VM console keeps pausing to the point of being unusable

** This mail originated from OUTSIDE the Oakford corporate network. Treat 
hyperlinks and attachments in this email with caution. **

Hello,

We have an annoying issue, when using the cloudstack console to access a 
running instance the viewer will keep pausing every 10 to 15 mins, then its 
fine for a bit.
I have found the cloud.log from the console proxy which is showing java 
exceptions queue full, could this be related?

https://pastebin.com/YS6XCy6y

CS version 4.13.1.0

Any help appreciated

Thanks

Adam

Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked 
this e-mail and any attachments for viruses, we can not guarantee that they are 
virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by 
those to whom they are addressed. If you are not the intended recipient, please 
contact us, delete the message from your computer and destroy any copies. Any 
distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology 
Limited does not accept legal responsibility for this message. The recipient is 
responsible for verifying its authenticity before acting on the contents. Any 
views or opinions presented are solely those of the author and do not 
necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, The Manor House, Potterne, 
Wiltshire. SN10 5PN.
Registered in England and Wales No. 5971519

Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked 
this e-mail and any attachments for viruses, we can not guarantee that they are 
virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by 
those to whom they are addressed. If you are not the intended recipient, please 
contact us, delete the message from your computer and destroy any copies. Any 
distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology 
Limited does not accept legal responsibility for this message. The recipient is 
responsible for verifying its authenticity before acting on the contents. Any 
views or opinions presented are solely those of the author and do not 
necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, The Manor House, Potterne, 
Wiltshire. SN10 5PN.
Registered in England and Wales No. 5971519

Re: Changing the role of the account

[Wido den Hollander]

That's already possible: 
https://cloudstack.apache.org/api/apidocs-4.15/apis/updateAccount.html


roleid: "The UUID of the dynamic role to set for the account"

Isn't that what you are looking for?

Wido

On 22/04/2021 11:55, Rakesh Venkatesh wrote:

Hello folks

I dont think there is an endpoint right now to change the role of the
account. If I have to change that in db, that should be doable right by
changing the id's in two or three tables. I hope that wont break anything
else. Is it good to add a new parameter to updateAccount api to take the
new role id?

Changing the role of the account

[Rakesh Venkatesh]

Hello folks

I dont think there is an endpoint right now to change the role of the
account. If I have to change that in db, that should be doable right by
changing the id's in two or three tables. I hope that wont break anything
else. Is it good to add a new parameter to updateAccount api to take the
new role id?

-- 
Thanks and regards
Rakesh venkatesh

VM console keeps pausing to the point of being unusable

[Adam Witwicki]

Hello,

We have an annoying issue, when using the cloudstack console to access a 
running instance the viewer will keep pausing every 10 to 15 mins, then its 
fine for a bit.
I have found the cloud.log from the console proxy which is showing java 
exceptions queue full, could this be related?

https://pastebin.com/YS6XCy6y

CS version 4.13.1.0

Any help appreciated

Thanks

Adam

Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked 
this e-mail and any attachments for viruses, we can not guarantee that they are 
virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by 
those to whom they are addressed. If you are not the intended recipient, please 
contact us, delete the message from your computer and destroy any copies. Any 
distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology 
Limited does not accept legal responsibility for this message. The recipient is 
responsible for verifying its authenticity before acting on the contents. Any 
views or opinions presented are solely those of the author and do not 
necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, The Manor House, Potterne, 
Wiltshire. SN10 5PN.
Registered in England and Wales No. 5971519

Re: [RESULT][VOTE] New life to Terraform Provider CloudStack with Apache CloudStack project

[Rohit Yadav]

Update - I've created the new repository on Github: 
https://github.com/apache/cloudstack-terraform-provider

... and logged request with ASF infra:
https://issues.apache.org/jira/browse/INFRA-21772


Regards.


From: Rohit Yadav 
Sent: Thursday, April 22, 2021 13:05
To: d...@cloudstack.apache.org ; 
users@cloudstack.apache.org 
Subject: [RESULT][VOTE] New life to Terraform Provider CloudStack with Apache 
CloudStack project

Hi all,

After 120 hours, the vote [1] for accepting codebase of Terraform Provider for 
CloudStack by the Apache CloudStack project has passed with the following votes:

+1 (PMC / binding)
4 persons (Wido, Daan, Boris, Rohit)

+1 (non-binding)
5 persons (Nathan, Abhishek, Hari, Suresh, Nicolas)

0
none

-1 (non-binding)
1 person (Rene)

Thanks to everyone participating. Following this I'll request ASF infra with a 
new repository: https://github.com/apache/cloudstack-terraform-provider have 
access enabled for all committers/PMC with issues, pull request and wiki 
features.

[1] https://markmail.org/message/26yjsewj4tgq


Regards.

rohit.ya...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue




rohit.ya...@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 

RE: [!!Mass Mail]system VM cannot connect to cloudstack?

[Yordan Kostov]

Thank you Евгений! 

We made it work just as per your proposal.

Regards,
Jordan

-Original Message-
From: Дикевич Евгений Александрович  
Sent: Wednesday, April 21, 2021 5:31 PM
To: users@cloudstack.apache.org
Subject: RE: [!!Mass Mail]system VM cannot connect to cloudstack?


[X] This message came from outside your organization


I think you should add both servers (comma separated) and check 
management.network.cidr in global parameters.

-Original Message-
From: Yordan Kostov [mailto:yord...@nsogroup.com]
Sent: Wednesday, April 21, 2021 12:55 PM
To: users@cloudstack.apache.org
Subject: RE: [!!Mass Mail]system VM cannot connect to cloudstack?

It has 10.10.38.14.

I believe this should be changed either to having both addresses - 10.10.38.13 
and 14  or to the VIP address 10.10.38.10 of the HAproxy VIP?

Best regards,
Jordan

-Original Message-
From: Дикевич Евгений Александрович 
Sent: Wednesday, April 21, 2021 12:38 PM
To: users@cloudstack.apache.org
Subject: RE: [!!Mass Mail]system VM cannot connect to cloudstack?


[X] This message came from outside your organization


Hi.
What do you have in global setting in host variable?

-Original Message-
From: Yordan Kostov [mailto:yord...@nsogroup.com]
Sent: Wednesday, April 21, 2021 12:13 PM
To: users@cloudstack.apache.org
Subject: [!!Mass Mail]system VM cannot connect to cloudstack?

Dear all,

I have the following setup:

-  2 x Cloustack MGM servers 4.15 (firewall and selinux disabled) - 
10.10.38.13 and 10.10.38.14; (SSVM is 4.15)

-  Hypervisor XCPNG 8.2

-  NFS secondary storage

I brought up a zone and secondary VMs are created just fine but the management 
server reports that the agent is down. After some troubleshooting:

-  SSVMs can connect to 10.10.38.13 on 8250

-  SSVMs cannot connect to 10.10.38.14 on 8250 due to time out.

SSVM logs:
2021-04-21 07:18:22,442 WARN  [utils.nio.NioConnection] (main:null) Unable to 
connect to remote: is there a server running on port 8250
2021-04-21 07:18:27,443 INFO  [cloud.agent.Agent] (main:null) Connecting to 
host:10.10.38.14

I cannot find the reason for this as SSVMs are on the same network as the MGM 
servers.

-  telnet from SSVMs to 38.14 on 8250 reports timeout

-  telnet from any other VM in the same network to 38.14 on 8250 is 
completely fine.

I am consulting with this article for SSVM troubleshooting but so far nothing 
helps - 
https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM*2C*templates*2C*Secondary*storage*troubleshooting__;JSslKysr!!A6UyJA!3Z7dCJiMwsIF9d8__V12zgvf17v3tmxwAE_yGbkqtDEClvxQoFOszpTl8hjwpCLWoMqI3sUHJCNP$

This is an empty environment straight out of the box so I don't 
understand why this behavior occurs.
Any advice is appreciated.

Best regards,
Jordan
Внимание!
Это электронное письмо и все прикрепленные к нему файлы являются 
конфиденциальными и предназначены исключительно для использования лицом 
(лицами), которому (которым) оно предназначено. Если Вы не являетесь лицом 
(лицами), которому (которым) предназначено это письмо, не копируйте и не 
разглашайте его содержимое и удалите это сообщение и все вложения из Вашей 
почтовой системы. Любое несанкционированное использование, распространение, 
раскрытие, печать или копирование этого электронного письма и прикрепленных к 
нему файлов, кроме как лицом (лицами) которому (которым) они предназначены, 
является незаконным и запрещено. Принимая во внимание, что передача данных 
посредством Интернет не является безопасной, мы не несем никакой 
ответственности за любой потенциальный ущерб, причиненный в результате ошибок 
при передаче данных или этим сообщением и прикрепленными к нему файлами.

Attention!
This email and all attachments to it are confidential and are intended solely 
for use by the person (or persons) referred to (mentioned) as the intended 
recipient (recipients). If you are not the intended recipient of this email, do 
not copy or disclose its contents and delete the message and any attachments to 
it from your e-mail system. Any unauthorized use, dissemination, disclosure, 
printing or copying of this e-mail and files attached to it, except by the 
intended recipient, is illegal and is prohibited. Taking into account that data 
transmission via Internet is not secure, we assume no responsibility for any 
potential damage caused by data transmission errors or this message and the 
files attached to it.

[RESULT][VOTE] New life to Terraform Provider CloudStack with Apache CloudStack project

[Rohit Yadav]

Hi all,

After 120 hours, the vote [1] for accepting codebase of Terraform Provider for 
CloudStack by the Apache CloudStack project has passed with the following votes:

+1 (PMC / binding)
4 persons (Wido, Daan, Boris, Rohit)

+1 (non-binding)
5 persons (Nathan, Abhishek, Hari, Suresh, Nicolas)

0
none

-1 (non-binding)
1 person (Rene)

Thanks to everyone participating. Following this I'll request ASF infra with a 
new repository: https://github.com/apache/cloudstack-terraform-provider have 
access enabled for all committers/PMC with issues, pull request and wiki 
features.

[1] https://markmail.org/message/26yjsewj4tgq


Regards.

rohit.ya...@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 

Re: [VOTE] New life to Terraform Provider CloudStack with Apache CloudStack project

[Rohit Yadav]

+1 (binding) - I'll add my vote as well.


Regards.


From: Nicolas Vazquez 
Sent: Tuesday, April 20, 2021 16:42
To: d...@cloudstack.apache.org ; 
priv...@cloudstack.apache.org ; 
users@cloudstack.apache.org 
Subject: Re: [VOTE] New life to Terraform Provider CloudStack with Apache 
CloudStack project

+1


Regards,

Nicolas Vazquez


From: Rohit Yadav 
Sent: Thursday, April 15, 2021 6:05 AM
To: d...@cloudstack.apache.org ; 
priv...@cloudstack.apache.org 
Cc: users@cloudstack.apache.org 
Subject: [VOTE] New life to Terraform Provider CloudStack with Apache 
CloudStack project

Hi All,

Following the discussion thread on Terraform [1], I would like to start a vote 
to gather consensus on the following actions:

  1.  Create a new "cloudstack-terraform-provider" repository based on Apache 
Licence v2.0 using re-licensed codebase of the archived/former terraform 
cloudstack provider repository: 
https://github.com/hashicorp/terraform-provider-cloudstack (note: re-licensing 
from MPL to AL will be done by Hashicorp)
  2.  Request ASF infra to enable issues, PR, and wiki features on the 
repository
  3.  Work with the community towards any further maintenance, development, and 
releases of the provider
  4.  Publish official releases on the official registry [2] if/after Apache 
CloudStack project gets a verified account (published by PMC members with 
access to the registry, or following guidelines from ASF infra if they've any)

The vote will be open for 120 hours, until Wed 21 April 2021.
For sanity in tallying the vote, can PMC members please be sure to indicate 
"(binding)" with their vote?

[ ] +1  approve
[ ] +0  no opinion
[ ] -1  disapprove (and reason why)

[1] https://markmail.org/message/iuggxin7kj6ri4hb
[2] https://registry.terraform.io/browse/providers


Regards.

rohit.ya...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue




nicolas.vazq...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue




rohit.ya...@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue