Change source-nat IP - Isolated Network

[cristian.c]

Hello,

 

Is there a way to change a source-NAT IP for an Isolated network? I ask
because I have multiple ranges, and the current one used for source-NAT must
be removed.

Cloudstack 4.18.1

Regards,
Cristian

RE: Failed to execute post certificate renewal command - KVM 4.18.1

[cristian.c]

Hello,

  Yes, I added with ssh password, and I want to reprovision because the status 
is unsecure in cloudstack UI.

Regards,
Cristian

-Original Message-
From: Vivek Kumar  
Sent: Saturday, March 30, 2024 4:33 PM
To: CloudStack Users Mailing list 
Subject: Re: Failed to execute post certificate renewal command - KVM 4.18.1

Hello Cristian,

Have you added the hosts in cloudstack, and it’s showing unsecure in hosts 
section, ? And then you are trying to reprovision the security keys ? Is this 
the case ? 

Vivek Kumar
Sr. Manager - Cloud & DevOps
TechOps | Indiqus Technologies

vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com>
www.indiqus.com <https://www.indiqus.com/>




> On 29-Mar-2024, at 9:53 PM, cristian.c@istream.today wrote:
> 
> Hello,
> 
> 
> 
>   Any idea how to get rid of this error? I try to provision security 
> keys for this kvm host, but I end with this error :
> 
> 
> 
>   This is a new setup, with 4.18.1.0 on ubuntu 22.04, 1x host for 
> management and 1x  KVM host.
> 
> 
> 2024-03-29 16:16:19,879 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) (logid:0c5006ba) Asking libvirt to 
> refresh storage pool 552ecff0-1806-407d-8db6-f1c9da18c971
> 
> ^C
> 
> root@hv-001:/etc/cloudstack/agent# nano agent.properties
> 
> root@hv-001:/etc/cloudstack/agent# systemctl restart cloudstack-agent
> 
> root@hv-001:/etc/cloudstack/agent# tail -f 
> /var/log/cloudstack/agent/agent.log
> 
> 2024-03-29 16:17:56,744 INFO  [cloud.agent.Agent] 
> (Agent-Handler-2:null)
> (logid:) Process agent startup answer, agent id = 0
> 
> 2024-03-29 16:17:56,745 INFO  [cloud.agent.Agent] 
> (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 
> 2024-03-29 16:17:56,750 INFO  [cloud.agent.Agent] 
> (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
> 
> 2024-03-29 16:17:56,965 INFO  [cloud.agent.Agent]
> (agentRequest-Handler-3:null) (logid:5d6aaa47) Processing agent ready 
> command, agent id = 1
> 
> 2024-03-29 16:17:56,965 INFO  [cloud.agent.Agent]
> (agentRequest-Handler-3:null) (logid:5d6aaa47) Set agent id 1
> 
> 2024-03-29 16:17:56,965 INFO  [cloud.agent.Agent]
> (agentRequest-Handler-3:null) (logid:5d6aaa47) Ready command is 
> processed for agent id = 1
> 
> 2024-03-29 16:17:57,085 INFO  [cloud.agent.Agent]
> (agentRequest-Handler-5:null) (logid:5d6aaa47) Processing agent ready 
> command, agent id = 1
> 
> 2024-03-29 16:17:57,086 INFO  [cloud.agent.Agent]
> (agentRequest-Handler-5:null) (logid:5d6aaa47) Set agent id 1
> 
> 2024-03-29 16:17:57,088 INFO  [cloud.agent.Agent]
> (agentRequest-Handler-5:null) (logid:5d6aaa47) Processed new 
> management server list: 51.xxx.xxx.69,10.111.0.1@static
> 
> 2024-03-29 16:17:57,088 INFO  [cloud.agent.Agent]
> (agentRequest-Handler-5:null) (logid:5d6aaa47) Ready command is 
> processed for agent id = 1
> 
> 2024-03-29 16:18:20,157 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-2:null) (logid:4be26651) Trying to fetch storage 
> pool
> 552ecff0-1806-407d-8db6-f1c9da18c971 from libvirt
> 
> 2024-03-29 16:18:20,181 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-2:null) (logid:4be26651) Asking libvirt to 
> refresh storage pool 552ecff0-1806-407d-8db6-f1c9da18c971
> 
> 2024-03-29 16:18:43,714 INFO
> [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] 
> (Certificate Renewal Timer:null) (logid:7f9421ae) Restarting libvirt 
> after certificate provisioning/renewal
> 
> 2024-03-29 16:18:43,729 INFO  [cloud.agent.Agent] 
> (AgentShutdownThread:null)
> (logid:) Stopping the agent: Reason = sig.kill
> 
> 2024-03-29 16:18:43,730 WARN  [cloud.agent.Agent] (Certificate Renewal
> Timer:null) (logid:7f9421ae) Failed to execute post certificate 
> renewal
> command:
> 
> java.lang.IllegalStateException: Shutdown in progress
> 
>at
> java.base/java.lang.ApplicationShutdownHooks.remove(ApplicationShutdow
> nHooks
> .java:82)
> 
>at 
> java.base/java.lang.Runtime.removeShutdownHook(Runtime.java:244)
> 
>at
> com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.ja
> va:115
> 6)
> 
>at
> org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInC
> ontext
> (ManagedContextTimerTask.java:30)
> 
>at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man
> agedCo
> ntextRunnable.java:48)
> 
>at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal
> l(Defa
> ultManagedContext.java:55)
> 
>at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW
> ithCon
> te

Failed to execute post certificate renewal command - KVM 4.18.1

[cristian.c]

Hello,

 

   Any idea how to get rid of this error? I try to provision security keys
for this kvm host, but I end with this error :



   This is a new setup, with 4.18.1.0 on ubuntu 22.04, 1x host for
management and 1x  KVM host.  


2024-03-29 16:16:19,879 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:0c5006ba) Asking libvirt to refresh
storage pool 552ecff0-1806-407d-8db6-f1c9da18c971

^C

root@hv-001:/etc/cloudstack/agent# nano agent.properties

root@hv-001:/etc/cloudstack/agent# systemctl restart cloudstack-agent

root@hv-001:/etc/cloudstack/agent# tail -f
/var/log/cloudstack/agent/agent.log

2024-03-29 16:17:56,744 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Process agent startup answer, agent id = 0

2024-03-29 16:17:56,745 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Set agent id 0

2024-03-29 16:17:56,750 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Startup Response Received: agent id = 0

2024-03-29 16:17:56,965 INFO  [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:5d6aaa47) Processing agent ready
command, agent id = 1

2024-03-29 16:17:56,965 INFO  [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:5d6aaa47) Set agent id 1

2024-03-29 16:17:56,965 INFO  [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:5d6aaa47) Ready command is processed
for agent id = 1

2024-03-29 16:17:57,085 INFO  [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:5d6aaa47) Processing agent ready
command, agent id = 1

2024-03-29 16:17:57,086 INFO  [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:5d6aaa47) Set agent id 1

2024-03-29 16:17:57,088 INFO  [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:5d6aaa47) Processed new management
server list: 51.xxx.xxx.69,10.111.0.1@static

2024-03-29 16:17:57,088 INFO  [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:5d6aaa47) Ready command is processed
for agent id = 1

2024-03-29 16:18:20,157 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:4be26651) Trying to fetch storage pool
552ecff0-1806-407d-8db6-f1c9da18c971 from libvirt

2024-03-29 16:18:20,181 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:4be26651) Asking libvirt to refresh
storage pool 552ecff0-1806-407d-8db6-f1c9da18c971

2024-03-29 16:18:43,714 INFO
[resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] (Certificate
Renewal Timer:null) (logid:7f9421ae) Restarting libvirt after certificate
provisioning/renewal

2024-03-29 16:18:43,729 INFO  [cloud.agent.Agent] (AgentShutdownThread:null)
(logid:) Stopping the agent: Reason = sig.kill

2024-03-29 16:18:43,730 WARN  [cloud.agent.Agent] (Certificate Renewal
Timer:null) (logid:7f9421ae) Failed to execute post certificate renewal
command:

java.lang.IllegalStateException: Shutdown in progress

at
java.base/java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks
.java:82)

at java.base/java.lang.Runtime.removeShutdownHook(Runtime.java:244)

at
com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:115
6)

at
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext
(ManagedContextTimerTask.java:30)

at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:48)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:55)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:102)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:52)

at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:45)

at
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedCon
textTimerTask.java:32)

at java.base/java.util.TimerThread.mainLoop(Timer.java:556)

at java.base/java.util.TimerThread.run(Timer.java:506)

2024-03-29 16:18:45,061 INFO  [cloud.agent.AgentShell] (main:null) (logid:)
Agent started

2024-03-29 16:18:45,063 INFO  [cloud.agent.AgentShell] (main:null) (logid:)
Implementation Version is 4.18.1.0

2024-03-29 16:18:45,064 INFO  [cloud.agent.AgentShell] (main:null) (logid:)
agent.properties found at /etc/cloudstack/agent/agent.properties

2024-03-29 16:18:45,091 INFO  [cloud.agent.AgentShell] (main:null) (logid:)
Defaulting to using properties file for storage

2024-03-29 16:18:45,092 INFO  [cloud.agent.AgentShell] (main:null) (logid:)
Defaulting to the constant time backoff algorithm

2024-03-29 16:18:45,121 INFO  [cloud.utils.LogUtils] (main:null) (logid:)
log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml

2024-03-29 16:18:45,121 INFO  [cloud.agent.AgentShell] (main:null) (logid:)
Using default Java settings for IPv6 preference for agent connection

2024-03-29 

Unable to add the host: Cannot find the server resources at

[cristian.c]

Hello,

 

   Any workaround for this bug? "Unable to add the host: Cannot find the
server resources at
http://06.hostx.host/ZONE-001/DC-001//mub-in-001.hostx.host
 "

  

When attempting to add a second host to an existing cluster, I encounter an
error. I suspect this may be caused by a duplicate "/" symbol, which occurs
even when using the API. 

 

However, if I set up a new cluster without specifying any hosts, it operates
correctly as it automatically discovers the hosts.

 

This problem is present in CloudStack version 4.18.1.


Regards,

Cristian

  

Failed to authenticate Veeam API service due to:Connection reset - Veeam 12

[cristian.c]

Hello,

 

I'm struggling to link CloudStack with Veeam 12 and I keep encountering the
same issue: "Failed to authenticate Veeam API service due to: Connection
reset."

My question is, should I be connecting to the API on the Veeam Enterprise
Manager or the Backup and Replication server? I'm asking because the
instructions aren't clear on this point.


  - Veeam B server URL. Default: http://
 :9398/api/ but
B API does not run on 9398 it is on 9419, only the Veeam Backup Ent
Manager API runs on 9398 and 9399. 

 

Anyone using with Veeam 12.

 

Thank you,

Cristian

RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 11: Link is closed

[cristian.c]

Update:

  After I updated the mgmt._server_id for all the hosts, looks like the tasks 
are running now.   I still have some issues with some system vms, I will 
recreate and see if everything is ok.


Regards,
Cristian

-Original Message-
From: cristian.c@istream.today  
Sent: Wednesday, January 17, 2024 5:03 PM
To: users@cloudstack.apache.org
Subject: RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 
11: Link is closed

Hi Wei,

   Please see :

root@acs-apac:/etc/cloudstack/management# hostname -f acs-apac

One thing I saw right now in table hosts, the mgmt._server_id is still 
corresponding to previous server  

Right now, all the nodes are with state Up but with a timeout in logs. I 
changed the mgmt._server_id for one host to the new id, current server, and the 
state was updated to Disconnected. Now I'm a bit confused; should not all hosts 
have the new mgmt._server_id, or is this updated over time?

For clarification, the hosts are up and reachable from cloudstack and vice 
versa.  

Regards,
Cristian



-Original Message-
From: Wei ZHOU 
Sent: Wednesday, January 17, 2024 4:56 PM
To: users@cloudstack.apache.org
Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 
11: Link is closed

The msid should be same after restart.

Can you check if "hostname -f" works?


在 2024年1月17日星期三，  写道：

> I already have a headache. Anyone here who can provide support for 
> this (not for a fee)? I need to restore this CloudStack, but it is 
> impossible on this version.
>
>
> Now any task I execute I see repetitive :
>
> 2024-01-17 14:41:20,892 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-9:null) (logid:) Seq 6-5811613844144914446:
> Forwarding Seq 6-5811613844144914446:  { Cmd , MgmtId: 345050149016, via:
> 6, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.
> RebootCommand":{"vmName":"s-24960-VM","executeInSequence":
> "true","wait":"0","bypassHostMaintenance":"false"}}] } to 52239141482
>
> So, I only see this in the log, one after another. Here I just tried 
> to reboot a secondary storage to see if it works. No matter what I 
> try, I see a similar repetitive event in the log, and it will not stop 
> until I restart the cloudstack-management.
>
>
> Regards,
> Cristian
>
>
>
>
> -Original Message-
> From: cristian.c@istream.today 
> Sent: Wednesday, January 17, 2024 12:46 PM
> To: users@cloudstack.apache.org
> Subject: RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable:
> Host 11: Link is closed
>
> Hi Wei,
>
>   I just did this, now I set this for both entries :
>
> |id |msid   |runid|name|uuid
>   |state|version |service_ip   |service_port|last_update
> |removed|alert_count|
> |---|---|-||
> |-||-|--
> --|---|---|---|
> |1  |52,239,141,482 
> |1,704,875,017,724|acs-apac|bd77b0ae-66d4-4783-bc17-5c3707cc1e49|Down
> |4.18.1.0|181.41.xxx.12|9,090   |2024-01-16 08:00:01|2024-01-17
> 12:36:18|0  |
> |2  |345,050,133,919|1,705,475,667,602|acs-apac|5fd7fd16-
> 8134-4408-b5b7-fa11327ebe29|Down |4.18.1.0|181.41.xxx.12|9,090
>  |2024-01-17 07:20:02|2024-01-17 12:36:23|0  |
>
>
> And after start I see this :
>
> |id |msid   |runid|name|uuid
>   |state|version |service_ip   |service_port|last_update
> |removed|alert_count|
> |---|---|-||
> |-||-|--
> --|---|---|---|
> |1  |52,239,141,482 
> |1,704,875,017,724|acs-apac|bd77b0ae-66d4-4783-bc17-5c3707cc1e49|Down
> |4.18.1.0|181.41.xxx.12|9,090   |2024-01-16 08:00:01|2024-01-17
> 12:36:18|0  |
> |2
> ||345,050,133,919|1,705,487,986,668|acs-apac|5fd7fd16-8134-4408-b5b7-f
> |a11327ebe29|Up
>  |4.18.1.0|181.41.xxx.12|9,090   |2024-01-17 10:41:05|
>  |0  |
>
>
> This is what I have in mshost status :
>
> |id |ms_id   |last_jvm_start
>  |last_jvm_stop  |last_system_boot   |os_distribution
>  |java_name|java_version   |updated
> |created|removed|
> |---||--
> -|---|---|--
> |-|---|-
> --|---|---|
> |2  |bd77b0ae-66d4-4783-bc17-5c3707cc1e49|2024-01-10
> |08:23:25|2024-01-10
> 08:16:12|2

RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 11: Link is closed

[cristian.c]

Hi Wei,

   Please see :

root@acs-apac:/etc/cloudstack/management# hostname -f
acs-apac

One thing I saw right now in table hosts, the mgmt._server_id is still 
corresponding to previous server  

Right now, all the nodes are with state Up but with a timeout in logs. I 
changed the mgmt._server_id for one host to the new id, current server, and the 
state was updated to Disconnected. Now I'm a bit confused; should not all hosts 
have the new mgmt._server_id, or is this updated over time?

For clarification, the hosts are up and reachable from cloudstack and vice 
versa.  

Regards,
Cristian



-Original Message-
From: Wei ZHOU  
Sent: Wednesday, January 17, 2024 4:56 PM
To: users@cloudstack.apache.org
Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 
11: Link is closed

The msid should be same after restart.

Can you check if "hostname -f" works?


在 2024年1月17日星期三，  写道：

> I already have a headache. Anyone here who can provide support for 
> this (not for a fee)? I need to restore this CloudStack, but it is 
> impossible on this version.
>
>
> Now any task I execute I see repetitive :
>
> 2024-01-17 14:41:20,892 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-9:null) (logid:) Seq 6-5811613844144914446:
> Forwarding Seq 6-5811613844144914446:  { Cmd , MgmtId: 345050149016, via:
> 6, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.
> RebootCommand":{"vmName":"s-24960-VM","executeInSequence":
> "true","wait":"0","bypassHostMaintenance":"false"}}] } to 52239141482
>
> So, I only see this in the log, one after another. Here I just tried 
> to reboot a secondary storage to see if it works. No matter what I 
> try, I see a similar repetitive event in the log, and it will not stop 
> until I restart the cloudstack-management.
>
>
> Regards,
> Cristian
>
>
>
>
> -Original Message-
> From: cristian.c@istream.today 
> Sent: Wednesday, January 17, 2024 12:46 PM
> To: users@cloudstack.apache.org
> Subject: RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable:
> Host 11: Link is closed
>
> Hi Wei,
>
>   I just did this, now I set this for both entries :
>
> |id |msid   |runid|name|uuid
>   |state|version |service_ip   |service_port|last_update
> |removed|alert_count|
> |---|---|-||
> |-||-|--
> --|---|---|---|
> |1  |52,239,141,482 
> |1,704,875,017,724|acs-apac|bd77b0ae-66d4-4783-bc17-5c3707cc1e49|Down
> |4.18.1.0|181.41.xxx.12|9,090   |2024-01-16 08:00:01|2024-01-17
> 12:36:18|0  |
> |2  |345,050,133,919|1,705,475,667,602|acs-apac|5fd7fd16-
> 8134-4408-b5b7-fa11327ebe29|Down |4.18.1.0|181.41.xxx.12|9,090
>  |2024-01-17 07:20:02|2024-01-17 12:36:23|0  |
>
>
> And after start I see this :
>
> |id |msid   |runid|name|uuid
>   |state|version |service_ip   |service_port|last_update
> |removed|alert_count|
> |---|---|-||
> |-||-|--
> --|---|---|---|
> |1  |52,239,141,482 
> |1,704,875,017,724|acs-apac|bd77b0ae-66d4-4783-bc17-5c3707cc1e49|Down
> |4.18.1.0|181.41.xxx.12|9,090   |2024-01-16 08:00:01|2024-01-17
> 12:36:18|0  |
> |2  
> ||345,050,133,919|1,705,487,986,668|acs-apac|5fd7fd16-8134-4408-b5b7-f
> |a11327ebe29|Up
>  |4.18.1.0|181.41.xxx.12|9,090   |2024-01-17 10:41:05|
>  |0  |
>
>
> This is what I have in mshost status :
>
> |id |ms_id   |last_jvm_start
>  |last_jvm_stop  |last_system_boot   |os_distribution
>  |java_name|java_version   |updated
> |created|removed|
> |---||--
> -|---|---|--
> |-|---|-
> --|---|---|
> |2  |bd77b0ae-66d4-4783-bc17-5c3707cc1e49|2024-01-10 
> |08:23:25|2024-01-10
> 08:16:12|2024-01-10 08:23:15|Ubuntu 20.04.6 LTS|Ubuntu
>  |11.0.21+9-post-Ubuntu-0ubuntu120.04|2024-01-16 07:59:59|2022-09-30
> 07:40:52|   |
> |3  |5fd7fd16-8134-4408-b5b7-fa11327ebe29|2024-01-17 
> |07:14:16|2024-01-17
> 07:14:14|2024-01-16 21:34:09|Ubuntu 22.04.3 LTS|Ubuntu
>  |11.0.21+9-post-Ubuntu-0ubuntu122.04|2024-01-17 07:19:46|2024-01-16
> 11:53:18|   |
>
> And I have the same issue, host timeout.
>
> 

RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 11: Link is closed

[cristian.c]

I already have a headache. Anyone here who can provide support for this (not 
for a fee)? I need to restore this CloudStack, but it is impossible on this 
version.


Now any task I execute I see repetitive :

2024-01-17 14:41:20,892 DEBUG [c.c.a.m.ClusteredAgentAttache] 
(AgentManager-Handler-9:null) (logid:) Seq 6-5811613844144914446: Forwarding 
Seq 6-5811613844144914446:  { Cmd , MgmtId: 345050149016, via: 6, Ver: v1, 
Flags: 100111, 
[{"com.cloud.agent.api.RebootCommand":{"vmName":"s-24960-VM","executeInSequence":"true","wait":"0","bypassHostMaintenance":"false"}}]
 } to 52239141482

So, I only see this in the log, one after another. Here I just tried to reboot 
a secondary storage to see if it works. No matter what I try, I see a similar 
repetitive event in the log, and it will not stop until I restart the 
cloudstack-management.


Regards,
Cristian 




-Original Message-
From: cristian.c@istream.today  
Sent: Wednesday, January 17, 2024 12:46 PM
To: users@cloudstack.apache.org
Subject: RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 
11: Link is closed

Hi Wei,

  I just did this, now I set this for both entries :

|id |msid   |runid|name|uuid
|state|version |service_ip   |service_port|last_update|removed  
  |alert_count|
|---|---|-|||-||-||---|---|---|
|1  |52,239,141,482 
|1,704,875,017,724|acs-apac|bd77b0ae-66d4-4783-bc17-5c3707cc1e49|Down 
|4.18.1.0|181.41.xxx.12|9,090   |2024-01-16 08:00:01|2024-01-17 12:36:18|0  
|
|2  
|345,050,133,919|1,705,475,667,602|acs-apac|5fd7fd16-8134-4408-b5b7-fa11327ebe29|Down
 |4.18.1.0|181.41.xxx.12|9,090   |2024-01-17 07:20:02|2024-01-17 12:36:23|0 
 |


And after start I see this :

|id |msid   |runid|name|uuid
|state|version |service_ip   |service_port|last_update|removed  
  |alert_count|
|---|---|-|||-||-||---|---|---|
|1  |52,239,141,482 
|1,704,875,017,724|acs-apac|bd77b0ae-66d4-4783-bc17-5c3707cc1e49|Down 
|4.18.1.0|181.41.xxx.12|9,090   |2024-01-16 08:00:01|2024-01-17 12:36:18|0  
|
|2  
|345,050,133,919|1,705,487,986,668|acs-apac|5fd7fd16-8134-4408-b5b7-fa11327ebe29|Up
   |4.18.1.0|181.41.xxx.12|9,090   |2024-01-17 10:41:05|   
|0  |


This is what I have in mshost status :

|id |ms_id   |last_jvm_start |last_jvm_stop 
 |last_system_boot   |os_distribution   |java_name|java_version 
  |updated|created|removed|
|---||---|---|---|--|-|---|---|---|---|
|2  |bd77b0ae-66d4-4783-bc17-5c3707cc1e49|2024-01-10 08:23:25|2024-01-10 
08:16:12|2024-01-10 08:23:15|Ubuntu 20.04.6 LTS|Ubuntu   
|11.0.21+9-post-Ubuntu-0ubuntu120.04|2024-01-16 07:59:59|2022-09-30 07:40:52|   
|
|3  |5fd7fd16-8134-4408-b5b7-fa11327ebe29|2024-01-17 07:14:16|2024-01-17 
07:14:14|2024-01-16 21:34:09|Ubuntu 22.04.3 LTS|Ubuntu   
|11.0.21+9-post-Ubuntu-0ubuntu122.04|2024-01-17 07:19:46|2024-01-16 11:53:18|   
|

And I have the same issue, host timeout.  

I'm out of ideas.


Thank you,
Cristian


-Original Message-
From: Wei ZHOU 
Sent: Wednesday, January 17, 2024 12:19 PM
To: users@cloudstack.apache.org
Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 
11: Link is closed

Hi,

Can you mark the mgmt server as Down (or set "removed" to "now()") in the 
"mshost" table , and retry?

-Wei

On Wed, 17 Jan 2024 at 10:29,  wrote:

> Looks like restoring on the exact same OS version is not fixing the 
> issue... at starting the cloudstack I get this error :
>
> 2024-01-17 09:26:26,067 DEBUG [c.c.u.s.Script] (main:null) (logid:) 
> System
> resource:
> file:/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2024-01-17 09:26:26,067 DEBUG [c.c.u.s.Script] (main:null) (logid:) 
> Absolute path = 
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2024-01-17 09:26:26,068 DEBUG [c.c.s.ConfigurationServerImpl]
> (main:null)
> (logid:) Executing: /bin/bash
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/lib/cloudstack/management/.ssh/id_rsa
> 2024-01-17 09:26:26,070 DEBUG [c.c.s.ConfigurationServerImpl]
> (main:null)
> (logid:) Executing

RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 11: Link is closed

[cristian.c]

java:379)
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
> at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
> at
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at org.eclipse.jetty.server.Server.start(Server.java:423)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at org.eclipse.jetty.server.Server.doStart(Server.java:387)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:192)
> at 
> org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:107)
> 2024-01-17 09:26:26,137 INFO  [c.c.c.ClusterManagerImpl] (main:null)
> (logid:) Detected that another management node with the same IP 
> 10.60.0.2 is considered as running in DB, however it is not pingable, 
> we will continue cluster initialization with this management server 
> node
> 2024-01-17 09:26:26,137 INFO  [c.c.c.ClusterManagerImpl] (main:null)
> (logid:) Cluster manager is configured.
>
>
> Regards,
> Cristian
>
>
>
>
> -Original Message-
> From: cristian.c@istream.today 
> Sent: Wednesday, January 17, 2024 9:24 AM
> To: users@cloudstack.apache.org
> Subject: RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable:
> Host 11: Link is closed
>
> Hi Jithin,
>
>I changed the status to down and set as removed. I still have the 
> same issue...  So, I think I must create a new server with previous OS 
> version, in any case not sure where is the logic.
>
> Thank you!
> Cristian
>
> -Original Message-
> From: Jithin Raju 
> Sent: Wednesday, January 17, 2024 6:16 AM
> To: users@cloudstack.apache.org
> Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable:
> Host 11: Link is closed
>
> Hi Cristian,
>
> Could you locate any stale management server record in the mshost table?
> If there is any ,specifically the one in the error ? update it as removed.
>
> -Jithin
>
> From: Cristian Ciobanu 
> Date: Wednesday, 17 January 2024 at 4:16 AM
> To: users@cloudstack.apache.org 
> Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable:
> Host 11: Link is closed Looks like I have the same issue as I had some 
> time ago...
>
> "host time out after cloudstack management restore". Old topic, 2022.
>  When I restored the backup on a different os, from Rocky to Ubuntu..  
> but now I used the same os... Different version..
>
> This looks like a bug. I'm trying to identify the route cause.
>
>
> Regards,
> Cristian
>
> On Wed, Jan 17, 2024, 00:20  wrote:
>
> > Hello,
> >
> >
> >
> > I have a CloudStack 4.18.1 restored from a DB backup due to server 
> > failure. The CloudStack is running; I see all the hosts with state 
> > 'Up', but it looks like it's stuck, nothing is working. I get a 
> > similar error for all hosts: “MgmtId 345050133919: Req: Resource
> [Host:11] is unreachable:
> > Host 11: Link is closed,” even if the hosts are reachable from 
> > CloudStack and vSphere, where they are connected and manageable. I 
> > do not see any events in vSphere.
> >
> >
> >
> > Any suggestions as to why CloudStack does not see that the servers 
> > are reachable or why I get a timeout?
> >
> >
> >
> > Thank you,
> >
> > Cristian
> >
>
>
>
>
>
>

RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 11: Link is closed

[cristian.c]

)
at 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:232)
at 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:116)
at 
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:78)
at 
org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
at 
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:70)
at 
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.(CloudStackSpringContext.java:57)
at 
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.(CloudStackSpringContext.java:61)
at 
org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
at 
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at 
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at 
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at 
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
at 
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at 
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at 
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at 
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at 
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at 
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:192)
at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:107)
2024-01-17 09:26:26,137 INFO  [c.c.c.ClusterManagerImpl] (main:null) (logid:) 
Detected that another management node with the same IP 10.60.0.2 is considered 
as running in DB, however it is not pingable, we will continue cluster 
initialization with this management server node
2024-01-17 09:26:26,137 INFO  [c.c.c.ClusterManagerImpl] (main:null) (logid:) 
Cluster manager is configured.


Regards,
Cristian




-Original Message-
From: cristian.c@istream.today  
Sent: Wednesday, January 17, 2024 9:24 AM
To: users@cloudstack.apache.org
Subject: RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 
11: Link is closed

Hi Jithin,

   I changed the status to down and set as removed. I still have the same 
issue...  So, I think I must create a new server with previous OS version, in 
any case not sure where is the logic.

Thank you!
Cristian 

-Original Message-
From: Jithin Raju 
Sent: Wednesday, January 17, 2024 6:16 AM
To: users@cloudstack.apache.org
Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 
11: Link is closed

Hi Cristian,

Could you locate any stale management server record in the mshost table? If 
there is any ,specifically the one in the error ? update it as removed.

-Jithin

From: Cristian Ciobanu 
Date: Wednesday, 17 January 2024 at 4:16 AM
To: users@cloudstack.apache.org 
Subject: Re: MgmtId 345050133919: Req: Resource

RE: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 11: Link is closed

[cristian.c]

Hi Jithin,

   I changed the status to down and set as removed. I still have the same 
issue...  So, I think I must create a new server with previous OS version, in 
any case not sure where is the logic.

Thank you!
Cristian 

-Original Message-
From: Jithin Raju  
Sent: Wednesday, January 17, 2024 6:16 AM
To: users@cloudstack.apache.org
Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 
11: Link is closed

Hi Cristian,

Could you locate any stale management server record in the mshost table? If 
there is any ,specifically the one in the error ? update it as removed.

-Jithin

From: Cristian Ciobanu 
Date: Wednesday, 17 January 2024 at 4:16 AM
To: users@cloudstack.apache.org 
Subject: Re: MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 
11: Link is closed Looks like I have the same issue as I had some time ago...

"host time out after cloudstack management restore". Old topic, 2022.
 When I restored the backup on a different os, from Rocky to Ubuntu..  but now 
I used the same os... Different version..

This looks like a bug. I'm trying to identify the route cause.


Regards,
Cristian

On Wed, Jan 17, 2024, 00:20  wrote:

> Hello,
>
>
>
> I have a CloudStack 4.18.1 restored from a DB backup due to server 
> failure. The CloudStack is running; I see all the hosts with state 
> 'Up', but it looks like it's stuck, nothing is working. I get a 
> similar error for all hosts: “MgmtId 345050133919: Req: Resource [Host:11] is 
> unreachable:
> Host 11: Link is closed,” even if the hosts are reachable from 
> CloudStack and vSphere, where they are connected and manageable. I do 
> not see any events in vSphere.
>
>
>
> Any suggestions as to why CloudStack does not see that the servers are 
> reachable or why I get a timeout?
>
>
>
> Thank you,
>
> Cristian
>

 

MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 11: Link is closed

[cristian.c]

Hello,

 

I have a CloudStack 4.18.1 restored from a DB backup due to server failure.
The CloudStack is running; I see all the hosts with state 'Up', but it looks
like it's stuck, nothing is working. I get a similar error for all hosts:
"MgmtId 345050133919: Req: Resource [Host:11] is unreachable: Host 11: Link
is closed," even if the hosts are reachable from CloudStack and vSphere,
where they are connected and manageable. I do not see any events in vSphere.

 

Any suggestions as to why CloudStack does not see that the servers are
reachable or why I get a timeout?



Thank you,

Cristian

Force delete/remove instances from lost host

[cristian.c]

Hello,

 

   Is there a way to force delete instances from a lost host? I still see
the VMs in CloudStack with state running/stopping. This host cannot be
recovered, I must remove this also.

 

 

 

Regards,

Cristian

RE: Load balancer configurations - maxcon

[cristian.c]

I see, thank you. For now I will adjust in the haproxy.cnf.

 

Regards,

Cristian

 

From: Sina Kashipazha  
Sent: Monday, December 4, 2023 3:16 PM
To: users 
Subject: Re: Load balancer configurations - maxcon

 

I think it is related to the following open PR:

 

https://github.com/apache/cloudstack/pull/7484

 

Regards,

Sina

 

On Mon, Dec 4, 2023 at 14:11, mailto:On%20Mon,%20Dec%204,%202023%20at%2014:11,%20%20%3c%3ca%20href=> > wrote:

Hello,



Is there a way to adjust the maxcon for a LB? isolated network



I have found this
https://cwiki.apache.org/confluence/display/CLOUDSTACK/VR+haproxy+customizat
ion+in+CloudStack but there is no such option in the UI. 4.18.1

Regards,

Cristian

Load balancer configurations - maxcon

[cristian.c]

Hello,

 

  Is there a way to adjust the maxcon for a LB? isolated network

 

  I have found this
https://cwiki.apache.org/confluence/display/CLOUDSTACK/VR+haproxy+customizat
ion+in+CloudStack   but there is no such option in the UI. 4.18.1

Regards,

Cristian

RE: Failed to fetch any free - False - ACS 4.18.1

[cristian.c]

After I executed :

select * from cloud.user_ip_address where removed is null and allocated is null 
and state !='Free';

I saw this (export as insert):  

INSERT INTO cloud.user_ip_address 
(uuid,account_id,domain_id,public_ip_address,data_center_id,source_nat,allocated,vlan_db_id,one_to_one_nat,vm_id,state,mac_address,source_network_id,network_id,physical_network_id,is_system,vpc_id,dnat_vmip,is_portable,display,removed,created,rule_state,forsystemvms)
 VALUES
 
('fd4a170b-127e-444d-bf7d-b6ad5529122b',NULL,NULL,'217.xx.2xx.12',7,0,NULL,63,0,NULL,'Allocated',1163,247,NULL,213,0,NULL,NULL,0,1,NULL,NULL,NULL,0);

There was no date for Allocated, it was with NULL.

Thank you,
Cristian



-Original Message-
From: Daan Hoogland  
Sent: Thursday, November 16, 2023 5:33 PM
To: users@cloudstack.apache.org
Subject: Re: Failed to fetch any free - False - ACS 4.18.1

just to recap Christian,
You had one with state free and removed null and now it works again?
Even though there where free IP before? that should not have been a problem. I 
think that is a bug. Can you exact how you fixed it?

On Thu, Nov 16, 2023 at 4:26 PM  wrote:
>
> Hi Wei,
>
>   Thank you for this!  There was 1 IP with removed null and state free. I 
> fixed.
>
>
> Regards,
> Cristian
>
>
>
> -Original Message-
> From: Wei ZHOU 
> Sent: Thursday, November 16, 2023 12:49 PM
> To: users@cloudstack.apache.org
> Subject: Re: Failed to fetch any free - False - ACS 4.18.1
>
> Is it possible that an IP is not allocated but the state is not Free ?
>
> The sql query below should return empty results.
>
> select * from cloud.user_ip_address where removed is null and 
> allocated is null and state !='Free';
>
> -Wei
>
>
> On Thu, 16 Nov 2023 at 10:11, Daan Hoogland  wrote:
>
> > strange Christian,
> >
> > I checked the code and the only thing I can think of is that 
> > creating a DB lock fails.
> >
> > if (_ipAddressDao.lockRow(possibleAddr.getId(), true) != null) {
> > final IPAddressVO userIp =
> > _ipAddressDao.findById(possibleAddr.getId());
> > if (userIp.getState() == State.Free) {
> > possibleAddr.setState(State.Allocating);
> > if (_ipAddressDao.update(possibleAddr.getId(), possibleAddr)) {
> > finalAddress = possibleAddr;
> > break;
> > }
> > }
> > }
> >
> > Unfortunately, no further logging can be turned on to investigate.
> >
> > is this a persistent error (i.e. remains after restarting system 
> > components like the MS and the DB)?
> >
> > On Wed, Nov 15, 2023 at 5:52 PM  wrote:
> > >
> > >
> > > Hello,
> > >
> > >
> > >
> > > Today, I tried to deploy new VMs, but I encountered the following error:
> > > failed to fetch any free public IP address. This is incorrect, as 
> > > even on the dashboard, I can see 12/22 for Public and 172/212 for 
> > > Shared. For
> > this
> > > deployment, I used the Shared/guest network. I also checked the 
> > > database
> > and
> > > saw 41 free IPs for this zone.
> > >
> > >
> > > Any suggestions?
> > >
> > >
> > > 2023-11-15 11:18:53,441 INFO  [c.c.h.v.m.VirtualMachineMO]
> > > (DirectAgent-19:ctx-a91068e2 usa-nyc-001.test.host, cmd:
> > > GetVolumeStatsCommand) (logid:851b339a) Look for disk device info 
> > > from volume : ROOT-6215_2.vmdk with base name: ROOT-6215_2
> > >
> > > 2023-11-15 11:18:53,441 INFO  [c.c.h.v.m.VirtualMachineMO]
> > > (DirectAgent-19:ctx-a91068e2 usa-nyc-001.test.host, cmd:
> > > GetVolumeStatsCommand) (logid:851b339a) Test against disk device,
> > controller
> > > key: 1000, unit number: 0
> > >
> > > 2023-11-15 11:18:53,441 INFO  [c.c.h.v.m.VirtualMachineMO]
> > > (DirectAgent-19:ctx-a91068e2 usa-nyc-001.test.host, cmd:
> > > GetVolumeStatsCommand) (logid:851b339a) Test against disk backing :
> > [nyc001]
> > > i-2-6215-VMAMS/ROOT-6215_2.vmdk
> > >
> > > 2023-11-15 11:18:53,441 INFO  [c.c.h.v.m.VirtualMachineMO]
> > > (DirectAgent-19:ctx-a91068e2 usa-nyc-001.test.host, cmd:
> > > GetVolumeStatsCommand) (logid:851b339a) Disk backing : [nyc001] 
> > > i-2-6215-VMAMS/ROOT-6215_2.vmdk matches ==> scsi0:0
> > >
> > > 2023-11-15 11:18:54,194 INFO  [c.c.n.NetworkModelImpl]
> > > (qtp1444635922-323:ctx-a813b819 ctx-3b16875f) (logid:2b3f6d9b) 
> > > Checking permission for account admin
> > > (17274c5e-d492-11e5-86c3-000c298715c8) on network 
> > > TestHost-Public-NYC (1bece1dc-d9c1-4bff-b5aa-67333a6830ae)
> > >
> > > 2023-11-15 11:18:54,309 INFO  [c.c.v.VirtualMachineManagerImpl]
> > > (qtp1444635922-323:ctx-a813b819 ctx-3b16875f) (logid:2b3f6d9b) 
> > > allocating virtual machine from
> > > template:391d4eab-1506-44be-b743-5ae7bd0f4ad7 with 
> > > hostname:i-2-6262-VMAMS and 1 networks
> > >
> > > 2023-11-15 11:18:54,325 ERROR [c.c.n.IpAddressManagerImpl]
> > > (qtp1444635922-323:ctx-a813b819 ctx-3b16875f) (logid:2b3f6d9b) 
> > > Failed to fetch any free
> > >
> > > public IP address
> > >
> > > 2023-11-15 11:18:54,340 ERROR [c.c.a.ApiServer]
> > > (qtp1444635922-323:ctx-a813b819 ctx-3b16875f) 

RE: Failed to fetch any free - False - ACS 4.18.1

[cristian.c]

Hi Wei,

  Thank you for this!  There was 1 IP with removed null and state free. I fixed.


Regards,
Cristian



-Original Message-
From: Wei ZHOU  
Sent: Thursday, November 16, 2023 12:49 PM
To: users@cloudstack.apache.org
Subject: Re: Failed to fetch any free - False - ACS 4.18.1

Is it possible that an IP is not allocated but the state is not Free ?

The sql query below should return empty results.

select * from cloud.user_ip_address where removed is null and allocated is null 
and state !='Free';

-Wei


On Thu, 16 Nov 2023 at 10:11, Daan Hoogland  wrote:

> strange Christian,
>
> I checked the code and the only thing I can think of is that creating 
> a DB lock fails.
>
> if (_ipAddressDao.lockRow(possibleAddr.getId(), true) != null) {
> final IPAddressVO userIp =
> _ipAddressDao.findById(possibleAddr.getId());
> if (userIp.getState() == State.Free) {
> possibleAddr.setState(State.Allocating);
> if (_ipAddressDao.update(possibleAddr.getId(), possibleAddr)) {
> finalAddress = possibleAddr;
> break;
> }
> }
> }
>
> Unfortunately, no further logging can be turned on to investigate.
>
> is this a persistent error (i.e. remains after restarting system 
> components like the MS and the DB)?
>
> On Wed, Nov 15, 2023 at 5:52 PM  wrote:
> >
> >
> > Hello,
> >
> >
> >
> > Today, I tried to deploy new VMs, but I encountered the following error:
> > failed to fetch any free public IP address. This is incorrect, as 
> > even on the dashboard, I can see 12/22 for Public and 172/212 for 
> > Shared. For
> this
> > deployment, I used the Shared/guest network. I also checked the 
> > database
> and
> > saw 41 free IPs for this zone.
> >
> >
> > Any suggestions?
> >
> >
> > 2023-11-15 11:18:53,441 INFO  [c.c.h.v.m.VirtualMachineMO]
> > (DirectAgent-19:ctx-a91068e2 usa-nyc-001.test.host, cmd:
> > GetVolumeStatsCommand) (logid:851b339a) Look for disk device info 
> > from volume : ROOT-6215_2.vmdk with base name: ROOT-6215_2
> >
> > 2023-11-15 11:18:53,441 INFO  [c.c.h.v.m.VirtualMachineMO]
> > (DirectAgent-19:ctx-a91068e2 usa-nyc-001.test.host, cmd:
> > GetVolumeStatsCommand) (logid:851b339a) Test against disk device,
> controller
> > key: 1000, unit number: 0
> >
> > 2023-11-15 11:18:53,441 INFO  [c.c.h.v.m.VirtualMachineMO]
> > (DirectAgent-19:ctx-a91068e2 usa-nyc-001.test.host, cmd:
> > GetVolumeStatsCommand) (logid:851b339a) Test against disk backing :
> [nyc001]
> > i-2-6215-VMAMS/ROOT-6215_2.vmdk
> >
> > 2023-11-15 11:18:53,441 INFO  [c.c.h.v.m.VirtualMachineMO]
> > (DirectAgent-19:ctx-a91068e2 usa-nyc-001.test.host, cmd:
> > GetVolumeStatsCommand) (logid:851b339a) Disk backing : [nyc001] 
> > i-2-6215-VMAMS/ROOT-6215_2.vmdk matches ==> scsi0:0
> >
> > 2023-11-15 11:18:54,194 INFO  [c.c.n.NetworkModelImpl]
> > (qtp1444635922-323:ctx-a813b819 ctx-3b16875f) (logid:2b3f6d9b) 
> > Checking permission for account admin 
> > (17274c5e-d492-11e5-86c3-000c298715c8) on network 
> > TestHost-Public-NYC (1bece1dc-d9c1-4bff-b5aa-67333a6830ae)
> >
> > 2023-11-15 11:18:54,309 INFO  [c.c.v.VirtualMachineManagerImpl]
> > (qtp1444635922-323:ctx-a813b819 ctx-3b16875f) (logid:2b3f6d9b) 
> > allocating virtual machine from 
> > template:391d4eab-1506-44be-b743-5ae7bd0f4ad7 with 
> > hostname:i-2-6262-VMAMS and 1 networks
> >
> > 2023-11-15 11:18:54,325 ERROR [c.c.n.IpAddressManagerImpl]
> > (qtp1444635922-323:ctx-a813b819 ctx-3b16875f) (logid:2b3f6d9b) 
> > Failed to fetch any free
> >
> > public IP address
> >
> > 2023-11-15 11:18:54,340 ERROR [c.c.a.ApiServer]
> > (qtp1444635922-323:ctx-a813b819 ctx-3b16875f) (logid:2b3f6d9b) 
> > unhandled exception executing api command: 
> > [Ljava.lang.String;@4a8c9f6c
> >
> > com.cloud.utils.exception.CloudRuntimeException: Failed to fetch any 
> > free public IP address
> >
> > at
> >
> com.cloud.network.IpAddressManagerImpl.lambda$assignAndAllocateIpAddre
> ssEntr
> > y$0(IpAddressManagerImpl.java:374)
> >
> > at 
> > com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> >
> > at
> >
> com.cloud.network.IpAddressManagerImpl.assignAndAllocateIpAddressEntry
> (IpAdd
> > ressManagerImpl.java:339)
> >
> > at
> >
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManag
> erImpl
> > .java:821)
> >
> > at
> >
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManag
> erImpl
> > .java:810)
> >
> > at
> >
> com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddress
> Manage
> > rImpl.java:778)
> >
> > at
> >
> com.cloud.network.IpAddressManagerImpl$10.doInTransactionWithoutResult
> (IpAdd
> > ressManagerImpl.java:2240)
> >
> > at
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransa
> ction(
> > TransactionCallbackWithExceptionNoReturn.java:25)
> >
> > at
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransa
> ction(
> > 

Failed to fetch any free - False - ACS 4.18.1

[cristian.c]

Hello,

 

Today, I tried to deploy new VMs, but I encountered the following error:
failed to fetch any free public IP address. This is incorrect, as even on
the dashboard, I can see 12/22 for Public and 172/212 for Shared. For this
deployment, I used the Shared/guest network. I also checked the database and
saw 41 free IPs for this zone.


Any suggestions?


2023-11-15 11:18:53,441 INFO  [c.c.h.v.m.VirtualMachineMO]
(DirectAgent-19:ctx-a91068e2 usa-nyc-001.test.host, cmd:
GetVolumeStatsCommand) (logid:851b339a) Look for disk device info from
volume : ROOT-6215_2.vmdk with base name: ROOT-6215_2

2023-11-15 11:18:53,441 INFO  [c.c.h.v.m.VirtualMachineMO]
(DirectAgent-19:ctx-a91068e2 usa-nyc-001.test.host, cmd:
GetVolumeStatsCommand) (logid:851b339a) Test against disk device, controller
key: 1000, unit number: 0

2023-11-15 11:18:53,441 INFO  [c.c.h.v.m.VirtualMachineMO]
(DirectAgent-19:ctx-a91068e2 usa-nyc-001.test.host, cmd:
GetVolumeStatsCommand) (logid:851b339a) Test against disk backing : [nyc001]
i-2-6215-VMAMS/ROOT-6215_2.vmdk

2023-11-15 11:18:53,441 INFO  [c.c.h.v.m.VirtualMachineMO]
(DirectAgent-19:ctx-a91068e2 usa-nyc-001.test.host, cmd:
GetVolumeStatsCommand) (logid:851b339a) Disk backing : [nyc001]
i-2-6215-VMAMS/ROOT-6215_2.vmdk matches ==> scsi0:0

2023-11-15 11:18:54,194 INFO  [c.c.n.NetworkModelImpl]
(qtp1444635922-323:ctx-a813b819 ctx-3b16875f) (logid:2b3f6d9b) Checking
permission for account admin (17274c5e-d492-11e5-86c3-000c298715c8) on
network TestHost-Public-NYC (1bece1dc-d9c1-4bff-b5aa-67333a6830ae)

2023-11-15 11:18:54,309 INFO  [c.c.v.VirtualMachineManagerImpl]
(qtp1444635922-323:ctx-a813b819 ctx-3b16875f) (logid:2b3f6d9b) allocating
virtual machine from template:391d4eab-1506-44be-b743-5ae7bd0f4ad7 with
hostname:i-2-6262-VMAMS and 1 networks

2023-11-15 11:18:54,325 ERROR [c.c.n.IpAddressManagerImpl]
(qtp1444635922-323:ctx-a813b819 ctx-3b16875f) (logid:2b3f6d9b) Failed to
fetch any free

public IP address

2023-11-15 11:18:54,340 ERROR [c.c.a.ApiServer]
(qtp1444635922-323:ctx-a813b819 ctx-3b16875f) (logid:2b3f6d9b) unhandled
exception executing api command: [Ljava.lang.String;@4a8c9f6c

com.cloud.utils.exception.CloudRuntimeException: Failed to fetch any free
public IP address

at
com.cloud.network.IpAddressManagerImpl.lambda$assignAndAllocateIpAddressEntr
y$0(IpAddressManagerImpl.java:374)

at com.cloud.utils.db.Transaction.execute(Transaction.java:40)

at
com.cloud.network.IpAddressManagerImpl.assignAndAllocateIpAddressEntry(IpAdd
ressManagerImpl.java:339)

at
com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl
.java:821)

at
com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl
.java:810)

at
com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddressManage
rImpl.java:778)

at
com.cloud.network.IpAddressManagerImpl$10.doInTransactionWithoutResult(IpAdd
ressManagerImpl.java:2240)

at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(
TransactionCallbackWithExceptionNoReturn.java:25)

at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(
TransactionCallbackWithExceptionNoReturn.java:21)

at com.cloud.utils.db.Transaction.execute(Transaction.java:40)

at
com.cloud.network.IpAddressManagerImpl.allocateDirectIp(IpAddressManagerImpl
.java:2220)

at
com.cloud.network.guru.DirectNetworkGuru$1.doInTransactionWithoutResult(Dire
ctNetworkGuru.java:324)

at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(
TransactionCallbackWithExceptionNoReturn.java:25)

at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(
TransactionCallbackWithExceptionNoReturn.java:21)

at com.cloud.utils.db.Transaction.execute(Transaction.java:40)

at
com.cloud.network.guru.DirectNetworkGuru.allocateDirectIp(DirectNetworkGuru.
java:317)

at
com.cloud.network.guru.DirectNetworkGuru.allocate(DirectNetworkGuru.java:290
)

at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocateNic(N
etworkOrchestrator.java:1025)

at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.addRequestedN
icToNicListWithDeviceNumberAndRetrieveDefaultDevice(NetworkOrchestrator.java
:877)

at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocate(Netw
orkOrchestrator.java:850)

at
com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.ja
va:491)

at
org.apache.cloudstack.engine.orchestration.CloudOrchestrator.createVirtualMa
chine(CloudOrchestrator.java:255)

at
com.cloud.vm.UserVmManagerImpl.commitUserVm(UserVmManagerImpl.java:4527)

at
com.cloud.vm.UserVmManagerImpl.commitUserVm(UserVmManagerImpl.java:4630)

at
com.cloud.vm.UserVmManagerImpl.getUncheckedUserVmResource(UserVmManagerImpl.
java:4261)

at

Redundant router - Isolated Network

[cristian.c]

Hello,

 

 I'm trying to enable a redundant router in an isolated network. The
issue I have is that I do not find the option to do this in the UI 4.18.1.
Is this a bug or is it not possible? I remember that it was possible in the
previous versions.


Thanks,

Cristian

RE: Live Instance Backup

[cristian.c]

Hello Ludjon,

That sounds like a prudent approach to ensure that our hosting solutions align 
well with your audience's needs and expectations. We're on board with beginning 
our collaboration through a Sponsored Review article, a listing in one of your 
articles, and a promotion in your newsletter. Could you please share the 
associated costs for these services? We’re eager to understand the investment 
required and to initiate this partnership.

Warm regards,

Cristian
Shape.host Team

-Original Message-
From: Simon Weller  
Sent: Monday, October 30, 2023 3:07 AM
To: users@cloudstack.apache.org
Subject: Re: Live Instance Backup

Fariborz,

KVM does support live (running) snapshots, but it's dependent on the underlying 
storage you're using. Currently, only NFS supports live snaps, as KVM stores 
the memory in a QCOW2 image.

There's a global option to enable this feature in CloudStack.

-Si

On Sun, Oct 29, 2023, 6:02 AM Fariborz Navidan 
wrote:

> Hi CS community,
>
> We're running CS 4.15.0.0 and KVM hypervisor. It doesn't support live 
> instance backups. Does the latest version of CS and KVM/QEMU have this 
> feature in place?
>
> TIA
> Cheers.
>

RE: dnsmasq-dhcp - ignored

[cristian.c]

Hello,

   I added the missing classes manually to the config, and this fixed the issue.

   Should I consider this a bug and submit on github?  

Regards,
Cristian

-Original Message-
From: cristian.c@istream.today  
Sent: Friday, October 20, 2023 11:07 AM
To: users@cloudstack.apache.org
Subject: RE: dnsmasq-dhcp - ignored

Hello,

   

   If I'm right the problem is here : 

  

root@r-6143-VMAMS:/etc/dnsmasq.d# cat cloud.conf
listen-address=127.0.0.1,23.29.xxx.130
dhcp-range=set:interface-eth0-0,23.29.xxx.130,static
dhcp-option=tag:interface-eth0-0,15,shape.host
dhcp-option=tag:interface-eth0-0,6,23.29.xxx.130,8.8.8.8
dhcp-option=tag:interface-eth0-0,3,23.29.xxx.129
dhcp-option=eth0,26,1500
dhcp-option=tag:interface-eth0-0,1,255.255.255.224
root@r-6143-VMAMS:/etc/dnsmasq.d#


the dhcp-option are missing for the not working IP classes. I just compared 
with a different router where I have multiple cidr, and all are present with 
dhcp-options but different zone.

 Version
4.17.2
Software version
4.18.1.0


And on the router with the issue see this :
Version
4.18.1
Software version
4.18.1.0


Regards,
Cristian


-Original Message-
From: cristian.c@istream.today 
Sent: Friday, October 20, 2023 1:02 AM
To: users@cloudstack.apache.org
Subject: RE: dnsmasq-dhcp - ignored

Please see :

root@r-6143-VMAMS:~# grep -v '^#\|^$' /etc/dnsmasq.conf domain-needed 
bogus-priv resolv-file=/etc/dnsmasq-resolv.conf
local=/my.host/
except-interface=eth1
except-interface=eth2
except-interface=lo
no-dhcp-interface=eth1
no-dhcp-interface=eth2
expand-hosts
domain=my.host
domain=my.host
domain=my.host
dhcp-range=217.79.xxx.129,static
dhcp-hostsfile=/etc/dhcphosts.txt
dhcp-ignore=tag:!known
dhcp-option=15,"my.host "
dhcp-option=vendor:MSFT,2,1i
dhcp-boot=pxelinux.0
enable-tftp
tftp-root=/opt/tftpboot
dhcp-lease-max=2100
domain=shape.host
log-facility=/var/log/dnsmasq.log
conf-dir=/etc/dnsmasq.d
dhcp-optsfile=/etc/dhcpopts.txt
localise-queries
dhcp-option=option:router,217.79.xxx.129
dhcp-option=6,217.79.xxx.130,8.8.8.8
dhcp-client-update




root@r-6143-VMAMS:~# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group 
default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc mq state UP group 
default qlen 1000
link/ether 1e:00:e4:00:00:64 brd ff:ff:ff:ff:ff:ff
altname enp3s0
altname ens160
inet 217.79.xxx.130/25 brd 217.79.xxx.255 scope global eth0
   valid_lft forever preferred_lft forever
inet 23.227.xxx.35/27 brd 23.227.xxx.63 scope global eth0
   valid_lft forever preferred_lft forever
inet 217.79.xxx.12/28 brd 217.79.xxx.15 scope global eth0
   valid_lft forever preferred_lft forever
inet 66.165.xxx.92/29 brd 66.165.xxx.95 scope global eth0
   valid_lft forever preferred_lft forever
inet 217.79.xxx.28/29 brd 217.79.xxx.31 scope global eth0
   valid_lft forever preferred_lft forever
inet 190.102.xxx.116/29 brd 190.102.xxx.119 scope global eth0
   valid_lft forever preferred_lft forever
inet 66.206.xxx.179/29 brd 66.206.xxx.183 scope global eth0
   valid_lft forever preferred_lft forever
inet 23.29.xxx.130/27 brd 23.29.xxx.159 scope global eth0
   valid_lft forever preferred_lft forever
3: eth1:  mtu 1500 qdisc mq state UP group 
default qlen 1000
link/ether 02:07:00:ef:01:23 brd ff:ff:ff:ff:ff:ff
altname enp11s0
altname ens192
inet 10.102.0.152/24 brd 10.102.0.255 scope global eth1
   valid_lft forever preferred_lft forever



root@r-6143-VMAMS:/etc/dnsmasq.d# ls
cloud.conf  README
root@r-6143-VMAMS:/etc/dnsmasq.d# cat cloud.conf
listen-address=127.0.0.1,23.29.xxx.130
dhcp-range=set:interface-eth0-0,23.29.xxx.130,static
dhcp-option=tag:interface-eth0-0,15,my.host
dhcp-option=tag:interface-eth0-0,6,23.29.xxx.130,8.8.8.8
dhcp-option=tag:interface-eth0-0,3,23.29.xxx.129
dhcp-option=eth0,26,1500
dhcp-option=tag:interface-eth0-0,1,255.255.255.224
root@r-6143-VMAMS:/etc/dnsmasq.d#


Regards,
Cristian



-Original Message-
From: Wei ZHOU 
Sent: Thursday, October 19, 2023 11:24 PM
To: users@cloudstack.apache.org
Subject: Re: dnsmasq-dhcp - ignored

Can you share /etc/dnsmasq.conf and /etc/dnsmasq.d/*?

AND the output of "ip a"?

-Wei

在 2023年10月19日星期四，Cristian Ciobanu  写道：

> Hi Wei,
>
>
>
> I use shared network without security groups. I did a cleanup and same 
> issue after. I have shared network and advanced in multiple different 
> zones, but only in one zone I have this issue. I think it started 
> suddenly after a deploy.
>
> Here is a tcpdump from router, looks how ignore still. I turn off and 
> change to a different IP, and then I turn on and the IP is allocated.
>
> Request with ignored request for IPs from this range 23.227.xxx.32/27:
>
> 19:44:20.453093 1e:00:d5:00:00:79 > ff:ff:ff:ff:ff:ff, ethertype IPv4 

RE: dnsmasq-dhcp - ignored

[cristian.c]

Hello,


   If I'm right the problem is here : 


root@r-6143-VMAMS:/etc/dnsmasq.d# cat cloud.conf
listen-address=127.0.0.1,23.29.xxx.130
dhcp-range=set:interface-eth0-0,23.29.xxx.130,static
dhcp-option=tag:interface-eth0-0,15,shape.host
dhcp-option=tag:interface-eth0-0,6,23.29.xxx.130,8.8.8.8
dhcp-option=tag:interface-eth0-0,3,23.29.xxx.129
dhcp-option=eth0,26,1500
dhcp-option=tag:interface-eth0-0,1,255.255.255.224
root@r-6143-VMAMS:/etc/dnsmasq.d#


the dhcp-option are missing for the not working IP classes. I just compared 
with a different router where I have multiple cidr, and all are present with 
dhcp-options but different zone.

 Version
4.17.2
Software version
4.18.1.0


And on the router with the issue see this :
Version
4.18.1
Software version
4.18.1.0


Regards,
Cristian


-Original Message-
From: cristian.c@istream.today  
Sent: Friday, October 20, 2023 1:02 AM
To: users@cloudstack.apache.org
Subject: RE: dnsmasq-dhcp - ignored

Please see :

root@r-6143-VMAMS:~# grep -v '^#\|^$' /etc/dnsmasq.conf domain-needed 
bogus-priv resolv-file=/etc/dnsmasq-resolv.conf
local=/shape.host/
except-interface=eth1
except-interface=eth2
except-interface=lo
no-dhcp-interface=eth1
no-dhcp-interface=eth2
expand-hosts
domain=shape.host
domain=shape.host
domain=shape.host
dhcp-range=217.79.xxx.129,static
dhcp-hostsfile=/etc/dhcphosts.txt
dhcp-ignore=tag:!known
dhcp-option=15,"shape.host"
dhcp-option=vendor:MSFT,2,1i
dhcp-boot=pxelinux.0
enable-tftp
tftp-root=/opt/tftpboot
dhcp-lease-max=2100
domain=shape.host
log-facility=/var/log/dnsmasq.log
conf-dir=/etc/dnsmasq.d
dhcp-optsfile=/etc/dhcpopts.txt
localise-queries
dhcp-option=option:router,217.79.xxx.129
dhcp-option=6,217.79.xxx.130,8.8.8.8
dhcp-client-update




root@r-6143-VMAMS:~# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group 
default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc mq state UP group 
default qlen 1000
link/ether 1e:00:e4:00:00:64 brd ff:ff:ff:ff:ff:ff
altname enp3s0
altname ens160
inet 217.79.xxx.130/25 brd 217.79.xxx.255 scope global eth0
   valid_lft forever preferred_lft forever
inet 23.227.xxx.35/27 brd 23.227.xxx.63 scope global eth0
   valid_lft forever preferred_lft forever
inet 217.79.xxx.12/28 brd 217.79.xxx.15 scope global eth0
   valid_lft forever preferred_lft forever
inet 66.165.xxx.92/29 brd 66.165.xxx.95 scope global eth0
   valid_lft forever preferred_lft forever
inet 217.79.xxx.28/29 brd 217.79.xxx.31 scope global eth0
   valid_lft forever preferred_lft forever
inet 190.102.xxx.116/29 brd 190.102.xxx.119 scope global eth0
   valid_lft forever preferred_lft forever
inet 66.206.xxx.179/29 brd 66.206.xxx.183 scope global eth0
   valid_lft forever preferred_lft forever
inet 23.29.xxx.130/27 brd 23.29.xxx.159 scope global eth0
   valid_lft forever preferred_lft forever
3: eth1:  mtu 1500 qdisc mq state UP group 
default qlen 1000
link/ether 02:07:00:ef:01:23 brd ff:ff:ff:ff:ff:ff
altname enp11s0
altname ens192
inet 10.102.0.152/24 brd 10.102.0.255 scope global eth1
   valid_lft forever preferred_lft forever



root@r-6143-VMAMS:/etc/dnsmasq.d# ls
cloud.conf  README
root@r-6143-VMAMS:/etc/dnsmasq.d# cat cloud.conf
listen-address=127.0.0.1,23.29.xxx.130
dhcp-range=set:interface-eth0-0,23.29.xxx.130,static
dhcp-option=tag:interface-eth0-0,15,shape.host
dhcp-option=tag:interface-eth0-0,6,23.29.xxx.130,8.8.8.8
dhcp-option=tag:interface-eth0-0,3,23.29.xxx.129
dhcp-option=eth0,26,1500
dhcp-option=tag:interface-eth0-0,1,255.255.255.224
root@r-6143-VMAMS:/etc/dnsmasq.d#


Regards,
Cristian



-Original Message-
From: Wei ZHOU 
Sent: Thursday, October 19, 2023 11:24 PM
To: users@cloudstack.apache.org
Subject: Re: dnsmasq-dhcp - ignored

Can you share /etc/dnsmasq.conf and /etc/dnsmasq.d/*?

AND the output of "ip a"?

-Wei

在 2023年10月19日星期四，Cristian Ciobanu  写道：

> Hi Wei,
>
>
>
> I use shared network without security groups. I did a cleanup and same 
> issue after. I have shared network and advanced in multiple different 
> zones, but only in one zone I have this issue. I think it started 
> suddenly after a deploy.
>
> Here is a tcpdump from router, looks how ignore still. I turn off and 
> change to a different IP, and then I turn on and the IP is allocated.
>
> Request with ignored request for IPs from this range 23.227.xxx.32/27:
>
> 19:44:20.453093 1e:00:d5:00:00:79 > ff:ff:ff:ff:ff:ff, ethertype IPv4 
> (0x0800), length 334: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, 
> Request from 1e:00:d5:00:00:79, length 292
>
> 19:44:24.319207 1e:00:d5:00:00:79 > ff:ff:ff:ff:ff:ff, ethertype IPv4 
> (0x0800), length 334: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, 
> Request from 1e:00:d5:00:00:79, 

RE: dnsmasq-dhcp - ignored

[cristian.c]

Please see :

root@r-6143-VMAMS:~# grep -v '^#\|^$' /etc/dnsmasq.conf
domain-needed
bogus-priv
resolv-file=/etc/dnsmasq-resolv.conf
local=/shape.host/
except-interface=eth1
except-interface=eth2
except-interface=lo
no-dhcp-interface=eth1
no-dhcp-interface=eth2
expand-hosts
domain=shape.host
domain=shape.host
domain=shape.host
dhcp-range=217.79.xxx.129,static
dhcp-hostsfile=/etc/dhcphosts.txt
dhcp-ignore=tag:!known
dhcp-option=15,"shape.host"
dhcp-option=vendor:MSFT,2,1i
dhcp-boot=pxelinux.0
enable-tftp
tftp-root=/opt/tftpboot
dhcp-lease-max=2100
domain=shape.host
log-facility=/var/log/dnsmasq.log
conf-dir=/etc/dnsmasq.d
dhcp-optsfile=/etc/dhcpopts.txt
localise-queries
dhcp-option=option:router,217.79.xxx.129
dhcp-option=6,217.79.xxx.130,8.8.8.8
dhcp-client-update




root@r-6143-VMAMS:~# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group 
default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc mq state UP group 
default qlen 1000
link/ether 1e:00:e4:00:00:64 brd ff:ff:ff:ff:ff:ff
altname enp3s0
altname ens160
inet 217.79.xxx.130/25 brd 217.79.xxx.255 scope global eth0
   valid_lft forever preferred_lft forever
inet 23.227.xxx.35/27 brd 23.227.xxx.63 scope global eth0
   valid_lft forever preferred_lft forever
inet 217.79.xxx.12/28 brd 217.79.xxx.15 scope global eth0
   valid_lft forever preferred_lft forever
inet 66.165.xxx.92/29 brd 66.165.xxx.95 scope global eth0
   valid_lft forever preferred_lft forever
inet 217.79.xxx.28/29 brd 217.79.xxx.31 scope global eth0
   valid_lft forever preferred_lft forever
inet 190.102.xxx.116/29 brd 190.102.xxx.119 scope global eth0
   valid_lft forever preferred_lft forever
inet 66.206.xxx.179/29 brd 66.206.xxx.183 scope global eth0
   valid_lft forever preferred_lft forever
inet 23.29.xxx.130/27 brd 23.29.xxx.159 scope global eth0
   valid_lft forever preferred_lft forever
3: eth1:  mtu 1500 qdisc mq state UP group 
default qlen 1000
link/ether 02:07:00:ef:01:23 brd ff:ff:ff:ff:ff:ff
altname enp11s0
altname ens192
inet 10.102.0.152/24 brd 10.102.0.255 scope global eth1
   valid_lft forever preferred_lft forever



root@r-6143-VMAMS:/etc/dnsmasq.d# ls
cloud.conf  README
root@r-6143-VMAMS:/etc/dnsmasq.d# cat cloud.conf
listen-address=127.0.0.1,23.29.xxx.130
dhcp-range=set:interface-eth0-0,23.29.xxx.130,static
dhcp-option=tag:interface-eth0-0,15,shape.host
dhcp-option=tag:interface-eth0-0,6,23.29.xxx.130,8.8.8.8
dhcp-option=tag:interface-eth0-0,3,23.29.xxx.129
dhcp-option=eth0,26,1500
dhcp-option=tag:interface-eth0-0,1,255.255.255.224
root@r-6143-VMAMS:/etc/dnsmasq.d#


Regards,
Cristian



-Original Message-
From: Wei ZHOU  
Sent: Thursday, October 19, 2023 11:24 PM
To: users@cloudstack.apache.org
Subject: Re: dnsmasq-dhcp - ignored

Can you share /etc/dnsmasq.conf and /etc/dnsmasq.d/*?

AND the output of "ip a"?

-Wei

在 2023年10月19日星期四，Cristian Ciobanu  写道：

> Hi Wei,
>
>
>
> I use shared network without security groups. I did a cleanup and same 
> issue after. I have shared network and advanced in multiple different 
> zones, but only in one zone I have this issue. I think it started 
> suddenly after a deploy.
>
> Here is a tcpdump from router, looks how ignore still. I turn off and 
> change to a different IP, and then I turn on and the IP is allocated.
>
> Request with ignored request for IPs from this range 23.227.xxx.32/27:
>
> 19:44:20.453093 1e:00:d5:00:00:79 > ff:ff:ff:ff:ff:ff, ethertype IPv4 
> (0x0800), length 334: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, 
> Request from 1e:00:d5:00:00:79, length 292
>
> 19:44:24.319207 1e:00:d5:00:00:79 > ff:ff:ff:ff:ff:ff, ethertype IPv4 
> (0x0800), length 334: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, 
> Request from 1e:00:d5:00:00:79, length 292
>
> 19:44:28.051571 1e:00:d5:00:00:79 > ff:ff:ff:ff:ff:ff, ethertype IPv4 
> (0x0800), length 334: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, 
> Request from 1e:00:d5:00:00:79, length 292
>
> 19:44:35.599400 1e:00:d5:00:00:79 > ff:ff:ff:ff:ff:ff, ethertype IPv4 
> (0x0800), length 334: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, 
> Request from 1e:00:d5:00:00:79, length 292
>
> 19:44:52.334213 1e:00:d5:00:00:79 > ff:ff:ff:ff:ff:ff, ethertype IPv4 
> (0x0800), length 334: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, 
> Request from 1e:00:d5:00:00:79, length 292
>
>
>
>
>
> And same instance after I turn off and change again the IP/ turn on. 
> Here
> works:
>
>
>
> 19:48:38.224138 1e:00:d5:00:00:79 > ff:ff:ff:ff:ff:ff, ethertype IPv4 
> (0x0800), length 334: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, 
> Request from 1e:00:d5:00:00:79, length 292
>
> 19:48:38.224432 1e:00:19:00:00:64 > 1e:00:d5:00:00:79, ethertype IPv4 
> (0x0800), length 352: 23.29.xxx.130.67 > 23.29.xxx.131.68: BOOTP/DHCP, 
> Reply, length 310
>
> 19:48:38.235802 

dnsmasq-dhcp - ignored

[cristian.c]

Hello,

 

  I'm experiencing strange issue, the Guest router started today to ignore
the requests for specific IP class.

Oct 19 16:22:49 dnsmasq-dhcp[24437]: 49204406 available DHCP subnet:
217.79.xxx.129/255.255.255.128

Oct 19 16:22:49 dnsmasq-dhcp[24437]: 49204406 available DHCP subnet:
66.206.xxx.179/255.255.255.248

Oct 19 16:22:49 dnsmasq-dhcp[24437]: 49204406 client provides name: ubuntu

Oct 19 16:22:49 dnsmasq-dhcp[24437]: 49204406 DHCPDISCOVER(eth0)
1e:00:30:00:04:67 ignored

Oct 19 16:23:53 dnsmasq-dhcp[24437]: 49204406 available DHCP subnet:
217.79.xxx.129/255.255.255.128

Oct 19 16:23:53 dnsmasq-dhcp[24437]: 49204406 available DHCP subnet:
66.206.xxx.179/255.255.255.248

Oct 19 16:23:53 dnsmasq-dhcp[24437]: 49204406 client provides name: ubuntu

Oct 19 16:23:53 dnsmasq-dhcp[24437]: 49204406 DHCPDISCOVER(eth0)
1e:00:30:00:04:67 ignored

 

 

root@r-6137-VMAMS:~# cat /etc/dhcphosts.txt | grep "1e:00:30:00:04:67"

1e:00:30:00:04:67,23.227.xxx.54,testignore001,infinite

 

root@r-6137-VMAMS:~# cat /etc/dhcphosts.txt | wc -l

144

 

If I reboot any VM with an IP from this class 23.227.xxx.129, which was
allocated without any issue before, after reboot is I'm going to see ignored
and is going to be left without IP assigned.

Setup: ACS 4.18.1 upgrade from previous version with VMware.

Any suggestions?

Thank you.

K8S Cluster using CAPC - Autoscaling

[cristian.c]

Hello,

 

   Is there a way to implement autoscaling for K8S cluster deployed using
CAPC? I'm asking this because these cluster are not listed via ACS API, like
the clusters deployed via Kubernetes plugin, implemented in ACS.

 

 

Thank you,

Cristian

RE: Metrics not visible

[cristian.c]

Hello Bryan,

 

  You’re right, my bad.  I forgot to check this configuration. I saw that is 
enabled and I ignored the vm.stats.max.retention.time.  It was set to 1. I 
upgraded from previous version.

Thank you,

Cristian

 

From: Bryan Lima  
Sent: Tuesday, October 17, 2023 5:01 PM
To: users@cloudstack.apache.org; cristian.c@istream.today; 'Nux' 

Subject: Re: Metrics not visible

 

Hello, Cristian

The VM's stats visualization is controlled by the global configuration 
vm.stats.interval and vm.stats.max.retention.time, where the former defines the 
interval of the data collection, and the latter defines for how long this data 
will be persisted in the database until its removal. The default values for 
these configurations were 6 (milliseconds) and 1 (minute), respectively. 
With these values, you should see only 1 data point with the default filter of 
1 hour in the UI, as presented in the screenshot. However, in version 4.18.0.1 
the default value of the configuration vm.stats.max.retention.time was changed 
to 720 minutes (12 hours); if you upgraded your environment from a prior 
version, the value of the configuration was not changed. Can you check and 
report the values of the mentioned configurations?

Furthermore, issue #5935[1] presents the specification of this feature and may 
help you understand its behavior.

Best regards,
Bryan

[1]: https://github.com/apache/cloudstack/issues/5935

On 10/17/23 06:05, cristian.c@istream.today <mailto:cristian.c@istream.today>  
wrote:

Hi Nux,
 
   The ACS version is 4.18.1, I see the same issue in different environments
with vmware.
 
  Screenshots :
  https://prnt.sc/tG10_GOjiVLr
 
 
Regards,
Cristian
  
 
-Original Message-
From: Nux  <mailto:n...@li.nux.ro>  
Sent: Tuesday, October 17, 2023 11:59 AM
To: users@cloudstack.apache.org <mailto:users@cloudstack.apache.org> 
Cc: cristian.c@istream.today <mailto:cristian.c@istream.today> 
Subject: Re: Metrics not visible
 
Can't see the image, maybe link it elsewhere.
Also which Cloudstack version etc?
 
On 2023-10-17 09:49, cristian.c@istream.today <mailto:cristian.c@istream.today> 
 wrote:

Hello,
 
  Is this a bug? I see like this for all VMs.
 
Best regards,
 
Cristian

RE: Metrics not visible

[cristian.c]

Hi Nux,

   The ACS version is 4.18.1, I see the same issue in different environments
with vmware.

  Screenshots :
  https://prnt.sc/tG10_GOjiVLr


Regards,
Cristian
  

-Original Message-
From: Nux  
Sent: Tuesday, October 17, 2023 11:59 AM
To: users@cloudstack.apache.org
Cc: cristian.c@istream.today
Subject: Re: Metrics not visible

Can't see the image, maybe link it elsewhere.
Also which Cloudstack version etc?

On 2023-10-17 09:49, cristian.c@istream.today wrote:
> Hello,
> 
>   Is this a bug? I see like this for all VMs.
> 
> Best regards,
> 
> Cristian

Metrics not visible

[cristian.c]

Hello,

 

  Is this a bug? I see like this for all VMs.



Best regards,

Cristian

RE: ISO Detach Issue

[cristian.c]

Hi,

 

I have the 4.18.1 and I see the detach option. You must detach before creating 
a template from it.





Regards,

Cristian

From: Granwille Strauss  
Sent: Thursday, October 12, 2023 5:35 PM
To: users@cloudstack.apache.org
Subject: ISO Detach Issue

 

Hi Guys

Something weird is going on again. I am trying to launch a new VM via UI using 
a newly registered Almalinux 8 ISO. As soon as the VM instance is created, I 
can use console to begin the OS install process. However, checking the VM 
instance options in the UI, I see no option to "Detach ISO" from the VM when I 
complete the install. So now, when I reboot the instance it always boots the 
ISO first. I tried to detach ISO via virsh, I get successful eject message but 
VM remains booting in ISO mode when I reboot VM. This is for ACS 4.18.1.0  via 
UI. 

-- 




Regards / Groete


  

Granwille Strauss  //  Senior Systems Admin

e:   granwi...@namhost.com
m:   +264 81 323 1260
w:   www.namhost.com

     
  
  
 



 

 


Namhost Internet Services (Pty) Ltd, 

24 Black Eagle Rd, Hermanus, 7210, RSA

 


The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It is 
forbidden to copy, forward, or in any way reveal the contents of this message 
to anyone without our explicit consent. The integrity and security of this 
email cannot be guaranteed over the Internet. Therefore, the sender will not be 
held liable for any damage caused by the message. For our full privacy policy 
and disclaimers, please go to https://www.namhost.com/privacy-policy

   

 

 

RE: Index 0 out of bounds for length 0 - Scale VM

[cristian.c]

Hi Wei,

  There are no reason to migrate, there is enough capacity on the same host. No 
storage tags and we use local storage.

Regards,
Cristian

-Original Message-
From: Wei ZHOU  
Sent: Monday, September 18, 2023 11:24 AM
To: users@cloudstack.apache.org
Subject: Re: Index 0 out of bounds for length 0 - Scale VM

Hi,

it looks there is no suitable pool found for the migration. good to know you 
fixed by manual db changes.
However, If you expect the volume not be migrated in the future, it would be 
better to figure out why cloudstack tried to migrate it ( probably due to 
storage tags or capacity, etc)

-Wei

On Mon, 18 Sept 2023 at 09:42,  wrote:

> Hi Wei,
>
>Please see,.:
>
> 2023-09-18 02:07:13,556 INFO  [c.c.s.ManagementServerImpl]
> (API-Job-Executor-2:ctx-89843446 job-85583 ctx-db5354a6) 
> (logid:f25c5504) Volume Vol[709
> 8|name=ROOT-4591|vm=4591|ROOT] isn't attached to any running vm. 
> 8|Looking
> for storage pools in the cluster to which this volumes can be migrated.
> 2023-09-18 02:07:13,565 INFO  [c.c.s.StorageManagerImpl]
> (API-Job-Executor-2:ctx-89843446 job-85583 ctx-db5354a6) 
> (logid:f25c5504) Storage pool tro -ca-001.internal.host-local-dd761a4d 
> (36) does not supply IOPS capacity, assuming enough capacity
> 2023-09-18 02:07:13,577 INFO  [c.c.s.StorageManagerImpl]
> (API-Job-Executor-2:ctx-89843446 job-85583 ctx-db5354a6) 
> (logid:f25c5504) Storage pool tro
> -ca-002.internal.host-local-bc4e09c8 (41) does not supply IOPS 
> capacity, assuming enough capacity
> 2023-09-18 02:07:13,595 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-2:ctx-89843446 job-85583) (logid:f25c5504) 
> Unexpected exception while  executing 
> org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin
> java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
> at
> java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
> at
> java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
> at
> java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
> at java.base/java.util.Objects.checkIndex(Objects.java:372)
> at java.base/java.util.ArrayList.get(ArrayList.java:459)
> at
> com.cloud.storage.VolumeApiServiceImpl.changeDiskOfferingForVolumeInternal(VolumeApiServiceImpl.java:1898)
> at
> com.cloud.storage.VolumeApiServiceImpl.changeDiskOfferingForVolume(VolumeApiServiceImpl.java:1824)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> at com.sun.proxy.$Proxy232.changeDiskOfferingForVolume(Unknown
> Source)
> at
> com.cloud.vm.UserVmManagerImpl.changeDiskOfferingForRootVolume(UserVmManagerImpl.java:2131)
> at
> com.cloud.vm.UserVmManagerImpl.upgradeStoppedVirtualMachine(UserVmManagerImpl.java:1305)
> at
> com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1885)
> at
> com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1859)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> 

RE: Index 0 out of bounds for length 0 - Scale VM

[cristian.c]

Hi Wei,

   Please see,.:

2023-09-18 02:07:13,556 INFO  [c.c.s.ManagementServerImpl] 
(API-Job-Executor-2:ctx-89843446 job-85583 ctx-db5354a6) (logid:f25c5504) 
Volume Vol[709
8|name=ROOT-4591|vm=4591|ROOT] isn't attached to any running vm. Looking for 
storage pools in the cluster to which this volumes can be migrated.
2023-09-18 02:07:13,565 INFO  [c.c.s.StorageManagerImpl] 
(API-Job-Executor-2:ctx-89843446 job-85583 ctx-db5354a6) (logid:f25c5504) 
Storage pool tro
-ca-001.internal.host-local-dd761a4d (36) does not supply IOPS capacity, 
assuming enough capacity
2023-09-18 02:07:13,577 INFO  [c.c.s.StorageManagerImpl] 
(API-Job-Executor-2:ctx-89843446 job-85583 ctx-db5354a6) (logid:f25c5504) 
Storage pool tro
-ca-002.internal.host-local-bc4e09c8 (41) does not supply IOPS capacity, 
assuming enough capacity
2023-09-18 02:07:13,595 ERROR [c.c.a.ApiAsyncJobDispatcher] 
(API-Job-Executor-2:ctx-89843446 job-85583) (logid:f25c5504) Unexpected 
exception while
 executing org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin
java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at 
java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
at 
java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
at 
java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
at java.base/java.util.Objects.checkIndex(Objects.java:372)
at java.base/java.util.ArrayList.get(ArrayList.java:459)
at 
com.cloud.storage.VolumeApiServiceImpl.changeDiskOfferingForVolumeInternal(VolumeApiServiceImpl.java:1898)
at 
com.cloud.storage.VolumeApiServiceImpl.changeDiskOfferingForVolume(VolumeApiServiceImpl.java:1824)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at 
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at 
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy232.changeDiskOfferingForVolume(Unknown Source)
at 
com.cloud.vm.UserVmManagerImpl.changeDiskOfferingForRootVolume(UserVmManagerImpl.java:2131)
at 
com.cloud.vm.UserVmManagerImpl.upgradeStoppedVirtualMachine(UserVmManagerImpl.java:1305)
at 
com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1885)
at 
com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1859)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at 
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at 
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at 

Index 0 out of bounds for length 0 - Scale VM

[cristian.c]

Hello,

 

   I just upgraded to cloudstack 4.18.1 from 4.17.2, and looks like the 
offering change/scale still not working. Any suggestion how can I scale these 
VMs without rebuild?

 

   (localcloud)  > scale virtualmachine 
id=191178f7-4f55-460c-87e8-5261d962f32f 
serviceofferingid=c7822345-0f4f-4425-a544-48af4380c5a0

{

  "accountid": "17274c5e-d492-11e5-86c3-000c298715c8",

  "cmd": "org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin",

  "completed": "2023-09-18T02:08:22-0400",

  "created": "2023-09-18T02:08:22-0400",

  "jobid": "ca1dd5f2-985c-40d8-bcc7-0d0c60db156d",

  "jobinstanceid": "191178f7-4f55-460c-87e8-5261d962f32f",

  "jobinstancetype": "VirtualMachine",

  "jobprocstatus": 0,

  "jobresult": {

"errorcode": 431,

"errortext": "Failed to change offering for volume 
4f529ab2-e8f4-4844-8b8f-514fcc78647a since automigrate is set to false but 
volume needs to migrated"

  },

  "jobresultcode": 431,

  "jobresulttype": "object",

  "jobstatus": 2,

  "userid": "1727541a-d492-11e5-86c3-000c298715c8"

}

 Error: async API failed for job ca1dd5f2-985c-40d8-bcc7-0d0c60db156d

(localcloud)  > scale virtualmachine id=191178f7-4f55-460c-87e8-5261d962f32f 
serviceofferingid=c7822345-0f4f-4425-a544-48af4380c5a0 automigrate=true

{

  "accountid": "17274c5e-d492-11e5-86c3-000c298715c8",

  "cmd": "org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin",

  "completed": "2023-09-18T02:08:46-0400",

  "created": "2023-09-18T02:08:46-0400",

  "jobid": "d052fa4d-b62d-4539-98c7-d9e7a915feb9",

  "jobinstanceid": "191178f7-4f55-460c-87e8-5261d962f32f",

  "jobinstancetype": "VirtualMachine",

  "jobprocstatus": 0,

  "jobresult": {

"errorcode": 530,

"errortext": "Index 0 out of bounds for length 0"

  },

  "jobresultcode": 530,

  "jobresulttype": "object",

  "jobstatus": 2,

  "userid": "1727541a-d492-11e5-86c3-000c298715c8"

}

 Error: async API failed for job d052fa4d-b62d-4539-98c7-d9e7a915feb9


Thank you,

Cristian

Network ID is already under use by another Kubernetes cluster

[cristian.c]

Hello,

 

  I just tried to create a Kubernetes cluster and I get this error : Network
ID: fffdc800-1c6e-4fb7-a8b6-34b10218245e is already under use by another
Kubernetes cluster.   I'm not allowed to have multiple cluster in the same
network?  


  I'm asking this because I saw that there is no option to create multiple
node pools inside a cluster ( similar to digital ocean ).

Thank you,

Cristian

RE: set display_name as hostname to instances

[cristian.c]

The reason why I asked for this it is because you can not have duplicated VM
names even if are different account, under /ROOT let say VM name "Linux"
beside this you can not put a name like "domain.tld"   only domain-tld,
which is quite annoying...  
I understand the current purpose for display name,  this is the reason why I
asked if it is possible to set this as hostname on the VMs.

Regards,
Cristian



-Original Message-
From: Rohit Yadav  
Sent: Wednesday, August 9, 2023 7:03 PM
To: users@cloudstack.apache.org
Subject: Re: set display_name as hostname to instances

Historically, the use of display name in case of instances and other
resources (such as offering, templates etc) was to allow the operator/admin
to the resources by name and end user to see by display name in upper UI/CMP
layers such as CPBM etc. There was even API/response keys such as displayvm
where the idea was a root admin (operator account) can deploy resources
without getting them listed in such upper UI/CMP layers.

I think it's completely redundant now in the current UI implementation
neither is actually followed or used (I think), or even needed. Upper layers
and CMP/billing portals can decide however they want to use these params.


Regards.


From: Pearl d'Silva 
Sent: Wednesday, August 9, 2023 18:06
To: users@cloudstack.apache.org 
Subject: Re: set display_name as hostname to instances

Display name is solely for UI purposes. In order to set the host name of a
VM to a desired value, you'll need to pass the name parameter(same as that
of the displayname). If the name parameter is not passed, the hostname would
default to VM-.

Regards,
Pearl
________
From: cristian.c@istream.today 
Sent: August 4, 2023 11:35 AM
To: users@cloudstack.apache.org 
Subject: RE: set display_name as hostname to instances

The metadata does not show the display name for local-hostname,  for both
parameters I see the name value of the VM, the deploy was made via API and I
provided only the display_name, where the display name is "displayname01".

As you see here, public and local have the same value, from vm name.

{"vm_ip_address":"51.89.84.82","vm_metadata":[["userdata","user-data",null],
["metadata","service-offering","S-2"],["metadata","availability-zone","DE-00
1"],["metadata","local-ipv4","51.89.xx.xx"],["metadata","local-hostname","VM
-c8c113e0-3277-425e-882b-44937413fd4a"],["metadata","public-ipv4","51.89.xx.
xx"],["metadata","public-hostname","VM-c8c113e0-3277-425e-882b-44937413fd4a"
],["metadata","instance-id","c8c113e0-3277-425e-882b-44937413fd4a"],["metada
ta","vm-id","c8c113e0-3277-425e-882b-44937413fd4a"]


Thank you,
Cristian




 


-Original Message-
From: Pearl d'Silva 
Sent: Friday, August 4, 2023 6:02 PM
To: users@cloudstack.apache.org
Subject: Re: set display_name as hostname to instances

If you are using the UI to create the VM, set the name of the VM - this
field sets the hostname and the display name. So you should see this value
in the local-hostname file in the VR metadata folder.
If you are using the API, explicitly set the name field.

Let us know if you see the desired name in the metadata after setting the
name parameter.

Thanks,

From: cristian.c@istream.today 
Sent: August 4, 2023 10:22 AM
To: users@cloudstack.apache.org 
Subject: RE: set display_name as hostname to instances

Hi Pearl,

   I will test this when I upgrade to 4.18.x I'm waiting for 4.18.1.

Thank you,
Cristian




-Original Message-
From: Pearl d'Silva 
Sent: Friday, August 4, 2023 4:24 PM
To: users@cloudstack.apache.org
Subject: Re: set display_name as hostname to instances

Hi Cristian,

In addition, in 4.18 a new feature - managed User data was introduced. This
can be used to register userdata with custom parameters - here, the
hostname. And when deploying the VM, the custom parameters could be set to
the desired values. You could refer to
https://www.shapeblue.com/cloudstack-managed-user-data/ for more
information.

Thanks,
Pearl

From: Rohit Yadav 
Sent: August 4, 2023 9:12 AM
To: users@cloudstack.apache.org 
Subject: Re: set display_name as hostname to instances

Hi Cristian,

I'm not sure, but could you try user data (cloud-init data) when you deploy
a VM. In there, you'll have flexibility to set the hostname of the VM to
whatever you'd like.


Regards.


From: cristian.c@istream.today 
Sent: Thursday, August 3, 2023 20:28
To: users@cloudstack.apache.org 
Subject: set display_name as hostname to instances

Hello,



Is there a method to set as hostnames to the instances the display name? I
did not see the display name in the metadata.



I am asking this because the display name allows any format, such as
fqdn.tld, and it does not matter if it's a duplicate or not.

Thank you,

Cristian

RE: set display_name as hostname to instances

[cristian.c]

The metadata does not show the display name for local-hostname,  for both
parameters I see the name value of the VM, the deploy was made via API and I
provided only the display_name, where the display name is "displayname01".

As you see here, public and local have the same value, from vm name.

{"vm_ip_address":"51.89.84.82","vm_metadata":[["userdata","user-data",null],
["metadata","service-offering","S-2"],["metadata","availability-zone","DE-00
1"],["metadata","local-ipv4","51.89.xx.xx"],["metadata","local-hostname","VM
-c8c113e0-3277-425e-882b-44937413fd4a"],["metadata","public-ipv4","51.89.xx.
xx"],["metadata","public-hostname","VM-c8c113e0-3277-425e-882b-44937413fd4a"
],["metadata","instance-id","c8c113e0-3277-425e-882b-44937413fd4a"],["metada
ta","vm-id","c8c113e0-3277-425e-882b-44937413fd4a"]


Thank you,
Cristian

-Original Message-
From: Pearl d'Silva  
Sent: Friday, August 4, 2023 6:02 PM
To: users@cloudstack.apache.org
Subject: Re: set display_name as hostname to instances

If you are using the UI to create the VM, set the name of the VM - this
field sets the hostname and the display name. So you should see this value
in the local-hostname file in the VR metadata folder.
If you are using the API, explicitly set the name field.

Let us know if you see the desired name in the metadata after setting the
name parameter.

Thanks,

From: cristian.c@istream.today 
Sent: August 4, 2023 10:22 AM
To: users@cloudstack.apache.org 
Subject: RE: set display_name as hostname to instances

Hi Pearl,

   I will test this when I upgrade to 4.18.x I'm waiting for 4.18.1.

Thank you,
Cristian

 


-Original Message-
From: Pearl d'Silva 
Sent: Friday, August 4, 2023 4:24 PM
To: users@cloudstack.apache.org
Subject: Re: set display_name as hostname to instances

Hi Cristian,

In addition, in 4.18 a new feature - managed User data was introduced. This
can be used to register userdata with custom parameters - here, the
hostname. And when deploying the VM, the custom parameters could be set to
the desired values. You could refer to
https://www.shapeblue.com/cloudstack-managed-user-data/ for more
information.

Thanks,
Pearl

From: Rohit Yadav 
Sent: August 4, 2023 9:12 AM
To: users@cloudstack.apache.org 
Subject: Re: set display_name as hostname to instances

Hi Cristian,

I'm not sure, but could you try user data (cloud-init data) when you deploy
a VM. In there, you'll have flexibility to set the hostname of the VM to
whatever you'd like.


Regards.


From: cristian.c@istream.today 
Sent: Thursday, August 3, 2023 20:28
To: users@cloudstack.apache.org 
Subject: set display_name as hostname to instances

Hello,



Is there a method to set as hostnames to the instances the display name? I
did not see the display name in the metadata.



I am asking this because the display name allows any format, such as
fqdn.tld, and it does not matter if it's a duplicate or not.

Thank you,

Cristian

RE: set display_name as hostname to instances

[cristian.c]

Hi Pearl,

   I will test this when I upgrade to 4.18.x I'm waiting for 4.18.1.

Thank you,
Cristian

-Original Message-
From: Pearl d'Silva  
Sent: Friday, August 4, 2023 4:24 PM
To: users@cloudstack.apache.org
Subject: Re: set display_name as hostname to instances

Hi Cristian,

In addition, in 4.18 a new feature - managed User data was introduced. This
can be used to register userdata with custom parameters - here, the
hostname. And when deploying the VM, the custom parameters could be set to
the desired values. You could refer to
https://www.shapeblue.com/cloudstack-managed-user-data/ for more
information.

Thanks,
Pearl

From: Rohit Yadav 
Sent: August 4, 2023 9:12 AM
To: users@cloudstack.apache.org 
Subject: Re: set display_name as hostname to instances

Hi Cristian,

I'm not sure, but could you try user data (cloud-init data) when you deploy
a VM. In there, you'll have flexibility to set the hostname of the VM to
whatever you'd like.


Regards.


From: cristian.c@istream.today 
Sent: Thursday, August 3, 2023 20:28
To: users@cloudstack.apache.org 
Subject: set display_name as hostname to instances

Hello,



Is there a method to set as hostnames to the instances the display name? I
did not see the display name in the metadata.



I am asking this because the display name allows any format, such as
fqdn.tld, and it does not matter if it's a duplicate or not.

Thank you,

Cristian





 

RE: set display_name as hostname to instances

[cristian.c]

Hi Rohit,

  I didn't saw the display name in the metadata, this is why I ask:


{"vm_ip_address":"51.89.xx.xx","vm_metadata":[["userdata","user-data",null],
["metadata","service-offering","S-2"],["metadata","availability-zone","DE-00
1"],["metadata","local-ipv4","51.89.xx.xx"],["metadata","local-hostname","VM
-c8c113e0-3277-425e-882b-44937413fd4a"],["metadata","public-ipv4","51.89.xx.
xx"],["metadata","public-hostname","VM-c8c113e0-3277-425e-882b-44937413fd4a"
],["metadata","instance-id","c8c113e0-3277-425e-882b-44937413fd4a"],["metada
ta","vm-id","c8c113e0-3277-425e-882b-44937413fd4a"]


  Or I'm not looking at the right path? This is from metadata file, located
on the routers / processed.

Thank you,
Cristian



-Original Message-
From: Rohit Yadav  
Sent: Friday, August 4, 2023 4:13 PM
To: users@cloudstack.apache.org
Subject: Re: set display_name as hostname to instances

Hi Cristian,

I'm not sure, but could you try user data (cloud-init data) when you deploy
a VM. In there, you'll have flexibility to set the hostname of the VM to
whatever you'd like.


Regards.


From: cristian.c@istream.today 
Sent: Thursday, August 3, 2023 20:28
To: users@cloudstack.apache.org 
Subject: set display_name as hostname to instances

Hello,



Is there a method to set as hostnames to the instances the display name? I
did not see the display name in the metadata.



I am asking this because the display name allows any format, such as
fqdn.tld, and it does not matter if it's a duplicate or not.

Thank you,

Cristian


 

set display_name as hostname to instances

[cristian.c]

Hello,

 

Is there a method to set as hostnames to the instances the display name? I
did not see the display name in the metadata.

 

I am asking this because the display name allows any format, such as
fqdn.tld, and it does not matter if it's a duplicate or not.

Thank you,

Cristian

Kubernetes service Plugin or CAPI

[cristian.c]

Hello,

 

  I saw that there are 2 ways to deploy Kubernetes cluster on Cloudstack. Is
the integrated version going to be improved/removed? I'm confused.

 

1.
https://docs.cloudstack.apache.org/en/latest/plugins/cloudstack-kubernetes-s
ervice.html
2.  https://cluster-api-cloudstack.sigs.k8s.io/introduction.html




Thank you,

Cristian

RE: cloud usage - Not owner of usage job, skipping

[cristian.c]

Thank you for this workaround.

-Original Message-
From: Nux  
Sent: Tuesday, March 14, 2023 6:13 PM
To: users@cloudstack.apache.org
Cc: cristian.c@istream.today
Subject: Re: cloud usage - Not owner of usage job, skipping

I think this is fixed by https://github.com/apache/cloudstack/pull/6825

Try to create that dir/file yourself until the fix makes it into Cloudstack.
mkdir -p /usr/local/libexec
echo 1 > /usr/local/libexec/sanity-check-last-id

On 2023-03-14 13:10, cristian.c@istream.today wrote:
> Today I saw a strange error in the usage log:
> 
> 
> 2023-03-14 12:30:23,317 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) processed Usage for 0 RECENTLY DELETED 
> accts
> 2023-03-14 12:30:23,337 INFO  [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) usage job complete
> 2023-03-14 12:30:23,337 INFO  [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) starting usage job...
> 2023-03-14 12:30:23,339 INFO  [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) not parsing usage records since start time 
> mills
> (167879520) is on or after end time millis (167879519)
> 2023-03-14 12:30:23,343 INFO  [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) usage job complete
> 2023-03-14 12:39:06,192 ERROR [cloud.usage.UsageSanityChecker]
> (Usage-Sanity-1:null) (logid:)
> readLastCheckId:Exception:/usr/local/libexec/sanity-check-last-id (No 
> such file or directory)
> java.io.FileNotFoundException: /usr/local/libexec/sanity-check-last-id
> (No
> such file or directory)
> at java.base/java.io.FileInputStream.open0(Native Method)
> at
> java.base/java.io.FileInputStream.open(FileInputStream.java:219)
> at
> java.base/java.io.FileInputStream.(FileInputStream.java:157)
> at
> java.base/java.io.FileInputStream.(FileInputStream.java:112)
> at java.base/java.io.FileReader.(FileReader.java:60)
> at
> com.cloud.usage.UsageSanityChecker.readLastCheckId(UsageSanityChecker.
> java:1
> 73)
> at
> com.cloud.usage.UsageSanityChecker.runSanityCheck(UsageSanityChecker.j
> ava:21
> 1)
> at
> com.cloud.usage.UsageManagerImpl$SanityCheck.runInContext(UsageManager
> Impl.j
> ava:2161)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man
> agedCo
> ntextRunnable.java:48)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal
> l(Defa
> ultManagedContext.java:55)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW
> ithCon
> text(DefaultManagedContext.java:102)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi
> thCont
> ext(DefaultManagedContext.java:52)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag
> edCont
> extRunnable.java:45)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> s.java
> :515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> tureTa
> sk.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Execut
> or.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecu
> tor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2023-03-14 12:39:06,205 ERROR [cloud.usage.UsageSanityChecker]
> (Usage-Sanity-1:null) (logid:)
> checkItemCountByPstmt:Exception:Parameter
> index out of range (2 > number of parameters, which is 1).
> 
> 
> Any idea why?
> 
> Regards,
> Cristian
> 
> -Original Message-
> From: cristian.c@istream.today 
> Sent: Monday, March 13, 2023 3:47 PM
> To: users@cloudstack.apache.org
> Subject: RE: cloud usage - Not owner of usage job, skipping
> 
> Hello,
> 
> Just did this, looks like I had some jobs with null, removed and 
> restarted the suage fixed the issue.
> 
> Thank you,
> Cristian
> 
> -Original Message-
> From: Nux 
> Sent: Monday, March 13, 2023 2:26 PM
> To: users@cloudstack.apache.org
> Subject: Re: cloud usage - Not owner of usage job, skipping
> 
> Hello,
> 
> It's possible that due to an "unclean" shutdown there is probably a 
> usage server running that is not having the same pid as the last 
> scheduled job's (the linux pid is used in the job schedule table).
> To correct this, shut down the usage server and delete the last job 
> (the one that hasn't run yet).
> 
> HTH
> 
> On 2023-03

RE: cloud usage - Not owner of usage job, skipping

[cristian.c]

Today I saw a strange error in the usage log:


2023-03-14 12:30:23,317 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) processed Usage for 0 RECENTLY DELETED accts
2023-03-14 12:30:23,337 INFO  [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) usage job complete
2023-03-14 12:30:23,337 INFO  [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) starting usage job...
2023-03-14 12:30:23,339 INFO  [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) not parsing usage records since start time mills
(167879520) is on or after end time
millis (167879519)
2023-03-14 12:30:23,343 INFO  [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) usage job complete
2023-03-14 12:39:06,192 ERROR [cloud.usage.UsageSanityChecker]
(Usage-Sanity-1:null) (logid:)
readLastCheckId:Exception:/usr/local/libexec/sanity-check-last-id (No such
file or directory)
java.io.FileNotFoundException: /usr/local/libexec/sanity-check-last-id (No
such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at
java.base/java.io.FileInputStream.(FileInputStream.java:157)
at
java.base/java.io.FileInputStream.(FileInputStream.java:112)
at java.base/java.io.FileReader.(FileReader.java:60)
at
com.cloud.usage.UsageSanityChecker.readLastCheckId(UsageSanityChecker.java:1
73)
at
com.cloud.usage.UsageSanityChecker.runSanityCheck(UsageSanityChecker.java:21
1)
at
com.cloud.usage.UsageManagerImpl$SanityCheck.runInContext(UsageManagerImpl.j
ava:2161)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:45)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java
:515)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTa
sk.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut
or.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu
tor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-03-14 12:39:06,205 ERROR [cloud.usage.UsageSanityChecker]
(Usage-Sanity-1:null) (logid:) checkItemCountByPstmt:Exception:Parameter
index out of range (2 > number of parameters, which is 1).


Any idea why?

Regards,
Cristian

-Original Message-
From: cristian.c@istream.today  
Sent: Monday, March 13, 2023 3:47 PM
To: users@cloudstack.apache.org
Subject: RE: cloud usage - Not owner of usage job, skipping

Hello,

Just did this, looks like I had some jobs with null, removed and
restarted the suage fixed the issue.

Thank you,
Cristian

-Original Message-
From: Nux 
Sent: Monday, March 13, 2023 2:26 PM
To: users@cloudstack.apache.org
Subject: Re: cloud usage - Not owner of usage job, skipping

Hello,

It's possible that due to an "unclean" shutdown there is probably a usage
server running that is not having the same pid as the last scheduled job's
(the linux pid is used in the job schedule table).
To correct this, shut down the usage server and delete the last job (the one
that hasn't run yet).

HTH

On 2023-03-13 11:50, cristian.c@istream.today wrote:
> Hello,
> 
> 
> 
> 
> 
>I just found today that my cloud usage stopped to collect data from 
> 2022, for sure is related to some upgrade, but it is quite impossible 
> to find out which one.  my question is, how can I fix this?  Can I 
> reset the cloudstack usage db? And start collection from today?
> 
> "Usage-HB-1:null) (logid:) it's been 17573256046 ms since last usage 
> job and
> 71243955 ms until next job, scheduling an immediate job to catch up 
> (aggregation duration is 1440 minutes)
> 
> 2023-03-13 05:27:36,045 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-HB-1:null) (logid:) Scheduling Usage job...
> 
> 2023-03-13 05:27:36,046 INFO  [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) starting usage job...
> 
> 2023-03-13 05:27:36,048 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) Not owner of usage job, skipping...
> 
> 2023-03-13 05:27:36,048 INFO  [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) usage job complete"
> 
> 
> Thank you,
> 
> Cristian

RE: cloud usage - Not owner of usage job, skipping

[cristian.c]

Hello,

Just did this, looks like I had some jobs with null, removed and
restarted the suage fixed the issue.

Thank you,
Cristian

-Original Message-
From: Nux  
Sent: Monday, March 13, 2023 2:26 PM
To: users@cloudstack.apache.org
Subject: Re: cloud usage - Not owner of usage job, skipping

Hello,

It's possible that due to an "unclean" shutdown there is probably a usage
server running that is not having the same pid as the last scheduled job's
(the linux pid is used in the job schedule table).
To correct this, shut down the usage server and delete the last job (the one
that hasn't run yet).

HTH

On 2023-03-13 11:50, cristian.c@istream.today wrote:
> Hello,
> 
> 
> 
> 
> 
>I just found today that my cloud usage stopped to collect data from 
> 2022, for sure is related to some upgrade, but it is quite impossible 
> to find out which one.  my question is, how can I fix this?  Can I 
> reset the cloudstack usage db? And start collection from today?
> 
> "Usage-HB-1:null) (logid:) it's been 17573256046 ms since last usage 
> job and
> 71243955 ms until next job, scheduling an immediate job to catch up 
> (aggregation duration is 1440 minutes)
> 
> 2023-03-13 05:27:36,045 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-HB-1:null) (logid:) Scheduling Usage job...
> 
> 2023-03-13 05:27:36,046 INFO  [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) starting usage job...
> 
> 2023-03-13 05:27:36,048 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) Not owner of usage job, skipping...
> 
> 2023-03-13 05:27:36,048 INFO  [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) usage job complete"
> 
> 
> Thank you,
> 
> Cristian

cloud usage - Not owner of usage job, skipping

[cristian.c]

Hello,

 

 

   I just found today that my cloud usage stopped to collect data from 2022,
for sure is related to some upgrade, but it is quite impossible to find out
which one.  my question is, how can I fix this?  Can I reset the cloudstack
usage db? And start collection from today?

"Usage-HB-1:null) (logid:) it's been 17573256046 ms since last usage job and
71243955 ms until next job, scheduling an immediate job to catch up
(aggregation duration is 1440 minutes)

2023-03-13 05:27:36,045 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-HB-1:null) (logid:) Scheduling Usage job...

2023-03-13 05:27:36,046 INFO  [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) starting usage job...

2023-03-13 05:27:36,048 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) Not owner of usage job, skipping...

2023-03-13 05:27:36,048 INFO  [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) usage job complete"


Thank you,

Cristian

New rootdisksize not set with Scale instance in 4.17.2

[cristian.c]

Hello,

 

Any idea why the new value for "rootdisksize" parameter does not have
any effect on the instance?

   I executed this via cmk (api):


scale virtualmachine id=a2b9aca8-4e67-4cb4-8526-e2507b8d3898
serviceofferingid=08f5f8fa-88d7-44c7-8ed5-7f06a6031888
details[0].cpuNumber=1 details[0].memory=1024 details[0].rootdisksize=25

   And the response is:



{

  "virtualmachine": {c API result

"account": "emeaclient_16_49",

"affinitygroup": [],

"cpunumber": 1,

"cpuspeed": 1600,

"created": "2023-03-07T10:21:04+",

"details": {

  "Message.ReservedCapacityFreed.Flag": "true",

  "cpuNumber": "1",

  "cpuOvercommitRatio": "2.0",

  "cpuSpeed": "1600",

  "dataDiskController": "osdefault",

  "keyboard": "us",

  "memory": "1024",

  "memoryOvercommitRatio": "2.0",

  "nicAdapter": "Vmxnet3",

  "rootDiskController": "scsi",

  "rootdisksize": "10"

},

"displayname": "Test-Scale-March-3",

"displayvm": true,

"domain": "cloud16",

"domainid": "793d17f0-30d3-46c1-adbf-657c4659a624",

"guestosid": "6428ffbd-1f58-11ec-b6e8-244bfeb8116f",

"haenable": false,

"hasannotations": false,

"hypervisor": "VMware",

"id": "a2b9aca8-4e67-4cb4-8526-e2507b8d3898",

"instancename": "i-74-1443-VM",

"isdynamicallyscalable": false,

"jobid": "701a1122-0185-43a1-b110-bfbee380f4dd",

"jobstatus": 0,

"lastupdated": "2023-03-07T10:30:43+",

"memory": 1024,

"name": "VM-a2b9aca8-4e67-4cb4-8526-e2507b8d3898",

"nic": [

  {

"broadcasturi": "vlan://untagged",

"deviceid": "0",

"extradhcpoption": [],

"gateway": "46.xxx.xxx.49",

"id": "4e23f2be-ecc5-4793-8cbd-c3bc15920cd1",

"ipaddress": "46.xxx.xxx.54",

"isdefault": true,

"isolationuri": "vlan://untagged",

"macaddress": "1e:00:cf:00:00:87",

"netmask": "255.255.255.240",

"networkid": "3f5fbbe4-fe38-42ae-af2d-01709d7a2e9c",

"networkname": "Public-NL-01",

"secondaryip": [],

"traffictype": "Guest",

"type": "Shared"

  }

],

"osdisplayname": "Other Linux (64-bit)",

"ostypeid": "6428ffbd-1f58-11ec-b6e8-244bfeb8116f",

"passwordenabled": true,

"pooltype": "VMFS",

"receivedbytes": 0,

"rootdeviceid": 0,

"rootdevicetype": "ROOT",

"securitygroup": [],

"sentbytes": 0,

"serviceofferingid": "08f5f8fa-88d7-44c7-8ed5-7f06a6031888",

"serviceofferingname": "S-Packages",

"state": "Stopped",

"tags": [

  {

"account": "emeaclient_16_49",

"domain": "cloud16",

"domainid": "793d17f0-30d3-46c1-adbf-657c4659a624",

"key": "service_type",

"resourceid": "a2b9aca8-4e67-4cb4-8526-e2507b8d3898",

"resourcetype": "UserVm",

"value": "S-2"

  },

  {

"account": "emeaclient_16_49",

"domain": "cloud16",

"domainid": "793d17f0-30d3-46c1-adbf-657c4659a624",

"key": "service_id",

"resourceid": "a2b9aca8-4e67-4cb4-8526-e2507b8d3898",

"resourcetype": "UserVm",

"value": "450"

  }

],

"templatedisplaytext": "RockyLinux-8",

"templateid": "be405fbf-9ca9-4b90-bf97-9806ebf75b7b",

"templatename": "RockyLinux-8",

"userid": "df0acf1e-52a2-4c4f-b4e6-535546cf6b12",

"username": "emeaclient_16_49",

"zoneid": "55f0123b-285c-4a70-8b34-565d8454393d",

"zonename": "EMEA-NL-01"

  }

}

  
I have also changed the value to "true" for
allow.diskoffering.change.during.scale.vm. The only difference is that I do
not see any more the warning in the log "Changing the disk offering of the
root volume during the compute offering change operation is disabled. Please
check the setting [allow.diskoffering.change.during.scale.vm]"

Any suggestion?

Thank you,

Cristian
   

 

RE: cloud usage Unable to load db configuration

[cristian.c]

Yes, after reinstall the cloudstack-usage.

-Original Message-
From: Suresh Kumar Anaparti  
Sent: Friday, August 12, 2022 10:55 AM
To: users@cloudstack.apache.org
Subject: Re: cloud usage Unable to load db configuration

Ok, issue resolved after reinstallation?

On Fri, Aug 12, 2022 at 12:40 PM  wrote:
>
> Looks like there was a issue with db.properties, after I reinstalled, the 
> symlink was created, before was without symlink.
>
> lrwxrwxrwx 1 root root   40 Aug 11 18:56 db.properties -> 
> /etc/cloudstack/management/db.properties
> lrwxrwxrwx 1 root root   30 Aug 11 18:56 key -> /etc/cloudstack/management/key
>
>
> Regards,
> Cristian
>
>
> -Original Message-
> From: Suresh Kumar Anaparti 
> Sent: Friday, August 12, 2022 9:37 AM
> To: users@cloudstack.apache.org
> Subject: Re: cloud usage Unable to load db configuration
>
> Ok, Please check the parameter "db.cloud.maxActive" in 
> /etc/cloudstack/usage/db.properties as well.
>
> parameter parsing issue here:
> https://github.com/apache/cloudstack/blob/cad9332082a1f85eedc30cf547ae
> 28224be170c2/framework/db/src/main/java/com/cloud/utils/db/Transaction
> Legacy.java#L1030
>
> On Fri, Aug 12, 2022 at 12:17 AM  wrote:
> >
> > Hi Suresh,
> >
> >I only have these;
> >
> > root@acs-apac:/home/ubuntu# find / -name "db.properties"
> > /etc/cloudstack/usage/db.properties
> > /etc/cloudstack/management/db.properties
> > root@acs-apac:/home/ubuntu#
> >
> >
> > Regards,
> > Cristian
> >
> > -Original Message-
> > From: Suresh Kumar Anaparti 
> > Sent: Thursday, August 11, 2022 7:19 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: cloud usage Unable to load db configuration
> >
> > Hi Cristian,
> >
> > The listed db configuration parameters looks good.
> >
> > I think there is another db.properties file from where the configuration is 
> > loaded. Can you find whether any other db.properties file exists, and check 
> > if the parameter "db.cloud.maxActive" is missing or has invalid value. 
> > Please update / correct it to "db.cloud.maxActive=250". You can ignore the 
> > simulator db settings (simulator is used for testing purposes).
> >
> > Regards,
> > Suresh
> >
> > On Thu, Aug 11, 2022 at 4:55 PM  wrote:
> > >
> > > HI Suresh,
> > >
> > >
> > >   This is what I have in from /etc/management/db.properties
> > >
> > > # usage database tuning parameters
> > >
> > > db.cloud.port=3306
> > >
> > > db.cloud.maxActive=250
> > > db.cloud.maxIdle=30
> > > db.cloud.maxWait=1
> > >
> > >
> > > db.usage.port=3306
> > >
> > > db.usage.maxActive=100
> > > db.usage.maxIdle=30
> > > db.usage.maxWait=1
> > > db.usage.url.params=serverTimezone=UTC
> > >
> > >
> > > # Simulator database settings
> > > db.simulator.username=cloud
> > > db.simulator.password=cloud  ( I'm not aware about what is simulator for 
> > > ) here the pass is "cloud"
> > > db.simulator.host=localhost
> > > db.simulator.driver=jdbc:mysql
> > > db.simulator.port=3306
> > > db.simulator.name=simulator
> > > db.simulator.maxActive=250
> > > db.simulator.maxIdle=30
> > > db.simulator.maxWait=1
> > > db.simulator.autoReconnect=true
> > >
> > >
> > > Regards,
> > > Cristian
> > >
> > >
> > >
> > > -Original Message-
> > > From: Suresh Kumar Anaparti 
> > > Sent: Thursday, August 11, 2022 1:41 PM
> > > To: users@cloudstack.apache.org
> > > Subject: Re: cloud usage Unable to load db configuration
> > >
> > > Hi Cristian,
> > >
> > > It seems to be issue with parameter parsing in the db.properties file, 
> > > not the password. Please check the value of the parameter 
> > > "db.cloud.maxActive" in db.properties file, it should be a valid integer.
> > >
> > > Also, make sure the below parameters in db.properties file have valid 
> > > integer values.
> > >
> > > db.cloud.port
> > > db.cloud.maxActive
> > > db.cloud.maxIdle
> > > db.cloud.maxWait
> > >
> > > db.usage.port
> > > db.usage.maxActive
> > > db.usage.maxIdle
> > > db.usage.maxWait
> > >
> > > db.simulator.port
> > > db.simulator.maxActive
> > > db.simulator.maxIdle
> > > db.simulator.maxWait
> > >
> > > Regards,
> > > Suresh
> > >
> > > On Thu, Aug 11, 2022 at 3:27 PM  wrote:
> > > >
> > > > Hello,
> > > >
> > > >
> > > >
> > > >Any idea why I see the bellow error?  I have the right 
> > > > password in /management/db.properties and in 
> > > > /usage/db.properties
> > > > (encrypted)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 2022-08-11 09:52:40,160 INFO  [utils.component.ComponentContext]
> > > > (main:null)
> > > > (logid:) Configuring
> > > > com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_ed194661
> > > >
> > > > 2022-08-11 09:52:40,160 INFO  [utils.component.ComponentContext]
> > > > (main:null)
> > > > (logid:) Configuring
> > > > com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_c8031ed8
> > > >
> > > > 2022-08-11 09:52:40,160 DEBUG [cloud.usage.UsageManagerImpl]
> > > > (main:null)
> > > > (logid:) Checking to see if usage.vmops.pid exists.
> > > >
> > > > 2022-08-11 

RE: cloud usage Unable to load db configuration

[cristian.c]

Looks like there was a issue with db.properties, after I reinstalled, the 
symlink was created, before was without symlink.

lrwxrwxrwx 1 root root   40 Aug 11 18:56 db.properties -> 
/etc/cloudstack/management/db.properties
lrwxrwxrwx 1 root root   30 Aug 11 18:56 key -> /etc/cloudstack/management/key


Regards,
Cristian


-Original Message-
From: Suresh Kumar Anaparti  
Sent: Friday, August 12, 2022 9:37 AM
To: users@cloudstack.apache.org
Subject: Re: cloud usage Unable to load db configuration

Ok, Please check the parameter "db.cloud.maxActive" in 
/etc/cloudstack/usage/db.properties as well.

parameter parsing issue here:
https://github.com/apache/cloudstack/blob/cad9332082a1f85eedc30cf547ae28224be170c2/framework/db/src/main/java/com/cloud/utils/db/TransactionLegacy.java#L1030

On Fri, Aug 12, 2022 at 12:17 AM  wrote:
>
> Hi Suresh,
>
>I only have these;
>
> root@acs-apac:/home/ubuntu# find / -name "db.properties"
> /etc/cloudstack/usage/db.properties
> /etc/cloudstack/management/db.properties
> root@acs-apac:/home/ubuntu#
>
>
> Regards,
> Cristian
>
> -Original Message-
> From: Suresh Kumar Anaparti 
> Sent: Thursday, August 11, 2022 7:19 PM
> To: users@cloudstack.apache.org
> Subject: Re: cloud usage Unable to load db configuration
>
> Hi Cristian,
>
> The listed db configuration parameters looks good.
>
> I think there is another db.properties file from where the configuration is 
> loaded. Can you find whether any other db.properties file exists, and check 
> if the parameter "db.cloud.maxActive" is missing or has invalid value. Please 
> update / correct it to "db.cloud.maxActive=250". You can ignore the simulator 
> db settings (simulator is used for testing purposes).
>
> Regards,
> Suresh
>
> On Thu, Aug 11, 2022 at 4:55 PM  wrote:
> >
> > HI Suresh,
> >
> >
> >   This is what I have in from /etc/management/db.properties
> >
> > # usage database tuning parameters
> >
> > db.cloud.port=3306
> >
> > db.cloud.maxActive=250
> > db.cloud.maxIdle=30
> > db.cloud.maxWait=1
> >
> >
> > db.usage.port=3306
> >
> > db.usage.maxActive=100
> > db.usage.maxIdle=30
> > db.usage.maxWait=1
> > db.usage.url.params=serverTimezone=UTC
> >
> >
> > # Simulator database settings
> > db.simulator.username=cloud
> > db.simulator.password=cloud  ( I'm not aware about what is simulator for ) 
> > here the pass is "cloud"
> > db.simulator.host=localhost
> > db.simulator.driver=jdbc:mysql
> > db.simulator.port=3306
> > db.simulator.name=simulator
> > db.simulator.maxActive=250
> > db.simulator.maxIdle=30
> > db.simulator.maxWait=1
> > db.simulator.autoReconnect=true
> >
> >
> > Regards,
> > Cristian
> >
> >
> >
> > -Original Message-
> > From: Suresh Kumar Anaparti 
> > Sent: Thursday, August 11, 2022 1:41 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: cloud usage Unable to load db configuration
> >
> > Hi Cristian,
> >
> > It seems to be issue with parameter parsing in the db.properties file, not 
> > the password. Please check the value of the parameter "db.cloud.maxActive" 
> > in db.properties file, it should be a valid integer.
> >
> > Also, make sure the below parameters in db.properties file have valid 
> > integer values.
> >
> > db.cloud.port
> > db.cloud.maxActive
> > db.cloud.maxIdle
> > db.cloud.maxWait
> >
> > db.usage.port
> > db.usage.maxActive
> > db.usage.maxIdle
> > db.usage.maxWait
> >
> > db.simulator.port
> > db.simulator.maxActive
> > db.simulator.maxIdle
> > db.simulator.maxWait
> >
> > Regards,
> > Suresh
> >
> > On Thu, Aug 11, 2022 at 3:27 PM  wrote:
> > >
> > > Hello,
> > >
> > >
> > >
> > >Any idea why I see the bellow error?  I have the right password 
> > > in /management/db.properties and in /usage/db.properties 
> > > (encrypted)
> > >
> > >
> > >
> > >
> > >
> > > 2022-08-11 09:52:40,160 INFO  [utils.component.ComponentContext]
> > > (main:null)
> > > (logid:) Configuring
> > > com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_ed194661
> > >
> > > 2022-08-11 09:52:40,160 INFO  [utils.component.ComponentContext]
> > > (main:null)
> > > (logid:) Configuring
> > > com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_c8031ed8
> > >
> > > 2022-08-11 09:52:40,160 DEBUG [cloud.usage.UsageManagerImpl]
> > > (main:null)
> > > (logid:) Checking to see if usage.vmops.pid exists.
> > >
> > > 2022-08-11 09:52:40,160 INFO  [cloud.usage.UsageManagerImpl]
> > > (main:null)
> > > (logid:) Implementation Version is 4.16.1.0
> > >
> > > 2022-08-11 09:52:51,433 DEBUG
> > > [utils.crypt.EncryptionSecretKeyChecker]
> > > (main:null) (logid:) Encryption Type: null
> > >
> > > 2022-08-11 09:52:51,435 INFO  [db.Transaction.Transaction]
> > > (main:null)
> > > (logid:) Is Data Base High Availiability enabled? Ans : false
> > >
> > > 2022-08-11 09:52:51,449 WARN  [db.Transaction.Transaction]
> > > (main:null)
> > > (logid:) Unable to load db configuration, using defaults with 5 
> > > connections.
> > > Falling back on assumed datasource on localhost:3306 

RE: cloud usage Unable to load db configuration

[cristian.c]

Hi Suresh,

   I only have these; 

root@acs-apac:/home/ubuntu# find / -name "db.properties"
/etc/cloudstack/usage/db.properties
/etc/cloudstack/management/db.properties
root@acs-apac:/home/ubuntu#


Regards,
Cristian

-Original Message-
From: Suresh Kumar Anaparti  
Sent: Thursday, August 11, 2022 7:19 PM
To: users@cloudstack.apache.org
Subject: Re: cloud usage Unable to load db configuration

Hi Cristian,

The listed db configuration parameters looks good.

I think there is another db.properties file from where the configuration is 
loaded. Can you find whether any other db.properties file exists, and check if 
the parameter "db.cloud.maxActive" is missing or has invalid value. Please 
update / correct it to "db.cloud.maxActive=250". You can ignore the simulator 
db settings (simulator is used for testing purposes).

Regards,
Suresh

On Thu, Aug 11, 2022 at 4:55 PM  wrote:
>
> HI Suresh,
>
>
>   This is what I have in from /etc/management/db.properties
>
> # usage database tuning parameters
>
> db.cloud.port=3306
>
> db.cloud.maxActive=250
> db.cloud.maxIdle=30
> db.cloud.maxWait=1
>
>
> db.usage.port=3306
>
> db.usage.maxActive=100
> db.usage.maxIdle=30
> db.usage.maxWait=1
> db.usage.url.params=serverTimezone=UTC
>
>
> # Simulator database settings
> db.simulator.username=cloud
> db.simulator.password=cloud  ( I'm not aware about what is simulator for ) 
> here the pass is "cloud"
> db.simulator.host=localhost
> db.simulator.driver=jdbc:mysql
> db.simulator.port=3306
> db.simulator.name=simulator
> db.simulator.maxActive=250
> db.simulator.maxIdle=30
> db.simulator.maxWait=1
> db.simulator.autoReconnect=true
>
>
> Regards,
> Cristian
>
>
>
> -Original Message-
> From: Suresh Kumar Anaparti 
> Sent: Thursday, August 11, 2022 1:41 PM
> To: users@cloudstack.apache.org
> Subject: Re: cloud usage Unable to load db configuration
>
> Hi Cristian,
>
> It seems to be issue with parameter parsing in the db.properties file, not 
> the password. Please check the value of the parameter "db.cloud.maxActive" in 
> db.properties file, it should be a valid integer.
>
> Also, make sure the below parameters in db.properties file have valid integer 
> values.
>
> db.cloud.port
> db.cloud.maxActive
> db.cloud.maxIdle
> db.cloud.maxWait
>
> db.usage.port
> db.usage.maxActive
> db.usage.maxIdle
> db.usage.maxWait
>
> db.simulator.port
> db.simulator.maxActive
> db.simulator.maxIdle
> db.simulator.maxWait
>
> Regards,
> Suresh
>
> On Thu, Aug 11, 2022 at 3:27 PM  wrote:
> >
> > Hello,
> >
> >
> >
> >Any idea why I see the bellow error?  I have the right password 
> > in /management/db.properties and in /usage/db.properties (encrypted)
> >
> >
> >
> >
> >
> > 2022-08-11 09:52:40,160 INFO  [utils.component.ComponentContext]
> > (main:null)
> > (logid:) Configuring
> > com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_ed194661
> >
> > 2022-08-11 09:52:40,160 INFO  [utils.component.ComponentContext]
> > (main:null)
> > (logid:) Configuring
> > com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_c8031ed8
> >
> > 2022-08-11 09:52:40,160 DEBUG [cloud.usage.UsageManagerImpl]
> > (main:null)
> > (logid:) Checking to see if usage.vmops.pid exists.
> >
> > 2022-08-11 09:52:40,160 INFO  [cloud.usage.UsageManagerImpl]
> > (main:null)
> > (logid:) Implementation Version is 4.16.1.0
> >
> > 2022-08-11 09:52:51,433 DEBUG 
> > [utils.crypt.EncryptionSecretKeyChecker]
> > (main:null) (logid:) Encryption Type: null
> >
> > 2022-08-11 09:52:51,435 INFO  [db.Transaction.Transaction] 
> > (main:null)
> > (logid:) Is Data Base High Availiability enabled? Ans : false
> >
> > 2022-08-11 09:52:51,449 WARN  [db.Transaction.Transaction] 
> > (main:null)
> > (logid:) Unable to load db configuration, using defaults with 5 connections.
> > Falling back on assumed datasource on localhost:3306 using 
> > username:password=cloud:cloud. Please check your configuration
> >
> > java.lang.NumberFormatException: null
> >
> > at java.base/java.lang.Integer.parseInt(Integer.java:614)
> >
> > at java.base/java.lang.Integer.parseInt(Integer.java:770)
> >
> > at
> > com.cloud.utils.db.TransactionLegacy.initDataSource(TransactionLegacy.
> > java:1
> > 030)
> >
> > at
> > com.cloud.utils.db.TransactionLegacy.(TransactionLegacy.java
> > :1
> > 008)
> >
> > at
> > com.cloud.utils.db.TransactionContextBuilder.interceptStart(Transact
> > io
> > nConte
> > xtBuilder.java:49)
> >
> > at
> > com.cloud.utils.component.ComponentInstantiationPostProcessor$Interc
> > ep
> > torDis
> > patcher.intercept(ComponentInstantiationPostProcessor.java:116)
> >
> > at
> > com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudS
> > ta
> > ck_ff5
> > 9469d.createPartialSelectSql()
> >
> > at
> > com.cloud.utils.db.GenericDaoBase.(GenericDaoBase.java:239)
> >
> > at
> > com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl.(UsageVMSn
> > ap
> > shotOn
> > 

RE: cloud usage Unable to load db configuration

[cristian.c]

HI Suresh,


  This is what I have in from /etc/management/db.properties

# usage database tuning parameters

db.cloud.port=3306

db.cloud.maxActive=250
db.cloud.maxIdle=30
db.cloud.maxWait=1


db.usage.port=3306

db.usage.maxActive=100
db.usage.maxIdle=30
db.usage.maxWait=1
db.usage.url.params=serverTimezone=UTC


# Simulator database settings
db.simulator.username=cloud  
db.simulator.password=cloud  ( I'm not aware about what is simulator for ) here 
the pass is "cloud"
db.simulator.host=localhost
db.simulator.driver=jdbc:mysql
db.simulator.port=3306
db.simulator.name=simulator
db.simulator.maxActive=250
db.simulator.maxIdle=30
db.simulator.maxWait=1
db.simulator.autoReconnect=true


Regards,
Cristian



-Original Message-
From: Suresh Kumar Anaparti  
Sent: Thursday, August 11, 2022 1:41 PM
To: users@cloudstack.apache.org
Subject: Re: cloud usage Unable to load db configuration

Hi Cristian,

It seems to be issue with parameter parsing in the db.properties file, not the 
password. Please check the value of the parameter "db.cloud.maxActive" in 
db.properties file, it should be a valid integer.

Also, make sure the below parameters in db.properties file have valid integer 
values.

db.cloud.port
db.cloud.maxActive
db.cloud.maxIdle
db.cloud.maxWait

db.usage.port
db.usage.maxActive
db.usage.maxIdle
db.usage.maxWait

db.simulator.port
db.simulator.maxActive
db.simulator.maxIdle
db.simulator.maxWait

Regards,
Suresh

On Thu, Aug 11, 2022 at 3:27 PM  wrote:
>
> Hello,
>
>
>
>Any idea why I see the bellow error?  I have the right password in 
> /management/db.properties and in /usage/db.properties (encrypted)
>
>
>
>
>
> 2022-08-11 09:52:40,160 INFO  [utils.component.ComponentContext] 
> (main:null)
> (logid:) Configuring
> com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_ed194661
>
> 2022-08-11 09:52:40,160 INFO  [utils.component.ComponentContext] 
> (main:null)
> (logid:) Configuring
> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_c8031ed8
>
> 2022-08-11 09:52:40,160 DEBUG [cloud.usage.UsageManagerImpl] 
> (main:null)
> (logid:) Checking to see if usage.vmops.pid exists.
>
> 2022-08-11 09:52:40,160 INFO  [cloud.usage.UsageManagerImpl] 
> (main:null)
> (logid:) Implementation Version is 4.16.1.0
>
> 2022-08-11 09:52:51,433 DEBUG [utils.crypt.EncryptionSecretKeyChecker]
> (main:null) (logid:) Encryption Type: null
>
> 2022-08-11 09:52:51,435 INFO  [db.Transaction.Transaction] (main:null)
> (logid:) Is Data Base High Availiability enabled? Ans : false
>
> 2022-08-11 09:52:51,449 WARN  [db.Transaction.Transaction] (main:null)
> (logid:) Unable to load db configuration, using defaults with 5 connections.
> Falling back on assumed datasource on localhost:3306 using 
> username:password=cloud:cloud. Please check your configuration
>
> java.lang.NumberFormatException: null
>
> at java.base/java.lang.Integer.parseInt(Integer.java:614)
>
> at java.base/java.lang.Integer.parseInt(Integer.java:770)
>
> at
> com.cloud.utils.db.TransactionLegacy.initDataSource(TransactionLegacy.
> java:1
> 030)
>
> at
> com.cloud.utils.db.TransactionLegacy.(TransactionLegacy.java:1
> 008)
>
> at
> com.cloud.utils.db.TransactionContextBuilder.interceptStart(Transactio
> nConte
> xtBuilder.java:49)
>
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercep
> torDis
> patcher.intercept(ComponentInstantiationPostProcessor.java:116)
>
> at
> com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudSta
> ck_ff5
> 9469d.createPartialSelectSql()
>
> at 
> com.cloud.utils.db.GenericDaoBase.(GenericDaoBase.java:239)
>
> at
> com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl.(UsageVMSnap
> shotOn
> PrimaryDaoImpl.java:38)
>
> at
> com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudSta
> ck_ff5
> 9469d.()
>
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstan
> ce0(Na
> tive Method)
>
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstan
> ce(Nat
> iveConstructorAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newIn
> stance
> (DelegatingConstructorAccessorImpl.java:45)
>
> at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:4
> 90)
>
> at 
> net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:291)
>
> at
> net.sf.cglib.proxy.Enhancer$EnhancerFactoryData.newInstance(Enhancer.j
> ava:44
> 8)
>
> at net.sf.cglib.proxy.Enhancer.nextInstance(Enhancer.java:700)
>
> at
> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:
> 298)
>
> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:480)
>
> at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:305)
>
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor.postProc
> essBef
> 

cloud usage Unable to load db configuration

[cristian.c]

Hello,

 

   Any idea why I see the bellow error?  I have the right password in
/management/db.properties and in /usage/db.properties (encrypted) 

 

 

2022-08-11 09:52:40,160 INFO  [utils.component.ComponentContext] (main:null)
(logid:) Configuring
com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_ed194661

2022-08-11 09:52:40,160 INFO  [utils.component.ComponentContext] (main:null)
(logid:) Configuring
com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_c8031ed8

2022-08-11 09:52:40,160 DEBUG [cloud.usage.UsageManagerImpl] (main:null)
(logid:) Checking to see if usage.vmops.pid exists.

2022-08-11 09:52:40,160 INFO  [cloud.usage.UsageManagerImpl] (main:null)
(logid:) Implementation Version is 4.16.1.0

2022-08-11 09:52:51,433 DEBUG [utils.crypt.EncryptionSecretKeyChecker]
(main:null) (logid:) Encryption Type: null

2022-08-11 09:52:51,435 INFO  [db.Transaction.Transaction] (main:null)
(logid:) Is Data Base High Availiability enabled? Ans : false

2022-08-11 09:52:51,449 WARN  [db.Transaction.Transaction] (main:null)
(logid:) Unable to load db configuration, using defaults with 5 connections.
Falling back on assumed datasource on localhost:3306 using
username:password=cloud:cloud. Please check your configuration

java.lang.NumberFormatException: null

at java.base/java.lang.Integer.parseInt(Integer.java:614)

at java.base/java.lang.Integer.parseInt(Integer.java:770)

at
com.cloud.utils.db.TransactionLegacy.initDataSource(TransactionLegacy.java:1
030)

at
com.cloud.utils.db.TransactionLegacy.(TransactionLegacy.java:1008)

at
com.cloud.utils.db.TransactionContextBuilder.interceptStart(TransactionConte
xtBuilder.java:49)

at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDis
patcher.intercept(ComponentInstantiationPostProcessor.java:116)

at
com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_ff5
9469d.createPartialSelectSql()

at com.cloud.utils.db.GenericDaoBase.(GenericDaoBase.java:239)

at
com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl.(UsageVMSnapshotOn
PrimaryDaoImpl.java:38)

at
com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_ff5
9469d.()

at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Na
tive Method)

at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Nat
iveConstructorAccessorImpl.java:62)

at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance
(DelegatingConstructorAccessorImpl.java:45)

at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)

at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:291)

at
net.sf.cglib.proxy.Enhancer$EnhancerFactoryData.newInstance(Enhancer.java:44
8)

at net.sf.cglib.proxy.Enhancer.nextInstance(Enhancer.java:700)

at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:
298)

at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:480)

at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:305)

at
com.cloud.utils.component.ComponentInstantiationPostProcessor.postProcessBef
oreInstantiation(ComponentInstantiationPostProcessor.java:92)

at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory
.applyBeanPostProcessorsBeforeInstantiation(AbstractAutowireCapableBeanFacto
ry.java:1144)

at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory
.resolveBeforeInstantiation(AbstractAutowireCapableBeanFactory.java:1119)

at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory
.createBean(AbstractAutowireCapableBeanFactory.java:520)

at
org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBe
an$0(AbstractBeanFactory.java:335)

at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSi
ngleton(DefaultSingletonBeanRegistry.java:234)

at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(Abst
ractBeanFactory.java:333)

at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abstra
ctBeanFactory.java:208)

at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInst
antiateSingletons(DefaultListableBeanFactory.java:944)

at
org.springframework.context.support.AbstractApplicationContext.finishBeanFac
toryInitialization(AbstractApplicationContext.java:923)

at
org.springframework.context.support.AbstractApplicationContext.refresh(Abstr
actApplicationContext.java:588)

at
org.springframework.context.support.ClassPathXmlApplicationContext.(Cl
assPathXmlApplicationContext.java:144)

at
org.springframework.context.support.ClassPathXmlApplicationContext.(Cl
assPathXmlApplicationContext.java:85)

at 

host time out after cloudstack management restore

[cristian.c]

Hello,

 

  I had a server fail today, because of that I restored the Cloudstack
management on a fresh install and used the last db backup I had.  Everything
looks fine in the UI, but not working, not sure why I see time out for all
hosts, this is not true, I can reach all the hosts via ip or hostname from
cloudstack management. ( all the hosts are connected in vCenter)

 

I see these events in the logs for all hosts.

 

2022-08-08 15:53:16,533 WARN  [c.c.v.UserVmManagerImpl]
(StatsCollector-2:ctx-faa5a90c) (logid:0c0862da) Unable to obtain VM
statistics.

2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache]
(StatsCollector-2:ctx-faa5a90c) (logid:0c0862da) Seq 42-8639592935156875294:
Forwarding null to 39908814623087

2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: Routing
from 2130706433

2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: Link is
closed

2022-08-08 15:53:16,548 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-8:null) (logid:) Seq 42-8639592935156875294: MgmtId
2130706433: Req: Resource [Host:42] is unreachable: Host 42: Link is closed

 

 

 

2022-08-08 15:54:13,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-3:null) (logid:) Seq 37--1: MgmtId 2130706433: Req:
Routing to peer

2022-08-08 15:54:13,213 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-4:null) (logid:) Seq 37--1: MgmtId 2130706433: Req:
Cancel request received

2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache]
(AgentManager-Handler-4:null) (logid:) Seq 37-3913346601208250383:
Cancelling.

2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache]
(StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq 37-3913346601208250383:
Waiting some more time because this is the current command

2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache]
(StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq 37-3913346601208250383:
Waiting some more time because this is the current command

2022-08-08 15:54:13,213 INFO  [c.c.u.e.CSExceptionErrorCode]
(StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for
exceptions

2022-08-08 15:54:13,213 WARN  [c.c.a.m.AgentAttache]
(StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq 37-3913346601208250383:
Timed out on null

2022-08-08 15:54:13,213 DEBUG [c.c.a.m.AgentAttache]
(StatsCollector-6:ctx-353c6d50) (logid:7f9ce0ca) Seq 37-3913346601208250383:
Cancelling.

 

 

 

   The only difference right now, is the OS, I restored on rocky Linux 8,
previously I had ubuntu 20.04.

 

  The cloudstack version is 4.16.1.0

Regards,

Cristian

No input filed for IP address in UI when adding new Zone

[cristian.c]

Hello,

 

   Did anyone added a new zone via UI in cloudstack 4.17 with success?  I
was not able to finish the setup, no option to continue /add ip address. 

 

   I created a bug report here :
https://github.com/apache/cloudstack/issues/6479

 

Regards,

Cristian

 

   

RE: Migration fail - Unable to find the target datastore

[cristian.c]

Hello,

   Yes, everything else works, I have many VMs. This error is persistent in
all my 3 cloudstack setups.

This is the ID da097676-7fa7-4b9c-b962-a2fe467fe565 in cloudstack with
this cloud.uuid : da097676-7fa7-4b9c-b962-a2fe467fe565 in VMware.



Regards,
Cristian

-Original Message-
From: Abhishek Kumar  
Sent: Thursday, March 31, 2022 8:43 AM
To: users@cloudstack.apache.org
Subject: Re: Migration fail - Unable to find the target datastore

Hi Cristian,

Are you able to deploy a new VM on this particular storage pool - UUID:
da097676-7fa7-4b9c-b962-a2fe467fe565?

Also, in your vCenter can you please check if target host (to which VM is
being migrated) does have a datastore attached with name -
da0976767fa74b9cb962a2fe467fe565.
Essentially, here CloudStack VMware plugin is failing to find the
corresponding datastore for the storage pool that you selected for
migration.

Regards,
Abhishek

From: cristian.c@istream.today 
Sent: 30 March 2022 21:19
To: 'Nux' ; users@cloudstack.apache.org

Subject: RE: Migration fail - Unable to find the target datastore

Hi,

  No, this is a different issue " Unable to find the target datastore".



Regards,
Cristian


---Original Message-
From: Nux 
Sent: Wednesday, March 30, 2022 5:08 PM
To: users@cloudstack.apache.org
Cc: cristian.c@istream.today
Subject: Re: Migration fail - Unable to find the target datastore

Hi,

Isn't it this all over again?
https://github.com/apache/cloudstack/issues/4187

---
Nux!
www.nux.ro

On 2022-03-29 20:08, cristian.c@istream.today wrote:
> I forgot to mention that I was trying first to migrate the volume and 
> start the VM, but I get error on trying to migrate the volume ( Vmware 
> local storge ) Unable to access the virtual machine configuration:
> Unable to access file [ro002] i-2-351-VM/i-2-351-VM.vmx,  then I was 
> trying with "Migrate instance to another primary storage" which gives 
> the bellow error..
>
>
>
> From: cristian.c@istream.today 
> Sent: Tuesday, March 29, 2022 9:21 PM
> To: users@cloudstack.apache.org
> Subject: Migration fail - Unable to find the target datastore
>
>
>
> Hello,
>
>
>
>Any idea why the migration fail with this error?  I have the same 
> error starting with CloudStack 4.16  ( also had some erros with 4.15, 
> worked before with older ACS ) I just did an update today to 4.16.1 
> and I see the same error..
>
>
>
>
>
> Error :
>
> 022-03-29 18:11:04,052 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-13:null) (logid:) SeqA 3-1551701: Sending Seq
> 3-1551701:  { Ans: , MgmtId: 39908814623087, via: 3, Ver: v1, Flags:
> 100010,
>
[{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypa
> ssHostMaintenance":"false"}}] }
>
> 2022-03-29 18:11:04,067 ERROR [c.c.h.v.r.VmwareResource] 
> (DirectAgent-256:ctx-43438d6c spb-ro-002.domain.host, 
> job-2647/job-2648,
> cmd: MigrateVmToPoolCommand) (logid:e122e3d6) Unable to find the 
> target
> datastore: da097676-7fa7-4b9c-b962-a2fe467fe565 in datacenter: RO-01 
> to execute migration
>
> 2022-03-29 18:11:04,118 ERROR [c.c.h.v.r.VmwareResource] 
> (DirectAgent-256:ctx-43438d6c spb-ro-002.domain.host, 
> job-2647/job-2648,
> cmd: MigrateVmToPoolCommand) (logid:e122e3d6) Change data store for VM 
> i-2-351-VM failed: Unable to find the target datastore:
> da097676-7fa7-4b9c-b962-a2fe467fe565 in datacenter: EU-ro-01 to 
> execute migration
>
> 2022-03-29 18:11:04,120 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-256:ctx-43438d6c) (logid:e122e3d6) Seq
> 25-1196268651020288049:
> Response Received:
>
> 2022-03-29 18:11:04,121 DEBUG [c.c.a.t.Request]
> (DirectAgent-256:ctx-43438d6c) (logid:e122e3d6) Seq
> 25-1196268651020288049:
> Processing:  { Ans: , MgmtId: 39908814623087, via:
> 25(spb-ro-002.domain.host), Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":"false","details":"Exception:
> com.cloud.utils.exception.CloudRuntimeException
>
> Message: Change data store for VM i-2-351-VM failed
>
> Stack: com.cloud.utils.exception.CloudRuntimeException: Change data 
> store for VM i-2-351-VM failed
>
> at
>
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareR
> esource.java:4550)
>
> at
>
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.j
> ava:4507)
>
> at
>
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareRes
> ource.java:501)
>
> at
>
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentA

RE: Migration fail - Unable to find the target datastore

[cristian.c]

Hi,

  No, this is a different issue " Unable to find the target datastore".  



Regards,
Cristian


---Original Message-
From: Nux  
Sent: Wednesday, March 30, 2022 5:08 PM
To: users@cloudstack.apache.org
Cc: cristian.c@istream.today
Subject: Re: Migration fail - Unable to find the target datastore

Hi,

Isn't it this all over again?
https://github.com/apache/cloudstack/issues/4187

---
Nux!
www.nux.ro

On 2022-03-29 20:08, cristian.c@istream.today wrote:
> I forgot to mention that I was trying first to migrate the volume and 
> start the VM, but I get error on trying to migrate the volume ( Vmware 
> local storge ) Unable to access the virtual machine configuration: 
> Unable to access file [ro002] i-2-351-VM/i-2-351-VM.vmx,  then I was 
> trying with "Migrate instance to another primary storage" which gives 
> the bellow error..
> 
> 
> 
> From: cristian.c@istream.today 
> Sent: Tuesday, March 29, 2022 9:21 PM
> To: users@cloudstack.apache.org
> Subject: Migration fail - Unable to find the target datastore
> 
> 
> 
> Hello,
> 
> 
> 
>Any idea why the migration fail with this error?  I have the same 
> error starting with CloudStack 4.16  ( also had some erros with 4.15, 
> worked before with older ACS ) I just did an update today to 4.16.1 
> and I see the same error..
> 
> 
> 
> 
> 
> Error :
> 
> 022-03-29 18:11:04,052 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-13:null) (logid:) SeqA 3-1551701: Sending Seq
> 3-1551701:  { Ans: , MgmtId: 39908814623087, via: 3, Ver: v1, Flags: 
> 100010,
>
[{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypa
> ssHostMaintenance":"false"}}] }
> 
> 2022-03-29 18:11:04,067 ERROR [c.c.h.v.r.VmwareResource]
> (DirectAgent-256:ctx-43438d6c spb-ro-002.domain.host, 
> job-2647/job-2648,
> cmd: MigrateVmToPoolCommand) (logid:e122e3d6) Unable to find the target
> datastore: da097676-7fa7-4b9c-b962-a2fe467fe565 in datacenter: RO-01 to
> execute migration
> 
> 2022-03-29 18:11:04,118 ERROR [c.c.h.v.r.VmwareResource]
> (DirectAgent-256:ctx-43438d6c spb-ro-002.domain.host, 
> job-2647/job-2648,
> cmd: MigrateVmToPoolCommand) (logid:e122e3d6) Change data store for VM
> i-2-351-VM failed: Unable to find the target datastore:
> da097676-7fa7-4b9c-b962-a2fe467fe565 in datacenter: EU-ro-01 to execute
> migration
> 
> 2022-03-29 18:11:04,120 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-256:ctx-43438d6c) (logid:e122e3d6) Seq 
> 25-1196268651020288049:
> Response Received:
> 
> 2022-03-29 18:11:04,121 DEBUG [c.c.a.t.Request]
> (DirectAgent-256:ctx-43438d6c) (logid:e122e3d6) Seq 
> 25-1196268651020288049:
> Processing:  { Ans: , MgmtId: 39908814623087, via:
> 25(spb-ro-002.domain.host), Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":"false","details":"Exception:
> com.cloud.utils.exception.CloudRuntimeException
> 
> Message: Change data store for VM i-2-351-VM failed
> 
> Stack: com.cloud.utils.exception.CloudRuntimeException: Change data 
> store
> for VM i-2-351-VM failed
> 
> at
>
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareR
> esource.java:4550)
> 
> at
>
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.j
> ava:4507)
> 
> at
>
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareRes
> ource.java:501)
> 
> at
>
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAtta
> che.java:315)
> 
> at
>
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
> ntextRunnable.java:48)
> 
> at
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
> ultManagedContext.java:55)
> 
> at
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
> text(DefaultManagedContext.java:102)
> 
> at
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
> ext(DefaultManagedContext.java:52)
> 
> at
>
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
> extRunnable.java:45)
> 
> at
>
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java
> :515)
> 
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> 
> at
>
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTa
> sk.run(ScheduledThreadPoolExecutor.java:304)
> 
> at
>
java.base/java.util.concurrent.ThreadPoolExecutor.runWorke

RE: Migration fail - Unable to find the target datastore

[cristian.c]

I forgot to mention that I was trying first to migrate the volume and start
the VM, but I get error on trying to migrate the volume ( Vmware local
storge ) Unable to access the virtual machine configuration: Unable to
access file [ro002] i-2-351-VM/i-2-351-VM.vmx,  then I was trying with
"Migrate instance to another primary storage" which gives the bellow error..

 

From: cristian.c@istream.today  
Sent: Tuesday, March 29, 2022 9:21 PM
To: users@cloudstack.apache.org
Subject: Migration fail - Unable to find the target datastore 

 

Hello,

 

   Any idea why the migration fail with this error?  I have the same error
starting with CloudStack 4.16  ( also had some erros with 4.15, worked
before with older ACS ) I just did an update today to 4.16.1 and I see the
same error..

 

 

Error :

022-03-29 18:11:04,052 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-13:null) (logid:) SeqA 3-1551701: Sending Seq
3-1551701:  { Ans: , MgmtId: 39908814623087, via: 3, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypa
ssHostMaintenance":"false"}}] }

2022-03-29 18:11:04,067 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-256:ctx-43438d6c spb-ro-002.domain.host, job-2647/job-2648,
cmd: MigrateVmToPoolCommand) (logid:e122e3d6) Unable to find the target
datastore: da097676-7fa7-4b9c-b962-a2fe467fe565 in datacenter: RO-01 to
execute migration

2022-03-29 18:11:04,118 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-256:ctx-43438d6c spb-ro-002.domain.host, job-2647/job-2648,
cmd: MigrateVmToPoolCommand) (logid:e122e3d6) Change data store for VM
i-2-351-VM failed: Unable to find the target datastore:
da097676-7fa7-4b9c-b962-a2fe467fe565 in datacenter: EU-ro-01 to execute
migration

2022-03-29 18:11:04,120 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-256:ctx-43438d6c) (logid:e122e3d6) Seq 25-1196268651020288049:
Response Received:

2022-03-29 18:11:04,121 DEBUG [c.c.a.t.Request]
(DirectAgent-256:ctx-43438d6c) (logid:e122e3d6) Seq 25-1196268651020288049:
Processing:  { Ans: , MgmtId: 39908814623087, via:
25(spb-ro-002.domain.host), Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"Exception:
com.cloud.utils.exception.CloudRuntimeException

Message: Change data store for VM i-2-351-VM failed

Stack: com.cloud.utils.exception.CloudRuntimeException: Change data store
for VM i-2-351-VM failed

at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareR
esource.java:4550)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.j
ava:4507)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareRes
ource.java:501)

at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAtta
che.java:315)

at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:48)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:55)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:102)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:52)

at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:45)

at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java
:515)

at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTa
sk.run(ScheduledThreadPoolExecutor.java:304)

at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut
or.java:1128)

at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu
tor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to find
the target datastore: da097676-7fa7-4b9c-b962-a2fe467fe565 in datacenter:
EU-ro-01 to execute migration

at
com.cloud.hypervisor.vmware.resource.VmwareResource.relocateVirtualMachine(V
mwareResource.java:7588)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareR
esource.java:4545)

... 14 more

 

 

I have tested with different hosts, zones, all with 4.16.x and VMware 6.5
u3.   Any suggestions? 

 

 

 

Thank you,

Cristian

 

 

Migration fail - Unable to find the target datastore

[cristian.c]

Hello,

 

   Any idea why the migration fail with this error?  I have the same error
starting with CloudStack 4.16  ( also had some erros with 4.15, worked
before with older ACS ) I just did an update today to 4.16.1 and I see the
same error..

 

 

Error :

022-03-29 18:11:04,052 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-13:null) (logid:) SeqA 3-1551701: Sending Seq
3-1551701:  { Ans: , MgmtId: 39908814623087, via: 3, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypa
ssHostMaintenance":"false"}}] }

2022-03-29 18:11:04,067 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-256:ctx-43438d6c spb-ro-002.domain.host, job-2647/job-2648,
cmd: MigrateVmToPoolCommand) (logid:e122e3d6) Unable to find the target
datastore: da097676-7fa7-4b9c-b962-a2fe467fe565 in datacenter: RO-01 to
execute migration

2022-03-29 18:11:04,118 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-256:ctx-43438d6c spb-ro-002.domain.host, job-2647/job-2648,
cmd: MigrateVmToPoolCommand) (logid:e122e3d6) Change data store for VM
i-2-351-VM failed: Unable to find the target datastore:
da097676-7fa7-4b9c-b962-a2fe467fe565 in datacenter: EU-ro-01 to execute
migration

2022-03-29 18:11:04,120 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-256:ctx-43438d6c) (logid:e122e3d6) Seq 25-1196268651020288049:
Response Received:

2022-03-29 18:11:04,121 DEBUG [c.c.a.t.Request]
(DirectAgent-256:ctx-43438d6c) (logid:e122e3d6) Seq 25-1196268651020288049:
Processing:  { Ans: , MgmtId: 39908814623087, via:
25(spb-ro-002.domain.host), Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"Exception:
com.cloud.utils.exception.CloudRuntimeException

Message: Change data store for VM i-2-351-VM failed

Stack: com.cloud.utils.exception.CloudRuntimeException: Change data store
for VM i-2-351-VM failed

at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareR
esource.java:4550)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.j
ava:4507)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareRes
ource.java:501)

at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAtta
che.java:315)

at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:48)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:55)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:102)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:52)

at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:45)

at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java
:515)

at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTa
sk.run(ScheduledThreadPoolExecutor.java:304)

at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut
or.java:1128)

at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu
tor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to find
the target datastore: da097676-7fa7-4b9c-b962-a2fe467fe565 in datacenter:
EU-ro-01 to execute migration

at
com.cloud.hypervisor.vmware.resource.VmwareResource.relocateVirtualMachine(V
mwareResource.java:7588)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareR
esource.java:4545)

... 14 more

 

 

I have tested with different hosts, zones, all with 4.16.x and VMware 6.5
u3.   Any suggestions? 

 

 

 

Thank you,

Cristian

 

 

RE: New zone in ACS 4.16 - Unable to add the external cluster

[cristian.c]

Hi Suresh,

  I was able to reproduce the error with "Unable to add the external cluster"  
when you create the zone, for some reasons the "vSwitch" labels are not saved 
for all the traffic type interface/s (in wizard).   This is why adding a 
cluster, fails.
  
   If you close the wizard and you update the labels from UI for the partial 
created zone, you can add the cluster with success.  

  
Regards,
Cristian
 

-Original Message-
From: Suresh Anaparti  
Sent: Wednesday, January 19, 2022 9:30 AM
To: users@cloudstack.apache.org
Subject: Re: New zone in ACS 4.16 - Unable to add the external cluster

Hi Cristian,

Have you noticed the same issue as this: 
https://github.com/apache/cloudstack/issues/5732 (this issue is already fixed, 
and will be available in 4.16.1). If not, you can log a ticket. Thanks.


Regards,
Suresh

On 19/01/22, 12:48 PM, "Suresh Anaparti"  wrote:

Hi Cristian,

Thanks for the update. Can you log a ticket (with the UI problem you 
noticed) here: https://github.com/apache/cloudstack/issues.


Regards,
Suresh

On 19/01/22, 12:40 PM, "cristian.c@istream.today" 
 wrote:

Hi  Suresh,

I think you can ignore this, I removed the zone from UI and added again 
and works fine...  btw, at the step where you can add multiple physical 
networks, if I add more than 1 there is no way to click on "next button" is not 
working, tired multiple times. ( I had to add after I added the zone)

Regards,
Cristian


-Original Message-
From: Suresh Anaparti  
Sent: Tuesday, January 18, 2022 4:14 PM
To: users@cloudstack.apache.org
Subject: Re: New zone in ACS 4.16 - Unable to add the external cluster

Hi Cristian,

It seems server discoverer couldn't detect the hosts on the cluster in 
VMware DC. Can you check the management port group in the hosts is same as in 
the global setting 'vmware.management.portgroup', and traffic label configured 
in the zone is in sync with the virtual switch on the hosts.


Regards,
    Suresh

On 18/01/22, 7:27 PM, "cristian.c@istream.today" 
 wrote:

Hi Suresh,

I just tested via API, same error, but something is strange in 
the log:

2022-01-18 08:44:21,900 INFO  [c.c.h.v.u.VmwareContext] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) New 
VmwareContext object, current outstanding count: 14
2022-01-18 08:44:21,904 ERROR [c.c.h.v.u.VmwareContext] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) Path 
element points to an un-existing inventory entity
2022-01-18 08:44:21,904 ERROR [c.c.h.v.m.VmwareManagerImpl] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Unable to find host from inventory path:
2022-01-18 08:44:21,906 INFO  [c.c.h.v.VmwareServerDiscoverer] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Found 0 hosts.
2022-01-18 08:44:21,906 ERROR [c.c.h.v.VmwareServerDiscoverer] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Unable to find host or cluster based on url:
2022-01-18 08:44:21,906 INFO  [c.c.h.v.u.VmwareContext] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Disconnecting VMware session
2022-01-18 08:44:21,908 WARN  [c.c.r.ResourceManagerImpl] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Unable to find the server resources at https://vmware-ams-002.domain.com
2022-01-18 08:44:21,909 INFO  [c.c.u.e.CSExceptionErrorCode] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Could not find exception: com.cloud.exception.DiscoveryException in error code 
list for exceptions
2022-01-18 08:44:21,918 WARN  [o.a.c.a.c.a.c.AddClusterCmd] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Exception:
com.cloud.exception.DiscoveryException: Unable to add the external 
cluster


I see " Unable to find host from inventory path:" but no path .

Regards,
Cristian

-Original Message-
From: Suresh Anaparti  
Sent: Tuesday, January 18, 2022 3:02 PM
To: users@cloudstack.apache.org
Subject: Re: New zone in ACS 4.16 - Unable to add the external 
cluster

Hi Cristian,

I'm not sure if this is a bug. Have you tried with api/cmk?

Can you share the complete log of zone creation. If possible, 
please create a ticket with details here: 
https://github.com/apache/cloudstack/issues.


    Regards,
Suresh

On 18/01/22, 2:41 PM, "cristian.c@istream.today" 
 wrote:

Hello,



   

RE: New zone in ACS 4.16 - Unable to add the external cluster

[cristian.c]

Hi  Suresh,

I think you can ignore this, I removed the zone from UI and added again and 
works fine...  btw, at the step where you can add multiple physical networks, 
if I add more than 1 there is no way to click on "next button" is not working, 
tired multiple times. ( I had to add after I added the zone)

Regards,
Cristian


-Original Message-
From: Suresh Anaparti  
Sent: Tuesday, January 18, 2022 4:14 PM
To: users@cloudstack.apache.org
Subject: Re: New zone in ACS 4.16 - Unable to add the external cluster

Hi Cristian,

It seems server discoverer couldn't detect the hosts on the cluster in VMware 
DC. Can you check the management port group in the hosts is same as in the 
global setting 'vmware.management.portgroup', and traffic label configured in 
the zone is in sync with the virtual switch on the hosts.

 
Regards,
Suresh

On 18/01/22, 7:27 PM, "cristian.c@istream.today"  
wrote:

Hi Suresh,

I just tested via API, same error, but something is strange in the log:

2022-01-18 08:44:21,900 INFO  [c.c.h.v.u.VmwareContext] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) New 
VmwareContext object, current outstanding count: 14
2022-01-18 08:44:21,904 ERROR [c.c.h.v.u.VmwareContext] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) Path 
element points to an un-existing inventory entity
2022-01-18 08:44:21,904 ERROR [c.c.h.v.m.VmwareManagerImpl] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Unable to find host from inventory path:
2022-01-18 08:44:21,906 INFO  [c.c.h.v.VmwareServerDiscoverer] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Found 0 hosts.
2022-01-18 08:44:21,906 ERROR [c.c.h.v.VmwareServerDiscoverer] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Unable to find host or cluster based on url:
2022-01-18 08:44:21,906 INFO  [c.c.h.v.u.VmwareContext] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Disconnecting VMware session
2022-01-18 08:44:21,908 WARN  [c.c.r.ResourceManagerImpl] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Unable to find the server resources at https://vmware-ams-002.domain.com
2022-01-18 08:44:21,909 INFO  [c.c.u.e.CSExceptionErrorCode] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Could not find exception: com.cloud.exception.DiscoveryException in error code 
list for exceptions
2022-01-18 08:44:21,918 WARN  [o.a.c.a.c.a.c.AddClusterCmd] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Exception:
com.cloud.exception.DiscoveryException: Unable to add the external cluster


I see " Unable to find host from inventory path:" but no path .

Regards,
Cristian

-Original Message-
From: Suresh Anaparti  
Sent: Tuesday, January 18, 2022 3:02 PM
To: users@cloudstack.apache.org
Subject: Re: New zone in ACS 4.16 - Unable to add the external cluster

Hi Cristian,

I'm not sure if this is a bug. Have you tried with api/cmk?

Can you share the complete log of zone creation. If possible, please create 
a ticket with details here: https://github.com/apache/cloudstack/issues.


Regards,
Suresh

On 18/01/22, 2:41 PM, "cristian.c@istream.today"  
wrote:

Hello,



Is this a bug?   I'm trying to add a new ZONE in a upgraded 
cloudstack
environment but fails. I have already 4 active zones which are created 
long
time ago.



Unable to connect to Vmware vSphere server. service address:
vmware-ams-002.domain.com. java.lang.NumberFormatException: For input
string: "null"

2022-01-18 04:03:30,849 INFO  [c.c.h.v.u.VmwareContext]
(qtp2053591126-636108:ctx-62b88abf ctx-06a37bc9) (logid:91c422ef)
Disconnecting VMware session

2022-01-18 04:03:30,851 WARN  [c.c.r.ResourceManagerImpl]
(qtp2053591126-636108:ctx-62b88abf ctx-06a37bc9) (logid:91c422ef) 
Unable to
find the server resources at
http://vmware-ams-002.domain.com/AMS-BR-01/SAO-01

2022-01-18 04:03:30,851 INFO  [c.c.u.e.CSExceptionErrorCode]
(qtp2053591126-636108:ctx-62b88abf ctx-06a37bc9) (logid:91c422ef) Could 
not
find exception: com.cloud.exception.DiscoveryException in error code 
list
for exceptions

2022-01-18 04:03:30,863 WARN  [o.a.c.a.c.a.c.AddClusterCmd]
(qtp2053591126-636108:ctx-62b88abf ctx-06a37bc9) (logid:91c422ef) 
Exception:

com.cloud.exception.DiscoveryException: Unable to add the external 
cluster

at

com.cloud.resource.ResourceManagerImpl.discoverCluster(ResourceManagerImpl.j
ava:586)

at 
jdk.internal.reflect.GeneratedMethodAccessor2020.invoke(Unknown

RE: New zone in ACS 4.16 - Unable to add the external cluster

[cristian.c]

Hi Suresh,

I just tested via API, same error, but something is strange in the log:

2022-01-18 08:44:21,900 INFO  [c.c.h.v.u.VmwareContext] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) New 
VmwareContext object, current outstanding count: 14
2022-01-18 08:44:21,904 ERROR [c.c.h.v.u.VmwareContext] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) Path 
element points to an un-existing inventory entity
2022-01-18 08:44:21,904 ERROR [c.c.h.v.m.VmwareManagerImpl] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Unable to find host from inventory path:
2022-01-18 08:44:21,906 INFO  [c.c.h.v.VmwareServerDiscoverer] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Found 0 hosts.
2022-01-18 08:44:21,906 ERROR [c.c.h.v.VmwareServerDiscoverer] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Unable to find host or cluster based on url:
2022-01-18 08:44:21,906 INFO  [c.c.h.v.u.VmwareContext] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Disconnecting VMware session
2022-01-18 08:44:21,908 WARN  [c.c.r.ResourceManagerImpl] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Unable to find the server resources at https://vmware-ams-002.domain.com
2022-01-18 08:44:21,909 INFO  [c.c.u.e.CSExceptionErrorCode] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Could not find exception: com.cloud.exception.DiscoveryException in error code 
list for exceptions
2022-01-18 08:44:21,918 WARN  [o.a.c.a.c.a.c.AddClusterCmd] 
(qtp2053591126-20:ctx-3ffe2578 ctx-d0c69f6d ctx-5e23a0d3) (logid:812d3e06) 
Exception:
com.cloud.exception.DiscoveryException: Unable to add the external cluster


I see " Unable to find host from inventory path:" but no path .

Regards,
Cristian

-Original Message-
From: Suresh Anaparti  
Sent: Tuesday, January 18, 2022 3:02 PM
To: users@cloudstack.apache.org
Subject: Re: New zone in ACS 4.16 - Unable to add the external cluster

Hi Cristian,

I'm not sure if this is a bug. Have you tried with api/cmk?

Can you share the complete log of zone creation. If possible, please create a 
ticket with details here: https://github.com/apache/cloudstack/issues.

 
Regards,
Suresh

On 18/01/22, 2:41 PM, "cristian.c@istream.today"  
wrote:

Hello,



Is this a bug?   I'm trying to add a new ZONE in a upgraded cloudstack
environment but fails. I have already 4 active zones which are created long
time ago.



Unable to connect to Vmware vSphere server. service address:
vmware-ams-002.domain.com. java.lang.NumberFormatException: For input
string: "null"

2022-01-18 04:03:30,849 INFO  [c.c.h.v.u.VmwareContext]
(qtp2053591126-636108:ctx-62b88abf ctx-06a37bc9) (logid:91c422ef)
Disconnecting VMware session

2022-01-18 04:03:30,851 WARN  [c.c.r.ResourceManagerImpl]
(qtp2053591126-636108:ctx-62b88abf ctx-06a37bc9) (logid:91c422ef) Unable to
find the server resources at
http://vmware-ams-002.domain.com/AMS-BR-01/SAO-01

2022-01-18 04:03:30,851 INFO  [c.c.u.e.CSExceptionErrorCode]
(qtp2053591126-636108:ctx-62b88abf ctx-06a37bc9) (logid:91c422ef) Could not
find exception: com.cloud.exception.DiscoveryException in error code list
for exceptions

2022-01-18 04:03:30,863 WARN  [o.a.c.a.c.a.c.AddClusterCmd]
(qtp2053591126-636108:ctx-62b88abf ctx-06a37bc9) (logid:91c422ef) Exception:

com.cloud.exception.DiscoveryException: Unable to add the external cluster

at
com.cloud.resource.ResourceManagerImpl.discoverCluster(ResourceManagerImpl.j
ava:586)

at jdk.internal.reflect.GeneratedMethodAccessor2020.invoke(Unknown
Source)

at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Delegatin
gMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)





  I get this error over an over no matter what, I'm not sure why is
saying that is unable to find the server resources at .. 



 ( I use the same vCenter for other zones and I do not have any issues
with this, version 6.5 u3.)



  I highly appreciate any suggestions.



Thank you,

Cristian





 

New zone in ACS 4.16 - Unable to add the external cluster

[cristian.c]

Hello,

 

Is this a bug?   I'm trying to add a new ZONE in a upgraded cloudstack
environment but fails. I have already 4 active zones which are created long
time ago.

 

Unable to connect to Vmware vSphere server. service address:
vmware-ams-002.domain.com. java.lang.NumberFormatException: For input
string: "null"

2022-01-18 04:03:30,849 INFO  [c.c.h.v.u.VmwareContext]
(qtp2053591126-636108:ctx-62b88abf ctx-06a37bc9) (logid:91c422ef)
Disconnecting VMware session

2022-01-18 04:03:30,851 WARN  [c.c.r.ResourceManagerImpl]
(qtp2053591126-636108:ctx-62b88abf ctx-06a37bc9) (logid:91c422ef) Unable to
find the server resources at
http://vmware-ams-002.domain.com/AMS-BR-01/SAO-01

2022-01-18 04:03:30,851 INFO  [c.c.u.e.CSExceptionErrorCode]
(qtp2053591126-636108:ctx-62b88abf ctx-06a37bc9) (logid:91c422ef) Could not
find exception: com.cloud.exception.DiscoveryException in error code list
for exceptions

2022-01-18 04:03:30,863 WARN  [o.a.c.a.c.a.c.AddClusterCmd]
(qtp2053591126-636108:ctx-62b88abf ctx-06a37bc9) (logid:91c422ef) Exception:

com.cloud.exception.DiscoveryException: Unable to add the external cluster

at
com.cloud.resource.ResourceManagerImpl.discoverCluster(ResourceManagerImpl.j
ava:586)

at jdk.internal.reflect.GeneratedMethodAccessor2020.invoke(Unknown
Source)

at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Delegatin
gMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)

 

 

  I get this error over an over no matter what, I'm not sure why is
saying that is unable to find the server resources at .. 

 

 ( I use the same vCenter for other zones and I do not have any issues
with this, version 6.5 u3.)

 

  I highly appreciate any suggestions.

 

Thank you,

Cristian

 

Failed to update VM, due to: Invalid cpu speed value, specify a value between 1 and 2147483647

[cristian.c]

Hello,

 

When I want to save a new value for CPU speed I get this error "Failed
to update VM, due to: Invalid cpu speed value, specify a value between 1 and
2147483647"  in the UI, no matter what I set, I get this in the new
CloudStack version 4.16.

 

 

Regards,

Cristian

RE: Failed to find datastore for pool UUID 4.15.2 VMware migration

[cristian.c]

Hi Harikrishna,

  Yes, I just try a simple migration for stopped VM(VMware) with local
storage to a different storage/host in the same cluster, I did this before
without any issue in 4.15.1 as I remember.

Regards,
Cristian

   

-Original Message-
From: Harikrishna Patnala  
Sent: Tuesday, November 9, 2021 7:48 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to find datastore for pool UUID 4.15.2 VMware migration

Hi Cristian,

Based on the error that you have shared, looks like this won't be an issue
in 4.16, I did some quick check in code base. You can test with the latest
4.16 RC3 artifacts. May I know what is the exact operation that you are
trying to do here? is it migrating a stopped or running VM to a different
pool in the same cluster or across cluster?

Thanks,
Harikrishna


From: cristian.c@istream.today 
Sent: Sunday, November 7, 2021 3:15 PM
To: users@cloudstack.apache.org 
Subject: RE: Failed to find datastore for pool UUID 4.15.2 VMware migration

Update:



   Correction " Not able to find the destination:



Regards,

Cristian



From: cristian.c@istream.today 
Sent: Sunday, November 7, 2021 11:41 AM
To: users@cloudstack.apache.org
Subject: Failed to find datastore for pool UUID 4.15.2 VMware migration



Hello,





 I just tested the migration in 4.15.2 and I get the following error
"Failed to find datastore for pool UUID" Looks like is not able to find the
source, which does not make sense, if I migrate from VMware between host
works perfectly, also the deploy VM deploy in acs works fine on any host, is
this a bug?





2021-11-07 09:29:10,837 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-396:ctx-0333aa69 ldn-uk-002.shape.host, job-2412/job-2413, cmd:
MigrateVmToPoolCommand) (logid:72a9c33a) Failed to find datastore for pool
UUID: ce27d8fc-27cf-4754-8a82-e27497c390a7

2021-11-07 09:29:10,839 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-396:ctx-0333aa69) (logid:72a9c33a) Seq 15-6715711469339541698:
Response Received:

2021-11-07 09:29:10,840 DEBUG [c.c.a.t.Request]
(DirectAgent-396:ctx-0333aa69) (logid:72a9c33a) Seq 15-6715711469339541698:
Processing:  { Ans: , MgmtId: 39908814623087, via:
15(ldn-uk-002.shape.host), Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"Exception:
com.cloud.utils.exception.CloudRuntimeException

Message: Failed to find datastore for pool UUID:
ce27d8fc-27cf-4754-8a82-e27497c390a7

Stack: com.cloud.utils.exception.CloudRuntimeException: Failed to find
datastore for pool UUID: ce27d8fc-27cf-4754-8a82-e27497c390a7

at
com.cloud.hypervisor.vmware.resource.VmwareResource.getTargetDatastoreMORefe
rence(VmwareResource.java:4522)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareR
esource.java:4416)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.j
ava:4401)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareRes
ource.java:494)

at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAtta
che.java:315)

at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:48)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:55)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:102)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:52)

at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:45)

at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java
:515)

at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTa
sk.run(ScheduledThreadPoolExecutor.java:304)

at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut
or.java:1128)

at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu
tor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

","wait":"0","bypassHostMaintenance":"false"}}] }

2021-11-07 09:29:10,840 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-396:ctx-0333aa69) (logid:72a9c33a) Seq 15-6715711469339541698:
No more commands found

2021-11-07 09:29:10,840 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-6:ctx-2cfa416a job-2412/job-2413 ctx-d6941fcf)
(logid:72a9c33a) Seq 15-6715711469339541698: Received:  { Ans: , MgmtId:
39908814623087, via: 15(ldn-uk-002.shape.host), Ver: v1, Flags: 110, {
Answer } }





Regards,

Cristian


 

RE: Failed to find datastore for pool UUID 4.15.2 VMware migration

[cristian.c]

Update:

 

   Correction " Not able to find the destination:

 

Regards,

Cristian

 

From: cristian.c@istream.today  
Sent: Sunday, November 7, 2021 11:41 AM
To: users@cloudstack.apache.org
Subject: Failed to find datastore for pool UUID 4.15.2 VMware migration

 

Hello,

 

 

 I just tested the migration in 4.15.2 and I get the following error
"Failed to find datastore for pool UUID" Looks like is not able to find the
source, which does not make sense, if I migrate from VMware between host
works perfectly, also the deploy VM deploy in acs works fine on any host, is
this a bug?

 

 

2021-11-07 09:29:10,837 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-396:ctx-0333aa69 ldn-uk-002.shape.host, job-2412/job-2413, cmd:
MigrateVmToPoolCommand) (logid:72a9c33a) Failed to find datastore for pool
UUID: ce27d8fc-27cf-4754-8a82-e27497c390a7

2021-11-07 09:29:10,839 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-396:ctx-0333aa69) (logid:72a9c33a) Seq 15-6715711469339541698:
Response Received:

2021-11-07 09:29:10,840 DEBUG [c.c.a.t.Request]
(DirectAgent-396:ctx-0333aa69) (logid:72a9c33a) Seq 15-6715711469339541698:
Processing:  { Ans: , MgmtId: 39908814623087, via:
15(ldn-uk-002.shape.host), Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"Exception:
com.cloud.utils.exception.CloudRuntimeException

Message: Failed to find datastore for pool UUID:
ce27d8fc-27cf-4754-8a82-e27497c390a7

Stack: com.cloud.utils.exception.CloudRuntimeException: Failed to find
datastore for pool UUID: ce27d8fc-27cf-4754-8a82-e27497c390a7

at
com.cloud.hypervisor.vmware.resource.VmwareResource.getTargetDatastoreMORefe
rence(VmwareResource.java:4522)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareR
esource.java:4416)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.j
ava:4401)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareRes
ource.java:494)

at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAtta
che.java:315)

at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:48)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:55)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:102)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:52)

at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:45)

at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java
:515)

at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTa
sk.run(ScheduledThreadPoolExecutor.java:304)

at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut
or.java:1128)

at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu
tor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

","wait":"0","bypassHostMaintenance":"false"}}] }

2021-11-07 09:29:10,840 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-396:ctx-0333aa69) (logid:72a9c33a) Seq 15-6715711469339541698:
No more commands found

2021-11-07 09:29:10,840 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-6:ctx-2cfa416a job-2412/job-2413 ctx-d6941fcf)
(logid:72a9c33a) Seq 15-6715711469339541698: Received:  { Ans: , MgmtId:
39908814623087, via: 15(ldn-uk-002.shape.host), Ver: v1, Flags: 110, {
Answer } }

 

 

Regards,

Cristian

Failed to find datastore for pool UUID 4.15.2 VMware migration

[cristian.c]

Hello,

 

 

 I just tested the migration in 4.15.2 and I get the following error
"Failed to find datastore for pool UUID" Looks like is not able to find the
source, which does not make sense, if I migrate from VMware between host
works perfectly, also the deploy VM deploy in acs works fine on any host, is
this a bug?

 

 

2021-11-07 09:29:10,837 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-396:ctx-0333aa69 ldn-uk-002.shape.host, job-2412/job-2413, cmd:
MigrateVmToPoolCommand) (logid:72a9c33a) Failed to find datastore for pool
UUID: ce27d8fc-27cf-4754-8a82-e27497c390a7

2021-11-07 09:29:10,839 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-396:ctx-0333aa69) (logid:72a9c33a) Seq 15-6715711469339541698:
Response Received:

2021-11-07 09:29:10,840 DEBUG [c.c.a.t.Request]
(DirectAgent-396:ctx-0333aa69) (logid:72a9c33a) Seq 15-6715711469339541698:
Processing:  { Ans: , MgmtId: 39908814623087, via:
15(ldn-uk-002.shape.host), Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"Exception:
com.cloud.utils.exception.CloudRuntimeException

Message: Failed to find datastore for pool UUID:
ce27d8fc-27cf-4754-8a82-e27497c390a7

Stack: com.cloud.utils.exception.CloudRuntimeException: Failed to find
datastore for pool UUID: ce27d8fc-27cf-4754-8a82-e27497c390a7

at
com.cloud.hypervisor.vmware.resource.VmwareResource.getTargetDatastoreMORefe
rence(VmwareResource.java:4522)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareR
esource.java:4416)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.j
ava:4401)

at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareRes
ource.java:494)

at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAtta
che.java:315)

at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:48)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:55)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:102)

at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:52)

at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:45)

at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java
:515)

at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTa
sk.run(ScheduledThreadPoolExecutor.java:304)

at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut
or.java:1128)

at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu
tor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

","wait":"0","bypassHostMaintenance":"false"}}] }

2021-11-07 09:29:10,840 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-396:ctx-0333aa69) (logid:72a9c33a) Seq 15-6715711469339541698:
No more commands found

2021-11-07 09:29:10,840 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-6:ctx-2cfa416a job-2412/job-2413 ctx-d6941fcf)
(logid:72a9c33a) Seq 15-6715711469339541698: Received:  { Ans: , MgmtId:
39908814623087, via: 15(ldn-uk-002.shape.host), Ver: v1, Flags: 110, {
Answer } }

 

 

Regards,

Cristian

Copy Template - client denied by server configuration

[cristian.c]

Hello,

 

  New zone, new error .  This error just started after we added new ZONE, 
when we are trying to copy a template from Zone A or B to Zone C,  I get this 
error : HTTP Server returned 403 (expected 200 OK), if I check the logs (Apache 
logs ) on source secondary storage, I see this : AH01797: client denied by 
server configuration: 
/var/www/html/copy/SecStorage/883d712b-425d-3954-b0f1-146c929b798e/template/tmpl/2/204/851d353a-32aa-3e65-8796-f8a22247bb5d.ova

 

 I didn’t changed any configuration, it is the first time when I see this 
type of error and I do not understand the reason why.

 

 If I execute the : /usr/local/cloud/systemvm/ssvm-check.sh on both 
storages, everything is OK.

 

 Any suggestions?

 

 

Regards,

Cristian

RE: Multiple Network labels - custom

[cristian.c]

Update:

I had add tags to all networks/in each zone, looks like right now is 
working, I'm not sure if this is a coincidence or not.


Regards,
Cristian

-Original Message-
From: cristian.c@istream.today  
Sent: Tuesday, October 12, 2021 4:28 PM
To: users@cloudstack.apache.org
Subject: RE: Multiple Network labels - custom

Let's say this is true, I cannot deploy here because the the KVM is disabled, 
but there is no such think in my setup which is allowing to deploy systemvms 
only on KVM.

In order to avoid this, I just removed the KVM host from this setup, not sure 
why is trying to deploy the router on KVM over and over.

- Restarted the network, the router deploy works fine, I can use the networks 
with tags in my test environment with 1 zone. (cloudstack 4.15.2)

Now I have to figure out why the same configuration is not working in a 
different environment, the only difference is multiple zones and cloudstack 
version (cloudstack 4.15.1 only with VMware)

Thank you,
Cristian

-Original Message-
From: Wei ZHOU 
Sent: Tuesday, October 12, 2021 4:01 PM
To: users 
Subject: Re: Multiple Network labels - custom

Hi Cristian,

The configuration looks good.

It looks like cloudstack cannot start the router because the router is 
allocated to be deployed on a KVM hypervisor but the only kvm host is disabled. 
so it is a capacity issue.

-Wei




On Tue, 12 Oct 2021 at 14:41,  wrote:

> Hi Wei,
>
>
>
>   Yes, of course  
>
>
>
>   I'm able to create the network with any of the network offering, 
> which are with tags, the problem is when the router is starting to 
> deploy, I get the same error without to many details, tested in 2 
> cloudstack environment
> 4.15.1 and 4.15.2, in 1 I have multiple zones and in the test one I 
> have only 1 zone with 2 hosts KVM and VMware.
>
>
>
> For me is pretty simple:
>
>
>
> Zone X:
>
>
>
> 1 . Assign a tag to each physical network, for the default 1 and for 
> any additional network.
>
> 2. Create a network offering with the corresponding tags, I also saw 
> that you can update the tags for default network offering.
>
> 3. Create network with any of those offerings
>
>
>
> Here I attached screenshots : https://imgur.com/a/VtvYsvSI do not see
> any issue in my configuration.
>
>
>
> Regards,
>
> Cristian
>
>
>
> -Original Message-
> From: Wei ZHOU 
> Sent: Tuesday, October 12, 2021 3:21 PM
> To: users 
> Subject: Re: Multiple Network labels - custom
>
>
>
> Hi Cristian,
>
>
>
> I think it is good. You do not need to change the physical networks.
>
>
>
> Have you created a new "network offering" with a tag (A or B) ?
>
>
>
> -Wei
>
>
>
> On Tue, 12 Oct 2021 at 14:02, < <mailto:cristian.c@istream.today> 
> cristian.c@istream.today> wrote:
>
>
>
> > I have only 1 network by default :
>
> >
>
> >  1. Management / Storage / Public / Guest   - Tag A
>
> >  2. Guest  - Tag B
>
> >
>
> > Now, if I understand right, this is not working in this way? I 
> > should
>
> > have separated network for guest from the beginning?
>
> >
>
> > 1 . Management / Storage /Public  - Tag A 2. Guest - Tag B 3. Guest
> > -
>
> > Tag C
>
> >
>
> >
>
> > Regarding running networks, I will try to rebuild after I have the
>
> > current issue fixed.
>
> >
>
> > Zone, I think the same, but not sure yet
>
> >
>
> > Thank you,
>
> > Cristian
>
> >
>
> >
>
>

RE: Multiple Network labels - custom

[cristian.c]

Let's say this is true, I cannot deploy here because the the KVM is disabled, 
but there is no such think in my setup which is allowing to deploy systemvms 
only on KVM.

In order to avoid this, I just removed the KVM host from this setup, not sure 
why is trying to deploy the router on KVM over and over.

- Restarted the network, the router deploy works fine, I can use the networks 
with tags in my test environment with 1 zone. (cloudstack 4.15.2)

Now I have to figure out why the same configuration is not working in a 
different environment, the only difference is multiple zones and cloudstack 
version (cloudstack 4.15.1 only with VMware)

Thank you,
Cristian

-Original Message-
From: Wei ZHOU  
Sent: Tuesday, October 12, 2021 4:01 PM
To: users 
Subject: Re: Multiple Network labels - custom

Hi Cristian,

The configuration looks good.

It looks like cloudstack cannot start the router because the router is 
allocated to be deployed on a KVM hypervisor but the only kvm host is disabled. 
so it is a capacity issue.

-Wei




On Tue, 12 Oct 2021 at 14:41,  wrote:

> Hi Wei,
>
>
>
>   Yes, of course  
>
>
>
>   I'm able to create the network with any of the network offering, 
> which are with tags, the problem is when the router is starting to 
> deploy, I get the same error without to many details, tested in 2 
> cloudstack environment
> 4.15.1 and 4.15.2, in 1 I have multiple zones and in the test one I 
> have only 1 zone with 2 hosts KVM and VMware.
>
>
>
> For me is pretty simple:
>
>
>
> Zone X:
>
>
>
> 1 . Assign a tag to each physical network, for the default 1 and for 
> any additional network.
>
> 2. Create a network offering with the corresponding tags, I also saw 
> that you can update the tags for default network offering.
>
> 3. Create network with any of those offerings
>
>
>
> Here I attached screenshots : https://imgur.com/a/VtvYsvSI do not see
> any issue in my configuration.
>
>
>
> Regards,
>
> Cristian
>
>
>
> -Original Message-
> From: Wei ZHOU 
> Sent: Tuesday, October 12, 2021 3:21 PM
> To: users 
> Subject: Re: Multiple Network labels - custom
>
>
>
> Hi Cristian,
>
>
>
> I think it is good. You do not need to change the physical networks.
>
>
>
> Have you created a new "network offering" with a tag (A or B) ?
>
>
>
> -Wei
>
>
>
> On Tue, 12 Oct 2021 at 14:02, < <mailto:cristian.c@istream.today> 
> cristian.c@istream.today> wrote:
>
>
>
> > I have only 1 network by default :
>
> >
>
> >  1. Management / Storage / Public / Guest   - Tag A
>
> >  2. Guest  - Tag B
>
> >
>
> > Now, if I understand right, this is not working in this way? I 
> > should
>
> > have separated network for guest from the beginning?
>
> >
>
> > 1 . Management / Storage /Public  - Tag A 2. Guest - Tag B 3. Guest 
> > -
>
> > Tag C
>
> >
>
> >
>
> > Regarding running networks, I will try to rebuild after I have the
>
> > current issue fixed.
>
> >
>
> > Zone, I think the same, but not sure yet
>
> >
>
> > Thank you,
>
> > Cristian
>
> >
>
> >
>
>

RE: Multiple Network labels - custom

[cristian.c]

Hi Wei,

 

  Yes, of course  

  

  I'm able to create the network with any of the network offering, which are 
with tags, the problem is when the router is starting to deploy, I get the same 
error without to many details, tested in 2 cloudstack environment 4.15.1 and 
4.15.2, in 1 I have multiple zones and in the test one I have only 1 zone with 
2 hosts KVM and VMware.

 

For me is pretty simple:

 

Zone X:

 

1 . Assign a tag to each physical network, for the default 1 and for any 
additional network.

2. Create a network offering with the corresponding tags, I also saw that you 
can update the tags for default network offering.

3. Create network with any of those offerings

 

Here I attached screenshots : https://imgur.com/a/VtvYsvSI do not see any 
issue in my configuration.

 

Regards,

Cristian

 

-Original Message-
From: Wei ZHOU  
Sent: Tuesday, October 12, 2021 3:21 PM
To: users 
Subject: Re: Multiple Network labels - custom

 

Hi Cristian,

 

I think it is good. You do not need to change the physical networks.

 

Have you created a new "network offering" with a tag (A or B) ?

 

-Wei

 

On Tue, 12 Oct 2021 at 14:02, < <mailto:cristian.c@istream.today> 
cristian.c@istream.today> wrote:

 

> I have only 1 network by default :

> 

>  1. Management / Storage / Public / Guest   - Tag A

>  2. Guest  - Tag B

> 

> Now, if I understand right, this is not working in this way? I should 

> have separated network for guest from the beginning?

> 

> 1 . Management / Storage /Public  - Tag A 2. Guest - Tag B 3. Guest - 

> Tag C

> 

> 

> Regarding running networks, I will try to rebuild after I have the 

> current issue fixed.

> 

> Zone, I think the same, but not sure yet

> 

> Thank you,

> Cristian

> 

> 

RE: Multiple Network labels - custom

[cristian.c]

orkOrchestrator.prepa
> > re
> > Nic(NetworkOrchestrator.java:1840)
> >
> > at
> > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepa
> > re
> > (NetworkOrchestrator.java:1774)
> >
> > at
> > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachi
> > ne
> > ManagerImpl.java:1158)
> >
> > at
> > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachi
> > ne
> > ManagerImpl.java:5502)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Nati
> > ve
> > Method)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Nativ
> > eM
> > ethodAccessorImpl.java:62)
> >
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(D
> > el
> > egatingMethodAccessorImpl.java:43)
> >
> > at 
> > java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerP
> > ro
> > xy.java:107)
> >
> > at
> > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachin
> > eM
> > anagerImpl.java:5669)
> >
> > at
> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102
> > )
> >
> > at
> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runI
> > nC
> > ontext(AsyncJobManagerImpl.java:620)
> >
> > at
> > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
> > an
> > agedContextRunnable.java:48)
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
> > al
> > l(DefaultManagedContext.java:55)
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
> > lW
> > ithContext(DefaultManagedContext.java:102)
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
> > Wi
> > thContext(DefaultManagedContext.java:52)
> >
> > at
> > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
> > ag
> > edContextRunnable.java:45)
> >
> > at
> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(
> > As
> > yncJobManagerImpl.java:568)
> >
> > at
> > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Execut
> > or
> > s.java:515)
> >
> > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPo
> > ol
> > Executor.java:1128)
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadP
> > oo
> > lExecutor.java:628)
> >
> > at java.base/java.lang.Thread.run(Thread.java:829)
> >
> > "
> >
> >
> >
> > I do not understand what is wrong, doesn't make any sense..
> >
> >
> >
> > 1. I add a second network with the name "Physical Network 2" with 
> > traffic Guest.
> >
> > 2. I enable the network and providers from DB
> >
> > 3. I add tag to the Physical Network 1 and Physical Network 2
> >
> > 4. I create a share network offering with tag for both networks.
> >
> > 5. Enable the new created network offering
> >
> > 6. Create a network with any of the network offering with tag
> >
> > 7. Deploy VM, restart network, clean network = fail with the same error.
> >
> >
> >
> > This issue is happening only when I use tags.
> >
> >
> >
> >
> >
> > Any suggestions?
> >
> >
> >
> >
> >
> > Thank you,
> >
> > Cristian
> >
> >
> >
> >
> >
> > -Original Message-
> > From: vas...@gmx.de 
> > Sent: Friday, October 8, 2021 12:36 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: Multiple Network labels - custom
> >
> >
> >
> > Does the offering "default a" work as expected - meaning Cloudstack 
> > is deploying the network as expected?
> >
> > I can remember that i had some struggle with this, too.
> >
> > Maybe, just for verification
> >
> > a) Check that the service offering is enabled ( :-) )
> >
&g

RE: Multiple Network labels - custom

[cristian.c]

perImpl.java:358)
> >
> > at
> >
> com.cloud.network.router.NetworkHelperImpl.startRouters(NetworkHelperI
> mpl.java:343)
> >
> > at
> >
> org.cloud.network.router.deployment.RouterDeploymentDefinition.deployV
> irtualRouter(RouterDeploymentDefinition.java:206)
> >
> > at
> >
> com.cloud.network.element.VirtualRouterElement.prepare(VirtualRouterEl
> ement.java:285)
> >
> > at
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare
> Element(NetworkOrchestrator.java:1485)
> >
> > at
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare
> Nic(NetworkOrchestrator.java:1840)
> >
> > at
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare
> (NetworkOrchestrator.java:1774)
> >
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachine
> ManagerImpl.java:1158)
> >
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachine
> ManagerImpl.java:5502)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Nati
> > ve
> > Method)
> >
> > at
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> ethodAccessorImpl.java:62)
> >
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> egatingMethodAccessorImpl.java:43)
> >
> > at 
> > java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerPro
> xy.java:107)
> >
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineM
> anagerImpl.java:5669)
> >
> > at
> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102
> > )
> >
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInC
> ontext(AsyncJobManagerImpl.java:620)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man
> agedContextRunnable.java:48)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal
> l(DefaultManagedContext.java:55)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW
> ithContext(DefaultManagedContext.java:102)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi
> thContext(DefaultManagedContext.java:52)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag
> edContextRunnable.java:45)
> >
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(As
> yncJobManagerImpl.java:568)
> >
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> s.java:515)
> >
> > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1128)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:628)
> >
> > at java.base/java.lang.Thread.run(Thread.java:829)
> >
> > "
> >
> >
> >
> > I do not understand what is wrong, doesn't make any sense..
> >
> >
> >
> > 1. I add a second network with the name "Physical Network 2" with
> traffic
> > Guest.
> >
> > 2. I enable the network and providers from DB
> >
> > 3. I add tag to the Physical Network 1 and Physical Network 2
> >
> > 4. I create a share network offering with tag for both networks.
> >
> > 5. Enable the new created network offering
> >
> > 6. Create a network with any of the network offering with tag
> >
> > 7. Deploy VM, restart network, clean network = fail with the same error.
> >
> >
> >
> > This issue is happening only when I use tags.
> >
> >
> >
> >
> >
> > Any suggestions?
> >
> >
> >
> >
> >
> > Thank you,
> >
> > Cristian
> >
> >
> >
> >
> >
> > -Original Message-
> > From: vas...@gmx.de 
>

RE: Multiple Network labels - custom

[cristian.c]

impl.AsyncJobManagerImpl$5.run(As
> yncJobManagerImpl.java:568)
>
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> s.java:515)
>
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:829)
>
> "
>
>
>
> I do not understand what is wrong, doesn't make any sense..
>
>
>
> 1. I add a second network with the name "Physical Network 2" with  
> traffic Guest.
>
> 2. I enable the network and providers from DB
>
> 3. I add tag to the Physical Network 1 and Physical Network 2
>
> 4. I create a share network offering with tag for both networks.
>
> 5. Enable the new created network offering
>
> 6. Create a network with any of the network offering with tag
>
> 7. Deploy VM, restart network, clean network = fail with the same error.
>
>
>
> This issue is happening only when I use tags.
>
>
>
>
>
> Any suggestions?
>
>
>
>
>
> Thank you,
>
> Cristian
>
>
>
>
>
> -Original Message-
> From: vas...@gmx.de 
> Sent: Friday, October 8, 2021 12:36 PM
> To: users@cloudstack.apache.org
> Subject: Re: Multiple Network labels - custom
>
>
>
> Does the offering "default a" work as expected - meaning Cloudstack is 
> deploying the network as expected?
>
> I can remember that i had some struggle with this, too.
>
> Maybe, just for verification
>
> a) Check that the service offering is enabled ( :-) )
>
> b) that the "offering access" is configured correct
>
>
>
> Maybe just create a test-account und give this account direct access 
> to the offerings instead provide it public?
>
>
>
> regarding the log you provided: Guess it looks good under the hood.
>
> As written above, when the offering is not showing for a account it 
> was in my case normaly that the account had no acccess rights for the 
> offering.
>
>
>
> Am Fr., 8. Okt. 2021 um 10:13 Uhr schrieb <  cristian.c@istream.today> cristian.c@istream.today>:
>
>
>
> > Hi,
>
> >
>
> > Right now I have added a second Physical network for Guest 
> > traffic
>
> > with tag "DefaultB" and added "DefaultA" to the one which was 
> > already
>
> > present, I have created new 2 network offering, one for each tag.
>
> > When I try to create a new Guest network in 4.15.1, I see the 
> > offering
>
> > for tag Default A but not for B when I select the second physical
>
> > network, nothing visible, I deleted the network, offering, created
> again, same thing.
>
> >
>
> > "Found physical network id=203 based on requested tags DefaultB
>
> > 2021-10-08 08:12:32,178 DEBUG [c.c.a.ApiServlet]
>
> > (qtp182531396-17:ctx-ecf8c295 ctx-1af2acec) (logid:5a80a500) 
> > ===END===
>
> > 86.125.230.37 -- GET
>
> >
> zoneid=c3b5e5fa-c3e8-49f0-8094-573456a45c00=Enabled=DefaultB=Shared=listNetworkOfferings=json"
>
> >
>
> > I'm doing something wrong?
>
> >
>
> >
>
> > Regards,
>
> > Cristian
>
> >
>
> > -Original Message-
>
> > From:  <mailto:vas...@gmx.de> vas...@gmx.de < <mailto:vas...@gmx.de>
> vas...@gmx.de>
>
> > Sent: Thursday, October 7, 2021 6:37 PM
>
> > To:  <mailto:users@cloudstack.apache.org> 
> > users@cloudstack.apache.org
>
> > Subject: Re: Multiple Network labels - custom
>
> >
>
> > it should work , as i am am using this for providing some "special"
>
> > networks myself in my environment.
>
> >
>
> > maybe for a better understanding you can take a look at the 
> > following
>
> >  
> > <https://www.shapeblue.com/understanding-cloudstacks-physical-networ
> > kin>
> https://www.shapeblue.com/understanding-cloudstacks-physical-networkin
>
> > g-architecture/ and there the section "advanced network traffic".
>
> > there you'll find a diagramm of a scenario, where they provide an 
> > mpls
>
> > network for guest traffic.
>
> >
>
> > what i had done to achieve this is (it works but i don't know if 
> > this
>
> > is all best practice):
>
> >
>
> > WARNING: When introducing another physical network for e.g guest
>

RE: Multiple Network labels - custom

[cristian.c]

heck that the service offering is enabled ( :-) )

b) that the "offering access" is configured correct

 

Maybe just create a test-account und give this account direct access to the 
offerings instead provide it public?

 

regarding the log you provided: Guess it looks good under the hood.

As written above, when the offering is not showing for a account it was in my 
case normaly that the account had no acccess rights for the offering.

 

Am Fr., 8. Okt. 2021 um 10:13 Uhr schrieb < <mailto:cristian.c@istream.today> 
cristian.c@istream.today>:

 

> Hi,

> 

> Right now I have added a second Physical network for Guest traffic 

> with tag "DefaultB" and added "DefaultA" to the one which was already 

> present, I have created new 2 network offering, one for each tag.  

> When I try to create a new Guest network in 4.15.1, I see the offering 

> for tag Default A but not for B when I select the second physical 

> network, nothing visible, I deleted the network, offering, created again, 
> same thing.

> 

> "Found physical network id=203 based on requested tags DefaultB

> 2021-10-08 08:12:32,178 DEBUG [c.c.a.ApiServlet]

> (qtp182531396-17:ctx-ecf8c295 ctx-1af2acec) (logid:5a80a500) ===END===

> 86.125.230.37 -- GET

> zoneid=c3b5e5fa-c3e8-49f0-8094-573456a45c00=Enabled=DefaultB=Shared=listNetworkOfferings=json"

> 

> I'm doing something wrong?

> 

> 

> Regards,

> Cristian

> 

> -Original Message-

> From:  <mailto:vas...@gmx.de> vas...@gmx.de < <mailto:vas...@gmx.de> 
> vas...@gmx.de>

> Sent: Thursday, October 7, 2021 6:37 PM

> To:  <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org

> Subject: Re: Multiple Network labels - custom

> 

> it should work , as i am am using this for providing some "special"

> networks myself in my environment.

> 

> maybe for a better understanding you can take a look at the following 

>  <https://www.shapeblue.com/understanding-cloudstacks-physical-networkin> 
> https://www.shapeblue.com/understanding-cloudstacks-physical-networkin

> g-architecture/ and there the section "advanced network traffic". 

> there you'll find a diagramm of a scenario, where they provide an mpls 

> network for guest traffic.

> 

> what i had done to achieve this is (it works but i don't know if this 

> is all best practice):

> 

> WARNING: When introducing another physical network for e.g guest 

> traffic, the "default" network offerings won't work anymore. CS has no 

> default-allocation to an specified network - At least this is my experience.

> You will need to implement tags and create "custome" default network 

> offerings for further usage!

> 

> - create a new physical network in the zone

> - add traffic type "guest"

> - set the networklabel for matching purpose with the nics on the host

> - define tags for ALL physical networks (at least i needed to. if i am 

> correct if you start tagging, you will have to implement it for all 

> physical networks)

> - create 2 network offerings each using one of the tags of the 

> physical networks - traffic type guest

> 

> Then you can create networks, using the new network offerings, which 

> will use the "tagged" physical network --> use the the matching nics 

> on your hosts

> 

> 

> Am Do., 7. Okt. 2021 um 16:38 Uhr schrieb Cristian Ciobanu

> < <mailto:cristian.c@istream.today> cristian.c@istream.today>:

> 

> > Hi,

> >

> >In a much simpler way.

> >

> >I have 2 networks, 1 shared and 1 isolated, the problem i have 

> > here, both are using the same guest traffic label, because of this, 

> > I'm not able to use these on different labels/nics, both are using 

> > the same traffic type. Even if I add an additional physical network 

> > i will have only one type of guest traffic...

> >

> >I would like to specify a custom traffic type ( guest x) and use 

> > for specific network, shared or isolated.

> >

> >   I'm not sure if I can replicate this by using tags, will this work?

> >

> >

> > Regards

> > Cristian

> >

> > On Thu, Oct 7, 2021, 15:22  <mailto:vas...@gmx.de> vas...@gmx.de < 
> > <mailto:vas...@gmx.de> vas...@gmx.de> wrote:

> >

> >> just my thoughts.

> >>

> >> if i am understanding your intention correctly, you want to use a 

> >> dedicated physical network on the hosts  for "customized" guest 

> >> traffic, correct?

> >>

> >> You will need to add a "new" physical network 

RE: Multiple Network labels - custom

[cristian.c]

Hi,

Right now I have added a second Physical network for Guest traffic with tag 
"DefaultB" and added "DefaultA" to the one which was already present, I have 
created new 2 network offering, one for each tag.  When I try to create a new 
Guest network in 4.15.1, I see the offering for tag Default A but not for B 
when I select the second physical network, nothing visible, I deleted the 
network, offering, created again, same thing.

"Found physical network id=203 based on requested tags DefaultB
2021-10-08 08:12:32,178 DEBUG [c.c.a.ApiServlet] (qtp182531396-17:ctx-ecf8c295 
ctx-1af2acec) (logid:5a80a500) ===END===  86.125.230.37 -- GET  
zoneid=c3b5e5fa-c3e8-49f0-8094-573456a45c00=Enabled=DefaultB=Shared=listNetworkOfferings=json"

I'm doing something wrong?


Regards,
Cristian

-Original Message-
From: vas...@gmx.de  
Sent: Thursday, October 7, 2021 6:37 PM
To: users@cloudstack.apache.org
Subject: Re: Multiple Network labels - custom

it should work , as i am am using this for providing some "special"
networks myself in my environment.

maybe for a better understanding you can take a look at the following 
https://www.shapeblue.com/understanding-cloudstacks-physical-networking-architecture/
and there the section "advanced network traffic". there you'll find a diagramm 
of a scenario, where they provide an mpls network for guest traffic.

what i had done to achieve this is (it works but i don't know if this is all 
best practice):

WARNING: When introducing another physical network for e.g guest traffic, the 
"default" network offerings won't work anymore. CS has no default-allocation to 
an specified network - At least this is my experience.
You will need to implement tags and create "custome" default network offerings 
for further usage!

- create a new physical network in the zone
- add traffic type "guest"
- set the networklabel for matching purpose with the nics on the host
- define tags for ALL physical networks (at least i needed to. if i am correct 
if you start tagging, you will have to implement it for all physical networks)
- create 2 network offerings each using one of the tags of the physical 
networks - traffic type guest

Then you can create networks, using the new network offerings, which will use 
the "tagged" physical network --> use the the matching nics on your hosts


Am Do., 7. Okt. 2021 um 16:38 Uhr schrieb Cristian Ciobanu
:

> Hi,
>
>In a much simpler way.
>
>I have 2 networks, 1 shared and 1 isolated, the problem i have 
> here, both are using the same guest traffic label, because of this, 
> I'm not able to use these on different labels/nics, both are using the 
> same traffic type. Even if I add an additional physical network i will 
> have only one type of guest traffic...
>
>I would like to specify a custom traffic type ( guest x) and use 
> for specific network, shared or isolated.
>
>   I'm not sure if I can replicate this by using tags, will this work?
>
>
> Regards
> Cristian
>
> On Thu, Oct 7, 2021, 15:22 vas...@gmx.de  wrote:
>
>> just my thoughts.
>>
>> if i am understanding your intention correctly, you want to use a 
>> dedicated physical network on the hosts  for "customized" guest 
>> traffic, correct?
>>
>> You will need to add a "new" physical network to the zone with the 
>> networklabel, assaign the traffic type "guest" and start to use tags 
>> for the physical networks.
>> Afterwards you would need to implement a dedicated network service 
>> offering for this network - by using a tag to associate the network 
>> offering to the physical network.
>>
>> Your idea would currently not work cause the "physical network" in a 
>> zone is a 1:1 representation of the physical network on the hosts. 
>> afterwards you have some like  a 1:m (one CS physical network - many 
>> various traffic types possible) but not n:1 (many physical networks : 
>> one traffic type - even "worse" you would have different "flavours" 
>> of one traffic type).
>>
>> Maybe another way to display the relation (physical Network on host - 
>> phyical network in a zone - traffic type):
>> Currently used in CS:  1 - 1 - m
>>
>> Not supported in in CS: 1 - n - m
>>
>> what i understand you are looking for: 2 - 1 - 1 (while the traffic 
>> type guest would be segmentet into "default" and "custome")
>>
>> Hope that someone can imagine what i mean :-D
>>
>>
>>
>>
>> Am Do., 7. Okt. 2021 um 08:37 Uhr schrieb :
>>
>> > Hello,
>> >
>> >
>> >
>> > Is there a way to use multiple network labels for the same 
>> > network type?
>> > for example; I have Guest traffic with network label "vSwitch1, but 
>> > I
>> also
>> > want to have a vSwitch0 or anything else.  If this is not possible, 
>> > is there a way to create custom networks traffic types using the 
>> > same type of network offering but create under a different network.
>> >
>> > label?
>> >
>> >
>> >
>> >I want to have 2 traffic types for Guest, (Guest And 
>> > GuestCustom)  using the same network offering but 

RE: Multiple Network labels - custom

[cristian.c]

Hi,

Thank you for the provided information, I will test.


Regards,
Cristian

-Original Message-
From: vas...@gmx.de  
Sent: Thursday, October 7, 2021 6:37 PM
To: users@cloudstack.apache.org
Subject: Re: Multiple Network labels - custom

it should work , as i am am using this for providing some "special"
networks myself in my environment.

maybe for a better understanding you can take a look at the following 
https://www.shapeblue.com/understanding-cloudstacks-physical-networking-architecture/
and there the section "advanced network traffic". there you'll find a diagramm 
of a scenario, where they provide an mpls network for guest traffic.

what i had done to achieve this is (it works but i don't know if this is all 
best practice):

WARNING: When introducing another physical network for e.g guest traffic, the 
"default" network offerings won't work anymore. CS has no default-allocation to 
an specified network - At least this is my experience.
You will need to implement tags and create "custome" default network offerings 
for further usage!

- create a new physical network in the zone
- add traffic type "guest"
- set the networklabel for matching purpose with the nics on the host
- define tags for ALL physical networks (at least i needed to. if i am correct 
if you start tagging, you will have to implement it for all physical networks)
- create 2 network offerings each using one of the tags of the physical 
networks - traffic type guest

Then you can create networks, using the new network offerings, which will use 
the "tagged" physical network --> use the the matching nics on your hosts


Am Do., 7. Okt. 2021 um 16:38 Uhr schrieb Cristian Ciobanu
:

> Hi,
>
>In a much simpler way.
>
>I have 2 networks, 1 shared and 1 isolated, the problem i have 
> here, both are using the same guest traffic label, because of this, 
> I'm not able to use these on different labels/nics, both are using the 
> same traffic type. Even if I add an additional physical network i will 
> have only one type of guest traffic...
>
>I would like to specify a custom traffic type ( guest x) and use 
> for specific network, shared or isolated.
>
>   I'm not sure if I can replicate this by using tags, will this work?
>
>
> Regards
> Cristian
>
> On Thu, Oct 7, 2021, 15:22 vas...@gmx.de  wrote:
>
>> just my thoughts.
>>
>> if i am understanding your intention correctly, you want to use a 
>> dedicated physical network on the hosts  for "customized" guest 
>> traffic, correct?
>>
>> You will need to add a "new" physical network to the zone with the 
>> networklabel, assaign the traffic type "guest" and start to use tags 
>> for the physical networks.
>> Afterwards you would need to implement a dedicated network service 
>> offering for this network - by using a tag to associate the network 
>> offering to the physical network.
>>
>> Your idea would currently not work cause the "physical network" in a 
>> zone is a 1:1 representation of the physical network on the hosts. 
>> afterwards you have some like  a 1:m (one CS physical network - many 
>> various traffic types possible) but not n:1 (many physical networks : 
>> one traffic type - even "worse" you would have different "flavours" 
>> of one traffic type).
>>
>> Maybe another way to display the relation (physical Network on host - 
>> phyical network in a zone - traffic type):
>> Currently used in CS:  1 - 1 - m
>>
>> Not supported in in CS: 1 - n - m
>>
>> what i understand you are looking for: 2 - 1 - 1 (while the traffic 
>> type guest would be segmentet into "default" and "custome")
>>
>> Hope that someone can imagine what i mean :-D
>>
>>
>>
>>
>> Am Do., 7. Okt. 2021 um 08:37 Uhr schrieb :
>>
>> > Hello,
>> >
>> >
>> >
>> > Is there a way to use multiple network labels for the same 
>> > network type?
>> > for example; I have Guest traffic with network label "vSwitch1, but 
>> > I
>> also
>> > want to have a vSwitch0 or anything else.  If this is not possible, 
>> > is there a way to create custom networks traffic types using the 
>> > same type of network offering but create under a different network.
>> >
>> > label?
>> >
>> >
>> >
>> >I want to have 2 traffic types for Guest, (Guest And 
>> > GuestCustom)  using the same network offering but create the under 
>> > the different network
>> label.
>> > The idea is to have the possibility to create/duplicate same type 
>> > of traffic but deploy under different network label (vSwitchX).
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Regards,
>> >
>> > Cristian
>> >
>> >
>>
>

Multiple Network labels - custom

[cristian.c]

Hello,

 

Is there a way to use multiple network labels for the same network type?
for example; I have Guest traffic with network label "vSwitch1, but I also
want to have a vSwitch0 or anything else.  If this is not possible, is there
a way to create custom networks traffic types using the same type of network
offering but create under a different network.

label?

 

   I want to have 2 traffic types for Guest, (Guest And GuestCustom)   using
the same network offering but create the under the different network label.
The idea is to have the possibility to create/duplicate same type of traffic
but deploy under different network label (vSwitchX).

 

  

 

Regards,

Cristian

RE: Unable to read/process CSR: Command failed due to Exception: java.lang.Exception

[cristian.c]

Hi,

  Yes, it is the server, but I already started to check the BIOS ... and I 
have very interesting question, why with KVM is working?  What is OS different 
between KVM and VMware when comes to systemvm patching/inject keys? I think 
here I have the answer to may error.

Regards,
Cristian

-Original Message-
From: Daan Hoogland  
Sent: Tuesday, September 28, 2021 12:14 PM
To: users 
Subject: Re: Unable to read/process CSR: Command failed due to Exception: 
java.lang.Exception

ok, than I would like to add an idea to your "it's the server". Not saying you 
are wrong but location points to network i.e. firewall(s).
Hope you strike luck,

On Tue, Sep 28, 2021 at 10:58 AM  wrote:

> Hi,
>
> We already tested in all the ways is possible, this problem is 
> related to these specific servers, 100%.
>
>The last test we did, was to add the ESXI server to the zone we 
> already have and works, we disabled all hosts from that zone and 
> destroyed the SSVM to force a redeploy on this ESXI, of course, we 
> have ended with the same error.
>
>I want to mention that we have the exact same ESXI version and 
> hardware, for me looks like there is a BIOS/HW/DISK issue.
>
>
> FYI:
>
>"I did this in 3 different new Setup (new Management install, NFS, 
> etc), no matter if I add as first zone or second zone.
>
>" I have tested this with multiple Cloudstack versions, 4.15.0, 
> 4.15.1,
> 4.15.2 and VMware, 6.5 and with 6.7 different patches and with the 
> last patches.  I'm 100% that is related to these 2 servers, I do not 
> understand what is wrong with these servers, this is the problem.
>
> We have 2 identical servers' hardware/ESXI, the only difference is 
> the location, the not working, are in UK, and any other location we 
> have, DE, NYC, FR, CA, etc. works fine. But we tested with the exact 
> same servers which are from DE(Germany) works perfect, with any ACS or 
> VMware version (also same CIDR range/size for public and private), 
> when we test with UK, we get same error no matter what."
>
>I want to mention that we have these servers (UK, DE, FR, CA ) from 
> OVH in vRack  and the UK, are not working.  ( I manage these server 
> for more than 5 years )"
>
> Regards,
> Cristian
>
> -Original Message-
> From: Daan Hoogland 
> Sent: Tuesday, September 28, 2021 11:00 AM
> To: users 
> Subject: Re: Unable to read/process CSR: Command failed due to Exception:
> java.lang.Exception
>
> Christian, did you solve this (i just encountered your mail and see it 
> is yet unanswered) the problem is with the certificate generation, it 
> seems to try and generate from an empty string (the csr/certificate 
> signing request) Not being able to sign in to the SVM is a problem but 
> on vmware you would use a specific command as described in [1]. Did 
> you try `ssh -i /opt/xensource/bin/id_rsa --p 3922 root@privateIP 
>  OfTheHost`?
>
> [1]
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templat
> es%2C+Secondary+storage+troubleshooting
>
> On Thu, Sep 23, 2021 at 11:31 AM  wrote:
>
> > Hello,
> >
> >
> >
> >I'm not sure why I get this error (added new vmware zone), 
> > there are not permission issues for systemvms folder, and I do not 
> > see any error above this. If I try to login to login into SSVM/Proxy 
> > console via cloudstack management, I get this access denied.
> >
> >
> >
> >Any suggestion?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 021-09-23 05:22:15,012 ERROR [c.c.u.s.SshHelper] 
> > (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, 
> > job-34950/job-34972,
> > cmd: SetupKeyStoreCommand) (logid:f3cd2252) Failed to authentication 
> > SSH user root on host 10.15.0.160
> >
> > 2021-09-23 05:22:15,012 INFO  [c.c.h.v.u.VmwareHelper] 
> > (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, 
> > job-34950/job-34972,
> > cmd: SetupKeyStoreCommand) (logid:f3cd2252) [ignored]failed to get 
> > message for exception: Failed to authentication SSH user root on 
> > host
> > 10.15.0.160
> >
> > 2021-09-23 05:22:15,012 ERROR [c.c.h.v.r.VmwareResource] 
> > (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, 
> > job-34950/job-34972,
> > cmd: SetupKeyStoreCommand) (logid:f3cd2252) Command failed due to
> > Exception:
> > java.lang.Exception
> >
> > Message: Failed to authentication SSH user root on host 10.15.0.160
> >
> >
> >
> > 2021-09-23 05:22:15,012 DEBUG [c.c.h.v.r.VmwareResource] 
> > (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, 
> > job-34950/job-34972,
> > cmd: SetupKeyStoreCommand) (logid:f3cd2252) keystore-setup execution
> > result:
> > false
> >
> > 2021-09-23 05:22:15,014 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-283:ctx-b1e493bd) (logid:f3cd2252) Seq
> 40-1394708509601300577:
> > Response Received:
> >
> > 2021-09-23 05:22:15,016 DEBUG [c.c.a.t.Request]
> > (DirectAgent-283:ctx-b1e493bd) (logid:f3cd2252) Seq
> 40-1394708509601300577:
> > Processing:  { Ans: , MgmtId: 345049356158, via:
> > 

RE: Unable to read/process CSR: Command failed due to Exception: java.lang.Exception

[cristian.c]

Hi,

We already tested in all the ways is possible, this problem is related to 
these specific servers, 100%.

   The last test we did, was to add the ESXI server to the zone we already have 
and works, we disabled all hosts from that zone and destroyed the SSVM to force 
a redeploy on this ESXI, of course, we have ended with the same error.

   I want to mention that we have the exact same ESXI version and hardware, for 
me looks like there is a BIOS/HW/DISK issue.


FYI:

   "I did this in 3 different new Setup (new Management install, NFS, etc), no 
matter if I add as first zone or second zone.

   " I have tested this with multiple Cloudstack versions, 4.15.0, 4.15.1,
4.15.2 and VMware, 6.5 and with 6.7 different patches and with the last 
patches.  I'm 100% that is related to these 2 servers, I do not understand what 
is wrong with these servers, this is the problem.  
 
We have 2 identical servers' hardware/ESXI, the only difference is the 
location, the not working, are in UK, and any other location we have, DE, NYC, 
FR, CA, etc. works fine. But we tested with the exact same servers which are 
from DE(Germany) works perfect, with any ACS or VMware version (also same CIDR 
range/size for public and private), when we test with UK, we get same error no 
matter what."

   I want to mention that we have these servers (UK, DE, FR, CA ) from OVH in 
vRack  and the UK, are not working.  ( I manage these server for more than 5 
years )"

Regards,
Cristian

-Original Message-
From: Daan Hoogland  
Sent: Tuesday, September 28, 2021 11:00 AM
To: users 
Subject: Re: Unable to read/process CSR: Command failed due to Exception: 
java.lang.Exception

Christian, did you solve this (i just encountered your mail and see it is yet 
unanswered) the problem is with the certificate generation, it seems to try and 
generate from an empty string (the csr/certificate signing request) Not being 
able to sign in to the SVM is a problem but on vmware you would use a specific 
command as described in [1]. Did you try `ssh -i /opt/xensource/bin/id_rsa --p 
3922 root@privateIP  OfTheHost`?

[1]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templates%2C+Secondary+storage+troubleshooting

On Thu, Sep 23, 2021 at 11:31 AM  wrote:

> Hello,
>
>
>
>I'm not sure why I get this error (added new vmware zone), 
> there are not permission issues for systemvms folder, and I do not see 
> any error above this. If I try to login to login into SSVM/Proxy 
> console via cloudstack management, I get this access denied.
>
>
>
>Any suggestion?
>
>
>
>
>
>
>
>
>
> 021-09-23 05:22:15,012 ERROR [c.c.u.s.SshHelper] 
> (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, 
> job-34950/job-34972,
> cmd: SetupKeyStoreCommand) (logid:f3cd2252) Failed to authentication 
> SSH user root on host 10.15.0.160
>
> 2021-09-23 05:22:15,012 INFO  [c.c.h.v.u.VmwareHelper] 
> (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, 
> job-34950/job-34972,
> cmd: SetupKeyStoreCommand) (logid:f3cd2252) [ignored]failed to get 
> message for exception: Failed to authentication SSH user root on host 
> 10.15.0.160
>
> 2021-09-23 05:22:15,012 ERROR [c.c.h.v.r.VmwareResource] 
> (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, 
> job-34950/job-34972,
> cmd: SetupKeyStoreCommand) (logid:f3cd2252) Command failed due to
> Exception:
> java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.160
>
>
>
> 2021-09-23 05:22:15,012 DEBUG [c.c.h.v.r.VmwareResource] 
> (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, 
> job-34950/job-34972,
> cmd: SetupKeyStoreCommand) (logid:f3cd2252) keystore-setup execution
> result:
> false
>
> 2021-09-23 05:22:15,014 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-283:ctx-b1e493bd) (logid:f3cd2252) Seq 40-1394708509601300577:
> Response Received:
>
> 2021-09-23 05:22:15,016 DEBUG [c.c.a.t.Request]
> (DirectAgent-283:ctx-b1e493bd) (logid:f3cd2252) Seq 40-1394708509601300577:
> Processing:  { Ans: , MgmtId: 345049356158, via: 
> 40(lnd-uk-001.shape.host),
> Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.ca
> .SetupKeystoreAnswer":{"result":"true","wait":"0"
> ,"bypassHostMaintenance":"false"}}] }
>
> 2021-09-23 05:22:15,016 DEBUG [c.c.a.t.Request] 
> (Work-Job-Executor-14:ctx-10c822ed job-34950/job-34972 ctx-67aa0df0)
> (logid:f3cd2252) Seq 40-1394708509601300577: Received:  { Ans: , MgmtId:
> 345049356158, via: 40(lnd-uk-001.test.host), Ver: v1, Flags: 10, { 
> SetupKeystoreAnswer } }
>
> 2021-09-23 05:22:15,022 ERROR [c.c.v.VirtualMachineManagerImpl] 
> (Work-Job-Executor-14:ctx-10c822ed job-34950/job-34972 ctx-67aa0df0)
> (logid:f3cd2252) Retrying after catching exception while trying to 
> secure agent for systemvm id=1955
>
> com.cloud.utils.exception.CloudRuntimeException: Unable to 
> read/process
> CSR:
> Command failed due to Exception: java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.160
>
>
>
> at

RE: Failed to authentication SSH user root on host - repetitive error

[cristian.c]

Hello,

   I did this in 3 different new Setup (new Management install, NFS, etc),
no matter if I add as first zone or second zone.

   " I have tested this with multiple Cloudstack versions, 4.15.0, 4.15.1,
4.15.2 and VMware, 6.5 and with 6.7 different patches and with the last
patches.  I'm 100% that is related to these 2 servers, I do not understand
what is wrong with these servers, this is the problem.  
 
We have 2 identical servers' hardware/ESXI, the only difference is the
location, the not working, are in UK, and any other location we have, DE,
NYC, FR, CA, etc. works fine. But we tested with the exact same servers
which are from DE(Germany) works perfect, with any ACS or VMware version
(also same CIDR range/size for public and private), when we test with UK, we
get same error no matter what."

   I want to mention that we have these servers (UK, DE, FR, CA ) from OVH
in vRack  and the UK, are not working.  ( I manage these server for more
than 5 years )


Regards,
Cristian

-Original Message-
From: Harikrishna Patnala  
Sent: Monday, September 27, 2021 1:29 PM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error

Hi Crisitian,

To my understanding, you are trying to add a new zone and getting issues
with the system VM having some SSH failures.

May I ask you to start everything fresh?

I mean to prepare new secondary storage, seed the system VM template into
new secondary storage and then deploy the zone from the beginning of the
zone creation wizard. Make sure your management server has the latest
packages and systemvm.iso is also as expected to the CloudStack version you
are trying.

Thanks,
Harikrishna




From: Cristian Ciobanu 
Sent: Friday, September 24, 2021 5:19 PM
To: users@cloudstack.apache.org 
Subject: Re: Failed to authentication SSH user root on host - repetitive
error

Hi Pearl,


   I did upgrades multiple times, I always add the template before upgrade.
If was something related to this, i was expecting not to work for all zones,
but right now is working for 2 zones from the same setup, environment.

Regards,
Cristian

On Fri, Sep 24, 2021, 14:40 Pearl d'Silva 
wrote:

> Hi Cristian,
>
> After going through your earlier mails again, and correct me if I am 
> wrong, but I noticed in one of your mails you mentioned that the env 
> that's causing issues was upgraded from a lower version, right? So, 
> ideally when doing upgrades, prior to initiating the upgrade we first 
> take a backup of the DB,  register the template (new systemVM 
> template) following this 
> https://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade
> -4.15.html#update-system-vm-templates
> and then upgrade the environment. However, you mentioned that you used 
> the cloud-install-sys-tmplt script to seed and register the template. 
> I am not familiar if using that script during upgrades works (it may, 
> but I haven't ever used that approach).
> If it is possible, it would be advisable to rollback your env to the 
> earlier version (disable your zone, destroy the systemVMs before that) 
> and then follow the upgrade steps as mentioned in the documentation:
> https://docs.cloudstack.apache.org/en/latest/upgrading/index.html
>
> Thanks,
> Pearl
>
>
> 
> From: cristian.c@istream.today 
> Sent: Friday, September 24, 2021 3:08 PM
> To: users@cloudstack.apache.org 
> Subject: RE: Failed to authentication SSH user root on host - 
> repetitive error
>
> Hi Pearl,
>
> I just did a SSVM redeploy in the same environment, but different zone 
> and and I see the following differences :
>
> "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)"
>
>
> Working zone - https://pastebin.com/raw/88N6Fg6z
>
> "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)
>
>  
> datastore-6588","managed":"false","storagePort":"0","volumeSize":"(2.4
> 4
> GB)
>
> 021-09-24 05:22:23,959 INFO  [c.c.h.v.r.VmwareResource] 
> (DirectAgent-437:ctx-2f1519ba fra-de-001.test.host, 
> job-25099/job-35371,
> cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not 
> the same as the cloned VM datastore UUID: fra-de-001 -
> 9d679070c7a6441592646bfbb82ef1b3
>
>
>
> Not working zone - https://pastebin.com/raw/Um2uHgWM
>
> "isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes)"here is 0
> bytes...
>
>  datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0
> bytes)
> here is 0 

RE: SSCM and CSVM status starting till abort / libvirt: invalid connection pointer in virConnectGetVersion

[cristian.c]

Hi,

   First, enable the debug mode in cloudstack agent and restart the agent.


Regards,
Cristian

-Original Message-
From: vas...@gmx.de  
Sent: Friday, September 24, 2021 12:07 PM
To: users@cloudstack.apache.org
Subject: SSCM and CSVM status starting till abort / libvirt: invalid connection 
pointer in virConnectGetVersion

Hi,

i am running into problems within my new setup - and i currently don't know 
where to look for to debug this.
Environment:
Newly installed
CS 4.15.2
Ubuntu 20.04 + KVM
1x Management Server + 1 Host
Deploying Advanced Networking Zone

Up to this point everything is running fine. Didn't found errors in the log 
files.
However: When enabling the zone the SSVM and CSVM are changeing to started but 
never reach "running".

Taking a look into the agent.log i stumbled upon this ("..." =  repetion of
entries):

2021-09-24 01:01:55,311 INFO  [utils.nio.NioClient] (main:null) (logid:)
SSL: Handshake done
2021-09-24 01:01:55,311 INFO  [utils.nio.NioClient] (main:null) (logid:) 
Connected to 172.17.1.2:8250
2021-09-24 01:01:55,373 INFO  [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) (logid:) Attempting to create storage pool 
8362564e-11fd-4dd8-bae1-f90d53e52dab (Filesystem) in libvirt
2021-09-24 01:01:55,379 ERROR [kvm.resource.LibvirtConnection]
(Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid 
connection pointer in virConnectGetVersion
2021-09-24 01:01:55,381 WARN  [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) (logid:) Storage pool 
8362564e-11fd-4dd8-bae1-f90d53e52dab was not found running in libvirt. Need to 
create it.
2021-09-24 01:01:55,381 INFO  [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) (logid:) Didn't find an existing storage pool 
8362564e-11fd-4dd8-bae1-f90d53e52dab by UUID, checking for pools with duplicate 
paths
2021-09-24 01:01:55,383 INFO  [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) (logid:) Trying to fetch storage pool 
8362564e-11fd-4dd8-bae1-f90d53e52dab from libvirt
2021-09-24 01:01:55,424 INFO  [cloud.serializer.GsonHelper]
(Agent-Handler-1:null) (logid:) Default Builder inited.
2021-09-24 01:01:55,470 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Proccess agent startup answer, agent id = 0
2021-09-24 01:01:55,471 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Set agent id 0
2021-09-24 01:01:55,476 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Startup Response Received: agent id = 0
2021-09-24 01:01:55,701 INFO  [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:9e52178e) Processing agent ready command, 
agent id = 1
2021-09-24 01:01:55,702 INFO  [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:9e52178e) Set agent id 1
2021-09-24 01:01:55,707 INFO  [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:9e52178e) Ready command is processed for 
agent id = 1
2021-09-24 01:01:55,807 INFO  [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:9e52178e) Processing agent ready command, 
agent id = 1
2021-09-24 01:01:55,807 INFO  [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:9e52178e) Set agent id 1
2021-09-24 01:01:55,808 INFO  [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:9e52178e) Processed new management server 
list: 172.17.1.2@static
2021-09-24 01:01:55,809 INFO  [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:9e52178e) Ready command is processed for 
agent id = 1
2021-09-24 01:02:24,490 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:b81c6af6) Attempting to create storage 
pool 8bc1feaa-a099-3590-aa2b-2d12df0e242b (NetworkFilesystem) in libvirt
2021-09-24 01:02:24,510 WARN  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:b81c6af6) Storage pool 
8bc1feaa-a099-3590-aa2b-2d12df0e242b was not found running in libvirt. Need to 
create it.
2021-09-24 01:02:24,510 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:b81c6af6) Didn't find an existing storage 
pool 8bc1feaa-a099-3590-aa2b-2d12df0e242b by UUID, checking for pools with 
duplicate paths
2021-09-24 01:02:24,708 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:b81c6af6) Trying to fetch storage pool 
8bc1feaa-a099-3590-aa2b-2d12df0e242b from libvirt
2021-09-24 01:02:40,975 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:null) (logid:74acba6b) Trying to fetch storage pool 
8bc1feaa-a099-3590-aa2b-2d12df0e242b from libvirt
2021-09-24 01:02:41,001 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:null) (logid:74acba6b) Asking libvirt to refresh 
storage pool 8bc1feaa-a099-3590-aa2b-2d12df0e242b
...
2021-09-24 01:08:41,382 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:94789a51) Trying to fetch storage pool 
8bc1feaa-a099-3590-aa2b-2d12df0e242b from libvirt
2021-09-24 01:08:41,405 INFO  [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:94789a51) 

RE: Failed to authentication SSH user root on host - repetitive error

[cristian.c]

Hi Pearl,

I just did a SSVM redeploy in the same environment, but different zone and
and I see the following differences :

"isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)"


Working zone - https://pastebin.com/raw/88N6Fg6z   

"isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)

 datastore-6588","managed":"false","storagePort":"0","volumeSize":"(2.44 GB)

021-09-24 05:22:23,959 INFO  [c.c.h.v.r.VmwareResource]
(DirectAgent-437:ctx-2f1519ba fra-de-001.test.host, job-25099/job-35371,
cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the
same as the cloned VM datastore UUID: fra-de-001 -
9d679070c7a6441592646bfbb82ef1b3



Not working zone - https://pastebin.com/raw/Um2uHgWM  

"isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes)"here is 0
bytes...
 
 datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0 bytes)
here is 0 bytes...

2021-09-24 05:00:55,594 INFO  [c.c.h.v.r.VmwareResource]
(DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd:
StartCommand) (logid:d442432d) Mapped disk size is not the same as the
cloned VM disk size: 0 - 262144( here I see this line, with disk
size 0 )

2021-09-24 05:00:55,594 INFO  [c.c.h.v.r.VmwareResource]
(DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd:
StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the same as
the cloned VM datastore UUID: lnd-uk-002 - fb7b551a584e4b5192b8f371af56b303



I do not understand why I have here the size "0"


Regards,
Cristian

-Original Message-
From: Pearl d'Silva  
Sent: Friday, September 24, 2021 9:44 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error

By any chance, are there any old systemvm.isos in the secondary store, if
yes, can you please delete them. Based on the information you've provided,
it seems like, the key has been injected into systemvm.iso, but during boot
up, a script (cloud-early-config) that sets up the VM before bootstrapping /
patching isn't copying the auth key.
Can you try the steps mentioned under the VMWare section of the doc:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso
and see if it helps.
SystemVm.iso - Apache Cloudstack - Apache Software
Foundation<https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.i
so>
Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated
differently for different hypervisors. It comes packaged as part of the
Cloudstack rpm or is built on dev environments with the command "mvn clean
install -P developer,systemvm" (More info on building cloudstack) .Once the
iso is at the appropriate location depending on the hypervisor it is
inserted as a cd drive and the ...
cwiki.apache.org
Thanks,
Pearl


From: cristian.c@istream.today 
Sent: Friday, September 24, 2021 11:45 AM
To: users@cloudstack.apache.org 
Subject: RE: Failed to authentication SSH user root on host - repetitive
error

Hi Pearl,

  I already did a check via console into the secondary storage vm, I saw
that there are not keys in authorized_keys.

 If I try to inject, I see this : /bin/bash
/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
/var/cloudstack/management/.ssh/id_rsa.pub
/var/cloudstack/management/.ssh/id_rsa
/usr/share/cloudstack-common/vms/systemvm.iso
mount: /dev/loop0 is write-protected, mounting read-only New public key is
the same as the one in the systemvm.iso, not injecting it, not modifying
systemvm.iso


[root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls agent.zip
authorized_keys  cloud-scripts.tgz [root@cloud-emea iso]# cat
authorized_keys ssh-rsa
B3NzaC1yc2EBIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl
BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+x
xx+Bd1Y2U1Tx
xx+lYee
+zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ
+26AP
IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
cl...@cloud-emea.test.host[root@cloud-emea iso]#

is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub

(For security reason I have replaced characters from ssh pub key)


  I'm stuck here, I do not understand the logic.


Regards,
Cristian

 


-Original Message-
From: Pearl d'Silva 
Sent: Friday, September 24, 2021 8:05 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error

Hi Cristian,

The exception you are 

RE: Failed to authentication SSH user root on host - repetitive error

[cristian.c]

Hi Wei,

   I think here is the problem, when the key are injected, the SSVM is not 
running yet, is starting few seconds later So, the answer is that I do not 
have what to STOP, I can only delete the SSVM instance from VMware side..

Thank you,
Cristian

-Original Message-
From: Wei ZHOU  
Sent: Friday, September 24, 2021 9:43 AM
To: users 
Subject: Re: Failed to authentication SSH user root on host - repetitive error

Hi Cristian,

When the new public key has been injected into systemvm.iso, please stop/start 
or destroy SSVM.

-Wei

On Fri, 24 Sept 2021 at 08:16,  wrote:

> Hi Pearl,
>
>   I already did a check via console into the secondary storage vm, 
> I saw that there are not keys in authorized_keys.
>
>  If I try to inject, I see this : /bin/bash 
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/cloudstack/management/.ssh/id_rsa.pub
> /var/cloudstack/management/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso
> mount: /dev/loop0 is write-protected, mounting read-only New public 
> key is the same as the one in the systemvm.iso, not injecting it, not 
> modifying systemvm.iso
>
>
> [root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls 
> agent.zip  authorized_keys  cloud-scripts.tgz [root@cloud-emea iso]# 
> cat authorized_keys ssh-rsa
>
> B3NzaC1yc2EBIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZh
> cPueIl
>
> BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxx
> xx
>
> xx+Bd1Y2U1
> xx+TxlYee
>
> +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIM
> +tZ26AP
> IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
> cl...@cloud-emea.test.host[root@cloud-emea iso]#
>
> is the same key as I have in : 
> /var/cloudstack/management/.ssh/id_rsa.pub
>
> (For security reason I have replaced characters from ssh pub key)
>
>
>   I'm stuck here, I do not understand the logic.
>
>
> Regards,
> Cristian
>
> -Original Message-
> From: Pearl d'Silva 
> Sent: Friday, September 24, 2021 8:05 AM
> To: users@cloudstack.apache.org
> Subject: Re: Failed to authentication SSH user root on host - 
> repetitive error
>
> Hi Cristian,
>
> The exception you are seeing is most likely to happen if the 
> systemvm.iso hasn't been injected with the Management server's key. 
> One way to validate it, would be to go to your secondary store - I 
> presume you are working on a VMware setup - mount the 
> systemvm-4.15.2.iso and verify the authorized_keys data with what's 
> present on the MS at /var/cloudstack/management/.ssh/id_rsa.pub.
>
> Thanks,
> Pearl
>
> 
> From: cristian.c@istream.today 
> Sent: Thursday, September 23, 2021 9:50 PM
> To: users@cloudstack.apache.org 
> Subject: Failed to authentication SSH user root on host - repetitive 
> error
>
> Hello,
>
>
>
>
>
>   I was not able to fix this error "Failed to authentication SSH" 
> and looks like a bug for me, I will explain here why.
>
>
>
>   My setup before adding new ZONE:
>
>
>
> 1.  Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone.  ( This
> is
> an old setup, it was upgraded frequently, I think from 4.2 )
> 2.  I add new VMware zone, but this one with Advanced Networking, I end
> with this repetitive error and nothing else.
> 3.  I decide to add a new zone, the same as I have at (2) with Advanced
> Networking, everything works perfect, no issue at systemvm deploy, 
> routers, instances.
> 4.  I delete the zone from point (2)
> 5.  I add again the same zone, end with the same error.
>
>
>
>I have tried multiple things, delete the template, add again, 
> inject, nothing works, I end with the same error, over and over, and I 
> do not see the why.
>
>
>
>
>
> 021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
> (DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, 
> job-35231/job-35263,
> cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to
> Exception:
> java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.204
>
>
>
> 2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl] 
> (Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
> (logid:df4131be) Retrying after catching exception while trying to 
> secure agent for systemvm id=2025
>
> com.cloud.utils.exception.CloudRuntimeException: Unable to 
> read/process
> CSR:
> Command failed due to Exception: java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.204
>
>
>
>
>
>   Here is the full log : https://pastebin.com/raw/fFfmquVB
>
>
>
>
>
>   Maybe someone have a hint for me .
>
>
>
> Regards,
>
> Cristian
>
>
>
>
>
>
>
>
>
>

RE: Failed to authentication SSH user root on host - repetitive error

[cristian.c]

Hi Pearl,

   I did the following multiple times:

1. remove systemvm.iso from secondary storage
2. remove vmdk template from ESXI host and from database spool ref.
3. I have deleted the old systemvm.iso.bak
4 restarted the MS

I didn't tried to change the minor version, how can I do this? In DB?
   
Is there a problem with this template?
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmp
lt -m /var/cloudstack/mnt/345049356158.7cfbc14b -u
http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/systemvmtemplate/4.
15/systemvmtemplate-4.15.1-vmware.ova -h vmware -F

Thank you,
Cristian

-Original Message-
From: Pearl d'Silva  
Sent: Friday, September 24, 2021 9:44 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error

By any chance, are there any old systemvm.isos in the secondary store, if
yes, can you please delete them. Based on the information you've provided,
it seems like, the key has been injected into systemvm.iso, but during boot
up, a script (cloud-early-config) that sets up the VM before bootstrapping /
patching isn't copying the auth key.
Can you try the steps mentioned under the VMWare section of the doc:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso
and see if it helps.
SystemVm.iso - Apache Cloudstack - Apache Software
Foundation<https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.i
so>
Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated
differently for different hypervisors. It comes packaged as part of the
Cloudstack rpm or is built on dev environments with the command "mvn clean
install -P developer,systemvm" (More info on building cloudstack) .Once the
iso is at the appropriate location depending on the hypervisor it is
inserted as a cd drive and the ...
cwiki.apache.org
Thanks,
Pearl

____
From: cristian.c@istream.today 
Sent: Friday, September 24, 2021 11:45 AM
To: users@cloudstack.apache.org 
Subject: RE: Failed to authentication SSH user root on host - repetitive
error

Hi Pearl,

  I already did a check via console into the secondary storage vm, I saw
that there are not keys in authorized_keys.

 If I try to inject, I see this : /bin/bash
/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
/var/cloudstack/management/.ssh/id_rsa.pub
/var/cloudstack/management/.ssh/id_rsa
/usr/share/cloudstack-common/vms/systemvm.iso
mount: /dev/loop0 is write-protected, mounting read-only New public key is
the same as the one in the systemvm.iso, not injecting it, not modifying
systemvm.iso


[root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls agent.zip
authorized_keys  cloud-scripts.tgz [root@cloud-emea iso]# cat
authorized_keys ssh-rsa
B3NzaC1yc2EBIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl
BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+x
xx+Bd1Y2U1Tx
xx+lYee
+zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ
+26AP
IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
cl...@cloud-emea.test.host[root@cloud-emea iso]#

is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub

(For security reason I have replaced characters from ssh pub key)


  I'm stuck here, I do not understand the logic.


Regards,
Cristian

 


-Original Message-
From: Pearl d'Silva 
Sent: Friday, September 24, 2021 8:05 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error

Hi Cristian,

The exception you are seeing is most likely to happen if the systemvm.iso
hasn't been injected with the Management server's key. One way to validate
it, would be to go to your secondary store - I presume you are working on a
VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys
data with what's present on the MS at
/var/cloudstack/management/.ssh/id_rsa.pub.

Thanks,
Pearl

____
From: cristian.c@istream.today 
Sent: Thursday, September 23, 2021 9:50 PM
To: users@cloudstack.apache.org 
Subject: Failed to authentication SSH user root on host - repetitive error

Hello,





  I was not able to fix this error "Failed to authentication SSH" and
looks like a bug for me, I will explain here why.



  My setup before adding new ZONE:



1.  Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone.  ( This is
an old setup, it was upgraded frequently, I think from 4.2 )
2.  I add new VMware zone, but this one with Advanced Networking, I end
with this repetitive error and nothing else.
3.  I decide to add a new zone, the same as I have at (2) with Advanced
Networking, everything works perfect, no issue at systemvm deploy, routers,
instances.
4.  I 

RE: Failed to authentication SSH user root on host - repetitive error

[cristian.c]

Hi Pearl,

  I already did a check via console into the secondary storage vm, I saw
that there are not keys in authorized_keys.

 If I try to inject, I see this : /bin/bash
/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
/var/cloudstack/management/.ssh/id_rsa.pub
/var/cloudstack/management/.ssh/id_rsa
/usr/share/cloudstack-common/vms/systemvm.iso
mount: /dev/loop0 is write-protected, mounting read-only
New public key is the same as the one in the systemvm.iso, not injecting it,
not modifying systemvm.iso 


[root@cloud-emea systemvm]# cd /mnt/iso/
[root@cloud-emea iso]# ls
agent.zip  authorized_keys  cloud-scripts.tgz
[root@cloud-emea iso]# cat authorized_keys
ssh-rsa
B3NzaC1yc2EBIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl
BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+x
xx+Bd1Y2U1TxlYee
+zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ26AP
IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
cl...@cloud-emea.test.host[root@cloud-emea iso]#

is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub

(For security reason I have replaced characters from ssh pub key)


  I'm stuck here, I do not understand the logic.


Regards,
Cristian

-Original Message-
From: Pearl d'Silva  
Sent: Friday, September 24, 2021 8:05 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error

Hi Cristian,

The exception you are seeing is most likely to happen if the systemvm.iso
hasn't been injected with the Management server's key. One way to validate
it, would be to go to your secondary store - I presume you are working on a
VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys
data with what's present on the MS at
/var/cloudstack/management/.ssh/id_rsa.pub.

Thanks,
Pearl


From: cristian.c@istream.today 
Sent: Thursday, September 23, 2021 9:50 PM
To: users@cloudstack.apache.org 
Subject: Failed to authentication SSH user root on host - repetitive error

Hello,





  I was not able to fix this error "Failed to authentication SSH" and
looks like a bug for me, I will explain here why.



  My setup before adding new ZONE:



1.  Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone.  ( This is
an old setup, it was upgraded frequently, I think from 4.2 )
2.  I add new VMware zone, but this one with Advanced Networking, I end
with this repetitive error and nothing else.
3.  I decide to add a new zone, the same as I have at (2) with Advanced
Networking, everything works perfect, no issue at systemvm deploy, routers,
instances.
4.  I delete the zone from point (2)
5.  I add again the same zone, end with the same error.



   I have tried multiple things, delete the template, add again, inject,
nothing works, I end with the same error, over and over, and I do not see
the why.





021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263,
cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to Exception:
java.lang.Exception

Message: Failed to authentication SSH user root on host 10.15.0.204



2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
(logid:df4131be) Retrying after catching exception while trying to secure
agent for systemvm id=2025

com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR:
Command failed due to Exception: java.lang.Exception

Message: Failed to authentication SSH user root on host 10.15.0.204





  Here is the full log : https://pastebin.com/raw/fFfmquVB





  Maybe someone have a hint for me .



Regards,

Cristian






 

Failed to authentication SSH user root on host - repetitive error

[cristian.c]

Hello,

 

 

  I was not able to fix this error "Failed to authentication SSH" and
looks like a bug for me, I will explain here why.

 

  My setup before adding new ZONE:

  

1.  Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone.  ( This is
an old setup, it was upgraded frequently, I think from 4.2 )
2.  I add new VMware zone, but this one with Advanced Networking, I end
with this repetitive error and nothing else.
3.  I decide to add a new zone, the same as I have at (2) with Advanced
Networking, everything works perfect, no issue at systemvm deploy, routers,
instances.
4.  I delete the zone from point (2)
5.  I add again the same zone, end with the same error.

 

   I have tried multiple things, delete the template, add again, inject,
nothing works, I end with the same error, over and over, and I do not see
the why.

 

 

021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263,
cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to Exception:
java.lang.Exception

Message: Failed to authentication SSH user root on host 10.15.0.204

 

2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
(logid:df4131be) Retrying after catching exception while trying to secure
agent for systemvm id=2025

com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR:
Command failed due to Exception: java.lang.Exception

Message: Failed to authentication SSH user root on host 10.15.0.204

 

 

  Here is the full log : https://pastebin.com/raw/fFfmquVB

 

 

  Maybe someone have a hint for me .

 

Regards,

Cristian

 

   

Unable to read/process CSR: Command failed due to Exception: java.lang.Exception

[cristian.c]

Hello,

 

   I'm not sure why I get this error (added new vmware zone), there are
not permission issues for systemvms folder, and I do not see any error above
this. If I try to login to login into SSVM/Proxy console via cloudstack
management, I get this access denied.

 

   Any suggestion? 

 

 

 

 

021-09-23 05:22:15,012 ERROR [c.c.u.s.SshHelper]
(DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, job-34950/job-34972,
cmd: SetupKeyStoreCommand) (logid:f3cd2252) Failed to authentication SSH
user root on host 10.15.0.160

2021-09-23 05:22:15,012 INFO  [c.c.h.v.u.VmwareHelper]
(DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, job-34950/job-34972,
cmd: SetupKeyStoreCommand) (logid:f3cd2252) [ignored]failed to get message
for exception: Failed to authentication SSH user root on host 10.15.0.160

2021-09-23 05:22:15,012 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, job-34950/job-34972,
cmd: SetupKeyStoreCommand) (logid:f3cd2252) Command failed due to Exception:
java.lang.Exception

Message: Failed to authentication SSH user root on host 10.15.0.160

 

2021-09-23 05:22:15,012 DEBUG [c.c.h.v.r.VmwareResource]
(DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, job-34950/job-34972,
cmd: SetupKeyStoreCommand) (logid:f3cd2252) keystore-setup execution result:
false

2021-09-23 05:22:15,014 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-283:ctx-b1e493bd) (logid:f3cd2252) Seq 40-1394708509601300577:
Response Received:

2021-09-23 05:22:15,016 DEBUG [c.c.a.t.Request]
(DirectAgent-283:ctx-b1e493bd) (logid:f3cd2252) Seq 40-1394708509601300577:
Processing:  { Ans: , MgmtId: 345049356158, via: 40(lnd-uk-001.shape.host),
Ver: v1, Flags: 10,
[{"org.apache.cloudstack.ca.SetupKeystoreAnswer":{"result":"true","wait":"0"
,"bypassHostMaintenance":"false"}}] }

2021-09-23 05:22:15,016 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-14:ctx-10c822ed job-34950/job-34972 ctx-67aa0df0)
(logid:f3cd2252) Seq 40-1394708509601300577: Received:  { Ans: , MgmtId:
345049356158, via: 40(lnd-uk-001.test.host), Ver: v1, Flags: 10, {
SetupKeystoreAnswer } }

2021-09-23 05:22:15,022 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-14:ctx-10c822ed job-34950/job-34972 ctx-67aa0df0)
(logid:f3cd2252) Retrying after catching exception while trying to secure
agent for systemvm id=1955

com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR:
Command failed due to Exception: java.lang.Exception

Message: Failed to authentication SSH user root on host 10.15.0.160

 

at
org.apache.cloudstack.ca.provider.RootCAProvider.generateCertificateUsingCsr
(RootCAProvider.java:170)

at
org.apache.cloudstack.ca.provider.RootCAProvider.issueCertificate(RootCAProv
ider.java:230)

at
org.apache.cloudstack.ca.CAManagerImpl.issueCertificate(CAManagerImpl.java:1
64)

at jdk.internal.reflect.GeneratedMethodAccessor318.invoke(Unknown
Source)

at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Delegatin
gMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)

at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:344)

at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:198)

at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:163)

at
org.apache.cloudstack.network.contrail.management.EventUtils$EventIntercepto
r.invoke(EventUtils.java:107)

at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:175)

at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51
)

at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:175)

at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:97)

at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:186)

at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:215)

at com.sun.proxy.$Proxy292.issueCertificate(Unknown Source)

at
com.cloud.vm.VirtualMachineManagerImpl.setupAgentSecurity(VirtualMachineMana
gerImpl.java:970)

at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage
rImpl.java:1241)

at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage
rImpl.java:5502)

at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)

at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodA
ccessorImpl.java:62)

at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Delegatin
gMethodAccessorImpl.java:43)

at 

RE: The RAM MEM inside KVM instance is not the same as it was set

[cristian.c]

HI Wei,

Thank you for the info.

Is there a way to do overprovisioning like we do with VMware?   If I have 
an over provisioning of 1.5x or 2x, is there a way not to show the min value 
inside the instance, and show the max?

Regards,
Cristian

-Original Message-
From: Wei ZHOU  
Sent: Wednesday, September 22, 2021 1:33 PM
To: users 
Subject: Re: The RAM MEM inside KVM instance is not the same as it was set

Hi Cristian,

4.15 branch only accepts bug fixes. I think it is not a bug but an enhancement.

-Wei

On Wed, 22 Sept 2021 at 12:19,  wrote:

> Hi,
>
> is there a way to apply this to 4.15.2 ?
> https://github.com/apache/cloudstack/commit/6531ee587198ca2789ab0f53e9
> f3c3dccdd009d4
>
>
>
> Regards,
> Cristian
>
> -Original Message-
> From: Daniel Augusto Veronezi Salvador 
> Sent: Friday, September 17, 2021 7:55 PM
> To: users@cloudstack.apache.org
> Subject: Re: The RAM MEM inside KVM instance is not the same as it was 
> set
>
> Hi, Cristian,
>
> When ACS is deploying a VM, it defines a minimum memory|CPU to the VM, 
> based on a calculation through the overprovisioning (defined mem|cpu 
> speed / overprovisioning factor); Therefore, it will define a range of
> memory|CPU, even if it does not use scalable (dynamic) service offerings.
>
> For now, the option to avoid this behavior is to keep memory 
> overprovisioning as "1", as Wei suggested. However, in version 4.16.0, 
> there'll be two new global settings 
> ("vm.min.memory.equals.memory.divided.by.mem.overprovisioning.factor" 
> and
> "vm.min.cpu.speed.equals.cpu.speed.divided.by.cpu.overprovisioning.fac
> tor")
> to avoid it.
>
> This is the PR which introduced it:
> https://github.com/apache/cloudstack/pull/4722
>
> Best regards,
> Daniel
>
> On 2021/09/17 15:07:55, Wei ZHOU  wrote:
> > For kvm environments, I suggest to set the value to 1 (in global 
> > setting and cluster setting).
> >
> > -Wei
> >
> >
> > On Fri, 17 Sept 2021 at 16:19,  wrote:
> >
> > > Hi,
> > >
> > >   Yes, I have found that this is why i have this value.
> > >
> > >
> > > Regards,
> > > Cristian
> > >
> > > -Original Message-
> > > From: Wei ZHOU 
> > > Sent: Friday, September 17, 2021 3:34 PM
> > > To: users 
> > > Subject: Re: The RAM MEM inside KVM instance is not the same as it 
> > > was set
> > >
> > > Hi,
> > >
> > > Did you set mem.overprovisioning.factor in global setting or 
> > > cluster setting ?
> > >
> > > -Wei
> > >
> > > On Fri, 17 Sept 2021 at 13:47,  wrote:
> > >
> > > > Hello,
> > > >
> > > >
> > > >
> > > >From where do I get this minRam value?   I used a compute
> offering
> > > > with 10GB of RAM, if I check the VM I see 4GB Ram.
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> "[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":"4156","name":"i-2-4156-VM"
> > > >
> > > > ,"state":"Starting","type":"User","cpus":"4","minSpeed":"760","m
> > > > ax
> > > > Spee
> > > > d":"19
> > > > 00","minRam":"(4.00 GB) 4294967296","maxRam":"(10.00 GB) 
> > > > 10737418240","arch":"x86_64","os":"CentOS
> > > > 7","platformEmulator":"CentOS
> > > >
> > > > 7","bootArgs":"","enableHA":"false","limitCpuUse":"false","enabl
> > > > eD
> > > > ynam
> > > > ically
> > > >
> > > >
> ScaleVm":"false","vncPassword":"NutaRZIRWThdE3aU_IkJAA","vncAddr":"10.
> > > > 100.22
> > > > .10""
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Regards,
> > > >
> > > > Cristian
> > > >
> > > >
> > >
> > >
> >
>
>

RE: The RAM MEM inside KVM instance is not the same as it was set

[cristian.c]

Hi,

is there a way to apply this to 4.15.2 
?https://github.com/apache/cloudstack/commit/6531ee587198ca2789ab0f53e9f3c3dccdd009d4

  

Regards,
Cristian

-Original Message-
From: Daniel Augusto Veronezi Salvador  
Sent: Friday, September 17, 2021 7:55 PM
To: users@cloudstack.apache.org
Subject: Re: The RAM MEM inside KVM instance is not the same as it was set

Hi, Cristian,

When ACS is deploying a VM, it defines a minimum memory|CPU to the VM, based on 
a calculation through the overprovisioning (defined mem|cpu speed / 
overprovisioning factor); Therefore, it will define a range of memory|CPU, even 
if it does not use scalable (dynamic) service offerings.

For now, the option to avoid this behavior is to keep memory overprovisioning 
as "1", as Wei suggested. However, in version 4.16.0, there'll be two new 
global settings 
("vm.min.memory.equals.memory.divided.by.mem.overprovisioning.factor" and 
"vm.min.cpu.speed.equals.cpu.speed.divided.by.cpu.overprovisioning.factor") to 
avoid it.

This is the PR which introduced it: 
https://github.com/apache/cloudstack/pull/4722

Best regards,
Daniel

On 2021/09/17 15:07:55, Wei ZHOU  wrote: 
> For kvm environments, I suggest to set the value to 1 (in global 
> setting and cluster setting).
> 
> -Wei
> 
> 
> On Fri, 17 Sept 2021 at 16:19,  wrote:
> 
> > Hi,
> >
> >   Yes, I have found that this is why i have this value.
> >
> >
> > Regards,
> > Cristian
> >
> > -Original Message-
> > From: Wei ZHOU 
> > Sent: Friday, September 17, 2021 3:34 PM
> > To: users 
> > Subject: Re: The RAM MEM inside KVM instance is not the same as it 
> > was set
> >
> > Hi,
> >
> > Did you set mem.overprovisioning.factor in global setting or cluster 
> > setting ?
> >
> > -Wei
> >
> > On Fri, 17 Sept 2021 at 13:47,  wrote:
> >
> > > Hello,
> > >
> > >
> > >
> > >From where do I get this minRam value?   I used a compute offering
> > > with 10GB of RAM, if I check the VM I see 4GB Ram.
> > >
> > >
> > >
> > >
> > >
> > "[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":"4156","name":"i-2-4156-VM"
> > >
> > > ,"state":"Starting","type":"User","cpus":"4","minSpeed":"760","max
> > > Spee
> > > d":"19
> > > 00","minRam":"(4.00 GB) 4294967296","maxRam":"(10.00 GB) 
> > > 10737418240","arch":"x86_64","os":"CentOS
> > > 7","platformEmulator":"CentOS
> > >
> > > 7","bootArgs":"","enableHA":"false","limitCpuUse":"false","enableD
> > > ynam
> > > ically
> > >
> > > ScaleVm":"false","vncPassword":"NutaRZIRWThdE3aU_IkJAA","vncAddr":"10.
> > > 100.22
> > > .10""
> > >
> > >
> > >
> > >
> > >
> > > Regards,
> > >
> > > Cristian
> > >
> > >
> >
> >
> 

RE: The RAM MEM inside KVM instance is not the same as it was set

[cristian.c]

Hi,

  Yes, I have found that this is why i have this value.


Regards,
Cristian

-Original Message-
From: Wei ZHOU  
Sent: Friday, September 17, 2021 3:34 PM
To: users 
Subject: Re: The RAM MEM inside KVM instance is not the same as it was set

Hi,

Did you set mem.overprovisioning.factor in global setting or cluster setting ?

-Wei

On Fri, 17 Sept 2021 at 13:47,  wrote:

> Hello,
>
>
>
>From where do I get this minRam value?   I used a compute offering
> with 10GB of RAM, if I check the VM I see 4GB Ram.
>
>
>
>
> "[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":"4156","name":"i-2-4156-VM"
>
> ,"state":"Starting","type":"User","cpus":"4","minSpeed":"760","maxSpee
> d":"19
> 00","minRam":"(4.00 GB) 4294967296","maxRam":"(10.00 GB) 
> 10737418240","arch":"x86_64","os":"CentOS 
> 7","platformEmulator":"CentOS
>
> 7","bootArgs":"","enableHA":"false","limitCpuUse":"false","enableDynam
> ically
>
> ScaleVm":"false","vncPassword":"NutaRZIRWThdE3aU_IkJAA","vncAddr":"10.
> 100.22
> .10""
>
>
>
>
>
> Regards,
>
> Cristian
>
>

The RAM MEM inside KVM instance is not the same as it was set

[cristian.c]

Hello,

 

   From where do I get this minRam value?   I used a compute offering
with 10GB of RAM, if I check the VM I see 4GB Ram.

 

"[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":"4156","name":"i-2-4156-VM"
,"state":"Starting","type":"User","cpus":"4","minSpeed":"760","maxSpeed":"19
00","minRam":"(4.00 GB) 4294967296","maxRam":"(10.00 GB)
10737418240","arch":"x86_64","os":"CentOS 7","platformEmulator":"CentOS
7","bootArgs":"","enableHA":"false","limitCpuUse":"false","enableDynamically
ScaleVm":"false","vncPassword":"NutaRZIRWThdE3aU_IkJAA","vncAddr":"10.100.22
.10""

 

 

Regards,

Cristian

RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm

[cristian.c]

I didn't test with lower values, but with 256 RAM and 500Mhz CPU is not working 
for me. (KVM ubuntu 20.04 ACS 4.15.1 )

Regards,
Cristian

-Original Message-
From: Daan Hoogland  
Sent: Friday, September 17, 2021 11:14 AM
To: users 
Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm

thanks for the update Christian,
sounds strange that the router would require that much to start but glad it 
worked.

On Wed, Sep 15, 2021 at 1:32 PM  wrote:

> Hello,
>
>   I have found the issue,   after I have changed the CPU and memory for
> this " System Offering For Software Router - Local Storage' system 
> offering the router was deployed in few seconds without any issue.
>
>   CPU mhz from 500 to 2000 and RAM from 256 to 2048.
>
>
>
> Regards,
> Cristian
>
> -Original Message-
> From: cristian.c@istream.today 
> Sent: Wednesday, September 15, 2021 10:18 AM
> To: users@cloudstack.apache.org
> Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with 
> kvm
>
> Any suggestion on this issue?
>
> Thank you.
>
> -Original Message-
> From: cristian.c@istream.today 
> Sent: Monday, September 13, 2021 5:40 PM
> To: users@cloudstack.apache.org
> Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with 
> kvm
>
> Hi,
>
>
>Yes, please see :
>
> root@usa-nyc-kvm-001:/var/lib/libvirt/images# libvirtd -v
> 2021-09-13 14:33:40.944+: 75403: info : libvirt version: 6.0.0,
> package: 0ubuntu8.13 (Frode Nordahl  Fri, 
> 16 Jul 2021 05:16:36 +)
> 2021-09-13 14:33:40.944+: 75403: info : hostname: usa-nyc-kvm-001
> 2021-09-13 14:33:40.944+: 75403: debug : virLogParseOutputs:1654 :
> outputs=1:stderr
> 2021-09-13 14:33:40.944+: 75403: debug : virLogParseOutput:1492 :
> output=1:stderr
> 2021-09-13 14:33:40.944+: 75403: info : libvirt version: 6.0.0,
> package: 0ubuntu8.13 (Frode Nordahl  Fri, 
> 16 Jul 2021 05:16:36 +)
> 2021-09-13 14:33:40.944+: 75403: info : hostname: usa-nyc-kvm-001
> 2021-09-13 14:33:40.944+: 75403: info : virObjectNew:251 : OBJECT_NEW:
> obj=0x55935a296b60 classname=virAccessManager
> 2021-09-13 14:33:40.944+: 75403: debug : 
> virAccessManagerNewDriver:111
> : Initialized with stack
> 2021-09-13 14:33:40.944+: 75403: info : virObjectNew:251 : OBJECT_NEW:
> obj=0x55935a297f90 classname=virAccessManager
> 2021-09-13 14:33:40.944+: 75403: debug : 
> virAccessManagerNewDriver:111
> : Initialized with none
> 2021-09-13 14:33:40.944+: 75403: info : virObjectRef:386 : OBJECT_REF:
> obj=0x55935a296b60
> 2021-09-13 14:33:40.944+: 75403: info : virObjectUnref:348 :
> OBJECT_UNREF: obj=0x55935a296b60
> 2021-09-13 14:33:40.944+: 75403: debug : main:1177 : Decided on 
> pid file path '/run/libvirtd.pid'
> 2021-09-13 14:33:40.944+: 75403: debug : main:1187 : Decided on 
> socket paths '/run/libvirt/libvirt-sock', 
> '/run/libvirt/libvirt-sock-ro' and '/run/libvirt/libvirt-admin-sock'
> 2021-09-13 14:33:40.944+: 75403: error : virPidFileAcquirePath:367 :
> Failed to acquire pid file '/run/libvirtd.pid': Resource temporarily 
> unavailable
> 2021-09-13 14:33:40.944+: 75403: debug : virFileClose:110 : Closed 
> fd 3
>
>
> root@usa-nyc-kvm-001:/var/lib/libvirt/images# 
> /usr/bin/qemu-system-x86_64 --version QEMU emulator version 4.2.1 
> (Debian 1:4.2-3ubuntu6.17) Copyright
> (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
>
>
> root@usa-nyc-kvm-001:/var/lib/libvirt/images# lsb_release -a No LSB 
> modules are available.
> Distributor ID: Ubuntu
> Description:Ubuntu 20.04.2 LTS
> Release:20.04
> Codename:   focal
>
>
> CPU :
> vendor_id   : AuthenticAMD
> cpu family  : 23
> model   : 49
> model name  : AMD EPYC 7552 48-Core Processor
>
>
> BTW: the SSVM and Proxy console is running fine on this node..
>
>
>
> Thank you!
>
> -Original Message-
> From: Andrija Panic 
> Sent: Monday, September 13, 2021 4:40 PM
> To: users 
> Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with 
> kvm
>
> Can you share qemu/libvirt version/flavour?
>
> On Mon, 13 Sept 2021 at 14:18,  wrote:
>
> > Hello,
> >
> >
> >
> >  I'm trying to understand this error,  I created new network 
> > with VPC in a new zone where I added KVM hypervisor, and I have this issue.
> >
> >
> >
> >
> >
> > 2021-09-13 11:36:18,621 DEBUG
> > [resource.virtualnetwork.VirtualRoutingResource]
> > (agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to
> > 169.254.52.234
> >
> > 2021-09-13 11:36:21,684 D

RE: RE: All systemVMs fail to start

[cristian.c]

Hello,


Check your network, there is a timeout.

c2e7140c71de:mon_host=192.168.204.61:auth_supported=cephx:id=cloudstack:key=AQApuDdhfpp7GRAAWOajg52D4rwuQFXsGuFewg==:rbd_default_format=2:client_mount_timeout=30:
 error while converting raw: error rbd create: File exists
2021-09-16 12:57:55,197 ERROR [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:9fbe695c) Failed to convert from 
/mnt/665b111e-49d0-357e-9d7a-63c5dd8d603d/82f24e9e-b7e6-4830-ab15-c2e7140c71de.qcow2
 to 
rbd:cloudstack/82f24e9e-b7e6-4830-ab15-c2e7140c71de:mon_host=192.168.204.61:auth_supported=cephx:id=cloudstack:key=AQApuDdhfpp7GRAAWOajg52D4rwuQFXsGuFewg==:rbd_default_format=2:client_mount_timeout=30
 the error was: qemu-img: 
rbd:cloudstack/82f24e9e-b7e6-4830-ab15-c2e7140c71de:mon_host=192.168.204.61:auth_supported=cephx:id=cloudstack:key=AQApuDdhfpp7GRAAWOajg52D4rwuQFXsGuFewg==:rbd_default_format=2:client_mount_timeout=30:
 error while converting raw: error rbd create: File exists
2021-09-16 12:57:55,197 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:9fbe695c) Attempting to remove storage 
pool 665b111e-49d0-357e-9d7a-63c5dd8d603d from libvirt
2021-09-16 12:57:55,197 DEBUG [kvm.resource.LibvirtConnection] 
(agentRequest-Handler-3:null) (logid:9fbe695c) Looking for libvirtd connection 
at: qemu:///system
2021-09-16 12:57:55,201 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:9fbe695c) Storage pool 
665b111e-49d0-357e-9d7a-63c5dd8d603d has no corresponding secret. Not removing 
any secret.
2021-09-16 12:57:55,268 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:9fbe695c) Storage pool 
665b111e-49d0-357e-9d7a-63c5dd8d603d was succesfully removed from libvirt.
2021-09-16 12:57:55,270 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) 
(logid:9fbe695c) Seq 3-8924445611588059147:  { Ans: , MgmtId: 90520741394843, 
via: 3, Ver: v1, Flags: 110, 
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
 Failed to copy 
/mnt/665b111e-49d0-357e-9d7a-63c5dd8d603d/82f24e9e-b7e6-4830-ab15-c2e7140c71de.qcow2
 to 
82f24e9e-b7e6-4830-ab15-c2e7140c71de","wait":"0","bypassHostMaintenance":"false"}}]
 }
2021-09-16 12:57:57,310 DEBUG [resource.wrapper.LibvirtStopCommandWrapper] 
(agentRequest-Handler-2:null) (logid:20082d80) Failed to backup cmdline file 
due to There was a problem while connecting to 169.254.204.209:3922


Regards,
Cristian


-Original Message-
From: Peter Stine  
Sent: Thursday, September 16, 2021 8:02 PM
To: users@cloudstack.apache.org
Subject: Re: RE: All systemVMs fail to start

Here's the log from the agent.

https://gist.github.com/PeterS-gd/04a279c5a4d4645ee7f2cd9971238b17

On 2021/09/16 06:52:36,  wrote: 
> Hello,
> 
>  Can you enable the debug log for KVM cloudstack agent and provide the 
> log?
> 
> 
> Regards,
> Cristian
> 
> -Original Message-
> From: Peter Stine  
> Sent: Thursday, September 16, 2021 1:10 AM
> To: users@cloudstack.apache.org
> Subject: All systemVMs fail to start
> 
> I have been trying to launch Cloudstack 4.15.1.0. I am running the management 
> server virtually on a Ubuntu 20.04 server using KVM for my virtualization for 
> the system VMs.
> 
> I have been following the install process listed in the apache docs.
> 
> After seeding the System VM templates and registering I started the system, 
> the VM Images are appearing in the images section, but the SSVM, 
> proxyconsole, and virtual router will not start. They get stuck at starting, 
> crash, retry, and repeat. When trying to start the VMs manually, I get an 
> "error 431 cannot find uuid". I do not see anything in the logs that 
> specifies what uuid it is trying to look for. There are two unused uuids in 
> the database that are the seeded system template. It almost looks like it 
> tries to call them, but cannot copy them.
> Is there a reason for this?
> 
> Here are the logs as best I can get them.
> https://gist.github.com/PeterS-gd/b7606fbdc06dcd7a173901bc0af9b1d0
> 
> 

root disk resize KVM

[cristian.c]

Hello,

 

   It is possible to use root disk resize on the wizard page? Or via API
when you deploy a KVM instance?

 

 

Regards,

Cristian

RE: All systemVMs fail to start

[cristian.c]

Hello,

 Can you enable the debug log for KVM cloudstack agent and provide the log?


Regards,
Cristian

-Original Message-
From: Peter Stine  
Sent: Thursday, September 16, 2021 1:10 AM
To: users@cloudstack.apache.org
Subject: All systemVMs fail to start

I have been trying to launch Cloudstack 4.15.1.0. I am running the management 
server virtually on a Ubuntu 20.04 server using KVM for my virtualization for 
the system VMs.

I have been following the install process listed in the apache docs.

After seeding the System VM templates and registering I started the system, the 
VM Images are appearing in the images section, but the SSVM, proxyconsole, and 
virtual router will not start. They get stuck at starting, crash, retry, and 
repeat. When trying to start the VMs manually, I get an "error 431 cannot find 
uuid". I do not see anything in the logs that specifies what uuid it is trying 
to look for. There are two unused uuids in the database that are the seeded 
system template. It almost looks like it tries to call them, but cannot copy 
them.
Is there a reason for this?

Here are the logs as best I can get them.
https://gist.github.com/PeterS-gd/b7606fbdc06dcd7a173901bc0af9b1d0

RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm

[cristian.c]

Hello,

  I have found the issue,   after I have changed the CPU and memory for this " 
System Offering For Software Router - Local Storage' system offering the router 
was deployed in few seconds without any issue.

  CPU mhz from 500 to 2000 and RAM from 256 to 2048.



Regards,
Cristian

-Original Message-
From: cristian.c@istream.today  
Sent: Wednesday, September 15, 2021 10:18 AM
To: users@cloudstack.apache.org
Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm

Any suggestion on this issue?

Thank you.

-Original Message-
From: cristian.c@istream.today 
Sent: Monday, September 13, 2021 5:40 PM
To: users@cloudstack.apache.org
Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm

Hi,


   Yes, please see :  

root@usa-nyc-kvm-001:/var/lib/libvirt/images# libvirtd -v
2021-09-13 14:33:40.944+: 75403: info : libvirt version: 6.0.0, package: 
0ubuntu8.13 (Frode Nordahl  Fri, 16 Jul 2021 
05:16:36 +)
2021-09-13 14:33:40.944+: 75403: info : hostname: usa-nyc-kvm-001
2021-09-13 14:33:40.944+: 75403: debug : virLogParseOutputs:1654 : 
outputs=1:stderr
2021-09-13 14:33:40.944+: 75403: debug : virLogParseOutput:1492 : 
output=1:stderr
2021-09-13 14:33:40.944+: 75403: info : libvirt version: 6.0.0, package: 
0ubuntu8.13 (Frode Nordahl  Fri, 16 Jul 2021 
05:16:36 +)
2021-09-13 14:33:40.944+: 75403: info : hostname: usa-nyc-kvm-001
2021-09-13 14:33:40.944+: 75403: info : virObjectNew:251 : OBJECT_NEW: 
obj=0x55935a296b60 classname=virAccessManager
2021-09-13 14:33:40.944+: 75403: debug : virAccessManagerNewDriver:111 : 
Initialized with stack
2021-09-13 14:33:40.944+: 75403: info : virObjectNew:251 : OBJECT_NEW: 
obj=0x55935a297f90 classname=virAccessManager
2021-09-13 14:33:40.944+: 75403: debug : virAccessManagerNewDriver:111 : 
Initialized with none
2021-09-13 14:33:40.944+: 75403: info : virObjectRef:386 : OBJECT_REF: 
obj=0x55935a296b60
2021-09-13 14:33:40.944+: 75403: info : virObjectUnref:348 : OBJECT_UNREF: 
obj=0x55935a296b60
2021-09-13 14:33:40.944+: 75403: debug : main:1177 : Decided on pid file 
path '/run/libvirtd.pid'
2021-09-13 14:33:40.944+: 75403: debug : main:1187 : Decided on socket 
paths '/run/libvirt/libvirt-sock', '/run/libvirt/libvirt-sock-ro' and 
'/run/libvirt/libvirt-admin-sock'
2021-09-13 14:33:40.944+: 75403: error : virPidFileAcquirePath:367 : Failed 
to acquire pid file '/run/libvirtd.pid': Resource temporarily unavailable
2021-09-13 14:33:40.944+: 75403: debug : virFileClose:110 : Closed fd 3


root@usa-nyc-kvm-001:/var/lib/libvirt/images# /usr/bin/qemu-system-x86_64 
--version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.17) Copyright (c) 
2003-2019 Fabrice Bellard and the QEMU Project developers


root@usa-nyc-kvm-001:/var/lib/libvirt/images# lsb_release -a No LSB modules are 
available.
Distributor ID: Ubuntu
Description:Ubuntu 20.04.2 LTS
Release:20.04
Codename:   focal


CPU :
vendor_id   : AuthenticAMD
cpu family  : 23
model   : 49
model name  : AMD EPYC 7552 48-Core Processor


BTW: the SSVM and Proxy console is running fine on this node..



Thank you!

-Original Message-
From: Andrija Panic 
Sent: Monday, September 13, 2021 4:40 PM
To: users 
Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm

Can you share qemu/libvirt version/flavour?

On Mon, 13 Sept 2021 at 14:18,  wrote:

> Hello,
>
>
>
>  I'm trying to understand this error,  I created new network with 
> VPC in a new zone where I added KVM hypervisor, and I have this issue.
>
>
>
>
>
> 2021-09-13 11:36:18,621 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to
> 169.254.52.234
>
> 2021-09-13 11:36:21,684 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Could not connect to
> 169.254.52.234
>
>
>
> Here is the full log : https://pastebin.com/0NwsccUc
>
>
>
> root@usa-nyc-kvm-001:~# telnet 169.254.156.179 3922
>
> Trying 169.254.156.179...
>
> telnet: Unable to connect to remote host: Connection refused  ( before 
> connection refused, I get no route to host, for sure the nic is not up
> )
>
> root@usa-nyc-kvm-001:~# virsh list
>
> Id   NameState
>
> ---
>
> 2v-4048-VM   running
>
> 4s-4050-VM   running
>
> 7r-4092-VM   running
>
>
>
> root@usa-nyc-kvm-001:~#
>
>
>
>
>
> If I check the log inside the router, I see this (the router is also 
> responding very slow, waiting a lot to connect on it via virsh console):
>
>
>
> r-4092-VM login: root
>
> Password:
>
>
>
> Login timed out after 60 seconds.
>
>
>
>
>
>   

Image is not in qcow2 format - 1 zone with KVM and Vmware

[cristian.c]

Hello,

 

This looks like a bug, why is trying to use the VMware systemvm template
for a KVM host?  The KVM system template is available and ready for use.

 

This is a test environment with 1 Zone with VMware and 2 clusters, 1 for
VMware and 1 with KVM hypervisor, I can use the hypervisor for instances but
not for system vms because of this issue.   I wanted to reproduce in this
environment the systemvm deploy issue which I have in a different
environment (virtual router stuck on starting - cloudstack 4.15.1 with kvm)
and here I have this issue.  

 

 

2021-09-15 08:06:35,641 WARN  [resource.wrapper.LibvirtStartCommandWrapper]
(agentRequest-Handler-4:null) (logid:63f6a09d) LibvirtException

org.libvirt.LibvirtException: internal error: process exited while
connecting to monitor: 2021-09-15T08:06:35.264910Z qemu-system-x86_64:
-blockdev
{"node-name":"libvirt-3-format","read-only":true,"cache":{"direct":true,"no-
flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null}:
Image is not in qcow2 format

at org.libvirt.ErrorHandler.processError(Unknown Source)

at org.libvirt.ErrorHandler.processError(Unknown Source)

at org.libvirt.Connect.domainCreateXML(Unknown Source)

at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtCo
mputingResource.java:1614)

at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute
(LibvirtStartCommandWrapper.java:85)

at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute
(LibvirtStartCommandWrapper.java:45)

at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(Libv
irtRequestWrapper.java:78)

at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(Li
bvirtComputingResource.java:1646)

at com.cloud.agent.Agent.processRequest(Agent.java:661)

at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079)

at com.cloud.utils.nio.Task.call(Task.java:83)

at com.cloud.utils.nio.Task.call(Task.java:29)

at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut
or.java:1128)

at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu
tor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

 

 

Regards,

Cristian

RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm

[cristian.c]

Any suggestion on this issue?

Thank you.

-Original Message-
From: cristian.c@istream.today  
Sent: Monday, September 13, 2021 5:40 PM
To: users@cloudstack.apache.org
Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm

Hi,


   Yes, please see :  

root@usa-nyc-kvm-001:/var/lib/libvirt/images# libvirtd -v
2021-09-13 14:33:40.944+: 75403: info : libvirt version: 6.0.0, package: 
0ubuntu8.13 (Frode Nordahl  Fri, 16 Jul 2021 
05:16:36 +)
2021-09-13 14:33:40.944+: 75403: info : hostname: usa-nyc-kvm-001
2021-09-13 14:33:40.944+: 75403: debug : virLogParseOutputs:1654 : 
outputs=1:stderr
2021-09-13 14:33:40.944+: 75403: debug : virLogParseOutput:1492 : 
output=1:stderr
2021-09-13 14:33:40.944+: 75403: info : libvirt version: 6.0.0, package: 
0ubuntu8.13 (Frode Nordahl  Fri, 16 Jul 2021 
05:16:36 +)
2021-09-13 14:33:40.944+: 75403: info : hostname: usa-nyc-kvm-001
2021-09-13 14:33:40.944+: 75403: info : virObjectNew:251 : OBJECT_NEW: 
obj=0x55935a296b60 classname=virAccessManager
2021-09-13 14:33:40.944+: 75403: debug : virAccessManagerNewDriver:111 : 
Initialized with stack
2021-09-13 14:33:40.944+: 75403: info : virObjectNew:251 : OBJECT_NEW: 
obj=0x55935a297f90 classname=virAccessManager
2021-09-13 14:33:40.944+: 75403: debug : virAccessManagerNewDriver:111 : 
Initialized with none
2021-09-13 14:33:40.944+: 75403: info : virObjectRef:386 : OBJECT_REF: 
obj=0x55935a296b60
2021-09-13 14:33:40.944+: 75403: info : virObjectUnref:348 : OBJECT_UNREF: 
obj=0x55935a296b60
2021-09-13 14:33:40.944+: 75403: debug : main:1177 : Decided on pid file 
path '/run/libvirtd.pid'
2021-09-13 14:33:40.944+: 75403: debug : main:1187 : Decided on socket 
paths '/run/libvirt/libvirt-sock', '/run/libvirt/libvirt-sock-ro' and 
'/run/libvirt/libvirt-admin-sock'
2021-09-13 14:33:40.944+: 75403: error : virPidFileAcquirePath:367 : Failed 
to acquire pid file '/run/libvirtd.pid': Resource temporarily unavailable
2021-09-13 14:33:40.944+: 75403: debug : virFileClose:110 : Closed fd 3


root@usa-nyc-kvm-001:/var/lib/libvirt/images# /usr/bin/qemu-system-x86_64 
--version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.17) Copyright (c) 
2003-2019 Fabrice Bellard and the QEMU Project developers


root@usa-nyc-kvm-001:/var/lib/libvirt/images# lsb_release -a No LSB modules are 
available.
Distributor ID: Ubuntu
Description:Ubuntu 20.04.2 LTS
Release:20.04
Codename:   focal


CPU :
vendor_id   : AuthenticAMD
cpu family  : 23
model   : 49
model name  : AMD EPYC 7552 48-Core Processor


BTW: the SSVM and Proxy console is running fine on this node..



Thank you!

-Original Message-
From: Andrija Panic 
Sent: Monday, September 13, 2021 4:40 PM
To: users 
Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm

Can you share qemu/libvirt version/flavour?

On Mon, 13 Sept 2021 at 14:18,  wrote:

> Hello,
>
>
>
>  I'm trying to understand this error,  I created new network with 
> VPC in a new zone where I added KVM hypervisor, and I have this issue.
>
>
>
>
>
> 2021-09-13 11:36:18,621 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to
> 169.254.52.234
>
> 2021-09-13 11:36:21,684 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Could not connect to
> 169.254.52.234
>
>
>
> Here is the full log : https://pastebin.com/0NwsccUc
>
>
>
> root@usa-nyc-kvm-001:~# telnet 169.254.156.179 3922
>
> Trying 169.254.156.179...
>
> telnet: Unable to connect to remote host: Connection refused  ( before 
> connection refused, I get no route to host, for sure the nic is not up
> )
>
> root@usa-nyc-kvm-001:~# virsh list
>
> Id   NameState
>
> ---
>
> 2v-4048-VM   running
>
> 4s-4050-VM   running
>
> 7r-4092-VM   running
>
>
>
> root@usa-nyc-kvm-001:~#
>
>
>
>
>
> If I check the log inside the router, I see this (the router is also 
> responding very slow, waiting a lot to connect on it via virsh console):
>
>
>
> r-4092-VM login: root
>
> Password:
>
>
>
> Login timed out after 60 seconds.
>
>
>
>
>
>__?.o/  Apache CloudStack SystemVM 4.15.1
>
>   (  )#https://cloudstack.apache.org
>
> (___(_)   Debian GNU/Linux 10 r-4093-VM ttyS0
>
>
>
> r-4093-VM login: root
>
> Password:
>
> Linux r-4093-VM 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30)
> x86_64
>
>
>
>
>
>
>
> root@r-4093-VM:~# cat /var/log/cloud.log
>
> Mon 13 Sep 2021 11:50:55 AM UTC Executing cloud-early-config
>
> Mon 13 Sep 2021 11:50:55 AM UTC Bootstrapping s

bash: /opt/cloud/bin/keystore-setup: No such file or directory

[cristian.c]

I do not understand anymore what is wrong with this KVM setup, now I try to
rebuild again the Secondary storage and console because the virtual router
is not working but I get a different error "bash:
/opt/cloud/bin/keystore-setup: No such file or directory"

 

021-09-14 07:06:39,255 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:defd9839) Processing command:
org.apache.cloudstack.ca.SetupKeyStoreCommand

2021-09-14 07:06:39,256 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:defd9839) Executing:
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
keystore-setup 169.254.61.247
/usr/local/cloud/systemvm/conf/agent.properties
/usr/local/cloud/systemvm/conf/

2021-09-14 07:06:39,257 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:defd9839) Executing while with timeout
: 90

2021-09-14 07:06:39,447 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:defd9839) Exit value is 127

2021-09-14 07:06:39,448 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:defd9839) bash:
/opt/cloud/bin/keystore-setup: No such file or directory

2021-09-14 07:06:39,448 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:defd9839) Executing script in VR:
keystore-setup

2021-09-14 07:06:39,449 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:defd9839) Seq
106277-3015159950524547147:  { Ans: , MgmtId: 345048618557, via: 106277,
Ver: v1, Flags: 10,
[{"org.apache.cloudstack.ca.SetupKeystoreAnswer":{"result":"true","wait":"0"
,"bypassHostMaintenance":"false"}}] }

 

 

Is this a bug?   I deleted the templated and added again,  the system
template is 4.15.1 KVM like the cloudstack version.

 

 

BTW, I have the same version of KVM but in a cluster, and the systemvms are
hosted on vmware, I do not have any problems with that one for instances but
I didn't tested for system vms.

 

 

Thank you

RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm

[cristian.c]

Hi,


   Yes, please see :  

root@usa-nyc-kvm-001:/var/lib/libvirt/images# libvirtd -v
2021-09-13 14:33:40.944+: 75403: info : libvirt version: 6.0.0, package: 
0ubuntu8.13 (Frode Nordahl  Fri, 16 Jul 2021 
05:16:36 +)
2021-09-13 14:33:40.944+: 75403: info : hostname: usa-nyc-kvm-001
2021-09-13 14:33:40.944+: 75403: debug : virLogParseOutputs:1654 : 
outputs=1:stderr
2021-09-13 14:33:40.944+: 75403: debug : virLogParseOutput:1492 : 
output=1:stderr
2021-09-13 14:33:40.944+: 75403: info : libvirt version: 6.0.0, package: 
0ubuntu8.13 (Frode Nordahl  Fri, 16 Jul 2021 
05:16:36 +)
2021-09-13 14:33:40.944+: 75403: info : hostname: usa-nyc-kvm-001
2021-09-13 14:33:40.944+: 75403: info : virObjectNew:251 : OBJECT_NEW: 
obj=0x55935a296b60 classname=virAccessManager
2021-09-13 14:33:40.944+: 75403: debug : virAccessManagerNewDriver:111 : 
Initialized with stack
2021-09-13 14:33:40.944+: 75403: info : virObjectNew:251 : OBJECT_NEW: 
obj=0x55935a297f90 classname=virAccessManager
2021-09-13 14:33:40.944+: 75403: debug : virAccessManagerNewDriver:111 : 
Initialized with none
2021-09-13 14:33:40.944+: 75403: info : virObjectRef:386 : OBJECT_REF: 
obj=0x55935a296b60
2021-09-13 14:33:40.944+: 75403: info : virObjectUnref:348 : OBJECT_UNREF: 
obj=0x55935a296b60
2021-09-13 14:33:40.944+: 75403: debug : main:1177 : Decided on pid file 
path '/run/libvirtd.pid'
2021-09-13 14:33:40.944+: 75403: debug : main:1187 : Decided on socket 
paths '/run/libvirt/libvirt-sock', '/run/libvirt/libvirt-sock-ro' and 
'/run/libvirt/libvirt-admin-sock'
2021-09-13 14:33:40.944+: 75403: error : virPidFileAcquirePath:367 : Failed 
to acquire pid file '/run/libvirtd.pid': Resource temporarily unavailable
2021-09-13 14:33:40.944+: 75403: debug : virFileClose:110 : Closed fd 3


root@usa-nyc-kvm-001:/var/lib/libvirt/images# /usr/bin/qemu-system-x86_64 
--version
QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.17)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers


root@usa-nyc-kvm-001:/var/lib/libvirt/images# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 20.04.2 LTS
Release:20.04
Codename:   focal


CPU :
vendor_id   : AuthenticAMD
cpu family  : 23
model   : 49
model name  : AMD EPYC 7552 48-Core Processor


BTW: the SSVM and Proxy console is running fine on this node..



Thank you!

-Original Message-
From: Andrija Panic  
Sent: Monday, September 13, 2021 4:40 PM
To: users 
Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm

Can you share qemu/libvirt version/flavour?

On Mon, 13 Sept 2021 at 14:18,  wrote:

> Hello,
>
>
>
>  I'm trying to understand this error,  I created new network with 
> VPC in a new zone where I added KVM hypervisor, and I have this issue.
>
>
>
>
>
> 2021-09-13 11:36:18,621 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to
> 169.254.52.234
>
> 2021-09-13 11:36:21,684 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Could not connect to
> 169.254.52.234
>
>
>
> Here is the full log : https://pastebin.com/0NwsccUc
>
>
>
> root@usa-nyc-kvm-001:~# telnet 169.254.156.179 3922
>
> Trying 169.254.156.179...
>
> telnet: Unable to connect to remote host: Connection refused  ( before 
> connection refused, I get no route to host, for sure the nic is not up 
> )
>
> root@usa-nyc-kvm-001:~# virsh list
>
> Id   NameState
>
> ---
>
> 2v-4048-VM   running
>
> 4s-4050-VM   running
>
> 7r-4092-VM   running
>
>
>
> root@usa-nyc-kvm-001:~#
>
>
>
>
>
> If I check the log inside the router, I see this (the router is also 
> responding very slow, waiting a lot to connect on it via virsh console):
>
>
>
> r-4092-VM login: root
>
> Password:
>
>
>
> Login timed out after 60 seconds.
>
>
>
>
>
>__?.o/  Apache CloudStack SystemVM 4.15.1
>
>   (  )#https://cloudstack.apache.org
>
> (___(_)   Debian GNU/Linux 10 r-4093-VM ttyS0
>
>
>
> r-4093-VM login: root
>
> Password:
>
> Linux r-4093-VM 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30)
> x86_64
>
>
>
>
>
>
>
> root@r-4093-VM:~# cat /var/log/cloud.log
>
> Mon 13 Sep 2021 11:50:55 AM UTC Executing cloud-early-config
>
> Mon 13 Sep 2021 11:50:55 AM UTC Bootstrapping systemvm appliance
>
> Mon 13 Sep 2021 11:50:57 AM UTC Starting guest services for kvm
>
> Mon 13 Sep 2021 11:51:04 AM UTC acpiphp and pci_hotplug module already 
> compiled in
>
> Mon 13 Sep 2021 11:52:59 AM UTC Received a new non-empty cmdline file 
> from qemu-guest-agent
>
> Mon 13 Sep 2021 11:52:59 AM UTC Booting from cloudstack, remove old 
> configuration files in /etc/cloudstack/
>
> Mon 13 Sep 2021 11:54:30 AM UTC Configuring systemvm type=vpcrouter
>
> Mon 13 Sep 2021 11:54:30 AM UTC Setting up VPC virtual router system 
> vm
>
> Mon 

virtual router stuck on starting - cloudstack 4.15.1 with kvm

[cristian.c]

Hello,

 

 I'm trying to understand this error,  I created new network with VPC in
a new zone where I added KVM hypervisor, and I have this issue.

 

 

2021-09-13 11:36:18,621 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to
169.254.52.234

2021-09-13 11:36:21,684 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-1:null) (logid:42d715f2) Could not connect to
169.254.52.234

 

Here is the full log : https://pastebin.com/0NwsccUc 

 

root@usa-nyc-kvm-001:~# telnet 169.254.156.179 3922

Trying 169.254.156.179...

telnet: Unable to connect to remote host: Connection refused  ( before
connection refused, I get no route to host, for sure the nic is not up )

root@usa-nyc-kvm-001:~# virsh list

Id   NameState

---

2v-4048-VM   running

4s-4050-VM   running

7r-4092-VM   running

 

root@usa-nyc-kvm-001:~#

 

 

If I check the log inside the router, I see this (the router is also
responding very slow, waiting a lot to connect on it via virsh console):

 

r-4092-VM login: root

Password:

 

Login timed out after 60 seconds.

 

 

   __?.o/  Apache CloudStack SystemVM 4.15.1

  (  )#https://cloudstack.apache.org

(___(_)   Debian GNU/Linux 10 r-4093-VM ttyS0

 

r-4093-VM login: root

Password:

Linux r-4093-VM 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64

 

 

 

root@r-4093-VM:~# cat /var/log/cloud.log

Mon 13 Sep 2021 11:50:55 AM UTC Executing cloud-early-config

Mon 13 Sep 2021 11:50:55 AM UTC Bootstrapping systemvm appliance

Mon 13 Sep 2021 11:50:57 AM UTC Starting guest services for kvm

Mon 13 Sep 2021 11:51:04 AM UTC acpiphp and pci_hotplug module already
compiled in

Mon 13 Sep 2021 11:52:59 AM UTC Received a new non-empty cmdline file from
qemu-guest-agent

Mon 13 Sep 2021 11:52:59 AM UTC Booting from cloudstack, remove old
configuration files in /etc/cloudstack/

Mon 13 Sep 2021 11:54:30 AM UTC Configuring systemvm type=vpcrouter

Mon 13 Sep 2021 11:54:30 AM UTC Setting up VPC virtual router system vm

Mon 13 Sep 2021 11:54:31 AM UTC Setting up apache web server for VPC

Mon 13 Sep 2021 11:55:05 AM UTC Processors = 1  Enable service  = 0

Mon 13 Sep 2021 11:55:05 AM UTC cloud: disable rp_filter

Mon 13 Sep 2021 11:55:05 AM UTC disable rpfilter

Mon 13 Sep 2021 11:55:05 AM UTC cloud: enable_fwding = 1

Mon 13 Sep 2021 11:55:05 AM UTC enable_fwding = 1

Mon 13 Sep 2021 11:55:05 AM UTC cloud: enabling passive FTP for guest VMs

Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm

Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm

2021-09-13 11:55:13,823 INFO update_config.py :: Processing incoming
file => cmd_line.json

2021-09-13 11:55:13,856 INFO Processing JSON file cmd_line.json

2021-09-13 11:55:13,858 INFO Continuing with the processing of file
'/var/cache/cloud/cmd_line.json'

2021-09-13 11:55:13,877 INFO Command of type cmdline received

2021-09-13 11:55:13,897 INFO Command of type ips received

2021-09-13 11:55:13,923 INFO Executing: ip addr show dev eth0

2021-09-13 11:55:13,975 INFO Executing: ip addr show dev eth0

2021-09-13 11:55:13,979 INFO Address found in DataBag ==> {u'public_ip':
u'169.254.41.129', u'one_to_one_nat': False, u'nic_dev_id': u'0',
u'network': u'169.254.0.0/16', u'netmask': u'255.255.0.0', u'source_nat':
False, u'broadcast': u'169.254.255.255', u'add': True, u'nw_type':
u'control', u'device': u'eth0', u'cidr': u'169.254.41.129/16', u'gateway':
u'', u'size': u'16'}

2021-09-13 11:55:13,979 INFO Address 169.254.41.129/16 on device eth0
already configured

2021-09-13 11:55:13,984 INFO Executing: ip addr show |grep -v secondary

2021-09-13 11:55:14,010 INFO Wrote edited file /etc/dnsmasq.d/cloud.conf

2021-09-13 11:55:14,010 INFO Updated file in-cache configuration

2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcphosts.txt
file did not change

2021-09-13 11:55:14,010 INFO Nothing to commit. The
/var/lib/misc/dnsmasq.leases file did not change

2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcpopts.txt
file did not change

2021-09-13 11:55:14,010 INFO Attempting to delete entries from
dnsmasq.leases file for VMs which are not on dhcphosts file

2021-09-13 11:55:14,010 ERRORCaught error while trying to delete entries
from dnsmasq.leases file: [Errno 2] No such file or directory:
'/etc/dhcphosts.txt'

2021-09-13 11:55:14,034 INFO Wrote edited file /etc/hosts

2021-09-13 11:55:14,044 INFO Updated file in-cache configuration

2021-09-13 11:55:14,044 INFO Updated hosts file

2021-09-13 11:55:14,046 INFO Executing: systemctl restart dnsmasq

2021-09-13 11:55:15,409 INFO Service dnsmasq restart

2021-09-13 11:55:15,453 INFO Executing: ip addr show |grep -v secondary

2021-09-13 11:55:15,832 INFO Nothing to commit. The
/etc/dnsmasq.d/cloud.conf file did not change

RE: Timed out /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch.sh

[cristian.c]

HI Wei,

  You were right, something was wrong with the template, I re added.


Thank you.

-Original Message-
From: Wei ZHOU  
Sent: Friday, September 10, 2021 5:15 PM
To: users 
Subject: Re: Timed out 
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch.sh

Hi Cristian,

Can you check if the SSVM uses the correct systemvm template ?

-Wei

On Fri, 10 Sept 2021 at 16:11,  wrote:

> Hello,
>
>
>
>  Any idea from where I get this time out?
>
>
>
> 2021-09-10 14:09:52,630 WARN  [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) (logid:e7bed45c) Timed out:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch.sh -n 
> s-4041-VM -c  template=domP type=secstorage 
> host=10.100.25.250,172.11.0.2 port=8250 name=s-4041-VM zone=5 pod=5 
> guid=s-4041-VM workers=5 
> resource=com.cloud.storage.resource.PremiumSecondaryStorageResource
> instance=SecStorage sslcopy=false role=templateProcessor mtu=1500 
> eth2ip=23.29.xxx.xxx eth2mask=255.255.255.224 gateway=23.29.xxx.xxx
> public.network.device=eth2 eth0ip=169.254.159.216 eth0mask=255.255.0.0
> eth1ip=10.100.22.165 eth1mask=255.255.255.0 localgw=10.100.22.4
> private.network.device=eth1 internaldns1=10.100.22.4 dns1=8.8.8.8 
> nfsVersion=null .  Output is:
>
> 2021-09-10 14:09:52,632 ERROR [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) (logid:e7bed45c) Passing cmdline 
> failed:timeout
>
>
>
>
>
> Regards,
>
> Cristian
>
>

Timed out /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch.sh

[cristian.c]

Hello,

 

 Any idea from where I get this time out? 

 

2021-09-10 14:09:52,630 WARN  [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) (logid:e7bed45c) Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch.sh -n s-4041-VM
-c  template=domP type=secstorage host=10.100.25.250,172.11.0.2 port=8250
name=s-4041-VM zone=5 pod=5 guid=s-4041-VM workers=5
resource=com.cloud.storage.resource.PremiumSecondaryStorageResource
instance=SecStorage sslcopy=false role=templateProcessor mtu=1500
eth2ip=23.29.xxx.xxx eth2mask=255.255.255.224 gateway=23.29.xxx.xxx
public.network.device=eth2 eth0ip=169.254.159.216 eth0mask=255.255.0.0
eth1ip=10.100.22.165 eth1mask=255.255.255.0 localgw=10.100.22.4
private.network.device=eth1 internaldns1=10.100.22.4 dns1=8.8.8.8
nfsVersion=null .  Output is:

2021-09-10 14:09:52,632 ERROR [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) (logid:e7bed45c) Passing cmdline
failed:timeout

 

 

Regards,

Cristian