Re: Rocky 9 and CS 4.17.1.0
Hi Simon, I can confirm that the cpu mode is host-passthrough FYI - I run KVM for my Hypervisor If this is any help I took a screenshot of the console. 6.1522631 Call Trace: 6.1525591 6.1528651 dump_stack_lvl+0x34/0x48 panic+0x102/0x2d4 6.153139] 6.153438] do_exit.cold+0x14/0x9f do_group_exit+0x33/Oxa0 6.1537371 6.1540621 6.154393] ? exc_page_fault+0x62/0x150 6.1547091 entry_SYSCALL_64_after_hwframe+0x63/Oxcd 6.1550791 RIP: 0033:0x78 acobb?c2d 6.1554101 Code: c3 Of 11 84 00 00 00 00 00 f3 Of 1e fa be e7 00 00 00 ba 3cl 00 00 00 eb od 89 do or 05 48 3d 00 fo ff ff 27 1c f4 89 fo of 05 <48› 3d 00 fo ff ff 76 e7 f7 d8 89 05 ff fe 00 00 eb dd of 1f 44 00 6.1565621 RSP: 002h: 7ffcf88f6428 EFLAGS: 0246 ORIG RAX: 000 000e7 6.1570511 RAX: fida RBX: 7 dc0bb76130 RCX: 7fdc0bb7c2d1 6.1575451 RDX: 003c RSI: 00€7 RDI: 007f 6.1580151 RB: 7ffcf88f6eb0 R08: 7ffcf8816899 R09: 6.1586231 R10: R11: 0246 R12: 7f dcObb55000 6.1591971 R13: 00230007 R14: 0007 R15: 7ffcf88f6ec0 6.1617161 Kernel Offset: 0x2fc0 from Ox8100 (relocation ran ge: Ox8000-Oxbfff) 6.1626471 -[ end Kernel panic - not syncing: Attempted to kill init? exit code=0x7f 00 Thanks On 2/2/23, 3:11 PM, "Simon Weller" wrote: Jeremy and Robert, Out of interest, which cpu profile are you passing through to the guest? I've seen a lot of reports with 9.x and kernel panics when certain CPU flags aren't being passed through, in particular x16 lahf popcnt sse4_1 sse4_2 ssse3. Do you have the ability to set your guest.cpu.mode to "host-passthrough" for the purposes of testing? You'd do that by editing /etc/cloudstack/agent/agent.properties on your host. Look for guest.cpu.mode and replace it with = guest.cpu.mode=host-passthrough Then restart the agent. -Si On Thu, Feb 2, 2023 at 9:32 AM Robert Ward wrote: > Hi Rohit, > > I can report that installing Oracle 9.1 on a VM results in a kernel panic > right off the bat. I hear there were some major changes between EL8 and 9. > My SaSS customers are holding off on upgrades as well because of potential > concerns. > > Thanks > > On 2/2/23, 10:17 AM, "Rohit Yadav" wrote: > > Jeremy, > > For EL9, can you try Alma Linux 9 or Oracle Linux 9 for guest VM? I > have had bad experiences with Rocky Linux 9 in terms of stability and > mirror/repo accessibility even though Rocky Linux 9 claims to be EL9 > compatible. > > Regards. > > > From: Jeremy Hansen > Sent: Saturday, December 10, 2022 7:53 PM > To: Vivek Kumar via users > Subject: Rocky 9 and CS 4.17.1.0 > > I’m running Cloudstack 4.17.1.0 and for unknown reasons, I’m having > issues running Rocky 9. Kernel begins to boot and then it looks like it > fails on loading initrd and I get a kernel oops. Just curious if this is a > known issue or if there’s a work around. I tried using the qcow2 image from > Rocky as well and just using the install iso to create a new image. Same > result. > > Rocky 8 works fine. > > Anyone running Rocky 9? > > Thanks > -jeremy > > > > > > > > >
Re: Rocky 9 and CS 4.17.1.0
Hi Rohit, I can report that installing Oracle 9.1 on a VM results in a kernel panic right off the bat. I hear there were some major changes between EL8 and 9. My SaSS customers are holding off on upgrades as well because of potential concerns. Thanks On 2/2/23, 10:17 AM, "Rohit Yadav" wrote: Jeremy, For EL9, can you try Alma Linux 9 or Oracle Linux 9 for guest VM? I have had bad experiences with Rocky Linux 9 in terms of stability and mirror/repo accessibility even though Rocky Linux 9 claims to be EL9 compatible. Regards. From: Jeremy Hansen Sent: Saturday, December 10, 2022 7:53 PM To: Vivek Kumar via users Subject: Rocky 9 and CS 4.17.1.0 I’m running Cloudstack 4.17.1.0 and for unknown reasons, I’m having issues running Rocky 9. Kernel begins to boot and then it looks like it fails on loading initrd and I get a kernel oops. Just curious if this is a known issue or if there’s a work around. I tried using the qcow2 image from Rocky as well and just using the install iso to create a new image. Same result. Rocky 8 works fine. Anyone running Rocky 9? Thanks -jeremy
VM Console Freeze/disconnect
Hello all,
Have a perplexing problem with the console function. I can launch consoles
without any issue but after about 10 seconds or so the console will freeze and
then disconnect shortly thereafter. I can relaunch the console and the same
event occurs. Any help to point me in the direction of resolving this is
appreciated.
Some detail:
CS 16.0
Management and Agent Firewalls are disabled
Can ping the public IP’s assigned to consoles and when console is up I can ping
back to my browsers address
Same symptoms on multiple browser types
Have disabled novnc and have same results
A snippet of the management logs:
2021-12-09 17:54:41,027 DEBUG [c.c.s.ConsoleProxyServlet]
(qtp2063763486-338:null) (logid:) Port info 192.168.110.129
2021-12-09 17:54:41,027 INFO [c.c.s.ConsoleProxyServlet]
(qtp2063763486-338:null) (logid:) Parse host info returned from executing
GetVNCPortCommand. host info: 192.168.110.129
2021-12-09 17:54:41,034 DEBUG [c.c.s.ConsoleProxyServlet]
(qtp2063763486-338:null) (logid:) Compose console url:
//172.16.1.2/resource/noVNC/vnc.html?autoconnect=true=8080=Tg19XlwRmIvDMUgrhjPASFEyJFK8tbCF734GWBjpgssBdqsn1JcBKJe0WfGj9A6Bc8FlCmC5zRf9KcAc9HmC_RnqOjtVgonVYlzAQ0pbBojSLHKKwwv68yXyXyj53CMYxBp75-sY5CZI7VsjEI3oweJyr8Soig4KXfzaVvktLh4orgL44QoR2m_KTElhkt_APmazCoMAFK82Sm3QRrQM9xIDJ5wndoAnBe7d_-qdYa-6EnYMiGp2bAB1GYMF88Wqlm0o98dPstgdNQzZTXEvt_bvKH0lXdRD8VYs-Z8TH1uimKbL_bnjUXuJHm15hdZm
2021-12-09 17:54:41,034 DEBUG [c.c.s.ConsoleProxyServlet]
(qtp2063763486-338:null) (logid:) the console url is ::
v-1-VM
2021-12-09 17:54:41,192 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-9:null) (logid:) SeqA 2-610: Processing Seq 2-610: { Cmd
, MgmtId: -1, via: 2, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleAccessAuthenticationCommand":{"_host":"192.168.110.129","_port":"5900","_vmId":"90037422-3b2e-4d72-bbb9-b55c7621c137","_sid":"6DSI5KehKwLK-4MoVd_LWA","_ticket":"dvOoDSrKGm7MjZjKtyw8r/QynSI=","_isReauthenticating":"false","wait":"0","bypassHostMaintenance":"false"}}]
}
2021-12-09 17:54:41,192 DEBUG [c.c.c.AgentHookBase]
(AgentManager-Handler-9:null) (logid:) Console authentication. Ticket in url
for 192.168.110.129:5900-90037422-3b2e-4d72-bbb9-b55c7621c137 is
dvOoDSrKGm7MjZjKtyw8r/QynSI=
2021-12-09 17:54:41,193 DEBUG [c.c.c.AgentHookBase]
(AgentManager-Handler-9:null) (logid:) Console authentication. Ticket in 1
minute boundary for 192.168.110.129:5900-90037422-3b2e-4d72-bbb9-b55c7621c137
is dvOoDSrKGm7MjZjKtyw8r/QynSI=
2021-12-09 17:54:41,199 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-9:null) (logid:) SeqA 2-610: Sending Seq 2-610: { Ans: ,
MgmtId: 3232263808, via: 2, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.ConsoleAccessAuthenticationAnswer":{"_success":"true","_isReauthenticating":"false","_port":"0","result":"true","wait":"0","bypassHostMaintenance":"false"}}]
}
2021-12-09 17:54:41,205 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-11:null) (logid:) SeqA 2-611: Processing Seq 2-611: {
Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"1","_loadInfo":"{
"connections": [
{
"id": 11,
"clientInfo": "",
"host": "192.168.110.129",
"port": 5900,
"tag": "90037422-3b2e-4d72-bbb9-b55c7621c137",
"createTime": 1639090481205,
"lastUsedTime": 1639090481205
}
]
}","wait":"0","bypassHostMaintenance":"false"}}] }
2021-12-09 17:54:41,234 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-11:null) (logid:) SeqA 2-611: Sending Seq 2-611: { Ans:
, MgmtId: 3232263808, via: 2, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
}
2021-12-09 17:54:42,347 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-6:ctx-a3c35eda) (logid:b26972af) HA health check
task is running...
2021-12-09 17:54:43,004 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-10:null) (logid:) Ping from 3(s-2-VM)
2021-12-09 17:54:43,072 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-12:null) (logid:) Ping from Routing host
1(agent01.cloudnet.xcubia.com)
2021-12-09 17:54:43,072 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-12:null) (logid:) Process host VM state report from ping
process. host: 1
2021-12-09 17:54:43,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-12:null) (logid:) Process VM state report. host: 1,
number of records in report: 3
2021-12-09 17:54:43,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-12:null) (logid:) VM state report. host: 1, vm id: 1,
power state: PowerOn
2021-12-09 17:54:43,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-12:null) (logid:) VM state report. host: 1, vm id: 2,
power state: PowerOn
2021-12-09 17:54:43,087 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-12:null) (logid:) VM state report. host: 1, vm id: 12,
Re: RE: Local-link address change issue
Hi Adam, Already tried restarts. No luck. I did actually "hard code" the ip info in the agent.properties file as suggested here: https://github.com/apache/cloudstack/issues/3488 But still no change. Thanks, Robert On 2020/08/24 08:34:44, Adam Witwicki wrote: > Hi Robert > > I think you may need to restart your agents - the new value should be written > to /etc/cloudstack/agent/agent.properties > > control.cidr= > > Thanks > > Adam > > -Original Message- > From: Robert Ward > Sent: 24 August 2020 04:22 > To: users@cloudstack.apache.org > Subject: Local-link address change issue > > ** This mail originated from OUTSIDE the Oakford corporate network. Treat > hyperlinks and attachments in this email with caution. ** > > Hello all, > > I am attempting to change the KVM System VM's local-link address and am not > having much success. I have changed the global setting: control.cidr (CS > 4.14.0) to both 10.254.0.0/16 and 172.31.0.0/16, restarted the MS, destroyed > and re-created the VM's but I am still stuck in the 169.254.0.0/16 subnet. > Not seeing anything in the logs that would indicate an issue. > > Can anyone suggest where I might be going wrong here? > > Thanks, > > Robert > Disclaimer Notice: > This email has been sent by Oakford Technology Limited, while we have checked > this e-mail and any attachments for viruses, we can not guarantee that they > are virus-free. You must therefore take full responsibility for virus > checking. > This message and any attachments are confidential and should only be read by > those to whom they are addressed. If you are not the intended recipient, > please contact us, delete the message from your computer and destroy any > copies. Any distribution or copying without our prior permission is > prohibited. > Internet communications are not always secure and therefore Oakford > Technology Limited does not accept legal responsibility for this message. The > recipient is responsible for verifying its authenticity before acting on the > contents. Any views or opinions presented are solely those of the author and > do not necessarily represent those of Oakford Technology Limited. > Registered address: Oakford Technology Limited, The Manor House, Potterne, > Wiltshire. SN10 5PN. > Registered in England and Wales No. 5971519 > >
Local-link address change issue
Hello all, I am attempting to change the KVM System VM's local-link address and am not having much success. I have changed the global setting: control.cidr (CS 4.14.0) to both 10.254.0.0/16 and 172.31.0.0/16, restarted the MS, destroyed and re-created the VM's but I am still stuck in the 169.254.0.0/16 subnet. Not seeing anything in the logs that would indicate an issue. Can anyone suggest where I might be going wrong here? Thanks, Robert
Re: [VOTE] Apache CloudStack 4.14.0.0 RC1
Hi Liridon,
I think CS has the wrong timezone because the default project is set to
GMT/UTC. I believe you can change that in the global settings. I have the same
situation but my management logs have the correct timestamp.
Thanks,
Robert
On 2020/05/01 17:52:46, "Ismaili, Liridon (SWISS TXT)"
wrote:
> @Robert I did add the default-time-zone parameter to the my.cnf file but the
> dates inside CloudStack are still in UTC format. select now() however gives
> me the correct output:
> mysql> select now();
> +-+
> | now() |
> +-+
> | 2020-05-01 19:48:38 |
> +-+
> 1 row in set (0.00 sec)
> Does anyone else have such wrong dates? All tables in the cloud DB seems to
> be affected.
> Regards Liridon
>
> -Original Message-
> From: Andrija Panic
> mailto:andrija%20panic%20%3candrija.pa...@gmail.com%3e>>
> Reply-To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
> To: users
> mailto:users%20%3cus...@cloudstack.apache.org%3e>>,
> Rohit Yadav
> mailto:rohit%20yadav%20%3crohit.ya...@shapeblue.com%3e>>
> Subject: Re: [VOTE] Apache CloudStack 4.14.0.0 RC1
> Date: Fri, 01 May 2020 19:08:16 +0200
>
>
> @Rohit Yadav <
>
> <mailto:rohit.ya...@shapeblue.com>
>
> rohit.ya...@shapeblue.com
>
> > can you possibly advice on the
>
> time zone issue? I've seen this on another ML thread as well. We are mostly
>
> in UTC (test envs) so this might be the reason we didn't see this...could
>
> be just a documentation update that is needed.
>
>
> @Robert is there any specifics to your NFS server (i.e. a forbidden NFSv3
>
> or similar)? Also, can you please open a GitHub issue and provide the same
>
> there? So we can track it and collaborate - I've not seen this one before.
>
> What packages did you use for the installation? On the issue you'll raise,
>
> please also include the relevant output from the /var/log/cloud.log from
>
> inside the SSVM
>
> Also not sure what is the relation between MySQL and JRE at all - they
>
> should have nothing in common?
>
>
> Thanks!
>
>
> Andrija
>
>
>
> On Fri, 1 May 2020 at 17:49, Robert Ward <
>
> <mailto:rww...@gmail.com>
>
> rww...@gmail.com
>
> > wrote:
>
>
> Hi all,
>
>
> I too have run into an issues while performing a clean install:
>
> Centos 7, MySQL 5.6, JRE 11
>
>
> JRE 11 and MySQL 5.6 don't see to play well on two points:
>
>
> Installing JRE before MySQL causes MYSQL to "lockup" on startup. I have
>
> found that if I install JRE after MySQL that issue is resolved.
>
>
> MySQL chokes with timezone error @ cloudstack-setup-management. In my case
>
> MySQL doesn't like "EDT" as a timezone. Resolved by adding this to
>
> etc/my.cnf:
>
>
> default-time-zone= "-05:00"
>
>
> On my last point CS has difficulty with mounting secondary storage. I have
>
> clipped the logs to the point where I think the problem lies:
>
>
> 2020-05-01 10:55:07,871 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>
> (StatsCollector-5:ctx-35c593ba) (logid:aad9ed2a) getCommandHostDelegation:
>
> class com.cloud.agent.api.GetStorageStatsCommand
>
> 2020-05-01 10:55:07,871 DEBUG [c.c.h.XenServerGuru]
>
> (StatsCollector-5:ctx-35c593ba) (logid:aad9ed2a) We are returning the
>
> default host to execute commands because the command is not of Copy type.
>
> 2020-05-01 10:55:07,910 DEBUG [c.c.a.t.Request]
>
> (AgentManager-Handler-8:null) (logid:) Seq 2-4356951164504244991:
>
> Processing: { Ans: , MgmtId: 144345337593, via: 2, Ver: v1, Flags: 10,
>
> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
>
> GetRootDir for nfs://192.168.25.1/export/secondary failed due to
>
> com.cloud.utils.exception.CloudRuntimeException: Unable to mount
>
> 192.168.25.1:/export/secondary at
>
> /mnt/SecStorage/b7e9e158-ed80-3a62-a5d7-1992c991d829 due to mount.nfs:
>
> requested NFS version or transport protocol is not supported\n\tat
>
> org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.getRootDir(NfsSecondaryStorageResource.java:2458)\n\tat
>
> org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:2208)\n\tat
>
> org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:292)\n\tat
>
> com.cloud.storage.resource.PremiumSec
Re: [VOTE] Apache CloudStack 4.14.0.0 RC1
Hi Andrija,
On the JRE/MySQL/Timezone issues these are more observations than anything
(comparing to my experience with fresh installs of 4.13). These may be more an
install documentation note than anything.
As for NFS issue: This install follows my tried and true practices for 4.13
installs using no variances from recommend steps in 4.13
documentationmeaning I'm not doing anything fancy :>)
Since I'm relatively new at testing feedback I will do as you requested and put
everything in GitHub but please pardon any rookie mistakes in advance.
Thanks for you quick reply.
Robert
On 2020/05/01 17:08:16, Andrija Panic wrote:
> @Rohit Yadav can you possibly advice on the
> time zone issue? I've seen this on another ML thread as well. We are mostly
> in UTC (test envs) so this might be the reason we didn't see this...could
> be just a documentation update that is needed.
>
> @Robert is there any specifics to your NFS server (i.e. a forbidden NFSv3
> or similar)? Also, can you please open a GitHub issue and provide the same
> there? So we can track it and collaborate - I've not seen this one before.
> What packages did you use for the installation? On the issue you'll raise,
> please also include the relevant output from the /var/log/cloud.log from
> inside the SSVM
> Also not sure what is the relation between MySQL and JRE at all - they
> should have nothing in common?
>
> Thanks!
>
> Andrija
>
>
> On Fri, 1 May 2020 at 17:49, Robert Ward wrote:
>
> > Hi all,
> >
> > I too have run into an issues while performing a clean install:
> > Centos 7, MySQL 5.6, JRE 11
> >
> > JRE 11 and MySQL 5.6 don't see to play well on two points:
> >
> > Installing JRE before MySQL causes MYSQL to "lockup" on startup. I have
> > found that if I install JRE after MySQL that issue is resolved.
> >
> > MySQL chokes with timezone error @ cloudstack-setup-management. In my case
> > MySQL doesn't like "EDT" as a timezone. Resolved by adding this to
> > etc/my.cnf:
> >
> > default-time-zone= "-05:00"
> >
> > On my last point CS has difficulty with mounting secondary storage. I have
> > clipped the logs to the point where I think the problem lies:
> >
> > 2020-05-01 10:55:07,871 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > (StatsCollector-5:ctx-35c593ba) (logid:aad9ed2a) getCommandHostDelegation:
> > class com.cloud.agent.api.GetStorageStatsCommand
> > 2020-05-01 10:55:07,871 DEBUG [c.c.h.XenServerGuru]
> > (StatsCollector-5:ctx-35c593ba) (logid:aad9ed2a) We are returning the
> > default host to execute commands because the command is not of Copy type.
> > 2020-05-01 10:55:07,910 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-8:null) (logid:) Seq 2-4356951164504244991:
> > Processing: { Ans: , MgmtId: 144345337593, via: 2, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> > GetRootDir for nfs://192.168.25.1/export/secondary failed due to
> > com.cloud.utils.exception.CloudRuntimeException: Unable to mount
> > 192.168.25.1:/export/secondary at
> > /mnt/SecStorage/b7e9e158-ed80-3a62-a5d7-1992c991d829 due to mount.nfs:
> > requested NFS version or transport protocol is not supported\n\tat
> > org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.getRootDir(NfsSecondaryStorageResource.java:2458)\n\tat
> > org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:2208)\n\tat
> > org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:292)\n\tat
> > com.cloud.storage.resource.PremiumSecondaryStorageResource.defaultAction(PremiumSecondaryStorageResource.java:64)\n\tat
> > com.cloud.storage.resource.PremiumSecondaryStorageResource.executeRequest(PremiumSecondaryStorageResource.java:60)\n\tat
> > com.cloud.agent.Agent.processRequest(Agent.java:644)\n\tat
> > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1057)\n\tat
> > com.cloud.utils.nio.Task.call(Task.java:83)\n\tat
> > com.cloud.utils.nio.Task.call(Task.java:29)\n\tat
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat
> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat
> > java.base/java.lang.Thread.run(Thread.java:834)\n","wait":0}}] }
> >
> > My troubleshooting steps:
> >
> > restart SSVM
> > restart MS
> > restart agent
> &
Re: [VOTE] Apache CloudStack 4.14.0.0 RC1
Hi all,
I too have run into an issues while performing a clean install:
Centos 7, MySQL 5.6, JRE 11
JRE 11 and MySQL 5.6 don't see to play well on two points:
Installing JRE before MySQL causes MYSQL to "lockup" on startup. I have found
that if I install JRE after MySQL that issue is resolved.
MySQL chokes with timezone error @ cloudstack-setup-management. In my case
MySQL doesn't like "EDT" as a timezone. Resolved by adding this to etc/my.cnf:
default-time-zone= "-05:00"
On my last point CS has difficulty with mounting secondary storage. I have
clipped the logs to the point where I think the problem lies:
2020-05-01 10:55:07,871 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(StatsCollector-5:ctx-35c593ba) (logid:aad9ed2a) getCommandHostDelegation:
class com.cloud.agent.api.GetStorageStatsCommand
2020-05-01 10:55:07,871 DEBUG [c.c.h.XenServerGuru]
(StatsCollector-5:ctx-35c593ba) (logid:aad9ed2a) We are returning the default
host to execute commands because the command is not of Copy type.
2020-05-01 10:55:07,910 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:null)
(logid:) Seq 2-4356951164504244991: Processing: { Ans: , MgmtId: 144345337593,
via: 2, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
GetRootDir for nfs://192.168.25.1/export/secondary failed due to
com.cloud.utils.exception.CloudRuntimeException: Unable to mount
192.168.25.1:/export/secondary at
/mnt/SecStorage/b7e9e158-ed80-3a62-a5d7-1992c991d829 due to mount.nfs:
requested NFS version or transport protocol is not supported\n\tat
org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.getRootDir(NfsSecondaryStorageResource.java:2458)\n\tat
org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:2208)\n\tat
org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:292)\n\tat
com.cloud.storage.resource.PremiumSecondaryStorageResource.defaultAction(PremiumSecondaryStorageResource.java:64)\n\tat
com.cloud.storage.resource.PremiumSecondaryStorageResource.executeRequest(PremiumSecondaryStorageResource.java:60)\n\tat
com.cloud.agent.Agent.processRequest(Agent.java:644)\n\tat
com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1057)\n\tat
com.cloud.utils.nio.Task.call(Task.java:83)\n\tat
com.cloud.utils.nio.Task.call(Task.java:29)\n\tat
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat
java.base/java.lang.Thread.run(Thread.java:834)\n","wait":0}}] }
My troubleshooting steps:
restart SSVM
restart MS
restart agent
verify I can manually mount secondary storage on agent and navigate through all
directories.
change secondary storage version from null to V4 in CS global settings
Any input is greatly appreciated on how to resolve.
Thanks,
Robert
On 2020/04/29 14:38:44, Andrija Panic wrote:
> Hi All,
>
> I've created a 4.14.0.0 release (RC1), with the following artefacts up for
> testing and a vote:
>
> Git Branch and Commit SH:
> https://gitbox.apache.org/repos/asf?p=cloudstack.git;a=shortlog;h=refs/heads/4.14.0.0-RC20200429T1355
> Commit: 2c39b7180a9fb40cbdcad5e6a58be64a86913771
>
> Source release (checksums and signatures are available at the same
> location):
> https://dist.apache.org/repos/dist/dev/cloudstack/4.14.0.0/
>
> PGP release keys (signed using 3DC01AE8):
> https://dist.apache.org/repos/dist/release/cloudstack/KEYS
>
> The vote will be open until Wednesday 06.May.2020.
>
> For sanity in tallying the vote, can PMC members please be sure to indicate
> "(binding)" with their vote?
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove (and reason why)
>
> Additional information:
>
> For users' convenience, I've built packages from
> 2c39b7180a9fb40cbdcad5e6a58be64a86913771 and published RC1 repository here
> (CentOS 7 and Debian/generic):
> http://packages.shapeblue.com/testing/41400rc1/
> and here (Ubuntu 18.04 specific, thanks to Gabriel):
> https://download.cloudstack.org/testing/4.14.0.0-RC20200429T1355/ubuntu/bionic/
>
> The release notes are still work-in-progress, but for the upgrade
> instructions (including new systemVM templates) you may refer the following
> URL:
> https://acs-www.shapeblue.com/docs/WIP-PROOFING/pr87/upgrading/index.html
>
> 4.14.0.0 systemVM templates are available from here:
> http://download.cloudstack.org/systemvm/4.14/
>
> Regards,
> Andrija Panić
>
Re: VM SSL error caught during wrap data
Hi Richard, New install...was working on a new layout with the mngt, primary, and secondary storage networks on their own interfaces/subnets. 30.2 = mngt server 30.53 = systemvm 30.58 = storagevm It started with troubleshooting why ports 3922/443 would not listen. In one of the attempts to troubleshoot I deleted and rebuilt both vm's and that's when the ssl issue raised it's ugly head. I found that rebooting both the mngt and agent servers (at the same time) fixed the problem but thanks so much for the tip! I'll add that to my troubleshooting bag. Thanks, Robert On 2020/02/06 09:54:24, Richard Lawley wrote: > What did you do leading up to this problem? Is this a new > install/upgrade? If upgrade, from what to what? I presume 192.168.30.2 is > your mgmt server - what are 192.168.30.53/.58? > > You can temporarily disable strictness which is akin to disabling SSL > validation by changing the setting ca.plugin.root.auth.strictness to false > (no restart needed when setting to false, restarted needed when setting it > back to true). I've seen this problem before when my hosts or system VMs > were connecting to the mgmt server via NAT - the IP they appear to connect > from does not match the IP in the certificate they're presenting. > > Regards, > > Richard > > On Wed, 5 Feb 2020 at 17:06, Robert Ward wrote: > > > Hello all, > > > > I have been struggling with this. > > > > 2020-02-04 23:59:53,904 ERROR [c.c.u.n.Link] > > (AgentManager-SSLHandshakeHandler-4:null) (logid:) SSL error caught during > > wrap data: null cert chain, for local address=/192.168.30.2:8250, remote > > address=/192.168.30.53:49126. > > 2020-02-04 23:59:53,928 ERROR [c.c.u.n.Link] > > (AgentManager-SSLHandshakeHandler-7:null) (logid:) SSL error caught during > > wrap data: null cert chain, for local address=/192.168.30.2:8250, remote > > address=/192.168.30.58:35316. > > > > I've tried all my troubleshooting bag of tricks but have come up empty. > > Can someone enlighten me on how to resolve this? > > > > Thanks, > > > > Robert > > > > > > > > >
VM SSL error caught during wrap data
Hello all, I have been struggling with this. 2020-02-04 23:59:53,904 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-4:null) (logid:) SSL error caught during wrap data: null cert chain, for local address=/192.168.30.2:8250, remote address=/192.168.30.53:49126. 2020-02-04 23:59:53,928 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-7:null) (logid:) SSL error caught during wrap data: null cert chain, for local address=/192.168.30.2:8250, remote address=/192.168.30.58:35316. I've tried all my troubleshooting bag of tricks but have come up empty. Can someone enlighten me on how to resolve this? Thanks, Robert
Re: Update for those having database upgrade issues recently
Since this issue is causing a lot of consternation among users I thought I would add this to help. Instead of using wget or pip to downgrade mysql or the faulty modules I suggest you do this after you have installed Mysql and CS but before you run the setup databases script (if you downgrade after running the script it will fail the DB upgrade process during CS setup): yum downgrade mysql-connector-python yum downgrade mysql-connector-java This should downgrade the modules to 8.0.18 and 5.1.25 respectively. On 2020/01/23 03:41:33, Robert Ward wrote: > Hello, > > If you have been one of the unfortunate ones with the recent database upgrade > problem I have stumbled across a possible solution...at least in my case. > > Looking a little closer at the logs it seems there was some sort of java > disconnect triggering the DB upgrade issue so I decided to downgrade the > mysql-connector-java module. In my case that did the trick! Everything seems > to working ok so far. > > BTW - I am running mysql-connector-python-8.0.19 so that may have not been > the culprit after all. > > Robert >
Update for those having database upgrade issues recently
Hello, If you have been one of the unfortunate ones with the recent database upgrade problem I have stumbled across a possible solution...at least in my case. Looking a little closer at the logs it seems there was some sort of java disconnect triggering the DB upgrade issue so I decided to downgrade the mysql-connector-java module. In my case that did the trick! Everything seems to working ok so far. BTW - I am running mysql-connector-python-8.0.19 so that may have not been the culprit after all. Robert
Re: Management server setup issue
Hi Andrija,
You were spot on about the mysql-connector-python issue. A critical bug was
raised on 8.0.19 2 days ago concerning the issue.
Following your instructions, I went back to 8.0.18 and
cloudstack-setup-management completes as expected however a new issue has
popped up. It seems there is an issue when CSMS attempts to upgrade the
database. I am hoping you can pinpoint what the issue is and provide a
recommendation.
020-01-17 14:35:36,858 INFO [o.e.j.s.h.ContextHandler] (Thread-1:null)
(logid:) Stopped
o.e.j.w.WebAppContext@6e38921c{/client,null,UNAVAILABLE}{/usr/share/cloudstack-management/webapp}
2020-01-17 14:35:38,419 INFO [o.a.c.ServerDaemon] (main:null) (logid:) Server
configuration file found: /etc/cloudstack/management/server.properties
2020-01-17 14:35:38,430 INFO [o.a.c.ServerDaemon] (main:null) (logid:)
Initializing server daemon on null, with http.enable=true, http.port=8080,
https.enable=false, https.port=8443, context.path=/client
2020-01-17 14:35:38,443 INFO [o.e.j.u.log] (main:null) (logid:) Logging
initialized @1005ms to org.eclipse.jetty.util.log.Slf4jLog
2020-01-17 14:35:38,684 INFO [o.e.j.s.Server] (main:null) (logid:)
jetty-9.4.z-SNAPSHOT, build timestamp: 2017-11-21T16:27:37-05:00, git hash:
82b8fb23f757335bb3329d540ce37a2a2615f0a8
2020-01-17 14:35:38,715 INFO [o.e.j.s.AbstractNCSARequestLog] (main:null)
(logid:) Opened /var/log/cloudstack/management/access.log
2020-01-17 14:35:38,812 INFO [o.e.j.w.StandardDescriptorProcessor] (main:null)
(logid:) NO JSP Support for /client, did not find
org.eclipse.jetty.jsp.JettyJspServlet
2020-01-17 14:35:38,841 INFO [o.e.j.s.session] (main:null) (logid:)
DefaultSessionIdManager workerName=node0
2020-01-17 14:35:38,841 INFO [o.e.j.s.session] (main:null) (logid:) No
SessionScavenger set, using defaults
2020-01-17 14:35:38,843 INFO [o.e.j.s.session] (main:null) (logid:) Scavenging
every 60ms
2020-01-17 14:35:54,716 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Loading module context [bootstrap] from URL
[jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.13.0.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context.xml]
2020-01-17 14:35:54,716 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Loading module context [bootstrap] from URL
[jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.13.0.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml]
2020-01-17 14:35:54,959 DEBUG [c.c.u.c.EncryptionSecretKeyChecker] (main:null)
(logid:) Encryption Type: file
2020-01-17 14:35:55,013 INFO [c.c.u.LogUtils] (main:null) (logid:) log4j
configuration found at /etc/cloudstack/management/log4j-cloud.xml
2020-01-17 14:35:55,046 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Loaded module context [bootstrap] in 331 ms
2020-01-17 14:35:55,047 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: bootstrap
2020-01-17 14:35:55,047 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: system
2020-01-17 14:35:55,047 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: core
2020-01-17 14:35:55,048 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: allocator
2020-01-17 14:35:55,048 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: host-allocator-random
2020-01-17 14:35:55,048 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: planner
2020-01-17 14:35:55,048 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: api-planner
2020-01-17 14:35:55,048 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: baremetal-planner
2020-01-17 14:35:55,048 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: explicit-dedication
2020-01-17 14:35:55,049 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: host-affinity
2020-01-17 14:35:55,049 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: host-anti-affinity
2020-01-17 14:35:55,049 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: implicit-dedication
2020-01-17 14:35:55,049 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: server-planner
2020-01-17 14:35:55,049 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: skip-heurestics
2020-01-17 14:35:55,049 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:) Module Hierarchy: user-concentrated-pod
2020-01-17 14:35:55,050 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(main:null) (logid:)
Management server setup issue
Hello all, I have run into a problem with the CS MGNT setup. When I run cloudstack-setup-management command I get back: Traceback (most recent call last): File "/usr/bin/cloudstack-setup-management", line 24, in from cloudutils.serviceConfigServer import cloudManagementConfig File "/usr/lib64/python2.7/site-packages/cloudutils/serviceConfigServer.py", line 17, in from db import Database File "/usr/lib64/python2.7/site-packages/cloudutils/db.py", line 20, in import mysql.connector File "/usr/lib64/python2.7/site-packages/mysql/connector/__init__.py", line 41, in import dns.resolver ImportError: No module named dns.resolver I have installed the Management server in the past without any issue and this just started happening recently. FYI, my core setup: Centos 7.7 MySQL 5.7 CS 4.13 I have used both the CS and SB repos and I get the same result. Any help with solving this is most appreciated.
