Hi Antonine, >I also tried to define the storage traffic type with VLAN 53; the VLAN/VNI >column shows blank,
I see the same issue with CS 4.18 too, this appears to be a bug could you report this in github? In my testing the VLAN is present in cloud. dc_storage_network_ip_range record and the same is present in the API response for API listStorageNetworkIpRange as well. Just that it is not displayed in the UI. Could you try removing the VLAN tag 53 from the bridge Cloudbr53 and let cloudstack configure the storage VLAN? >Is the storage definition only or mainly used for the SSVM? Only for SSVM. -Jithin From: Antoine Boucher <[email protected]> Date: Wednesday, 21 June 2023 at 12:14 AM To: [email protected] <[email protected]>, users <[email protected]> Subject: Re: SSVM routing issue Hi Stanley, You will find the answers below. Antoine Boucher [email protected] [o] +1-226-505-9734 www.haltondc.com<http://www.haltondc.com> “Data security made simple” > On May 24, 2023, at 8:59 PM, Stanley Burkee <[email protected]> wrote: > > Hi Antoine, > > > Please share the cloudstack version you are using. Also check if you have > connectivity between your management network & storage network. 4.17.2.0 > > Please share the management server logs & your zone cloudbr0 & other > interfaces configurations. Here is my CentOS network config: (Management Server and Some Clusters) [root@nimbus network-scripts]# cat ifcfg* DEVICE=bond0 ONBOOT=yes BONDING_OPTS="mode=6" BRIDGE=cloudbr0 NM_CONTROLLED=no DEVICE=bond0.53 VLAN=yes BOOTPROTO=static ONBOOT=yes TYPE=Unknown BRIDGE=cloudbr53 DEVICE=bond1 ONBOOT=yes BONDING_OPTS="mode=6" BRIDGE=cloudbr1 NM_CONTROLLED=no DEVICE=cloudbr0 ONBOOT=yes TYPE=Bridge IPADDR=10.101.2.40 NETMASK=255.255.252.0 GATEWAY=10.101.0.1 DOMAIN="haltondc.net" DEFROUTE=yes NM_CONTROLLED=no DELAY=0 DEVICE=cloudbr1 ONBOOT=yes TYPE=Bridge NM_CONTROLLED=no DELAY=0 DEVICE=cloudbr53 ONBOOT=yes TYPE=Bridge VLAN=yes IPADDR=10.101.6.40 #GATEWAY=10.101.6.1 NETMASK=255.255.254.0 NM_CONTROLLED=no DELAY=0 DEVICE=eno1 TYPE=Ethernet USERCTL=no MASTER=bond1 SLAVE=yes BOOTPROTO=none NM_CONTROLLED=no ONBOOT=yes DEVICE=eno2 TYPE=Ethernet USERCTL=no MASTER=bond1 SLAVE=yes BOOTPROTO=none NM_CONTROLLED=no ONBOOT=yes DEVICE=eno3 TYPE=Ethernet USERCTL=no MASTER=bond1 SLAVE=yes BOOTPROTO=none NM_CONTROLLED=no ONBOOT=yes DEVICE=eno4 TYPE=Ethernet USERCTL=no #MASTER=bond1 #SLAVE=yes #BOOTPROTO=none NM_CONTROLLED=no ONBOOT=no DEVICE=ens2f0 TYPE=Ethernet USERCTL=no MASTER=bond0 SLAVE=yes BOOTPROTO=none NM_CONTROLLED=no ONBOOT=yes DEVICE=ens2f1 TYPE=Ethernet USERCTL=no MASTER=bond0 SLAVE=yes BOOTPROTO=none NM_CONTROLLED=no ONBOOT=yes DEVICE=lo IPADDR=127.0.0.1 NETMASK=255.0.0.0 NETWORK=127.0.0.0 # If you're having problems with gated making 127.0.0.0/8 a martian, # you can change this to something else (255.255.255.255, for example) BROADCAST=127.255.255.255 ONBOOT=yes NAME=loopback Here is my Ubuntu 20/22 network config: (Most Clusters) root@cs-kvm01:~# cat /etc/netplan/00-installer-config.yaml network: version: 2 ethernets: eno1: {} eno2: {} ens2f0: mtu: 1500 ens2f1: mtu: 1500 bonds: bond0: interfaces: - ens2f0 - ens2f1 mtu: 1500 parameters: mode: balance-alb bond1: interfaces: - eno1 - eno2 nameservers: addresses: [] search: [] parameters: mode: balance-alb vlans: bond0.53: id: 53 link: bond0 mtu: 1500 bridges: cloudbr0: interfaces: [bond0] mtu: 1500 addresses: - 10.101.2.42/22 gateway4: 10.101.0.1 nameservers: addresses: - 10.101.0.1 search: - haltondc.net - haltondc.com dhcp4: no dhcp6: no cloudbr1: interfaces: [bond1] mtu: 1500 dhcp4: no dhcp6: no cloudbr53: interfaces: [bond0.53] mtu: 1500 addresses: - 10.101.6.42/23 dhcp4: no dhcp6: no > > Thanks. > > Best Regards > Stanley > > On Mon, 15 May 2023, 6:00 pm Antoine Boucher, <[email protected] > <mailto:[email protected]>> wrote: > >> Hello, >> >> Would anyone have clues on my on going SSVM issue below? >> >> However, I can work around the issue by deleting my Storage Network >> traffic definition and recreating the SSVM.. >> >> What would be the impact of deleting the Storage Network traffic >> definition on other part of the system? My Primary Storage configuration >> seems to all be done part of my hosts static configuration. >> >> Regards, >> Antoine >> >> >>> On May 11, 2023, at 10:27 AM, Antoine Boucher <[email protected]> >> wrote: >>> >>> Good morning/afternoon/evening, >>> >>> I am following up with my SSVM routing issue when a Storage Network is >> defined. >>> >>> I have a zone with Xen and KVM servers that have a Storage Network >> defined as Cloudbr53 with a storage network-specific subnet (Cloudbr0 is >> also defined for Management and Cloudbr1 for Guests) >>> >>> The Cloudbr53 bridge is “hard coded” to VLAN 53 on all hosts within the >> specific storage ip subnet range. The Storage traffic type for the Zone is >> defined with Cloudbr53 and VLAN as blank. >>> >>> You will see that the storage network route on the SSVM is pointed to >> the wrong eth1 interface as it should be eth3 >>> >>> 10.101.6.0 cloudrouter01.n 255.255.254.0 UG 0 0 0 eth1 >>> >>> root@s-394-VM:~# route >>> Kernel IP routing table >>> Destination Gateway Genmask Flags Metric Ref Use Iface >>> default 148.59.36.49 0.0.0.0 UG 0 0 0 eth2 >>> 10.0.0.0 cloudrouter01.n 255.0.0.0 UG 0 0 0 eth1 >>> 10.91.0.0 cloudrouter01.n 255.255.254.0 UG 0 0 0 eth1 >>> 10.91.6.0 cloudrouter01.n 255.255.255.0 UG 0 0 0 eth1 >>> 10.101.0.0 0.0.0.0 255.255.252.0 U 0 0 0 eth1 >>> nimbus.haltondc 10.101.6.1 255.255.255.255 UGH 0 0 0 eth3 >>> 10.101.6.0 cloudrouter01.n 255.255.254.0 UG 0 0 0 eth1 >>> 148.59.36.48 0.0.0.0 255.255.255.240 U 0 0 0 eth2 >>> link-local 0.0.0.0 255.255.0.0 U 0 0 0 eth0 >>> 172.16.0.0 cloudrouter01.n 255.240.0.0 UG 0 0 0 eth1 >>> 192.168.0.0 cloudrouter01.n 255.255.0.0 UG 0 0 0 eth1 >>> >>> >>> I also tried to define the storage traffic type with VLAN 53; the >> VLAN/VNI column shows blank, but It looks to be changing the routing to >> eth3; however, I experienced the same overall communication issue. When >> communicating to the management network is from the source IP on the >> storage network and dies coming back since I have no routing between the >> two networks. >>> >>> However, as a workaround, if I remove the storage traffic definition on >> the Zone, all traffic will be routed through the management network. All is >> well if I allow my secondary storage (NFS) on the management network. >>> >>> >>> >>> I’m using the host-configured “storage network” for primary storage on >> all my Zones without issues. >>> >>> What would be the potential issues of deleting the Storage Network >> definition traffic type in my zones, assuming I would keep all my secondary >> storage on or accessible on the management network and recreating the SSVMs? >>> >>> Is the storage definition only or mainly used for the SSVM? >>> >>> Regards, >>> Antoine >>> >>> >>> Confidentiality Warning: This message and any attachments are intended >> only for the use of the intended recipient(s), are confidential, and may be >> privileged. If you are not the intended recipient, you are hereby notified >> that any review, retransmission, conversion to hard copy, copying, >> circulation or other use of this message and any attachments is strictly >> prohibited. If you are not the intended recipient, please notify the sender >> immediately by return e-mail, and delete this message and any attachments >> from your system. >>> >>> >>>> On Feb 28, 2023, at 11:39 AM, Antoine Boucher <[email protected]> >> wrote: >>>> >>>> # root@s-340-VM:~# cat /var/cache/cloud/cmdline >>>> >>>> template=domP type=secstorage host=10.101.2.40 port=8250 name=s-340-VM >> zone=1 pod=1 guid=s-340-VM workers=5 authorized_key=**** >>>> resource=com.cloud.storage.resource.PremiumSecondaryStorageResource >> instance=SecStorage sslcopy=true role=templateProcessor mtu=1500 >> eth2ip=148.59.36.60 eth2mask=255.255.255.240 gateway=148.59.36.49 >> public.network.device=eth2 eth0ip=169.254.211.29 eth0mask=255.255.0.0 >> eth1ip=10.101.3.231 eth1mask=255.255.252.0 mgmtcidr=10.101.0.0/22 >> localgw=10.101.0.1 private.network.device=eth1 eth3ip=10.101.7.212 >> eth3mask=255.255.254.0 storageip=10.101.7.212 storagenetmask=255.255.254.0 >> storagegateway=10.101.6.1 internaldns1=10.101.0.1 dns1=1.1.1.1 dns2=8.8.8.8 >> nfsVersion=null keystore_password=***** >>>> >>>> >>>> # cat /var/log/cloudstack/management/management-server.log.2023-02-*.gz >> | zgrep SecStorageSetupCommand >>>> >>>> 2023-02-18 14:35:38,699 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-290:ctx-cf94f90e) (logid:6dc1b961) Seq >> 47-6546545008336437249: Sending { Cmd , MgmtId: 130593671224, via: >> 47(s-292-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.91.6.5/volume1/ACS_Backup06","_role":"Image"}},"secUrl":"nfs:// >> 10.91.6.5/volume1/ACS_Backup06","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-18 14:35:42,024 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentConnectTaskPool-290:ctx-cf94f90e) (logid:6dc1b961) Details from >> executing class com.cloud.agent.api.SecStorageSetupCommand: success >>>> 2023-02-20 11:12:33,345 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-4:ctx-3b91dfcf) (logid:50ea75a6) Seq >> 43-719450040472436737: Sending { Cmd , MgmtId: 130593671224, via: >> 43(s-289-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.101.6.40/export/secondary","_role":"Image"}},"secUrl":"nfs:// >> 10.101.6.40/export/secondary","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-20 11:12:34,389 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-5:ctx-5834440f) (logid:2d16c04c) Seq >> 47-4507540277044445185: Sending { Cmd , MgmtId: 130593671224, via: >> 47(s-292-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.91.6.5/volume1/ACS_Backup06","_role":"Image"}},"secUrl":"nfs:// >> 10.91.6.5/volume1/ACS_Backup06","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-20 11:12:37,406 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentConnectTaskPool-5:ctx-5834440f) (logid:2d16c04c) Details from >> executing class com.cloud.agent.api.SecStorageSetupCommand: success >>>> 2023-02-20 11:59:14,400 WARN [c.c.a.m.AgentAttache] >> (AgentConnectTaskPool-4:ctx-3b91dfcf) (logid:50ea75a6) Seq >> 43-719450040472436737: Timed out on Seq 43-719450040472436737: { Cmd , >> MgmtId: 130593671224, via: 43(s-289-VM), Ver: v1, Flags: 10100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.101.6.40/export/secondary","_role":"Image"}},"secUrl":"nfs:// >> 10.101.6.40/export/secondary","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-20 12:25:40,060 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-16:ctx-236c5a52) (logid:e701c43f) Seq >> 48-8498011021871415297: Sending { Cmd , MgmtId: 130593671224, via: >> 48(s-311-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.101.6.40/export/secondary","_role":"Image"}},"secUrl":"nfs:// >> 10.101.6.40/export/secondary","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-20 12:25:43,138 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentConnectTaskPool-16:ctx-236c5a52) (logid:e701c43f) Details from >> executing class com.cloud.agent.api.SecStorageSetupCommand: success >>>> 2023-02-20 12:25:43,159 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-16:ctx-236c5a52) (logid:e701c43f) Seq >> 48-8498011021871415298: Sending { Cmd , MgmtId: 130593671224, via: >> 48(s-311-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.101.6.23/mnt/Store08/CSBackup08","_role":"Image"}},"secUrl":"nfs:// >> 10.101.6.23/mnt/Store08/CSBackup08","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-20 12:25:45,730 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentConnectTaskPool-16:ctx-236c5a52) (logid:e701c43f) Details from >> executing class com.cloud.agent.api.SecStorageSetupCommand: success >>>> 2023-02-20 12:53:59,308 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-10:ctx-bf90573f) (logid:dc87251d) Seq >> 48-3231051257661620225: Sending { Cmd , MgmtId: 130593671224, via: >> 48(s-311-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.101.6.40/export/secondary","_role":"Image"}},"secUrl":"nfs:// >> 10.101.6.40/export/secondary","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-20 12:54:01,842 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentConnectTaskPool-10:ctx-bf90573f) (logid:dc87251d) Details from >> executing class com.cloud.agent.api.SecStorageSetupCommand: success >>>> 2023-02-20 12:54:01,871 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-10:ctx-bf90573f) (logid:dc87251d) Seq >> 48-3231051257661620226: Sending { Cmd , MgmtId: 130593671224, via: >> 48(s-311-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.101.6.23/mnt/Store08/CSBackup08","_role":"Image"}},"secUrl":"nfs:// >> 10.101.6.23/mnt/Store08/CSBackup08","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-20 12:54:04,295 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentConnectTaskPool-10:ctx-bf90573f) (logid:dc87251d) Details from >> executing class com.cloud.agent.api.SecStorageSetupCommand: success >>>> 2023-02-22 15:23:33,561 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-23:ctx-22a41bf8) (logid:894a50f0) Seq >> 50-2542563464627355649: Sending { Cmd , MgmtId: 130593671224, via: >> 50(s-324-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.91.6.5/volume1/ACS_Backup06","_role":"Image"}},"secUrl":"nfs:// >> 10.91.6.5/volume1/ACS_Backup06","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-22 15:23:36,815 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentConnectTaskPool-23:ctx-22a41bf8) (logid:894a50f0) Details from >> executing class com.cloud.agent.api.SecStorageSetupCommand: success >>>> 2023-02-26 14:46:39,737 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-26:ctx-50d91205) (logid:4c7783a0) Seq >> 52-8409064929230848001: Sending { Cmd , MgmtId: 130593671224, via: >> 52(s-339-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.101.6.40/export/secondary","_role":"Image"}},"secUrl":"nfs:// >> 10.101.6.40/export/secondary","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-26 14:46:42,926 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentConnectTaskPool-26:ctx-50d91205) (logid:4c7783a0) Details from >> executing class com.cloud.agent.api.SecStorageSetupCommand: success >>>> 2023-02-26 14:46:42,945 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-26:ctx-50d91205) (logid:4c7783a0) Seq >> 52-8409064929230848002: Sending { Cmd , MgmtId: 130593671224, via: >> 52(s-339-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.101.6.23/mnt/Store08/CSBackup08","_role":"Image"}},"secUrl":"nfs:// >> 10.101.6.23/mnt/Store08/CSBackup08","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-26 14:46:45,435 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentConnectTaskPool-26:ctx-50d91205) (logid:4c7783a0) Details from >> executing class com.cloud.agent.api.SecStorageSetupCommand: success >>>> 2023-02-26 15:07:11,934 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-27:ctx-8f2e92c1) (logid:4a947f5e) Seq >> 52-7356067041356283905: Sending { Cmd , MgmtId: 130593671224, via: >> 52(s-339-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.101.6.40/export/secondary","_role":"Image"}},"secUrl":"nfs:// >> 10.101.6.40/export/secondary","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-26 15:07:14,985 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentConnectTaskPool-27:ctx-8f2e92c1) (logid:4a947f5e) Details from >> executing class com.cloud.agent.api.SecStorageSetupCommand: success >>>> 2023-02-26 15:07:15,001 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-27:ctx-8f2e92c1) (logid:4a947f5e) Seq >> 52-7356067041356283906: Sending { Cmd , MgmtId: 130593671224, via: >> 52(s-339-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.101.6.23/mnt/Store08/CSBackup08","_role":"Image"}},"secUrl":"nfs:// >> 10.101.6.23/mnt/Store08/CSBackup08","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-26 15:07:17,516 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentConnectTaskPool-27:ctx-8f2e92c1) (logid:4a947f5e) Details from >> executing class com.cloud.agent.api.SecStorageSetupCommand: success >>>> 2023-02-26 16:03:33,807 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-28:ctx-8b4b3cb8) (logid:6ae260b3) Seq >> 53-603482350067646465: Sending { Cmd , MgmtId: 130593671224, via: >> 53(s-340-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.101.6.40/export/secondary","_role":"Image"}},"secUrl":"nfs:// >> 10.101.6.40/export/secondary","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-26 16:03:37,126 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentConnectTaskPool-28:ctx-8b4b3cb8) (logid:6ae260b3) Details from >> executing class com.cloud.agent.api.SecStorageSetupCommand: success >>>> 2023-02-26 16:03:37,142 DEBUG [c.c.a.t.Request] >> (AgentConnectTaskPool-28:ctx-8b4b3cb8) (logid:6ae260b3) Seq >> 53-603482350067646466: Sending { Cmd , MgmtId: 130593671224, via: >> 53(s-340-VM), Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// >> 10.101.6.23/mnt/Store08/CSBackup08","_role":"Image"}},"secUrl":"nfs:// >> 10.101.6.23/mnt/Store08/CSBackup08","certs":{},"postUploadKey":"89HVaWPMPwbWI-QGJrI5jzoGULt3lyZzHN4pnc-kn36Le5Hy_Hh3l6ZABLMgJXeBlA4vspDa-NyrxtBbJGj20A","wait":"0","bypassHostMaintenance":"false"}}] >> } >>>> 2023-02-26 16:03:39,890 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentConnectTaskPool-28:ctx-8b4b3cb8) (logid:6ae260b3) Details from >> executing class com.cloud.agent.api.SecStorageSetupCommand: success >>>> >>>> >>>> Antoine Boucher >>>> >>>> >>>>> On Feb 28, 2023, at 4:47 AM, Wei ZHOU <[email protected]> wrote: >>>>> >>>>> The routes should use eth3 not eth1. >>>>> >>>>> Can you share the `/var/cache/cloud/cmdline` file in SSVM, and filter >>>>> management-server.log by keyword `SecStorageSetupCommand` ? >>>>> >>>>> >>>>> -Wei >>>>> >>>>> On Tue, 28 Feb 2023 at 10:42, Granwille Strauss >>>>> <[email protected] <mailto:[email protected]> >>>>> <mailto:[email protected]>> >> wrote: >>>>> >>>>>> I recently had a similar issue and now that I look at my routing >> tables, >>>>>> storage goes via eth1 and not eth3. Full details on this here: >>>>>> >> https://github.com/apache/cloudstack/issues/7244#issuecomment-1434755523 >>>>>> This, therefore, also explains why I randomly got this error with my >> SSVM. >>>>>> On 2/28/23 11:35, Daan Hoogland wrote: >>>>>> >>>>>> does sound like a bug Antoine, >>>>>> I did the network calculation and it seems you are right. >>>>>> I wonder about the last two routes as well. did you do anything for >> those? >>>>>> >>>>>> On Sun, Feb 26, 2023 at 9:47?PM Antoine Boucher < >> [email protected] <mailto:[email protected]> >> <mailto:[email protected]>> < >> [email protected] <mailto:[email protected]> >> <mailto:[email protected]>> >>>>>> wrote: >>>>>> >>>>>> >>>>>> Hello, >>>>>> >>>>>> I'm having a networking issue on SSVMs, I have the following networks >>>>>> defined in “Zone 1”. >>>>>> >>>>>> Management: 10.101.0.0/22 >>>>>> Storage: 10.101.6.0/23 >>>>>> >>>>>> All worked well until we decided to configure new storage devices on >>>>>> 10.101.7.x, the hosts and management server have no issue but the >> SSVM is >>>>>> not able to reach it. Here are the defined interfaces of the SSVM >> and the >>>>>> routing table of the SSVM: >>>>>> >>>>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN >> group >>>>>> default qlen 1000 >>>>>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >>>>>> inet 127.0.0.1/8 scope host lo >>>>>> valid_lft forever preferred_lft forever >>>>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast >> state >>>>>> UP group default qlen 1000 >>>>>> link/ether 0e:00:a9:fe:72:ec brd ff:ff:ff:ff:ff:ff >>>>>> altname enp0s3 >>>>>> altname ens3 >>>>>> inet 169.254.114.236/16 brd 169.254.255.255 scope global eth0 >>>>>> valid_lft forever preferred_lft forever >>>>>> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast >> state >>>>>> UP group default qlen 1000 >>>>>> link/ether 1e:00:a1:00:00:06 brd ff:ff:ff:ff:ff:ff >>>>>> altname enp0s4 >>>>>> altname ens4 >>>>>> inet 10.101.3.205/22 brd 10.101.3.255 scope global eth1 >>>>>> valid_lft forever preferred_lft forever >>>>>> 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast >> state >>>>>> UP group default qlen 1000 >>>>>> link/ether 1e:00:6b:00:00:3d brd ff:ff:ff:ff:ff:ff >>>>>> altname enp0s5 >>>>>> altname ens5 >>>>>> inet 148.59.36.61/28 brd 148.59.36.63 scope global eth2 >>>>>> valid_lft forever preferred_lft forever >>>>>> 5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast >> state >>>>>> UP group default qlen 1000 >>>>>> link/ether 1e:00:09:00:00:6b brd ff:ff:ff:ff:ff:ff >>>>>> altname enp0s6 >>>>>> altname ens6 >>>>>> inet 10.101.7.226/23 brd 10.101.7.255 scope global eth3 >>>>>> valid_lft forever preferred_lft forever >>>>>> >>>>>> default via 148.59.36.49 dev eth210.0.0.0/8 via 10.101.0.1 dev >> eth110.91.0.0/23 via 10.101.0.1 dev eth110.91.6.0/24 via 10.101.0.1 dev >> eth110.101.0.0/22 dev eth1 proto kernel scope link src >> 10.101.3.25310.101.6.0/23 via 10.101.0.1 dev eth1148.59.36.48/28 dev eth2 >> proto kernel scope link src 148.59.36.61169.254.0.0/16 dev eth0 proto >> kernel scope link src 169.254.232.208172.16.0.0/12 via 10.101.0.1 dev >> eth1192.168.0.0/16 via 10.101.0.1 dev eth1 >>>>>> >>>>>> Why is the routing for 10.101.6.0/23 routing via eth1, shoudn’nt it >> be >>>>>> using eth3? The router seems to be bypassing the routing rules for >>>>>> 10.101.6.x since I see no traffic going through the gateway but I see >>>>>> traffic going through the gateway when the destination is 10.101.7.x >>>>>> >>>>>> If I modify the routing for 10.101.6.0/23 to eth3 all is well. >>>>>> >>>>>> Is this by design? >>>>>> >>>>>> Regards, >>>>>> Antoine Boucher >>>>>> >>>>>> >>>>>> -- >>>>>> Regards / Groete >>>>>> >>>>>> <https://www.namhost.com <https://www.namhost.com/> >>>>>> <https://www.namhost.com/>> Granwille >> Strauss // Senior Systems Admin >>>>>> >>>>>> *e:* [email protected] <mailto:[email protected]> >>>>>> <mailto:[email protected]> >>>>>> *m:* +264 81 323 1260 <+264813231260> >>>>>> *w:* www.namhost.com<http://www.namhost.com> <http://www.namhost.com/> >>>>>> <http://www.namhost.com/> >>>>>> >>>>>> <https://www.facebook.com/namhost> <https://twitter.com/namhost> >>>>>> <https://www.instagram.com/namhostinternetservices/> >>>>>> <https://www.linkedin.com/company/namhos> >>>>>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> >>>>>> >>>>>> >>>>>> < >> https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner >>> >>>>>> >>>>>> Namhost Internet Services (Pty) Ltd, >>>>>> >>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA >>>>>> >>>>>> >>>>>> >>>>>> The content of this message is confidential. If you have received it >> by >>>>>> mistake, please inform us by email reply and then delete the message. >> It is >>>>>> forbidden to copy, forward, or in any way reveal the contents of this >>>>>> message to anyone without our explicit consent. The integrity and >> security >>>>>> of this email cannot be guaranteed over the Internet. Therefore, the >> sender >>>>>> will not be held liable for any damage caused by the message. For our >> full >>>>>> privacy policy and disclaimers, please go to >>>>>> https://www.namhost.com/privacy-policy >>>>>> >>>>>> [image: Powered by AdSigner] >>>>>> < >> https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818
