Hi guys, I'm trying to fresh install a CS 4.14 with an S3 secondary storage. For the systemvm template download part (cloud-install-sys-tmplt), I have used an NFS server. When I tried to define S3 secondary storage, I have added the mentioned NFS server as a staging NFS server. But when I check systemvm template status, I get this message: DOWNLOAD_ERROR: Timeout waiting for response from storage host Notice that I'm in the initialization state of CS installation, so there is no SSVM. The environment consists of CS, NFS and S3. I have checked that S3 is visible from NFS and CS. I have also analyzed the CS traffic at the time of adding secondary storage with Wireshark. It seems that CS doesn't even try to resolve the domain name of S3. So I'm in doubt that CS is really designed to use S3 as its secondary storage in initialization state.
Here is the debug log: 2021-02-19 01:25:03,396 DEBUG [c.c.a.ApiServlet] (qtp1165303897-18:ctx-16d3f513) (logid:943c72e7) ===START=== 172.24.12.2 -- GET command=addImageStore&response=json&name=s3&provider=S3&details%5B2%5D.key=bucket&details%5B2%5D.value=cloudstack&details%5B3%5D.key=usehttps&details%5B3%5D.value=true&details%5B4%5D.key=endpoint&details%5B4%5D.value= s3.example.com&_=1613685218312 2021-02-19 01:25:03,412 DEBUG [c.c.a.ApiServer] (qtp1165303897-18:ctx-16d3f513 ctx-a2aaa2fe) (logid:943c72e7) CIDRs from which account 'Acct[f014cf6a-71d4-11eb-8b92-0050569b9c1c-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0 2021-02-19 01:25:03,423 INFO [o.a.c.s.d.l.S3ImageStoreLifeCycleImpl] (qtp1165303897-18:ctx-16d3f513 ctx-a2aaa2fe) (logid:943c72e7) Trying to add a S3 store with endpoint: s3.example.com 2021-02-19 01:25:03,435 DEBUG [c.c.a.ApiServlet] (qtp1165303897-22:ctx-3fca0f55) (logid:21439e18) ===START=== 172.24.12.2 -- GET command=createSecondaryStagingStore&response=json&provider=NFS&zoneid=dff0c654-ac46-4ef7-83cb-a08872a9bc67&url=nfs%3A%2F%2Fscst01.cloud.local%2Fstorage&_=1613685218313 2021-02-19 01:25:03,450 DEBUG [c.c.a.ApiServer] (qtp1165303897-22:ctx-3fca0f55 ctx-4c2109d6) (logid:21439e18) CIDRs from which account 'Acct[f014cf6a-71d4-11eb-8b92-0050569b9c1c-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0 2021-02-19 01:25:03,464 INFO [o.a.c.s.d.l.CloudStackImageStoreLifeCycleImpl] (qtp1165303897-22:ctx-3fca0f55 ctx-4c2109d6) (logid:21439e18) Trying to add a new data store at nfs://scst01.cloud.local/storage to data center 1 2021-02-19 01:25:03,483 DEBUG [c.c.a.ApiServlet] (qtp1165303897-22:ctx-3fca0f55 ctx-4c2109d6) (logid:21439e18) ===END=== 172.24.12.2 -- GET command=createSecondaryStagingStore&response=json&provider=NFS&zoneid=dff0c654-ac46-4ef7-83cb-a08872a9bc67&url=nfs%3A%2F%2Fscst01.cloud.local%2Fstorage&_=1613685218313 2021-02-19 01:25:03,514 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (qtp1165303897-18:ctx-16d3f513 ctx-a2aaa2fe) (logid:943c72e7) template 8 is already in store:22, type:Image 2021-02-19 01:25:03,523 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] (qtp1165303897-18:ctx-16d3f513 ctx-a2aaa2fe) (logid:943c72e7) Downloading template to data store 22 2021-02-19 01:25:03,545 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] (qtp1165303897-18:ctx-16d3f513 ctx-a2aaa2fe) (logid:943c72e7) Performing image store createTemplate async callback 2021-02-19 01:25:03,589 INFO [o.a.c.s.i.d.TemplateDataStoreDaoImpl] (qtp1165303897-18:ctx-16d3f513 ctx-a2aaa2fe) (logid:943c72e7) Duplicate 0 template cache store records to region store 2021-02-19 01:25:03,593 INFO [o.a.c.s.i.d.SnapshotDataStoreDaoImpl] (qtp1165303897-18:ctx-16d3f513 ctx-a2aaa2fe) (logid:943c72e7) Duplicate 0 snapshot cache store records to region store 2021-02-19 01:25:03,601 INFO [o.a.c.s.i.d.VolumeDataStoreDaoImpl] (qtp1165303897-18:ctx-16d3f513 ctx-a2aaa2fe) (logid:943c72e7) Duplicate 0 volume cache store records to region store 2021-02-19 01:25:03,607 DEBUG [c.c.a.ApiServlet] (qtp1165303897-18:ctx-16d3f513 ctx-a2aaa2fe) (logid:943c72e7) ===END=== 172.24.12.2 -- GET command=addImageStore&response=json&name=s3&provider=S3&details%5B2%5D.key=bucket&details%5B2%5D.value=cloudstack&details%5B3%5D.key=usehttps&details%5B3%5D.value=true&details%5B4%5D.key=endpoint&details%5B4%5D.value= s3.example.com&_=1613685218312 2021-02-19 01:25:22,746 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-a0d7a7bf) (logid:18edf562) System vm template is not ready at data center 1, wait until it is ready to launch console proxy vm 2021-02-19 01:25:22,746 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-a0d7a7bf) (logid:18edf562) Zone 1 is not ready to launch console proxy yet 2021-02-19 01:25:22,994 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-f16b34fc) (logid:cd9caf82) System vm template is not ready at data center 1, wait until it is ready to launch secondary storage vm 2021-02-19 01:25:22,994 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-f16b34fc) (logid:cd9caf82) Zone 1 is not ready to launch secondary storage VM yet 2021-02-19 01:25:33,071 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-c9bc2b66) (logid:be12bbeb) Found 0 routers to update status. 2021-02-19 01:25:33,073 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-c9bc2b66) (logid:be12bbeb) Found 0 VPC's to update Redundant State. 2021-02-19 01:25:33,075 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-c9bc2b66) (logid:be12bbeb) Found 0 networks to update RvR status. 2021-02-19 01:25:33,546 DEBUG [c.c.s.d.DownloadListener] (Timer-6:ctx-e2a01191) (logid:a7da0124) Scheduling timeout at 30000 ms, TEMPLATE: 8 at host 0 2021-02-19 01:25:33,553 DEBUG [c.c.s.d.DownloadListener] (Timer-6:ctx-e2a01191) (logid:a7da0124) NOT_DOWNLOADED first timeout: checking again , TEMPLATE: 8 at host 0 2021-02-19 01:25:52,742 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-dc7b5ccc) (logid:e6bcf824) System vm template is not ready at data center 1, wait until it is ready to launch console proxy vm 2021-02-19 01:25:52,742 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-dc7b5ccc) (logid:e6bcf824) Zone 1 is not ready to launch console proxy yet 2021-02-19 01:25:52,993 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-5df03173) (logid:66f517c5) System vm template is not ready at data center 1, wait until it is ready to launch secondary storage vm 2021-02-19 01:25:52,993 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-5df03173) (logid:66f517c5) Zone 1 is not ready to launch secondary storage VM yet 2021-02-19 01:26:03,070 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a1d56fa8) (logid:e1875feb) Found 0 routers to update status. 2021-02-19 01:26:03,072 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a1d56fa8) (logid:e1875feb) Found 0 VPC's to update Redundant State. 2021-02-19 01:26:03,074 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a1d56fa8) (logid:e1875feb) Found 0 networks to update RvR status. 2021-02-19 01:26:03,547 DEBUG [c.c.s.d.DownloadListener] (Timer-6:ctx-b90910ce) (logid:316adaf4) timeout: transitioning to download error state, currstate=NOT_DOWNLOADED, TEMPLATE: 8 at host 0 2021-02-19 01:26:03,547 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] (Timer-6:ctx-b90910ce) (logid:316adaf4) Performing image store createTemplate async callback 2021-02-19 01:26:03,570 WARN [c.c.a.AlertManagerImpl] (Timer-6:ctx-b90910ce) (logid:316adaf4) AlertType:: 28 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Failed to register template: d34dd2fc-9a02-40aa-9726-70f1d9b70532 with error: Timeout waiting for response from storage host 2021-02-19 01:26:03,577 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (Timer-6:ctx-b90910ce) (logid:316adaf4) Failed to register template: d34dd2fc-9a02-40aa-9726-70f1d9b70532 with error: Timeout waiting for response from storage host 2021-02-19 01:26:03,578 WARN [c.c.s.d.DownloadListener] (Timer-6:ctx-b90910ce) (logid:316adaf4) Entering download error state: timeout waiting for response from storage host, TEMPLATE: 8 at host 0
