Here is what i have learned about the local upload issue for templates and iso.
* Inconsistently works. * Sometimes a reboot of the SSVM will fix the upload for a while. * Sometimes logging out of ACS and back in will fix the upload for a while. * not related to size of upload, name of the uploaded file, name of the template * doesn't matter if 1 management server is up or both ________________________________ From: Brian Fossmeyer <bfossme...@answersingenesis.org> Sent: Friday, January 5, 2024 3:23:19 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload. Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload as SSVM is either destroyed or SSVM agent not in 'Up' state Hello Perl, Thank you for getting back to me. We are SSL enabled in our environment, but I did try what you suggested and got an forbidden: Forbidden You don't have permission to access this resource. Thanks, Brian From: Pearl d'Silva <pearl.dsi...@shapeblue.com> Date: Friday, January 5, 2024 at 3:16 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload. Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload as SSVM is either destroyed or SSVM agent not in 'Up' state Hi Brian, Is it an SSL enabled environment? If not, can you prior uploading the iso from local, try placing https://<public_ip<https://%3cpublic_ip> of ssvm> on the URL tab of the browser and accept the risk and then attempt to upload the iso. Let us know if it uploads the iso successfully then? Regards, ________________________________ From: Brian Fossmeyer <bfossme...@answersingenesis.org.INVALID> Sent: January 5, 2024 3:03 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload. Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload as SSVM is either destroyed or SSVM agent not in 'Up' state Also, running the ssvm-check.sh shows that all is well with the SSVM root@s-230-VM:/usr/local/cloud/systemvm# ./ssvm-check.sh ================================================ First DNS server is 10.30.50.1 PING 10.30.50.1 (10.30.50.1): 56 data bytes 64 bytes from 10.30.50.1: icmp_seq=0 ttl=254 time=0.344 ms 64 bytes from 10.30.50.1: icmp_seq=1 ttl=254 time=0.539 ms --- 10.30.50.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.344/0.442/0.539/0.098 ms Good: Can ping DNS server ================================================ Good: DNS resolves cloudstack.apache.org ================================================ nfs is currently mounted Mount point is /mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f Good: Can write to mount point ================================================ Management server is 10.50.11.10,10.50.11.20. Checking connectivity. Good: Can connect to management server 10.50.11.10 port 8250 Good: Can connect to management server 10.50.11.20 port 8250 ================================================ Good: Java process is running ================================================ Tests Complete. Look for ERROR or WARNING above. Thanks, Brian From: Brian Fossmeyer <bfossme...@answersingenesis.org> Date: Friday, January 5, 2024 at 11:33 AM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload. Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload as SSVM is either destroyed or SSVM agent not in 'Up' state Also, as you can see from the SSVM, the qcow2 file is there, see the bottom. root@s-230-VM:/var/log# tail -f cloud.log 2024-01-05 16:12:45,395 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:12:55,755 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:13:45,639 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-4:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:13:56,099 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:14:45,879 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:14:56,345 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:15:46,167 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:15:56,611 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-4:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:16:46,363 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:16:56,951 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:17:46,610 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:17:46,928 INFO [handler.logging.LoggingHandler] (nioEventLoopGroup-2-1:null) [id: 0xc2744c38, L:/0.0.0.0:8210] READ: [id: 0x1bc34b72, L:/127.0.0.1:8210 - R:/127.0.0.1:33898] 2024-01-05 16:17:46,930 INFO [handler.logging.LoggingHandler] (nioEventLoopGroup-2-1:null) [id: 0xc2744c38, L:/0.0.0.0:8210] READ COMPLETE 2024-01-05 16:17:46,936 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-7:null) HEADER: SIGNATURE=xRbt2tIf27eRHe/H9MqxowS9QMI= 2024-01-05 16:17:46,937 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-7:null) HEADER: METADATA=JvY7F8jeDr1zGrJiWz4RAx6XllfhiaRie1t/mp30Kri+I9rYaBaQieJQ4LUshDFPdQ+UsOmecgrc9Udeh5mLPU0VK+sJLhbmAVQFT6wcdMTgycHI/tvIPwQZYARuKsL2tjc8TwsiXkIuK6BeAro9+MHWvh8Z0H1XTrasyU4UAOvPL4hEOBVoGqalnz/XcC++4l00AjtDuoJwuhVn9sv20zV/4x7UiwVPyBAZCnyGvTh+whf5+hpj9eUKBAljxOetN00bW3Mcp8XNlwEnGAcr33n/CjdGeiMrN5+cMIKiBocfTQXOgeN7koDn7Aqvu0MOFXUOMG5F8AjaFb+D7I+YC9JEjlTippGKmRipKJ2/NCHV7DozS8vwb2O19u6VtvIxcpxsvrO8NVppAOojpFY4/ZrpEd0fpazrLAd3aJTI1jtxOQLtFjLE4iMVvvAGFPEntm7PqFKEjKCMBXuTESnz5QXCh4kg3bXKtSMxRKOm6cDDYLiXw++aWYK3w/bSke0sw/PBYfKGkhnyqLUPxqo+Drqacl+dnQErX/bTUFa9oAXH4LWftvqtqX8B7pwJ5bqz9NuJKXDxJzUSFORpC117CADegTo2B9eYLTw= 2024-01-05 16:17:46,938 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-7:null) HEADER: EXPIRES=2024-01-05T17:17:46.775Z 2024-01-05 16:17:46,938 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-7:null) HEADER: HOST=216-120-178-249.aigcloud.io 2024-01-05 16:17:46,938 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-7:null) HEADER: CONTENT_LENGTH=5509021929 2024-01-05 16:17:46,938 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-7:null) URI: uuid=a69107bb-fc76-4439-b553-c270f792945a 2024-01-05 16:17:46,941 INFO [storage.resource.NfsSecondaryStorageResource] (nioEventLoopGroup-3-7:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:17:46,964 INFO [storage.resource.NfsSecondaryStorageResource] (nioEventLoopGroup-3-7:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:17:49,141 INFO [storage.resource.NfsSecondaryStorageResource] (nioEventLoopGroup-3-7:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:17:49,310 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-7:null) base directory: /mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239 2024-01-05 16:17:57,260 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:18:46,821 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-4:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:18:57,878 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:19:47,679 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:19:59,514 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:20:49,108 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:21:00,692 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-4:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:21:50,637 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:22:02,156 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:22:51,242 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:23:03,209 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:23:52,487 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-4:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:24:04,091 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-7:null) BODY FileUpload: FileUpload: content-disposition: form-data; name="files[]"; filename="1b4f2ce0-0999-4314-ab69-348953432514.qcow2" content-type: application/octet-stream; charset=UTF-8 content-length: 5509021696 Completed: true IsInMemory: false RealFile: /mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239/FUp_9513061883467712232_upload DefaultDeleteAfter: true 2024-01-05 16:24:04,597 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) Determined host 10.22.1.10 corresponds to IP 10.22.1.10 2024-01-05 16:24:05,335 INFO [storage.template.OVAProcessor] (nioEventLoopGroup-3-7:null) Template processing. templatePath: /mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239, templateName: a9ba3e9f-16dc-375b-9781-1edb79159336 2024-01-05 16:24:05,360 INFO [storage.template.OVAProcessor] (nioEventLoopGroup-3-7:null) Unable to find the vmware template file: /mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239/a9ba3e9f-16dc-375b-9781-1edb79159336.ova 2024-01-05 16:24:05,361 INFO [storage.template.VmdkProcessor] (nioEventLoopGroup-3-7:null) Template processing. templatePath: /mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239, templateName: a9ba3e9f-16dc-375b-9781-1edb79159336 2024-01-05 16:24:05,385 INFO [storage.template.VmdkProcessor] (nioEventLoopGroup-3-7:null) Unable to find the vmware template file: /mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239/a9ba3e9f-16dc-375b-9781-1edb79159336.vmdk 2024-01-05 16:24:05,538 WARN [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-7:null) io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74) at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138) at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76) at io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284) at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952) at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935) at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-05 16:24:05,540 WARN [netty.channel.AbstractChannelHandlerContext] (nioEventLoopGroup-3-7:null) An exception 'java.lang.NullPointerException' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74) at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138) at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76) at io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284) at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952) at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935) at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ^Z [1]+ Stopped tail -f cloud.log root@s-230-VM:/var/log# root@s-230-VM:/var/log# cd /mnt root@s-230-VM:/mnt# ls SecStorage root@s-230-VM:/mnt# cd SecStorage/ root@s-230-VM:/mnt/SecStorage# ls c7d010a7-ec39-344f-afab-f7bff31a309f extractmnt root@s-230-VM:/mnt/SecStorage# cd c7d010a7-ec39-344f-afab-f7bff31a309f/ root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f# ls snapshots template volumes root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f# cd template root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template# ls tmpl root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template# cd templ -bash: cd: templ: No such file or directory root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template# ls -la total 0 drwxr-xr-x 3 root root 0 Dec 4 17:11 . drwxrwxrwt 5 root root 0 Dec 27 18:09 .. drwxr-xr-x 5 root root 0 Dec 27 16:25 tmpl root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template# cd tmpl root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl# ls 1 2 7 root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl# cd 2 root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2# ls 205 208 218 221 225 232 233 234 238 239 root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2# cd 239 root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239# ls a9ba3e9f-16dc-375b-9781-1edb79159336.qcow2 template.properties root@s-230-VM:/mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/239# Thanks, Brian From: Brian Fossmeyer <bfossme...@answersingenesis.org.INVALID> Date: Friday, January 5, 2024 at 5:41 AM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload. Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload as SSVM is either destroyed or SSVM agent not in 'Up' state . No, that was the entire upload process. There are no other logs in that log file and no errors. Any place else I can look? ________________________________ From: Jithin Raju Sent: Thursday, January 4, 2024 11:44:30 PM To: users@cloudstack.apache.org Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload. Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload as SSVM is either destroyed or SSVM agent not in 'Up' state Hi Brian, These stack traces are not expected and reported recently in github: https://github.com/apache/cloudstack/issues/8376 But as per this issue ticket, this stack trace is not the one causing the upload error mostly. Do you see anything else in cloud.log with an ERROR in the upload process? -Jithin From: Brian Fossmeyer Date: Friday, 5 January 2024 at 12:09 AM To: users@cloudstack.apache.org Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload. Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload as SSVM is either destroyed or SSVM agent not in 'Up' state Thanks for the heads up on the cloud.log. I have added the log portion from the upload below. The thing that stands out to me is that the error states that there is no vmware template and we are not using vmnware? 2024-01-04 18:32:08,970 INFO [handler.logging.LoggingHandler] (nioEventLoopGroup-2-1:null) [id: 0xc2744c38, L:/0.0.0.0:8210] READ: [id: 0x9ea99be8, L:/127.0.0.1:8210 - R:/127.0.0.1:60612] 2024-01-04 18:32:08,971 INFO [handler.logging.LoggingHandler] (nioEventLoopGroup-2-1:null) [id: 0xc2744c38, L:/0.0.0.0:8210] READ COMPLETE 2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-5:null) HEADER: SIGNATURE=8ehXP50nEe++Y8kK9uLWFS4ez5s= 2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-5:null) HEADER: METADATA=zgZ2nuZ5Ml9E+OYDsKzLoZw0YIBzqwfH7i/wlz6vG3If1P4CLO/+MTvgAfYwBtUj+MUmfrgX6nKaQJJ9V92H0qyPXCfwu7e2JLOOIvTcrXOUYOQOApZDWLNGcqlSVtcNoWN0qlYhN1Ot/JHL6UnU3oGaGvuA03bTlAHIajTpLopXSJC0QAND5J7pdMSve5GeR+yDpN2mQMdDQZ25swM0zUd1SnAPuR3sprQO/MrE+eh7OOFbp6M86v7cWhq1Z6TYdaaD58woNaeyxr8kVCJBuLbG7mfMLbtFQ840nrZ9SC5ejATbMKUQvxYbHWjpyheePRHLhC9LR+nPI7TY05F3PMjkw1Ipt2Vh8Io1Q6Bb62D5ixFM5fWHn++dVi1onlg0oJL/F1BO2mX0+WB+46t8vfvZgJxf/00eoq7TdTaGgzoanj7JYNqLznVjCD+gzAVgoIjjMyVjDsIPuvXr40yraX7Cb7/1KbHLJtqybup5++euF7qbJ9hzqAUWVtVD+8VfseVMjnTHB2/c4WB2NhGxJ55sr82sIBN6as6WpveLTCDR//r2Hl+j8BfvwVbIyix5rhiCT4vc/Gk4/g0FWd/xjxXIEPsYw/izjA== 2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-5:null) HEADER: EXPIRES=2024-01-04T19:32:08.683Z 2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-5:null) HEADER: HOST=216-120-178-249.aigcloud.io 2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-5:null) HEADER: CONTENT_LENGTH=2133391587 2024-01-04 18:32:08,978 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-5:null) URI: uuid=330b70a5-258e-4e7c-a698-ad8ccfe2246f 2024-01-04 18:32:10,895 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-5:null) base directory: /mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234 2024-01-04 18:34:07,578 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-5:null) BODY FileUpload: FileUpload: content-disposition: form-data; name="files[]"; filename="ubuntu-22.04.3-live-server-amd64.iso" content-type: application/octet-stream; charset=UTF-8 content-length: 2133391360 Completed: true IsInMemory: false RealFile: /mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234/FUp_12970119428300643144_upload DefaultDeleteAfter: true 2024-01-04 18:34:08,645 INFO [storage.template.OVAProcessor] (nioEventLoopGroup-3-5:null) Template processing. templatePath: /mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234, templateName: 330b70a5-258e-4e7c-a698-ad8ccfe2246f 2024-01-04 18:34:08,671 INFO [storage.template.OVAProcessor] (nioEventLoopGroup-3-5:null) Unable to find the vmware template file: /mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234/330b70a5-258e-4e7c-a698-ad8ccfe2246f.ova 2024-01-04 18:34:08,671 INFO [storage.template.VmdkProcessor] (nioEventLoopGroup-3-5:null) Template processing. templatePath: /mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234, templateName: 330b70a5-258e-4e7c-a698-ad8ccfe2246f 2024-01-04 18:34:08,695 INFO [storage.template.VmdkProcessor] (nioEventLoopGroup-3-5:null) Unable to find the vmware template file: /mnt/SecStorage/c7d010a7-ec39-344f-afab-f7bff31a309f/template/tmpl/2/234/330b70a5-258e-4e7c-a698-ad8ccfe2246f.vmdk 2024-01-04 18:34:08,883 WARN [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-5:null) io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74) at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138) at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76) at io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284) at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952) at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935) at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-04 18:34:08,885 WARN [netty.channel.AbstractChannelHandlerContext] (nioEventLoopGroup-3-5:null) An exception 'java.lang.NullPointerException' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74) at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138) at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76) at io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284) at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952) at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935) at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) Thanks, Brian From: Jithin Raju Date: Wednesday, January 3, 2024 at 11:14 PM To: users@cloudstack.apache.org Subject: Re: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload. Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload as SSVM is either destroyed or SSVM agent not in 'Up' state Hi Brian, Does the SSVM agent state shows ‘Up’? you should be able to find more information in the cloud.log of the SSVM. -Jithin From: Brian Fossmeyer Date: Thursday, 4 January 2024 at 12:39 AM To: users@cloudstack.apache.org Subject: FW: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload. Error details: Template f2d39f0f-dc31-47a3-88a5-e02f0b892bcb failed to upload as SSVM is either destroyed or SSVM agent not in 'Up' state Hello, I am trying to upload ISO’s from my desktop and having this issue. The ISO will upload to 100%, but then will generate this error and then will not ever get into a ready state. Both of the System VMs are in a running and in an up state. Template failed to upload. Error details: Template failed to upload as SSVM is either destroyed or SSVM agent not in 'Up' state Thanks Brian Brian Fossmeyer Senior Systems Engineer, Technology [Answers in Genesis] [Creation Museum] [Ark Encounter]
