hanisirfan opened a new issue, #10044:
URL: https://github.com/apache/cloudstack/issues/10044
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete
the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
* Improvement Request
* Enhancement Request
* Feature Idea
* Documentation Report
* Other
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
Infrastructure > Object Store
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main
branch.
-->
~~~
4.20.0.0
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network,
advanced networking, etc. N/A otherwise
-->
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
Ceph version: 19.2.0 squid (stable)
##### SUMMARY
<!-- Explain the problem/feature briefly -->
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal
test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
1. Create a Object Gateway user on Ceph Dashboard.
2. Try to add object store using the Ceph provider.
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
Being able to add a Ceph object store to my infrastructure.
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
Failed with error:
Request failed. (530)
Failed to add object store: Error while initializing Ceph RGW Object Store.
Invalid credentials or URL
~~~
##### ERROR LOG
```
2024-12-05 05:54:03,154 DEBUG [c.c.a.ApiServlet]
(qtp2038105753-397:[ctx-d6792894]) (logid:de1e0941) ===START=== 10.0.113.1 --
GET
name=Ceph+Test+1&provider=Ceph&url=https:%2F%2F<URL_HERE>%2F&command=addObjectStoragePool&response=json&sessionkey=_IgdzVMijOi6635kzbViqZtkR2A
2024-12-05 05:54:03,154 DEBUG [c.c.a.ApiServlet]
(qtp2038105753-397:[ctx-d6792894]) (logid:de1e0941) Two factor authentication
is already verified for the user 2, so skipping
2024-12-05 05:54:03,161 DEBUG [c.c.a.ApiServer]
(qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) CIDRs from
which account 'Account
[{"accountName":"admin","id":2,"uuid":"c2a81095-5469-11ef-8cfa-005056a0391c"}]'
is allowed to perform API calls: 0.0.0.0/0,::/0
2024-12-05 05:54:03,163 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker]
(qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) RoleService
is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-12-05 05:54:03,163 DEBUG [o.a.c.r.ApiRateLimitServiceImpl]
(qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) API rate
limiting is disabled. We will not use ApiRateLimitService.
2024-12-05 05:54:03,176 INFO [o.a.c.s.d.l.CephObjectStoreLifeCycleImpl]
(qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Attempting to
connect to Ceph RGW at https://<URL_HERE>/ with access key <ACCESS_KEY_HERE>
2024-12-05 05:54:03,199 DEBUG [o.a.c.s.d.l.CephObjectStoreLifeCycleImpl]
(qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Error while
initializing Ceph RGW Object Store: Cannot invoke "java.util.List.stream()"
because the return value of
"org.twonote.rgwadmin4j.impl.RgwAdminImpl.listBucketInfo()" is null
2024-12-05 05:54:03,199 DEBUG [c.c.s.StorageManagerImpl]
(qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Failed to add
object store: Error while initializing Ceph RGW Object Store. Invalid
credentials or URL java.lang.RuntimeException: Error while initializing Ceph
RGW Object Store. Invalid credentials or URL
at
org.apache.cloudstack.storage.datastore.lifecycle.CephObjectStoreLifeCycleImpl.initialize(CephObjectStoreLifeCycleImpl.java:88)
at
com.cloud.storage.StorageManagerImpl.discoverObjectStore(StorageManagerImpl.java:4140)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy130.discoverObjectStore(Unknown
Source)
at
org.apache.cloudstack.api.command.admin.storage.AddObjectStoragePoolCmd.execute(AddObjectStoragePoolCmd.java:115)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:831)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:652)
at
com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:354)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:157)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:154)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:108)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at
org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
at
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:840)
2024-12-05 05:54:03,200 WARN [o.a.c.m.w.WebhookServiceImpl]
(qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Skipping
delivering event [ID: null, description:
{"event":"OBJECT.STORE.CREATE","status":"Completed"}] to any webhook as account
ID is missing
2024-12-05 05:54:03,202 WARN [o.a.c.f.e.EventDistributorImpl]
(qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Failed to
publish event [category: ActionEvent, type: OBJECT.STORE.CREATE] on bus
webhookEventBus
2024-12-05 05:54:03,213 ERROR [o.a.c.a.c.a.s.AddObjectStoragePoolCmd]
(qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Exception:
com.cloud.utils.exception.CloudRuntimeException: Failed to add object store:
Error while initializing Ceph RGW Object Store. Invalid credentials or URL
at
com.cloud.storage.StorageManagerImpl.discoverObjectStore(StorageManagerImpl.java:4145)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy130.discoverObjectStore(Unknown
Source)
at
org.apache.cloudstack.api.command.admin.storage.AddObjectStoragePoolCmd.execute(AddObjectStoragePoolCmd.java:115)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:831)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:652)
at
com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:354)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:157)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:154)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:108)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at
org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
at
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.RuntimeException: Error while initializing Ceph RGW
Object Store. Invalid credentials or URL
at
org.apache.cloudstack.storage.datastore.lifecycle.CephObjectStoreLifeCycleImpl.initialize(CephObjectStoreLifeCycleImpl.java:88)
at
com.cloud.storage.StorageManagerImpl.discoverObjectStore(StorageManagerImpl.java:4140)
... 67 more
2024-12-05 05:54:03,213 INFO [c.c.a.ApiServer]
(qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Failed to add
object store: Error while initializing Ceph RGW Object Store. Invalid
credentials or URL
2024-12-05 05:54:03,214 INFO [c.c.a.ApiServlet]
(qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) (userId=2
accountId=2 sessionId=node0a1m8rqhb8vwl17sdi3x5937fp3) 10.0.113.1 -- GET
name=Ceph+Test+1&provider=Ceph&url=https:%2F%2F<URL_HERE>%2F&command=addObjectStoragePool&response=json&sessionkey=_IgdzVMijOi6635kzbViqZtkR2A
530 Failed to add object store: Error while initializing Ceph RGW Object
Store. Invalid credentials or URL
2024-12-05 05:54:03,214 DEBUG [c.c.a.ApiServlet]
(qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) ===END===
10.0.113.1 -- GET
name=Ceph+Test+1&provider=Ceph&url=https:%2F%2F<URL_HERE>%2F&command=addObjectStoragePool&response=json&sessionkey=_IgdzVMijOi6635kzbViqZtkR2A
```
It might be something related to
`org.twonote.rgwadmin4j.impl.RgwAdminImpl.listBucketInfo()`. Note that I'm
inserting the base URL of my Ceph RGW endpoint. The endpoint should support
both path and virtual host based URL (tested with WinSCP). No bucket created at
all for the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
