There are indeed 4 networks with NULL broadcast_uri that are causing the problem. I think those are systemic for cloudstack. +-----+-----------------------------+---------------+ | id | name | broadcast_uri | +-----+-----------------------------+---------------+ | 200 | NULL | NULL | | 201 | NULL | NULL | | 202 | NULL | NULL | | 203 | NULL | NULL |
select id,name,broadcast_uri,broadcast_domain_type,guru_name,state,gateway,cidr from networks where id < 204; +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ | id | name | broadcast_uri | broadcast_domain_type | guru_name | state | gateway | cidr | +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ | 200 | NULL | NULL | Vxlan | PublicNetworkGuru | Setup | NULL | NULL | | 201 | NULL | NULL | Vxlan | PodBasedNetworkGuru | Setup | NULL | NULL | | 202 | NULL | NULL | LinkLocal | ControlNetworkGuru | Setup | 169.254.0.1 | 169.254.0.0/16 | | 203 | NULL | NULL | Native | StorageNetworkGuru | Setup | NULL | NULL | +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ I can see some vnics associated with them: select instance_id,network_id,state,broadcast_uri,isolation_uri from nics where network_id<204 and removed is null; +-------------+------------+----------+-----------------+-----------------+ | instance_id | network_id | state | broadcast_uri | isolation_uri | +-------------+------------+----------+-----------------+-----------------+ | 1649 | 202 | Reserved | NULL | NULL | | 1649 | 200 | Reserved | vlan://untagged | vlan://untagged | | 1650 | 202 | Reserved | NULL | NULL | | 1650 | 200 | Reserved | vlan://untagged | vlan://untagged | | 1651 | 202 | Reserved | NULL | NULL | | 1651 | 200 | Reserved | vlan://untagged | vlan://untagged | | 1653 | 202 | Reserved | NULL | NULL | | 1653 | 200 | Reserved | vlan://untagged | vlan://untagged | | 1656 | 202 | Reserved | NULL | NULL | | 1656 | 200 | Reserved | vlan://untagged | vlan://untagged | | 1659 | 202 | Reserved | NULL | NULL | | 1659 | 200 | Reserved | vlan://untagged | vlan://untagged | | 1669 | 200 | Reserved | vlan://untagged | vlan://untagged | | 1669 | 202 | Reserved | NULL | NULL | | 1669 | 201 | Reserved | NULL | NULL | | 1670 | 200 | Reserved | vlan://untagged | vlan://untagged | | 1670 | 202 | Reserved | NULL | NULL | | 1670 | 201 | Reserved | NULL | NULL | | 1671 | 202 | Reserved | NULL | NULL | | 1671 | 200 | Reserved | vlan://untagged | vlan://untagged | | 1672 | 202 | Reserved | NULL | NULL | | 1672 | 200 | Reserved | vlan://untagged | vlan://untagged | | 1673 | 202 | Reserved | NULL | NULL | | 1673 | 200 | Reserved | vlan://untagged | vlan://untagged | | 1684 | 202 | Reserved | NULL | NULL | | 1684 | 200 | Reserved | vlan://untagged | vlan://untagged | | 1690 | 202 | Reserved | NULL | NULL | | 1690 | 200 | Reserved | vlan://untagged | vlan://untagged | +-------------+------------+----------+-----------------+-----------------+ 28 rows in set (0.002 sec) Hmm, maybe the solution is to have vlan://untagged in the broadcast_uri field for those 4 in the networks table ? On Thu, Jun 13, 2024 at 8:19 PM Wei ZHOU <ustcweiz...@gmail.com> wrote: > Can you run the following sql query ? > > select id,name,broadcast_uri from networks where broadcast_uri IS NOT > NULL and removed is NULL; > > -Wei > > On Thu, Jun 13, 2024 at 11:35 AM Curious Pandora <p4nd...@gmail.com> > wrote: > > > > Hello, for vnet null there seems to be none: > > > > select * from op_dc_vnet_alloc where vnet IS NULL; > > Empty set (0.001 sec) > > > > For API CALL: > > GET > listall=true&page=1&pagesize=20&command=listGuestVlans&response=json > > > > For vlan table: > > select * from vlan where removed is null; > > > +----+--------------------------------------+-----------------+--------------+---------------+---------------------------+----------------+----------------+------------+---------------------+----------------------+------------------------+-----------+---------+---------------------+ > > | id | uuid | vlan_id | > > vlan_gateway | vlan_netmask | ip4_range | vlan_type > | > > data_center_id | network_id | physical_network_id | ip6_gateway > | > > ip6_cidr | ip6_range | removed | created | > > > +----+--------------------------------------+-----------------+--------------+---------------+---------------------------+----------------+----------------+------------+---------------------+----------------------+------------------------+-----------+---------+---------------------+ > > | 2 | 25169e64-8ab5-48b5-9420-ed2d4201a06d | vlan://untagged | > xx.xx.xx.1 > > | 255.255.254.0 | xx.xx.xx.2-xx.xx.xx.254 | VirtualNetwork | > > 2 | 200 | 200 | NULL | NULL > > | NULL | NULL | 2019-12-28 16:34:39 | > > | 35 | 860f0e1e-b3a2-4183-9d0d-5469a409f5b5 | vlan://untagged | NULL > > | NULL | NULL | VirtualNetwork | > > 2 | 200 | 200 | 2a0f:xxxx:xxxx:xxxx::1 | > > 2a0f:xxxx:xxx:xxxx::/64 | NULL | NULL | 2022-06-17 17:19:35 | > > > +----+--------------------------------------+-----------------+--------------+---------------+---------------------------+----------------+----------------+------------+---------------------+----------------------+------------------------+-----------+---------+---------------------+ > > > > and op_dc_vnet_allocs: > > select * from op_dc_vnet_alloc where account_id > 1; > > > +-------+-------+---------------------+----------------+--------------------------------------+------------+---------------------+---------------------+ > > | id | vnet | physical_network_id | data_center_id | reservation_id > > | account_id | taken | > account_vnet_map_id | > > > +-------+-------+---------------------+----------------+--------------------------------------+------------+---------------------+---------------------+ > > | 9685 | 30000 | 200 | 2 | > > 52cdb05f-7124-4b48-9948-3e17af5280ea | 106 | 2024-05-01 07:56:23 | > > NULL | > > | 11637 | 30244 | 200 | 2 | > > 46148850-2956-45e7-88e6-26dd86e9001c | 23 | 2020-03-11 00:22:02 | > > NULL | > > | 11769 | 30238 | 200 | 2 | > > 2bd525bf-f714-41f2-a94b-e8ef7fe44994 | 16 | 2020-03-20 00:20:08 | > > NULL | > > | 11857 | 30229 | 200 | 2 | > > 7c761f3d-8465-4669-988c-fa6f7a84ebd0 | 10 | 2020-01-06 19:07:32 | > > NULL | > > | 12131 | 30293 | 200 | 2 | > > 64900dfd-48d4-4101-9b04-291e4bb23e81 | 96 | 2022-11-01 15:49:36 | > > NULL | > > | 12203 | 30289 | 200 | 2 | > > 8c0da5da-b4f0-4da2-b7e5-4971b750e96d | 93 | 2022-07-27 05:37:07 | > > NULL | > > | 12269 | 30273 | 200 | 2 | > > 18eec34a-639b-4f3b-af3e-f00392062361 | 80 | 2021-11-09 22:24:42 | > > NULL | > > | 12491 | 30380 | 200 | 2 | > > eb599359-6ce9-46eb-912c-08de03e78ea8 | 101 | 2023-12-23 05:07:56 | > > NULL | > > | 18463 | 54996 | 200 | 2 | > > 5213b778-9687-4b29-b0cc-1dfaaf6bb935 | 8 | 2020-01-09 15:54:38 | > > 16 | > > | 18467 | 54995 | 200 | 2 | > > 5c21ade2-241f-4b2b-8612-6f51a28d729d | 8 | 2020-03-10 21:10:18 | > > 16 | > > | 18471 | 54994 | 200 | 2 | > > 93a51305-3b17-4773-92b7-349e6ee75668 | 8 | 2023-12-23 05:04:49 | > > 16 | > > | 18475 | 54993 | 200 | 2 | > > 8bbb8c50-5301-4ee8-a195-444d13cf17e8 | 8 | 2020-01-09 15:54:09 | > > 16 | > > | 18481 | 54999 | 200 | 2 | > > de553e5e-5764-4cc8-8c9d-41a06f77aec9 | 8 | 2020-01-15 20:15:38 | > > 16 | > > | 18485 | 54998 | 200 | 2 | > > 87d967c6-4972-47dc-b55c-2ff740aa9293 | 8 | 2020-01-07 14:15:14 | > > 16 | > > | 18489 | 54997 | 200 | 2 | > > fb62bd1d-7314-4a7b-a3b8-a53036916e01 | 8 | 2020-01-15 20:17:10 | > > 16 | > > | 19215 | 30999 | 200 | 2 | > > 9e4f1ae0-064d-4400-b1b5-d12b5a350ba6 | 89 | 2023-12-23 05:40:11 | > > NULL | > > | 35639 | 55000 | 200 | 2 | > > e4fc5468-6739-4ffc-970f-1a6a77d6c9e7 | 8 | 2020-01-15 20:16:26 | > > 16 | > > > +-------+-------+---------------------+----------------+--------------------------------------+------------+---------------------+---------------------+ > > > > select count(*) from op_dc_vnet_alloc; > > +----------+ > > | count(*) | > > +----------+ > > | 25001 | > > +----------+ > > > > > > On Thu, Jun 13, 2024 at 11:50 AM Wei ZHOU <ustcweiz...@gmail.com> wrote: > > > > > can you check "op_dc_vnet_alloc" table if there is a record with empty > > > "vnet" ? > > > > > > select * from op_dc_vnet_alloc where vnet IS NULL; > > > > > > > > > -Wei > > > > > > On Thu, Jun 13, 2024 at 10:02 AM Curious Pandora <p4nd...@gmail.com> > > > wrote: > > > > > > > > Hello, > > > > > > > > we get a 503 error when trying to list guest vlans (both from web > and api > > > > call). > > > > > > > > The relevant error log in the management server is: > > > > > > > > ERROR [c.c.a.ApiServer] (qtp1278254413-1728:ctx-d3eec0e8 > ctx-7b04643e) > > > > (logid:6a2d2a44) unhandled exception executing api command: > > > > [Ljava.lang.String;@1a84b3d5 > > > > java.lang.NullPointerException > > > > at > > > > > > > > com.cloud.network.dao.NetworkDaoImpl.listByZoneAndUriAndGuestType(NetworkDaoImpl.java:464) > > > > at > jdk.internal.reflect.GeneratedMethodAccessor666.invoke(Unknown > > > > Source) > > > > at > > > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > > > at > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) > > > > at > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > > > > at > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > at > > > > > > > > com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) > > > > at > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) > > > > at > > > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > > > > at > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > > > > at > > > > > > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) > > > > at > com.sun.proxy.$Proxy52.listByZoneAndUriAndGuestType(Unknown > > > > Source) > > > > at > > > > > > > > com.cloud.api.ApiResponseHelper.createGuestVlanResponse(ApiResponseHelper.java:4943) > > > > at > > > > > > > > org.apache.cloudstack.api.command.admin.network.ListGuestVlansCmd.execute(ListGuestVlansCmd.java:98) > > > > at > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172) > > > > at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803) > > > > at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624) > > > > at > > > > com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342) > > > > at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149) > > > > at > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > > > > at > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > > > > at > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > > > > at > com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146) > > > > at com.cloud.api.ApiServlet.doGet(ApiServlet.java:100) > > > > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:645) > > > > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:750) > > > > at > > > > > > > > org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450) > > > > at > > > > > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) > > > > at > > > > > > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) > > > > at > > > > > > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > > > > at > > > > > > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) > > > > at > > > > > > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > > > > at > > > > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > > > > at > > > > > > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) > > > > at > > > > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > > > > at > > > > > > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) > > > > at > > > > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) > > > > at > > > > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) > > > > at > > > > > > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) > > > > at > > > > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) > > > > at > > > > > > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) > > > > at > > > > > > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > > > > at > > > > > > > > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) > > > > at > > > > > > > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) > > > > at > > > > > > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > > > > at org.eclipse.jetty.server.Server.handle(Server.java:516) > > > > at > > > > > > > > org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) > > > > at > > > > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) > > > > at > > > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) > > > > at > > > > > > > > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) > > > > at > > > > org.eclipse.jetty.io > > > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > > > > at org.eclipse.jetty.io > > > .FillInterest.fillable(FillInterest.java:105) > > > > at > > > > org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) > > > > at > > > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) > > > > at > > > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) > > > > at > > > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > > > at > > > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) > > > > at > > > > > > > > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) > > > > at > > > > > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > > > at > > > > > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > > > at java.base/java.lang.Thread.run(Thread.java:829) > > > > > > > > The relevant table in database (op_dc_vnet_alloc) seems to be > populated > > > > correctly. > > > > > > > > Any ideas how to debug this ? > > > > > > > > -- > > > > p4nd0ra - the curious > > > > > > > > > -- > > p4nd0ra - the curious > -- p4nd0ra - the curious
