from the code https://github.com/apache/cloudstack/blob/cb9b3134f7fff972b63d8565a4d021f8ea918903/engine/schema/src/main/java/com/cloud/network/dao/NetworkDaoImpl.java#L464
it looks like either broadcast_uri of a network or vnet of a record op_dc_vnet_alloc is null or unsupported. can you share the following ? select id,vnet from op_dc_vnet_alloc; select id,broadcast_uri from networks where data_center_id is not null and broadcast_uri is not null; -Wei On Fri, Jun 14, 2024 at 1:39 PM Curious Pandora <p4nd...@gmail.com> wrote: > > @Wei ZHOU any thoughts ? > > On Thu, Jun 13, 2024 at 9:29 PM Curious Pandora <p4nd...@gmail.com> wrote: > > > There are indeed 4 networks with NULL broadcast_uri that are causing the > > problem. I think those are systemic for cloudstack. > > +-----+-----------------------------+---------------+ > > | id | name | broadcast_uri | > > +-----+-----------------------------+---------------+ > > | 200 | NULL | NULL | > > | 201 | NULL | NULL | > > | 202 | NULL | NULL | > > | 203 | NULL | NULL | > > > > select > > id,name,broadcast_uri,broadcast_domain_type,guru_name,state,gateway,cidr > > from networks where id < 204; > > > > +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ > > | id | name | broadcast_uri | broadcast_domain_type | guru_name > > | state | gateway | cidr | > > > > +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ > > | 200 | NULL | NULL | Vxlan | PublicNetworkGuru > > | Setup | NULL | NULL | > > | 201 | NULL | NULL | Vxlan | PodBasedNetworkGuru > > | Setup | NULL | NULL | > > | 202 | NULL | NULL | LinkLocal | ControlNetworkGuru > > | Setup | 169.254.0.1 | 169.254.0.0/16 | > > | 203 | NULL | NULL | Native | StorageNetworkGuru > > | Setup | NULL | NULL | > > > > +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ > > > > I can see some vnics associated with them: > > > > select instance_id,network_id,state,broadcast_uri,isolation_uri from nics > > where network_id<204 and removed is null; > > +-------------+------------+----------+-----------------+-----------------+ > > | instance_id | network_id | state | broadcast_uri | isolation_uri | > > +-------------+------------+----------+-----------------+-----------------+ > > | 1649 | 202 | Reserved | NULL | NULL | > > | 1649 | 200 | Reserved | vlan://untagged | vlan://untagged | > > | 1650 | 202 | Reserved | NULL | NULL | > > | 1650 | 200 | Reserved | vlan://untagged | vlan://untagged | > > | 1651 | 202 | Reserved | NULL | NULL | > > | 1651 | 200 | Reserved | vlan://untagged | vlan://untagged | > > | 1653 | 202 | Reserved | NULL | NULL | > > | 1653 | 200 | Reserved | vlan://untagged | vlan://untagged | > > | 1656 | 202 | Reserved | NULL | NULL | > > | 1656 | 200 | Reserved | vlan://untagged | vlan://untagged | > > | 1659 | 202 | Reserved | NULL | NULL | > > | 1659 | 200 | Reserved | vlan://untagged | vlan://untagged | > > | 1669 | 200 | Reserved | vlan://untagged | vlan://untagged | > > | 1669 | 202 | Reserved | NULL | NULL | > > | 1669 | 201 | Reserved | NULL | NULL | > > | 1670 | 200 | Reserved | vlan://untagged | vlan://untagged | > > | 1670 | 202 | Reserved | NULL | NULL | > > | 1670 | 201 | Reserved | NULL | NULL | > > | 1671 | 202 | Reserved | NULL | NULL | > > | 1671 | 200 | Reserved | vlan://untagged | vlan://untagged | > > | 1672 | 202 | Reserved | NULL | NULL | > > | 1672 | 200 | Reserved | vlan://untagged | vlan://untagged | > > | 1673 | 202 | Reserved | NULL | NULL | > > | 1673 | 200 | Reserved | vlan://untagged | vlan://untagged | > > | 1684 | 202 | Reserved | NULL | NULL | > > | 1684 | 200 | Reserved | vlan://untagged | vlan://untagged | > > | 1690 | 202 | Reserved | NULL | NULL | > > | 1690 | 200 | Reserved | vlan://untagged | vlan://untagged | > > +-------------+------------+----------+-----------------+-----------------+ > > 28 rows in set (0.002 sec) > > > > Hmm, maybe the solution is to have vlan://untagged in the broadcast_uri > > field for those 4 in the networks table ? > > > > > > > > > > On Thu, Jun 13, 2024 at 8:19 PM Wei ZHOU <ustcweiz...@gmail.com> wrote: > > > >> Can you run the following sql query ? > >> > >> select id,name,broadcast_uri from networks where broadcast_uri IS NOT > >> NULL and removed is NULL; > >> > >> -Wei > >> > >> On Thu, Jun 13, 2024 at 11:35 AM Curious Pandora <p4nd...@gmail.com> > >> wrote: > >> > > >> > Hello, for vnet null there seems to be none: > >> > > >> > select * from op_dc_vnet_alloc where vnet IS NULL; > >> > Empty set (0.001 sec) > >> > > >> > For API CALL: > >> > GET > >> listall=true&page=1&pagesize=20&command=listGuestVlans&response=json > >> > > >> > For vlan table: > >> > select * from vlan where removed is null; > >> > > >> +----+--------------------------------------+-----------------+--------------+---------------+---------------------------+----------------+----------------+------------+---------------------+----------------------+------------------------+-----------+---------+---------------------+ > >> > | id | uuid | vlan_id | > >> > vlan_gateway | vlan_netmask | ip4_range | vlan_type > >> | > >> > data_center_id | network_id | physical_network_id | ip6_gateway > >> | > >> > ip6_cidr | ip6_range | removed | created | > >> > > >> +----+--------------------------------------+-----------------+--------------+---------------+---------------------------+----------------+----------------+------------+---------------------+----------------------+------------------------+-----------+---------+---------------------+ > >> > | 2 | 25169e64-8ab5-48b5-9420-ed2d4201a06d | vlan://untagged | > >> xx.xx.xx.1 > >> > | 255.255.254.0 | xx.xx.xx.2-xx.xx.xx.254 | VirtualNetwork | > >> > 2 | 200 | 200 | NULL | NULL > >> > | NULL | NULL | 2019-12-28 16:34:39 | > >> > | 35 | 860f0e1e-b3a2-4183-9d0d-5469a409f5b5 | vlan://untagged | NULL > >> > | NULL | NULL | VirtualNetwork | > >> > 2 | 200 | 200 | 2a0f:xxxx:xxxx:xxxx::1 | > >> > 2a0f:xxxx:xxx:xxxx::/64 | NULL | NULL | 2022-06-17 17:19:35 | > >> > > >> +----+--------------------------------------+-----------------+--------------+---------------+---------------------------+----------------+----------------+------------+---------------------+----------------------+------------------------+-----------+---------+---------------------+ > >> > > >> > and op_dc_vnet_allocs: > >> > select * from op_dc_vnet_alloc where account_id > 1; > >> > > >> +-------+-------+---------------------+----------------+--------------------------------------+------------+---------------------+---------------------+ > >> > | id | vnet | physical_network_id | data_center_id | reservation_id > >> > | account_id | taken | > >> account_vnet_map_id | > >> > > >> +-------+-------+---------------------+----------------+--------------------------------------+------------+---------------------+---------------------+ > >> > | 9685 | 30000 | 200 | 2 | > >> > 52cdb05f-7124-4b48-9948-3e17af5280ea | 106 | 2024-05-01 07:56:23 > >> | > >> > NULL | > >> > | 11637 | 30244 | 200 | 2 | > >> > 46148850-2956-45e7-88e6-26dd86e9001c | 23 | 2020-03-11 00:22:02 > >> | > >> > NULL | > >> > | 11769 | 30238 | 200 | 2 | > >> > 2bd525bf-f714-41f2-a94b-e8ef7fe44994 | 16 | 2020-03-20 00:20:08 > >> | > >> > NULL | > >> > | 11857 | 30229 | 200 | 2 | > >> > 7c761f3d-8465-4669-988c-fa6f7a84ebd0 | 10 | 2020-01-06 19:07:32 > >> | > >> > NULL | > >> > | 12131 | 30293 | 200 | 2 | > >> > 64900dfd-48d4-4101-9b04-291e4bb23e81 | 96 | 2022-11-01 15:49:36 > >> | > >> > NULL | > >> > | 12203 | 30289 | 200 | 2 | > >> > 8c0da5da-b4f0-4da2-b7e5-4971b750e96d | 93 | 2022-07-27 05:37:07 > >> | > >> > NULL | > >> > | 12269 | 30273 | 200 | 2 | > >> > 18eec34a-639b-4f3b-af3e-f00392062361 | 80 | 2021-11-09 22:24:42 > >> | > >> > NULL | > >> > | 12491 | 30380 | 200 | 2 | > >> > eb599359-6ce9-46eb-912c-08de03e78ea8 | 101 | 2023-12-23 05:07:56 > >> | > >> > NULL | > >> > | 18463 | 54996 | 200 | 2 | > >> > 5213b778-9687-4b29-b0cc-1dfaaf6bb935 | 8 | 2020-01-09 15:54:38 > >> | > >> > 16 | > >> > | 18467 | 54995 | 200 | 2 | > >> > 5c21ade2-241f-4b2b-8612-6f51a28d729d | 8 | 2020-03-10 21:10:18 > >> | > >> > 16 | > >> > | 18471 | 54994 | 200 | 2 | > >> > 93a51305-3b17-4773-92b7-349e6ee75668 | 8 | 2023-12-23 05:04:49 > >> | > >> > 16 | > >> > | 18475 | 54993 | 200 | 2 | > >> > 8bbb8c50-5301-4ee8-a195-444d13cf17e8 | 8 | 2020-01-09 15:54:09 > >> | > >> > 16 | > >> > | 18481 | 54999 | 200 | 2 | > >> > de553e5e-5764-4cc8-8c9d-41a06f77aec9 | 8 | 2020-01-15 20:15:38 > >> | > >> > 16 | > >> > | 18485 | 54998 | 200 | 2 | > >> > 87d967c6-4972-47dc-b55c-2ff740aa9293 | 8 | 2020-01-07 14:15:14 > >> | > >> > 16 | > >> > | 18489 | 54997 | 200 | 2 | > >> > fb62bd1d-7314-4a7b-a3b8-a53036916e01 | 8 | 2020-01-15 20:17:10 > >> | > >> > 16 | > >> > | 19215 | 30999 | 200 | 2 | > >> > 9e4f1ae0-064d-4400-b1b5-d12b5a350ba6 | 89 | 2023-12-23 05:40:11 > >> | > >> > NULL | > >> > | 35639 | 55000 | 200 | 2 | > >> > e4fc5468-6739-4ffc-970f-1a6a77d6c9e7 | 8 | 2020-01-15 20:16:26 > >> | > >> > 16 | > >> > > >> +-------+-------+---------------------+----------------+--------------------------------------+------------+---------------------+---------------------+ > >> > > >> > select count(*) from op_dc_vnet_alloc; > >> > +----------+ > >> > | count(*) | > >> > +----------+ > >> > | 25001 | > >> > +----------+ > >> > > >> > > >> > On Thu, Jun 13, 2024 at 11:50 AM Wei ZHOU <ustcweiz...@gmail.com> > >> wrote: > >> > > >> > > can you check "op_dc_vnet_alloc" table if there is a record with empty > >> > > "vnet" ? > >> > > > >> > > select * from op_dc_vnet_alloc where vnet IS NULL; > >> > > > >> > > > >> > > -Wei > >> > > > >> > > On Thu, Jun 13, 2024 at 10:02 AM Curious Pandora <p4nd...@gmail.com> > >> > > wrote: > >> > > > > >> > > > Hello, > >> > > > > >> > > > we get a 503 error when trying to list guest vlans (both from web > >> and api > >> > > > call). > >> > > > > >> > > > The relevant error log in the management server is: > >> > > > > >> > > > ERROR [c.c.a.ApiServer] (qtp1278254413-1728:ctx-d3eec0e8 > >> ctx-7b04643e) > >> > > > (logid:6a2d2a44) unhandled exception executing api command: > >> > > > [Ljava.lang.String;@1a84b3d5 > >> > > > java.lang.NullPointerException > >> > > > at > >> > > > > >> > > > >> com.cloud.network.dao.NetworkDaoImpl.listByZoneAndUriAndGuestType(NetworkDaoImpl.java:464) > >> > > > at > >> jdk.internal.reflect.GeneratedMethodAccessor666.invoke(Unknown > >> > > > Source) > >> > > > at > >> > > > > >> > > > >> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > > > at > >> java.base/java.lang.reflect.Method.invoke(Method.java:566) > >> > > > at > >> > > > > >> > > > >> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) > >> > > > at > >> > > > > >> > > > >> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > >> > > > at > >> > > > > >> > > > >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > >> > > > at > >> > > > > >> > > > >> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) > >> > > > at > >> > > > > >> > > > >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) > >> > > > at > >> > > > > >> > > > >> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > >> > > > at > >> > > > > >> > > > >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > >> > > > at > >> > > > > >> > > > >> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) > >> > > > at > >> com.sun.proxy.$Proxy52.listByZoneAndUriAndGuestType(Unknown > >> > > > Source) > >> > > > at > >> > > > > >> > > > >> com.cloud.api.ApiResponseHelper.createGuestVlanResponse(ApiResponseHelper.java:4943) > >> > > > at > >> > > > > >> > > > >> org.apache.cloudstack.api.command.admin.network.ListGuestVlansCmd.execute(ListGuestVlansCmd.java:98) > >> > > > at > >> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172) > >> > > > at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803) > >> > > > at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624) > >> > > > at > >> > > > > >> com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342) > >> > > > at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149) > >> > > > at > >> > > > > >> > > > >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > >> > > > at > >> > > > > >> > > > >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > >> > > > at > >> > > > > >> > > > >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > >> > > > at > >> com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146) > >> > > > at com.cloud.api.ApiServlet.doGet(ApiServlet.java:100) > >> > > > at > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:645) > >> > > > at > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:750) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450) > >> > > > at > >> > > > > >> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) > >> > > > at > >> > > > > >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > >> > > > at org.eclipse.jetty.server.Server.handle(Server.java:516) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) > >> > > > at > >> > > > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) > >> > > > at > >> > > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) > >> > > > at > >> > > > org.eclipse.jetty.io > >> > > > >> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > >> > > > at org.eclipse.jetty.io > >> > > .FillInterest.fillable(FillInterest.java:105) > >> > > > at > >> > > > org.eclipse.jetty.io > >> .ChannelEndPoint$1.run(ChannelEndPoint.java:104) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > >> > > > at > >> > > > > >> > > > >> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > >> > > > at java.base/java.lang.Thread.run(Thread.java:829) > >> > > > > >> > > > The relevant table in database (op_dc_vnet_alloc) seems to be > >> populated > >> > > > correctly. > >> > > > > >> > > > Any ideas how to debug this ? > >> > > > > >> > > > -- > >> > > > p4nd0ra - the curious > >> > > > >> > > >> > > >> > -- > >> > p4nd0ra - the curious > >> > > > > > > -- > > p4nd0ra - the curious > > > > > -- > p4nd0ra - the curious
