All entries from select id,vnet from op_dc_vnet_alloc are populated. For the second query we have the same results as before:
+-----+-----------------------------+---------------+ | id | name | broadcast_uri | +-----+-----------------------------+---------------+ | 200 | NULL | NULL | | 201 | NULL | NULL | | 202 | NULL | NULL | | 203 | NULL | NULL | select id,name,broadcast_uri,broadcast_domain_type,guru_name,state,gateway,cidr from networks where id < 204; +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ | id | name | broadcast_uri | broadcast_domain_type | guru_name | state | gateway | cidr | +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ | 200 | NULL | NULL | Vxlan | PublicNetworkGuru | Setup | NULL | NULL | | 201 | NULL | NULL | Vxlan | PodBasedNetworkGuru | Setup | NULL | NULL | | 202 | NULL | NULL | LinkLocal | ControlNetworkGuru | Setup | 169.254.0.1 | 169.254.0.0/16 | | 203 | NULL | NULL | Native | StorageNetworkGuru | Setup | NULL | NULL | +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ By doing this: update networks set broadcast_uri='vlan://untagged' where id<204; the problem goes away. If i remember correctly the broadcast_uri for PublicNetworkGuru, PodBasedNetworkGuru, ControlNetworkGuru and StorageNetworkGuru is null by default. On Fri, Jun 14, 2024 at 3:39 PM Wei ZHOU <ustcweiz...@gmail.com> wrote: > from the code > > https://github.com/apache/cloudstack/blob/cb9b3134f7fff972b63d8565a4d021f8ea918903/engine/schema/src/main/java/com/cloud/network/dao/NetworkDaoImpl.java#L464 > > it looks like either broadcast_uri of a network or vnet of a record > op_dc_vnet_alloc is null or unsupported. > > can you share the following ? > > select id,vnet from op_dc_vnet_alloc; > select id,broadcast_uri from networks where data_center_id is not null > and broadcast_uri is not null; > > > -Wei > > On Fri, Jun 14, 2024 at 1:39 PM Curious Pandora <p4nd...@gmail.com> wrote: > > > > @Wei ZHOU any thoughts ? > > > > On Thu, Jun 13, 2024 at 9:29 PM Curious Pandora <p4nd...@gmail.com> > wrote: > > > > > There are indeed 4 networks with NULL broadcast_uri that are causing > the > > > problem. I think those are systemic for cloudstack. > > > +-----+-----------------------------+---------------+ > > > | id | name | broadcast_uri | > > > +-----+-----------------------------+---------------+ > > > | 200 | NULL | NULL | > > > | 201 | NULL | NULL | > > > | 202 | NULL | NULL | > > > | 203 | NULL | NULL | > > > > > > select > > > > id,name,broadcast_uri,broadcast_domain_type,guru_name,state,gateway,cidr > > > from networks where id < 204; > > > > > > > +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ > > > | id | name | broadcast_uri | broadcast_domain_type | guru_name > > > | state | gateway | cidr | > > > > > > > +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ > > > | 200 | NULL | NULL | Vxlan | > PublicNetworkGuru > > > | Setup | NULL | NULL | > > > | 201 | NULL | NULL | Vxlan | > PodBasedNetworkGuru > > > | Setup | NULL | NULL | > > > | 202 | NULL | NULL | LinkLocal | > ControlNetworkGuru > > > | Setup | 169.254.0.1 | 169.254.0.0/16 | > > > | 203 | NULL | NULL | Native | > StorageNetworkGuru > > > | Setup | NULL | NULL | > > > > > > > +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ > > > > > > I can see some vnics associated with them: > > > > > > select instance_id,network_id,state,broadcast_uri,isolation_uri from > nics > > > where network_id<204 and removed is null; > > > > +-------------+------------+----------+-----------------+-----------------+ > > > | instance_id | network_id | state | broadcast_uri | > isolation_uri | > > > > +-------------+------------+----------+-----------------+-----------------+ > > > | 1649 | 202 | Reserved | NULL | NULL > | > > > | 1649 | 200 | Reserved | vlan://untagged | > vlan://untagged | > > > | 1650 | 202 | Reserved | NULL | NULL > | > > > | 1650 | 200 | Reserved | vlan://untagged | > vlan://untagged | > > > | 1651 | 202 | Reserved | NULL | NULL > | > > > | 1651 | 200 | Reserved | vlan://untagged | > vlan://untagged | > > > | 1653 | 202 | Reserved | NULL | NULL > | > > > | 1653 | 200 | Reserved | vlan://untagged | > vlan://untagged | > > > | 1656 | 202 | Reserved | NULL | NULL > | > > > | 1656 | 200 | Reserved | vlan://untagged | > vlan://untagged | > > > | 1659 | 202 | Reserved | NULL | NULL > | > > > | 1659 | 200 | Reserved | vlan://untagged | > vlan://untagged | > > > | 1669 | 200 | Reserved | vlan://untagged | > vlan://untagged | > > > | 1669 | 202 | Reserved | NULL | NULL > | > > > | 1669 | 201 | Reserved | NULL | NULL > | > > > | 1670 | 200 | Reserved | vlan://untagged | > vlan://untagged | > > > | 1670 | 202 | Reserved | NULL | NULL > | > > > | 1670 | 201 | Reserved | NULL | NULL > | > > > | 1671 | 202 | Reserved | NULL | NULL > | > > > | 1671 | 200 | Reserved | vlan://untagged | > vlan://untagged | > > > | 1672 | 202 | Reserved | NULL | NULL > | > > > | 1672 | 200 | Reserved | vlan://untagged | > vlan://untagged | > > > | 1673 | 202 | Reserved | NULL | NULL > | > > > | 1673 | 200 | Reserved | vlan://untagged | > vlan://untagged | > > > | 1684 | 202 | Reserved | NULL | NULL > | > > > | 1684 | 200 | Reserved | vlan://untagged | > vlan://untagged | > > > | 1690 | 202 | Reserved | NULL | NULL > | > > > | 1690 | 200 | Reserved | vlan://untagged | > vlan://untagged | > > > > +-------------+------------+----------+-----------------+-----------------+ > > > 28 rows in set (0.002 sec) > > > > > > Hmm, maybe the solution is to have vlan://untagged in the broadcast_uri > > > field for those 4 in the networks table ? > > > > > > > > > > > > > > > On Thu, Jun 13, 2024 at 8:19 PM Wei ZHOU <ustcweiz...@gmail.com> > wrote: > > > > > >> Can you run the following sql query ? > > >> > > >> select id,name,broadcast_uri from networks where broadcast_uri IS NOT > > >> NULL and removed is NULL; > > >> > > >> -Wei > > >> > > >> On Thu, Jun 13, 2024 at 11:35 AM Curious Pandora <p4nd...@gmail.com> > > >> wrote: > > >> > > > >> > Hello, for vnet null there seems to be none: > > >> > > > >> > select * from op_dc_vnet_alloc where vnet IS NULL; > > >> > Empty set (0.001 sec) > > >> > > > >> > For API CALL: > > >> > GET > > >> listall=true&page=1&pagesize=20&command=listGuestVlans&response=json > > >> > > > >> > For vlan table: > > >> > select * from vlan where removed is null; > > >> > > > >> > +----+--------------------------------------+-----------------+--------------+---------------+---------------------------+----------------+----------------+------------+---------------------+----------------------+------------------------+-----------+---------+---------------------+ > > >> > | id | uuid | vlan_id | > > >> > vlan_gateway | vlan_netmask | ip4_range | vlan_type > > >> | > > >> > data_center_id | network_id | physical_network_id | ip6_gateway > > >> | > > >> > ip6_cidr | ip6_range | removed | created | > > >> > > > >> > +----+--------------------------------------+-----------------+--------------+---------------+---------------------------+----------------+----------------+------------+---------------------+----------------------+------------------------+-----------+---------+---------------------+ > > >> > | 2 | 25169e64-8ab5-48b5-9420-ed2d4201a06d | vlan://untagged | > > >> xx.xx.xx.1 > > >> > | 255.255.254.0 | xx.xx.xx.2-xx.xx.xx.254 | VirtualNetwork | > > >> > 2 | 200 | 200 | NULL | NULL > > >> > | NULL | NULL | 2019-12-28 16:34:39 | > > >> > | 35 | 860f0e1e-b3a2-4183-9d0d-5469a409f5b5 | vlan://untagged | NULL > > >> > | NULL | NULL | VirtualNetwork | > > >> > 2 | 200 | 200 | 2a0f:xxxx:xxxx:xxxx::1 | > > >> > 2a0f:xxxx:xxx:xxxx::/64 | NULL | NULL | 2022-06-17 17:19:35 > | > > >> > > > >> > +----+--------------------------------------+-----------------+--------------+---------------+---------------------------+----------------+----------------+------------+---------------------+----------------------+------------------------+-----------+---------+---------------------+ > > >> > > > >> > and op_dc_vnet_allocs: > > >> > select * from op_dc_vnet_alloc where account_id > 1; > > >> > > > >> > +-------+-------+---------------------+----------------+--------------------------------------+------------+---------------------+---------------------+ > > >> > | id | vnet | physical_network_id | data_center_id | > reservation_id > > >> > | account_id | taken | > > >> account_vnet_map_id | > > >> > > > >> > +-------+-------+---------------------+----------------+--------------------------------------+------------+---------------------+---------------------+ > > >> > | 9685 | 30000 | 200 | 2 | > > >> > 52cdb05f-7124-4b48-9948-3e17af5280ea | 106 | 2024-05-01 > 07:56:23 > > >> | > > >> > NULL | > > >> > | 11637 | 30244 | 200 | 2 | > > >> > 46148850-2956-45e7-88e6-26dd86e9001c | 23 | 2020-03-11 > 00:22:02 > > >> | > > >> > NULL | > > >> > | 11769 | 30238 | 200 | 2 | > > >> > 2bd525bf-f714-41f2-a94b-e8ef7fe44994 | 16 | 2020-03-20 > 00:20:08 > > >> | > > >> > NULL | > > >> > | 11857 | 30229 | 200 | 2 | > > >> > 7c761f3d-8465-4669-988c-fa6f7a84ebd0 | 10 | 2020-01-06 > 19:07:32 > > >> | > > >> > NULL | > > >> > | 12131 | 30293 | 200 | 2 | > > >> > 64900dfd-48d4-4101-9b04-291e4bb23e81 | 96 | 2022-11-01 > 15:49:36 > > >> | > > >> > NULL | > > >> > | 12203 | 30289 | 200 | 2 | > > >> > 8c0da5da-b4f0-4da2-b7e5-4971b750e96d | 93 | 2022-07-27 > 05:37:07 > > >> | > > >> > NULL | > > >> > | 12269 | 30273 | 200 | 2 | > > >> > 18eec34a-639b-4f3b-af3e-f00392062361 | 80 | 2021-11-09 > 22:24:42 > > >> | > > >> > NULL | > > >> > | 12491 | 30380 | 200 | 2 | > > >> > eb599359-6ce9-46eb-912c-08de03e78ea8 | 101 | 2023-12-23 > 05:07:56 > > >> | > > >> > NULL | > > >> > | 18463 | 54996 | 200 | 2 | > > >> > 5213b778-9687-4b29-b0cc-1dfaaf6bb935 | 8 | 2020-01-09 > 15:54:38 > > >> | > > >> > 16 | > > >> > | 18467 | 54995 | 200 | 2 | > > >> > 5c21ade2-241f-4b2b-8612-6f51a28d729d | 8 | 2020-03-10 > 21:10:18 > > >> | > > >> > 16 | > > >> > | 18471 | 54994 | 200 | 2 | > > >> > 93a51305-3b17-4773-92b7-349e6ee75668 | 8 | 2023-12-23 > 05:04:49 > > >> | > > >> > 16 | > > >> > | 18475 | 54993 | 200 | 2 | > > >> > 8bbb8c50-5301-4ee8-a195-444d13cf17e8 | 8 | 2020-01-09 > 15:54:09 > > >> | > > >> > 16 | > > >> > | 18481 | 54999 | 200 | 2 | > > >> > de553e5e-5764-4cc8-8c9d-41a06f77aec9 | 8 | 2020-01-15 > 20:15:38 > > >> | > > >> > 16 | > > >> > | 18485 | 54998 | 200 | 2 | > > >> > 87d967c6-4972-47dc-b55c-2ff740aa9293 | 8 | 2020-01-07 > 14:15:14 > > >> | > > >> > 16 | > > >> > | 18489 | 54997 | 200 | 2 | > > >> > fb62bd1d-7314-4a7b-a3b8-a53036916e01 | 8 | 2020-01-15 > 20:17:10 > > >> | > > >> > 16 | > > >> > | 19215 | 30999 | 200 | 2 | > > >> > 9e4f1ae0-064d-4400-b1b5-d12b5a350ba6 | 89 | 2023-12-23 > 05:40:11 > > >> | > > >> > NULL | > > >> > | 35639 | 55000 | 200 | 2 | > > >> > e4fc5468-6739-4ffc-970f-1a6a77d6c9e7 | 8 | 2020-01-15 > 20:16:26 > > >> | > > >> > 16 | > > >> > > > >> > +-------+-------+---------------------+----------------+--------------------------------------+------------+---------------------+---------------------+ > > >> > > > >> > select count(*) from op_dc_vnet_alloc; > > >> > +----------+ > > >> > | count(*) | > > >> > +----------+ > > >> > | 25001 | > > >> > +----------+ > > >> > > > >> > > > >> > On Thu, Jun 13, 2024 at 11:50 AM Wei ZHOU <ustcweiz...@gmail.com> > > >> wrote: > > >> > > > >> > > can you check "op_dc_vnet_alloc" table if there is a record with > empty > > >> > > "vnet" ? > > >> > > > > >> > > select * from op_dc_vnet_alloc where vnet IS NULL; > > >> > > > > >> > > > > >> > > -Wei > > >> > > > > >> > > On Thu, Jun 13, 2024 at 10:02 AM Curious Pandora < > p4nd...@gmail.com> > > >> > > wrote: > > >> > > > > > >> > > > Hello, > > >> > > > > > >> > > > we get a 503 error when trying to list guest vlans (both from > web > > >> and api > > >> > > > call). > > >> > > > > > >> > > > The relevant error log in the management server is: > > >> > > > > > >> > > > ERROR [c.c.a.ApiServer] (qtp1278254413-1728:ctx-d3eec0e8 > > >> ctx-7b04643e) > > >> > > > (logid:6a2d2a44) unhandled exception executing api command: > > >> > > > [Ljava.lang.String;@1a84b3d5 > > >> > > > java.lang.NullPointerException > > >> > > > at > > >> > > > > > >> > > > > >> > com.cloud.network.dao.NetworkDaoImpl.listByZoneAndUriAndGuestType(NetworkDaoImpl.java:464) > > >> > > > at > > >> jdk.internal.reflect.GeneratedMethodAccessor666.invoke(Unknown > > >> > > > Source) > > >> > > > at > > >> > > > > > >> > > > > >> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > >> > > > at > > >> java.base/java.lang.reflect.Method.invoke(Method.java:566) > > >> > > > at > > >> > > > > > >> > > > > >> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) > > >> > > > at > > >> > > > > > >> > > > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > > >> > > > at > > >> > > > > > >> > > > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > >> > > > at > > >> > > > > > >> > > > > >> > com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) > > >> > > > at > > >> > > > > > >> > > > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) > > >> > > > at > > >> > > > > > >> > > > > >> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > > >> > > > at > > >> > > > > > >> > > > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > > >> > > > at > > >> > > > > > >> > > > > >> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) > > >> > > > at > > >> com.sun.proxy.$Proxy52.listByZoneAndUriAndGuestType(Unknown > > >> > > > Source) > > >> > > > at > > >> > > > > > >> > > > > >> > com.cloud.api.ApiResponseHelper.createGuestVlanResponse(ApiResponseHelper.java:4943) > > >> > > > at > > >> > > > > > >> > > > > >> > org.apache.cloudstack.api.command.admin.network.ListGuestVlansCmd.execute(ListGuestVlansCmd.java:98) > > >> > > > at > > >> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172) > > >> > > > at > com.cloud.api.ApiServer.queueCommand(ApiServer.java:803) > > >> > > > at > com.cloud.api.ApiServer.handleRequest(ApiServer.java:624) > > >> > > > at > > >> > > > > > >> com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342) > > >> > > > at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149) > > >> > > > at > > >> > > > > > >> > > > > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > > >> > > > at > > >> > > > > > >> > > > > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > > >> > > > at > > >> > > > > > >> > > > > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > > >> > > > at > > >> com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146) > > >> > > > at com.cloud.api.ApiServlet.doGet(ApiServlet.java:100) > > >> > > > at > > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:645) > > >> > > > at > > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:750) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450) > > >> > > > at > > >> > > > > > >> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) > > >> > > > at > > >> > > > > > >> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > > >> > > > at > org.eclipse.jetty.server.Server.handle(Server.java:516) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) > > >> > > > at > > >> > > > > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) > > >> > > > at > > >> > > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) > > >> > > > at > > >> > > > org.eclipse.jetty.io > > >> > > > > >> > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > > >> > > > at org.eclipse.jetty.io > > >> > > .FillInterest.fillable(FillInterest.java:105) > > >> > > > at > > >> > > > org.eclipse.jetty.io > > >> .ChannelEndPoint$1.run(ChannelEndPoint.java:104) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > >> > > > at > > >> > > > > > >> > > > > >> > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > >> > > > at java.base/java.lang.Thread.run(Thread.java:829) > > >> > > > > > >> > > > The relevant table in database (op_dc_vnet_alloc) seems to be > > >> populated > > >> > > > correctly. > > >> > > > > > >> > > > Any ideas how to debug this ? > > >> > > > > > >> > > > -- > > >> > > > p4nd0ra - the curious > > >> > > > > >> > > > >> > > > >> > -- > > >> > p4nd0ra - the curious > > >> > > > > > > > > > -- > > > p4nd0ra - the curious > > > > > > > > > -- > > p4nd0ra - the curious > -- p4nd0ra - the curious
