@Wei ZHOU any thoughts ? On Thu, Jun 13, 2024 at 9:29 PM Curious Pandora <p4nd...@gmail.com> wrote:
> There are indeed 4 networks with NULL broadcast_uri that are causing the > problem. I think those are systemic for cloudstack. > +-----+-----------------------------+---------------+ > | id | name | broadcast_uri | > +-----+-----------------------------+---------------+ > | 200 | NULL | NULL | > | 201 | NULL | NULL | > | 202 | NULL | NULL | > | 203 | NULL | NULL | > > select > id,name,broadcast_uri,broadcast_domain_type,guru_name,state,gateway,cidr > from networks where id < 204; > > +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ > | id | name | broadcast_uri | broadcast_domain_type | guru_name > | state | gateway | cidr | > > +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ > | 200 | NULL | NULL | Vxlan | PublicNetworkGuru > | Setup | NULL | NULL | > | 201 | NULL | NULL | Vxlan | PodBasedNetworkGuru > | Setup | NULL | NULL | > | 202 | NULL | NULL | LinkLocal | ControlNetworkGuru > | Setup | 169.254.0.1 | 169.254.0.0/16 | > | 203 | NULL | NULL | Native | StorageNetworkGuru > | Setup | NULL | NULL | > > +-----+------+---------------+-----------------------+---------------------+-------+-------------+----------------+ > > I can see some vnics associated with them: > > select instance_id,network_id,state,broadcast_uri,isolation_uri from nics > where network_id<204 and removed is null; > +-------------+------------+----------+-----------------+-----------------+ > | instance_id | network_id | state | broadcast_uri | isolation_uri | > +-------------+------------+----------+-----------------+-----------------+ > | 1649 | 202 | Reserved | NULL | NULL | > | 1649 | 200 | Reserved | vlan://untagged | vlan://untagged | > | 1650 | 202 | Reserved | NULL | NULL | > | 1650 | 200 | Reserved | vlan://untagged | vlan://untagged | > | 1651 | 202 | Reserved | NULL | NULL | > | 1651 | 200 | Reserved | vlan://untagged | vlan://untagged | > | 1653 | 202 | Reserved | NULL | NULL | > | 1653 | 200 | Reserved | vlan://untagged | vlan://untagged | > | 1656 | 202 | Reserved | NULL | NULL | > | 1656 | 200 | Reserved | vlan://untagged | vlan://untagged | > | 1659 | 202 | Reserved | NULL | NULL | > | 1659 | 200 | Reserved | vlan://untagged | vlan://untagged | > | 1669 | 200 | Reserved | vlan://untagged | vlan://untagged | > | 1669 | 202 | Reserved | NULL | NULL | > | 1669 | 201 | Reserved | NULL | NULL | > | 1670 | 200 | Reserved | vlan://untagged | vlan://untagged | > | 1670 | 202 | Reserved | NULL | NULL | > | 1670 | 201 | Reserved | NULL | NULL | > | 1671 | 202 | Reserved | NULL | NULL | > | 1671 | 200 | Reserved | vlan://untagged | vlan://untagged | > | 1672 | 202 | Reserved | NULL | NULL | > | 1672 | 200 | Reserved | vlan://untagged | vlan://untagged | > | 1673 | 202 | Reserved | NULL | NULL | > | 1673 | 200 | Reserved | vlan://untagged | vlan://untagged | > | 1684 | 202 | Reserved | NULL | NULL | > | 1684 | 200 | Reserved | vlan://untagged | vlan://untagged | > | 1690 | 202 | Reserved | NULL | NULL | > | 1690 | 200 | Reserved | vlan://untagged | vlan://untagged | > +-------------+------------+----------+-----------------+-----------------+ > 28 rows in set (0.002 sec) > > Hmm, maybe the solution is to have vlan://untagged in the broadcast_uri > field for those 4 in the networks table ? > > > > > On Thu, Jun 13, 2024 at 8:19 PM Wei ZHOU <ustcweiz...@gmail.com> wrote: > >> Can you run the following sql query ? >> >> select id,name,broadcast_uri from networks where broadcast_uri IS NOT >> NULL and removed is NULL; >> >> -Wei >> >> On Thu, Jun 13, 2024 at 11:35 AM Curious Pandora <p4nd...@gmail.com> >> wrote: >> > >> > Hello, for vnet null there seems to be none: >> > >> > select * from op_dc_vnet_alloc where vnet IS NULL; >> > Empty set (0.001 sec) >> > >> > For API CALL: >> > GET >> listall=true&page=1&pagesize=20&command=listGuestVlans&response=json >> > >> > For vlan table: >> > select * from vlan where removed is null; >> > >> +----+--------------------------------------+-----------------+--------------+---------------+---------------------------+----------------+----------------+------------+---------------------+----------------------+------------------------+-----------+---------+---------------------+ >> > | id | uuid | vlan_id | >> > vlan_gateway | vlan_netmask | ip4_range | vlan_type >> | >> > data_center_id | network_id | physical_network_id | ip6_gateway >> | >> > ip6_cidr | ip6_range | removed | created | >> > >> +----+--------------------------------------+-----------------+--------------+---------------+---------------------------+----------------+----------------+------------+---------------------+----------------------+------------------------+-----------+---------+---------------------+ >> > | 2 | 25169e64-8ab5-48b5-9420-ed2d4201a06d | vlan://untagged | >> xx.xx.xx.1 >> > | 255.255.254.0 | xx.xx.xx.2-xx.xx.xx.254 | VirtualNetwork | >> > 2 | 200 | 200 | NULL | NULL >> > | NULL | NULL | 2019-12-28 16:34:39 | >> > | 35 | 860f0e1e-b3a2-4183-9d0d-5469a409f5b5 | vlan://untagged | NULL >> > | NULL | NULL | VirtualNetwork | >> > 2 | 200 | 200 | 2a0f:xxxx:xxxx:xxxx::1 | >> > 2a0f:xxxx:xxx:xxxx::/64 | NULL | NULL | 2022-06-17 17:19:35 | >> > >> +----+--------------------------------------+-----------------+--------------+---------------+---------------------------+----------------+----------------+------------+---------------------+----------------------+------------------------+-----------+---------+---------------------+ >> > >> > and op_dc_vnet_allocs: >> > select * from op_dc_vnet_alloc where account_id > 1; >> > >> +-------+-------+---------------------+----------------+--------------------------------------+------------+---------------------+---------------------+ >> > | id | vnet | physical_network_id | data_center_id | reservation_id >> > | account_id | taken | >> account_vnet_map_id | >> > >> +-------+-------+---------------------+----------------+--------------------------------------+------------+---------------------+---------------------+ >> > | 9685 | 30000 | 200 | 2 | >> > 52cdb05f-7124-4b48-9948-3e17af5280ea | 106 | 2024-05-01 07:56:23 >> | >> > NULL | >> > | 11637 | 30244 | 200 | 2 | >> > 46148850-2956-45e7-88e6-26dd86e9001c | 23 | 2020-03-11 00:22:02 >> | >> > NULL | >> > | 11769 | 30238 | 200 | 2 | >> > 2bd525bf-f714-41f2-a94b-e8ef7fe44994 | 16 | 2020-03-20 00:20:08 >> | >> > NULL | >> > | 11857 | 30229 | 200 | 2 | >> > 7c761f3d-8465-4669-988c-fa6f7a84ebd0 | 10 | 2020-01-06 19:07:32 >> | >> > NULL | >> > | 12131 | 30293 | 200 | 2 | >> > 64900dfd-48d4-4101-9b04-291e4bb23e81 | 96 | 2022-11-01 15:49:36 >> | >> > NULL | >> > | 12203 | 30289 | 200 | 2 | >> > 8c0da5da-b4f0-4da2-b7e5-4971b750e96d | 93 | 2022-07-27 05:37:07 >> | >> > NULL | >> > | 12269 | 30273 | 200 | 2 | >> > 18eec34a-639b-4f3b-af3e-f00392062361 | 80 | 2021-11-09 22:24:42 >> | >> > NULL | >> > | 12491 | 30380 | 200 | 2 | >> > eb599359-6ce9-46eb-912c-08de03e78ea8 | 101 | 2023-12-23 05:07:56 >> | >> > NULL | >> > | 18463 | 54996 | 200 | 2 | >> > 5213b778-9687-4b29-b0cc-1dfaaf6bb935 | 8 | 2020-01-09 15:54:38 >> | >> > 16 | >> > | 18467 | 54995 | 200 | 2 | >> > 5c21ade2-241f-4b2b-8612-6f51a28d729d | 8 | 2020-03-10 21:10:18 >> | >> > 16 | >> > | 18471 | 54994 | 200 | 2 | >> > 93a51305-3b17-4773-92b7-349e6ee75668 | 8 | 2023-12-23 05:04:49 >> | >> > 16 | >> > | 18475 | 54993 | 200 | 2 | >> > 8bbb8c50-5301-4ee8-a195-444d13cf17e8 | 8 | 2020-01-09 15:54:09 >> | >> > 16 | >> > | 18481 | 54999 | 200 | 2 | >> > de553e5e-5764-4cc8-8c9d-41a06f77aec9 | 8 | 2020-01-15 20:15:38 >> | >> > 16 | >> > | 18485 | 54998 | 200 | 2 | >> > 87d967c6-4972-47dc-b55c-2ff740aa9293 | 8 | 2020-01-07 14:15:14 >> | >> > 16 | >> > | 18489 | 54997 | 200 | 2 | >> > fb62bd1d-7314-4a7b-a3b8-a53036916e01 | 8 | 2020-01-15 20:17:10 >> | >> > 16 | >> > | 19215 | 30999 | 200 | 2 | >> > 9e4f1ae0-064d-4400-b1b5-d12b5a350ba6 | 89 | 2023-12-23 05:40:11 >> | >> > NULL | >> > | 35639 | 55000 | 200 | 2 | >> > e4fc5468-6739-4ffc-970f-1a6a77d6c9e7 | 8 | 2020-01-15 20:16:26 >> | >> > 16 | >> > >> +-------+-------+---------------------+----------------+--------------------------------------+------------+---------------------+---------------------+ >> > >> > select count(*) from op_dc_vnet_alloc; >> > +----------+ >> > | count(*) | >> > +----------+ >> > | 25001 | >> > +----------+ >> > >> > >> > On Thu, Jun 13, 2024 at 11:50 AM Wei ZHOU <ustcweiz...@gmail.com> >> wrote: >> > >> > > can you check "op_dc_vnet_alloc" table if there is a record with empty >> > > "vnet" ? >> > > >> > > select * from op_dc_vnet_alloc where vnet IS NULL; >> > > >> > > >> > > -Wei >> > > >> > > On Thu, Jun 13, 2024 at 10:02 AM Curious Pandora <p4nd...@gmail.com> >> > > wrote: >> > > > >> > > > Hello, >> > > > >> > > > we get a 503 error when trying to list guest vlans (both from web >> and api >> > > > call). >> > > > >> > > > The relevant error log in the management server is: >> > > > >> > > > ERROR [c.c.a.ApiServer] (qtp1278254413-1728:ctx-d3eec0e8 >> ctx-7b04643e) >> > > > (logid:6a2d2a44) unhandled exception executing api command: >> > > > [Ljava.lang.String;@1a84b3d5 >> > > > java.lang.NullPointerException >> > > > at >> > > > >> > > >> com.cloud.network.dao.NetworkDaoImpl.listByZoneAndUriAndGuestType(NetworkDaoImpl.java:464) >> > > > at >> jdk.internal.reflect.GeneratedMethodAccessor666.invoke(Unknown >> > > > Source) >> > > > at >> > > > >> > > >> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > > > at >> java.base/java.lang.reflect.Method.invoke(Method.java:566) >> > > > at >> > > > >> > > >> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) >> > > > at >> > > > >> > > >> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) >> > > > at >> > > > >> > > >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) >> > > > at >> > > > >> > > >> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) >> > > > at >> > > > >> > > >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) >> > > > at >> > > > >> > > >> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) >> > > > at >> > > > >> > > >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) >> > > > at >> > > > >> > > >> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) >> > > > at >> com.sun.proxy.$Proxy52.listByZoneAndUriAndGuestType(Unknown >> > > > Source) >> > > > at >> > > > >> > > >> com.cloud.api.ApiResponseHelper.createGuestVlanResponse(ApiResponseHelper.java:4943) >> > > > at >> > > > >> > > >> org.apache.cloudstack.api.command.admin.network.ListGuestVlansCmd.execute(ListGuestVlansCmd.java:98) >> > > > at >> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172) >> > > > at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803) >> > > > at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624) >> > > > at >> > > > >> com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342) >> > > > at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149) >> > > > at >> > > > >> > > >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) >> > > > at >> > > > >> > > >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) >> > > > at >> > > > >> > > >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) >> > > > at >> com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146) >> > > > at com.cloud.api.ApiServlet.doGet(ApiServlet.java:100) >> > > > at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:645) >> > > > at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:750) >> > > > at >> > > > >> > > >> org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450) >> > > > at >> > > > >> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) >> > > > at >> > > > >> > > >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >> > > > at >> > > > >> > > >> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) >> > > > at >> > > > >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) >> > > > at org.eclipse.jetty.server.Server.handle(Server.java:516) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) >> > > > at >> > > > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) >> > > > at >> > > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) >> > > > at >> > > > >> > > >> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) >> > > > at >> > > > org.eclipse.jetty.io >> > > >> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) >> > > > at org.eclipse.jetty.io >> > > .FillInterest.fillable(FillInterest.java:105) >> > > > at >> > > > org.eclipse.jetty.io >> .ChannelEndPoint$1.run(ChannelEndPoint.java:104) >> > > > at >> > > > >> > > >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) >> > > > at >> > > > >> > > >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) >> > > > at >> > > > >> > > >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) >> > > > at >> > > > >> > > >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) >> > > > at >> > > > >> > > >> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) >> > > > at >> > > > >> > > >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) >> > > > at >> > > > >> > > >> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) >> > > > at java.base/java.lang.Thread.run(Thread.java:829) >> > > > >> > > > The relevant table in database (op_dc_vnet_alloc) seems to be >> populated >> > > > correctly. >> > > > >> > > > Any ideas how to debug this ? >> > > > >> > > > -- >> > > > p4nd0ra - the curious >> > > >> > >> > >> > -- >> > p4nd0ra - the curious >> > > > -- > p4nd0ra - the curious > -- p4nd0ra - the curious
