Hi, I do not know what you have removed.
Can you list the hosts by id one by one to see which host causes the issue ? -Wei On Tue, Feb 18, 2025 at 1:24 AM Md Mosharaf Hossain < mosharaf.hoss...@bol-online.com> wrote: > Hi @Wei ZHOU <ustcweiz...@gmail.com> > We have removed all information from MySQL database for the host > cshost25.bol-online.com" but still unable to view the host option. > Kindly advise your expert opinion. > > > > Regards > Mosharaf Hossain > Manager, Product Development > Bangladesh Online (BOL) > > Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh > Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222 95757 > Cell: +880 1787 680828, Web: www.bol-online.com > > > > On Sun, Feb 16, 2025 at 3:36 PM Md Mosharaf Hossain < > mosharaf.hoss...@bol-online.com> wrote: > >> Hi @Wei ZHOU <ustcweiz...@gmail.com> >> Today, we attempted to remove the disconnected host using *Cloud Monkey >> tools*. As the issue began after removing the host *"cshost25.bol-online.com >> <http://cshost25.bol-online.com>"*, and we discovered *two records* in >> the database's host table. One record was deleted successfully without any >> errors, but we encountered the following error while trying to remove the >> other record. >> >> *Cloud Monkey Outputs : * >> ---------------------------------- >> (bol-cloud-admin) 🐱 > list hosts >> 🙈 Error: (HTTP 530, error code 9999) <nil> >> (bol-cloud-admin) 🐱 > delete host id=19714 forced=true >> { >> "success": true >> } >> (bol-cloud-admin) 🐱 > delete host id=19713 forced=true >> 🙈 Error: (HTTP 530, error code 9999) <nil> >> (bol-cloud-admin) 🐱 > >> >> _----------------------------------------------------------------------------------- >> >> DB query after deletion of Host ID : 19714 >> >> mysql> SELECT h.id AS host_id, h.name AS host_name, h.status, h.type, >> h.private_ip_address, h.cluster_id, c.name AS cluster_name FROM host h >> JOIN cluster c ON h.cluster | NU >> _id = c.id; >> +---------+-------------------------+--------------+-------- >> -+--------------------+------------+-------------------------+ 0 >> | host_id | host_name | status | type | >> private_ip_address | cluster_id | cluster_name | >> +---------+-------------------------+--------------+-------- >> -+--------------------+------------+-------------------------+.10.1 >> | 19198 | cshost3.bol-online.com | Up | Routing | >> 10.10.11.33 | 1 | cluster1.bol-online.com | | NU >> | 19439 | cshost5.bol-online.com | Up | Routing | >> 10.10.11.35 | 1 | cluster1.bol-online.com | >> | 19667 | cshost8.bol-online.com | Up | Routing | >> 10.10.11.38 | 1 | cluster1.bol-online.com | 0 >> | 19668 | cshost17.bol-online.com | Up | Routing | >> 10.10.11.47 | 5 | cluster2.bol-online.com | >> | 19670 | cph2n8.bol-online.com | Disconnected | Routing | >> 10.10.10.118 | 1 | cluster1.bol-online.com |.10.1 >> | 19675 | cshost18.bol-online.com | Up | Routing | >> 10.10.11.48 | 5 | cluster2.bol-online.com | | NU >> | 19676 | cshost19.bol-online.com | Up | Routing | >> 10.10.11.49 | 5 | cluster2.bol-online.com | >> | 19685 | cshost10.bol-online.com | Up | Routing | >> 10.10.11.41 | 1 | cluster1.bol-online.com | 0 >> | 19686 | cshost21.bol-online.com | Up | Routing | >> 10.10.11.51 | 5 | cluster2.bol-online.com | >> | 19687 | cshost11.bol-online.com | Up | Routing | >> 10.10.11.42 | 1 | cluster1.bol-online.com |.10.1 >> | 19691 | cshost7.bol-online.com | Up | Routing | >> 10.10.11.37 | 1 | cluster1.bol-online.com | | NU >> | 19696 | cshost6.bol-online.com | Up | Routing | >> 10.10.11.36 | 1 | cluster1.bol-online.com | >> | 19701 | cshost22.bol-online.com | Up | Routing | >> 10.10.11.52 | 6 | cluster3.bol-online.com | 0 >> | 19705 | cshost24.bol-online.com | Up | Routing | >> 10.10.11.54 | 1 | cluster1.bol-online.com | >> | 19706 | cshost23.bol-online.com | Up | Routing | >> 10.10.11.53 | 1 | cluster1.bol-online.com |.10.1 >> | 19707 | cshost2.bol-online.com | Up | Routing | >> 10.10.11.32 | 1 | cluster1.bol-online.com | | NU >> | 19709 | cshost4.bol-online.com | Up | Routing | >> 10.10.11.34 | 1 | cluster1.bol-online.com | >> | 19710 | cshost1.bol-online.com | Up | Routing | >> 10.10.11.31 | 1 | cluster1.bol-online.com | 0 >> | 19712 | cshost9.bol-online.com | Up | Routing | >> 10.10.11.39 | 1 | cluster1.bol-online.com | >> | 19713 | cshost25.bol-online.com | Disconnected | Routing | >> 10.10.11.55 | 1 | cluster1.bol-online.com |----- >> | 19715 | cshost26.bol-online.com | Up | Routing | >> 10.10.11.56 | 10 | cluster4.bol-online.com |-+--- >> | 19716 | cshost27.bol-online.com | Up | Routing | >> 10.10.11.57 | 12 | cluster5.bol-online.com |----- >> | 19717 | cshost28.bol-online.com | Up | Routing | >> 10.10.11.58 | 12 | cluster5.bol-online.com |----- >> | 19719 | cshost29.bol-online.com | Up | Routing | >> 10.10.11.59 | 12 | cluster5.bol-online.com | >> +---------+-------------------------+--------------+-------- >> -+--------------------+------------+-------------------------+ >> 24 rows in set (0.00 sec) >> _________________________________________________ >> >> *Additionally*, we enabled the debug log for *CloudMonkey*, which >> displayed the following error while listing the hosts. >> >> cloud-admin) 🐱 > list hosts id=19713 >> [debug] ExecLine line:list hosts id=19713 >> [debug] ExecCmd args: list, hosts, id=19713 >> [debug] NewAPIRequest API request URL: >> http://10.10.11.61:8080/client/api?apiKey=_A1conzDjN-7JrbMIeVyuEPU5Gw-d-yKYXj2gn9n7c8X85wdn0j_04pQZpZtI1fyv8QKDoFma0CF7gdMh5FaBA&command=listHosts&id=19713&response=json&signature=5RD%2FzDvE3w%2FQrdSkgpiA%2BFQNHH4%3D >> [debug] NewAPIRequest response status code:530 >> [debug] NewAPIRequest response >> body:{"listhostsresponse":{"uuidList":[],"errorcode":530,"cserrorcode":9999}} >> 🙈 Error: (HTTP 530, error code 9999) <nil> >> (bol-cloud-admin) 🐱 > >> >> >> >> >> Regards >> Mosharaf Hossain >> >> >> >> On Fri, Feb 14, 2025 at 9:41 AM Md Mosharaf Hossain < >> mosharaf.hoss...@bol-online.com> wrote: >> >>> Hi >>> I find the following items >>> >>> 2025-02-14 05:07:06,522 DEBUG [c.c.s.StorageManagerImpl] >>> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Total over provisioned >>> capacity of the pool Eskaton-hdd-rbd-cls1 id: 100 is (83.3988 TB) >>> 91697987977216 >>> 2025-02-14 05:07:06,524 ERROR [c.c.a.AlertManagerImpl] >>> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Caught exception in >>> recalculating capacity >>> java.lang.NullPointerException >>> 2025-02-14 05:07:06,544 DEBUG [c.c.a.AlertManagerImpl] >>> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) System Alert: Number of >>> unallocated shared network IPs is low in availability zone Gulshan-1 SAM >>> Tower >>> 2025-02-14 05:07:06,544 DEBUG [c.c.a.AlertManagerImpl] >>> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Number of unallocated >>> shared network IPs is low, total: 244.0, allocated: 221.0 (90.57%) >>> 2025-02-14 05:07:06,544 WARN [c.c.a.AlertManagerImpl] >>> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) alertType=[24] >>> dataCenterId=[1] podId=[null] clusterId=[null] message=[System Alert: >>> Number of unallocated shared network IPs is low in availability zone >>> Gulshan-1 SAM Tower]. >>> 2025-02-14 05:07:06,546 DEBUG [c.c.a.AlertManagerImpl] >>> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Have already sent: 1 emails >>> for alert type '24' -- skipping send email >>> 2025-02-14 05:07:06,693 DEBUG [c.c.a.AlertManagerImpl] >>> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Done running Capacity >>> Checker ... >>> 2025-02-14 05:07:07,413 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>> (AgentManager-Handler-13:null) (logid:) Seq 19198-6733725867849049292: >>> MgmtId 34072927583874: Resp: Routing to peer >>> >>> >>> Regards >>> Mosharaf Hossain >>> Manager, Product Development >>> Bangladesh Online (BOL) >>> >>> Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh >>> Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222 >>> 95757 >>> Cell: +880 1787 680828, Web: www.bol-online.com >>> >>> >>> >>> On Thu, Feb 13, 2025 at 2:39 PM Wei ZHOU <ustcweiz...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> Can you search "java.lang.NullPointerException" and paste the full log >>>> of the exception ? >>>> >>>> -Wei >>>> >>>> On Thu, Feb 13, 2025 at 9:37 AM Md Mosharaf Hossain < >>>> mosharaf.hoss...@bol-online.com> wrote: >>>> >>>>> Hi @Wei ZHOU <ustcweiz...@gmail.com> >>>>> The following report might help which I got as an error while >>>>> accessing the host. Please review the log and guide me to resolve. >>>>> >>>>> >>>>> 2025-02-13 14:33:37,395 DEBUG [c.c.a.q.QueryManagerImpl] >>>>> (qtp364604394-2433:ctx-93c55557 ctx-0bef80ad ctx-5ce8a856) >>>>> (logid:89bc4212) >>>>> >>>Searching for hosts>>> >>>>> 2025-02-13 14:33:37,412 DEBUG [c.c.a.q.QueryManagerImpl] >>>>> (qtp364604394-2433:ctx-93c55557 ctx-0bef80ad ctx-5ce8a856) >>>>> (logid:89bc4212) >>>>> >>>Generating Response>>> >>>>> 2025-02-13 14:33:37,871 ERROR [c.c.a.ApiServer] >>>>> (qtp364604394-2433:ctx-93c55557 ctx-0bef80ad ctx-5ce8a856) >>>>> (logid:89bc4212) >>>>> unhandled exception executing api command: [Ljava >>>>> .lang.String;@631ae634 >>>>> >>>>> >>>>> Regards >>>>> Mosharaf Hossain >>>>> Manager, Product Development >>>>> Bangladesh Online (BOL) >>>>> >>>>> Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh >>>>> Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222 >>>>> 95757 >>>>> Cell: +880 1787 680828, Web: www.bol-online.com >>>>> >>>>> >>>>> >>>>> On Thu, Feb 13, 2025 at 11:42 AM Md Mosharaf Hossain < >>>>> mosharaf.hoss...@bol-online.com> wrote: >>>>> >>>>>> Hi Wei >>>>>> Are you wanting to search the keyword "Exception" in the log. There >>>>>> are several entry of "exception" in the log file. >>>>>> What shall I need to do to proceed further? >>>>>> >>>>>> *Some exception log: * >>>>>> 2025-02-13 10:24:38,283 DEBUG [c.c.u.d.GenericDaoBase] >>>>>> (qtp364604394-2335:ctx-daa415e4 ctx-1e82f184) (logid:9d8686b6) >>>>>> ClassCastException when casting value to String. Setting the value of the >>>>>> object directly. >>>>>> 2025-02-13 10:24:38,283 DEBUG [c.c.u.d.GenericDaoBase] >>>>>> (qtp364604394-2335:ctx-daa415e4 ctx-1e82f184) (logid:9d8686b6) >>>>>> ClassCastException when casting value to String. Setting the value of the >>>>>> object directly. >>>>>> java.lang.NullPointerException >>>>>> java.lang.NullPointerException >>>>>> 2025-02-13 10:34:23,422 DEBUG [c.c.u.d.GenericDaoBase] >>>>>> (qtp364604394-2321:ctx-f41289d9 ctx-fc56cd3f) (logid:d6dfb17c) >>>>>> ClassCastException when casting value to String. Setting the value of the >>>>>> object directly. >>>>>> 2025-02-13 10:34:23,422 DEBUG [c.c.u.d.GenericDaoBase] >>>>>> (qtp364604394-2321:ctx-f41289d9 ctx-fc56cd3f) (logid:d6dfb17c) >>>>>> ClassCastException when casting value to String. Setting the value of the >>>>>> object directly. >>>>>> 2025-02-13 10:34:23,422 DEBUG [c.c.u.d.GenericDaoBase] >>>>>> (qtp364604394-2321:ctx-f41289d9 ctx-fc56cd3f) (logid:d6dfb17c) >>>>>> ClassCastException when castin >>>>>> >>>>>> Regards >>>>>> Mosharaf Hossain >>>>>> Manager, Product Development >>>>>> Bangladesh Online (BOL) >>>>>> >>>>>> Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh >>>>>> Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222 >>>>>> 95757 >>>>>> Cell: +880 1787 680828, Web: www.bol-online.com >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Feb 12, 2025 at 4:08 PM Wei ZHOU <ustcweiz...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Your log does not contain any log for listing hosts. >>>>>>> Can you check "Exception" in the log ? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 12, 2025 at 10:01 AM Md Mosharaf Hossain < >>>>>>> mosharaf.hoss...@bol-online.com> wrote: >>>>>>> >>>>>>>> Hi Wei >>>>>>>> >>>>>>>> The requested log is attached for your review. Please check it and >>>>>>>> let me know if you require any further information. >>>>>>>> >>>>>>>> cloudstack-management-server_10 Feb 2025.log >>>>>>>> <https://drive.google.com/open?id=1weLLgB9ib_T4ddjTvmBr8zkb8cyIKpJg> >>>>>>>> >>>>>>>> >>>>>>>> Regards >>>>>>>> Mosharaf Hossain >>>>>>>> Manager, Product Development >>>>>>>> Bangladesh Online (BOL) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Feb 10, 2025 at 2:20 PM Md Mosharaf Hossain < >>>>>>>> mosharaf.hoss...@bol-online.com> wrote: >>>>>>>> >>>>>>>>> Hi >>>>>>>>> I have attached logs from the management server, generated while >>>>>>>>> attempting to access the host list. Please review them. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> Mosharaf Hossain >>>>>>>>> Manager, Product Development >>>>>>>>> Bangladesh Online (BOL) >>>>>>>>> >>>>>>>>> Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, >>>>>>>>> Bangladesh >>>>>>>>> Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 >>>>>>>>> 2222 95757 >>>>>>>>> Cell: +880 1787 680828, Web: www.bol-online.com >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Feb 10, 2025 at 1:50 PM Wei ZHOU <ustcweiz...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> The management server log is located >>>>>>>>>> at /var/log/cloudstack/management/management-server.log >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -Wei >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Feb 10, 2025 at 8:47 AM Md Mosharaf Hossain < >>>>>>>>>> mosharaf.hoss...@bol-online.com> wrote: >>>>>>>>>> >>>>>>>>>> > What type of log you need? >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > Regards >>>>>>>>>> > Mosharaf Hossain >>>>>>>>>> > Manager, Product Development >>>>>>>>>> > Bangladesh Online (BOL) >>>>>>>>>> > >>>>>>>>>> > Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, >>>>>>>>>> Bangladesh >>>>>>>>>> > Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 >>>>>>>>>> 2 2222 95757 >>>>>>>>>> > Cell: +880 1787 680828, Web: www.bol-online.com >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > On Mon, Feb 10, 2025 at 1:27 PM Wei ZHOU <ustcweiz...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> > >>>>>>>>>> > > Hi, >>>>>>>>>> > > >>>>>>>>>> > > Can you share some logs ? >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > -Wei >>>>>>>>>> > > >>>>>>>>>> > > On Sun, Feb 9, 2025 at 12:06 PM Md Mosharaf Hossain < >>>>>>>>>> > > mosharaf.hoss...@bol-online.com> wrote: >>>>>>>>>> > > >>>>>>>>>> > > > Dear Users: >>>>>>>>>> > > > >>>>>>>>>> > > > I hope this message finds you well. I am currently using >>>>>>>>>> CloudStack >>>>>>>>>> > > > version 4.19.1.1 and have four clusters set up in my >>>>>>>>>> environment. >>>>>>>>>> > > Recently, >>>>>>>>>> > > > I have encountered an issue while trying to list hosts in >>>>>>>>>> *Cluster 1*. >>>>>>>>>> > > > >>>>>>>>>> > > > When attempting to list the hosts, I receive a *500 >>>>>>>>>> Internal Server >>>>>>>>>> > > Error*. >>>>>>>>>> > > > This issue is specific to Cluster 1, as the other clusters >>>>>>>>>> are >>>>>>>>>> > > functioning >>>>>>>>>> > > > as expected. I have tried troubleshooting the issue by >>>>>>>>>> checking the >>>>>>>>>> > logs >>>>>>>>>> > > > and ensuring that the hosts in Cluster 1 are properly >>>>>>>>>> connected, but >>>>>>>>>> > the >>>>>>>>>> > > > problem persists. >>>>>>>>>> > > > >>>>>>>>>> > > > Here are some additional details that might help in >>>>>>>>>> diagnosing the >>>>>>>>>> > issue: >>>>>>>>>> > > > >>>>>>>>>> > > > - >>>>>>>>>> > > > >>>>>>>>>> > > > *CloudStack Version*: 4.19.1.1 >>>>>>>>>> > > > - >>>>>>>>>> > > > >>>>>>>>>> > > > *Number of Clusters*: 4 >>>>>>>>>> > > > - >>>>>>>>>> > > > >>>>>>>>>> > > > *Affected Cluster*: Cluster 1 >>>>>>>>>> > > > - >>>>>>>>>> > > > >>>>>>>>>> > > > *Error Message*: 500 Internal Server Error >>>>>>>>>> > > > >>>>>>>>>> > > > Could you please provide guidance on how to resolve this >>>>>>>>>> issue? Any >>>>>>>>>> > > > suggestions or steps to further investigate the problem >>>>>>>>>> would be >>>>>>>>>> > greatly >>>>>>>>>> > > > appreciated. >>>>>>>>>> > > > >>>>>>>>>> > > > Thank you in advance for your support. I look forward to >>>>>>>>>> your response. >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > CloudStack login -> Infrastructure -> Host: SHowing the >>>>>>>>>> below error >>>>>>>>>> > > > [image: image.png] >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > Regards >>>>>>>>>> > > > Mosharaf Hossain >>>>>>>>>> > > > Manager, Product Development >>>>>>>>>> > > > Bangladesh Online (BOL) >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>
