Hi @Wei ZHOU <ustcweiz...@gmail.com> We have removed all information from MySQL database for the host cshost25.bol-online.com" but still unable to view the host option. Kindly advise your expert opinion.
Regards Mosharaf Hossain Manager, Product Development Bangladesh Online (BOL) Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222 95757 Cell: +880 1787 680828, Web: www.bol-online.com On Sun, Feb 16, 2025 at 3:36 PM Md Mosharaf Hossain < mosharaf.hoss...@bol-online.com> wrote: > Hi @Wei ZHOU <ustcweiz...@gmail.com> > Today, we attempted to remove the disconnected host using *Cloud Monkey > tools*. As the issue began after removing the host *"cshost25.bol-online.com > <http://cshost25.bol-online.com>"*, and we discovered *two records* in > the database's host table. One record was deleted successfully without any > errors, but we encountered the following error while trying to remove the > other record. > > *Cloud Monkey Outputs : * > ---------------------------------- > (bol-cloud-admin) 🐱 > list hosts > 🙈 Error: (HTTP 530, error code 9999) <nil> > (bol-cloud-admin) 🐱 > delete host id=19714 forced=true > { > "success": true > } > (bol-cloud-admin) 🐱 > delete host id=19713 forced=true > 🙈 Error: (HTTP 530, error code 9999) <nil> > (bol-cloud-admin) 🐱 > > > _----------------------------------------------------------------------------------- > > DB query after deletion of Host ID : 19714 > > mysql> SELECT h.id AS host_id, h.name AS host_name, h.status, h.type, > h.private_ip_address, h.cluster_id, c.name AS cluster_name FROM host h > JOIN cluster c ON h.cluster | NU > _id = c.id; > +---------+-------------------------+--------------+-------- > -+--------------------+------------+-------------------------+ 0 > | host_id | host_name | status | type | > private_ip_address | cluster_id | cluster_name | > +---------+-------------------------+--------------+-------- > -+--------------------+------------+-------------------------+.10.1 > | 19198 | cshost3.bol-online.com | Up | Routing | > 10.10.11.33 | 1 | cluster1.bol-online.com | | NU > | 19439 | cshost5.bol-online.com | Up | Routing | > 10.10.11.35 | 1 | cluster1.bol-online.com | > | 19667 | cshost8.bol-online.com | Up | Routing | > 10.10.11.38 | 1 | cluster1.bol-online.com | 0 > | 19668 | cshost17.bol-online.com | Up | Routing | > 10.10.11.47 | 5 | cluster2.bol-online.com | > | 19670 | cph2n8.bol-online.com | Disconnected | Routing | > 10.10.10.118 | 1 | cluster1.bol-online.com |.10.1 > | 19675 | cshost18.bol-online.com | Up | Routing | > 10.10.11.48 | 5 | cluster2.bol-online.com | | NU > | 19676 | cshost19.bol-online.com | Up | Routing | > 10.10.11.49 | 5 | cluster2.bol-online.com | > | 19685 | cshost10.bol-online.com | Up | Routing | > 10.10.11.41 | 1 | cluster1.bol-online.com | 0 > | 19686 | cshost21.bol-online.com | Up | Routing | > 10.10.11.51 | 5 | cluster2.bol-online.com | > | 19687 | cshost11.bol-online.com | Up | Routing | > 10.10.11.42 | 1 | cluster1.bol-online.com |.10.1 > | 19691 | cshost7.bol-online.com | Up | Routing | > 10.10.11.37 | 1 | cluster1.bol-online.com | | NU > | 19696 | cshost6.bol-online.com | Up | Routing | > 10.10.11.36 | 1 | cluster1.bol-online.com | > | 19701 | cshost22.bol-online.com | Up | Routing | > 10.10.11.52 | 6 | cluster3.bol-online.com | 0 > | 19705 | cshost24.bol-online.com | Up | Routing | > 10.10.11.54 | 1 | cluster1.bol-online.com | > | 19706 | cshost23.bol-online.com | Up | Routing | > 10.10.11.53 | 1 | cluster1.bol-online.com |.10.1 > | 19707 | cshost2.bol-online.com | Up | Routing | > 10.10.11.32 | 1 | cluster1.bol-online.com | | NU > | 19709 | cshost4.bol-online.com | Up | Routing | > 10.10.11.34 | 1 | cluster1.bol-online.com | > | 19710 | cshost1.bol-online.com | Up | Routing | > 10.10.11.31 | 1 | cluster1.bol-online.com | 0 > | 19712 | cshost9.bol-online.com | Up | Routing | > 10.10.11.39 | 1 | cluster1.bol-online.com | > | 19713 | cshost25.bol-online.com | Disconnected | Routing | > 10.10.11.55 | 1 | cluster1.bol-online.com |----- > | 19715 | cshost26.bol-online.com | Up | Routing | > 10.10.11.56 | 10 | cluster4.bol-online.com |-+--- > | 19716 | cshost27.bol-online.com | Up | Routing | > 10.10.11.57 | 12 | cluster5.bol-online.com |----- > | 19717 | cshost28.bol-online.com | Up | Routing | > 10.10.11.58 | 12 | cluster5.bol-online.com |----- > | 19719 | cshost29.bol-online.com | Up | Routing | > 10.10.11.59 | 12 | cluster5.bol-online.com | > +---------+-------------------------+--------------+-------- > -+--------------------+------------+-------------------------+ > 24 rows in set (0.00 sec) > _________________________________________________ > > *Additionally*, we enabled the debug log for *CloudMonkey*, which > displayed the following error while listing the hosts. > > cloud-admin) 🐱 > list hosts id=19713 > [debug] ExecLine line:list hosts id=19713 > [debug] ExecCmd args: list, hosts, id=19713 > [debug] NewAPIRequest API request URL: > http://10.10.11.61:8080/client/api?apiKey=_A1conzDjN-7JrbMIeVyuEPU5Gw-d-yKYXj2gn9n7c8X85wdn0j_04pQZpZtI1fyv8QKDoFma0CF7gdMh5FaBA&command=listHosts&id=19713&response=json&signature=5RD%2FzDvE3w%2FQrdSkgpiA%2BFQNHH4%3D > [debug] NewAPIRequest response status code:530 > [debug] NewAPIRequest response > body:{"listhostsresponse":{"uuidList":[],"errorcode":530,"cserrorcode":9999}} > 🙈 Error: (HTTP 530, error code 9999) <nil> > (bol-cloud-admin) 🐱 > > > > > > Regards > Mosharaf Hossain > > > > On Fri, Feb 14, 2025 at 9:41 AM Md Mosharaf Hossain < > mosharaf.hoss...@bol-online.com> wrote: > >> Hi >> I find the following items >> >> 2025-02-14 05:07:06,522 DEBUG [c.c.s.StorageManagerImpl] >> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Total over provisioned >> capacity of the pool Eskaton-hdd-rbd-cls1 id: 100 is (83.3988 TB) >> 91697987977216 >> 2025-02-14 05:07:06,524 ERROR [c.c.a.AlertManagerImpl] >> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Caught exception in >> recalculating capacity >> java.lang.NullPointerException >> 2025-02-14 05:07:06,544 DEBUG [c.c.a.AlertManagerImpl] >> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) System Alert: Number of >> unallocated shared network IPs is low in availability zone Gulshan-1 SAM >> Tower >> 2025-02-14 05:07:06,544 DEBUG [c.c.a.AlertManagerImpl] >> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Number of unallocated >> shared network IPs is low, total: 244.0, allocated: 221.0 (90.57%) >> 2025-02-14 05:07:06,544 WARN [c.c.a.AlertManagerImpl] >> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) alertType=[24] >> dataCenterId=[1] podId=[null] clusterId=[null] message=[System Alert: >> Number of unallocated shared network IPs is low in availability zone >> Gulshan-1 SAM Tower]. >> 2025-02-14 05:07:06,546 DEBUG [c.c.a.AlertManagerImpl] >> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Have already sent: 1 emails >> for alert type '24' -- skipping send email >> 2025-02-14 05:07:06,693 DEBUG [c.c.a.AlertManagerImpl] >> (CapacityChecker:ctx-86e3d26e) (logid:0cc0f918) Done running Capacity >> Checker ... >> 2025-02-14 05:07:07,413 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> (AgentManager-Handler-13:null) (logid:) Seq 19198-6733725867849049292: >> MgmtId 34072927583874: Resp: Routing to peer >> >> >> Regards >> Mosharaf Hossain >> Manager, Product Development >> Bangladesh Online (BOL) >> >> Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh >> Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222 >> 95757 >> Cell: +880 1787 680828, Web: www.bol-online.com >> >> >> >> On Thu, Feb 13, 2025 at 2:39 PM Wei ZHOU <ustcweiz...@gmail.com> wrote: >> >>> Hi, >>> >>> Can you search "java.lang.NullPointerException" and paste the full log >>> of the exception ? >>> >>> -Wei >>> >>> On Thu, Feb 13, 2025 at 9:37 AM Md Mosharaf Hossain < >>> mosharaf.hoss...@bol-online.com> wrote: >>> >>>> Hi @Wei ZHOU <ustcweiz...@gmail.com> >>>> The following report might help which I got as an error while accessing >>>> the host. Please review the log and guide me to resolve. >>>> >>>> >>>> 2025-02-13 14:33:37,395 DEBUG [c.c.a.q.QueryManagerImpl] >>>> (qtp364604394-2433:ctx-93c55557 ctx-0bef80ad ctx-5ce8a856) (logid:89bc4212) >>>> >>>Searching for hosts>>> >>>> 2025-02-13 14:33:37,412 DEBUG [c.c.a.q.QueryManagerImpl] >>>> (qtp364604394-2433:ctx-93c55557 ctx-0bef80ad ctx-5ce8a856) (logid:89bc4212) >>>> >>>Generating Response>>> >>>> 2025-02-13 14:33:37,871 ERROR [c.c.a.ApiServer] >>>> (qtp364604394-2433:ctx-93c55557 ctx-0bef80ad ctx-5ce8a856) (logid:89bc4212) >>>> unhandled exception executing api command: [Ljava >>>> .lang.String;@631ae634 >>>> >>>> >>>> Regards >>>> Mosharaf Hossain >>>> Manager, Product Development >>>> Bangladesh Online (BOL) >>>> >>>> Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh >>>> Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222 >>>> 95757 >>>> Cell: +880 1787 680828, Web: www.bol-online.com >>>> >>>> >>>> >>>> On Thu, Feb 13, 2025 at 11:42 AM Md Mosharaf Hossain < >>>> mosharaf.hoss...@bol-online.com> wrote: >>>> >>>>> Hi Wei >>>>> Are you wanting to search the keyword "Exception" in the log. There >>>>> are several entry of "exception" in the log file. >>>>> What shall I need to do to proceed further? >>>>> >>>>> *Some exception log: * >>>>> 2025-02-13 10:24:38,283 DEBUG [c.c.u.d.GenericDaoBase] >>>>> (qtp364604394-2335:ctx-daa415e4 ctx-1e82f184) (logid:9d8686b6) >>>>> ClassCastException when casting value to String. Setting the value of the >>>>> object directly. >>>>> 2025-02-13 10:24:38,283 DEBUG [c.c.u.d.GenericDaoBase] >>>>> (qtp364604394-2335:ctx-daa415e4 ctx-1e82f184) (logid:9d8686b6) >>>>> ClassCastException when casting value to String. Setting the value of the >>>>> object directly. >>>>> java.lang.NullPointerException >>>>> java.lang.NullPointerException >>>>> 2025-02-13 10:34:23,422 DEBUG [c.c.u.d.GenericDaoBase] >>>>> (qtp364604394-2321:ctx-f41289d9 ctx-fc56cd3f) (logid:d6dfb17c) >>>>> ClassCastException when casting value to String. Setting the value of the >>>>> object directly. >>>>> 2025-02-13 10:34:23,422 DEBUG [c.c.u.d.GenericDaoBase] >>>>> (qtp364604394-2321:ctx-f41289d9 ctx-fc56cd3f) (logid:d6dfb17c) >>>>> ClassCastException when casting value to String. Setting the value of the >>>>> object directly. >>>>> 2025-02-13 10:34:23,422 DEBUG [c.c.u.d.GenericDaoBase] >>>>> (qtp364604394-2321:ctx-f41289d9 ctx-fc56cd3f) (logid:d6dfb17c) >>>>> ClassCastException when castin >>>>> >>>>> Regards >>>>> Mosharaf Hossain >>>>> Manager, Product Development >>>>> Bangladesh Online (BOL) >>>>> >>>>> Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh >>>>> Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222 >>>>> 95757 >>>>> Cell: +880 1787 680828, Web: www.bol-online.com >>>>> >>>>> >>>>> >>>>> On Wed, Feb 12, 2025 at 4:08 PM Wei ZHOU <ustcweiz...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Your log does not contain any log for listing hosts. >>>>>> Can you check "Exception" in the log ? >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Feb 12, 2025 at 10:01 AM Md Mosharaf Hossain < >>>>>> mosharaf.hoss...@bol-online.com> wrote: >>>>>> >>>>>>> Hi Wei >>>>>>> >>>>>>> The requested log is attached for your review. Please check it and >>>>>>> let me know if you require any further information. >>>>>>> >>>>>>> cloudstack-management-server_10 Feb 2025.log >>>>>>> <https://drive.google.com/open?id=1weLLgB9ib_T4ddjTvmBr8zkb8cyIKpJg> >>>>>>> >>>>>>> >>>>>>> Regards >>>>>>> Mosharaf Hossain >>>>>>> Manager, Product Development >>>>>>> Bangladesh Online (BOL) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 10, 2025 at 2:20 PM Md Mosharaf Hossain < >>>>>>> mosharaf.hoss...@bol-online.com> wrote: >>>>>>> >>>>>>>> Hi >>>>>>>> I have attached logs from the management server, generated while >>>>>>>> attempting to access the host list. Please review them. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Regards >>>>>>>> Mosharaf Hossain >>>>>>>> Manager, Product Development >>>>>>>> Bangladesh Online (BOL) >>>>>>>> >>>>>>>> Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, >>>>>>>> Bangladesh >>>>>>>> Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 >>>>>>>> 2222 95757 >>>>>>>> Cell: +880 1787 680828, Web: www.bol-online.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Feb 10, 2025 at 1:50 PM Wei ZHOU <ustcweiz...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> The management server log is located >>>>>>>>> at /var/log/cloudstack/management/management-server.log >>>>>>>>> >>>>>>>>> >>>>>>>>> -Wei >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Feb 10, 2025 at 8:47 AM Md Mosharaf Hossain < >>>>>>>>> mosharaf.hoss...@bol-online.com> wrote: >>>>>>>>> >>>>>>>>> > What type of log you need? >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Regards >>>>>>>>> > Mosharaf Hossain >>>>>>>>> > Manager, Product Development >>>>>>>>> > Bangladesh Online (BOL) >>>>>>>>> > >>>>>>>>> > Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, >>>>>>>>> Bangladesh >>>>>>>>> > Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 >>>>>>>>> 2222 95757 >>>>>>>>> > Cell: +880 1787 680828, Web: www.bol-online.com >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > On Mon, Feb 10, 2025 at 1:27 PM Wei ZHOU <ustcweiz...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> > >>>>>>>>> > > Hi, >>>>>>>>> > > >>>>>>>>> > > Can you share some logs ? >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > -Wei >>>>>>>>> > > >>>>>>>>> > > On Sun, Feb 9, 2025 at 12:06 PM Md Mosharaf Hossain < >>>>>>>>> > > mosharaf.hoss...@bol-online.com> wrote: >>>>>>>>> > > >>>>>>>>> > > > Dear Users: >>>>>>>>> > > > >>>>>>>>> > > > I hope this message finds you well. I am currently using >>>>>>>>> CloudStack >>>>>>>>> > > > version 4.19.1.1 and have four clusters set up in my >>>>>>>>> environment. >>>>>>>>> > > Recently, >>>>>>>>> > > > I have encountered an issue while trying to list hosts in >>>>>>>>> *Cluster 1*. >>>>>>>>> > > > >>>>>>>>> > > > When attempting to list the hosts, I receive a *500 Internal >>>>>>>>> Server >>>>>>>>> > > Error*. >>>>>>>>> > > > This issue is specific to Cluster 1, as the other clusters >>>>>>>>> are >>>>>>>>> > > functioning >>>>>>>>> > > > as expected. I have tried troubleshooting the issue by >>>>>>>>> checking the >>>>>>>>> > logs >>>>>>>>> > > > and ensuring that the hosts in Cluster 1 are properly >>>>>>>>> connected, but >>>>>>>>> > the >>>>>>>>> > > > problem persists. >>>>>>>>> > > > >>>>>>>>> > > > Here are some additional details that might help in >>>>>>>>> diagnosing the >>>>>>>>> > issue: >>>>>>>>> > > > >>>>>>>>> > > > - >>>>>>>>> > > > >>>>>>>>> > > > *CloudStack Version*: 4.19.1.1 >>>>>>>>> > > > - >>>>>>>>> > > > >>>>>>>>> > > > *Number of Clusters*: 4 >>>>>>>>> > > > - >>>>>>>>> > > > >>>>>>>>> > > > *Affected Cluster*: Cluster 1 >>>>>>>>> > > > - >>>>>>>>> > > > >>>>>>>>> > > > *Error Message*: 500 Internal Server Error >>>>>>>>> > > > >>>>>>>>> > > > Could you please provide guidance on how to resolve this >>>>>>>>> issue? Any >>>>>>>>> > > > suggestions or steps to further investigate the problem >>>>>>>>> would be >>>>>>>>> > greatly >>>>>>>>> > > > appreciated. >>>>>>>>> > > > >>>>>>>>> > > > Thank you in advance for your support. I look forward to >>>>>>>>> your response. >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > CloudStack login -> Infrastructure -> Host: SHowing the >>>>>>>>> below error >>>>>>>>> > > > [image: image.png] >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > Regards >>>>>>>>> > > > Mosharaf Hossain >>>>>>>>> > > > Manager, Product Development >>>>>>>>> > > > Bangladesh Online (BOL) >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>> >>>>>>>>
