Re: Load Balancers and how?
I was able to have the group that configured the load balancer determine if the Remedy Service was started/running. -Original Message- From: LJ LongWing To: arslist Sent: Fri, Dec 10, 2010 10:51 am Subject: Load Balancers and how? ** Ok….I’m going down a road that I KNOW some of you have travelled before, I need your travel tips. We are setting up a server group on 7.5 Windows and we are putting a load balancer in front of it. How do you all tell if a node is up? There are 4 types of probes that I’m being told are available… PING HTTP TCP Script Ping - Obviously doesn’t tell you anything other than the host OS is up HTTP - Won’t work for telling if a remedy service is up TCP – My instincts are telling me that this one simply does something similar to a telnet to a given port, if the port allows you ‘in’, then the service is considered up. I don’t consider this one viable because it doesn’t ‘test’ much and I expect that my remedy server could respond to the port request without actually being functional Script – This one is the most promising in my mind, but I know nothing about the scripting language of TCL. How does the rest of the remedy world ‘verify’ that their remedy instance is online so that their load balancer can know to route traffic to it or not. I just this week finished a java servlet that allows the mid tier load balancer to know if an individual node is working properly, but the same can’t be said yet for Remedy….any and all help is appreciated _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
Re: Load Balancers and how?
Roger, What method did they use to determine it was started/running? I personally have found ‘other teams’ don’t usually have a concept of what ‘functional’ means in a remedy world. I don’t consider the fact that arserver.exe is in memory an indicator that my server is functional…I’ve had TOO many situations where it was in memory, but you couldn’t log onto it. From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Roger Justice Sent: Friday, December 10, 2010 9:00 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** I was able to have the group that configured the load balancer determine if the Remedy Service was started/running. -Original Message- From: LJ LongWing To: arslist Sent: Fri, Dec 10, 2010 10:51 am Subject: Load Balancers and how? ** Ok….I’m going down a road that I KNOW some of you have travelled before, I need your travel tips. We are setting up a server group on 7.5 Windows and we are putting a load balancer in front of it. How do you all tell if a node is up? There are 4 types of probes that I’m being told are available… PING HTTP TCP Script Ping - Obviously doesn’t tell you anything other than the host OS is up HTTP - Won’t work for telling if a remedy service is up TCP – My instincts are telling me that this one simply does something similar to a telnet to a given port, if the port allows you ‘in’, then the service is considered up. I don’t consider this one viable because it doesn’t ‘test’ much and I expect that my remedy server could respond to the port request without actually being functional Script – This one is the most promising in my mind, but I know nothing about the scripting language of TCL. How does the rest of the remedy world ‘verify’ that their remedy instance is online so that their load balancer can know to route traffic to it or not. I just this week finished a java servlet that allows the mid tier load balancer to know if an individual node is working properly, but the same can’t be said yet for Remedy….any and all help is appreciated _attend WWRUG11 <http://www.wwrug.com/> www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
Re: Load Balancers and how?
I provided the best information and then let the customer determine how they can monitor the system with the LB. I agree that both Mid-Tier and App Servers have issues with finding if the system is truly available. I know that the files arerror and armonitor can assist with this. -Original Message- From: LJ LongWing To: arslist Sent: Fri, Dec 10, 2010 11:12 am Subject: Re: Load Balancers and how? ** Roger, What method did they use to determine it was started/running? I personally have found ‘other teams’ don’t usually have a concept of what ‘functional’ means in a remedy world. I don’t consider the fact that arserver.exe is in memory an indicator that my server is functional…I’ve had TOO many situations where it was in memory, but you couldn’t log onto it. From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Roger Justice Sent: Friday, December 10, 2010 9:00 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** I was able to have the group that configured the load balancer determine if the Remedy Service was started/running. -Original Message- From: LJ LongWing To: arslist Sent: Fri, Dec 10, 2010 10:51 am Subject: Load Balancers and how? ** Ok….I’m going down a road that I KNOW some of you have travelled before, I need your travel tips. We are setting up a server group on 7.5 Windows and we are putting a load balancer in front of it. How do you all tell if a node is up? There are 4 types of probes that I’m being told are available… PING HTTP TCP Script Ping - Obviously doesn’t tell you anything other than the host OS is up HTTP - Won’t work for telling if a remedy service is up TCP – My instincts are telling me that this one simply does something similar to a telnet to a given port, if the port allows you ‘in’, then the service is considered up. I don’t consider this one viable because it doesn’t ‘test’ much and I expect that my remedy server could respond to the port request without actually being functional Script – This one is the most promising in my mind, but I know nothing about the scripting language of TCL. How does the rest of the remedy world ‘verify’ that their remedy instance is online so that their load balancer can know to route traffic to it or not. I just this week finished a java servlet that allows the mid tier load balancer to know if an individual node is working properly, but the same can’t be said yet for Remedy….any and all help is appreciated _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
Re: Load Balancers and how?
Hi, PING - Most firewalls don't allow ICMP traffic. HTTP - You can use this, with a formed url including a user name and pass, to use on the load balancer in front of the midtiers not the AR Servers. TCP - I have found that even though you can connect on the TCP port, it doesn't always mean its responsive e.g. ARERR 91 to 93 Reason being is that you don't connect to arserver.exe, you connect to arforkd Script - I don't know TCL but if you can run an executable, then you could have driver which is always installed in the AR Server dir and you could feed in a driver script with -x. Regards Danny From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of LJ LongWing Sent: 10 December 2010 15:50 To: arslist@ARSLIST.ORG Subject: Load Balancers and how? ** Ok..I'm going down a road that I KNOW some of you have travelled before, I need your travel tips. We are setting up a server group on 7.5 Windows and we are putting a load balancer in front of it. How do you all tell if a node is up? There are 4 types of probes that I'm being told are available. PING HTTP TCP Script Ping - Obviously doesn't tell you anything other than the host OS is up HTTP - Won't work for telling if a remedy service is up TCP - My instincts are telling me that this one simply does something similar to a telnet to a given port, if the port allows you 'in', then the service is considered up. I don't consider this one viable because it doesn't 'test' much and I expect that my remedy server could respond to the port request without actually being functional Script - This one is the most promising in my mind, but I know nothing about the scripting language of TCL. How does the rest of the remedy world 'verify' that their remedy instance is online so that their load balancer can know to route traffic to it or not. I just this week finished a java servlet that allows the mid tier load balancer to know if an individual node is working properly, but the same can't be said yet for Remedy..any and all help is appreciated _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
Re: Load Balancers and how?
** The LB itself should be able to determine that if it is configured correctly. In particular on Apache you would use mod_proxy and mod_proxy_balancer modules simultaneously with some ProxyPass directives defined. Any of your web server admins should have a very good idea of what I am talking about. If I were not typing this from my obnoxious touchscreen I would go into more detail. In any case if you need more precise help reply stating so and I will be happy to help on e I get to my desk.Cheerios!Sent from my Windows Phone From: LJ LongWingSent: Friday, December 10, 2010 10:49 AMTo: arslist@ARSLIST.ORGSubject: Load Balancers and how? > Ok..I'm going down a road that I KNOW some of you have travelled before, I > need your travel tips. We are setting up a server group on 7.5 Windows and > we are putting a load balancer in front of it. How do you all tell if a > node is up? There are 4 types of probes that I'm being told are available. > > > > PING > > HTTP > > TCP > > Script > > > > Ping - Obviously doesn't tell you anything other than the host OS is up > > HTTP - Won't work for telling if a remedy service is up > > TCP - My instincts are telling me that this one simply does something > similar to a telnet to a given port, if the port allows you 'in', then the > service is considered up. I don't consider this one viable because it > doesn't 'test' much and I expect that my remedy server could respond to the > port request without actually being functional > > Script - This one is the most promising in my mind, but I know nothing about > the scripting language of TCL. > > > > How does the rest of the remedy world 'verify' that their remedy instance is > online so that their load balancer can know to route traffic to it or not. > I just this week finished a java servlet that allows the mid tier load > balancer to know if an individual node is working properly, but the same > can't be said yet for Remedy..any and all help is appreciated > > > ___ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_
Re: Load Balancers and how?
LJ, Here are a few approaches that can be used. A product called SiteScope can be configured to access a form through the MidTier. This gives you end to end up/down information. We also use HP OpenView to monitor the services on the MidTier, app and Database servers. We also monitor the arerror, stderr.log and stdout log files on the app server, the Oracle logs on the DB server and the MidTier log files on the MidTier servers. I have also created a health check for email services. This simply sends an email out of the system to one of two email addresses we use for incoming ticket generation. The email creates a record in a receiving form that when the record is created updates the record on the sending form. If the record on the sending form does not generate within a specific time period an escalation will send us email and page us. Dave From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of LJ LongWing Sent: Friday, December 10, 2010 11:12 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** Roger, What method did they use to determine it was started/running? I personally have found 'other teams' don't usually have a concept of what 'functional' means in a remedy world. I don't consider the fact that arserver.exe is in memory an indicator that my server is functional...I've had TOO many situations where it was in memory, but you couldn't log onto it. From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Roger Justice Sent: Friday, December 10, 2010 9:00 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** I was able to have the group that configured the load balancer determine if the Remedy Service was started/running. -Original Message- From: LJ LongWing To: arslist Sent: Fri, Dec 10, 2010 10:51 am Subject: Load Balancers and how? ** OkI'm going down a road that I KNOW some of you have travelled before, I need your travel tips. We are setting up a server group on 7.5 Windows and we are putting a load balancer in front of it. How do you all tell if a node is up? There are 4 types of probes that I'm being told are available... PING HTTP TCP Script Ping - Obviously doesn't tell you anything other than the host OS is up HTTP - Won't work for telling if a remedy service is up TCP - My instincts are telling me that this one simply does something similar to a telnet to a given port, if the port allows you 'in', then the service is considered up. I don't consider this one viable because it doesn't 'test' much and I expect that my remedy server could respond to the port request without actually being functional Script - This one is the most promising in my mind, but I know nothing about the scripting language of TCL. How does the rest of the remedy world 'verify' that their remedy instance is online so that their load balancer can know to route traffic to it or not. I just this week finished a java servlet that allows the mid tier load balancer to know if an individual node is working properly, but the same can't be said yet for Remedyany and all help is appreciated _attend WWRUG11 www.wwrug.com<http://www.wwrug.com/> ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
Re: Load Balancers and how?
You will want to handle traffic for both rpcbind and arserver, and potentially your plugin servers, DSO, on the LB, depending on your implementation. The easiest check is just a port check. I think most LB devices provide this capability. As you recently saw with your Mid-Tier/Tomcat problems, a simple port check does not always fault the node if there is a problem. What kind of device are you using to balance the traffic? If you are using an F5 LTM, iRules can be written to do more extensive checks. I was part of a thread on the subject here that can provide some additional information: http://devcentral.f5.com/Forums/tabid/1082223/asg/52/showtab/groupforums/afv/topic/aff/31/aft/19000/Default.aspx On Fri, Dec 10, 2010 at 9:49 AM, LJ LongWing wrote: > ** > > Ok….I’m going down a road that I KNOW some of you have travelled before, I > need your travel tips. We are setting up a server group on 7.5 Windows and > we are putting a load balancer in front of it. How do you all tell if a > node is up? There are 4 types of probes that I’m being told are available… > > > > PING > > HTTP > > TCP > > Script > > > > Ping - Obviously doesn’t tell you anything other than the host OS is up > > HTTP - Won’t work for telling if a remedy service is up > > TCP – My instincts are telling me that this one simply does something > similar to a telnet to a given port, if the port allows you ‘in’, then the > service is considered up. I don’t consider this one viable because it > doesn’t ‘test’ much and I expect that my remedy server could respond to the > port request without actually being functional > > Script – This one is the most promising in my mind, but I know nothing > about the scripting language of TCL. > > > > How does the rest of the remedy world ‘verify’ that their remedy instance > is online so that their load balancer can know to route traffic to it or > not. I just this week finished a java servlet that allows the mid tier load > balancer to know if an individual node is working properly, but the same > can’t be said yet for Remedy….any and all help is appreciated > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
Re: Load Balancers and how?
The problem is that load balancers are only as smart as their implementation. We have been running for years with our load balancer checking a static html file on the tomcat server….while this provides a check of ‘is the tomcat server working’, it does not provide a check of ‘is the mid-tier application functioning’….so I respectfully disagree that the lb or the lb team know how to configure it properly. We are using Cisco’s CSM (Content Switching Module), and I don’t want to just rely on a port probe because I don’t trust that it means I’m functional. From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Michael Latham Sent: Friday, December 10, 2010 9:29 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** The LB itself should be able to determine that if it is configured correctly. In particular on Apache you would use mod_proxy and mod_proxy_balancer modules simultaneously with some ProxyPass directives defined. Any of your web server admins should have a very good idea of what I am talking about. If I were not typing this from my obnoxious touchscreen I would go into more detail. In any case if you need more precise help reply stating so and I will be happy to help on e I get to my desk. Cheerios! Sent from my Windows Phone _ From: LJ LongWing Sent: Friday, December 10, 2010 10:49 AM To: arslist@ARSLIST.ORG Subject: Load Balancers and how? > Ok..I'm going down a road that I KNOW some of you have travelled before, I > need your travel tips. We are setting up a server group on 7.5 Windows and > we are putting a load balancer in front of it. How do you all tell if a > node is up? There are 4 types of probes that I'm being told are available. > > > > PING > > HTTP > > TCP > > Script > > > > Ping - Obviously doesn't tell you anything other than the host OS is up > > HTTP - Won't work for telling if a remedy service is up > > TCP - My instincts are telling me that this one simply does something > similar to a telnet to a given port, if the port allows you 'in', then the > service is considered up. I don't consider this one viable because it > doesn't 'test' much and I expect that my remedy server could respond to the > port request without actually being functional > > Script - This one is the most promising in my mind, but I know nothing about > the scripting language of TCL. > > > > How does the rest of the remedy world 'verify' that their remedy instance is > online so that their load balancer can know to route traffic to it or not. > I just this week finished a java servlet that allows the mid tier load > balancer to know if an individual node is working properly, but the same > can't be said yet for Remedy..any and all help is appreciated > > > ___ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
Re: Load Balancers and how?
Dave, I also have similar monitoring on the app/db/web servers.but that doesn't solve the problem of having two app servers with a load balancer sitting in front of them, and ensuring that if node 1 goes offline, that the load balancer will stop sending traffic to it.that's what I'm needing. From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Shellman, David Sent: Friday, December 10, 2010 9:37 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** LJ, Here are a few approaches that can be used. A product called SiteScope can be configured to access a form through the MidTier. This gives you end to end up/down information. We also use HP OpenView to monitor the services on the MidTier, app and Database servers. We also monitor the arerror, stderr.log and stdout log files on the app server, the Oracle logs on the DB server and the MidTier log files on the MidTier servers. I have also created a health check for email services. This simply sends an email out of the system to one of two email addresses we use for incoming ticket generation. The email creates a record in a receiving form that when the record is created updates the record on the sending form. If the record on the sending form does not generate within a specific time period an escalation will send us email and page us. Dave _ From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of LJ LongWing Sent: Friday, December 10, 2010 11:12 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** Roger, What method did they use to determine it was started/running? I personally have found 'other teams' don't usually have a concept of what 'functional' means in a remedy world. I don't consider the fact that arserver.exe is in memory an indicator that my server is functional.I've had TOO many situations where it was in memory, but you couldn't log onto it. From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Roger Justice Sent: Friday, December 10, 2010 9:00 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** I was able to have the group that configured the load balancer determine if the Remedy Service was started/running. -Original Message- From: LJ LongWing To: arslist Sent: Fri, Dec 10, 2010 10:51 am Subject: Load Balancers and how? ** Ok..I'm going down a road that I KNOW some of you have travelled before, I need your travel tips. We are setting up a server group on 7.5 Windows and we are putting a load balancer in front of it. How do you all tell if a node is up? There are 4 types of probes that I'm being told are available. PING HTTP TCP Script Ping - Obviously doesn't tell you anything other than the host OS is up HTTP - Won't work for telling if a remedy service is up TCP - My instincts are telling me that this one simply does something similar to a telnet to a given port, if the port allows you 'in', then the service is considered up. I don't consider this one viable because it doesn't 'test' much and I expect that my remedy server could respond to the port request without actually being functional Script - This one is the most promising in my mind, but I know nothing about the scripting language of TCL. How does the rest of the remedy world 'verify' that their remedy instance is online so that their load balancer can know to route traffic to it or not. I just this week finished a java servlet that allows the mid tier load balancer to know if an individual node is working properly, but the same can't be said yet for Remedy..any and all help is appreciated _attend WWRUG11 <http://www.wwrug.com/> www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
Re: Load Balancers and how?
Axton, We are using Cisco's CSM as a load balancer. I'm surprised in this day and age, with remedy being around as long as it has been that we are having this discussion, but reading a post you made in that thread 'Another option I am considering is writing a program to listen on a port separate that of the application that can handle a request that contains a string. It would simply return a 0 or a 1 if the server is up or down.' I was just considering an option along those lines. To save me some time and maybe help the community, do you by chance have that daemon written? (you did write that post over a year ago) If not, I may need to pull out my eclipse (YET AGAIN) to solve load balancing issues in my environment. From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Axton Sent: Friday, December 10, 2010 9:54 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** You will want to handle traffic for both rpcbind and arserver, and potentially your plugin servers, DSO, on the LB, depending on your implementation. The easiest check is just a port check. I think most LB devices provide this capability. As you recently saw with your Mid-Tier/Tomcat problems, a simple port check does not always fault the node if there is a problem. What kind of device are you using to balance the traffic? If you are using an F5 LTM, iRules can be written to do more extensive checks. I was part of a thread on the subject here that can provide some additional information: http://devcentral.f5.com/Forums/tabid/1082223/asg/52/showtab/groupforums/afv /topic/aff/31/aft/19000/Default.aspx On Fri, Dec 10, 2010 at 9:49 AM, LJ LongWing wrote: ** Ok..I'm going down a road that I KNOW some of you have travelled before, I need your travel tips. We are setting up a server group on 7.5 Windows and we are putting a load balancer in front of it. How do you all tell if a node is up? There are 4 types of probes that I'm being told are available. PING HTTP TCP Script Ping - Obviously doesn't tell you anything other than the host OS is up HTTP - Won't work for telling if a remedy service is up TCP - My instincts are telling me that this one simply does something similar to a telnet to a given port, if the port allows you 'in', then the service is considered up. I don't consider this one viable because it doesn't 'test' much and I expect that my remedy server could respond to the port request without actually being functional Script - This one is the most promising in my mind, but I know nothing about the scripting language of TCL. How does the rest of the remedy world 'verify' that their remedy instance is online so that their load balancer can know to route traffic to it or not. I just this week finished a java servlet that allows the mid tier load balancer to know if an individual node is working properly, but the same can't be said yet for Remedy..any and all help is appreciated _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
Re: Load Balancers and how?
What I was working on at the time is available here: http://communities.bmc.com/communities/docs/DOC-9059 <http://communities.bmc.com/communities/docs/DOC-9059>This is a command line utility that will checks the availability of the server by issuing an ARVerifyUser call to the server. I was planning to take this and wrap it in something to make it available over the network, like a cgi, or maybe if there is a good tiny http server, embedding that into the project. On Fri, Dec 10, 2010 at 12:25 PM, LJ LongWing wrote: > ** > > Axton, > > We are using Cisco’s CSM as a load balancer. I’m surprised in this day and > age, with remedy being around as long as it has been that we are having this > discussion, but reading a post you made in that thread > > > > ‘Another option I am considering is writing a program to listen on a port > separate that of the application that can handle a request that contains a > string. It would simply return a 0 or a 1 if the server is up or down.’ > > > > I was just considering an option along those lines. To save me some time > and maybe help the community, do you by chance have that daemon written? > (you did write that post over a year ago) > > > > If not, I may need to pull out my eclipse (YET AGAIN) to solve load > balancing issues in my environment. > > *From:* Action Request System discussion list(ARSList) [mailto: > arsl...@arslist.org] *On Behalf Of *Axton > *Sent:* Friday, December 10, 2010 9:54 AM > > *To:* arslist@ARSLIST.ORG > *Subject:* Re: Load Balancers and how? > > > > ** You will want to handle traffic for both rpcbind and arserver, and > potentially your plugin servers, DSO, on the LB, depending on your > implementation. The easiest check is just a port check. I think most LB > devices provide this capability. As you recently saw with your > Mid-Tier/Tomcat problems, a simple port check does not always fault the node > if there is a problem. > > > > What kind of device are you using to balance the traffic? If you are using > an F5 LTM, iRules can be written to do more extensive checks. > > > > I was part of a thread on the subject here that can provide some additional > information: > > > http://devcentral.f5.com/Forums/tabid/1082223/asg/52/showtab/groupforums/afv/topic/aff/31/aft/19000/Default.aspx > > > > On Fri, Dec 10, 2010 at 9:49 AM, LJ LongWing > wrote: > > ** > > Ok….I’m going down a road that I KNOW some of you have travelled before, I > need your travel tips. We are setting up a server group on 7.5 Windows and > we are putting a load balancer in front of it. How do you all tell if a > node is up? There are 4 types of probes that I’m being told are available… > > > > PING > > HTTP > > TCP > > Script > > > > Ping - Obviously doesn’t tell you anything other than the host OS is up > > HTTP - Won’t work for telling if a remedy service is up > > TCP – My instincts are telling me that this one simply does something > similar to a telnet to a given port, if the port allows you ‘in’, then the > service is considered up. I don’t consider this one viable because it > doesn’t ‘test’ much and I expect that my remedy server could respond to the > port request without actually being functional > > Script – This one is the most promising in my mind, but I know nothing > about the scripting language of TCL. > > > > How does the rest of the remedy world ‘verify’ that their remedy instance > is online so that their load balancer can know to route traffic to it or > not. I just this week finished a java servlet that allows the mid tier load > balancer to know if an individual node is working properly, but the same > can’t be said yet for Remedy….any and all help is appreciated > > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > > > > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ > ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
Re: Load Balancers and how?
Some Load Balancers are able to snmp checks. I've found that this works pretty well with arsnmpd. -Dave LJ LongWing To Sent by: "Action arslist@ARSLIST.ORG Request System cc discussion list(ARSList)"Subject 12/10/2010 01:15 PM Please respond to arsl...@arslist.o RG ** Dave, I also have similar monitoring on the app/db/web servers…but that doesn’t solve the problem of having two app servers with a load balancer sitting in front of them, and ensuring that if node 1 goes offline, that the load balancer will stop sending traffic to it…that’s what I’m needing. From: Action Request System discussion list(ARSList) [ mailto:arsl...@arslist.org] On Behalf Of Shellman, David Sent: Friday, December 10, 2010 9:37 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** LJ, Here are a few approaches that can be used. A product called SiteScope can be configured to access a form through the MidTier. This gives you end to end up/down information. We also use HP OpenView to monitor the services on the MidTier, app and Database servers. We also monitor the arerror, stderr.log and stdout log files on the app server, the Oracle logs on the DB server and the MidTier log files on the MidTier servers. I have also created a health check for email services. This simply sends an email out of the system to one of two email addresses we use for incoming ticket generation. The email creates a record in a receiving form that when the record is created updates the record on the sending form. If the record on the sending form does not generate within a specific time period an escalation will send us email and page us. Dave From: Action Request System discussion list(ARSList) [ mailto:arsl...@arslist.org] On Behalf Of LJ LongWing Sent: Friday, December 10, 2010 11:12 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** Roger, What method did they use to determine it was started/running? I personally have found ‘other teams’ don’t usually have a concept of what ‘functional’ means in a remedy world. I don’t consider the fact that arserver.exe is in memory an indicator that my server is functional…I’ve had TOO many situations where it was in memory, but you couldn’t log onto it. From: Action Request System discussion list(ARSList) [ mailto:arsl...@arslist.org] On Behalf Of Roger Justice Sent: Friday, December 10, 2010 9:00 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** I was able to have the group that configured the load balancer determine if the Remedy Service was started/running. -Original Message- From: LJ LongWing To: arslist Sent: Fri, Dec 10, 2010 10:51 am Subject: Load Balancers and how? ** Ok….I’m going down a road that I KNOW some of you have travelled before, I need your travel tips. We are setting up a server group on 7.5 Windows and we are putting a load balancer in front of it. How do you all tell if a node is up? There are 4 types of probes that I’m being told are available… PING HTTP TCP Script Ping - Obviously doesn’t tell you anything other than the host OS is up HTTP - Won’t work for telling if a remedy service is up TCP – My instincts are telling me that this one simply does something similar to a telnet to a given port, if the port allows you ‘in’, then the service is considered up. I don’t consider this one viable because it doesn’t ‘test’ much and I expect that my remedy server could respond to the port request without actually being functional Script – This one is the most promising in my mind, but I know nothing about the scripting language of TCL. How does the rest of the remedy world ‘verify’ that their remedy instance is online so that their load balancer ca
Re: Load Balancers and how?
I'm not seeing that capability of the Cisco CSM unfortunately. -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of david.zifch...@apcc.com Sent: Friday, December 10, 2010 12:28 PM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? Some Load Balancers are able to snmp checks. I've found that this works pretty well with arsnmpd. -Dave LJ LongWing To Sent by: "Action arslist@ARSLIST.ORG Request System cc discussion list(ARSList)"Subject 12/10/2010 01:15 PM Please respond to arsl...@arslist.o RG ** Dave, I also have similar monitoring on the app/db/web servers…but that doesn’t solve the problem of having two app servers with a load balancer sitting in front of them, and ensuring that if node 1 goes offline, that the load balancer will stop sending traffic to it…that’s what I’m needing. From: Action Request System discussion list(ARSList) [ mailto:arsl...@arslist.org] On Behalf Of Shellman, David Sent: Friday, December 10, 2010 9:37 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** LJ, Here are a few approaches that can be used. A product called SiteScope can be configured to access a form through the MidTier. This gives you end to end up/down information. We also use HP OpenView to monitor the services on the MidTier, app and Database servers. We also monitor the arerror, stderr.log and stdout log files on the app server, the Oracle logs on the DB server and the MidTier log files on the MidTier servers. I have also created a health check for email services. This simply sends an email out of the system to one of two email addresses we use for incoming ticket generation. The email creates a record in a receiving form that when the record is created updates the record on the sending form. If the record on the sending form does not generate within a specific time period an escalation will send us email and page us. Dave From: Action Request System discussion list(ARSList) [ mailto:arsl...@arslist.org] On Behalf Of LJ LongWing Sent: Friday, December 10, 2010 11:12 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** Roger, What method did they use to determine it was started/running? I personally have found ‘other teams’ don’t usually have a concept of what ‘functional’ means in a remedy world. I don’t consider the fact that arserver.exe is in memory an indicator that my server is functional…I’ve had TOO many situations where it was in memory, but you couldn’t log onto it. From: Action Request System discussion list(ARSList) [ mailto:arsl...@arslist.org] On Behalf Of Roger Justice Sent: Friday, December 10, 2010 9:00 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** I was able to have the group that configured the load balancer determine if the Remedy Service was started/running. -Original Message- From: LJ LongWing To: arslist Sent: Fri, Dec 10, 2010 10:51 am Subject: Load Balancers and how? ** Ok….I’m going down a road that I KNOW some of you have travelled before, I need your travel tips. We are setting up a server group on 7.5 Windows and we are putting a load balancer in front of it. How do you all tell if a node is up? There are 4 types of probes that I’m being told are available… PING HTTP TCP Script Ping - Obviously doesn’t tell you anything other than the host OS is up HTTP - Won’t work for telling if a remedy service is up TCP – My instincts are telling me that this one simply does something similar to a telnet to a given port, if the port allows you ‘in’, then the service is considered up. I don’t consider this one viable because it doesn’t ‘test’ much and I e
Re: Load Balancers and how?
http://www.cisco.com/en/US/products/hw/modules/ps2706/prod_bulletin09186a0080107b2b.html "Toolkit Command Language (TCL) scripting—To support more flexible health-probing functionality, this feature gives the administrator the ability to upload and execute TCL scripts on the Cisco CSM. The administrator can create a "script probe" that the Cisco CSM periodically executes for each real server in any server farm associated with the probe. Depending upon the exit code of such a script, the real server is considered healthy, suspect, or failed. A wide variety of probing functions are possible using the flexibility of the TCL scripting environment. The Cisco CSM also supports execution of custom TCL scripts that are not directly associated with a particular server health probe. A "standalone script" dynamically executes a task at a specified interval." http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/csm/4.2.x/configuration/guide/scriptg.pdf On Fri, Dec 10, 2010 at 7:39 PM, LJ LongWing wrote: > I'm not seeing that capability of the Cisco CSM unfortunately. > > -Original Message- > From: Action Request System discussion list(ARSList) [mailto: > arsl...@arslist.org] On Behalf Of david.zifch...@apcc.com > Sent: Friday, December 10, 2010 12:28 PM > To: arslist@ARSLIST.ORG > Subject: Re: Load Balancers and how? > > Some Load Balancers are able to snmp checks. I've found that this works > pretty well with arsnmpd. > > -Dave > > > > > LJ LongWing > L.COM> To > Sent by: "Action arslist@ARSLIST.ORG > Request System cc > discussion > list(ARSList)"Subject > ORG> > > > 12/10/2010 01:15 > PM > > > Please respond to > arsl...@arslist.o >RG > > > > > > > ** > Dave, > I also have similar monitoring on the app/db/web servers…but that doesn’t > solve the problem of having two app servers with a load balancer sitting in > front of them, and ensuring that if node 1 goes offline, that the load > balancer will stop sending traffic to it…that’s what I’m needing. > > From: Action Request System discussion list(ARSList) [ > mailto:arsl...@arslist.org] On Behalf Of Shellman, David > Sent: Friday, December 10, 2010 9:37 AM > To: arslist@ARSLIST.ORG > Subject: Re: Load Balancers and how? > > ** > LJ, > > Here are a few approaches that can be used. > > A product called SiteScope can be configured to access a form through the > MidTier. This gives you end to end up/down information. > > We also use HP OpenView to monitor the services on the MidTier, app and > Database servers. We also monitor the arerror, stderr.log and stdout log > files on the app server, the Oracle logs on the DB server and the MidTier > log files on the MidTier servers. > > I have also created a health check for email services. This simply sends > an email out of the system to one of two email addresses we use for > incoming ticket generation. The email creates a record in a receiving form > that when the record is created updates the record on the sending form. If > the record on the sending form does not generate within a specific time > period an escalation will send us email and page us. > > Dave > > > From: Action Request System discussion list(ARSList) [ > mailto:arsl...@arslist.org] On Behalf Of LJ LongWing > Sent: Friday, December 10, 2010 11:12 AM > To: arslist@ARSLIST.ORG > Subject: Re: Load Balancers and how? > ** > Roger, > What method did they use to determine it was started/running? I personally > have found ‘other teams’ don’t usually have a concept of what ‘functional’ > means in a remedy world. I don’t consider the fact that arserver.exe is in > memory an indicator that my server is functional…I’ve had TOO many > situations where it was in memory, but you couldn’t log onto it. > > From: Action Request System discussion list(ARSList) [ > mailto:arsl...@arslist.org] On Behalf Of Roger Justice > Sent: Friday, December 10, 2010 9:00 AM > To: arslist@ARSLIST.ORG > Subject: Re: Load Balancers and how? > > ** I was able to have the group that configured the load balancer determine > if the Remedy Service was started/running. > > -Original Message- > From: LJ LongWing > To: arslist > Sent: Fri, Dec 10, 2010 10:51 am > Subject: Load Balancers and how? > ** > Ok….I’m going down a road that I KNOW some of you have travelled before, I &g
Re: Load Balancers and how?
Yes..so I guess the question then would be Does anyone have a TCL script laying around they use to monitor the health of their remedy server? From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Axton Sent: Friday, December 10, 2010 7:49 PM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** <http://www.cisco.com/en/US/products/hw/modules/ps2706/prod_bulletin09186a00 80107b2b.html> http://www.cisco.com/en/US/products/hw/modules/ps2706/prod_bulletin09186a008 0107b2b.html "Toolkit Command Language (TCL) scripting-To support more flexible health-probing functionality, this feature gives the administrator the ability to upload and execute TCL scripts on the Cisco CSM. The administrator can create a "script probe" that the Cisco CSM periodically executes for each real server in any server farm associated with the probe. Depending upon the exit code of such a script, the real server is considered healthy, suspect, or failed. A wide variety of probing functions are possible using the flexibility of the TCL scripting environment. The Cisco CSM also supports execution of custom TCL scripts that are not directly associated with a particular server health probe. A "standalone script" dynamically executes a task at a specified interval." http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/csm/4.2. x/configuration/guide/scriptg.pdf On Fri, Dec 10, 2010 at 7:39 PM, LJ LongWing wrote: I'm not seeing that capability of the Cisco CSM unfortunately. -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of david.zifch...@apcc.com Sent: Friday, December 10, 2010 12:28 PM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? Some Load Balancers are able to snmp checks. I've found that this works pretty well with arsnmpd. -Dave LJ LongWing To Sent by: "Action arslist@ARSLIST.ORG Request System cc discussion list(ARSList)"Subject 12/10/2010 01:15 PM Please respond to arsl...@arslist.o RG ** Dave, I also have similar monitoring on the app/db/web servers.but that doesn't solve the problem of having two app servers with a load balancer sitting in front of them, and ensuring that if node 1 goes offline, that the load balancer will stop sending traffic to it.that's what I'm needing. From: Action Request System discussion list(ARSList) [ mailto:arsl...@arslist.org] On Behalf Of Shellman, David Sent: Friday, December 10, 2010 9:37 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** LJ, Here are a few approaches that can be used. A product called SiteScope can be configured to access a form through the MidTier. This gives you end to end up/down information. We also use HP OpenView to monitor the services on the MidTier, app and Database servers. We also monitor the arerror, stderr.log and stdout log files on the app server, the Oracle logs on the DB server and the MidTier log files on the MidTier servers. I have also created a health check for email services. This simply sends an email out of the system to one of two email addresses we use for incoming ticket generation. The email creates a record in a receiving form that when the record is created updates the record on the sending form. If the record on the sending form does not generate within a specific time period an escalation will send us email and page us. Dave From: Action Request System discussion list(ARSList) [ mailto:arsl...@arslist.org] On Behalf Of LJ LongWing Sent: Friday, December 10, 2010 11:12 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** Roger, What method did they use to determine it was started/running? I personally have found 'other teams' don't usually have a concept of what 'functional' means in a remedy world. I don't consider the fact that arserver.exe is in memory an indicator that my server is functional.I've had TOO many situations where it was in memory, but you couldn't log onto it. From: Action Request System discussion list(ARSList) [ mailto:arsl...@arslist.org] On Behalf Of Roger Justice Sent: Friday, December 10, 2010 9:00 AM To: arslist@ARSLIST.ORG Subject: Re: Load Balancers and how? ** I was able to have the group that configured the load balancer determine if the Remedy Service was started/running. -Original Message- From: LJ LongWing To: arslist Sent: Fri, Dec 10, 2010 10:51 am Subject: Load Balancers and how? ** Ok..I'm going down a road that I KNOW some of you have travelled before, I need your t
