Group emails slow
Schlumberger-Public Hi all, I am wondering if anyone else has experienced the issues that I am seeing. Group emails will fly out of the system with no issue for about 4 days, then on that 5th day, they start lagging and taking longer and longer to leave the system. The form that I am looking at is the NTE:SYS-NT Process Control form. So, I know that there are escalations around this form depending on if it is an individual email or a group email. The individual ones will continue to go out at a fast pace, it is just the group emails that take longer. I understand that it has to loop through a table with all of the people that are assigned to that group, but like I said in the first 4 days after a restart of the Remedy service, they go out at a fast pace. Just to give you an idea, I have seen the group emails get 20-30 minutes behind. I am thinking this is more of an escalation issue as if I restart just the email service, it does not help any. It does not matter if I restart the Remedy service on a Monday at 8am or a Thursday at 2pm, within 7 days, the group emails will be slow in going out. Also during this time, we will see the CPU of this server go up to around 30% or above and stay there, where in the first 4 days it is between 10-20%. I have relayed all of this to BMC and they have tons of logs (we have been working for 6 month on this 1 issue). They have asked me to increase escalation thread, move the group escalations around, and move the escalations from one server to another. None of this has helped at all. I have asked for BMC to escalate the issue and the same person keeps calling me and says they are working with the escalated person - starting to have doubts about that. Here is our environment: ARS - 7.6.04 SP3 ITSM (Incident, Change, and Asset) - 7.6.04 SP2 Database - Oracle 11g - 64bit Application server - 2 Linux servers that are VMs that are in a server group MidTier - 3 servers - 7.6.04 SP4 with the March 4, 2013 Hotfix ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Group emails slow
Greg, Do you notice the problem with a particular group or is it all groups? Also, there are a couple of NTE:SYS-NT... forms- I think a Log form and of course the Processing form, how many records are in the forms when it starts to slow down? Have you seen any change in the count of records on that form after you restart ARS? Also, I wonder if you are running into indexing problems with all of the push fields that occur during this processing event and just the sheer volume of records that end up in the notification forms. Janie On Mon, Feb 24, 2014 at 8:03 AM, Greg Donalson cdonal...@slb.com wrote: Schlumberger-Public Hi all, I am wondering if anyone else has experienced the issues that I am seeing. Group emails will fly out of the system with no issue for about 4 days, then on that 5th day, they start lagging and taking longer and longer to leave the system. The form that I am looking at is the NTE:SYS-NT Process Control form. So, I know that there are escalations around this form depending on if it is an individual email or a group email. The individual ones will continue to go out at a fast pace, it is just the group emails that take longer. I understand that it has to loop through a table with all of the people that are assigned to that group, but like I said in the first 4 days after a restart of the Remedy service, they go out at a fast pace. Just to give you an idea, I have seen the group emails get 20-30 minutes behind. I am thinking this is more of an escalation issue as if I restart just the email service, it does not help any. It does not matter if I restart the Remedy service on a Monday at 8am or a Thursday at 2pm, within 7 days, the group emails will be slow in going out. Also during this time, we will see the CPU of this server go up to around 30% or above and stay there, where in the first 4 days it is between 10-20%. I have relayed all of this to BMC and they have tons of logs (we have been working for 6 month on this 1 issue). They have asked me to increase escalation thread, move the group escalations around, and move the escalations from one server to another. None of this has helped at all. I have asked for BMC to escalate the issue and the same person keeps calling me and says they are working with the escalated person - starting to have doubts about that. Here is our environment: ARS - 7.6.04 SP3 ITSM (Incident, Change, and Asset) - 7.6.04 SP2 Database - Oracle 11g - 64bit Application server - 2 Linux servers that are VMs that are in a server group MidTier - 3 servers - 7.6.04 SP4 with the March 4, 2013 Hotfix ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Group emails slow
Hi, What do you see in the AR System Email Messages form when the lag occurs, are there a large number of unprocessed records (emails) waiting to be sent? Are there records waiting to be processed in the Notification Engine when the bottleneck occurs? Best to determine where the bottleneck is and go from there. The Email Engine maybe getting behind and AR Server queuing up the processing, so a restart of the AR Server maybe resetting the queuing. The Notification Engine is designed to be self deleting, and therefore does a good job at cleaning out Notifications when successfully processed - the AR System Email Messages however does not delete messages. Kind Regards, Carl Wilson http://www.missingpiecessoftware.com/ -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Greg Donalson Sent: 24 February 2014 16:03 To: arslist@ARSLIST.ORG Subject: Group emails slow Schlumberger-Public Hi all, I am wondering if anyone else has experienced the issues that I am seeing. Group emails will fly out of the system with no issue for about 4 days, then on that 5th day, they start lagging and taking longer and longer to leave the system. The form that I am looking at is the NTE:SYS-NT Process Control form. So, I know that there are escalations around this form depending on if it is an individual email or a group email. The individual ones will continue to go out at a fast pace, it is just the group emails that take longer. I understand that it has to loop through a table with all of the people that are assigned to that group, but like I said in the first 4 days after a restart of the Remedy service, they go out at a fast pace. Just to give you an idea, I have seen the group emails get 20-30 minutes behind. I am thinking this is more of an escalation issue as if I restart just the email service, it does not help any. It does not matter if I restart the Remedy service on a Monday at 8am or a Thursday at 2pm, within 7 days, the group emails will be slow in going out. Also during this time, we will see the CPU of this server go up to around 30% or above and stay there, where in the first 4 days it is between 10-20%. I have relayed all of this to BMC and they have tons of logs (we have been working for 6 month on this 1 issue). They have asked me to increase escalation thread, move the group escalations around, and move the escalations from one server to another. None of this has helped at all. I have asked for BMC to escalate the issue and the same person keeps calling me and says they are working with the escalated person - starting to have doubts about that. Here is our environment: ARS - 7.6.04 SP3 ITSM (Incident, Change, and Asset) - 7.6.04 SP2 Database - Oracle 11g - 64bit Application server - 2 Linux servers that are VMs that are in a server group MidTier - 3 servers - 7.6.04 SP4 with the March 4, 2013 Hotfix ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Group emails slow
HI What are you group sizes like? BMC recommends that for groups larger than 25 that you use the group email option and send to a DL. This way you have one email going out rather than 25. Mark -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Carl Wilson Sent: Monday, February 24, 2014 12:39 PM To: arslist@ARSLIST.ORG Subject: Re: Group emails slow Hi, What do you see in the AR System Email Messages form when the lag occurs, are there a large number of unprocessed records (emails) waiting to be sent? Are there records waiting to be processed in the Notification Engine when the bottleneck occurs? Best to determine where the bottleneck is and go from there. The Email Engine maybe getting behind and AR Server queuing up the processing, so a restart of the AR Server maybe resetting the queuing. The Notification Engine is designed to be self deleting, and therefore does a good job at cleaning out Notifications when successfully processed - the AR System Email Messages however does not delete messages. Kind Regards, Carl Wilson http://www.missingpiecessoftware.com/ -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Greg Donalson Sent: 24 February 2014 16:03 To: arslist@ARSLIST.ORG Subject: Group emails slow Schlumberger-Public Hi all, I am wondering if anyone else has experienced the issues that I am seeing. Group emails will fly out of the system with no issue for about 4 days, then on that 5th day, they start lagging and taking longer and longer to leave the system. The form that I am looking at is the NTE:SYS-NT Process Control form. So, I know that there are escalations around this form depending on if it is an individual email or a group email. The individual ones will continue to go out at a fast pace, it is just the group emails that take longer. I understand that it has to loop through a table with all of the people that are assigned to that group, but like I said in the first 4 days after a restart of the Remedy service, they go out at a fast pace. Just to give you an idea, I have seen the group emails get 20-30 minutes behind. I am thinking this is more of an escalation issue as if I restart just the email service, it does not help any. It does not matter if I restart the Remedy service on a Monday at 8am or a Thursday at 2pm, within 7 days, the group emails will be slow in going out. Also during this time, we will see the CPU of this server go up to around 30% or above and stay there, where in the first 4 days it is between 10-20%. I have relayed all of this to BMC and they have tons of logs (we have been working for 6 month on this 1 issue). They have asked me to increase escalation thread, move the group escalations around, and move the escalations from one server to another. None of this has helped at all. I have asked for BMC to escalate the issue and the same person keeps calling me and says they are working with the escalated person - starting to have doubts about that. Here is our environment: ARS - 7.6.04 SP3 ITSM (Incident, Change, and Asset) - 7.6.04 SP2 Database - Oracle 11g - 64bit Application server - 2 Linux servers that are VMs that are in a server group MidTier - 3 servers - 7.6.04 SP4 with the March 4, 2013 Hotfix ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Group emails slow
** Greg,I’ve seen tis exact symptom before. Support is on the right track.There are two escalations that process records in this form. One turns Notifications addressed to a group into a bunch of Notifications addressed to individual people, and the other processes the individual messages taking into account the user’s preference, performing keyword substitution and sending the message to AR System Email Messages. The problem you’re seeing happens when a large number of notifications are to be processed and the amount of time taken by the first escalation, plus the amount of time for other escalations running in the same pool, takes longer than the escalation interval. In that condition the first escalation is skipped or delayed until the next time for it to be called comes around. This delays the expansion of the group messages while individual ones continue tone processed every minute.For example, if it takes 5 minutes to process the group notifications and another 7 minutes to process all the individual messages, it is 12 minutes before the group escalation could be considered again. There are other escalations running in the same pool (or if you don’t use pools, all on the same escalation thread). So the group escalation is delayed.As I recall the group escalation runs every 10 minutes and the individual one every minute. if you turn on escalation logging you can see the order in which the escalations fire, and that one has to run all the way through before the next one starts, and that ALL the escalations due to run at a specific time have to be processed before the system gets the list again.To deal with this we did three things.First we added several pools for escalations, and moved the one that processes individual messages to its own pool. This means nothing gets in its way. The second was to move the group escalation also to its own pool, that way nothing delays it either.The third thing was to add multiple threads for 390603. This means that in cases where both of these escalations are due to run at the same time they can in fact be processed in parallel.In this configuration that big escalation that closes old incidents or removed processed email messages will not tie up an escalation thread that needs to process outgoing messages. I suspect you’ll find something that does a large scale operation on incidents or people records or some such.And… of course, anything that your DBA tells you about bad queries looking up people’s email address or their user preferences for notifications should be looked at, but changes the the huber of users with notification preferences, as well as the number of pending group notifications, are both going to be fairly small numbers.Feel free to contact me off list if you need some more detail.Doug BlairOn Feb 24, 2014, at 10:03 AM, Greg Donalson cdonal...@slb.com wrote:Schlumberger-PublicHi all,I am wondering if anyone else has experienced the issues that I am seeing. Group emails will flyout of the system with no issue for about 4 days, then on that 5th day, they start lagging andtaking longer and longer to leave the system. The form that I am looking at is the NTE:SYS-NTProcess Control form. So, I know that there are escalations around this form depending on if it isan individual email or a group email. The individual ones will continue to go out at a fast pace,it is just the group emails that take longer. I understand that it has to loop through a table withall of the people that are assigned to that group, but like I said in the first 4 days after arestart of the Remedy service, they go out at a fast pace. Just to give you an idea, I have seenthe group emails get 20-30 minutes behind. I am thinking this is more of an escalation issue as ifI restart just the email service, it does not help any. It does not matter if I restart the Remedyservice on a Monday at 8am or a Thursday at 2pm, within 7 days, the group emails will be slow ingoing out. Also during this time, we will see the CPU of this server go up to around 30% or aboveand stay there, where in the first 4 days it is between 10-20%. I have relayed all of this to BMCand they have tons of logs (we have been working for 6 month on this 1 issue). They have asked meto increase escalation thread, move the group escalations around, and move the escalations from oneserver to another. None of this has helped at all. I have asked for BMC to escalate the issue andthe same person keeps calling me and says they are working with the escalated person - starting tohave doubts about that.Here is our environment:ARS - 7.6.04 SP3ITSM (Incident, Change, and Asset) - 7.6.04 SP2Database - Oracle 11g - 64bitApplication server - 2 Linux servers that are VMs that are in a server groupMidTier - 3 servers - 7.6.04 SP4 with the March 4, 2013 Hotfix___UNSUBSCRIBE or access ARSlist Archives at www.arslist.org"Where the
Re: Group emails slow
Schlumberger-Public Hi all, Thanks to all that have given answers. Let me try to answer some of these questions: It does not matter which group it is trying to get the individuals from, it is still slow. Also, as soon as I restart the Remedy service the group emails are gone within a minute. In the AR System Email Messages form, they are sent fast even during this slowness. There is no issue here. We do have them deleting once they are sent. I also go through the ones that have errors and delete those on a regular basis, so the AR System Email Messages form is not holding old data. There are some groups that have lots of members, but for a number of these we have turned off the group email. Also, we have multiple escalation threads and have the individual ones on their own thread and the group ones on their own thread. The issue is that through 4 days or so, they are sending out fast and are gone out of the NTE:SYS-NT Process Control form within 2 minutes because we have the group emails sending every minute as well as the individual ones sending every minute (meaning the escalation is running every minute for each of those). Thanks! Greg From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Doug Blair Sent: Monday, February 24, 2014 1:19 PM To: arslist@ARSLIST.ORG Subject: Re: Group emails slow ** Greg, I've seen tis exact symptom before. Support is on the right track. There are two escalations that process records in this form. One turns Notifications addressed to a group into a bunch of Notifications addressed to individual people, and the other processes the individual messages taking into account the user's preference, performing keyword substitution and sending the message to AR System Email Messages. The problem you're seeing happens when a large number of notifications are to be processed and the amount of time taken by the first escalation, plus the amount of time for other escalations running in the same pool, takes longer than the escalation interval. In that condition the first escalation is skipped or delayed until the next time for it to be called comes around. This delays the expansion of the group messages while individual ones continue tone processed every minute. For example, if it takes 5 minutes to process the group notifications and another 7 minutes to process all the individual messages, it is 12 minutes before the group escalation could be considered again. There are other escalations running in the same pool (or if you don't use pools, all on the same escalation thread). So the group escalation is delayed. As I recall the group escalation runs every 10 minutes and the individual one every minute. if you turn on escalation logging you can see the order in which the escalations fire, and that one has to run all the way through before the next one starts, and that ALL the escalations due to run at a specific time have to be processed before the system gets the list again. To deal with this we did three things. First we added several pools for escalations, and moved the one that processes individual messages to its own pool. This means nothing gets in its way. The second was to move the group escalation also to its own pool, that way nothing delays it either. The third thing was to add multiple threads for 390603. This means that in cases where both of these escalations are due to run at the same time they can in fact be processed in parallel. In this configuration that big escalation that closes old incidents or removed processed email messages will not tie up an escalation thread that needs to process outgoing messages. I suspect you'll find something that does a large scale operation on incidents or people records or some such. And... of course, anything that your DBA tells you about bad queries looking up people's email address or their user preferences for notifications should be looked at, but changes the the huber of users with notification preferences, as well as the number of pending group notifications, are both going to be fairly small numbers. Feel free to contact me off list if you need some more detail. Doug Blair On Feb 24, 2014, at 10:03 AM, Greg Donalson cdonal...@slb.commailto:cdonal...@slb.com wrote: Schlumberger-Public Hi all, I am wondering if anyone else has experienced the issues that I am seeing. Group emails will fly out of the system with no issue for about 4 days, then on that 5th day, they start lagging and taking longer and longer to leave the system. The form that I am looking at is the NTE:SYS-NT Process Control form. So, I know that there are escalations around this form depending on if it is an individual email or a group email. The individual ones will continue to go out at a fast pace, it is just the group emails that take longer. I understand
