Expiry Overhead
We've had some performance problems in a synchronous app lately. It's a VB-NT client connected to a Solaris concentrator. Messages are put to a queues that go thru WMQI on Solaris. Flows convert XML to Cobol, and forward them to the mainframe. CICS programs do some DB2 work, and send replies that follow the same path in the opposite direction. We've noticed that these round trips slow way, way down at noon and 4:00 PM on Monday, Thursday and Friday. But we don't see this on Tuesday and Wednesday. After much research, we found a Perl script that runs on the WMQI server every day at noon and 4:00. The script goes through a queue and does GETs for a length of 0. This queue is an output only, archive queue, with messages that expire 3 days after they're put. The purpose of the script is to allow messages older than three days to expire. Our theory is that the overhead of expiration is noticeably slowing down WMQI. The reason Tuesday and Wednesday are OK is because far fewer messages expire on those days (since the only messages eligible to expire were from the previous weekend). We turned this script off, and will know on Friday if this is the problem. Our WMQI server is a dual processor machine loaded with memory. It's very surprising to me this script could hurt throughput this much, but it certainly looks that way. Has anyone seen this? Does IBM have any info as to what sort of overhead is involved in expiring messages? Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
Re: Expiry Overhead
Jim - Out of curiosity: Are these persistent messages? Are they relatively large? I would expect that the actual, physical deletion of a persistent, expired message would require the same logging overhead as any normal MQGET of the same persistent message. If you are deleting a large volume of persistent messages this might have a very noticeable impact on the logger. - Steve Gies -Original Message- From: Jim Ford [mailto:[EMAIL PROTECTED]] Sent: Tuesday, November 26, 2002 11:40 AM To: [EMAIL PROTECTED] Subject: Expiry Overhead We've had some performance problems in a synchronous app lately. It's a VB-NT client connected to a Solaris concentrator. Messages are put to a queues that go thru WMQI on Solaris. Flows convert XML to Cobol, and forward them to the mainframe. CICS programs do some DB2 work, and send replies that follow the same path in the opposite direction. We've noticed that these round trips slow way, way down at noon and 4:00 PM on Monday, Thursday and Friday. But we don't see this on Tuesday and Wednesday. After much research, we found a Perl script that runs on the WMQI server every day at noon and 4:00. The script goes through a queue and does GETs for a length of 0. This queue is an output only, archive queue, with messages that expire 3 days after they're put. The purpose of the script is to allow messages older than three days to expire. Our theory is that the overhead of expiration is noticeably slowing down WMQI. The reason Tuesday and Wednesday are OK is because far fewer messages expire on those days (since the only messages eligible to expire were from the previous weekend). We turned this script off, and will know on Friday if this is the problem. Our WMQI server is a dual processor machine loaded with memory. It's very surprising to me this script could hurt throughput this much, but it certainly looks that way. Has anyone seen this? Does IBM have any info as to what sort of overhead is involved in expiring messages? Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
Re: Expiry Overhead
No, it's using _INPUT_AS_Q_DEF and _BROWSE. Plus the queue is written to by a WMQI message flow, which means that WMQI already has the queue opened. So the script could only access it in a shared mode. DiLauro, Nick [EMAIL PROTECTED]To: [EMAIL PROTECTED] cc: Sent by: MQSeriesSubject: Re: Expiry Overhead List MQSERIES@AKH-WIE N.AC.AT 11/26/2002 02:41 PM Please respond to MQSeries List Just a thought. Could the Perl script be opening the queue with exclusive access and somehow slowing some process in the application which also accesses this queue? -Original Message- From: Jim Ford [mailto:[EMAIL PROTECTED]] Sent: Tuesday, November 26, 2002 11:40 AM To: [EMAIL PROTECTED] Subject: Expiry Overhead We've had some performance problems in a synchronous app lately. It's a VB-NT client connected to a Solaris concentrator. Messages are put to a queues that go thru WMQI on Solaris. Flows convert XML to Cobol, and forward them to the mainframe. CICS programs do some DB2 work, and send replies that follow the same path in the opposite direction. We've noticed that these round trips slow way, way down at noon and 4:00 PM on Monday, Thursday and Friday. But we don't see this on Tuesday and Wednesday. After much research, we found a Perl script that runs on the WMQI server every day at noon and 4:00. The script goes through a queue and does GETs for a length of 0. This queue is an output only, archive queue, with messages that expire 3 days after they're put. The purpose of the script is to allow messages older than three days to expire. Our theory is that the overhead of expiration is noticeably slowing down WMQI. The reason Tuesday and Wednesday are OK is because far fewer messages expire on those days (since the only messages eligible to expire were from the previous weekend). We turned this script off, and will know on Friday if this is the problem. Our WMQI server is a dual processor machine loaded with memory. It's very surprising to me this script could hurt throughput this much, but it certainly looks that way. Has anyone seen this? Does IBM have any info as to what sort of overhead is involved in expiring messages? Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
Re: Expiry Overhead
Yes, they are persistent. They're small, though, only 334 bytes. Still, I imagine you're right that the logger is where the bottleneck resides. I suppose that the logger process is single threaded, and gets overwhelmed by the work that's required to delete the messages. It's not something I thought to worry about, but I guess I'd better. GIES, STEVE [EMAIL PROTECTED]To: [EMAIL PROTECTED] M cc: Sent by: MQSeriesSubject: Re: Expiry Overhead List MQSERIES@AKH-WIE N.AC.AT 11/26/2002 03:00 PM Please respond to MQSeries List Jim - Out of curiosity: Are these persistent messages? Are they relatively large? I would expect that the actual, physical deletion of a persistent, expired message would require the same logging overhead as any normal MQGET of the same persistent message. If you are deleting a large volume of persistent messages this might have a very noticeable impact on the logger. - Steve Gies -Original Message- From: Jim Ford [mailto:[EMAIL PROTECTED]] Sent: Tuesday, November 26, 2002 11:40 AM To: [EMAIL PROTECTED] Subject: Expiry Overhead We've had some performance problems in a synchronous app lately. It's a VB-NT client connected to a Solaris concentrator. Messages are put to a queues that go thru WMQI on Solaris. Flows convert XML to Cobol, and forward them to the mainframe. CICS programs do some DB2 work, and send replies that follow the same path in the opposite direction. We've noticed that these round trips slow way, way down at noon and 4:00 PM on Monday, Thursday and Friday. But we don't see this on Tuesday and Wednesday. After much research, we found a Perl script that runs on the WMQI server every day at noon and 4:00. The script goes through a queue and does GETs for a length of 0. This queue is an output only, archive queue, with messages that expire 3 days after they're put. The purpose of the script is to allow messages older than three days to expire. Our theory is that the overhead of expiration is noticeably slowing down WMQI. The reason Tuesday and Wednesday are OK is because far fewer messages expire on those days (since the only messages eligible to expire were from the previous weekend). We turned this script off, and will know on Friday if this is the problem. Our WMQI server is a dual processor machine loaded with memory. It's very surprising to me this script could hurt throughput this much, but it certainly looks that way. Has anyone seen this? Does IBM have any info as to what sort of overhead is involved in expiring messages? Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
Re: Expiry Overhead
Jim, I think I remember reading in the announcment of MQ V5.3 that you no longer have to write these scavenger routines to get rid of expired messages, but that the queue manager itself will take care of that. Which version of MQ are you currently on? I don't know if my memory serves me well, but if this processing is causing you trouble because of massive log activity you might see some remedy after your next upgrade. Surely doesn't help much right now, though. Can somebody confirm? Cheers, Stefan From: Jim Ford [EMAIL PROTECTED] Reply-To: MQSeries List [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Expiry Overhead Date: Tue, 26 Nov 2002 13:39:32 -0600 We've had some performance problems in a synchronous app lately. It's a VB-NT client connected to a Solaris concentrator. Messages are put to a queues that go thru WMQI on Solaris. Flows convert XML to Cobol, and forward them to the mainframe. CICS programs do some DB2 work, and send replies that follow the same path in the opposite direction. We've noticed that these round trips slow way, way down at noon and 4:00 PM on Monday, Thursday and Friday. But we don't see this on Tuesday and Wednesday. After much research, we found a Perl script that runs on the WMQI server every day at noon and 4:00. The script goes through a queue and does GETs for a length of 0. This queue is an output only, archive queue, with messages that expire 3 days after they're put. The purpose of the script is to allow messages older than three days to expire. Our theory is that the overhead of expiration is noticeably slowing down WMQI. The reason Tuesday and Wednesday are OK is because far fewer messages expire on those days (since the only messages eligible to expire were from the previous weekend). We turned this script off, and will know on Friday if this is the problem. Our WMQI server is a dual processor machine loaded with memory. It's very surprising to me this script could hurt throughput this much, but it certainly looks that way. Has anyone seen this? Does IBM have any info as to what sort of overhead is involved in expiring messages? Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive _ The new MSN 8: smart spam protection and 2 months FREE* http://join.msn.com/?page=features/junkmail Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
Re: Expiry Overhead
I'm on 5.2 now, so you've provided some hope. Stefan Sievert stefansievert@HOTo: [EMAIL PROTECTED] TMAIL.COM cc: Sent by: MQSeriesSubject: Re: Expiry Overhead List MQSERIES@AKH-WIE N.AC.AT 11/26/2002 05:17 PM Please respond to MQSeries List Jim, I think I remember reading in the announcment of MQ V5.3 that you no longer have to write these scavenger routines to get rid of expired messages, but that the queue manager itself will take care of that. Which version of MQ are you currently on? I don't know if my memory serves me well, but if this processing is causing you trouble because of massive log activity you might see some remedy after your next upgrade. Surely doesn't help much right now, though. Can somebody confirm? Cheers, Stefan From: Jim Ford [EMAIL PROTECTED] Reply-To: MQSeries List [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Expiry Overhead Date: Tue, 26 Nov 2002 13:39:32 -0600 We've had some performance problems in a synchronous app lately. It's a VB-NT client connected to a Solaris concentrator. Messages are put to a queues that go thru WMQI on Solaris. Flows convert XML to Cobol, and forward them to the mainframe. CICS programs do some DB2 work, and send replies that follow the same path in the opposite direction. We've noticed that these round trips slow way, way down at noon and 4:00 PM on Monday, Thursday and Friday. But we don't see this on Tuesday and Wednesday. After much research, we found a Perl script that runs on the WMQI server every day at noon and 4:00. The script goes through a queue and does GETs for a length of 0. This queue is an output only, archive queue, with messages that expire 3 days after they're put. The purpose of the script is to allow messages older than three days to expire. Our theory is that the overhead of expiration is noticeably slowing down WMQI. The reason Tuesday and Wednesday are OK is because far fewer messages expire on those days (since the only messages eligible to expire were from the previous weekend). We turned this script off, and will know on Friday if this is the problem. Our WMQI server is a dual processor machine loaded with memory. It's very surprising to me this script could hurt throughput this much, but it certainly looks that way. Has anyone seen this? Does IBM have any info as to what sort of overhead is involved in expiring messages? Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive _ The new MSN 8: smart spam protection and 2 months FREE* http://join.msn.com/?page=features/junkmail Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
Re: Expiry Overhead
Sorry to disappoint, but I'm fairly certain that this new feature is OS/390 version only. It was implement to clean up shared queues in the coupling facility. I don't think it was implemented in the distributed code base. - Steve Gies -Original Message- From: Jim Ford [mailto:[EMAIL PROTECTED]] Sent: Tuesday, November 26, 2002 3:50 PM To: [EMAIL PROTECTED] Subject: Re: Expiry Overhead I'm on 5.2 now, so you've provided some hope. Stefan Sievert stefansievert@HOTo: [EMAIL PROTECTED] TMAIL.COM cc: Sent by: MQSeriesSubject: Re: Expiry Overhead List MQSERIES@AKH-WIE N.AC.AT 11/26/2002 05:17 PM Please respond to MQSeries List Jim, I think I remember reading in the announcment of MQ V5.3 that you no longer have to write these scavenger routines to get rid of expired messages, but that the queue manager itself will take care of that. Which version of MQ are you currently on? I don't know if my memory serves me well, but if this processing is causing you trouble because of massive log activity you might see some remedy after your next upgrade. Surely doesn't help much right now, though. Can somebody confirm? Cheers, Stefan From: Jim Ford [EMAIL PROTECTED] Reply-To: MQSeries List [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Expiry Overhead Date: Tue, 26 Nov 2002 13:39:32 -0600 We've had some performance problems in a synchronous app lately. It's a VB-NT client connected to a Solaris concentrator. Messages are put to a queues that go thru WMQI on Solaris. Flows convert XML to Cobol, and forward them to the mainframe. CICS programs do some DB2 work, and send replies that follow the same path in the opposite direction. We've noticed that these round trips slow way, way down at noon and 4:00 PM on Monday, Thursday and Friday. But we don't see this on Tuesday and Wednesday. After much research, we found a Perl script that runs on the WMQI server every day at noon and 4:00. The script goes through a queue and does GETs for a length of 0. This queue is an output only, archive queue, with messages that expire 3 days after they're put. The purpose of the script is to allow messages older than three days to expire. Our theory is that the overhead of expiration is noticeably slowing down WMQI. The reason Tuesday and Wednesday are OK is because far fewer messages expire on those days (since the only messages eligible to expire were from the previous weekend). We turned this script off, and will know on Friday if this is the problem. Our WMQI server is a dual processor machine loaded with memory. It's very surprising to me this script could hurt throughput this much, but it certainly looks that way. Has anyone seen this? Does IBM have any info as to what sort of overhead is involved in expiring messages? Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive _ The new MSN 8: smart spam protection and 2 months FREE* http://join.msn.com/?page=features/junkmail Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
