Dan, Thanks for the pointers... I'm pretty certain it's just a case of nailing down the correct kernel parameters.
However, I couldn't get to your doc because the site requires cookies enabled, which our local security policy prevents. Could you email the doc directly to me? Thanks, tonyB. -----Original Message----- Sent: Monday 9/8/2003 2:23:24 PM Subject: Re: Large number of QMgrs on a Solaris box... >Antony, > >Here is some info that I put on the BMC support web site at >http://service.bmc.com/bmc_mq/kb/view_article/0,,4960%2B4977%2B4990%2B4991%2 >B10423,00.html > >Overview > >FDCs are found in /var/mqm/errors. Also the error logs are there. >AMQERR01.LOG is the current error log. The FDC filenames look like this: >AMQ10450.0.FDC. Where 10450 is the pid of the process throwing the FDC. >This resolution describes when tuning is needed due to a lack of semaphore >undo structures. The FDCs will have these characteristics: > >Probe Id: XY324192 >Probe Description :- AMQ6119: An internal MQSeries error has occurred ('22 >| - Invalid argument' from semop.) > >This problem is seen most frequently with Solaris. It is not common with >HP-UX and I have never seen the problem with AIX. > >Solution > >Recommendation: >1. For Solaris, increase the value of SEMUME and SEMMNU in /etc/system and >boot. > >2. Make sure you do not delete (ipcrm) MQSeries semaphores while MQSeries is >running. This is not likely to be the problem but I have seen it. >====== >DETAILS: >The following are details on how to come to the conclusion above. >At a Solaris computer type: > >man semop > >And see: > > EINVAL The semid argument is not a valid semaphore iden- > tifier, or the number of individual semaphores for > which the calling process requests a SEM_UNDO > would exceed the limit. > >The EINVAL is E = error and INVAL = INVALID. > >So the FDC is saying that the semop call returned EINVAL. Looks like the >number of SEM_UNDOs being requested is too large for the current kernel. >Looks like you need to increase the number of SEM_UNDOs allowed. > >Looking at >http://www.sun.com/sun-on-net/itworld/UIR960101perf.html we see that the >kernel parm to increase is semsys:seminfo_semume. The kernel parms can be >increased by changing /etc/system and then booting. How much should you >increase it? I usually double it until I get to 4096 and then increase by >smaller amounts. > >In the past I have seen increasing increasing just SEMUME helped with the >EINVAL problem. But, to be safe you may wish to increase both SEMUME and >SEMMNU. > >I usually double the values to start with. It is difficult to come up with >exact recommendations. However, here is some info that will help. See >http://www-3.ibm.com/software/ts/mqseries/txppacs/supportpacs/mp00.pdf >Page 60 of 71. Here is the quote: >"SEMMNU , semaphore undo's was increased from 256 to 2048. Without this >only approx 90 clients could be MQCONNected." >So based on that I would say that in addition to SEMUME (number of semaphore >undos per process), also we have SEMNU (total number of undos in the >system). It sounds like SEMMNU=256 handles approximately 90 MQCONNs. You >could extrapolate from there. > >ANOTHER POSSIBILITY: Notice that the error explanation for EINVAL includes >two possibilities. This article describes how to fix one possible problem. >The other possibility is that the semaphore identifier being used by >MQSeries is invalid. This probably means that something or someone has >deleted the semaphore. Please examine for this, are there any jobs that use >ipcrm to delete, to clean up, semaphores? It is ok to delete semaphores >that are not MQSeries semaphores or to cleanup MQSeries semaphores while >MQSeries is down. But if you delete (ipcrm) semaphores while MQSeries is >running this error will occur. > >Dan Egner >IBM WebSphere MQ V5.3 System Administration Certified >Product Support >BMC Software, Inc > >-----Original Message----- >From: Antony Boggis [mailto:[EMAIL PROTECTED] >Sent: Monday, September 08, 2003 2:02 PM >To: [EMAIL PROTECTED] >Subject: Large number of QMgrs on a Solaris box... > > >I am in the midst of troubleshooting an issue where I am getting huge >numbers of FDC files generated on one of our development Solaris machines. >Some of these FDC files are are REALLY large too. > >The machine in question is an 8 CPU box with 32GB RAM. > >There are currently 70+ queue managers *DEFINED* on the box, many in a >"local" cluster (ie the CLUSTER of queue managers is all on the same box - >this is a DEVELOPMENT machine... in the real world the cluster members would >be on different physical machines). > >At the moment I am showing 27 active queue managers. > >The kernel parameters have been updated to the followiung values: > >set shmsys:shminfo_shmmax = 4294967295 >set shmsys:shminfo_shmseg = 2048 >set shmsys:shminfo_shmmni = 2048 >set semsys:seminfo_semaem = 16384 >set semsys:seminfo_semmni = 1024 >set semsys:seminfo_semmap = 1026 >set semsys:seminfo_semmns = 16384 >set semsys:seminfo_semmsl = 10000 >set semsys:seminfo_semopm = 100 >set semsys:seminfo_semmnu = 2048 >set semsys:seminfo_semume = 256 >set msgsys:msginfo_msgmap = 1026 >set msgsys:msginfo_msgmax = 4096 > >In addition to the FDC files in /var/mqm/errors, the AMQERROR0x files are >also filled with: > >AMQ6119: An internal MQSeries error has occurred ('22 - Invalid argument' >from semop.) > >If anyone can shed some more light on the problem, I'd apreciate it. I am in >the process of reviewing the solaris kernel params to see if that (likely) >is the root of the problem. > >Regards, > >tonyB. > >Instructions for managing your mailing list subscription are provided in >the Listserv General Users Guide available at http://www.lsoft.com >Archive: http://vm.akh-wien.ac.at/MQSeries.archive > >Instructions for managing your mailing list subscription are provided in >the Listserv General Users Guide available at http://www.lsoft.com >Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
