Hi Stefan, At the time the servers were installed Redhat 6.4 was our standard. The servers need to be upgraded to the latest 6.x release first before we can upgrade TSM to 8.1.x. But before making any changes to the environment while we have a case open at IBM, I need to be sure there is a clear reason to do so. If there is any indication that the issue is related to the OS level or the TSM level, I will plan an upgrade of course. Another reason is that the server which is having the most issues is running a 7.1 release prior to new security 'enhancements'. I would like to prevent extra issues caused by certificates until after we found and fixed the performance issues. Today we have a meeting with IBM and all departments and managers to plan an onsite workshop on how we/IBM will tackle the issue. Finally...
Kind regards, Eric van Loon Air France/KLM Storage & Backup -----Original Message----- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Stefan Folkerts Sent: dinsdag 3 september 2019 09:41 To: ADSM-L@VM.MARIST.EDU Subject: Re: TSM server performance continuing Very strange issue indeed Eric, I have never heard of a specific client causing slowdowns, I would understand if it where a massive amount of oracle sessions starting but if it's really platform and not session count related it would have to be something that IBM needs to look into I guess. May I ask, is there a specific reason you are running fairly old server code in the environment? On Tue, Aug 27, 2019 at 1:01 PM Loon, Eric van (ITOP NS) - KLM < eric-van.l...@klm.com> wrote: > Hi Stefan, > > There is no noticable impact on the TSM performance when we use the > tsmdiskperf tool on the database volumes. > We recently discovered the issue seems to be related to the TDP for > Oracle clients. When none are running the server response is OK, even > during a reasonable load. But as soon as the TDP for Oracle clients > start kicking in, the server becomes slow. We brought it to the > attention of the developers, but no response yet... > Thanks for your help! > > Kind regards, > Eric van Loon > Air France/KLM Storage & Backup > > -----Original Message----- > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf > Of Stefan Folkerts > Sent: maandag 26 augustus 2019 10:39 > To: ADSM-L@VM.MARIST.EDU > Subject: Re: TSM server performance continuing > > Eric, > > What happens when you benchmark the DB volumes using the tool provided > with the blueprints on an idle system, does the system also slow down > with commands such as q stgpool or does it stay fast when the > benchmark is running on all volumes? > Also, what kind of a result does the benchmark give you and what do > you use for your database storage? > > Regards, > Stefan > > > > On Thu, Aug 22, 2019 at 3:36 PM PAC Brion Arnaud < > arnaud.br...@panalpina.com> > wrote: > > > Hi Eric, > > > > These seem to be default values when installing Red Hat Enterprise > > Linux Server 7.3. > > I checked several other machines, all are having the same value. > > > > And following page > > https://stackoverflow.com/questions/55428812/how-the-values-of-kerne > > l- > > parameters-are-defined > > seems to confirm my statement. > > > > Cheers. > > > > Arnaud > > > > > > > > ******************************************************************** > > ** > > ********************************************************** > > Backup and Recovery Systems Administrator Panalpina Management Ltd., > > Basle, Switzerland, CIT Department Viadukstrasse 42, P.O. Box 4002 > > Basel/CH > > Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01 > > Direct: +41 (61) 226 19 78 > > e-mail: arnaud.br...@panalpina.com > > This electronic message transmission contains information from > > Panalpina and is confidential or privileged. > > This information is intended only for the person (s) named above. If > > you are not the intended recipient, any disclosure, copying, > > distribution or use or any other action based on the contents of > > this information is strictly prohibited. > > > > If you receive this electronic transmission in error, please notify > > the sender by e-mail, telephone or fax at the numbers listed above. > Thank you. > > > > ******************************************************************** > > ** > > ********************************************************** > > www.panalpina.com > > > > > > > > -----Original Message----- > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On > > Behalf Of Loon, Eric van (ITOP NS) - KLM > > Sent: Thursday, August 22, 2019 2:31 PM > > To: ADSM-L@VM.MARIST.EDU > > Subject: Re: TSM server performance continuing > > > > Hi Arnoud, > > > > I do not understand why your max seg size (kbytes) = > > 18014398509465599 and max total shared memory (kbytes) = 18014397435740096 > > are so huge. > > If you have 256 GB RAM, like I do, I would expect max seg size > > (kbytes) = > > 268435456 and max total shared memory (kbytes) = 268435456. > > Thanks for your help! > > > > Kind regards, > > Eric van Loon > > Air France/KLM Storage & Backup > > > > > > -----Original Message----- > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On > > Behalf Of PAC Brion Arnaud > > Sent: donderdag 22 augustus 2019 09:02 > > To: ADSM-L@VM.MARIST.EDU > > Subject: Re: TSM server performance continuing > > > > Hi Eric, > > > > Running 4 TSM servers at 8.1.6.1 (PowerLinux servers with 256 GB > > RAM), and making use of directory-container storage pools only. > > No server performance or responsiveness issues here anymore, since > > we changed our storage to make use of IBM Isilon. > > > > Here my values : > > > > vmstat 1 > > procs -----------memory---------- ---swap-- -----io---- -system-- > > ------cpu----- > > r b swpd free buff cache si so bi bo in cs us sy > id > > wa st > > 13 0 1420032 1152064 1429248 243848064 0 0 7500 3766 0 0 > > 24 3 61 11 0 > > 13 1 1420032 1046016 1428288 243943488 0 0 614256 80 22427 > 42342 > > 50 4 39 7 0 > > 12 0 1420032 1144640 1426688 243861952 0 0 542840 0 19109 > 33340 > > 49 2 43 6 0 > > 12 4 1420032 883904 1427136 244133824 0 0 577108 0 19777 34954 > > 49 2 44 6 0 > > 12 1 1420032 936576 1425344 244072768 0 0 613152 372 21110 37559 > > 49 2 43 6 0 > > 11 5 1420032 1028096 1425600 243977344 0 0 549024 0 24599 > 47296 > > 48 2 42 8 0 > > 12 0 1420032 1028032 1424832 243983424 0 0 566180 16 21693 > 39976 > > 49 2 43 6 0 > > > > > > ipcs -l > > > > ------ Messages Limits -------- > > max queues system wide = 250880 > > max size of message (bytes) = 65536 > > default max size of queue (bytes) = 65536 > > > > ------ Shared Memory Limits -------- max number of segments = 62720 > > max seg size (kbytes) = 18014398509465599 max total shared memory > > (kbytes) = 18014397435740096 min seg size (bytes) = 1 > > > > ------ Semaphore Limits -------- > > max number of arrays = 62720 > > max semaphores per array = 250 > > max semaphores system wide = 256000 > > max ops per semop call = 32 > > semaphore max value = 32767 > > > > > > Specific tuning done : disabling CPU virtualization (led to sever > > hangs when active) > > > > ppc64_cpu --smt=off > > > > > > Cheers. > > > > Arnaud > > > > > > ******************************************************************** > > ** > > ********************************************************** > > Backup and Recovery Systems Administrator Panalpina Management Ltd., > > Basle, Switzerland, CIT Department Viadukstrasse 42, P.O. Box 4002 > > Basel/CH > > Phone: +41 (61) 226 11 11, FAX: +41 (61) 226 17 01 > > Direct: +41 (61) 226 19 78 > > e-mail: arnaud.br...@panalpina.com > > This electronic message transmission contains information from > > Panalpina and is confidential or privileged. > > This information is intended only for the person (s) named above. If > > you are not the intended recipient, any disclosure, copying, > > distribution or use or any other action based on the contents of > > this information is strictly prohibited. > > > > If you receive this electronic transmission in error, please notify > > the sender by e-mail, telephone or fax at the numbers listed above. > Thank you. > > > > ******************************************************************** > > ** > > ********************************************************** > > www.panalpina.com > > > > > > -----Original Message----- > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On > > Behalf Of Loon, Eric van (ITOP NS) - KLM > > Sent: Wednesday, August 21, 2019 5:39 PM > > To: ADSM-L@VM.MARIST.EDU > > Subject: TSM server performance continuing > > > > Hi guys, > > > > A few weeks ago I already wrote about the severe performance issues > > we have with our TSM 7.1 servers. In the 'old days' we used to back > > up our clients to TSM 6.3 servers with Data Domains attached. > > Smaller clients backed up through the LAN, large ones through the SAN. > > Our newer servers use LAN-only with directory containers and the > > performance of these servers really sucks. Setting up a session > > takes sometimes almost one minute and a q stg also takes 30 to 50 > > seconds. I noticed that performance is OK when there are no TDP for > > Oracle sessions running, but as soon as they are started the > > performance starts to drop drastically. > > We are really lost on where to look for the cause. I sent numerous > > logs and traces to IBM, but I guess they are out of ideas too since > > I don't hear anything back from them lately. The only thing is that > > supports notices delays in DB2, but they don't know why... > > What I noticed on my TSM server is that as soon as there is a load > > on the server, the blocked queue starts to rise. I would like to > > know if that's something to focus on or not. > > Can some of you please run the "vmstat 1" command on their Linux > > server (preferably one with directory containers too) and let me > > know if you too see values other than 0 in the B column? > > Thank you very much for your help in advance! > > > > Kind regards, > > Eric van Loon > > Air France/KLM Storage & Backup > > ******************************************************** > > For information, services and offers, please visit our web site: > > http://www.klm.com. This e-mail and any attachment may contain > > confidential and privileged material intended for the addressee only. > > If you are not the addressee, you are notified that no part of the > > e-mail or any attachment may be disclosed, copied or distributed, > > and that any other action related to this e-mail or attachment is > > strictly prohibited, and may be unlawful. If you have received this > > e-mail by error, please notify the sender immediately by return > > e-mail, and delete > this message. > > > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries > > and/or its employees shall not be liable for the incorrect or > > incomplete transmission of this e-mail or any attachments, nor > > responsible for any > delay in receipt. > > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal > > Dutch > > Airlines) is registered in Amstelveen, The Netherlands, with > > registered number 33014286 > > ******************************************************** > > ******************************************************** > > For information, services and offers, please visit our web site: > > http://www.klm.com. This e-mail and any attachment may contain > > confidential and privileged material intended for the addressee only. > > If you are not the addressee, you are notified that no part of the > > e-mail or any attachment may be disclosed, copied or distributed, > > and that any other action related to this e-mail or attachment is > > strictly prohibited, and may be unlawful. If you have received this > > e-mail by error, please notify the sender immediately by return > > e-mail, and delete > this message. > > > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries > > and/or its employees shall not be liable for the incorrect or > > incomplete transmission of this e-mail or any attachments, nor > > responsible for any > delay in receipt. > > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal > > Dutch > > Airlines) is registered in Amstelveen, The Netherlands, with > > registered number 33014286 > > ******************************************************** > > > ******************************************************** > For information, services and offers, please visit our web site: > http://www.klm.com. This e-mail and any attachment may contain > confidential and privileged material intended for the addressee only. > If you are not the addressee, you are notified that no part of the > e-mail or any attachment may be disclosed, copied or distributed, and > that any other action related to this e-mail or attachment is strictly > prohibited, and may be unlawful. If you have received this e-mail by > error, please notify the sender immediately by return e-mail, and delete this > message. > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or > its employees shall not be liable for the incorrect or incomplete > transmission of this e-mail or any attachments, nor responsible for any delay > in receipt. > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal > Dutch > Airlines) is registered in Amstelveen, The Netherlands, with > registered number 33014286 > ******************************************************** > ******************************************************** For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message. Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt. Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286 ********************************************************
