Re: AIX large pages with TSM
Hi, my experience is : AIX 6.1, TSM 6.3.5.100, with 256 Virtual Drives 64 K is better size than 16MB and 4K. we are tested the three values and the best behavior is when we establish 64K size. Regards Fran, Good Luck and Thanks , Wanda. De: Rhodes, Richard L. rrho...@firstenergycorp.com Para: ADSM-L@VM.MARIST.EDU Enviado: Jueves 12 de marzo de 2015 12:18 Asunto: Re: AIX large pages with TSM Later versions of AIX have two larger page sizes available and so the number of locks to be taken and released drops dramatically for the typical 256KB tape transfer. This is what I'm really confused about, and our AIX admins don't have an answer. Our Power systems with AIX supports 4k, 64k and 16mb pages. The IBM presentation makes the case about minimizing lock contation b using 16mb non-pagable pages. But it also notes that using 64k pages can greatly minimize the contention. When I check svmon, most of our BSS heap segments are already using both small(4k) and medium(64k) pages (the sm in the listing below). svmon -P 10944554 | grep -i bss heap d811d8 12 work text data BSS heap sm 57518 0 0 57518 ab892b 13 work text data BSS heap sm 48006 0 0 48006 f8317a 14 work text data BSS heap sm 37204 0 0 37204 bc0f3c 11 work text data BSS heap sm 34200 0 0 34200 f50e75 10 clnt text data BSS heap, s 28667 0 - - 991d9b 15 work text data BSS heap sm 26213 0 0 26213 9e279c 16 work text data BSS heap sm 9023 0 0 9023 92d013 17 work text data BSS heap sm 3652 0 0 3652 I don't see any way to tell if the I/O is locking 4k pages, 64k pages, or a mix. The presentation gave 2 different LDR_CNTRL variables depending upon if you want 16mb pages or 64k pages. 16mb pages: LDR_CNTRL=TEXTPSIZE=64K@STACKPSIZE=64K@SHMPSIZE=64K@LARGE_PAGE_DATA=Y dsmserv 64k pages: LDR_CNTRL=TEXTPSIZE=64K@STACKPSIZE=64K@SHMPSIZE=64K@DATAPSIZE=64K dsmserv My assumption is that the 64k LDR_CNTRL will make all the BSS heaps use only 64k pages. I'm tempted to try this first since it doesn't require an AIX reboot, just a TSM shutdown/startup. Rick - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
Re: AIX large pages with TSM
Richard
I just edit a copy of the rc.dsmserv script. As I recall I had to
anyway as there was an error in it, something to do with setting the
home directory under AIX.
I just added the LDR_CONTROL variable to the line where dsmserv is
invoked.
On 19/03/2015 12:13 AM, Rhodes, Richard L. wrote:
I've been experimenting with LARGE PAGES. At this time I've just
been trying to get a TSM instance to come up with 64k pages. I'll
play with 16mb pages later.
This WORKS - manual startup after changing to the db2 owner.
Su - tsmuser (db2 owner)
LDR_CNTRL=TEXTPSIZE=64K@STACKPSIZE=64K@SHMPSIZE=64K@DATAPSIZE=64K
./dsmserv -i /tsmdata/tsmoc/config
This DOESN'T work - starting it via the rc.dsmserv script.
become root
LDR_CNTRL=TEXTPSIZE=64K@STACKPSIZE=64K@SHMPSIZE=64K@DATAPSIZE=64K
/opt/tivoli/tsm/server/bin/rc.dsmserv -u tsmuser -i
/tsmdata/${tsm}/config -q
I understand why - the rc.dsmserv script performs a su - to the
db2 instance owner. This creates a new login environment for the
instance owner, and thus doesn't pass the LDR_CNTRL env var. I
could use a custom script to start dsmserv, but I'm wondering how
you are starting it.
Q) How are you starting up dsmserv to enable LARGE PAGES?
Thanks
Rick
-
The information contained in this message is intended only for the
personal and confidential use of the recipient(s) named above. If
the reader of this message is not the intended recipient or an
agent responsible for delivering it to the intended recipient, you
are hereby notified that you have received this document in error
and that any review, dissemination, distribution, or copying of
this message is strictly prohibited. If you have received this
communication in error, please notify us immediately, and delete
the original message.
Re: AIX large pages with TSM
I've been experimenting with LARGE PAGES. At this time I've just been trying
to get a TSM instance to come up with 64k pages. I'll play with 16mb pages
later.
This WORKS - manual startup after changing to the db2 owner.
Su - tsmuser (db2 owner)
LDR_CNTRL=TEXTPSIZE=64K@STACKPSIZE=64K@SHMPSIZE=64K@DATAPSIZE=64K ./dsmserv -i
/tsmdata/tsmoc/config
This DOESN'T work - starting it via the rc.dsmserv script.
become root
LDR_CNTRL=TEXTPSIZE=64K@STACKPSIZE=64K@SHMPSIZE=64K@DATAPSIZE=64K
/opt/tivoli/tsm/server/bin/rc.dsmserv -u tsmuser -i /tsmdata/${tsm}/config -q
I understand why - the rc.dsmserv script performs a su - to the db2 instance
owner. This creates a new login environment for the instance owner, and thus
doesn't pass the LDR_CNTRL env var. I could use a custom script to start
dsmserv, but I'm wondering how you are starting it.
Q) How are you starting up dsmserv to enable LARGE PAGES?
Thanks
Rick
-
The information contained in this message is intended only for the personal and
confidential use of the recipient(s) named above. If the reader of this message
is not the intended recipient or an agent responsible for delivering it to the
intended recipient, you are hereby notified that you have received this
document in error and that any review, dissemination, distribution, or copying
of this message is strictly prohibited. If you have received this communication
in error, please notify us immediately, and delete the original message.
Re: AIX large pages with TSM
We have a script that runs out of init that boils down to:
export LDR_CNTRL=TEXTPSIZE=64K@STACKPSIZE=64K@SHMPSIZE=64K@LARGE_PAGE_DATA=Y
/opt/tivoli/tsm/server/bin/dsmserv -u instance id -i instance home dir
(note it runs dsmserv, not rc.dsmserv)
James.
-Original Message-
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of
Rhodes, Richard L.
Sent: 18 March 2015 13:14
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] AIX large pages with TSM
I've been experimenting with LARGE PAGES. At this time I've just been trying
to get a TSM instance to come up with 64k pages. I'll play with 16mb pages
later.
This WORKS - manual startup after changing to the db2 owner.
Su - tsmuser (db2 owner)
LDR_CNTRL=TEXTPSIZE=64K@STACKPSIZE=64K@SHMPSIZE=64K@DATAPSIZE=64K ./dsmserv -i
/tsmdata/tsmoc/config
This DOESN'T work - starting it via the rc.dsmserv script.
become root
LDR_CNTRL=TEXTPSIZE=64K@STACKPSIZE=64K@SHMPSIZE=64K@DATAPSIZE=64K
/opt/tivoli/tsm/server/bin/rc.dsmserv -u tsmuser -i /tsmdata/${tsm}/config -q
I understand why - the rc.dsmserv script performs a su - to the db2 instance
owner. This creates a new login environment for the instance owner, and thus
doesn't pass the LDR_CNTRL env var. I could use a custom script to start
dsmserv, but I'm wondering how you are starting it.
Q) How are you starting up dsmserv to enable LARGE PAGES?
Thanks
Rick
-
The information contained in this message is intended only for the personal and
confidential use of the recipient(s) named above. If the reader of this message
is not the intended recipient or an agent responsible for delivering it to the
intended recipient, you are hereby notified that you have received this
document in error and that any review, dissemination, distribution, or copying
of this message is strictly prohibited. If you have received this communication
in error, please notify us immediately, and delete the original message.
Re: AIX large pages with TSM
At least on x86, there's also a benefit in having smaller page tables. In particular, it makes the job of the TLB a lot easier for large memory systems. On 03/12/2015 03:59 AM, Steven Harris wrote: Hi Andrew The original reasoning on AIX was that as the data is transferred to tape AIX gets a spin lock on each 4k page and then has to release it, so that a system using tape heavily spends quite a lot of CPU acquiring and releasing locks. Later versions of AIX have two larger page sizes available and so the number of locks to be taken and released drops dramatically for the typical 256KB tape transfer. I'm not sure whether this reasoning applies to Linux. If it does then a larger page size might be beneficial there. Regards Steve Steven Harris TSM Admin Canberra Australia. On 12/03/2015 3:20 AM, Andrew Galloway wrote: We have 2 TSM 6.3.5 servers running on zLinux. Is this something we could leverage? __ Andrew Galloway Operational Systems Continuity Analyst SSC RCMP Technology Directorate CIO Building C1035 R.C.M.P. Headquarters 1200 Vanier Parkway Ottawa ON K1A 0R2 613.998.7707 andrew.gallo...@ssc-spc.gc.ca__ Rhodes, Richard L. rrho...@firstenergycorp.com 2015-03-11 7:15 AM I've been in meetings/training this week about AIX LPM and other stuff related with LPARS. We got talking about our TSM servers, and the IBM leader mentioned a presentation given a while ago at a IBM conference about using LARGE PAGES (16MB) with TSM to prevent lock contention. This is supposed to help if you drive some large number of tape drives concurrently. He mentioned one company that implemented this and saw a 20% increase in tape throughput. http://www-01.ibm.com/support/docview.wss?uid=swg21649823aid=1 I'm curious if anyone out there has heard of this and possibly implemented large pages for TSM. Thanks! Rick - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message. -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354
Re: AIX large pages with TSM
Later versions of AIX have two larger page sizes available and so the number of locks to be taken and released drops dramatically for the typical 256KB tape transfer. This is what I'm really confused about, and our AIX admins don't have an answer. Our Power systems with AIX supports 4k, 64k and 16mb pages. The IBM presentation makes the case about minimizing lock contation b using 16mb non-pagable pages. But it also notes that using 64k pages can greatly minimize the contention. When I check svmon, most of our BSS heap segments are already using both small(4k) and medium(64k) pages (the sm in the listing below). svmon -P 10944554 | grep -i bss heap d811d812 work text data BSS heap sm 57518 00 57518 ab892b13 work text data BSS heap sm 48006 00 48006 f8317a14 work text data BSS heap sm 37204 00 37204 bc0f3c11 work text data BSS heap sm 34200 00 34200 f50e7510 clnt text data BSS heap, s 28667 0- - 991d9b15 work text data BSS heap sm 26213 00 26213 9e279c16 work text data BSS heap sm 9023 00 9023 92d01317 work text data BSS heap sm 3652 00 3652 I don't see any way to tell if the I/O is locking 4k pages, 64k pages, or a mix. The presentation gave 2 different LDR_CNTRL variables depending upon if you want 16mb pages or 64k pages. 16mb pages: LDR_CNTRL=TEXTPSIZE=64K@STACKPSIZE=64K@SHMPSIZE=64K@LARGE_PAGE_DATA=Y dsmserv 64k pages: LDR_CNTRL=TEXTPSIZE=64K@STACKPSIZE=64K@SHMPSIZE=64K@DATAPSIZE=64K dsmserv My assumption is that the 64k LDR_CNTRL will make all the BSS heaps use only 64k pages. I'm tempted to try this first since it doesn't require an AIX reboot, just a TSM shutdown/startup. Rick - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
Re: AIX large pages with TSM
On Wed, Mar 11, 2015 at 12:20:42PM -0400, Andrew Galloway wrote: We have 2 TSM 6.3.5 servers running on zLinux. Is this something we could leverage? __ Andrew Galloway Operational Systems Continuity Analyst SSC RCMP Technology Directorate CIO Building C1035 R.C.M.P. Headquarters 1200 Vanier Parkway Ottawa ON K1A 0R2 613.998.7707 andrew.gallo...@ssc-spc.gc.ca__ Rhodes, Richard L. rrho...@firstenergycorp.com 2015-03-11 7:15 AM I've been in meetings/training this week about AIX LPM and other stuff related with LPARS. We got talking about our TSM servers, and the IBM leader mentioned a presentation given a while ago at a IBM conference about using LARGE PAGES (16MB) with TSM to prevent lock contention. This is supposed to help if you drive some large number of tape drives concurrently. He mentioned one company that implemented this and saw a 20% increase in tape throughput. http://www-01.ibm.com/support/docview.wss?uid=swg21649823aid=1 I'm curious if anyone out there has heard of this and possibly implemented large pages for TSM. Thanks! Rick - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message. Hello, we are using Large Pages with AIX 7.1.3 in TSM 7.1.1 Environment with 4 VIOServer 2.2.2.4 and 6 TSM Servers. . First - Performance is good - but: We see a problem of 100% System Wait on our Servers when Starting up TSM Server. Possible cause of this is a bug in combination using a Library Server and SAN Discovery. When the above comes together there is only one solution: - Rename/Move the SAN Discovery Binary - error pops up in TSM Activity Log (can't find Binary) - Rename/Move Back SAN Dicovery Binary After that everything went smooth and works as desired. A call to IBM TSm Support is still open. -- Han Solo: This is not going to work. Luke Skywalker: Why didn't you say so before? Han Solo: I did say so before!
Re: AIX large pages with TSM
Hi Andrew The original reasoning on AIX was that as the data is transferred to tape AIX gets a spin lock on each 4k page and then has to release it, so that a system using tape heavily spends quite a lot of CPU acquiring and releasing locks. Later versions of AIX have two larger page sizes available and so the number of locks to be taken and released drops dramatically for the typical 256KB tape transfer. I'm not sure whether this reasoning applies to Linux. If it does then a larger page size might be beneficial there. Regards Steve Steven Harris TSM Admin Canberra Australia. On 12/03/2015 3:20 AM, Andrew Galloway wrote: We have 2 TSM 6.3.5 servers running on zLinux. Is this something we could leverage? __ Andrew Galloway Operational Systems Continuity Analyst SSC RCMP Technology Directorate CIO Building C1035 R.C.M.P. Headquarters 1200 Vanier Parkway Ottawa ON K1A 0R2 613.998.7707 andrew.gallo...@ssc-spc.gc.ca__ Rhodes, Richard L. rrho...@firstenergycorp.com 2015-03-11 7:15 AM I've been in meetings/training this week about AIX LPM and other stuff related with LPARS. We got talking about our TSM servers, and the IBM leader mentioned a presentation given a while ago at a IBM conference about using LARGE PAGES (16MB) with TSM to prevent lock contention. This is supposed to help if you drive some large number of tape drives concurrently. He mentioned one company that implemented this and saw a 20% increase in tape throughput. http://www-01.ibm.com/support/docview.wss?uid=swg21649823aid=1 I'm curious if anyone out there has heard of this and possibly implemented large pages for TSM. Thanks! Rick - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
Re: AIX large pages with TSM
We have 2 TSM 6.3.5 servers running on zLinux. Is this something we could leverage? __ Andrew Galloway Operational Systems Continuity Analyst SSC RCMP Technology Directorate CIO Building C1035 R.C.M.P. Headquarters 1200 Vanier Parkway Ottawa ON K1A 0R2 613.998.7707 andrew.gallo...@ssc-spc.gc.ca__ Rhodes, Richard L. rrho...@firstenergycorp.com 2015-03-11 7:15 AM I've been in meetings/training this week about AIX LPM and other stuff related with LPARS. We got talking about our TSM servers, and the IBM leader mentioned a presentation given a while ago at a IBM conference about using LARGE PAGES (16MB) with TSM to prevent lock contention. This is supposed to help if you drive some large number of tape drives concurrently. He mentioned one company that implemented this and saw a 20% increase in tape throughput. http://www-01.ibm.com/support/docview.wss?uid=swg21649823aid=1 I'm curious if anyone out there has heard of this and possibly implemented large pages for TSM. Thanks! Rick - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
AIX large pages with TSM
I've been in meetings/training this week about AIX LPM and other stuff related with LPARS. We got talking about our TSM servers, and the IBM leader mentioned a presentation given a while ago at a IBM conference about using LARGE PAGES (16MB) with TSM to prevent lock contention. This is supposed to help if you drive some large number of tape drives concurrently. He mentioned one company that implemented this and saw a 20% increase in tape throughput. http://www-01.ibm.com/support/docview.wss?uid=swg21649823aid=1 I'm curious if anyone out there has heard of this and possibly implemented large pages for TSM. Thanks! Rick - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
Re: AIX large pages with TSM
We've been using them on RHEL x86_64 via Linux's anonhugepages mechanism for a while now. I'm not sure that they improve performance but they certainly haven't hurt anything that I can tell. On Wed, Mar 11, 2015 at 11:15:30AM +, Rhodes, Richard L. wrote: I've been in meetings/training this week about AIX LPM and other stuff related with LPARS. We got talking about our TSM servers, and the IBM leader mentioned a presentation given a while ago at a IBM conference about using LARGE PAGES (16MB) with TSM to prevent lock contention. This is supposed to help if you drive some large number of tape drives concurrently. He mentioned one company that implemented this and saw a 20% increase in tape throughput. http://www-01.ibm.com/support/docview.wss?uid=swg21649823aid=1 I'm curious if anyone out there has heard of this and possibly implemented large pages for TSM. Thanks! Rick -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine
Re: AIX large pages with TSM
Yep implemented this on my new v6 servers. Seems to work as advertised. They aren't CPU stressed enough yet to see any real difference though. Regards Steve Steven Harris TSM Admin, Canberra Australia. On 11/03/2015 10:15 PM, Rhodes, Richard L. wrote: I've been in meetings/training this week about AIX LPM and other stuff related with LPARS. We got talking about our TSM servers, and the IBM leader mentioned a presentation given a while ago at a IBM conference about using LARGE PAGES (16MB) with TSM to prevent lock contention. This is supposed to help if you drive some large number of tape drives concurrently. He mentioned one company that implemented this and saw a 20% increase in tape throughput. http://www-01.ibm.com/support/docview.wss?uid=swg21649823aid=1 I'm curious if anyone out there has heard of this and possibly implemented large pages for TSM. Thanks! Rick - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
Re: AIX large pages with TSM
We have implemented large pages on our v6 servers (now v7 servers). I agree it seems to work as suggested by the presentation. It requires some tuning to begin with to get the ideal number. It does mean that the memory reserved for large pages is not available for other processes so like so many things implementing it is a bit of a balancing act. If you make large pages mandatory then do not run out of large page space or you'll get out of memory errors. With it set to yes TSM will fall back to using smaller pages when large page space is exhausted. We think we saw a drive performance improvement but it was hard to measure, it certainly didn't make things worse. We can see that TSM is using the large pages. We have 38 TS1140 drives across two libraries. James. -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Steven Harris Sent: 11 March 2015 12:14 To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] AIX large pages with TSM Yep implemented this on my new v6 servers. Seems to work as advertised. They aren't CPU stressed enough yet to see any real difference though. Regards Steve Steven Harris TSM Admin, Canberra Australia. On 11/03/2015 10:15 PM, Rhodes, Richard L. wrote: I've been in meetings/training this week about AIX LPM and other stuff related with LPARS. We got talking about our TSM servers, and the IBM leader mentioned a presentation given a while ago at a IBM conference about using LARGE PAGES (16MB) with TSM to prevent lock contention. This is supposed to help if you drive some large number of tape drives concurrently. He mentioned one company that implemented this and saw a 20% increase in tape throughput. http://www-01.ibm.com/support/docview.wss?uid=swg21649823aid=1 I'm curious if anyone out there has heard of this and possibly implemented large pages for TSM. Thanks! Rick - The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
