Zoltan, I need to look at the complete output from a "show threads" command. Rather than filling up the listserv with stuff, send it to my work ID. ( ddca...@us.ibm.com). We need to figure out what is hanging up the import from canceling.
Dave Canan IBM ATS TSM Performance ddcananATUSDOTIBMDOTCOM 916-723-2410 > On Mon, May 2, 2011 at 12:01 PM, Zoltan Forray <zforray....@gmail.com>wrote: > Unless you have some kinda "force" command, it will not cancel. Tried > numerous times. A "q sess" shows this. The numbers are misleading since > at > one point it was so large (remember, this has been running/hung since > 03/07/2011) it has now rolled-over and started again. > > 2:53:38 PM SUN : q sess > Sess Comm. Sess Wait Bytes Bytes Sess Platform Client Name > > Number Method State Time Sent Recvd Type > ------- ------ ------ ------ ------- ------- ----- -------- > ------------------- > 55,704 Tcp/Ip Run 0 S 461.4 K 4.5 T Admin Linux/x- ZFORRAY > > 86_64 > > 55,705 Memory IdleW 111.1 883.8 K 4.9 M Admin Server ZFORRAY > > IPC H > > > 2:55:57 PM SUN : q sess 55705 f=d > Sess Number: 55,705 > Comm. Method: Memory IPC > Sess State: IdleW > Wait Time: 111.1 H > Bytes Sent: 883.8 K > Bytes Recvd: 4.9 M > Sess Type: Admin > Platform: Server > Client Name: ZFORRAY > Media Access Status: > User Name: > Date/Time First Data Sent: > Proxy By Storage Agent: > Actions: > > Yes it is currently happening. Log usage is up to 50% of 60GB. We run DB > backups daily. It has been slowly creeping up. > > 2:58:00 PM SUN : q log f=d > Total Space(MB): 65,536 > Used Space(MB): 34,224 > Free Space(MB): 30,992 > Active Log Directory: /tsmlog > Archive Log Directory: /tsmarchlog > Mirror Log Directory: > Archive Failover Log Directory: > > > > On Mon, May 2, 2011 at 2:48 PM, Dave Canan <ddca...@gmail.com> wrote: > > > So from this, it looks like session # 55705 would be the one we would > want > > to cancel in order to hopefully clear this out. So is the problem > currently > > happening? If it is, the next question is whether or not this session is > > going to cancel. Let me know what happens. If it won't cancel let me > know. > > > > > > Dave Canan > > IBM ATS TSM Performance > > ddcananATUSDOTIBMDOTCOM > > 916-723-2410 > > > > On Mon, May 2, 2011 at 10:27 AM, Zoltan Forray <zforray....@gmail.com > > >wrote: > > > > > Aha - that is why there was so much info. Here is just that piece plus > > the > > > thread for the "parent" and his "parent" > > > > > > Thread 94, Parent 1: AcceptorThread, Storage 77938528, AllocCnt 420814 > > > HighWaterAmt 77938528 > > > tid=1116109120, ptid=1971136576, det=1, zomb=0, join=0, result=0, > sess=0 > > > > > > Thread 137116, Parent 94: psSessionThread, Storage 41352, AllocCnt > > 2464414 > > > HighWaterAmt 1424944 > > > tid=1203480896, ptid=1116109120, det=1, zomb=0, join=0, result=0, > > > sess=55704 > > > > > > Thread 137117, Parent 137116: XiSessionThread, Storage 406459533, > > AllocCnt > > > 204215 HighWaterAmt 2147307969 > > > tid=1237166400, ptid=1203480896, det=1, zomb=0, join=0, result=0, > > > sess=55705 > > > Awaiting cond xiMaster->cSent (0x0x2b787908e870), using mutex > > > xiMaster->mutex > > > (0x0xff84ae8), at xicomm.c(551) > > > > > > On Mon, May 2, 2011 at 1:14 PM, Dave Canan <ddca...@gmail.com> wrote: > > > > > > > There are no parameters on the "show threads" command; that is the > > > complete > > > > command. When you issue that command, do you see any lines that start > > > with > > > > "Thread 137117" ? That will be the output we are looking for. . Then > > > look > > > > for the "sess=" part of the output from that thread. That is the > > session > > > we > > > > are looking for. > > > > > > > > > > > > Dave Canan > > > > IBM ATS TSM Performance > > > > ddcananATUSDOTIBMDOTCOM > > > > 916-723-2410 > > > > > > > > > > > > > On Mon, May 2, 2011 at 9:50 AM, Zoltan Forray <zforray....@gmail.com > > > > > > wrote: > > > > > > > > > Dave, > > > > > > > > > > Thanks for the response. > > > > > > > > > > As I suspect, the oldest thread is from 03/07/2011. I have > 2-IMPORT > > > > > processes that hung (one started 03/07/2011 the other 04/19/2011) > > > since > > > > > the > > > > > exporting server had a tape-mount problem and the exports died. I > > have > > > > > long > > > > > since tried to cancel the imports, to no avail. A "q proc" just > > shows > > > > > "cancel in process". > > > > > > > > > > I wasn't sure of the proper syntax for the "show threads" command > and > > > > just > > > > > used the ThreadID and used it. In this case: > > > > > > > > > > Start ThreadId=137117, Timestamp=03/07/2011 12:35:37 PM, > > > > > (the oldest) > > > > > > > > > > and did a "show threads 137117" > > > > > > > > > > to get: > > > > > > > > > > ------------------------------------------------- > > > > > 12:34:20 PM SUN : show threads 137117 > > > > > Server PID: 8520 > > > > > Active threads: 126 > > > > > Zombie threads: 0 > > > > > Cached descriptors: 6 > > > > > Thread 1, Parent 0: main, Storage 6203696, AllocCnt 8342 > HighWaterAmt > > > > > 6203696 > > > > > tid=1971136576, ptid=<none>, det=0, zomb=0, join=0, result=0, > sess=0 > > > > > Thread 2, Parent 1: TmDeadlockDetector, Storage 0, AllocCnt 0 > > > > HighWaterAmt > > > > > 0 > > > > > tid=1086736704, ptid=1971136576, det=1, zomb=0, join=0, result=0, > > > sess=0 > > > > > Thread 3, Parent 1: TbCacheMonitorThread, Storage 0, AllocCnt > 2081959 > > > > > HighWaterAmt 0 > > > > > tid=1099311424, ptid=1971136576, det=0, zomb=0, join=0, result=0, > > > sess=0 > > > > > Thread 4, Parent 1: AdmActivityLogThread, Storage 0, AllocCnt > > 20132093 > > > > > HighWaterAmt 114297 > > > > > tid=1105582400, ptid=1971136576, det=1, zomb=0, join=0, result=0, > > > sess=0 > > > > > Awaiting cond descP->outReady (0x0x2b7879076ee0), using mutex > > > > OUTV->mutex > > > > > (0x0xb5de3e8), at output.c(3292) > > > > > Thread 13, Parent 1: NaDiscoverThread, Storage 733, AllocCnt 14 > > > > > HighWaterAmt > > > > > 37761 > > > > > tid=1100364096, ptid=1971136576, det=1, zomb=0, join=0, result=0, > > > sess=0 > > > > > Awaiting cond NAV->checkConfig (0x0x2b78790770a0), using mutex > > > > NAV->mutex > > > > > (0x0xb7be7b8), at nadiscvr.c(2137) > > > > > Thread 14, Parent 1: RemAgent, Storage 0, AllocCnt 0 HighWaterAmt 0 > > > > > tid=1102469440, ptid=1971136576, det=1, zomb=0, join=0, result=0, > > > sess=0 > > > > > Awaiting cond MMSV->remCond (0x0x2b7879077180), using mutex > > > MMSV->mutex > > > > > (0x0x2aaab0009638), at mmsop.c(1409) > > > > > Thread 15, Parent 1: PollAgent, Storage 0, AllocCnt 0 HighWaterAmt > 0 > > > > > tid=1101416768, ptid=1971136576, det=1, zomb=0, join=0, result=0, > > > sess=0 > > > > > > > > > > (lot of snippage to not anger the listserv > gods....................) > > > > > > > > > > > > > > > Awaiting cond descP->outReady (0x0x2b78790900f0), using mutex > > > > OUTV->mutex > > > > > (0x0xb5de3e8), at output.c(3292) > > > > > Thread 517131, Parent 517130: SmAdminCommandThread, Storage 253205, > > > > > AllocCnt > > > > > 4574 HighWaterAmt 321609 > > > > > tid=1120319808, ptid=1201375552, det=0, zomb=0, join=0, result=0, > > > sess=0 > > > > > Holding mutex THRV->mutex (0x0xb5ea958), acquired at > > pkthread.c(2397) > > > > > Server PID: 8520 > > > > > Active threads: 126 > > > > > Zombie threads: 0 > > > > > Cached descriptors: 6 > > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > > > > > > I know there are fixes to import in 6.2.2.2 so I am going straight > to > > > > > 6.2.2.30 > > > > > > > > > > On Mon, May 2, 2011 at 12:26 PM, Dave Canan <ddca...@gmail.com> > > wrote: > > > > > > > > > > > Zoltan, there is no "show logpinned" command in V6.2.2, but it is > > > still > > > > > > possible to figure out which session and transaction is causing > the > > > log > > > > > to > > > > > > be pinned. Here is an example of what you would need to do: > > > > > > This isn't quite as straight-forward as the old show command, but > > > this > > > > > will > > > > > > work. Call or re-post if you have questions. > > > > > > > > > > > > > > > > > > 1. You need to start with doing a "show txnt" command. Here is an > > > > example > > > > > > of > > > > > > one: > > > > > > > > > > > > > > > > > > Session established with server xxxxxxx: AIX > > > > > > Server Version 6, Release 2, Level 2.0 > > > > > > Server date/time: 05/02/11 12:09:43 Last access: 05/02/11 > > > > 12:09:30 > > > > > > > > > > > > > > > > > > tsm: xxxxxxx>show txnt > > > > > > > > > > > > Transaction hash table contents (slots=256): > > > > > > slot -> 38: > > > > > > Tsn=0:688678, Resurrected=False, InFlight=True, > Distributed=False, > > > > > > Persistent=True, Addr 113001ab8 > > > > > > Start ThreadId=17986, Timestamp=05/02/11 12:10:06, > > > > > Creator=imdmgr.c(3552) > > > > > > Last known in use by ThreadId=17986 > > > > > > Participants=1, summaryVote=ReadOnly > > > > > > EndInFlight False, endThreadId 0, tmidx 0 0, processBatchCount > 0, > > > > > > mustAbort > > > > > > False. > > > > > > Participant DB: voteReceived=False, ackReceived=False > > > > > > DB: Txn 11295a158, ReadOnly(YES), connP=1128ab1f8, > > > > applHandle=23208, > > > > > > openTbls=2: > > > > > > DB: --> OpenP=115facf38 for table=Management.Classes. > > > > > > DB: --> OpenP=116eba978 for table=Management.Class.Ids. > > > > > > DB: --> RegSqlId=0x03000000 SELECT for table=Backup.Objects, > > > > > > executed(Yes). > > > > > > slot -> 39: > > > > > > Tsn=0:688679, Resurrected=False, InFlight=True, > Distributed=False, > > > > > > Persistent=True, Addr 115646a78 > > > > > > Start ThreadId=17986, Timestamp=05/02/11 12:10:06, > > > > > Creator=imdmgr.c(3673) > > > > > > Last known in use by ThreadId=17986 > > > > > > Participants=2, summaryVote=ReadOnly > > > > > > EndInFlight False, endThreadId 0, tmidx 0 0, processBatchCount > 0, > > > > > > mustAbort > > > > > > False. > > > > > > Participant DB: voteReceived=False, ackReceived=False > > > > > > DB: Txn 112f855b8, ReadOnly(NO), connP=112ef88f8, > > > > applHandle=23205, > > > > > > openTbls=24: > > > > > > DB: --> OpenP=1147eded8 for table=Activity.Summary. > > > > > > DB: --> OpenP=118b7e038 for table=Group.Objects.ByLeader. > > > > > > DB: --> OpenP=112fbdcf8 for table=Backup.Objects. > > > > > > DB: --> OpenP=11300d3d8 for table=Group.Leaders. > > > > > > DB: --> OpenP=11730b3b8 for table=AS.Volume.Status. > > > > > > DB: --> OpenP=11307a7b8 for table=AF.Vol.Clusters. > > > > > > DB: --> OpenP=11743acd8 for table=AF.Clusters. > > > > > > DB: --> OpenP=117335198 for table=BF.Dereferenced.Chunks. > > > > > > DB: --> OpenP=112fa2bb8 for table=DF.Segments. > > > > > > DB: --> OpenP=113084658 for table=AS.Segments. > > > > > > DB: --> OpenP=1160b65b8 for table=AF.Damaged. > > > > > > DB: --> OpenP=112fe8b78 for table=BF.Bitfile.Extents. > > > > > > DB: --> OpenP=115641438 for table=AF.Segments. > > > > > > DB: --> OpenP=115f7d578 for table=AF.Bitfiles. > > > > > > DB: --> OpenP=114536ad8 for table=DF.Bitfiles. > > > > > > DB: --> OpenP=11303ff78 for table=BF.Aggregate.Attributes. > > > > > > DB: --> OpenP=118c62eb8 for table=BF.Aggregated.Bitfiles. > > > > > > DB: --> OpenP=1170581d8 for table=ARCH_EXPDIRS. > > > > > > DB: --> OpenP=1147ec498 for table=Archive.Objects. > > > > > > DB: --> OpenP=11604ddb8 for table=Management.Class.Ids. > > > > > > DB: --> OpenP=116e91c98 for table=Management.Classes. > > > > > > DB: --> OpenP=113034e98 for table=Filespaces. > > > > > > DB: --> OpenP=1144d3e38 for table=Nodes. > > > > > > DB: --> OpenP=114d1ce58 for table=Policy.Domain.Members. > > > > > > Participant IM: voteReceived=False, ackReceived=False > > > > > > Locks held by Tsn=0:688679 : > > > > > > Type=19002(im node filespace), NameSpace=0, SummMode=xLock, > > > > > Mode=xLock, > > > > > > Key='154.1' > > > > > > Type=19001(im node), NameSpace=0, SummMode=ixLock, > Mode=ixLock, > > > > > > Key='154' > > > > > > Type=19000(im universe), NameSpace=0, SummMode=ixLock, > > > Mode=ixLock, > > > > > > Key='' > > > > > > slot -> 165: > > > > > > Tsn=0:692901, Resurrected=False, InFlight=True, > Distributed=False, > > > > > > Persistent=True, Addr 114532178 > > > > > > Start ThreadId=17984, Timestamp=05/02/11 12:10:07, > > > > > Creator=imdmgr.c(2460) > > > > > > Last known in use by ThreadId=17984 > > > > > > Participants=1, summaryVote=ReadOnly > > > > > > EndInFlight False, endThreadId 0, tmidx 0 0, processBatchCount > 0, > > > > > > mustAbort > > > > > > False. > > > > > > > > > > > > 2. What you're going to want to do is look at the lines that > state > > > > "Start > > > > > > Threadid=nnnnn, Timestamp=". You need to find the oldest > Timestamp. > > > > That > > > > > > will also give you the threadid. From there you do a "show > threads" > > > > > > command. > > > > > > Here is a piece from that command : > > > > > > > > > > > > Thread 17980, Parent 42: psSessionThread, Storage 0, > AllocCnt > > > 490 > > > > > > HighWaterAmt > > > > > > 174080 > > > > > > tid=253c, ptid=1e2a, det=1, zomb=0, join=0, result=0, sess=4612 > > > > > > Holding mutex OUTV->mutex (0x112018018), acquired at > > output.c(3267) > > > > > > > > > > > > Stack trace: > > > > > > 0x0900000000595f30 _cond_wait_global > > > > > > 0x0900000000596abc _cond_wait > > > > > > 0x09000000005977ac pthread_cond_wait > > > > > > 0x0000000100008414 pkWaitConditionTracked > > > > > > 0x00000001000130f8 outGetNext > > > > > > 0x0000000100870e58 SmAdminOutput > > > > > > 0x000000010086ffb8 SmExecuteAdminCommand > > > > > > 0x000000010086f888 SmAdminSession > > > > > > 0x00000001003f123c DoAdminGeneral > > > > > > 0x00000001003ec9bc smExecuteSession > > > > > > 0x000000010009ad48 psSessionThread > > > > > > 0x000000010001b688 StartThread > > > > > > > > > > > > > > > > > > 3. Look for the thread Id for the longest running transaction in > > the > > > > > first > > > > > > step. This will also point you to the session matching that > thread. > > > > From > > > > > > there, the session (that is the longest running transaction), can > > be > > > > > > canceled. > > > > > > > > > > > > > > > > > > Dave Canan > > > > > > IBM ATS TSM Performance > > > > > > ddcananATUSDOTIBMDOTCOM > > > > > > 916-723-2410 > > > > > > > > > > > > > > > > > > On Fri, Apr 29, 2011 at 5:50 AM, Zoltan Forray < > > > zforray....@gmail.com > > > > > > >wrote: > > > > > > > > > > > > > Yes, I realize 6.x is DB2 but I was wondering is there is an > > > > equivalent > > > > > > to > > > > > > > "show logpinned". I seem to be in some kind of situation where > > the > > > > logs > > > > > > are > > > > > > > not being released and growing daily, eventhough I have > performed > > > > > > numerous > > > > > > > full DB backups and backup volhist. > > > > > > > > > > > > > > I am going to have to bounce it soon, if I can't kill whatever > is > > > > > holding > > > > > > > things up. > > > > > > > > > > > > > > My suspicion is a hung import. I have been moving numerous > nodes > > > > from > > > > > an > > > > > > > old 5.5 server and along the way at least 2-export/imports have > > > > crashed > > > > > > for > > > > > > > various reasons and left zombie imports on the 6.2.2 server > (yes > > I > > > > > tried > > > > > > > cancel proc with no results other than "cancel in progress") > > > > > > > > > > > > > > -- > > > > > > > Zoltan Forray > > > > > > > TSM Software & Hardware Administrator > > > > > > > Virginia Commonwealth University > > > > > > > UCC/Office of Technology Services > > > > > > > zfor...@vcu.edu - 804-828-4807 > > > > > > > Don't be a phishing victim - VCU and other reputable > > organizations > > > > will > > > > > > > never use email to request that you reply with your password, > > > social > > > > > > > security number or confidential personal information. For more > > > > details > > > > > > > visit > > > > > > > http://infosecurity.vcu.edu/phishing.html > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Zoltan Forray > > > > > TSM Software & Hardware Administrator > > > > > Virginia Commonwealth University > > > > > UCC/Office of Technology Services > > > > > zfor...@vcu.edu - 804-828-4807 > > > > > Don't be a phishing victim - VCU and other reputable organizations > > will > > > > > never use email to request that you reply with your password, > social > > > > > security number or confidential personal information. For more > > details > > > > > visit > > > > > http://infosecurity.vcu.edu/phishing.html > > > > > > > > > > > > > > > > > > > > > -- > > > Zoltan Forray > > > TSM Software & Hardware Administrator > > > Virginia Commonwealth University > > > UCC/Office of Technology Services > > > zfor...@vcu.edu - 804-828-4807 > > > Don't be a phishing victim - VCU and other reputable organizations will > > > never use email to request that you reply with your password, social > > > security number or confidential personal information. For more details > > > visit > > > http://infosecurity.vcu.edu/phishing.html > > > > > > > > > -- > Zoltan Forray > TSM Software & Hardware Administrator > Virginia Commonwealth University > UCC/Office of Technology Services > zfor...@vcu.edu - 804-828-4807 > Don't be a phishing victim - VCU and other reputable organizations will > never use email to request that you reply with your password, social > security number or confidential personal information. For more details > visit > http://infosecurity.vcu.edu/phishing.html >