Re: Hung Admin Sessions
Hello Tom, Are these sessions created by the admin command line or by the web? This could be caused by the APAR IC30203: "ADMINISTRATIVE COMMAND-LINE SESSIONS MAY HANG ON THE TSM SERVER IF THE SESSIONS ARE CANCELLED WHILE AWAITING SERVER RESPONSE " The APAR is fixed in the 4.1.5.0 server Regards, Vo An Nguyen Email: [EMAIL PROTECTED] Tom Melton <[EMAIL PROTECTED]>@VM.MARIST.EDU> on 01/02/2002 02:41:26 PM Please respond to "ADSM: Dist Stor Manager" <[EMAIL PROTECTED]> Sent by: "ADSM: Dist Stor Manager" <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] cc: Subject: Hung Admin Sessions Storage Management Server for AIX-RS/6000 - Version 4, Release 1, Level 4.0 I have about 10 hung admin sessions, searched the archive and found many people have this problem. Has there been a resolution found? Is there a way to cancel these sessions, as the cancel command fails to work? Thanks in advance... Tom Melton Emory HealthCare
Hung Admin Sessions
Storage Management Server for AIX-RS/6000 - Version 4, Release 1, Level 4.0 I have about 10 hung admin sessions, searched the archive and found many people have this problem. Has there been a resolution found? Is there a way to cancel these sessions, as the cancel command fails to work? Thanks in advance... Tom Melton Emory HealthCare
Re: Hung Admin Sessions
Jennifer, Yes, I get these too... They appear to happen whenever I have a session doing anything that takes a long time (moving data w/ wait=yes, big sql query, etc.). The 'ghost' sessions don't appear to harm anything, although I have noticed that eventually the server will start acting generally strange until a halt and restart are done. I've also noticed that the longer a dsmadmc program is running (even if the session times out w/ the server) the slower the client responds. No real apparent reason for this, either. I too am running 4.1.2 on AIX 4.3.3 +---+ | Brian T. HuntleySystems Administrator | | Campus Information Services, Clarkson University | | Ph/FAX: 315.268-6723/6570 | | [EMAIL PROTECTED] www.clarkson.edu/cis | +---+ UNIX *is* user friendly. It's just selective about who its friends are. PGP Public Key available. finger [EMAIL PROTECTED] On Fri, 23 Feb 2001, Jennifer I. Moore wrote: // Gentle Greetings, // // Has anyone noticed any situations where Admin sessions on the server get // hung and become "ghost" sessions that will not go away? Cancelling has no // effect. Opening a separate Admin session and cancelling from there has no // effect. In some cases, subsequent sessions get hung and a reboot is // necessary. And in other cases, there seems to be no adverse side effects. // // In this case, I was doing some SQL queries on the database, and two user // Admin sessions hung and two Query Admin sessions hung. (Fortunately, // everything else is continuing to run smoothly.) Below is the output from // "show threads" related to these 4 sessions. // // We are running 4.1.2 Server on an IBM M80 running AIX 4.3.3. // // Thank you in advance for any help and/or suggestions. // // Jennifer // // // Thread 68: SessionThread // tid=4437, ktid=34577, ptid=52, det=1, zomb=0, join=0, result=0, sess=31636 // Awaiting cond 5c355b04 (mutex 301a65b0) at 10009b38 (outGetNext) // // Thread 101: SessionThread // tid=653d, ktid=69811, ptid=52, det=1, zomb=0, join=0, result=0, sess=31768 //Awaiting cond 5c33a484 (mutex 301a65b0) at 10009b38 (outGetNext) // // Thread 105: SessionThread // tid=6963, ktid=72547, ptid=52, det=1, zomb=0, join=0, result=0, sess=30446 //Awaiting cond 5c286354 (mutex 301a65b0) at 10009b38 (outGetNext) // // Thread 108: SessionThread // tid=6acf, ktid=52079, ptid=52, det=1, zomb=0, join=0, result=0, sess=31704 //Awaiting cond 5c1c69d4 (mutex 301a65b0) at 10009b38 (outGetNext) // //
Re: Hung Admin Sessions
Yes and its getting rather annoying ... but we noticed this when we went to ECA015 on our 3466 that's 3.7.2 from 3.1.2.20 ... joe.f. Joseph A Faracchio, Systems Programmer, UC Berkeley On Fri, 23 Feb 2001, Jennifer I. Moore wrote: > Gentle Greetings, > > Has anyone noticed any situations where Admin sessions on the server get > hung and become "ghost" sessions that will not go away? Cancelling has no > effect. Opening a separate Admin session and cancelling from there has no > effect. In some cases, subsequent sessions get hung and a reboot is > necessary. And in other cases, there seems to be no adverse side effects. > > In this case, I was doing some SQL queries on the database, and two user > Admin sessions hung and two Query Admin sessions hung. (Fortunately, > everything else is continuing to run smoothly.) Below is the output from > "show threads" related to these 4 sessions. > > We are running 4.1.2 Server on an IBM M80 running AIX 4.3.3. > > Thank you in advance for any help and/or suggestions. > > Jennifer > > > Thread 68: SessionThread > tid=4437, ktid=34577, ptid=52, det=1, zomb=0, join=0, result=0, sess=31636 > Awaiting cond 5c355b04 (mutex 301a65b0) at 10009b38 (outGetNext) > > Thread 101: SessionThread > tid=653d, ktid=69811, ptid=52, det=1, zomb=0, join=0, result=0, sess=31768 >Awaiting cond 5c33a484 (mutex 301a65b0) at 10009b38 (outGetNext) > > Thread 105: SessionThread > tid=6963, ktid=72547, ptid=52, det=1, zomb=0, join=0, result=0, sess=30446 >Awaiting cond 5c286354 (mutex 301a65b0) at 10009b38 (outGetNext) > > Thread 108: SessionThread > tid=6acf, ktid=52079, ptid=52, det=1, zomb=0, join=0, result=0, sess=31704 >Awaiting cond 5c1c69d4 (mutex 301a65b0) at 10009b38 (outGetNext) >
Re: Hung Admin Sessions
We see a similar problem occasionally, but can't reproduce it on demand. A long-running SQL query will run fine one day, hang the next. I think it MAY be related to having another long-running process (such as reclamation) in flight while the query is running. (But this could be coincidence.) You can cancel any active processes, and you can cancel the session that started the long-running query, but that session never goes away, and never times out. Then any additional SQL queries also hang. Eventually ALL queries hang, although backups continue to run just fine. You can leave the situation there for days and it will never clear up on its on. Eventually you have to kill the server and restart it to clear the condition. We are TSM 3.7.2 on AIX 4.3.3, 20 GB data base. Doesn't happen on our other TSM 3.7.2 server, which is exactly the same config, but only as a 3 GB data base. Wanda Prather The Johns Hopkins Applied Physics Lab 443-778-8769 [EMAIL PROTECTED] "Intelligence has much less practical application than you'd think" - Scott Adams/Dilbert -Original Message- From: George Lesho [mailto:[EMAIL PROTECTED]] Sent: Monday, February 26, 2001 11:55 AM To: [EMAIL PROTECTED] Subject: Re: Hung Admin Sessions Jennifer, The same type problem occurs in threads on my version of TSM 3.7.3.6 for drives waiting on instructions to dismount tapes... you would probably do us a favor by bringing this issue to TSM for resolution... Don't the sessions eventually "time out" and disappear? George Lesho Storage/System Admin AFC Enterprises "Jennifer I. Moore" <[EMAIL PROTECTED]> on 02/23/2001 06:32:14 PM Please respond to "ADSM: Dist Stor Manager" <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] cc: (bcc: George Lesho/Partners/AFC) Fax to: Subject: Hung Admin Sessions Gentle Greetings, Has anyone noticed any situations where Admin sessions on the server get hung and become "ghost" sessions that will not go away? Cancelling has no effect. Opening a separate Admin session and cancelling from there has no effect. In some cases, subsequent sessions get hung and a reboot is necessary. And in other cases, there seems to be no adverse side effects. In this case, I was doing some SQL queries on the database, and two user Admin sessions hung and two Query Admin sessions hung. (Fortunately, everything else is continuing to run smoothly.) Below is the output from "show threads" related to these 4 sessions. We are running 4.1.2 Server on an IBM M80 running AIX 4.3.3. Thank you in advance for any help and/or suggestions. Jennifer Thread 68: SessionThread tid=4437, ktid=34577, ptid=52, det=1, zomb=0, join=0, result=0, sess=31636 Awaiting cond 5c355b04 (mutex 301a65b0) at 10009b38 (outGetNext) Thread 101: SessionThread tid=653d, ktid=69811, ptid=52, det=1, zomb=0, join=0, result=0, sess=31768 Awaiting cond 5c33a484 (mutex 301a65b0) at 10009b38 (outGetNext) Thread 105: SessionThread tid=6963, ktid=72547, ptid=52, det=1, zomb=0, join=0, result=0, sess=30446 Awaiting cond 5c286354 (mutex 301a65b0) at 10009b38 (outGetNext) Thread 108: SessionThread tid=6acf, ktid=52079, ptid=52, det=1, zomb=0, join=0, result=0, sess=31704 Awaiting cond 5c1c69d4 (mutex 301a65b0) at 10009b38 (outGetNext)
Hung Admin Sessions
Hi Jennifer,
We have see this problem of Web Admin sessions hanging in earlier
releases (APAR IC28349). We are now running 3.7.3.8 and are not
presently experiencing this. But we have seen some other problems
running SQL queries. One issue is that TSM does not support correlated
sub-queries. When I've issued these complex SQL queries, the query may
run for a very long time or appear to loop and then crash the server.
Another issue we've seen with SQL queries, is that they use database
space for their calculations. We've had some problems with not have
enough contiguous space for the transaction. This is partially true for
us, because we have an old, large, highly fragmented database.
Good luck getting to the bottom of this issue.
Dennis
Subject:
Hung Admin Sessions
Date:
Fri, 23 Feb 2001 19:32:14 -0500
From:
"Jennifer I. Moore" <[EMAIL PROTECTED]>
Gentle Greetings,
Has anyone noticed any situations where Admin sessions on the server get
hung and become "ghost" sessions that will not go away? Cancelling has
no
effect. Opening a separate Admin session and cancelling from there has
no
effect. In some cases, subsequent sessions get hung and a reboot is
necessary. And in other cases, there seems to be no adverse side
effects.
In this case, I was doing some SQL queries on the database, and two user
Admin sessions hung and two Query Admin sessions hung. (Fortunately,
everything else is continuing to run smoothly.) Below is the output
from
"show threads" related to these 4 sessions.
We are running 4.1.2 Server on an IBM M80 running AIX 4.3.3.
Thank you in advance for any help and/or suggestions.
Jennifer
Thread 68: SessionThread
tid=4437, ktid=34577, ptid=52, det=1, zomb=0, join=0, result=0,
sess=31636
Awaiting cond 5c355b04 (mutex 301a65b0) at 10009b38 (outGetNext)
Thread 101: SessionThread
tid=653d, ktid=69811, ptid=52, det=1, zomb=0, join=0, result=0,
sess=31768
Awaiting cond 5c33a484 (mutex 301a65b0) at 10009b38 (outGetNext)
Thread 105: SessionThread
tid=6963, ktid=72547, ptid=52, det=1, zomb=0, join=0, result=0,
sess=30446
Awaiting cond 5c286354 (mutex 301a65b0) at 10009b38 (outGetNext)
Thread 108: SessionThread
tid=6acf, ktid=52079, ptid=52, det=1, zomb=0, join=0, result=0,
sess=31704
Awaiting cond 5c1c69d4 (mutex 301a65b0) at 10009b38 (outGetNext)
--
Dennis Riley
Qwest Communications International Inc.
450 110th NE, Room 103
Bellevue, WA 98004
work: 425-451-5236
fax: 425-453-3589
pager: 800-946-4646.1434047
cell: 425-351-7049
begin:vcard
n:Riley;Dennis
tel;pager:(800)946-4646.1434047
tel;cell:(425)351-7049
tel;fax:(425)453-3589
tel;work:(425)451-5236
x-mozilla-html:FALSE
org:EWEBARS;
adr:;;450 110th NE, Room 103;Bellevue;Washington;98004;
version:2.1
email;internet:[EMAIL PROTECTED]
title:TSM Server Specialist
x-mozilla-cpt:;-18080
fn:Riley, Dennis
end:vcard
Re: Hung Admin Sessions
Each of these threads will have at least one other thread associated with it. You should find an admin command thread that has a ptid= the thread number of the session thread. What this session thread is waiting for is more output from that command thread. You should probably open a problem record on this. There have been some situations in the past where there was some lock contention with certain queries. It might be possible to figure this out from just some additional query and show commands or it might require a kill -11 so a core dump can be generated and those threads can be examined. The point to start with is what those command threads are waiting for and continue diagnosis from there. David Bohm TSM server development email - [EMAIL PROTECTED] >Gentle Greetings, > >Has anyone noticed any situations where Admin sessions on the server get >hung and become "ghost" sessions that will not go away? Cancelling has no >effect. Opening a separate Admin session and cancelling from there has no >effect. In some cases, subsequent sessions get hung and a reboot is >necessary. And in other cases, there seems to be no adverse side effects. > >In this case, I was doing some SQL queries on the database, and two user >Admin sessions hung and two Query Admin sessions hung. (Fortunately, >everything else is continuing to run smoothly.) Below is the output from >"show threads" related to these 4 sessions. > >We are running 4.1.2 Server on an IBM M80 running AIX 4.3.3. > >Thank you in advance for any help and/or suggestions. > >Jennifer > > >Thread 68: SessionThread >tid=4437, ktid=34577, ptid=52, det=1, zomb=0, join=0, result=0, sess=31636 > Awaiting cond 5c355b04 (mutex 301a65b0) at 10009b38 (outGetNext) > >Thread 101: SessionThread > tid=653d, ktid=69811, ptid=52, det=1, zomb=0, join=0, result=0, sess=31768 > Awaiting cond 5c33a484 (mutex 301a65b0) at 10009b38 (outGetNext) > >Thread 105: SessionThread > tid=6963, ktid=72547, ptid=52, det=1, zomb=0, join=0, result=0, sess=30446 > Awaiting cond 5c286354 (mutex 301a65b0) at 10009b38 (outGetNext) > >Thread 108: SessionThread > tid=6acf, ktid=52079, ptid=52, det=1, zomb=0, join=0, result=0, sess=31704 > Awaiting cond 5c1c69d4 (mutex 301a65b0) at 10009b38 (outGetNext)
Re: Hung Admin Sessions
Jennifer, The same type problem occurs in threads on my version of TSM 3.7.3.6 for drives waiting on instructions to dismount tapes... you would probably do us a favor by bringing this issue to TSM for resolution... Don't the sessions eventually "time out" and disappear? George Lesho Storage/System Admin AFC Enterprises "Jennifer I. Moore" <[EMAIL PROTECTED]> on 02/23/2001 06:32:14 PM Please respond to "ADSM: Dist Stor Manager" <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] cc:(bcc: George Lesho/Partners/AFC) Fax to: Subject: Hung Admin Sessions Gentle Greetings, Has anyone noticed any situations where Admin sessions on the server get hung and become "ghost" sessions that will not go away? Cancelling has no effect. Opening a separate Admin session and cancelling from there has no effect. In some cases, subsequent sessions get hung and a reboot is necessary. And in other cases, there seems to be no adverse side effects. In this case, I was doing some SQL queries on the database, and two user Admin sessions hung and two Query Admin sessions hung. (Fortunately, everything else is continuing to run smoothly.) Below is the output from "show threads" related to these 4 sessions. We are running 4.1.2 Server on an IBM M80 running AIX 4.3.3. Thank you in advance for any help and/or suggestions. Jennifer Thread 68: SessionThread tid=4437, ktid=34577, ptid=52, det=1, zomb=0, join=0, result=0, sess=31636 Awaiting cond 5c355b04 (mutex 301a65b0) at 10009b38 (outGetNext) Thread 101: SessionThread tid=653d, ktid=69811, ptid=52, det=1, zomb=0, join=0, result=0, sess=31768 Awaiting cond 5c33a484 (mutex 301a65b0) at 10009b38 (outGetNext) Thread 105: SessionThread tid=6963, ktid=72547, ptid=52, det=1, zomb=0, join=0, result=0, sess=30446 Awaiting cond 5c286354 (mutex 301a65b0) at 10009b38 (outGetNext) Thread 108: SessionThread tid=6acf, ktid=52079, ptid=52, det=1, zomb=0, join=0, result=0, sess=31704 Awaiting cond 5c1c69d4 (mutex 301a65b0) at 10009b38 (outGetNext)
Re: Hung Admin Sessions
Are you seeing any BA client sessions going thru? Is expiration running? It certainly sounds as if you're creating deadlock with the SQL quieries. On Fri, 23 Feb 2001, Jennifer I. Moore wrote: > Gentle Greetings, > > Has anyone noticed any situations where Admin sessions on the server get > hung and become "ghost" sessions that will not go away? Cancelling has no > effect. Opening a separate Admin session and cancelling from there has no > effect. In some cases, subsequent sessions get hung and a reboot is > necessary. And in other cases, there seems to be no adverse side effects. > > In this case, I was doing some SQL queries on the database, and two user > Admin sessions hung and two Query Admin sessions hung. (Fortunately, > everything else is continuing to run smoothly.) Below is the output from > "show threads" related to these 4 sessions. > > We are running 4.1.2 Server on an IBM M80 running AIX 4.3.3. > > Thank you in advance for any help and/or suggestions. > > Jennifer > > > Thread 68: SessionThread > tid=4437, ktid=34577, ptid=52, det=1, zomb=0, join=0, result=0, sess=31636 > Awaiting cond 5c355b04 (mutex 301a65b0) at 10009b38 (outGetNext) > > Thread 101: SessionThread > tid=653d, ktid=69811, ptid=52, det=1, zomb=0, join=0, result=0, sess=31768 >Awaiting cond 5c33a484 (mutex 301a65b0) at 10009b38 (outGetNext) > > Thread 105: SessionThread > tid=6963, ktid=72547, ptid=52, det=1, zomb=0, join=0, result=0, sess=30446 >Awaiting cond 5c286354 (mutex 301a65b0) at 10009b38 (outGetNext) > > Thread 108: SessionThread > tid=6acf, ktid=52079, ptid=52, det=1, zomb=0, join=0, result=0, sess=31704 >Awaiting cond 5c1c69d4 (mutex 301a65b0) at 10009b38 (outGetNext) >
Hung Admin Sessions
Gentle Greetings, Has anyone noticed any situations where Admin sessions on the server get hung and become "ghost" sessions that will not go away? Cancelling has no effect. Opening a separate Admin session and cancelling from there has no effect. In some cases, subsequent sessions get hung and a reboot is necessary. And in other cases, there seems to be no adverse side effects. In this case, I was doing some SQL queries on the database, and two user Admin sessions hung and two Query Admin sessions hung. (Fortunately, everything else is continuing to run smoothly.) Below is the output from "show threads" related to these 4 sessions. We are running 4.1.2 Server on an IBM M80 running AIX 4.3.3. Thank you in advance for any help and/or suggestions. Jennifer Thread 68: SessionThread tid=4437, ktid=34577, ptid=52, det=1, zomb=0, join=0, result=0, sess=31636 Awaiting cond 5c355b04 (mutex 301a65b0) at 10009b38 (outGetNext) Thread 101: SessionThread tid=653d, ktid=69811, ptid=52, det=1, zomb=0, join=0, result=0, sess=31768 Awaiting cond 5c33a484 (mutex 301a65b0) at 10009b38 (outGetNext) Thread 105: SessionThread tid=6963, ktid=72547, ptid=52, det=1, zomb=0, join=0, result=0, sess=30446 Awaiting cond 5c286354 (mutex 301a65b0) at 10009b38 (outGetNext) Thread 108: SessionThread tid=6acf, ktid=52079, ptid=52, det=1, zomb=0, join=0, result=0, sess=31704 Awaiting cond 5c1c69d4 (mutex 301a65b0) at 10009b38 (outGetNext)
