I bounced the server and successfully turned off replication for the node with the problem and deleted it's filespaces.
However, now I can't get ANY replication to run - even when trying to replicate a single node. I get these errors (I have tried Googling but did not find much and what I did find talked about earlier versions of TSM server): 9/1/2016 10:30:14 AM ANR2017I Administrator ZFORRAY issued command: REPLICATE NODE orarcat.vcu.edu maxsessions=2 9/1/2016 10:30:15 AM ANR2032E REPLICATE NODE: Command failed - internal server error detected. 9/1/2016 10:30:15 AM ANR9999D Thread<846> issued message 2032 from: 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x00000000dde43a outTextf 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x0000000087cb66 BuildFSReplicationList 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x0000000087e3ee imReplicateNode 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x000000005a0506 AdmReplicateNode 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x000000004eb03e AdmCommandLocal 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x000000004ec28e admCommand 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x00000000ca7fca SmAdminCommandThread 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x00000000e7c6ea StartThread 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x00003b7d207aa1 *UNKNOWN* 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x00003b7cee8aad *UNKNOWN* 9/1/2016 10:30:15 AM ANR1930E REPLICATE NODE: There are no eligible filespaces for node ORARCAT.VCU.EDU. 9/1/2016 10:30:15 AM ANR9999D_0413043177 GetCommandRc(admrepl.c:4727) Thread<846>: unexpected rc=1114 9/1/2016 10:30:15 AM ANR9999D Thread<846> issued message 9999 from: 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x00000000de4373 OutDiagToCons 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x00000000de7175 outDiagfExt 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x0000000059c6e4 GetCommandRc 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x000000005a025f AdmReplicateNode 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x000000004eb03e AdmCommandLocal 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x000000004ec28e admCommand 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x00000000ca7fca SmAdminCommandThread 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x00000000e7c6ea StartThread 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x00003b7d207aa1 *UNKNOWN* 9/1/2016 10:30:15 AM ANR9999D Thread<846> 0x00003b7cee8aad *UNKNOWN* On Thu, Sep 1, 2016 at 8:58 AM, Zoltan Forray <zfor...@vcu.edu> wrote: > I forgot to mention I already did that but while it was stuck issuing > errors. Just tried it again and it confirms it is not being replicated. > Time to restart the TSM server. Hoping I won't have to bounce the whole > box. The last time a replication process got stuck on a target server, > halting the TSM server would not clear it. The server would not shut down > so I bounced the whole thing. > > 8:51:18 AM HADES : remove replnode compile.vcu.edu > ANR1630W REMOVE REPLNODE: COMPILE.VCU.EDU is not configured for > replication. > ANR2679E REMOVE REPLNODE: All of the nodes provided are invalid. > ANR1632E REMOVE REPLNODE: Command failed. Replication state information > for > the specified nodes could not be removed. > ANS8001I Return code 11. > > > On Thu, Sep 1, 2016 at 8:40 AM, David Ehresman < > david.ehres...@louisville.edu> wrote: > >> On the source server, try "remove replnode NODENAME", then try deleting >> its filespaces. >> >> David >> >> -----Original Message----- >> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of >> Zoltan Forray >> Sent: Thursday, September 01, 2016 8:26 AM >> To: ADSM-L@VM.MARIST.EDU >> Subject: [ADSM-L] How to delete a node/filespaces when target replica >> server is unavailable >> >> We are just starting to roll out replication and I have a problem. >> >> Many months ago, I tested replication/configuration on a single node. Now >> I need to delete the node and all of its filespaces. >> >> But the target server used for replication testing has been taken >> down/unavailable. >> >> Now when I issued delete filespaces for the node on the source server, the >> delete process has hung. Attempting to cancel the delete process has been >> hung for >24-hours and is starting to cause log pinning issues. >> >> I am ready to bounce this server (RH Linux with 6.3.5.100 server) but >> still >> don't know how to delete these filespaces without hanging things up, >> again. >> >> I have turned off replication on the source server. Tried to update the >> node to disable replication but that didn't seem to help/take. >> >> -- >> *Zoltan Forray* >> TSM Software & Hardware Administrator >> Xymon Monitor Administrator >> VMware Administrator (in training) >> Virginia Commonwealth University >> UCC/Office of Technology Services >> www.ucc.vcu.edu >> zfor...@vcu.edu - 804-828-4807 >> Don't be a phishing victim - VCU and other reputable organizations will >> never use email to request that you reply with your password, social >> security number or confidential personal information. For more details >> visit https://urldefense.proofpoint.com/v2/url?u=http-3A__infosecu >> rity.vcu.edu_phishing.html&d=AwIBaQ&c=SgMrq23dbjbGX6e0ZsSHg >> EZX6A4IAf1SO3AJ2bNrHlk&r=dOGCMY197NTNH1k_wcsrWS3_fxedKW4rpKJ >> 8cHCD2L8&m=q25GyWPNnS9scC8LnF8VhVZP_CBMMjSimusH4Kjy5gk&s=T7Y >> D4SfFzcqmGHMpT9uZYJqBhEZ3v5MaKBWoelfQ5Pc&e= >> > > > > -- > *Zoltan Forray* > TSM Software & Hardware Administrator > Xymon Monitor Administrator > VMware Administrator (in training) > Virginia Commonwealth University > UCC/Office of Technology Services > www.ucc.vcu.edu > zfor...@vcu.edu - 804-828-4807 > Don't be a phishing victim - VCU and other reputable organizations will > never use email to request that you reply with your password, social > security number or confidential personal information. For more details > visit http://infosecurity.vcu.edu/phishing.html > -- *Zoltan Forray* TSM Software & Hardware Administrator Xymon Monitor Administrator VMware Administrator (in training) Virginia Commonwealth University UCC/Office of Technology Services www.ucc.vcu.edu zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://infosecurity.vcu.edu/phishing.html