Team Just would like to update status for replication issue reported earlier. IBM development team managed to find the root cause and provided solution for intermittent replication hung. We have applied those recommended settings and didn’t see any hung for the past 3 weeks.
Regards Sarav +65 9857 8665 > On 24 Sep 2020, at 12:12 AM, Sasa Drnjevic <sasa.drnje...@srce.hr> wrote: > > Hi Sarav and all, > > I forgot to inform you earlier, but I have solved the problem with HUNG > or more precisely EXTREMELY slow replication. > > Linux nodees were ALL replicating as before the upgrade, without any > problems. > > The main culprit was Windows Systemstate. Ordinary NTFS filespaces > replicated fine. > > But when I have replicated all of the windows nodes manually one-by-one > (not as a nodegroup) the second and all successive runs were normal as > before. So, now everything runs fine as before the upgrade to > v8.1.10.100 (on RHEL 7.8). > > All windows would take much more to replicate when run for the first > time after the upgrade, some of them took 8 hours instead of less than > 10 minutes. > > IBM support blamed all of it on canceled replication because they have > found some msgs like: > > ANR1650W The server detected partially replicated data > from a previous replication operation. This might result > in extended processing time for process 2 while the > server is replicating node nodename... > > > Yes, I have canceled some, but maybe only 5-6 windows nodes, but all of > the 40+ windows nodes took ages to replicate for the first time after > upgrade. > > Most of the windows nodes used some readonly tapes (scanning only?) with > data more than 2 months old which was surely replicated long time ago. > > Some Linux nodes (out of 180+) were canceled too, but later replicated > with normal throughput. > > So, be careful :-) > > Regards, > > -- > Sasa Drnjevic > www.srce.unizg.hr/en/ > > > > > > > > >> On 9.9.2020. 1:08, Saravanan Palanisamy wrote: >> Hi Sasa >> >> Thanks for the update. We are still waiting for updates from dev team. >> >> Regards >> Sarav >> +65 9857 8665 >> >> >>>> On 9 Sep 2020, at 4:55 AM, Sasa Drnjevic <sasa.drnje...@srce.hr> wrote: >>> >>> On 2.9.2020. 1:30, Saravanan Palanisamy wrote: >>>> Hi Eric >>>> >>>> We have applied efix to address APAR IT33437 instead of patch v8.1.10.100 >>>> and we haven’t seen any server crash after >efix. However our intermittent >>>> replication issue case still in progress. >>> >>> >>> Hi Sarav & all. >>> >>> Last week I have upgraded my target server to v8.1.10.100, and node >>> replication was going without any problems from source v8.1.1.000 to >>> target v8.1.10.100. Both servers are RHEL 7.8. >>> >>> So, this morning I took my chances and upgraded my source server to >>> v8.1.10.100. The upgrade went fine. >>> >>> But, replication hangs :-( >>> >>> I have opened a case with IBM support and did a first step supplying >>> some logs. >>> >>> But, while trying to find out what is causing the replication to hang I >>> did some 30 manual single node replication jobs and ALL of the Linux >>> nodes replicated with SUCCESS. >>> >>> I have node groups which are replicated with schedules, and of course >>> some are mixed with Linux and Windows nodes. >>> >>> So, when a windows node hangs it hangs the whole node group. >>> >>> And it seems that the main culprit is the Windows Systemstate. >>> >>> Would bet on that, and I mentioned that to IBM support. >>> >>> We'll see tomorrow. >>> >>> Hope it helps. >>> >>> Rgds, >>> >>> -- >>> Sasa Drnjevic >>> www.srce.unizg.hr/en/ >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>> >>>> Regards >>>> Sarav >>>> +65 9857 8665 >>>> >>>> >>>>>> On 1 Sep 2020, at 10:38 PM, Loon, Eric van (ITOP NS) - KLM >>>>>> <eric-van.l...@klm.com> wrote: >>>>> >>>>> Hi Sarav, >>>>> >>>>> On page https://www.ibm.com/support/pages/node/6212043#8110100 I do find >>>>> APAR IT33437 listed. >>>>> >>>>> Kind regards, >>>>> Eric van Loon >>>>> Air France/KLM Storage & Backup >>>>> >>>>> -----Original Message----- >>>>> From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of >>>>> Saravanan Palanisamy >>>>> Sent: maandag 31 augustus 2020 23:10 >>>>> To: ADSM-L@VM.MARIST.EDU >>>>> Subject: Re: Replication Hung after upgrade to v8.1.10 >>>>> >>>>> Hi Eric >>>>> >>>>> Thanks. I have gone through v8.1.10.100 APAR details but I don’t find my >>>>> issue here. I am still waiting for IBM on my case. >>>>> >>>>> Regards >>>>> Sarav >>>>> +65 9857 8665 >>>>> >>>>> >>>>>> On 31 Aug 2020, at 3:00 PM, Loon, Eric van (ITOP NS) - KLM >>>>>> <eric-van.l...@klm.com> wrote: >>>>>> >>>>>> Hi Sarav, >>>>>> >>>>>> You probably noticed it already, but IBM released 8.1.10.100 containing >>>>>> a fix for your (and several other) issues. >>>>>> >>>>>> Kind regards, >>>>>> Eric van Loon >>>>>> Air France/KLM Storage & Backup >>>>>> >>>>>> -----Original Message----- >>>>>> From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of >>>>>> Saravanan Palanisamy >>>>>> Sent: donderdag 20 augustus 2020 17:42 >>>>>> To: ADSM-L@VM.MARIST.EDU >>>>>> Subject: Re: Replication Hung after upgrade to v8.1.10 >>>>>> >>>>>> IT33437 >>>>>> >>>>>> Regards >>>>>> Sarav >>>>>> +65 9857 8665 >>>>>> >>>>>> >>>>>>>> On 20 Aug 2020, at 11:05 PM, Loon, Eric van (ITOP NS) - KLM >>>>>>>> <eric-van.l...@klm.com> wrote: >>>>>>> >>>>>>> Hi Sarav, >>>>>>> >>>>>>> You stated " There is also one more issue on v8.1.10 where server will >>>>>>> crash abruptly if you use directory container. It appears to be known >>>>>>> issue and we have got efix to address it". I find this very alarming, >>>>>>> because I'm about to start production on a new server running this >>>>>>> level. Do you have an APAR number? >>>>>>> >>>>>>> Kind regards, >>>>>>> Eric van Loon >>>>>>> Air France/KLM Storage & Backup >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of >>>>>>> Saravanan Palanisamy >>>>>>> Sent: donderdag 20 augustus 2020 16:58 >>>>>>> To: ADSM-L@VM.MARIST.EDU >>>>>>> Subject: Re: Replication Hung after upgrade to v8.1.10 >>>>>>> >>>>>>> Thanks Erwann for the update. >>>>>>> >>>>>>> We are still facing intermittent hang and no progress on our case. If >>>>>>> it’s one server then we could say it’s environment issue but 3 >>>>>>> different pair and all are in different countries as well. We are >>>>>>> holding upgrade for rest of countries until find out root cause. >>>>>>> >>>>>>> There is also one more issue on v8.1.10 where server will crash >>>>>>> abruptly if you use directory container. It appears to be known issue >>>>>>> and we have got efix to address it. >>>>>>> >>>>>>> I will keep you posted if I hear any updates on my case. >>>>>>> >>>>>>> Regards >>>>>>> Sarav >>>>>>> +65 9857 8665 >>>>>>> >>>>>>> >>>>>>>>> On 20 Aug 2020, at 10:10 PM, Erwann SIMON <erwann.si...@free.fr> >>>>>>>>> wrote: >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I've just upgraded some servers to that level and I do not encounter >>>>>>>> this king of issue (right now...). >>>>>>>> >>>>>>>> -- >>>>>>>> Best regards / Cordialement / مع تحياتي >>>>>>>> Erwann SIMON >>>>>>>> >>>>>>>> ----- Mail original ----- >>>>>>>> De: "Saravanan Palanisamy" <evergreen.sa...@gmail.com> >>>>>>>> À: ADSM-L@VM.MARIST.EDU >>>>>>>> Envoyé: Mardi 21 Juillet 2020 00:37:41 >>>>>>>> Objet: [ADSM-L] Replication Hung after upgrade to v8.1.10 >>>>>>>> >>>>>>>> Dear Team >>>>>>>> >>>>>>>> Is there anyone facing intermittent replication hung issue after >>>>>>>> upgrade to v8.1.10? >>>>>>>> >>>>>>>> We have upgraded TSM server from 8.1.9.300 to 8.1.10 and replication >>>>>>>> process is hung intermittently on all 3 servers. Just want to see >>>>>>>> whether anyone else impacted. >>>>>>>> >>>>>>>> Once restart TSM instance then replication works but again replication >>>>>>>> node process will get hung. But we don’t see any hung for protect stg >>>>>>>> and it works fine. >>>>>>>> >>>>>>>> We have raised case with IBM and working on it. >>>>>>>> >>>>>>>> Regards >>>>>>>> Sarav >>>>>>>> +65 9857 8665 >>>>>>> ******************************************************** >>>>>>> For information, services and offers, please visit our web site: >>>>>>> http://www.klm.com. This e-mail and any attachment may contain >>>>>>> confidential and privileged material intended for the addressee only. >>>>>>> If you are not the addressee, you are notified that no part of the >>>>>>> e-mail or any attachment may be disclosed, copied or distributed, and >>>>>>> that any other action related to this e-mail or attachment is strictly >>>>>>> prohibited, and may be unlawful. If you have received this e-mail by >>>>>>> error, please notify the sender immediately by return e-mail, and >>>>>>> delete this message. >>>>>>> >>>>>>> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or >>>>>>> its employees shall not be liable for the incorrect or incomplete >>>>>>> transmission of this e-mail or any attachments, nor responsible for any >>>>>>> delay in receipt. >>>>>>> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch >>>>>>> Airlines) is registered in Amstelveen, The Netherlands, with registered >>>>>>> number 33014286 >>>>>>> ******************************************************** >>>>>> ******************************************************** >>>>>> For information, services and offers, please visit our web site: >>>>>> http://www.klm.com. This e-mail and any attachment may contain >>>>>> confidential and privileged material intended for the addressee only. If >>>>>> you are not the addressee, you are notified that no part of the e-mail >>>>>> or any attachment may be disclosed, copied or distributed, and that any >>>>>> other action related to this e-mail or attachment is strictly >>>>>> prohibited, and may be unlawful. If you have received this e-mail by >>>>>> error, please notify the sender immediately by return e-mail, and delete >>>>>> this message. >>>>>> >>>>>> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or >>>>>> its employees shall not be liable for the incorrect or incomplete >>>>>> transmission of this e-mail or any attachments, nor responsible for any >>>>>> delay in receipt. >>>>>> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch >>>>>> Airlines) is registered in Amstelveen, The Netherlands, with registered >>>>>> number 33014286 >>>>>> ******************************************************** >>>>> ******************************************************** >>>>> For information, services and offers, please visit our web site: >>>>> http://www.klm.com. This e-mail and any attachment may contain >>>>> confidential and privileged material intended for the addressee only. If >>>>> you are not the addressee, you are notified that no part of the e-mail or >>>>> any attachment may be disclosed, copied or distributed, and that any >>>>> other action related to this e-mail or attachment is strictly prohibited, >>>>> and may be unlawful. If you have received this e-mail by error, please >>>>> notify the sender immediately by return e-mail, and delete this message. >>>>> >>>>> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its >>>>> employees shall not be liable for the incorrect or incomplete >>>>> transmission of this e-mail or any attachments, nor responsible for any >>>>> delay in receipt. >>>>> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch >>>>> Airlines) is registered in Amstelveen, The Netherlands, with registered >>>>> number 33014286 >>>>> ********************************************************
