We rebooted it yesterday. I just did a virgin install of the 8.1.0.2 package on a new VM and we are going to compare all of the files in *baclient* and *Common Files* to see if there are any discrepancies. Any other directories that contain files we should be looking at?
On Fri, Oct 6, 2017 at 12:56 PM, Andrew Raibeck <stor...@us.ibm.com> wrote: > Hi Zoltan, > > I do not think the problem is really with MSVCR110.dll, and the stack > traces in the dsmcrash.log file are not going to be particularly insightful > (other than knowing when a crash occurs, how often, and exception type). I > suppose if you upgraded the client but did not reboot if necessary, that > *might* be a contributing factor. But we would have to see the > corresponding dsmcrash.dmp file to get a better sense of why it occurs. > > Regards, > > Andy > > ____________________________________________________________ > ________________ > > Andrew Raibeck | IBM Spectrum Protect Level 3 | stor...@us.ibm.com > > IBM Tivoli Storage Manager links: > Product support: > https://www.ibm.com/support/entry/portal/product/tivoli/ > tivoli_storage_manager > > Online documentation: > http://www.ibm.com/support/knowledgecenter/SSGSG7/ > landing/welcome_ssgsg7.html > > Product Wiki: > https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli% > 20Storage%20Manager > > "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 2017-10-06 > 12:28:07: > > > From: Zoltan Forray <zfor...@vcu.edu> > > To: ADSM-L@VM.MARIST.EDU > > Date: 2017-10-06 12:29 > > Subject: Re: Problems after upgrading client from 7.1.6.3 to 8.1.0.2 > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> > > > > Andy, > > > > From the dsmcrash.log file: > > > > 2017/10/04 10:25:54 > > IBM Spectrum Protect > > Version: 8.1.0.2 > > Build date: Mon May 01 16:15:45 2017 > > > > dsmcsvc.exe caused exception C0000005 (EXCEPTION_ACCESS_VIOLATION) at > > 00000000722C7C65 > > > > with pointers to MSVCR110.dll. > > > > Can you tell me what level/version this dll is supposed to be at? I > > suspect something didn't get updated when the client was upgraded. > > > > > > On Fri, Oct 6, 2017 at 11:26 AM, Andrew Raibeck <stor...@us.ibm.com> > wrote: > > > > > Hi Zoltan, > > > > > > It sounds like you are seeing multiple issues, best to deal with them > one > > > at a time. > > > > > > 1) For the crash dump messages, it is possible that there is only one > dump, > > > and the message just keeps repeating because the client detects the > same > > > file over and over again. Check the corresponding dsmcrash.log file > (start > > > at the bottom of that file) to see if crashes are constantly occurring, > and > > > when they occurred. If the crashes are persistent, you should open a > PMR > > > and upload the dump, so it can be reviewed. If it is a one-time crash, > you > > > can do the same thing, or you can delete the dump file (dsmcrash.dmp), > > > which should eliminate the messages. If the problem reoccurs, open a > PMR, > > > etc. > > > > > > 2) I am not sure what the hung sessions are. Sessions in IdleW state > mean > > > that the session is not in transaction with the server. Most likely > these > > > are consumer sessions waiting for work, possibly related to item 3, > below. > > > > > > 3) You mentioned the backups run longer, and you observe "Process Dirs" > as > > > the category with the most time in it. "Process Dirs" is the time the > > > client spends traversing the file system, searching for changed files. > A > > > lengthy "Process Dirs" time is not necessarily a "problem" for file > systems > > > with a large number, e.g., hundreds of thousands, or millions, of > files; > > > and that number can be the largest if the number of changed files is > very > > > small in comparison. This could also explain consumer sessions in IdleW > > > state (item 2, above). > > > > > > Does the dsminstr.log file include data from when the client level was > > > 7.1.6.3? If yes, then you can compare the latest 7.1.6.3 metrics with > the > > > earliest 8.1.0.2 metrics to see if there was truly some big difference > > > introduced at the time you upgraded the client. Off the top of my head, > I > > > cannot think of anything in the client code itself that would suddenly > > > cause such a performance issue. > > > > > > Without actual data to see what is going on (complete dsmsched.log > file, > > > dsmerror.log file, dsminstr.log file, dsm.opt, etc.) it is hard to > provide > > > more specific suggestions. > > > > > > Regards, > > > > > > Andy > > > > > > ____________________________________________________________ > > > ________________ > > > > > > Andrew Raibeck | IBM Spectrum Protect Level 3 | stor...@us.ibm.com > > > > > > IBM Tivoli Storage Manager links: > > > Product support: > > > https://www.ibm.com/support/entry/portal/product/tivoli/ > > > tivoli_storage_manager > > > > > > Online documentation: > > > http://www.ibm.com/support/knowledgecenter/SSGSG7/ > > > landing/welcome_ssgsg7.html > > > > > > Product Wiki: > > > https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli% > > > 20Storage%20Manager > > > > > > "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 2017-10-06 > > > 10:36:15: > > > > > > > From: Zoltan Forray <zfor...@vcu.edu> > > > > To: ADSM-L@VM.MARIST.EDU > > > > Date: 2017-10-06 10:37 > > > > Subject: Problems after upgrading client from 7.1.6.3 to 8.1.0.2 > > > > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> > > > > > > > > We have this server that is used to backup DFS mounts/files. It has > 30+ > > > > TSM nodes configured to it. > > > > > > > > In the past, everything has been running like it should, albeit a > little > > > > slow sometimes due to the size of some of the DFS mounts but the > backups > > > > would usually finish. > > > > > > > > After upgrading the client to 8.1.0.2, we are experiencing constant > > > > problems. Many of the backups are not completing, some node sessions > are > > > > running 10+ hours to examine 1000 files and backup 7MB in changes, > etc. > > > > > > > > Looking at the logs, we are seeing numerous "A IBM Spectrum Protect > core > > > > file or crash report was found:" crash-dumps for most of the nodes, > often > > > > numerous times within the same backup session. > > > > > > > > Instrumentation statistics on a few nodes I examined say the largest > time > > > > consumer is "Process Dirs". > > > > > > > > From the SP server perspective, these nodes are in a constant "Idle > > > Wait". > > > > Session info shows lots of metadata send to the client, but very > little > > > > coming from the client. > > > > > > > > The only change has been the SP client upgrade. Yes we have rebooted > the > > > > server (Windows 2012) multiple times to clear the hung sessions > > > (sometimes > > > > running multiple days and never ending). > > > > > > > > Any way to determine is going on? > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *Zoltan Forray* > > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > > > Xymon Monitor Administrator > > > > VMware Administrator > > > > Virginia Commonwealth University > > > > UCC/Office of Technology Services > > > > www.ucc.vcu.edu > > > > zfor...@vcu.edu - 804-828-4807 > > > > Don't be a phishing victim - VCU and other reputable organizations > will > > > > never use email to request that you reply with your password, social > > > > security number or confidential personal information. For more > details > > > > visit https://urldefense.proofpoint.com/v2/url? > > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > > > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=bnJkpUfY38- > > > > > > > nYutnVeKi_lreUncb4usTMD65HpEa4ko&s=hvKEF7hPnJ83ZRrIKoqxy35CfELFHG > > > srMocoLUGDXZU&e= > > > > > > > > > > > > > > > > > > -- > > *Zoltan Forray* > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > > Xymon Monitor Administrator > > VMware Administrator > > Virginia Commonwealth University > > UCC/Office of Technology Services > > www.ucc.vcu.edu > > zfor...@vcu.edu - 804-828-4807 > > Don't be a phishing victim - VCU and other reputable organizations will > > never use email to request that you reply with your password, social > > security number or confidential personal information. For more details > > visit https://urldefense.proofpoint.com/v2/url? > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx- > > siA1ZOg&r=Ij6DLy1l7wDpCbTfcDkLC_KknvhyGdCy_RnAGnhV37I&m=SSNQ9Hyzp- > > > uXTAnZNWKcnvvYR9klhqPdbivWnLFH6Gk&s=pOYK0915YykOlToqNC0KdOnwwh2r93 > uz68JThlrbepU&e= > > > > -- *Zoltan Forray* Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon Monitor Administrator VMware Administrator Virginia Commonwealth University UCC/Office of Technology Services www.ucc.vcu.edu zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://phishing.vcu.edu/