Hi Eric and all, We are aware of this situation and putting additional focus and attention on it.
Del ---------------------------------------------------- "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 07/19/2019 09:35:37 AM: > From: "Loon, Eric van (ITOP NS) - KLM" <eric-van.l...@klm.com> > To: ADSM-L@VM.MARIST.EDU > Date: 07/19/2019 09:36 AM > Subject: [EXTERNAL] Re: deletion performance of large deduplicated files > Sent by: "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> > > Hi Rick and others! > > I replicated the data of the test TDP client to multiple servers, > running 7.1.7, 7.1.9 and even 8.1.8: the performance sucks on all servers. > We do not use client replication as part of our server protection. > We need a real time replication over the datacenter and thus we rely > on host-based replication for all our servers. I tested with and > without replication: there is no noticeable difference. > As a work-around I will install a non-deduplicated file stgpool on > the worst performing server next week. I expect the server to > perform better as soon as all container pool objects are expired, in > about 3 weeks from now. > In the meantime I will keep on pursuing IBM until it is fixed or > else we might need to replace the product altogether... > > Kind regards, > Eric van Loon > Air France/KLM Storage & Backup > > -----Original Message----- > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On > Behalf Of Rick Adamson > Sent: vrijdag 19 juli 2019 14:45 > To: ADSM-L@VM.MARIST.EDU > Subject: Re: deletion performance of large deduplicated files > > Eric, Michael, > I have been working through similar struggles and as I read your > posts had to wonder, can you provide some details on your server and > client versions? > Basically I now have experience/exposure to every version/ > maintenance pack/patch SP has put out since 7.1.3.x > > My servers are now running on 8.1.8 (windows) and has seemed to > have stabilized (for now). > One thing that was causing me a lot of grief was using directory > storage pools with client side deduplication enabled, particularly > on data protection products (all of them). > Afterwards there was a lot of cleanup; auditing containers, > confirming that protect storage was completing successfully, and > finally doing the same for replicate node processes. > > I found that understandably server performance takes a severe nose > dive if you are trying to process (protect/replicate) damaged > containers, and most likely restores will be compromised as well. > > > Thank you, > -Rick Adamson > > > -----Original Message----- > From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of Michael Prix > Sent: Friday, July 19, 2019 6:11 AM > To: ADSM-L@VM.MARIST.EDU > Subject: Re: [ADSM-L] deletion performance of large deduplicated files > > * This email originated outside of the organization. Use caution > when opening attachments or clicking links. * > > ---------------------------------------------------------------------- > Hello Eric, > > > > welcome to my nightmares. Take a seat, wanna have a drink? > > > > I had the pleasure of performance and data corruption PMRs during the last two > > years with TDP Oracle. Yes, at first the customer got blamed for not adhering > > completely to to blueprints, but after some weeks it boild down to ... > > silence. > > Data corruption was because of what ended in IT28096 - now fixed. > > Performance is interesting, but resembles to what you have written. We work > > with MAXPIECESIZE settings on RMAN to keep the backup pieces small and got > > some interesting values, pending further observation, but we might be on a > > cheerful way. I'm talking about database sizes of 50TB here, warehouse style. > > In between we moved the big DBs to a dedicated server to prove that the > > performance drop is because of the big DBs, and the remaining "small" DBs - > > size of 500MB up to 5TB - didn't put any measurable stress on the DB in terms > > of expiration and protect stgpool. Even the big DBs on their dedicated server > > performed better in terms of expiration and protect stgpool, which might have > > been a coincidence of these DBs holding nearly the same data and having the > > same retention period. > > > > What I can't observe is a slowness of the DB. Queries are answered in the > > normal time - depending on the query. a count(*) from backupobjects naturally > > takes some time, considerably longer when you use dedup, but the daily queries > > are answered in the "normal" timeframe. > > > > What helped immediately was some tuning: > > - More LUNS and filesystems for the TSM-DB > > - smaller disks, but more of them, for each filesystem. > > changing the disks from 100GB to 2 x 50GB for each DB-filesystem got me a > > performance boost of 200% in expiration and backup db. Unbelievable,but true. > > Yes, I'm using SSD. And SVC. And multiple storage systems. Performance isn't > > the problem, we are measuring 2ms respone time for write AND read. > > - stripeset for each fileset > > > > > > -- > > Michael Prix > > > > On Fri, 2019-07-19 at 07:29 +0000, Loon, Eric van (ITOP NS) - KLM wrote: > > > Hi TSM/SP-ers, > > > > > > We are struggling with the performance of our TSM servers for months now. We > > > are running several servers with hardware (Data Domain) dedup for years > > > without any problems, but on our new servers with directory container pools > > > performance is really, really bad. > > > The servers and storage are designed according to the Blueprints and they > > > are working fine as long as you do not add large database (Oracle and SAP) > > > client to them. As soon as you do, the overall server performance becomes > > > very bad: client and admin session initiation takes 20 to 40 seconds, SQL > > > queries run for minutes where they should take a few seconds and q query > > > stgpool sometimes takes more than a minute to respond! > > > I have two cases open for this. In one case we focused a lot on the OS and > > > disk performance, but during that process I noticed that the performance is > > > most likely caused by the way TSM processes large (few hundred MB) files. I > > > performed a large amount of tests and came to the conclusion that it takes > > > TSM a huge amount of time to delete large deduplicated files, both in > > > container pools as deduplicated file pools. As test I use an TDP for Oracle > > > client which uses a backup piece size of 900 MB. The client contains about > > > 5000 files. Deleting the files from a container pool takes more than an > > > hour. When you run a delete object for the files individually I see that > > > most files take more than a second(!) to delete. If I put that same data in > > > a non-deduplicated file pool, a delete filespace takes about 15 seconds... > > > The main issue is that the TDP clients are doing the exact same thing: as > > > soon as a backup file is no longer needed, it's removed from the RMAN > > > catalog and deleted from TSM. Since we have several huge database clients > > > (multiple TB's each) these Oracle delete jobs tend to run for hours. These > > > delete jobs also seem to slow down each other, because when there are > > > several of those jobs running at the same time, they become even more slow. > > > At this point I have one server where these jobs are running 24 hours per > > > day! This server is at the moment the worst performing TSM server I have > > > ever seen. On the other container pool servers I was able to move the Oracle > > > and SAP server away to the old servers (the ones with the Data Domain), but > > > on this one I can't because of Data Domain capacity reasons. > > > For this file deletion performance I also have a case open, but there is > > > absolutely no progress. I proved IBM how bad the performance is and I even > > > offered them a copy of our database so they can see for themselves, but only > > > silence from development... > > > One thing I do not understand: I find it very hard to believe that we are > > > the only one suffering from this issue. There must be dozens of TSM users > > > out there that backup large databases to TSM container pools? > > > > > > Kind regards, > > > Eric van Loon > > > Air France/KLM Storage & Backup > > > ******************************************************** > > > For information, services and offers, please visit our web site: > > > https://urldefense.proofpoint.com/v2/url? > u=http-3A__www.klm.com&d=DwICaQ&c=AzgFQeXLLKhxSQaoFCm29A&r=eqh5PzQPIsPArLoI_uV1mKvhIpcNP1MsClDPSJjFfxw&m=OAsoTBlnuy- > Km69JdbqTRXQFPP3_U4a9gj4CSO0lbFw&s=MSJs- > FT5ZG_GgtiqMOCzRHNo0Bw0PiD3C7-s_xuNgBg&e= . This e-mail and any > attachment may contain confidential > > > and privileged material intended for the addressee only. If you are not the > > > addressee, you are notified that no part of the e-mail or any attachment may > > > be disclosed, copied or distributed, and that any other action related to > > > this e-mail or attachment is strictly prohibited, and may be unlawful. If > > > you have received this e-mail by error, please notify the sender immediately > > > by return e-mail, and delete this message. > > > > > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its > > > employees shall not be liable for the incorrect or incomplete transmission > > > of this e-mail or any attachments, nor responsible for any delay in receipt. > > > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch > > > Airlines) is registered in Amstelveen, The Netherlands, with registered > > > number 33014286 > > > ******************************************************** > > > **CONFIDENTIALITY NOTICE** This electronic message contains > information from Southeastern Grocers, Inc and is intended only for > the use of the addressee. This message may contain information that > is privileged, confidential and/or exempt from disclosure under > applicable Law. This message may not be read, used, distributed, > forwarded, reproduced or stored by any other than the intended > recipient. If you are not the intended recipient, please delete and > notify the sender. > ******************************************************** > For information, services and offers, please visit our web site: > https://urldefense.proofpoint.com/v2/url? > u=http-3A__www.klm.com&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=0hq2JX5c3TEZNriHEs7Zf7HrkY2fNtONOrEOM8Txvk8&m=sawlFLhDPFJ01kK39aEsF5p- > P9m243WQvtv2i8G_tG0&s=THD1AsuKVK6V1PFwHzYqwMwt5qW9rRgIaXYIEhTqyUk&e= > . This e-mail and any attachment may contain confidential and > privileged material intended for the addressee only. If you are not > the addressee, you are notified that no part of the e-mail or any > attachment may be disclosed, copied or distributed, and that any > other action related to this e-mail or attachment is strictly > prohibited, and may be unlawful. If you have received this e-mail by > error, please notify the sender immediately by return e-mail, and > delete this message. > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/ > or its employees shall not be liable for the incorrect or incomplete > transmission of this e-mail or any attachments, nor responsible for > any delay in receipt. > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal > Dutch Airlines) is registered in Amstelveen, The Netherlands, with > registered number 33014286 > ******************************************************** >
