[U2] [UV] Large File Operations Kill Linux
Looking for some ideas on how to keep Linux from becoming largely unresponsive when creating large files. What happens is as the new file is being created the I/O buffer cache quickly fills up with dirty buffers. Until the kernel can flush these out to disk there is no avail buffers for I/O operations from other processes. . The most troubling manifestation of this is the transaction logging check point daemon gets *way* behind putting us as risk if we were to have a failure of some kind. I have tried using ionice and renice to slow the file creation down as much as possible. This help a little but is still a big problem. Any ideas how to get CREATE.FILE/RESIZE to play nice on Linux? Thanks. Perry Perry Taylor Senior MV Architect ZirMed 888 West Market Street, Suite 400 Louisville, KY 40202 www.zirmed.comhttp://www.zirmed.com/ CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. ZirMed, Inc. has strict policies regarding the content of e-mail communications, specifically Protected Health Information, any communications containing such material will be returned to the originating party with such advisement noted. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] [UV] Large File Operations Kill Linux
A few questions - What linux version/distro are you on and what type of file system, and how much ram do you have -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Perry Taylor Sent: 04 February 2013 15:57 To: U2-Users List Subject: [U2] [UV] Large File Operations Kill Linux Looking for some ideas on how to keep Linux from becoming largely unresponsive when creating large files. What happens is as the new file is being created the I/O buffer cache quickly fills up with dirty buffers. Until the kernel can flush these out to disk there is no avail buffers for I/O operations from other processes. . The most troubling manifestation of this is the transaction logging check point daemon gets *way* behind putting us as risk if we were to have a failure of some kind. I have tried using ionice and renice to slow the file creation down as much as possible. This help a little but is still a big problem. Any ideas how to get CREATE.FILE/RESIZE to play nice on Linux? Thanks. Perry Perry Taylor Senior MV Architect ZirMed 888 West Market Street, Suite 400 Louisville, KY 40202 www.zirmed.comhttp://www.zirmed.com/ CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. ZirMed, Inc. has strict policies regarding the content of e-mail communications, specifically Protected Health Information, any communications containing such material will be returned to the originating party with such advisement noted. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users - No virus found in this message. Checked by AVG - www.avg.com Version: 2012.0.2238 / Virus Database: 2639/5579 - Release Date: 02/03/13 ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] [UV] Large File Operations Kill Linux
We're on RHEL5 (2.6.18-348.el5), ext3 and 132GB ram. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Symeon Breen Sent: Monday, February 04, 2013 9:23 AM To: 'U2 Users List' Subject: Re: [U2] [UV] Large File Operations Kill Linux A few questions - What linux version/distro are you on and what type of file system, and how much ram do you have -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Perry Taylor Sent: 04 February 2013 15:57 To: U2-Users List Subject: [U2] [UV] Large File Operations Kill Linux Looking for some ideas on how to keep Linux from becoming largely unresponsive when creating large files. What happens is as the new file is being created the I/O buffer cache quickly fills up with dirty buffers. Until the kernel can flush these out to disk there is no avail buffers for I/O operations from other processes. . The most troubling manifestation of this is the transaction logging check point daemon gets *way* behind putting us as risk if we were to have a failure of some kind. I have tried using ionice and renice to slow the file creation down as much as possible. This help a little but is still a big problem. Any ideas how to get CREATE.FILE/RESIZE to play nice on Linux? Thanks. Perry Perry Taylor Senior MV Architect ZirMed 888 West Market Street, Suite 400 Louisville, KY 40202 www.zirmed.comhttp://www.zirmed.com/ CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. ZirMed, Inc. has strict policies regarding the content of e-mail communications, specifically Protected Health Information, any communications containing such material will be returned to the originating party with such advisement noted. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users - No virus found in this message. Checked by AVG - www.avg.com Version: 2012.0.2238 / Virus Database: 2639/5579 - Release Date: 02/03/13 ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] [UV] Large File Operations Kill Linux
What's the value in /proc/sys/vm/swappiness? From: perry.tay...@zirmed.com To: u2-users@listserver.u2ug.org Date: Mon, 4 Feb 2013 20:53:13 + Subject: Re: [U2] [UV] Large File Operations Kill Linux We're on RHEL5 (2.6.18-348.el5), ext3 and 132GB ram. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Symeon Breen Sent: Monday, February 04, 2013 9:23 AM To: 'U2 Users List' Subject: Re: [U2] [UV] Large File Operations Kill Linux A few questions - What linux version/distro are you on and what type of file system, and how much ram do you have -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Perry Taylor Sent: 04 February 2013 15:57 To: U2-Users List Subject: [U2] [UV] Large File Operations Kill Linux Looking for some ideas on how to keep Linux from becoming largely unresponsive when creating large files. What happens is as the new file is being created the I/O buffer cache quickly fills up with dirty buffers. Until the kernel can flush these out to disk there is no avail buffers for I/O operations from other processes. . The most troubling manifestation of this is the transaction logging check point daemon gets *way* behind putting us as risk if we were to have a failure of some kind. I have tried using ionice and renice to slow the file creation down as much as possible. This help a little but is still a big problem. Any ideas how to get CREATE.FILE/RESIZE to play nice on Linux? Thanks. Perry Perry Taylor Senior MV Architect ZirMed 888 West Market Street, Suite 400 Louisville, KY 40202 www.zirmed.comhttp://www.zirmed.com/ CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. ZirMed, Inc. has strict policies regarding the content of e-mail communications, specifically Protected Health Information, any communications containing such material will be returned to the originating party with such advisement noted. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users - No virus found in this message. Checked by AVG - www.avg.com Version: 2012.0.2238 / Virus Database: 2639/5579 - Release Date: 02/03/13 ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] [UV] Large File Operations Kill Linux
On 04/02/13 21:05, Dan Fitzgerald wrote: What's the value in /proc/sys/vm/swappiness? How will that make any difference? 2.6.18-348 SOUNDS like an ancient (in linux terms) kernel. Are you on RedHat support? This is a problem with the linux kernel that was addressed recently, iirc. Large amounts of io from a single process can swamp the queue, and the latest kernels have it fixed. If you've got RH support, see if you can find out if that's been backported into your kernel. Cheers, Wol From: perry.tay...@zirmed.com To: u2-users@listserver.u2ug.org Date: Mon, 4 Feb 2013 20:53:13 + Subject: Re: [U2] [UV] Large File Operations Kill Linux We're on RHEL5 (2.6.18-348.el5), ext3 and 132GB ram. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Symeon Breen Sent: Monday, February 04, 2013 9:23 AM To: 'U2 Users List' Subject: Re: [U2] [UV] Large File Operations Kill Linux A few questions - What linux version/distro are you on and what type of file system, and how much ram do you have -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Perry Taylor Sent: 04 February 2013 15:57 To: U2-Users List Subject: [U2] [UV] Large File Operations Kill Linux Looking for some ideas on how to keep Linux from becoming largely unresponsive when creating large files. What happens is as the new file is being created the I/O buffer cache quickly fills up with dirty buffers. Until the kernel can flush these out to disk there is no avail buffers for I/O operations from other processes. . The most troubling manifestation of this is the transaction logging check point daemon gets *way* behind putting us as risk if we were to have a failure of some kind. I have tried using ionice and renice to slow the file creation down as much as possible. This help a little but is still a big problem. Any ideas how to get CREATE.FILE/RESIZE to play nice on Linux? Thanks. Perry Perry Taylor Senior MV Architect ZirMed 888 West Market Street, Suite 400 Louisville, KY 40202 www.zirmed.comhttp://www.zirmed.com/ ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] [UV] Large File Operations Kill Linux
70. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan Fitzgerald Sent: Monday, February 04, 2013 2:06 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] [UV] Large File Operations Kill Linux What's the value in /proc/sys/vm/swappiness? From: perry.tay...@zirmed.com To: u2-users@listserver.u2ug.org Date: Mon, 4 Feb 2013 20:53:13 + Subject: Re: [U2] [UV] Large File Operations Kill Linux We're on RHEL5 (2.6.18-348.el5), ext3 and 132GB ram. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Symeon Breen Sent: Monday, February 04, 2013 9:23 AM To: 'U2 Users List' Subject: Re: [U2] [UV] Large File Operations Kill Linux A few questions - What linux version/distro are you on and what type of file system, and how much ram do you have -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Perry Taylor Sent: 04 February 2013 15:57 To: U2-Users List Subject: [U2] [UV] Large File Operations Kill Linux Looking for some ideas on how to keep Linux from becoming largely unresponsive when creating large files. What happens is as the new file is being created the I/O buffer cache quickly fills up with dirty buffers. Until the kernel can flush these out to disk there is no avail buffers for I/O operations from other processes. . The most troubling manifestation of this is the transaction logging check point daemon gets *way* behind putting us as risk if we were to have a failure of some kind. I have tried using ionice and renice to slow the file creation down as much as possible. This help a little but is still a big problem. Any ideas how to get CREATE.FILE/RESIZE to play nice on Linux? Thanks. Perry Perry Taylor Senior MV Architect ZirMed 888 West Market Street, Suite 400 Louisville, KY 40202 www.zirmed.comhttp://www.zirmed.com/ CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. ZirMed, Inc. has strict policies regarding the content of e-mail communications, specifically Protected Health Information, any communications containing such material will be returned to the originating party with such advisement noted. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users - No virus found in this message. Checked by AVG - www.avg.com Version: 2012.0.2238 / Virus Database: 2639/5579 - Release Date: 02/03/13 ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] [UV] Large File Operations Kill Linux
Yes we are on RH support. I'll run it by them and see. Thanks. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wols Lists Sent: Monday, February 04, 2013 2:15 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] [UV] Large File Operations Kill Linux On 04/02/13 21:05, Dan Fitzgerald wrote: What's the value in /proc/sys/vm/swappiness? How will that make any difference? 2.6.18-348 SOUNDS like an ancient (in linux terms) kernel. Are you on RedHat support? This is a problem with the linux kernel that was addressed recently, iirc. Large amounts of io from a single process can swamp the queue, and the latest kernels have it fixed. If you've got RH support, see if you can find out if that's been backported into your kernel. Cheers, Wol From: perry.tay...@zirmed.com To: u2-users@listserver.u2ug.org Date: Mon, 4 Feb 2013 20:53:13 + Subject: Re: [U2] [UV] Large File Operations Kill Linux We're on RHEL5 (2.6.18-348.el5), ext3 and 132GB ram. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Symeon Breen Sent: Monday, February 04, 2013 9:23 AM To: 'U2 Users List' Subject: Re: [U2] [UV] Large File Operations Kill Linux A few questions - What linux version/distro are you on and what type of file system, and how much ram do you have -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Perry Taylor Sent: 04 February 2013 15:57 To: U2-Users List Subject: [U2] [UV] Large File Operations Kill Linux Looking for some ideas on how to keep Linux from becoming largely unresponsive when creating large files. What happens is as the new file is being created the I/O buffer cache quickly fills up with dirty buffers. Until the kernel can flush these out to disk there is no avail buffers for I/O operations from other processes. . The most troubling manifestation of this is the transaction logging check point daemon gets *way* behind putting us as risk if we were to have a failure of some kind. I have tried using ionice and renice to slow the file creation down as much as possible. This help a little but is still a big problem. Any ideas how to get CREATE.FILE/RESIZE to play nice on Linux? Thanks. Perry Perry Taylor Senior MV Architect ZirMed 888 West Market Street, Suite 400 Louisville, KY 40202 www.zirmed.comhttp://www.zirmed.com/ ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. ZirMed, Inc. has strict policies regarding the content of e-mail communications, specifically Protected Health Information, any communications containing such material will be returned to the originating party with such advisement noted. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] [UV] Large File Operations Kill Linux
On 04/02/13 21:34, Perry Taylor wrote: Yes we are on RH support. I'll run it by them and see. Again, this is from memory, but I think somebody noticed that copying a single very large file brought a system to its knees until the copy finished, and the whole thing spiralled from there. Probably about 6 months to a year ago. Chances are I picked up the story from LWN. Thanks. Cheers, Wol ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] [UV] Large File Operations Kill Linux
Other users could have been hanging at malloc. With a swappiness of 100 (on some kernels) or 100 (on others) or not 0 or 100(not sure which behavior you get on 2.6.18), pages wouldn't be getting freed up quickly enough duing the creation/copying of a large file. Another thing to look at (although I prefer the support route, since you have it), is /sys/kernel/mm/transparent_hugepage/defrag. Other people who have had this problem alleviated it by setting this to never. Of course, others fixed it by updating the kernel. My aged eyes read what you have as 2.6.8.1... Date: Mon, 4 Feb 2013 21:15:25 + From: antli...@youngman.org.uk To: u2-users@listserver.u2ug.org Subject: Re: [U2] [UV] Large File Operations Kill Linux On 04/02/13 21:05, Dan Fitzgerald wrote: What's the value in /proc/sys/vm/swappiness? How will that make any difference? 2.6.18-348 SOUNDS like an ancient (in linux terms) kernel. Are you on RedHat support? This is a problem with the linux kernel that was addressed recently, iirc. Large amounts of io from a single process can swamp the queue, and the latest kernels have it fixed. If you've got RH support, see if you can find out if that's been backported into your kernel. Cheers, Wol From: perry.tay...@zirmed.com To: u2-users@listserver.u2ug.org Date: Mon, 4 Feb 2013 20:53:13 + Subject: Re: [U2] [UV] Large File Operations Kill Linux We're on RHEL5 (2.6.18-348.el5), ext3 and 132GB ram. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Symeon Breen Sent: Monday, February 04, 2013 9:23 AM To: 'U2 Users List' Subject: Re: [U2] [UV] Large File Operations Kill Linux A few questions - What linux version/distro are you on and what type of file system, and how much ram do you have -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Perry Taylor Sent: 04 February 2013 15:57 To: U2-Users List Subject: [U2] [UV] Large File Operations Kill Linux Looking for some ideas on how to keep Linux from becoming largely unresponsive when creating large files. What happens is as the new file is being created the I/O buffer cache quickly fills up with dirty buffers. Until the kernel can flush these out to disk there is no avail buffers for I/O operations from other processes. . The most troubling manifestation of this is the transaction logging check point daemon gets *way* behind putting us as risk if we were to have a failure of some kind. I have tried using ionice and renice to slow the file creation down as much as possible. This help a little but is still a big problem. Any ideas how to get CREATE.FILE/RESIZE to play nice on Linux? Thanks. Perry Perry Taylor Senior MV Architect ZirMed 888 West Market Street, Suite 400 Louisville, KY 40202 www.zirmed.comhttp://www.zirmed.com/ ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
