Filesystem for Linux production database server?
Hi, friends that run oracle on linux: We are running some database on linux, some with UPS protection and some not, all using ext2. Currently we have a DW server running oracle 817/Redhat 7.2/Ext2. When server crash because of power supply, fsck took rather long time! And we are planning migrate to redhat as2.1/9.2, and I am thinking of what kind of filesystem shall I choose for the new server. The database is about 150G.I had planned to use ext3 for it is journal filesystem, But according to http://www.linuxjournal.com/article.php?sid=5841, ext3 is bad for performance, and some friend said he hit filesystem crash with ext3. So,what is your experience using filesystem for oracle in linux? Performance and avaliability is of most important. Thanks. Regards zhu chao msn:[EMAIL PROTECTED] www.cnoug.org -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: zhu chao INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
Re: Filesystem for Linux production database server?
Linux file systems usually do not support direct I/O (bypassing the buffer cache), which means that you're going to have double caching with almost everything except raw devices. You can have up to 256 raw devices on a Linux box and if your database is small enough (32-bit system, 2G limit applies), that will definitely be the fastest option. My recommendation is to go with IBM JFS because it is a battle tested, mature, well performing file system. I would advise against XFS because nobody uses it. As for availability, you'll have to go with some RAID controller and standby database or RAC. In case of RAC, your choice of file systems is clear (OCFS). Whatever you do, do not configure your RAID as RAID-5 but RAID 1+0 (BARF). which means that On 2003.06.29 11:44, zhu chao wrote: Hi, friends that run oracle on linux: We are running some database on linux, some with UPS protection and some not, all using ext2. Currently we have a DW server running oracle 817/Redhat 7.2/Ext2. When server crash because of power supply, fsck took rather long time! And we are planning migrate to redhat as2.1/9.2, and I am thinking of what kind of filesystem shall I choose for the new server. The database is about 150G.I had planned to use ext3 for it is journal filesystem, But according to http://www.linuxjournal.com/article.php?sid=5841, ext3 is bad for performance, and some friend said he hit filesystem crash with ext3. So,what is your experience using filesystem for oracle in linux? Performance and avaliability is of most important. Thanks. Regards zhu chao msn:[EMAIL PROTECTED] www.cnoug.org -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: zhu chao INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Mladen Gogala Oracle DBA -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Mladen Gogala INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
Re: Filesystem for Linux production database server?
Hi, Gogala: Thanks very much for your suggestions. some points: 1.Oracle 9i on linux 2.4 does support 64GB datafile/raw device. Linux 2.4 kernel support 2GB+ files and oracle 9i have 64bit filesystem io support.I have a rac database on raw with some datafiles 20Gb. 2.Maybe I will choose raid5, as it is dw database.Raid5 is pretty good for Read most database.Though raid10 is also good, it reduced the number of avaliable disks that are avaliable for read. 3.I think I will go for raw device. I am not familier with JFS yet.Raw make it easy when we upgrade to rac:).After all,pc server have limited power and rac is a possible solution. 4.Have anyone successfully compiled redhat as2.1 with LVM support? I tried several times and always fail with "VGDA in kernel and lvmtab are NOT consistent" after system reboot. Regards zhu chao msn:[EMAIL PROTECTED] www.cnoug.org - Original Message - To: "Multiple recipients of list ORACLE-L" <[EMAIL PROTECTED]> Sent: Monday, June 30, 2003 2:14 AM > Linux file systems usually do not support direct I/O (bypassing the buffer > cache), which means that you're going to have double caching with almost > everything except raw devices. You can have up to 256 raw devices on a Linux > box and if your database is small enough (32-bit system, 2G limit applies), > that will definitely be the fastest option. My recommendation is to go with > IBM JFS because it is a battle tested, mature, well performing file system. I > would advise against XFS because nobody uses it. > As for availability, you'll have to go with some RAID controller and standby > database or RAC. In case of RAC, your choice of file systems is clear (OCFS). > Whatever you do, do not configure your RAID as RAID-5 but RAID 1+0 (BARF). > > which means that > On 2003.06.29 11:44, zhu chao wrote: > > Hi, friends that run oracle on linux: > > We are running some database on linux, some with UPS protection and some > > not, all using ext2. Currently we have a DW server running oracle 817/Redhat > > 7.2/Ext2. When server crash because of power supply, fsck took rather long > > time! > > And we are planning migrate to redhat as2.1/9.2, and I am thinking of > > what kind of filesystem shall I choose for the new server. The database is > > about 150G.I had planned to use ext3 for it is journal filesystem, But > > according to http://www.linuxjournal.com/article.php?sid=5841, ext3 is bad > > for performance, and some friend said he hit filesystem crash with ext3. > > So,what is your experience using filesystem for oracle in linux? > > Performance and avaliability is of most important. > > Thanks. > > > > > > Regards > > zhu chao > > msn:[EMAIL PROTECTED] > > www.cnoug.org > > > > -- > > Please see the official ORACLE-L FAQ: http://www.orafaq.net > > -- > > Author: zhu chao > > INET: [EMAIL PROTECTED] > > > > Fat City Network Services-- 858-538-5051 http://www.fatcity.com > > San Diego, California-- Mailing list and web hosting services > > - > > To REMOVE yourself from this mailing list, send an E-Mail message > > to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in > > the message BODY, include a line containing: UNSUB ORACLE-L > > (or the name of mailing list you want to be removed from). You may > > also send the HELP command for other information (like subscribing). > > > > -- > Mladen Gogala > Oracle DBA > -- > Please see the official ORACLE-L FAQ: http://www.orafaq.net > -- > Author: Mladen Gogala > INET: [EMAIL PROTECTED] > > Fat City Network Services-- 858-538-5051 http://www.fatcity.com > San Diego, California-- Mailing list and web hosting services > - > To REMOVE yourself from this mailing list, send an E-Mail message > to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in > the message BODY, include a line containing: UNSUB ORACLE-L > (or the name of mailing list you want to be removed from). You may > also send the HELP command for other information (like subscribing). > > -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: zhu chao INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
Re: Filesystem for Linux production database server?
A database running on a single RAID 1 device on a 2 CPU Dell with ext3 on linux is still faster than a database running on a similarly configured Dell box with windows 2000, and several RAID 10 devices. No kidding. Jared Mladen Gogala <[EMAIL PROTECTED]> Sent by: [EMAIL PROTECTED] 06/29/2003 11:14 AM Please respond to ORACLE-L To: Multiple recipients of list ORACLE-L <[EMAIL PROTECTED]> cc: Subject:Re: Filesystem for Linux production database server? Linux file systems usually do not support direct I/O (bypassing the buffer cache), which means that you're going to have double caching with almost everything except raw devices. You can have up to 256 raw devices on a Linux box and if your database is small enough (32-bit system, 2G limit applies), that will definitely be the fastest option. My recommendation is to go with IBM JFS because it is a battle tested, mature, well performing file system. I would advise against XFS because nobody uses it. As for availability, you'll have to go with some RAID controller and standby database or RAC. In case of RAC, your choice of file systems is clear (OCFS). Whatever you do, do not configure your RAID as RAID-5 but RAID 1+0 (BARF). which means that On 2003.06.29 11:44, zhu chao wrote: > Hi, friends that run oracle on linux: > We are running some database on linux, some with UPS protection and some > not, all using ext2. Currently we have a DW server running oracle 817/Redhat > 7.2/Ext2. When server crash because of power supply, fsck took rather long > time! > And we are planning migrate to redhat as2.1/9.2, and I am thinking of > what kind of filesystem shall I choose for the new server. The database is > about 150G.I had planned to use ext3 for it is journal filesystem, But > according to http://www.linuxjournal.com/article.php?sid=5841, ext3 is bad > for performance, and some friend said he hit filesystem crash with ext3. > So,what is your experience using filesystem for oracle in linux? > Performance and avaliability is of most important. > Thanks. > > > Regards > zhu chao > msn:[EMAIL PROTECTED] > www.cnoug.org > > -- > Please see the official ORACLE-L FAQ: http://www.orafaq.net > -- > Author: zhu chao > INET: [EMAIL PROTECTED] > > Fat City Network Services-- 858-538-5051 http://www.fatcity.com > San Diego, California-- Mailing list and web hosting services > - > To REMOVE yourself from this mailing list, send an E-Mail message > to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in > the message BODY, include a line containing: UNSUB ORACLE-L > (or the name of mailing list you want to be removed from). You may > also send the HELP command for other information (like subscribing). > -- Mladen Gogala Oracle DBA -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Mladen Gogala INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
Re: Filesystem for Linux production database server?
> Linux file systems usually do not support direct I/O (bypassing the buffer > cache), which means that you're going to have double caching with almost This is no longer the case. Look at the O_DIRECT open option, which can be used with oracle. Make sure that your distribution has support for it. > everything except raw devices. You can have up to 256 raw devices on a Linux > box and if your database is small enough (32-bit system, 2G limit applies), > that will definitely be the fastest option. My recommendation is to go with > IBM JFS because it is a battle tested, mature, well performing file system. I > would advise against XFS because nobody uses it. Quite a few people use XFS. Its made it both into the -ac branch and into the 2.5 kernel. However, *I* would strongly recommend ext3 as many people run oracle on it and that OS/filesystem combination has been beaten to hell a lot more than linux/JFS. Also, when placed into data=writeback,noatime mode[1] performs *much* better than the default data=ordered. I recall a rerun of that benchmark in which ext3/writeback was pretty close to ext2's performance save that it has much faster recovery on boot. Second on the "most used" list is reiserfs, however ext3/writeback's direct/block io should be somewhat faster than it. Being that i don't have my own large-scale personal testing in a publishable form, i'll point you to someone else's work (he, unfortunately, doesn't appear to have had a raid array to beat up with via an SMP server) http://oregonstate.edu/~kveton/fs/page2.php I'll note that on my desktop running 2.5.70 the performance for bonnie resulted in the following order (all noatime) FASTEST:ext3/writeback ; XFS ; JFS ;SLOWEST but with only a single platter and one cpu, not much can be said :) Before you get concerned about what writeback means, read the man page for the filesystem you'll find that it is basically what most other filesystems (jfs, xfs, reiserfs, vxfs, etc) use for their default/performance options. however,do make sure that you're patched up to date if you go near data=journal. [1] The noatime mount parameter should be used for any filesystem backing oracle (ext[23],xfs,jfs,reiserfs, etc as there is no need to keep updating last access times, so you can save a whole pile of ~1k writes. One other thing: if you are using fibre attached storage and are *not* using async io, then put some usefully large value into /proc/sys/vm/max-readahead, remember it is power of 2 minus one, so youd want to look at values like 255, 511, and 1023. > As for availability, you'll have to go with some RAID controller and standby > database or RAC. In case of RAC, your choice of file systems is clear (OCFS). > Whatever you do, do not configure your RAID as RAID-5 but RAID 1+0 (BARF). what he said, save that i like raid10. > which means that > On 2003.06.29 11:44, zhu chao wrote: > > Hi, friends that run oracle on linux: > > We are running some database on linux, some with UPS protection and some > > not, all using ext2. Currently we have a DW server running oracle 817/Redhat > > 7.2/Ext2. When server crash because of power supply, fsck took rather long > > time! > > And we are planning migrate to redhat as2.1/9.2, and I am thinking of > > what kind of filesystem shall I choose for the new server. The database is > > about 150G.I had planned to use ext3 for it is journal filesystem, But > > according to http://www.linuxjournal.com/article.php?sid=5841, ext3 is bad > > for performance, and some friend said he hit filesystem crash with ext3. > > So,what is your experience using filesystem for oracle in linux? > > Performance and avaliability is of most important. > > Thanks. > > > > > > Regards > > zhu chao > > msn:[EMAIL PROTECTED] > > www.cnoug.org > > > > -- > > Please see the official ORACLE-L FAQ: http://www.orafaq.net > > -- > > Author: zhu chao > > INET: [EMAIL PROTECTED] > > > > Fat City Network Services-- 858-538-5051 http://www.fatcity.com > > San Diego, California-- Mailing list and web hosting services > > - > > To REMOVE yourself from this mailing list, send an E-Mail message > > to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in > > the message BODY, include a line containing: UNSUB ORACLE-L > > (or the name of mailing list you want to be removed from). You may > > also send the HELP command for other information (like subscribing). > > > > -- .-... . -.-. .-. . --- . ... ... .- --. . Craig I. Hagan hagan(at)cih.com "Never let your sense of morals prevent you from doing what is right." - Mayor Hardin from Isaac Asimov's Foundation -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Craig I. Hagan INET: [EMAIL PROTECTED] Fat City Networ
RE: Filesystem for Linux production database server?
Reiserfs' biggest strength is in its ability to deal with directories with huge numbers of files in a very speedy fashion. Obviously, for oracle this is less relevant. As far as the max-readahead option, I don't understand the specific relevance to fibre-attached storage. I would imagine, though admittedly I no longer have large storage arrays to play with, that setting readahead to be higher could damage performance against arrays with "intelligent" caching algorithms. Having the OS handle read-ahead rather than the array will likely fool the array into thinking that the i/o patterns are more sequential than they are. This will cause them to pre-allocate cache regions and pre-fetch more tracks off disk, which could adversely impact performance. Totally separate from that, Redhat strongly advises _against_ tuning that parameter. I happen to be doing I/O testing right now anyway, so maybe I'll gen up some workloads on different filesystems and go nuts. Thanks, Matt -- Matthew Zito GridApp Systems Email: [EMAIL PROTECTED] Cell: 646-220-3551 Phone: 212-358-8211 x 359 http://www.gridapp.com > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On > Behalf Of Craig I. Hagan > Sent: Wednesday, July 02, 2003 4:11 AM > To: Multiple recipients of list ORACLE-L > Subject: Re: Filesystem for Linux production database server? > > > > Linux file systems usually do not support direct I/O (bypassing the > > buffer cache), which means that you're going to have double caching > > with almost > > This is no longer the case. Look at the O_DIRECT open option, > which can be used with oracle. Make sure that your > distribution has support for it. > > One other thing: if you are using fibre attached storage and > are *not* using async io, then put some usefully large value > into /proc/sys/vm/max-readahead, remember it is power of 2 > minus one, so youd want to look at values like 255, 511, and 1023. > > > > As for availability, you'll have to go with some RAID > controller and > > standby > > database or RAC. In case of RAC, your choice of file > systems is clear (OCFS). > > Whatever you do, do not configure your RAID as RAID-5 but > RAID 1+0 (BARF). > > what he said, save that i like raid10. > > HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Matthew Zito INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
Re: Filesystem for Linux production database server?
On 2003.07.02 05:10, Craig I. Hagan wrote:
> Linux file systems usually do not support direct I/O (bypassing the buffer
> cache), which means that you're going to have double caching with almost
This is no longer the case. Look at the O_DIRECT open option, which can be
used
with oracle. Make sure that your distribution has support for it.
I'm running RH 8.0 with the latest version of the 2.20 kernel. O_DIRECT flag
is supported as of 2.4.10 but it is dependent on the file system. Each file
system has it's own implementation of the "open" system call and most of the
filesystems simply ignore the O_DIRECT specification. If you open a file with
O_DIRECT, you should record much greater disk I/O activity by using "sar -d"
then without it. Also, the program without O_DIRECT should execute much slower
because it shouldn't go through cache. The two programs are attached to this
message, the difference is only in O_DIRECT flag. You can see for yourself
that ext3 completely ignores O_DIRECT flag. My suspicion is that only OCFS is
using O_DIRECT on linux. Here are results of my tests:
$ time test1
Repetitions:5
6.30s real 0.00s user 1.41s system
$
$ time test1
Repetitions:5
6.30s real 0.01s user 1.31s system
$ time test2
Repetitions:5
6.37s real 0.02s user 1.48s system
$ time test2
Repetitions:5
6.32s real 0.00s user 1.38s system
$
O_DIRECT has always been tied to file system. Some honored it, most of them
have simply ignored it. JFS on AIX honors it and I hoped that it does so on
Linux as well. As for XFS, I must say that (and I am a former SGI user and a
life-long admirer) I don't know anybody using it.
--
Mladen Gogala
Oracle DBA#define _GNU_SOURCE
#include
#include
#include
#include
#include
#include
main() {
int fd=-1,count=0,rep=0;
char buff[8192];
fd=open("/tmp/",O_RDONLY|O_DIRECT);
if (fd == -1) {
fprintf(stderr,"Error:%s\n",strerror(errno));
exit(errno);
}
while ((count=read(fd,buff,8192)) > 0)
rep++;
printf("Repetitions:%d\n",rep);
}
#define _GNU_SOURCE
#include
#include
#include
#include
#include
#include
main() {
int fd=-1,count=0,rep=0;
char buff[8192];
fd=open("/tmp/",O_RDONLY);
if (fd == -1) {
fprintf(stderr,"Error:%s\n",strerror(errno));
exit(errno);
}
while ((count=read(fd,buff,8192)) > 0)
rep++;
printf("Repetitions:%d\n",rep);
}
RE: Filesystem for Linux production database server?
> As far as the max-readahead option, I don't understand the specific > relevance to fibre-attached storage. I would imagine, though admittedly > I no longer have large storage arrays to play with, that setting > readahead to be higher could damage performance against arrays with > "intelligent" caching algorithms. Having the OS handle read-ahead this has less to do with the caching than bw utilization. (someone ought to test for async) and/or for filesystem io. Also, this is only for reads (obviously). nutshell is that the default fs *max* readahead is the current block (4k) + 31blocks for a total of ~128k. If you mess with a lot of io to fibre arrays (through fs or raw) you'll find that 128k sized io's is right at the base of the ramp-up for efficiently using your bus bandwidth for streaming io's (e.g. range or fast full scans). If you look at what happens, you start with current block + min-readahead blocks (default is 7) for a total of 64k. if the readahead is successful, then it doubles the number of pages read ahead for that file. repeat until you are at max-readahead. This PITA is that you really are trading streaming io throughput for a percentage of random io with a caveat that a higher readahead change much if you don't do that much sequential io happening (e.g. an OLTP database). > rather than the array will likely fool the array into thinking that the > i/o patterns are more sequential than they are. This will cause them to > pre-allocate cache regions and pre-fetch more tracks off disk, which > could adversely impact performance. again, this comes back to *know* your io pattern. A large db_multiblock_read setting can yield similar results if used incorrectly (the trick with fs readahead is that there is variance) > Totally separate from that, Redhat strongly advises _against_ tuning > that parameter. if you are doing async io, yes. Likewise, if you know what is really going on under the hood and understand your io patterns, then it is handy to have. > I happen to be doing I/O testing right now anyway, so maybe I'll gen up > some workloads on different filesystems and go nuts. whatcha got for storage and machines? [reply to me offline] -- craig .-... . -.-. .-. . --- . ... ... .- --. . Craig I. Hagan hagan(at)cih.com "Never let your sense of morals prevent you from doing what is right." - Mayor Hardin from Isaac Asimov's Foundation -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Craig I. Hagan INET: [EMAIL PROTECTED] Fat City Network Services-- 858-538-5051 http://www.fatcity.com San Diego, California-- Mailing list and web hosting services - To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
