Re: [Ocfs2-users] ocfs2 vs ext3?
On Fri, May 01, 2009 at 08:55:21PM +1000, Andrew (Anything) wrote: > > echo 'stats'|debugfs.ocfs2 /dev/sdX|grep "Incompat" > > I re-read your reply with the Incompat flag check and it made me think to > check what mine were showing. > > I found that my format with max-features meant that it wasn't using all the > available features. > Some are probably quite important for performance. > > mkfs.ocfs2 -T mail --fs-feature-level=max-features /dev/sda > debugfs.ocfs2 stats was showing: > Feature Compat: 1 BackupSuper > Feature Incompat: 272 Sparse ExtendedSlotMap > > I downloaded the ocfs2-tools via git, instead of the 1.4.1 source file on > the ocfs website, and reformatted my partition with the following options > (just in case) Yeah, 1.4.1 tools doesn't know about many of the new features. > mkfs.ocfs2 -T mail --fs-feature-level=max-features > --fs-features=backup-super,sparse,unwritten,inline-data,noxattr /dev/sda You don't need to specify any of the features here - if mkfs knows about them, max-features will pull them in. The only thing on your fs-features line that isn't covered is noxattr (which turns off xattr, obviously). > debugfs.ocfs2 stats now shows: > Feature Compat: 3 BackupSuper JBD2SuperBlock > Feature Incompat: 2384 Sparse ExtendedSlotMap InlineData MetaECC > > > Performance is considerably better when tested with bonnie++ -n > 25:1024:0:10, probably because of the inline data. Inline-data will be the clear winner, yes. Also, mainline has some inode allocation changes that help with lots of files. Don't know if that affects you. > Are there any other features im missing that might effect performance? We just landed indexed directories in mainline. Tools support isn't done yet, though, so you can't enable it without grabbing the appropriate development code. > And what does the incompat flags mean? > I had mistaken them to mean that these features are available but > incompatible with this partition, ie: not used. Nope. If it's not used on the partition, it's not set. The feature set scheme comes from extN. "Incompat" features are features that can only be accessed by a filesystem driver that understands them. So if you load an older ocfs2.ko, and it does not understand an incompat feature, it will not mount the filesystem. Joel -- "Nobody loves me, Nobody seems to care. Troubles and worries, people, You know I've had my share." Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2 vs ext3?
Hi Tao > echo 'stats'|debugfs.ocfs2 /dev/sdX|grep "Incompat" I re-read your reply with the Incompat flag check and it made me think to check what mine were showing. I found that my format with max-features meant that it wasn't using all the available features. Some are probably quite important for performance. mkfs.ocfs2 -T mail --fs-feature-level=max-features /dev/sda debugfs.ocfs2 stats was showing: Feature Compat: 1 BackupSuper Feature Incompat: 272 Sparse ExtendedSlotMap I downloaded the ocfs2-tools via git, instead of the 1.4.1 source file on the ocfs website, and reformatted my partition with the following options (just in case) mkfs.ocfs2 -T mail --fs-feature-level=max-features --fs-features=backup-super,sparse,unwritten,inline-data,noxattr /dev/sda debugfs.ocfs2 stats now shows: Feature Compat: 3 BackupSuper JBD2SuperBlock Feature Incompat: 2384 Sparse ExtendedSlotMap InlineData MetaECC Performance is considerably better when tested with bonnie++ -n 25:1024:0:10, probably because of the inline data. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP before 25:1024:0/10 1721 39 + +++ 2655 13 1806 44 + +++ 1995 19 before 50:1024:0/10 1645 62 976 8 1586 11 1469 56 120 1 336 5 after 25:1024:0/10 4015 83 + +++ 3119 12 3872 80 + +++ 1953 13 after 50:1024:0/10 2486 80 99692 100 2737 14 2694 88 83541 98 615 5 Are there any other features im missing that might effect performance? And what does the incompat flags mean? I had mistaken them to mean that these features are available but incompatible with this partition, ie: not used. Andy.. ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2 vs ext3?
Andrew (Anything) wrote: > Ive been testing using bonnie++ -n 50:1024:0:10 -s 0. Is this a bad way to > test? > Some raw results follow later in case you want them. > > Obviously ocfs2 should be slower than ext3. > But I guess I expected a single node ocfs node to be only doing internal > stuff with kernel and dlm at really fast cpu speeds, and its only bottleneck > to be writing to the disk. For it to be so slow it must be doing heaps of > disk stuff instead? > > Had tried a few dd tests however oflag=direct seems to cause an instant > kernel panic, I don't know if I am to trust dd's results without directio. > > Andy.. > > > ext3, noatime, > bonnie++ -d /mnt/temp/ -n 50:1024:0:10 -s 0 > Version 1.03e --Sequential Create-- Random > Create > wombat -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files:max/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec > %CP >50:1024:0/10 10838 43 + +++ 20386 50 7147 28 + +++ 17248 > 44 > > > ocfs2, -T mail max-features, noatime,data=writeback, > bonnie++ -d /mnt/temp/ -n 50:1024:0:10 -s 0 > Version 1.03e --Sequential Create-- Random > Create > wombat -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files:max/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec > %CP >50:1024:0/10 1429 53 10849 32 1224 8 1354 51 205 2 292 > 4 > > I ran both a few times just in case. Yes, it is doing heaps more disk io compared to ext3 simply due to the fact the ext3's inode is 128 bytes whereas ocfs2's is 1 block. So choosing a smaller block size will improve create performance. However, that is not recommended because smaller block sizes will negatively affect the r/w performance. Have you tried running bonnie on multiple nodes concurrently? The create performance will scale up to the limit of your io subsystem. ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2 vs ext3?
Hi Tao. > > on a single node system (with no network latency, and only itself to > talk > > to) I had expected better results. > > is a 10x reduction in file creation/modification an expected result? > could you please talk a little more about the test case? > Ive been testing using bonnie++ -n 50:1024:0:10 -s 0. Is this a bad way to test? Some raw results follow later in case you want them. Obviously ocfs2 should be slower than ext3. But I guess I expected a single node ocfs node to be only doing internal stuff with kernel and dlm at really fast cpu speeds, and its only bottleneck to be writing to the disk. For it to be so slow it must be doing heaps of disk stuff instead? Had tried a few dd tests however oflag=direct seems to cause an instant kernel panic, I don't know if I am to trust dd's results without directio. Andy.. ext3, noatime, bonnie++ -d /mnt/temp/ -n 50:1024:0:10 -s 0 Version 1.03e --Sequential Create-- Random Create wombat -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 50:1024:0/10 10838 43 + +++ 20386 50 7147 28 + +++ 17248 44 ocfs2, -T mail max-features, noatime,data=writeback, bonnie++ -d /mnt/temp/ -n 50:1024:0:10 -s 0 Version 1.03e --Sequential Create-- Random Create wombat -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 50:1024:0/10 1429 53 10849 32 1224 8 1354 51 205 2 292 4 I ran both a few times just in case. ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2 vs ext3?
Dear Joel, so how slow of OCFS2 actually? _ Thomas Lau Product Engineer Product Development NTT Com Asia Limited Phone: +852 3793-0733 Address: 12/F., Guardian House, 32 Oi Kwan Road, Wan Chai, Hong Kong Email: thomas@ntt.com.hk Joel Becker wrote: On Wed, Apr 29, 2009 at 05:24:11PM +1000, Andrew (Anything) wrote: I just checked max-features, it doesn't include "local" which means that you still need to create dlm lock in your local node which will cost some delay. You can check whether your volume enable "local" by command echo 'stats'|debugfs.ocfs2 /dev/sdX|grep "Incompat" If the volume is mkfsed to used as local disk, you should see "local". Otherwise you need to add "--fs-features=local" to your mkfs. Yeah, 'local' is a feature, but it's not a feature in any default list because we generally expect to be used in a cluster :-) Thanks for your replies Tao and Thomas I still intend on using the filesystem in a 3 node cluster, but when I found it to be 25x slower than a single ext3 part I thought id start troubleshooting at the start. I just did a format of a small partition to give it a quick test anyway, and found that it performs just as well as ext3, in some test even better. Except if I understand correctly, local disables the ability for clustering completely? There are a lot of safety concerns we have to make just in case there are other nodes in the cluster. When in 'local' mode, those checks are all disabled, and that's why we can run as fast as any other local filesystem. In a cluster mode, we have to have those checks enabled, and it does cause a speed penalty. But remember, you're gaining the concurrent access on other nodes. That's the tradeoff. And I think we compare favorably in speed to other cluster filesystems. Joel ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2 vs ext3?
On Wed, Apr 29, 2009 at 05:24:11PM +1000, Andrew (Anything) wrote: > > I just checked max-features, it doesn't include "local" which means > > that you still need to create dlm lock in your local node which will > > cost some delay. You can check whether your volume enable "local" by > > command > > > > echo 'stats'|debugfs.ocfs2 /dev/sdX|grep "Incompat" > > > > If the volume is mkfsed to used as local disk, you should see "local". > > Otherwise you need to add "--fs-features=local" to your mkfs. Yeah, 'local' is a feature, but it's not a feature in any default list because we generally expect to be used in a cluster :-) > Thanks for your replies Tao and Thomas > > I still intend on using the filesystem in a 3 node cluster, but when I found > it to be 25x slower than a single ext3 part I thought id start > troubleshooting at the start. > > I just did a format of a small partition to give it a quick test anyway, and > found that it performs just as well as ext3, in some test even better. > > Except if I understand correctly, local disables the ability for clustering > completely? There are a lot of safety concerns we have to make just in case there are other nodes in the cluster. When in 'local' mode, those checks are all disabled, and that's why we can run as fast as any other local filesystem. In a cluster mode, we have to have those checks enabled, and it does cause a speed penalty. But remember, you're gaining the concurrent access on other nodes. That's the tradeoff. And I think we compare favorably in speed to other cluster filesystems. Joel -- "We'd better get back, `cause it'll be dark soon, and they mostly come at night. Mostly." Joel Becker Principal Software Developer Oracle E-mail: joel.bec...@oracle.com Phone: (650) 506-8127 ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2 vs ext3?
Andrew (Anything) wrote: >> Hi Andrew, >> I just checked max-features, it doesn't include "local" which means >> that you still need to create dlm lock in your local node which will >> cost some delay. You can check whether your volume enable "local" by >> command >> >> echo 'stats'|debugfs.ocfs2 /dev/sdX|grep "Incompat" >> >> If the volume is mkfsed to used as local disk, you should see "local". >> Otherwise you need to add "--fs-features=local" to your mkfs. >> >> Regards, >> Tao > > Thanks for your replies Tao and Thomas > > I still intend on using the filesystem in a 3 node cluster, but when I found > it to be 25x slower than a single ext3 part I thought id start > troubleshooting at the start. > > I just did a format of a small partition to give it a quick test anyway, and > found that it performs just as well as ext3, in some test even better. > > Except if I understand correctly, local disables the ability for clustering > completely? yes, local means you will use it locally, so other nodes can't use it. > >> Dear Andrew, >> >> I think the result should be normal because ocfs2 is a cluster fs ? >> > > Hi Thomas, > > > on a single node system (with no network latency, and only itself to talk > to) I had expected better results. > is a 10x reduction in file creation/modification an expected result? could you please talk a little more about the test case? Just FYI. I have just committed a new series of patches to 2.6.30 which will improve inode allocation a lot. See http://oss.oracle.com/pipermail/ocfs2-devel/2009-January/003799.html It will save you a lot of time when you create a large amount of file, delete them and then recreate. The above link can show you the test result. Regards, Tao ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2 vs ext3?
> > Hi Andrew, > I just checked max-features, it doesn't include "local" which means > that you still need to create dlm lock in your local node which will > cost some delay. You can check whether your volume enable "local" by > command > > echo 'stats'|debugfs.ocfs2 /dev/sdX|grep "Incompat" > > If the volume is mkfsed to used as local disk, you should see "local". > Otherwise you need to add "--fs-features=local" to your mkfs. > > Regards, > Tao Thanks for your replies Tao and Thomas I still intend on using the filesystem in a 3 node cluster, but when I found it to be 25x slower than a single ext3 part I thought id start troubleshooting at the start. I just did a format of a small partition to give it a quick test anyway, and found that it performs just as well as ext3, in some test even better. Except if I understand correctly, local disables the ability for clustering completely? > > Dear Andrew, > > I think the result should be normal because ocfs2 is a cluster fs ? > Hi Thomas, on a single node system (with no network latency, and only itself to talk to) I had expected better results. is a 10x reduction in file creation/modification an expected result? Andy.. ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2 vs ext3?
Dear Andrew, I think the result should be normal because ocfs2 is a cluster fs ? _ Thomas Lau Product Engineer Product Development NTT Com Asia Limited Phone: +852 3793-0733 Address: 12/F., Guardian House, 32 Oi Kwan Road, Wan Chai, Hong Kong Email: thomas@ntt.com.hk Andrew (Anything) wrote: > > Hi > > I’ve seen in various places that ocfs2 on a single node can perform > almost as well as en ext3 partition. > > > However on any test I do with bonnie++ my ocfs2 test is 10x slower > than ext3. > > ext3: 15000+ random file creations > > ocfs2: 1500 random file creations. > > I created the ext3 disk with defaults, and the ocfs2 disk with T mail > --fs-feature-level=max-features, and mounted with data=writeback,noatime. > > I updated to 2.6.29.1 before I ran any tests in case it helped. > > Can someone tell me where I am going wrong? > > Andy.. > > > > ___ > Ocfs2-users mailing list > Ocfs2-users@oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
Re: [Ocfs2-users] ocfs2 vs ext3?
Hi Andrew, I just checked max-features, it doesn't include "local" which means that you still need to create dlm lock in your local node which will cost some delay. You can check whether your volume enable "local" by command echo 'stats'|debugfs.ocfs2 /dev/sdX|grep "Incompat" If the volume is mkfsed to used as local disk, you should see "local". Otherwise you need to add "--fs-features=local" to your mkfs. Regards, Tao Andrew (Anything) wrote: > > > Hi > > > > I’ve seen in various places that ocfs2 on a single node can perform > almost as well as en ext3 partition. > > > However on any test I do with bonnie++ my ocfs2 test is 10x slower than > ext3. > > > > ext3: 15000+ random file creations > > ocfs2: 1500 random file creations. > > > > I created the ext3 disk with defaults, and the ocfs2 disk with T mail > --fs-feature-level=max-features, and mounted with data=writeback,noatime. > > I updated to 2.6.29.1 before I ran any tests in case it helped. > > > > Can someone tell me where I am going wrong? > > > > > > Andy.. > > > > > > > ___ > Ocfs2-users mailing list > Ocfs2-users@oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
[Ocfs2-users] ocfs2 vs ext3?
Hi I've seen in various places that ocfs2 on a single node can perform almost as well as en ext3 partition. However on any test I do with bonnie++ my ocfs2 test is 10x slower than ext3. ext3: 15000+ random file creations ocfs2: 1500 random file creations. I created the ext3 disk with defaults, and the ocfs2 disk with T mail --fs-feature-level=max-features, and mounted with data=writeback,noatime. I updated to 2.6.29.1 before I ran any tests in case it helped. Can someone tell me where I am going wrong? Andy.. ___ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
