Re: [Openstack] [Swift] Cache pressure tuning
Jonathan, we happen to use SN similar with yours and I could share you some performance testing data here: 1. 100 container with 1 objects(base) The performance is quite good and can hit HW bottleneck 2. 10kcontainer with 100M objects The performance is not so good, which dropped 80% compared with base 3. 1container with 1000M objects The performance is not so good, which dropped 95% compared with base The suspect reason we found are: 1) XFS's overhead W/ huge # of objects. Deleting some files wouldn't help since the inode allocation is quite sparse on the disks already and later inode_lookup should costs more disk seek time I guess. But this could be greatly improved by setting vfs_cache_presure to a lower value and it should be safe even we set it to 1 since Swift does use cache at all. If we could cache all the inode and the performance would become good again.We've done some tests with precached inode(simply run 'ls -R /srv/nodes') and verified the performance is quite good. 2) SQLite DB performance bottleneck when there are millions of records in a single DB. There is a BP to auto split large database but not implemented yet Hope this can help. -- Sincerely, Yuan On Tue, Jun 18, 2013 at 1:56 PM, Jonathan Lu jojokur...@gmail.com wrote: Hi, Huang Thanks a lot. I will try this test. One more question: In the 3 following situation, will the base line performance be quite different? 1. only 1 sontaienr with 10m objects; 2. 100,000 objects per container at 100 containers 3. 1,000 objects per container at 10,000 containers Cheers Jonathan Lu On 2013/6/18 12:54, Huang Zhiteng wrote: On Tue, Jun 18, 2013 at 12:35 PM, Jonathan Lu jojokur...@gmail.comwrote: Hi, Huang Thanks for you explanation. Does it mean that the storage cluster of specific processing ability will be slower and slower with more and more objects? Is there any test about the rate of the decline or is there any lower limit? For example, my environment is: Swift version : grizzly Tried on Ubuntu 12.04 3 Storage-nodes : each for 16GB-ram / CPU 4*2 / 3TB*12 The expected* *throughout is more than 100/s with uploaded objects of 50KB. At the beginning it works quite well and then it drops. If this degradation is unstoppable, I'm afraid that the performance will finally not be able to meet our needs no matter how I tuning other config. It won't be hard to do a base line performance (without inode cache) assessment of your system: populate your system with certain mount of objects with desired size (say 50k, 10million objects 1,000 objects per container at 10,000 containers), and *then drop VFS caches explicitly before testing*. Measure performance with your desired IO pattern and in the mean time drop VFS cache every once in a while (say every 60s). That's roughly the performance you can get when your storage system gets into a 'steady' state (i.e. objects # has out grown memory size). This will give you idea of pretty much the worst case. Jonathan Lu On 2013/6/18 11:05, Huang Zhiteng wrote: On Tue, Jun 18, 2013 at 10:42 AM, Jonathan Lu jojokur...@gmail.comwrote: On 2013/6/17 18:59, Robert van Leeuwen wrote: I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys /vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? Hi, When this is set to a lower value the kernel will try to keep the inode/dentry cache longer in memory. Since the swift replicator is scanning the filesystem continuously it will eat up a lot of iops if those are not in memory. To see if a lot of cache misses are happening, for xfs, you can look at xs_dir_lookup and xs_ig_missed. ( look at http://xfs.org/index.php/Runtime_Stats ) We greatly benefited from setting this to a low value but we have quite a lot of files on a node ( 30 million) Note that setting this to zero will result in the OOM killer killing the machine sooner or later. (especially if files are moved around due to a cluster change ;) Cheers, Robert van Leeuwen Hi, We set this to a low value(20) and the performance is better than before. It seems quite useful. According to your description, this issue is related with the object quantity in the storage. We delete all the objects in the storage but it doesn't help anything. The only method to recover is to format and re-mount the storage node. We try to install swift on different environment but this degradation problem seems to be an inevitable one. It's inode cache for each file(object) helps (reduce extra disk IOs). As long as your memory is big enough to hold inode information of those frequently accessed objects, you are good. And there's no need (no point) to limit # of objects for each storage node IMO. You may manually load inode information of objects into VFS cache if you
Re: [Openstack] [Swift] Cache pressure tuning
Jonathan, Yes we have 10,000 containers and we're using COSBench to do the tests. Sincerely, Yuan On Wed, Jun 19, 2013 at 9:24 AM, Jonathan Lu jojokur...@gmail.com wrote: Hi, Zhou, BTW, in test case 2, the number of container is 10,000 or just 10? Jonathan Lu On 2013/6/18 19:18, ZHOU Yuan wrote: Jonathan, we happen to use SN similar with yours and I could share you some performance testing data here: 1. 100 container with 1 objects(base) The performance is quite good and can hit HW bottleneck 2. 10kcontainer with 100M objects The performance is not so good, which dropped 80% compared with base 3. 1container with 1000M objects The performance is not so good, which dropped 95% compared with base The suspect reason we found are: 1) XFS's overhead W/ huge # of objects. Deleting some files wouldn't help since the inode allocation is quite sparse on the disks already and later inode_lookup should costs more disk seek time I guess. But this could be greatly improved by setting vfs_cache_presure to a lower value and it should be safe even we set it to 1 since Swift does use cache at all. If we could cache all the inode and the performance would become good again.We've done some tests with precached inode(simply run 'ls -R /srv/nodes') and verified the performance is quite good. 2) SQLite DB performance bottleneck when there are millions of records in a single DB. There is a BP to auto split large database but not implemented yet Hope this can help. -- Sincerely, Yuan On Tue, Jun 18, 2013 at 1:56 PM, Jonathan Lu jojokur...@gmail.com wrote: Hi, Huang Thanks a lot. I will try this test. One more question: In the 3 following situation, will the base line performance be quite different? 1. only 1 sontaienr with 10m objects; 2. 100,000 objects per container at 100 containers 3. 1,000 objects per container at 10,000 containers Cheers Jonathan Lu On 2013/6/18 12:54, Huang Zhiteng wrote: On Tue, Jun 18, 2013 at 12:35 PM, Jonathan Lu jojokur...@gmail.comwrote: Hi, Huang Thanks for you explanation. Does it mean that the storage cluster of specific processing ability will be slower and slower with more and more objects? Is there any test about the rate of the decline or is there any lower limit? For example, my environment is: Swift version : grizzly Tried on Ubuntu 12.04 3 Storage-nodes : each for 16GB-ram / CPU 4*2 / 3TB*12 The expected* *throughout is more than 100/s with uploaded objects of 50KB. At the beginning it works quite well and then it drops. If this degradation is unstoppable, I'm afraid that the performance will finally not be able to meet our needs no matter how I tuning other config. It won't be hard to do a base line performance (without inode cache) assessment of your system: populate your system with certain mount of objects with desired size (say 50k, 10million objects 1,000 objects per container at 10,000 containers), and *then drop VFS caches explicitly before testing*. Measure performance with your desired IO pattern and in the mean time drop VFS cache every once in a while (say every 60s). That's roughly the performance you can get when your storage system gets into a 'steady' state (i.e. objects # has out grown memory size). This will give you idea of pretty much the worst case. Jonathan Lu On 2013/6/18 11:05, Huang Zhiteng wrote: On Tue, Jun 18, 2013 at 10:42 AM, Jonathan Lu jojokur...@gmail.comwrote: On 2013/6/17 18:59, Robert van Leeuwen wrote: I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys /vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? Hi, When this is set to a lower value the kernel will try to keep the inode/dentry cache longer in memory. Since the swift replicator is scanning the filesystem continuously it will eat up a lot of iops if those are not in memory. To see if a lot of cache misses are happening, for xfs, you can look at xs_dir_lookup and xs_ig_missed. ( look at http://xfs.org/index.php/Runtime_Stats ) We greatly benefited from setting this to a low value but we have quite a lot of files on a node ( 30 million) Note that setting this to zero will result in the OOM killer killing the machine sooner or later. (especially if files are moved around due to a cluster change ;) Cheers, Robert van Leeuwen Hi, We set this to a low value(20) and the performance is better than before. It seems quite useful. According to your description, this issue is related with the object quantity in the storage. We delete all the objects in the storage but it doesn't help anything. The only method to recover is to format and re-mount the storage node. We try to install swift on different environment but this degradation problem seems to be an inevitable one.
[Openstack] [Swift] Cache pressure tuning
Hi, all, I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys/vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? I have scanned the Deployment Guide of OpenStack Swift http://docs.openstack.org/developer/swift/deployment_guide.html#general-server-configuration but cannot find anything about such tuning. Thanks, Jonathan Lu ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Cache pressure tuning
Check the slab usage of xfs_* might help . $ sudo slabtop Hope it help 從我的 iPhone 傳送 Jonathan Lu jojokur...@gmail.com 於 2013/6/17 下午3:25 寫道： Hi, all, I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys/vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? I have scanned the Deployment Guide of OpenStack Swift but cannot find anything about such tuning. Thanks, Jonathan Lu ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Cache pressure tuning
Can you say more about your environment and setup, and then about the measurements you are taking to categorize the performance issue? On Mon, Jun 17, 2013 at 3:25 AM, Jonathan Lu jojokur...@gmail.com wrote: Hi, all, I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys/vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? I have scanned the Deployment Guide of OpenStack Swift but cannot find anything about such tuning. Thanks, Jonathan Lu ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Cache pressure tuning
I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys /vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? Hi, When this is set to a lower value the kernel will try to keep the inode/dentry cache longer in memory. Since the swift replicator is scanning the filesystem continuously it will eat up a lot of iops if those are not in memory. To see if a lot of cache misses are happening, for xfs, you can look at xs_dir_lookup and xs_ig_missed. ( look at http://xfs.org/index.php/Runtime_Stats ) We greatly benefited from setting this to a low value but we have quite a lot of files on a node ( 30 million) Note that setting this to zero will result in the OOM killer killing the machine sooner or later. (especially if files are moved around due to a cluster change ;) Cheers, Robert van Leeuwen ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Cache pressure tuning
On 2013/6/17 18:59, Robert van Leeuwen wrote: I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys /vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? Hi, When this is set to a lower value the kernel will try to keep the inode/dentry cache longer in memory. Since the swift replicator is scanning the filesystem continuously it will eat up a lot of iops if those are not in memory. To see if a lot of cache misses are happening, for xfs, you can look at xs_dir_lookup and xs_ig_missed. ( look at http://xfs.org/index.php/Runtime_Stats ) We greatly benefited from setting this to a low value but we have quite a lot of files on a node ( 30 million) Note that setting this to zero will result in the OOM killer killing the machine sooner or later. (especially if files are moved around due to a cluster change ;) Cheers, Robert van Leeuwen Hi, We set this to a low value(20) and the performance is better than before. It seems quite useful. According to your description, this issue is related with the object quantity in the storage. We delete all the objects in the storage but it doesn't help anything. The only method to recover is to format and re-mount the storage node. We try to install swift on different environment but this degradation problem seems to be an inevitable one. Cheers, Jonathan Lu ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Cache pressure tuning
On Tue, Jun 18, 2013 at 10:42 AM, Jonathan Lu jojokur...@gmail.com wrote: On 2013/6/17 18:59, Robert van Leeuwen wrote: I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys /vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? Hi, When this is set to a lower value the kernel will try to keep the inode/dentry cache longer in memory. Since the swift replicator is scanning the filesystem continuously it will eat up a lot of iops if those are not in memory. To see if a lot of cache misses are happening, for xfs, you can look at xs_dir_lookup and xs_ig_missed. ( look at http://xfs.org/index.php/**Runtime_Statshttp://xfs.org/index.php/Runtime_Stats) We greatly benefited from setting this to a low value but we have quite a lot of files on a node ( 30 million) Note that setting this to zero will result in the OOM killer killing the machine sooner or later. (especially if files are moved around due to a cluster change ;) Cheers, Robert van Leeuwen Hi, We set this to a low value(20) and the performance is better than before. It seems quite useful. According to your description, this issue is related with the object quantity in the storage. We delete all the objects in the storage but it doesn't help anything. The only method to recover is to format and re-mount the storage node. We try to install swift on different environment but this degradation problem seems to be an inevitable one. It's inode cache for each file(object) helps (reduce extra disk IOs). As long as your memory is big enough to hold inode information of those frequently accessed objects, you are good. And there's no need (no point) to limit # of objects for each storage node IMO. You may manually load inode information of objects into VFS cache if you like (by simply 'ls' files), to _restore_ performance. But still memory size and object access pattern are the key to this kind of performance tuning, if memory is too small, inode cache will be invalided sooner or later. Cheers, Jonathan Lu __**_ Mailing list: https://launchpad.net/~**openstackhttps://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~**openstackhttps://launchpad.net/~openstack More help : https://help.launchpad.net/**ListHelphttps://help.launchpad.net/ListHelp -- Regards Huang Zhiteng ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Cache pressure tuning
Hi Jonathan , How did you perform delete all the objects in the storage ? Those deleted objects still consume inodes in tombstone status until the reclaim time. Would you mind to compare the result of $ sudo cat /proc/slabinfo | grep xfs , before/after set the vfs_cache_pressure spongebob@patrick1:~$ sudo cat /proc/slabinfo | grep xfs xfs_ili70153 70182216 181 : tunables00 0 : slabdata 3899 3899 0 xfs_inode 169738 170208 1024 164 : tunables000 : slabdata 10638 10638 0 xfs_efd_item 60 60400 202 : tunables000 : slabdata 3 3 0 xfs_buf_item 234234224 181 : tunables000 : slabdata 13 13 0 xfs_trans 28 28280 141 : tunables000 : slabdata 2 2 0 xfs_da_state 32 32488 162 : tunables000 : slabdata 2 2 0 xfs_btree_cur 38 38208 191 : tunables000 : slabdata 2 2 0 xfs_log_ticket40 40200 201 : tunables000 : slabdata 2 2 0 Hi Robert, The performance degradation still there even only main swift workers are running in storage node. ( stop replicator/updater/auditor ). In my knowing. I'll check xs_dir_lookup and xs_ig_missed here. Thanks +Hugo Kuo+ h...@swiftstack.com tonyt...@gmail.com +886 935004793 2013/6/18 Jonathan Lu jojokur...@gmail.com On 2013/6/17 18:59, Robert van Leeuwen wrote: I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys /vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? Hi, When this is set to a lower value the kernel will try to keep the inode/dentry cache longer in memory. Since the swift replicator is scanning the filesystem continuously it will eat up a lot of iops if those are not in memory. To see if a lot of cache misses are happening, for xfs, you can look at xs_dir_lookup and xs_ig_missed. ( look at http://xfs.org/index.php/**Runtime_Statshttp://xfs.org/index.php/Runtime_Stats) We greatly benefited from setting this to a low value but we have quite a lot of files on a node ( 30 million) Note that setting this to zero will result in the OOM killer killing the machine sooner or later. (especially if files are moved around due to a cluster change ;) Cheers, Robert van Leeuwen Hi, We set this to a low value(20) and the performance is better than before. It seems quite useful. According to your description, this issue is related with the object quantity in the storage. We delete all the objects in the storage but it doesn't help anything. The only method to recover is to format and re-mount the storage node. We try to install swift on different environment but this degradation problem seems to be an inevitable one. Cheers, Jonathan Lu __**_ Mailing list: https://launchpad.net/~**openstackhttps://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~**openstackhttps://launchpad.net/~openstack More help : https://help.launchpad.net/**ListHelphttps://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Cache pressure tuning
Hi Hugo, I know the tombstone mechanism. In my opinion after the reclaim time, the object of xxx.tombstone will be deleted at all. Is that right? Maybe I misunderstand the doc :( ... We try to colddown the swift system ( just wait for the reclaiming) and test, but the result is not satisfying. Thanks, Jonathan Lu On 2013/6/18 11:04, Kuo Hugo wrote: Hi Jonathan , How did you perform delete all the objects in the storage ? Those deleted objects still consume inodes in tombstone status until the reclaim time. Would you mind to compare the result of $ sudo cat /proc/slabinfo | grep xfs , before/after set the vfs_cache_pressure spongebob@patrick1:~$ sudo cat /proc/slabinfo | grep xfs xfs_ili 70153 70182 216 181 : tunables000 : slabdata 3899 3899 0 xfs_inode 169738 1702081024 164 : tunables000 : slabdata 10638 10638 0 xfs_efd_item 60 60400 202 : tunables00 0 : slabdata 3 3 0 xfs_buf_item 234234224 181 : tunables00 0 : slabdata 13 13 0 xfs_trans 28 28280 141 : tunables00 0 : slabdata 2 2 0 xfs_da_state 32 32488 162 : tunables00 0 : slabdata 2 2 0 xfs_btree_cur 38 38208 191 : tunables00 0 : slabdata 2 2 0 xfs_log_ticket40 40200 201 : tunables00 0 : slabdata 2 2 0 Hi Robert, The performance degradation still there even only main swift workers are running in storage node. ( stop replicator/updater/auditor ). In my knowing. I'll check xs_dir_lookup and xs_ig_missed here. Thanks +Hugo Kuo+ h...@swiftstack.com mailto:h...@swiftstack.com tonyt...@gmail.com mailto:tonyt...@gmail.com +886 935004793 2013/6/18 Jonathan Lu jojokur...@gmail.com mailto:jojokur...@gmail.com On 2013/6/17 18:59, Robert van Leeuwen wrote: I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys /vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? Hi, When this is set to a lower value the kernel will try to keep the inode/dentry cache longer in memory. Since the swift replicator is scanning the filesystem continuously it will eat up a lot of iops if those are not in memory. To see if a lot of cache misses are happening, for xfs, you can look at xs_dir_lookup and xs_ig_missed. ( look at http://xfs.org/index.php/Runtime_Stats ) We greatly benefited from setting this to a low value but we have quite a lot of files on a node ( 30 million) Note that setting this to zero will result in the OOM killer killing the machine sooner or later. (especially if files are moved around due to a cluster change ;) Cheers, Robert van Leeuwen Hi, We set this to a low value(20) and the performance is better than before. It seems quite useful. According to your description, this issue is related with the object quantity in the storage. We delete all the objects in the storage but it doesn't help anything. The only method to recover is to format and re-mount the storage node. We try to install swift on different environment but this degradation problem seems to be an inevitable one. Cheers, Jonathan Lu ___ Mailing list: https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Cache pressure tuning
Hi Huang, Storage nodes will run out of memory for caching inode eventually right? Have you ever measure the upper limit of caching capability of your storage nodes? Hi Jonathan, The default reclaiming time is 7 days. Did you wait for 7 days or just change the setting to 1 min in conf file ? Hugo +Hugo Kuo+ h...@swiftstack.com tonyt...@gmail.com +886 935004793 2013/6/18 Jonathan Lu jojokur...@gmail.com Hi Hugo, I know the tombstone mechanism. In my opinion after the reclaim time, the object of xxx.tombstone will be deleted at all. Is that right? Maybe I misunderstand the doc :( ... We try to colddown the swift system ( just wait for the reclaiming) and test, but the result is not satisfying. Thanks, Jonathan Lu On 2013/6/18 11:04, Kuo Hugo wrote: Hi Jonathan , How did you perform delete all the objects in the storage ? Those deleted objects still consume inodes in tombstone status until the reclaim time. Would you mind to compare the result of $ sudo cat /proc/slabinfo | grep xfs , before/after set the vfs_cache_pressure spongebob@patrick1:~$ sudo cat /proc/slabinfo | grep xfs xfs_ili70153 70182216 181 : tunables00 0 : slabdata 3899 3899 0 xfs_inode 169738 170208 1024 164 : tunables00 0 : slabdata 10638 10638 0 xfs_efd_item 60 60400 202 : tunables000 : slabdata 3 3 0 xfs_buf_item 234234224 181 : tunables000 : slabdata 13 13 0 xfs_trans 28 28280 141 : tunables000 : slabdata 2 2 0 xfs_da_state 32 32488 162 : tunables000 : slabdata 2 2 0 xfs_btree_cur 38 38208 191 : tunables000 : slabdata 2 2 0 xfs_log_ticket40 40200 201 : tunables000 : slabdata 2 2 0 Hi Robert, The performance degradation still there even only main swift workers are running in storage node. ( stop replicator/updater/auditor ). In my knowing. I'll check xs_dir_lookup and xs_ig_missed here. Thanks +Hugo Kuo+ h...@swiftstack.com tonyt...@gmail.com +886 935004793 2013/6/18 Jonathan Lu jojokur...@gmail.com On 2013/6/17 18:59, Robert van Leeuwen wrote: I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys /vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? Hi, When this is set to a lower value the kernel will try to keep the inode/dentry cache longer in memory. Since the swift replicator is scanning the filesystem continuously it will eat up a lot of iops if those are not in memory. To see if a lot of cache misses are happening, for xfs, you can look at xs_dir_lookup and xs_ig_missed. ( look at http://xfs.org/index.php/Runtime_Stats ) We greatly benefited from setting this to a low value but we have quite a lot of files on a node ( 30 million) Note that setting this to zero will result in the OOM killer killing the machine sooner or later. (especially if files are moved around due to a cluster change ;) Cheers, Robert van Leeuwen Hi, We set this to a low value(20) and the performance is better than before. It seems quite useful. According to your description, this issue is related with the object quantity in the storage. We delete all the objects in the storage but it doesn't help anything. The only method to recover is to format and re-mount the storage node. We try to install swift on different environment but this degradation problem seems to be an inevitable one. Cheers, Jonathan Lu ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Cache pressure tuning
Hi, Huang Thanks for you explanation. Does it mean that the storage cluster of specific processing ability will be slower and slower with more and more objects? Is there any test about the rate of the decline or is there any lower limit? For example, my environment is: Swift version : grizzly Tried on Ubuntu 12.04 3 Storage-nodes : each for 16GB-ram / CPU 4*2 / 3TB*12 The expected**throughout is more than 100/s with uploaded objects of 50KB. At the beginning it works quite well and then it drops. If this degradation is unstoppable, I'm afraid that the performance will finally not be able to meet our needs no matter how I tuning other config. Jonathan Lu On 2013/6/18 11:05, Huang Zhiteng wrote: On Tue, Jun 18, 2013 at 10:42 AM, Jonathan Lu jojokur...@gmail.com mailto:jojokur...@gmail.com wrote: On 2013/6/17 18:59, Robert van Leeuwen wrote: I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys /vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? Hi, When this is set to a lower value the kernel will try to keep the inode/dentry cache longer in memory. Since the swift replicator is scanning the filesystem continuously it will eat up a lot of iops if those are not in memory. To see if a lot of cache misses are happening, for xfs, you can look at xs_dir_lookup and xs_ig_missed. ( look at http://xfs.org/index.php/Runtime_Stats ) We greatly benefited from setting this to a low value but we have quite a lot of files on a node ( 30 million) Note that setting this to zero will result in the OOM killer killing the machine sooner or later. (especially if files are moved around due to a cluster change ;) Cheers, Robert van Leeuwen Hi, We set this to a low value(20) and the performance is better than before. It seems quite useful. According to your description, this issue is related with the object quantity in the storage. We delete all the objects in the storage but it doesn't help anything. The only method to recover is to format and re-mount the storage node. We try to install swift on different environment but this degradation problem seems to be an inevitable one. It's inode cache for each file(object) helps (reduce extra disk IOs). As long as your memory is big enough to hold inode information of those frequently accessed objects, you are good. And there's no need (no point) to limit # of objects for each storage node IMO. You may manually load inode information of objects into VFS cache if you like (by simply 'ls' files), to _restore_ performance. But still memory size and object access pattern are the key to this kind of performance tuning, if memory is too small, inode cache will be invalided sooner or later. Cheers, Jonathan Lu ___ Mailing list: https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp -- Regards Huang Zhiteng ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Cache pressure tuning
On Tue, Jun 18, 2013 at 12:35 PM, Jonathan Lu jojokur...@gmail.com wrote: Hi, Huang Thanks for you explanation. Does it mean that the storage cluster of specific processing ability will be slower and slower with more and more objects? Is there any test about the rate of the decline or is there any lower limit? For example, my environment is: Swift version : grizzly Tried on Ubuntu 12.04 3 Storage-nodes : each for 16GB-ram / CPU 4*2 / 3TB*12 The expected* *throughout is more than 100/s with uploaded objects of 50KB. At the beginning it works quite well and then it drops. If this degradation is unstoppable, I'm afraid that the performance will finally not be able to meet our needs no matter how I tuning other config. It won't be hard to do a base line performance (without inode cache) assessment of your system: populate your system with certain mount of objects with desired size (say 50k, 10million objects 1,000 objects per container at 10,000 containers), and *then drop VFS caches explicitly before testing*. Measure performance with your desired IO pattern and in the mean time drop VFS cache every once in a while (say every 60s). That's roughly the performance you can get when your storage system gets into a 'steady' state (i.e. objects # has out grown memory size). This will give you idea of pretty much the worst case. Jonathan Lu On 2013/6/18 11:05, Huang Zhiteng wrote: On Tue, Jun 18, 2013 at 10:42 AM, Jonathan Lu jojokur...@gmail.comwrote: On 2013/6/17 18:59, Robert van Leeuwen wrote: I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys /vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? Hi, When this is set to a lower value the kernel will try to keep the inode/dentry cache longer in memory. Since the swift replicator is scanning the filesystem continuously it will eat up a lot of iops if those are not in memory. To see if a lot of cache misses are happening, for xfs, you can look at xs_dir_lookup and xs_ig_missed. ( look at http://xfs.org/index.php/Runtime_Stats ) We greatly benefited from setting this to a low value but we have quite a lot of files on a node ( 30 million) Note that setting this to zero will result in the OOM killer killing the machine sooner or later. (especially if files are moved around due to a cluster change ;) Cheers, Robert van Leeuwen Hi, We set this to a low value(20) and the performance is better than before. It seems quite useful. According to your description, this issue is related with the object quantity in the storage. We delete all the objects in the storage but it doesn't help anything. The only method to recover is to format and re-mount the storage node. We try to install swift on different environment but this degradation problem seems to be an inevitable one. It's inode cache for each file(object) helps (reduce extra disk IOs). As long as your memory is big enough to hold inode information of those frequently accessed objects, you are good. And there's no need (no point) to limit # of objects for each storage node IMO. You may manually load inode information of objects into VFS cache if you like (by simply 'ls' files), to _restore_ performance. But still memory size and object access pattern are the key to this kind of performance tuning, if memory is too small, inode cache will be invalided sooner or later. Cheers, Jonathan Lu ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Regards Huang Zhiteng -- Regards Huang Zhiteng ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Cache pressure tuning
Hi, Huang Thanks a lot. I will try this test. One more question: In the 3 following situation, will the base line performance be quite different? 1. only 1 sontaienr with 10m objects; 2. 100,000 objects per container at 100 containers 3. 1,000 objects per container at 10,000 containers Cheers Jonathan Lu On 2013/6/18 12:54, Huang Zhiteng wrote: On Tue, Jun 18, 2013 at 12:35 PM, Jonathan Lu jojokur...@gmail.com mailto:jojokur...@gmail.com wrote: Hi, Huang Thanks for you explanation. Does it mean that the storage cluster of specific processing ability will be slower and slower with more and more objects? Is there any test about the rate of the decline or is there any lower limit? For example, my environment is: Swift version : grizzly Tried on Ubuntu 12.04 3 Storage-nodes : each for 16GB-ram / CPU 4*2 / 3TB*12 The expected**throughout is more than 100/s with uploaded objects of 50KB. At the beginning it works quite well and then it drops. If this degradation is unstoppable, I'm afraid that the performance will finally not be able to meet our needs no matter how I tuning other config. It won't be hard to do a base line performance (without inode cache) assessment of your system: populate your system with certain mount of objects with desired size (say 50k, 10million objects 1,000 objects per container at 10,000 containers), and *then drop VFS caches explicitly before testing*. Measure performance with your desired IO pattern and in the mean time drop VFS cache every once in a while (say every 60s). That's roughly the performance you can get when your storage system gets into a 'steady' state (i.e. objects # has out grown memory size). This will give you idea of pretty much the worst case. Jonathan Lu On 2013/6/18 11:05, Huang Zhiteng wrote: On Tue, Jun 18, 2013 at 10:42 AM, Jonathan Lu jojokur...@gmail.com mailto:jojokur...@gmail.com wrote: On 2013/6/17 18:59, Robert van Leeuwen wrote: I'm facing the issue about the performance degradation, and once I glanced that changing the value in /proc/sys /vm/vfs_cache_pressure will do a favour. Can anyone explain to me whether and why it is useful? Hi, When this is set to a lower value the kernel will try to keep the inode/dentry cache longer in memory. Since the swift replicator is scanning the filesystem continuously it will eat up a lot of iops if those are not in memory. To see if a lot of cache misses are happening, for xfs, you can look at xs_dir_lookup and xs_ig_missed. ( look at http://xfs.org/index.php/Runtime_Stats ) We greatly benefited from setting this to a low value but we have quite a lot of files on a node ( 30 million) Note that setting this to zero will result in the OOM killer killing the machine sooner or later. (especially if files are moved around due to a cluster change ;) Cheers, Robert van Leeuwen Hi, We set this to a low value(20) and the performance is better than before. It seems quite useful. According to your description, this issue is related with the object quantity in the storage. We delete all the objects in the storage but it doesn't help anything. The only method to recover is to format and re-mount the storage node. We try to install swift on different environment but this degradation problem seems to be an inevitable one. It's inode cache for each file(object) helps (reduce extra disk IOs). As long as your memory is big enough to hold inode information of those frequently accessed objects, you are good. And there's no need (no point) to limit # of objects for each storage node IMO. You may manually load inode information of objects into VFS cache if you like (by simply 'ls' files), to _restore_ performance. But still memory size and object access pattern are the key to this kind of performance tuning, if memory is too small, inode cache will be invalided sooner or later. Cheers, Jonathan Lu ___ Mailing list: https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp -- Regards Huang Zhiteng -- Regards Huang Zhiteng ___ Mailing list:
