Re: [Openstack] [SWIFT] raising network traffic on the storage node
If the replication traffic is responsible for this raising network traffic for only 1.200.000 objects, how much traffic I can expect if I have 100.000.000 objects stored? The average size of my mailobjects are 120 kB. It's planned to use all 12 hard drive slots of my DELL R720xd with 4 TB drives . I have 5 storage nodes and 2 balanced proxy nodes. Will the replication traffic kill my system? We are running with 400.000.000 objects across 11 object storage nodes. Total network traffic on any of those nodes is less then 10 MByte /second However we have seen slowdowns with lots of small files and really big disks. The issue is not related to the network but the local filesystem/disk. When the inode cache gets insufficient you can see terrible slow-downs. There have been a few threads about that in this list, having a lot of memory usually helps a bit. Cheers, Robert van Leeuwen ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [SWIFT] raising network traffic on the storage node
I measured the network traffic with darkstat: server In Out Total storagenode1699,118,562 679,077,971 1,378,196,533 storagenode2168,636,360 165,050,575 333,686,935 storagenode3166,583,442 164,405,402 330,988,844 storagenode4164,282,250 163,051,416 327,333,666 storagenode5164,000,162 162,840,370 326,840,532 proxynode1 7,339,629 31,253,205 38,592,834 proxynode2 8,236,128 12,517,594 20,753,722 This is a part of traffic to server storagenode3: PortIn Out Total Syns 600021,055,732 347,350,916 368,406,648 47,388 600119,717,608 18,090,656 37,808,264 31,549 6002494,124 316,830 810,954 883 36905 39,660 2,263 41,923 0 44687 33,056 1,944 35,000 0 47388 31,691 2,467 34,158 0 41999 30,626 1,788 32,414 0 34228 26,552 3,345 29,897 0 Is this correct configured? -Ursprüngliche Nachricht- Von: Openstack [mailto:openstack-bounces+klaus.schuermann=mediabeam@lists.launchpad.net] Im Auftrag von Robert van Leeuwen Gesendet: Dienstag, 9. Juli 2013 09:09 An: openstack@lists.launchpad.net Betreff: Re: [Openstack] [SWIFT] raising network traffic on the storage node If the replication traffic is responsible for this raising network traffic for only 1.200.000 objects, how much traffic I can expect if I have 100.000.000 objects stored? The average size of my mailobjects are 120 kB. It's planned to use all 12 hard drive slots of my DELL R720xd with 4 TB drives . I have 5 storage nodes and 2 balanced proxy nodes. Will the replication traffic kill my system? We are running with 400.000.000 objects across 11 object storage nodes. Total network traffic on any of those nodes is less then 10 MByte /second However we have seen slowdowns with lots of small files and really big disks. The issue is not related to the network but the local filesystem/disk. When the inode cache gets insufficient you can see terrible slow-downs. There have been a few threads about that in this list, having a lot of memory usually helps a bit. Cheers, Robert van Leeuwen ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [SWIFT] raising network traffic o the storage node
On Monday some more mailboxes store their Mails in the objectstorage. But that only increased the raising. Traffic Storagenode: http://www.schuermann.net/temp/storagenode2.png Traffic Proxyserver: http://www.schuermann.net/temp/proxyserver2.png -Ursprüngliche Nachricht- Von: Peter Portante [mailto:peter.a.porta...@gmail.com] Gesendet: Montag, 8. Juli 2013 16:04 An: Klaus Schürmann Cc: openstack@lists.launchpad.net Betreff: Re: [Openstack] [SWIFT] raising network traffic o the storage node Can you zoom in past the spike, most recent 2 or three weeks and see how it looks? My guess is that the proxy traffic is also rising. On Mon, Jul 8, 2013 at 9:50 AM, Klaus Schürmann klaus.schuerm...@mediabeam.com wrote: Hi, I use a swift storage as a mail-store. Now I have about 1.000.000 objects stored in the cluster. I'm wondering about the raising network traffic on my storage nodes. The traffic from the proxy-server has a normal characteristic. Traffic Storagenode: http://www.schuermann.net/temp/storagenode.png Traffic Proxyserver: http://www.schuermann.net/temp/proxyserver.png Can someone explain such behavior? Thanks Klaus ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [SWIFT] raising network traffic o the storage node
On Mon, 8 Jul 2013 13:50:38 + Klaus Schürmann klaus.schuerm...@mediabeam.com wrote: I use a swift storage as a mail-store. Now I have about 1.000.000 objects stored in the cluster. Traffic Storagenode: http://www.schuermann.net/temp/storagenode.png Traffic Proxyserver: http://www.schuermann.net/temp/proxyserver.png Can someone explain such behavior? At a guess, raising number of objects makes number of partitions to increase, and that increases the replication traffic, specifically the number of sent MD5s for partitions. It would be interesting to corellate the number of objects and number of non-empty and empty partitions with the amounts of traffic. If the increasing transfer of hashes is the reason, you could also verify by graphing the traffic to port 873 separately. Swift never replicates object bodies through its own HTTP interface, so this splits control traffic from data traffic for you. Data traffic should be driven by customer and node failures, not consistency checking. Be prepared to split up storage nodes, however. Even if there's no bug with replication, its aggregate traffic increases with the increases in object counts. -- Pete ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [SWIFT] raising network traffic on the storage node
If the replication traffic is responsible for this raising network traffic for only 1.200.000 objects, how much traffic I can expect if I have 100.000.000 objects stored? The average size of my mailobjects are 120 kB. It's planned to use all 12 hard drive slots of my DELL R720xd with 4 TB drives. I have 5 storage nodes and 2 balanced proxy nodes. Will the replication traffic kill my system? Here is a small part of my object-server-replicator log: Jul 9 06:48:16 storage-node1 object-replicator Starting object replication pass. Jul 9 06:49:13 storage-node1 object-replicator 9830/9830 (100.00%) partitions replicated in 57.56s (170.78/sec, 0s remaining) Jul 9 06:49:13 storage-node1 object-replicator 1234597 suffixes checked - 0.00% hashed, 0.00% synced Jul 9 06:49:13 storage-node1 object-replicator Partition times: max 0.0279s, min 0.0068s, med 0.0104s Jul 9 06:49:13 storage-node1 object-replicator Object replication complete. (0.96 minutes) Jul 9 06:49:43 storage-node1 object-replicator Starting object replication pass. Jul 9 06:50:41 storage-node1 object-replicator 9830/9830 (100.00%) partitions replicated in 57.69s (170.39/sec, 0s remaining) Jul 9 06:50:41 storage-node1 object-replicator 1234643 suffixes checked - 0.00% hashed, 0.00% synced Jul 9 06:50:41 storage-node1 object-replicator Partition times: max 0.0365s, min 0.0068s, med 0.0104s Jul 9 06:50:41 storage-node1 object-replicator Object replication complete. (0.96 minutes) Jul 9 06:51:11 storage-node1 object-replicator Starting object replication pass. Jul 9 06:52:09 storage-node1 object-replicator 9830/9830 (100.00%) partitions replicated in 58.31s (168.58/sec, 0s remaining) Jul 9 06:52:09 storage-node1 object-replicator 1234688 suffixes checked - 0.00% hashed, 0.00% synced Jul 9 06:52:09 storage-node1 object-replicator Partition times: max 0.0348s, min 0.0069s, med 0.0106s Jul 9 06:52:09 storage-node1 object-replicator Object replication complete. (0.97 minutes) -Ursprüngliche Nachricht- Von: Pete Zaitcev [mailto:zait...@redhat.com] Gesendet: Montag, 8. Juli 2013 19:22 An: Klaus Schürmann Cc: openstack@lists.launchpad.net Betreff: Re: [Openstack] [SWIFT] raising network traffic o the storage node On Mon, 8 Jul 2013 13:50:38 + Klaus Schürmann klaus.schuerm...@mediabeam.com wrote: I use a swift storage as a mail-store. Now I have about 1.000.000 objects stored in the cluster. Traffic Storagenode: http://www.schuermann.net/temp/storagenode.png Traffic Proxyserver: http://www.schuermann.net/temp/proxyserver.png Can someone explain such behavior? At a guess, raising number of objects makes number of partitions to increase, and that increases the replication traffic, specifically the number of sent MD5s for partitions. It would be interesting to corellate the number of objects and number of non-empty and empty partitions with the amounts of traffic. If the increasing transfer of hashes is the reason, you could also verify by graphing the traffic to port 873 separately. Swift never replicates object bodies through its own HTTP interface, so this splits control traffic from data traffic for you. Data traffic should be driven by customer and node failures, not consistency checking. Be prepared to split up storage nodes, however. Even if there's no bug with replication, its aggregate traffic increases with the increases in object counts. -- Pete ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
