Hi , 1. Correct ! Once you adding new devices and rebalance rings, portion of partitions will be reassigned to new devices. If those partitions were used by some objects, object-replicator is going to move data to new devices. You should see logs of object-replicator to transfer objects from one device to another by invoking rsync.
2. Regarding to busy swift-account-server, that's pretty abnormal tho. Is there any log indicating account-server doing any jobs? A possibility is the ring which includes wrong port number of other workers to account-server. Perhaps you can paste all your rings layout to http://paste.openstack.org/ . To use strace on account-server process may help to track the exercise. 3. In kind of deployment that outward-facing interface shares same network resource with cluster-facing interface, it definitely causes some race on network utilization. Hence the frontend traffic is under impact by replication traffic now. 4. To have a detail network topology diagram will help. Hugo Kuo 2014-05-29 1:06 GMT+08:00 Shyam Prasad N <nspmangal...@gmail.com>: > Hi, > > Confused about the right mailing list to ask this question. So including > both openstack and openstack-dev in the CC list. > > I'm running a swift cluster with 4 nodes. > All 4 nodes are symmetrical. i.e. proxy, object, container, and account > servers running on each with similar storage configuration and conf files. > The I/O traffic to this cluster is mainly to upload dynamic large objects > (typically 1GB chunks (sub-objects) and around 5-6 chunks under each large > object). > > The setup is running and serving data; but I've begun to see a few perf > issues, as the traffic increases. I want to understand the reason behind > some of these issues, and make sure that there is nothing wrong with the > setup configuration. > > 1. High CPU utilization from rsync. I have set replica count in each of > account, container, and object rings to 2. From what I've read, this > assigns 2 devices for each partition in the storage cluster. And for each > PUT, the 2 replicas should be written synchronously. And for GET, the I/O > is through one of the object servers. So nothing here should be > asynchronous in nature. Then what is causing the rsync traffic here? > > I recently ran a ring rebalance command after adding a node recently. > Could this be causing the issue? > > 2. High CPU utilization from swift-account-server threads. All my frontend > traffic use 1 account and 1 container on the servers. There are hundreds of > such objects in the same container. I don't understand what's keeping the > account servers busy. > > 3. I've started noticing that the 1GB object transfers of the frontend > traffic are taking significantly more time than they used to (more than > double the time). Could this be because i'm using the same subnet for both > the internal and the frontend traffic. > > 4. Can someone provide me some pointers/tips to improving perf for my > cluster configuration? (I guess I've given out most details above. Feel > free to ask if you need more details) > > As always, thanks in advance for your replies. Appreciate the support. :) > -- > -Shyam > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openst...@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev