Re: [Openstack] Swift slow write performance

2011-12-22 Thread Rustam Aliyev
Hi All, After few more days of checks I couldn't resolve it. I think I tried pretty much everything. I tend to think that the problem is CentOS/RHEL 5.x and the following error: Dec 22 05:03:24 ec01 object-server STDOUT: WARNING:root:Unable to locate fallocate in libc. Leaving as a

Re: [Openstack] Swift slow write performance

2011-12-20 Thread Rustam Aliyev
Hi Mike, Thanks, I didn't know that PUT operation also includes updating replica containers. That makes sense. I will check that. In the mean time I've added debug checkpoints into PUT operation to measure different steps. Modified code is here: http://paste.openstack.org/show/3899/

Re: [Openstack] Swift slow write performance

2011-12-20 Thread Ywang225
Yes, the step to update object list in container should be the source, as it will wait for http connection timeout. From the error log, all connection to port 6001are failed, so I'd presume the container servers aren't running. One straight testing is to comment out the call to

Re: [Openstack] Swift slow write performance

2011-12-20 Thread Ywang225
Both Metadata.update() and container_update() should apply on all replicas, and metadata.update() will write xattrs, and container_update() will write to sqlite db files. Surprisingly to see so long time to update both. --ywang 发自我的 iPhone 在 2011-12-20,19:13,Rustam Aliyev rus...@code.az 写道:

Re: [Openstack] Swift slow write performance

2011-12-19 Thread Rustam Aliyev
Hi All, Right now I tried everything, but PUT speed remains very slow. In order to speedup I tried various combination of workers, disabled disk mount check, enabled DEBUG log level, checked source code to understand logic, finally added another node. No errors, and I was able only to squeeze

Re: [Openstack] Swift slow write performance

2011-12-19 Thread Michael Barton
On Mon, Dec 19, 2011 at 6:21 AM, Rustam Aliyev rus...@code.az wrote: The only thing which looks suspicious to me are these errors: Dec 18 04:01:28 ec01 object-server ERROR container update failed with 10.0.1.3:6001/d01 (saving for async update later): Timeout (3s) (txn:

Re: [Openstack] Swift slow write performance

2011-12-16 Thread Ywang225
increasing concurrency (with -c)should boost write performance. 发自我的 iPhone 在 2011-12-16,11:20,Rustam Aliyev rus...@code.az 写道: Hi, I'm testing swift 1.4.4 setup with 4 nodes/zones on RHEL 5.7. I ran into the problem of slow writes. Using swift-bench we generated load, writing hundreds

[Openstack] Swift slow write performance

2011-12-15 Thread Rustam Aliyev
Hi, I'm testing swift 1.4.4 setup with 4 nodes/zones on RHEL 5.7. I ran into the problem of slow writes. Using swift-bench we generated load, writing hundreds of 4K files. Results: - Writes - ~3 PUTs/sec (very slow) - Reads - ~25 GETs/sec (ok) It's clear that writes are not limited by I/O.