Re: [Openstack] Swift sharding across containers

2013-10-10 Thread Chuck Thier
Hi Shri, On Thu, Oct 10, 2013 at 1:31 PM, Shrinand Javadekar wrote: > Thanks for the inputs Chuck. Please see my responses inline. > > > On Thu, Oct 10, 2013 at 7:56 AM, Chuck Thier wrote: > >> Hi Shri, >> >> I think your observations are fairly spot on. Here are a couple of >> thoughts/commen

Re: [Openstack] Swift sharding across containers

2013-10-10 Thread Shrinand Javadekar
Thanks for the inputs Chuck. Please see my responses inline. On Thu, Oct 10, 2013 at 7:56 AM, Chuck Thier wrote: > Hi Shri, > > I think your observations are fairly spot on. Here are a couple of > thoughts/comments. > > 1. I wonder if you are maxing out how much your client can push at 128 >

Re: [Openstack] Swift sharding across containers

2013-10-10 Thread Chuck Thier
Hi Shri, I think your observations are fairly spot on. Here are a couple of thoughts/comments. 1. I wonder if you are maxing out how much your client can push at 128 threads. If you were to increase the number threads (or number of clients) for the higher container counts, you could get more t

Re: [Openstack] Swift sharding across containers

2013-10-09 Thread Shrinand Javadekar
Thanks Chuck. In order to really measure this, I ran some tests on Rackspace; i.e. I got a VM on Rackspace and that VM was talking to a Rackspace Cloudfiles-US swift cluster. The VM and object store were both in the Chicago region. The downside of using a public object store is that I have little

Re: [Openstack] Swift sharding across containers

2013-09-03 Thread Chuck Thier
Hi Shri, The short answer is that sharding your data across containers in swift is generally a good idea. The limitations with containers has a lot more to do with overall concurrency rather than total objects in a container. The number of objects in a container can have an affect on that, but w

[Openstack] Swift sharding across containers

2013-09-01 Thread Shrinand Javadekar
Hi, There have been several articles which talk about keeping the number of objects in a container to about 1M. Beyond that sqlite starts becoming the bottleneck. I am going to make sure we abide by this number. However, has anyone measured whether putting objects among multiple containers right