Re: [Openstack] [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ?

2012-04-16 Thread Juan J.
On Thu, 2012-04-12 at 20:51 -0500, John Dickinson wrote: > Swift keeps total bytes, container, and object count (eventually) up-to-date > in the account metadata. There are also log processing tools (like slogging - > http://github.com/notmyname/slogging) that can provide usage information > (in

Re: [Openstack] [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ?

2012-04-13 Thread Dmitry Ukov
Please explain, how did you measure replication time? On Fri, Apr 13, 2012 at 7:23 AM, Alex Yang wrote: > In my view, the biggest problem of swift is not the new features but the > improvement of performance. > > At first, we knew that container-updater, *-auditor, *-replicator will > loop all t

Re: [Openstack] [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ?

2012-04-13 Thread Oleg Gelbukh
Alex, Thank you for important point and interesting information on large-scale Swift performance! Can you please explain a little what these times stand for? Is this a single process runtime, or the time needed to converge cluster in case of device failure, or something else? -- Best regards, Ol

[Openstack] [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ?

2012-04-12 Thread Alex Yang
In my view, the biggest problem of swift is not the new features but the improvement of performance. At first, we knew that container-updater, *-auditor, *-replicator will loop all the files in the disk during the running interval. When the number of files is very large, the process of replicator,

Re: [Openstack] [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ?

2012-04-12 Thread John Dickinson
I should also mention the summit session talking about this very topic led by Everett Toews. It's (currently) scheduled for 9am on wednesday. http://summit.openstack.org/sessions/view/81 --John On Apr 12, 2012, at 8:51 PM, John Dickinson wrote: > Swift keeps total bytes, container, and objec

Re: [Openstack] [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ?

2012-04-12 Thread John Dickinson
Swift keeps total bytes, container, and object count (eventually) up-to-date in the account metadata. There are also log processing tools (like slogging - http://github.com/notmyname/slogging) that can provide usage information (including bandwidth) based on swift logs. While I think that it's

Re: [Openstack] [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ?

2012-04-12 Thread Frederik Van Hecke
Hi Kuo, One option would be to keep the usage information (num files, num bytes, etc) per container / account in an sqlite DB, just like it is done for account and container info. To avoid having to loop through all data at regular intervals (to update the info), additional logic could be added t

[Openstack] [Swift] Does there any exist blueprint or sub-project of user's storage space quota or counting method for Swift ?

2012-04-12 Thread Kuo Hugo
Hi folks , I'm thinking about the better approach to manage "an user" or "an account" space usage quota in swift. Is there any related blueprint or sub-project even an idea around ? Any suggestion of benefits to be an external service or to be a middle-ware in swift-proxy ? I'm concerning about