During the past month, Swift contributors have gathered in Austin, Hong Kong, and online to discuss projects underway. There are some major efforts underway, and I hope to lay them out and tie them together here, so that we all know what the goals for the next six months are.
The biggest feature set is storage policies. Storage policies will give deployers and users incredible flexibility in how to manage their data in the storage cluster. There are three basic parts to storage policies. First, given the global set of hardware available in a single Swift cluster, choose which subset of hardware on which to store data. This can be done by geography (e.g. US-East vs EU vs APAC vs global) or by hardware properties (e.g. SATA vs SSDs). An obviously, the combination can give a lot of flexibility. Second, given the subset of hardware being used to store the data, choose how to encode the data across that set of hardware. For example, perhaps you have 2-replica, 3-replica, or erasure code policies. Combining this with the hardware possibilities, you get e.g. US-East reduced redundancy, global triple replicas, and EU erasure coded. Third, give the subset of hardware and how to store the data across that hardware, control how Swift talks to a particular storage volume. This may be optimized local file systems. This may be Gluster volumes. This may be non-POSIX volumes like Seagate's new Kinetic drives. We're well on our way to getting this set of work done. In Hong Kong there was a demo of the current state of multi-ring support (for parts one and two). We've also got a great start in refactoring the interface between Swift and on-disk files and databases (for part three). But there is still a ton of work to do. * we need to finalize the multi-ring work for the WSGI processes * we need to ensure that large objects work with storage policies * replication needs to be multi-ring aware * auditing needs to be multi-ring aware * updaters need to be multi-ring aware * we need to write a multi-ring reconciler * we need to merge all of this multi-ring stuff into master * we need to finalize the DiskFile refactoring * we need to properly refactor the DBBrokers * we need the make sure the daemons can be extended to support different DiskFiles The top-level blueprint for this is at https://blueprints.launchpad.net/swift/+spec/storage-policies Our target for the storage policy work is to support erasure coded data within Swift. After the general storage policy work above is done, we need to work on refactoring Swift's proxy server to make sure its interaction with the storage nodes allows for differing storage schemes. I'm tremendously excited about the potential for storage policies. I think it's the most significant development in Swift since the entire project was open-sourced. Storage policies allow Swift to grow from being the engine powering the world's largest storage clouds to a storage platform enabling broader use cases by offering the flexibility to very specifically match many different deployment patterns. Oh and if that's not enough to keep us all busy, there is other work going on in the community, too. Some has been merged into Swift, and some will stay in the ecosystem of tools for Swift, but they are all important to a storage system. We've got an improved replication platform merged into Swift, and it needs to be thoroughly tested and polished. Once it's stable, we'll be able to build on it to really improve Swift around MTTD and MTTR metrics. We're in the process of refactoring metadata so that it is strongly separate into stuff that a user can see and change and stuff the user can't see and change. There is also some serious effort being put forth by a few companies (including HP and IBM) to provide a way to add powerful metadata searching into a Swift cluster. The ZeroVM team is interested in extending Swift to better support large-scale data processing. The Barbican team is looking in to providing good ways to offer encryption for data stored in Swift. Others are looking in to how to grow clusters (changing the partition power) and extend clusters (transparently federating multiple Swift clusters). Not all of this is going to be done by the OpenStack Icehouse release. We cut stable releases of Swift fairly often, and these features will roll out in those releases as they are done. My goal for Icehouse is to see the storage policy work done and ready for production. --John
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev