Hi All, I did little research about Apache Zookeeper[1] and how to use it in airavata. Its really a nice way to achieve fault tolerance and reliable communication between our thrift services and clients. Zookeeper is a distributed, fault tolerant system to do a reliable communication between distributed applications. This is like an in-memory file system which has nodes in a tree structure and each node can have small amount of data associated with it and these nodes are called znodes. Clients can connect to a zookeeper server and add/delete and update these znodes.
In Apache Airavata we start multiple thrift services and these can go down for maintenance or these can crash, if we use zookeeper to store these configuration(thrift service configurations) we can achieve a very reliable system. Basically thrift clients can dynamically discover available service by using ephemeral znodes(Here we do not have to change the generated thrift client code but we have to change the locations we are invoking them). ephemeral znodes will be removed when the thrift service goes down and zookeeper guarantee the atomicity between these operations. With this approach we can have a node hierarchy for multiple of airavata, orchestrator,appcatalog and gfac thrift services. For specifically for gfac we can have different types of services for each provider implementation. This can be achieved by using the hierarchical support in zookeeper and providing some logic in gfac-thrift service to register it to a defined path. Using the same logic orchestrator can discover the provider specific gfac thrift service and route the message to the correct thrift service. With this approach I think we simply have write some client code in thrift services and clients and zookeeper server installation can be done as a separate process and it will be easier to keep the Zookeeper server separate from Airavata because installation of Zookeeper server little complex in production scenario. I think we have to make sure everything works fine when there is no Zookeeper running, ex: enable.zookeeper=false should works fine and users doesn't have to download and start zookeeper. [1]http://zookeeper.apache.org/ Thanks Lahiru -- System Analyst Programmer PTI Lab Indiana University
