Can I just say absolutely yes? I'm not clear on what issues zookeeper introduced, but the concept of having a central configuration is very appealing. I can definitely say that this is one of the items that will delay my employers transition from 0.9.x to 1.x.
Ralph On Jun 7, 2012, at 3:18 PM, Eric Sammer <[email protected]> wrote: > Flumers: > > Flume 0.9.x supported online reconfiguration and the intention was for the > 1.x branch to do so as well (it doesn't yet). I wanted to start a > discussion around whether people are interested in this kind of > functionality or if simply restarting the daemon(s) was sufficient for your > deployment. There are two ways of thinking about it: > > * Support reconfiguration. Agents may have multiple flows passing through > them and, ideally, adding new ones shouldn't interrupt existing flows. > Agent restarts interrupt collection and, for non-durable channels (i.e. > MemoryChannel), data *may* be lost. Reconfiguration will add significant > complexity and ultimately does not get around host level maintenance, > software upgrades, and the like. > > * Do no support reconfiguration. Accept the fact that agents may go down > eventually, so it should be supported as a first class case. In other > words, embrace the idea of failure / maintenance and handle it by > recommending topologies of agents that include multiple agents at each tier > and simply roundrobin / failover where necessary. The only downside is the > agent tier closest to the originating source (e.g. a log4j client); > restarting that agent means the client application needs to be able to find > another agent or buffer (which impacts durability or blocks the > application). > > We can optionally support some subset of online reconfiguration such as > only allowing new flows to be introduced or existing flows to be > "decommissioned," but not allow alteration of existing flows. Ultimately > this feature is a ton of work and adds a ton of complexity so if it's not > something folks are clambering for, we should spend our time worrying about > more pressing issues. > > Thoughts? Comments? > -- > Eric Sammer > twitter: esammer > data: www.cloudera.com
