Dears We are planning to use flume in consuming data from Gnip. We already configured a flume agent that uses a customized source to read from gnip and write the streamed data to HDSF.
This agent now is considered a single point of failure in the architecture. If this agent goes down for any reason, no data will be read and the system will not be updated. Is there a way to configure multiple agents doing the same job but only one is working and the others in something like standby mode. If the working agent is down for any reason, one of the other standby agents works automatically. We can achieve this using a monitoring tool like nagios that monitor the status of the agent and take an action based on that status. If the agent is down for 3 subsequent checks, start another agent or restart the same agent. Is there anything like this from flume side? something like a master flume agent monitoring the other agents and so on ? Best regards
