Here is a simple example of 2 netcat sources sending to 2 different elasticsearch clusters. The same principle would apply for HBase as well
#Flow definition agent.sources = netcat anothernetcat agent.channels = netcat-channel-es anothernetcat-channel-es agent.sinks = netcat-sink-es anothernetcat-sink-es # For each one of the sources, the type is defined agent.sources.netcat.type = netcat agent.sources.netcat.bind = 0.0.0.0 agent.sources.netcat.port = 5150 agent.sources.anothernetcat.type = netcat agent.sources.anothernetcat.bind = 0.0.0.0 agent.sources.anothernetcat.port = 5155 # The channels can be defined as follows. agent.sources.netcat.channels = netcat-channel-es agent.sources.anothernetcat.channels = anothernetcat-channel-es # Each sink's type must be defined agent.sinks.netcat-sink-es.type = elasticsearch agent.sinks.netcat-sink-es.hostNames = server1:9300,server2:9300,server3:9300 agent.sinks.netcat-sink-es.indexName = flume_netcat_index agent.sinks.netcat-sink-es.indexType = bar_type agent.sinks.netcat-sink-es.clusterName = netcat agent.sinks.netcat-sink-es.batchSize = 500 agent.sinks.netcat-sink-es.ttl = 5 agent.sinks.anothernetcat-sink-es.type = elasticsearch agent.sinks.anothernetcat-sink-es.hostNames = server4:9300,server5:9300,server6:9300 agent.sinks.anothernetcat-sink-es.indexName = flume_anothernetcat_index agent.sinks.anothernetcat-sink-es.indexType = bar_type agent.sinks.anothernetcat-sink-es.clusterName = anothernetcat agent.sinks.anothernetcat-sink-es.batchSize = 500 agent.sinks.anothernetcat-sink-es.ttl = 5 #Specify the channel the sink should use agent.sinks.netcat-sink-es.channel = netcat-channel-es agent.sinks.anothernetcat-sink-es.channel = anothernetcat-channel-es # Each channel's type is defined. agent.channels.netcat-channel-es.type = memory agent.channels.anothernetcat-channel-es.type = memory # Other config values specific to each type of channel(sink or source) # can be defined as well # In this case, it specifies the capacity of the memory channel agent.channels.netcat-channel-es.capacity = 1000000 agent.channels.anothernetcat-channel-es.capacity = 1000000 agent.channels.netcat-channel-es.transactionCapacity = 1000 agent.channels.anothernetcat-channel-es.transactionCapacity = 1000 On Thursday, March 6, 2014 11:03:08 PM, Chhaya Vishwakarma <[email protected]> wrote: Thank you for your reply do you have any sample for the same? From:Simeon Zaharici [mailto:[email protected]] Sent: Thursday, March 06, 2014 8:23 PM To: [email protected] Subject: Re: Multiple flume agent on single machine Hello You do not need to run multiple flume agents, although this is possible. You could just define multiple sources running on different ports in the same flume agent, each one writing to its own Hbase sink. Regards, Simeon On Wednesday, March 5, 2014 11:30:56 PM, Chhaya Vishwakarma <[email protected]> wrote: Hi I have a problem statement where i need to collect data from multiple machines(each machine forwarding data for different format) to centralized storage(Hbase). I need to make a different table for each type of log collected. How can i do it using flume? Is it possible to run multiple flume agent on same machine ? I can assign each type to one agent which will write to hbase. Regards, Chhaya Vishwakarma ________________________________ The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"
