Does anyone have a working example of using flume 1.7 or 1.8 with the use of “recursiveDirectorySearch”? My source (spooldir) has multiple subdirectories and I’ve read that version 1.7+ has the ability to work with subdirectories. I have this configured and flume starts up without error, but nothing seems to be moving out of flume.
Flume may not be the best way to handle this actually, but it was worth a shot. Here’s my flume.conf agent.sources = EventSpool agent.sources.EventSpool.type = spooldir agent.sources.EventSpool.spoolDir = /archive/home/orderfeed/ agent.sources.EventSpool.recursiveDirectorySearch = true agent.sources.EventSpool.channels = memoryChannel agent.sources.EventSpool.fileHeader = true agent.sources.EventSpool.deserializer.maxLineLength = 209715200 agent.sources.EventSpool.basenameHeader = true agent.sources.EventSpool.interceptors = itime agent.sources.EventSpool.deletePolicy = immediate #agent.sources.EventSpool.interceptors.itime.type = timestamp agent.channels = memoryChannel agent.channels.memoryChannel.type = memory agent.channels.memoryChannel.capacity = 15000 agent.channels.memoryChannel.transactionCapacity = 10000 agent.sinks = AvroSink agent.sinks.AvroSink.type = avro agent.sinks.AvroSink.channel = memoryChannel agent.sinks.AvroSink.hostname = [flume receiver hostname removed] agent.sinks.AvroSink.port = 5175 agent.sinks.AvroSink.recursiveDirectorySearch = true agent.sinks.AvroSink.batchSize = 0 agent.sinks.AvroSink.rollSize = 0 agent.sinks.AvroSink.rollInterval = 0 agent.sinks.AvroSink.rollCount = 0 agent.sinks.AvroSink.idleTimeout = 0
