Hi,all:
I submit a pull request to flume-1.7 on github. The address is
https://github.com/apache/flume/pull/54 .
The changes are as follows:
1. Support multiline. Users can define the start regex of multiline.
Add a parameter REGEX_START in
TaildirSourceConfigurationConstants.java.REGEX_START is used for generating
Flume events containing multiple lines in the body, per event. The parameter
determines the start of an event. Default value is "". If the value is set to
"", a line with the end of '\n' will be dealed into one flume event.
The sample usage:
agent.sources.taildirsource.lineStartRegex =
\\s?\\d\\d\\d\\d-\\d\\d-\\d\\d\\s\\d\\d:\\d\\d:\\d\\d,\\d\\d\\d
2. Support recursive directory. Wildcards are allowed in the directory
name.
Modify the function getMatchFiles() in ReliableTaildirEventReader.java
to support this functionality.
The sample usage:
agent.sources.taildirsource.filegroups.f1 =
/Users/wenqiao/work/flume/apache-flume-1.7.0-SNAPSHOT-bin/conf/*/01/[ab].log
3. Fix the bug if a line??s length exceeds 8192 bytes. Make the buffer
size be configured.
Add a parameter BUFFER_SIZE in
TaildirSourceConfigurationConstants.java.BUFFER_SIZE is used to define the max
number of bytes for one flume event body's content. Default size is 8192.
4. Put the filePath, hostname, IP into the headers of a flume event if the
headers do not contain the keys.
