[ https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roshan Naik updated FLUME-2498: ------------------------------- Attachment: FLUME-2498-5.patch Uploading Patch. - Changed the file consume order to be based on modification time (let me if you feel this is inappropriate choice, can easily switch to creation time if needed). - Also added to unit test to check file consume order - it includes changes from patch v4 submitted by Johny (can u please verify this [~jrufus] ?) -corrected the description of 'filegroups' setting in doc. [~iijima_satoshi] can you please validate ? Although i have a unit test to verify the consume order, would be good if someone does a review of it to ensure i didnt mess up. [~iijima_satoshi] johny, ashish or others ? I have kept my changes to bare minimal to reduce churn. > Implement Taildir Source > ------------------------ > > Key: FLUME-2498 > URL: https://issues.apache.org/jira/browse/FLUME-2498 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources > Reporter: Satoshi Iijima > Fix For: v1.7.0 > > Attachments: FLUME-2498-2.patch, FLUME-2498-3.patch, > FLUME-2498-4.patch, FLUME-2498-5.patch, FLUME-2498.patch > > > This is the proposal of implementing a new tailing source. > This source watches the specified files, and tails them in nearly real-time > once appends are detected to these files. > * This source is reliable and will not miss data even when the tailing files > rotate. > * It periodically writes the last read position of each file in a position > file using the JSON format. > * If Flume is stopped or down for some reason, it can restart tailing from > the position written on the existing position file. > * It can add event headers to each tailing file group. > A attached patch includes a config documentation of this. > This source requires Unix-style file system and Java 1.7 or later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)