This was posted and resolved on the user thread. Typo in my configuration was the issue.
Thanks Justin Sent from my iPhone > On Jan 17, 2017, at 12:42 AM, Tristan Stevens <tris...@cloudera.com> wrote: > > Hi Justin, > Please can you post your agent config and also any HDFS logs? Ideally you > should be seeing INFO logs as follows: “Closing Idle Bucketwriter”. > > Tristan > > Tristan Stevens > Senior Solutions Architect > Cloudera, Inc. | www.cloudera.com > m +44(0)7808 986422 | tris...@cloudera.com > > <hadoop10.png> Celebrating a decade of community accomplishments > cloudera.com/hadoop10 > #hadoop10 > >> On 12 January 2017 at 19:23:18, Justin Workman (justinjwork...@gmail.com) >> wrote: >> >> More details >> >> Flume 1.6 - Core Apache version. >> KafkaSource (0.8.2) -> File Channel -> HDFS Sink (CDH5.5.2). >> >> On Thu, Jan 12, 2017 at 12:20 PM, Justin Workman <justinjwork...@gmail.com> >> wrote: >> >> > sorry for cross posting to user and dev. I have recently set up a flume >> > configuration where we are using the regex_extractor interceptor to parse >> > the actual event date from the record flowing through the Flume source, >> > then using that date to build the HDFS sink bucket path. However, it >> > appears that the hdfs.idleTimeout value is not honored in this >> > configuration. It does work when using the timestamp interceptor you build >> > the output path. >> > >> > I have set the hdfs.idleTimeout value for the HDFS sink, but the files are >> > never closed or renamed until I restart or shutdown Flume. Our flume is >> > configured to roll based on size or output path, and the files >> > rename/close/roll fine based on size, however the last file in each output >> > path is always left with the .tmp extension until we restart Flume. I >> > would >> > expect that the file would be renamed and closed if there are no records >> > written to this file after the idleTimeout is reached. >> > >> > Could I be missing something, or is this a known bug with the >> > regex_extract interceptor? >> > >> > Thanks >> > Justin >> >
