Krishna Kumar created FLUME-2331:
------------------------------------
Summary: Large TMP files created and never closed
Key: FLUME-2331
URL: https://issues.apache.org/jira/browse/FLUME-2331
Project: Flume
Issue Type: Bug
Components: File Channel
Affects Versions: v1.4.0
Reporter: Krishna Kumar
We are currently writing files to Hadoop partitioned by year, month, day via
Flume. File rollovers are done every 5 minutes. Recently, we noticed that this
file rollover stops happening sometime during the day and that further data is
written to an open TMP file. Because there are no further file rollovers, this
TMP file becomes very large. At the end of the day, the TMP is not closed also
and Flume goes to the next day, creating new files.
We use a "." prefix to prevent Hive from complaining of the open TMP file.
Because of this issue where the TMP file is never closed, the file remains
hidden to Hive even after the day ends.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
