[
https://issues.apache.org/jira/browse/FLUME-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429982#comment-13429982
]
Hari Shreedharan commented on FLUME-1428:
-----------------------------------------
Assigning to myself for now. I am trying a couple of methods:
1. Move the reference counting to the Log class, and pass it to the queue while
checkpointing, and retrieve from queue before replaying. Replay handler
provides the refcounts from the replay and the log class applies this delta to
the original values in the checkpoint.
2. If 1 is too complex and causes issues, I am going to get something checked
in which will work reasonably fine. Include a configuration variable which sets
a cleanUpInterval and delete the "inactive" files after the clean up interval.
The downside is there might be extra "old" data, but this will work around the
current problem. Not yet sure how this will work on restarts - shutdown and
restart after several hours.
> File Channel should not consider a file as inactive until all takes are
> committed.
> ----------------------------------------------------------------------------------
>
> Key: FLUME-1428
> URL: https://issues.apache.org/jira/browse/FLUME-1428
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.2.0
> Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
> FlumeEventQueue removes fileID from fileIDCounts before commits are
> completed. See FLUME-1417 for background. This series of events would be
> problematic:
> File ( i )
> n puts
> commit
> n takes (file i is no longer in flume event queue fileIDCounts)
> —
> file roll -> current active file in directory (File i+1)
> background worker kicks in -> removes file i.
> Note that the commit for the n takes has not taken place.
> We do keep 2 files per directory, but if a commit/rollback does not come and
> 2 files are rolled, we might have some issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
