[jira] [Commented] (FLUME-3216) When I rename the file, it causes flume to read the data repeatedly.
[ https://issues.apache.org/jira/browse/FLUME-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846387#comment-16846387 ] Jeff Yang commented on FLUME-3216: -- I have resubmitted this patch. > When I rename the file, it causes flume to read the data repeatedly. > > > Key: FLUME-3216 > URL: https://issues.apache.org/jira/browse/FLUME-3216 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: 1.8.0 > Environment: centos-6.8 >Reporter: Jeff Yang >Priority: Critical > Fix For: 1.7.0, 1.8.0 > > Time Spent: 50m > Remaining Estimate: 0h > > When I use Taildir source, whenever I rename a file, it causes flume to read > the data repeatedly.This resulted in duplication of uploaded data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Commented] (FLUME-3216) When I rename the file, it causes flume to read the data repeatedly.
[ https://issues.apache.org/jira/browse/FLUME-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359326#comment-16359326 ] jifei_yang commented on FLUME-3216: --- I modified the method of judgment: Judge the log file only by inode. > When I rename the file, it causes flume to read the data repeatedly. > > > Key: FLUME-3216 > URL: https://issues.apache.org/jira/browse/FLUME-3216 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: 1.8.0 > Environment: centos-6.8 >Reporter: jifei_yang >Priority: Critical > Fix For: 1.7.0, 1.8.0 > > > When I use Taildir source, whenever I rename a file, it causes flume to read > the data repeatedly.This resulted in duplication of uploaded data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Commented] (FLUME-3216) When I rename the file, it causes flume to read the data repeatedly.
[ https://issues.apache.org/jira/browse/FLUME-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359324#comment-16359324 ] jifei_yang commented on FLUME-3216: --- flume monitor files in the specified directory, rename the file in the directory, flume will think the renamed file is a new file, will be read from the first log, resulting in the duplication of the log upload. > When I rename the file, it causes flume to read the data repeatedly. > > > Key: FLUME-3216 > URL: https://issues.apache.org/jira/browse/FLUME-3216 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: 1.8.0 > Environment: centos-6.8 >Reporter: jifei_yang >Priority: Critical > Fix For: 1.7.0, 1.8.0 > > > When I use Taildir source, whenever I rename a file, it causes flume to read > the data repeatedly.This resulted in duplication of uploaded data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Commented] (FLUME-3216) When I rename the file, it causes flume to read the data repeatedly.
[ https://issues.apache.org/jira/browse/FLUME-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356682#comment-16356682 ] Ferenc Szabo commented on FLUME-3216: - Hi [~highfei2...@126.com], could you please describe the scenario where this happens? What is the use case where someone would move out and move back the same file into the tailed folder and would not expect them to be read again? > When I rename the file, it causes flume to read the data repeatedly. > > > Key: FLUME-3216 > URL: https://issues.apache.org/jira/browse/FLUME-3216 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: 1.8.0 > Environment: centos-6.8 >Reporter: jifei_yang >Priority: Critical > Fix For: 1.7.0, 1.8.0 > > > When I use Taildir source, whenever I rename a file, it causes flume to read > the data repeatedly.This resulted in duplication of uploaded data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Commented] (FLUME-3216) When I rename the file, it causes flume to read the data repeatedly.
[ https://issues.apache.org/jira/browse/FLUME-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355061#comment-16355061 ] jifei_yang commented on FLUME-3216: --- Hi,[~fszabo] File name or the original file name, just move to a temporary directory, and then move back. > When I rename the file, it causes flume to read the data repeatedly. > > > Key: FLUME-3216 > URL: https://issues.apache.org/jira/browse/FLUME-3216 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: 1.8.0 > Environment: centos-6.8 >Reporter: jifei_yang >Priority: Critical > Fix For: 1.7.0, 1.8.0 > > > When I use Taildir source, whenever I rename a file, it causes flume to read > the data repeatedly.This resulted in duplication of uploaded data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Commented] (FLUME-3216) When I rename the file, it causes flume to read the data repeatedly.
[ https://issues.apache.org/jira/browse/FLUME-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350072#comment-16350072 ] Ferenc Szabo commented on FLUME-3216: - Renaming a file creates a new file, so the problem you describe here is the expected behavior. > When I rename the file, it causes flume to read the data repeatedly. > > > Key: FLUME-3216 > URL: https://issues.apache.org/jira/browse/FLUME-3216 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: 1.8.0 > Environment: centos-6.8 >Reporter: jifei_yang >Priority: Critical > Fix For: 1.7.0, 1.8.0 > > > When I use Taildir source, whenever I rename a file, it causes flume to read > the data repeatedly.This resulted in duplication of uploaded data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Commented] (FLUME-3216) When I rename the file, it causes flume to read the data repeatedly.
[ https://issues.apache.org/jira/browse/FLUME-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349776#comment-16349776 ] ASF GitHub Bot commented on FLUME-3216: --- GitHub user highfei2011 opened a pull request: https://github.com/apache/flume/pull/196 FLUME-3216.Avoid reading content repeatedly. When I use Taildir source, whenever I rename a file, it causes flume to read the data repeatedly.This resulted in duplication of uploaded data. You can merge this pull request into a Git repository by running: $ git pull https://github.com/highfei2011/flume trunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/196.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #196 commit db7996ca0e287aa78d68e460ee9287b731f0e5f1 Author: yangjifei Date: 2018-02-02T04:32:40Z FLUME-3216.Avoid reading content repeatedly. When I use Taildir source, whenever I rename a file, it causes flume to read the data repeatedly.This resulted in duplication of uploaded data. commit 574db14d91cdc10c6333ccfdf20c543d64303abc Author: yangjifei Date: 2018-02-02T04:38:02Z FLUME-3216.Avoid reading content repeatedly. When I use Taildir source, whenever I rename a file, it causes flume to read the data repeatedly.This resulted in duplication of uploaded data. > When I rename the file, it causes flume to read the data repeatedly. > > > Key: FLUME-3216 > URL: https://issues.apache.org/jira/browse/FLUME-3216 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: 1.8.0 > Environment: centos-6.8 >Reporter: jifei_yang >Priority: Critical > Fix For: 1.7.0, 1.8.0 > > > When I use Taildir source, whenever I rename a file, it causes flume to read > the data repeatedly.This resulted in duplication of uploaded data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
