[
https://issues.apache.org/jira/browse/FLUME-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
67287685 updated FLUME-3233:
----------------------------
Description:
we use the flume 1.7 to collect our log send to kafka.
our java application compressed the log at every day 00:00,the log name change
aaa.log -> aaa.20180503.tmp -> aaa.20180503.log.gz,the change while complete
less than 1 second time.
this have a problem ,when have some data into log at 23:59:59,the log while
compress fast,the flume not collect the new data,the log is become zip file,so
some data will lost.
To solve this problem，we Modify the java application and flume
1.modify java application
the log is compress delay 5 minutes，aaa.log --> aaa.20180503.tmp -> after 5 min
-> aaa.20180503.log.gz
the flume have enough time complete collect the data .
2.modify flume 1.7
use inode judge the file uniqueness，not use inode + filename ,when use inode +
filename，the flume while repeate collect when the aaa.log change to
aaa.20180503.tmp.this two file have the same inode,we will not repeat collect
the file.
Please help to see if our plan is feasible，thanks every .
was:
we use the flume 1.7 to collect our log send to kafka.
our java application compressed the log at every day 00:00,the log name change
aaa.log -> aaa.20180503.tmp -> aaa.20180503.log.gz,the change while complete
less than 1 second time.
this have a problem ,when have some data into log at 23:59:59,the log while
compress fast,the flume while not collect the new data,the log is become zip
file,the data is lost.
To solve this problem，we Modify the java application and flume
1.modify java application
the log is compress delay 5 minutes，aaa.log --> aaa.20180503.tmp -> after 5 min
-> aaa.20180503.log.gz,the flume while complete collect the data ,not lost.
2.modify flume 1.7
use inode judge the file uniqueness，not use inode + filename ,when use inode +
filename，the flume while repeate collect when the aaa.log -->
aaa.20180503.tmp,this two file have the same inode
Please help to see if our plan is feasible，thanks every .
> log roll to a new name,the flume collect repeat
> -----------------------------------------------
>
> Key: FLUME-3233
> URL: https://issues.apache.org/jira/browse/FLUME-3233
> Project: Flume
> Issue Type: Bug
> Affects Versions: 1.7.0
> Reporter: 67287685
> Priority: Major
>
> we use the flume 1.7 to collect our log send to kafka.
> our java application compressed the log at every day 00:00,the log name
> change aaa.log -> aaa.20180503.tmp -> aaa.20180503.log.gz,the change while
> complete less than 1 second time.
> this have a problem ,when have some data into log at 23:59:59,the log while
> compress fast,the flume not collect the new data,the log is become zip
> file,so some data will lost.
> To solve this problem，we Modify the java application and flume
> 1.modify java application
> the log is compress delay 5 minutes，aaa.log --> aaa.20180503.tmp -> after 5
> min -> aaa.20180503.log.gz
> the flume have enough time complete collect the data .
>
> 2.modify flume 1.7
> use inode judge the file uniqueness，not use inode + filename ,when use inode
> + filename，the flume while repeate collect when the aaa.log change to
> aaa.20180503.tmp.this two file have the same inode,we will not repeat collect
> the file.
>
> Please help to see if our plan is feasible，thanks every .
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
