Log and cry are as below: http://paste.debian.net/901028
https://paste.debian.net/901029 Actually there is no error or exception thrown. The problem is when observing 204 then flume stops consuming(only keep printing 'checkpoint not required' msg to log), resulting in the records in source and sink are not the same. This usually happens with consuming more than 15k records. When the number of records are not big e.g. 5k everything works fine (after moving data from source to sink, the record number is the same). And at this moment no transformation, etc. involves, just plain moving stuff from source to sink. Thanks for help. > On 05 December 2016 at 22:36 iain wright <[email protected]> wrote: > > seems http 204 is exepected response from S3 delete, can you pastebin > your log/cfg (please omit keys if present in cfg)? > > Does your sink stop consuming from the channel when that error is > encountered? > > > -- > Iain Wright > > This email message is confidential, intended only for the recipient(s) > named above and may contain information that is privileged, exempt from > disclosure under applicable law. If you are not the intended recipient, do > not disclose or disseminate the message to anyone except the intended > recipient. If you have received this message in error, or are not the named > recipient(s), please immediately notify the sender by return email, and > delete all copies of this message. > > On Mon, Dec 5, 2016 at 12:27 PM, Frank Norris <[email protected] > mailto:[email protected] > wrote: > > > > > > I use flume v1.6 and write data to s3 sink. However flume stops > > processing (no crash still print log message) when it receives http code > > 204. In log it shows flume was deleting files. Any places I can check to > > avoid this problem or further investigation? > > > > Thanks > > > > > >
