Xinyuan Liu created FLUME-3392:
----------------------------------
Summary: kafka channel flume hdfs file close failed cause retry
and data losse
Key: FLUME-3392
URL: https://issues.apache.org/jira/browse/FLUME-3392
Project: Flume
Issue Type: Bug
Affects Versions: 1.9.0
Environment: flume 1.9.0
kafka channel
hdfs sink
Reporter: Xinyuan Liu
Due to the failure of hdfs BucketWriter close file and unlimited retries, the
pressure of nenode is too high, and data may be lost due to subsequent
successful consumption and offset submission.
{code:java}
2021-05-30 02:33:45,045 (hdfs-s1-roll-timer-0) [WARN -
org.apache.flume.sink.hdfs.BucketWriter$CloseHandler.close(BucketWriter.java:348)]
Closing file:
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
failed. Will retry again in 180 seconds.2021-05-30 02:33:45,045
(hdfs-s1-roll-timer-0) [WARN -
org.apache.flume.sink.hdfs.BucketWriter$CloseHandler.close(BucketWriter.java:348)]
Closing file:
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
failed. Will retry again in 180 seconds.java.io.IOException: Unable to close
file because the last block
BP-1006090754-10.1.53.214-1490513887497:blk_6365012859_5293069088 does not have
enough number of replicas. at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2865)
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2810)
at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2794)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2737) at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:135) at
org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:319) at
org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:316) at
org.apache.flume.sink.hdfs.BucketWriter$8$1.run(BucketWriter.java:727) at
org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:724) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)2021-05-30 02:33:45,062
(hdfs-s1-call-runner-12) [INFO -
org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:681)] Renaming
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
to
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/t_note_phone.1622312610822.json2021-05-30
02:34:04,157 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hdfs.HDFSDataStream.configure(HDFSDataStream.java:57)]
Serializer = TEXT, UseRawLocalFileSystem = false2021-05-30 02:34:04,181
(SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:246)] Creating
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622313244158.json.tmp2021-05-30
02:36:45,047 (hdfs-s1-call-runner-2) [ERROR -
org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:269)]
Error while trying to hflushOrSync!2021-05-30 02:36:45,048
(hdfs-s1-roll-timer-0) [WARN -
org.apache.flume.sink.hdfs.BucketWriter$CloseHandler.close(BucketWriter.java:348)]
Closing file:
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
failed. Will retry again in 180
seconds.java.nio.channels.ClosedChannelException at
org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:2039)
at
org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:2461)
at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:2395) at
org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:138) at
sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:266)
at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:134) at
org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:319) at
org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:316) at
org.apache.flume.sink.hdfs.BucketWriter$8$1.run(BucketWriter.java:727) at
org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:724) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
