[
https://issues.apache.org/jira/browse/FLUME-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinyuan Liu updated FLUME-3392:
-------------------------------
Description:
Due to the failure of hdfs BucketWriter close file and unlimited retries, the
pressure of nenode is too high, and data may be lost due to subsequent
successful consumption and offset submission.
{code:java}
2021-05-30 02:33:45,045 (hdfs-s1-roll-timer-0) [WARN -
org.apache.flume.sink.hdfs.BucketWriter$CloseHandler.close(BucketWriter.java:348)]
Closing file:
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
failed. Will retry again in 180 seconds.2021-05-30 02:33:45,045
(hdfs-s1-roll-timer-0) [WARN -
org.apache.flume.sink.hdfs.BucketWriter$CloseHandler.close(BucketWriter.java:348)]
Closing file:
hdfs://x.x.x.x:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
failed. Will retry again in 180 seconds.java.io.IOException: Unable to close
file because the last block
BP-1006090754-10.1.53.214-1490513887497:blk_6365012859_5293069088 does not have
enough number of replicas. at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2865)
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2810)
at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2794)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2737) at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:135) at
org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:319) at
org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:316) at
org.apache.flume.sink.hdfs.BucketWriter$8$1.run(BucketWriter.java:727) at
org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:724) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)2021-05-30 02:33:45,062
(hdfs-s1-call-runner-12) [INFO -
org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:681)] Renaming
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
to
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/t_note_phone.1622312610822.json2021-05-30
02:34:04,157 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hdfs.HDFSDataStream.configure(HDFSDataStream.java:57)]
Serializer = TEXT, UseRawLocalFileSystem = false2021-05-30 02:34:04,181
(SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:246)] Creating
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622313244158.json.tmp2021-05-30
02:36:45,047 (hdfs-s1-call-runner-2) [ERROR -
org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:269)]
Error while trying to hflushOrSync!2021-05-30 02:36:45,048
(hdfs-s1-roll-timer-0) [WARN -
org.apache.flume.sink.hdfs.BucketWriter$CloseHandler.close(BucketWriter.java:348)]
Closing file:
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
failed. Will retry again in 180
seconds.java.nio.channels.ClosedChannelException at
org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:2039)
at
org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:2461)
at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:2395) at
org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:138) at
sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:266)
at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:134) at
org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:319) at
org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:316) at
org.apache.flume.sink.hdfs.BucketWriter$8$1.run(BucketWriter.java:727) at
org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:724) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
{code}
was:
Due to the failure of hdfs BucketWriter close file and unlimited retries, the
pressure of nenode is too high, and data may be lost due to subsequent
successful consumption and offset submission.
{code:java}
2021-05-30 02:33:45,045 (hdfs-s1-roll-timer-0) [WARN -
org.apache.flume.sink.hdfs.BucketWriter$CloseHandler.close(BucketWriter.java:348)]
Closing file:
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
failed. Will retry again in 180 seconds.2021-05-30 02:33:45,045
(hdfs-s1-roll-timer-0) [WARN -
org.apache.flume.sink.hdfs.BucketWriter$CloseHandler.close(BucketWriter.java:348)]
Closing file:
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
failed. Will retry again in 180 seconds.java.io.IOException: Unable to close
file because the last block
BP-1006090754-10.1.53.214-1490513887497:blk_6365012859_5293069088 does not have
enough number of replicas. at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2865)
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2810)
at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2794)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2737) at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:135) at
org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:319) at
org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:316) at
org.apache.flume.sink.hdfs.BucketWriter$8$1.run(BucketWriter.java:727) at
org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:724) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)2021-05-30 02:33:45,062
(hdfs-s1-call-runner-12) [INFO -
org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:681)] Renaming
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
to
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/t_note_phone.1622312610822.json2021-05-30
02:34:04,157 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hdfs.HDFSDataStream.configure(HDFSDataStream.java:57)]
Serializer = TEXT, UseRawLocalFileSystem = false2021-05-30 02:34:04,181
(SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:246)] Creating
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622313244158.json.tmp2021-05-30
02:36:45,047 (hdfs-s1-call-runner-2) [ERROR -
org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:269)]
Error while trying to hflushOrSync!2021-05-30 02:36:45,048
(hdfs-s1-roll-timer-0) [WARN -
org.apache.flume.sink.hdfs.BucketWriter$CloseHandler.close(BucketWriter.java:348)]
Closing file:
hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
failed. Will retry again in 180
seconds.java.nio.channels.ClosedChannelException at
org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:2039)
at
org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:2461)
at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:2395) at
org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:138) at
sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:266)
at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:134) at
org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:319) at
org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:316) at
org.apache.flume.sink.hdfs.BucketWriter$8$1.run(BucketWriter.java:727) at
org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:724) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
{code}
> kafka channel flume hdfs file close failed cause retry and data losse
> ---------------------------------------------------------------------
>
> Key: FLUME-3392
> URL: https://issues.apache.org/jira/browse/FLUME-3392
> Project: Flume
> Issue Type: Bug
> Affects Versions: 1.9.0
> Environment: flume 1.9.0
> kafka channel
> hdfs sink
> Reporter: Xinyuan Liu
> Priority: Blocker
>
>
> Due to the failure of hdfs BucketWriter close file and unlimited retries, the
> pressure of nenode is too high, and data may be lost due to subsequent
> successful consumption and offset submission.
> {code:java}
> 2021-05-30 02:33:45,045 (hdfs-s1-roll-timer-0) [WARN -
> org.apache.flume.sink.hdfs.BucketWriter$CloseHandler.close(BucketWriter.java:348)]
> Closing file:
> hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
> failed. Will retry again in 180 seconds.2021-05-30 02:33:45,045
> (hdfs-s1-roll-timer-0) [WARN -
> org.apache.flume.sink.hdfs.BucketWriter$CloseHandler.close(BucketWriter.java:348)]
> Closing file:
> hdfs://x.x.x.x:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
> failed. Will retry again in 180 seconds.java.io.IOException: Unable to close
> file because the last block
> BP-1006090754-10.1.53.214-1490513887497:blk_6365012859_5293069088 does not
> have enough number of replicas. at
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2865)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2810)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2794)
> at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2737) at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) at
> org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:135) at
> org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:319) at
> org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:316) at
> org.apache.flume.sink.hdfs.BucketWriter$8$1.run(BucketWriter.java:727) at
> org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
> at org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:724) at
> java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)2021-05-30 02:33:45,062
> (hdfs-s1-call-runner-12) [INFO -
> org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:681)]
> Renaming
> hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
> to
> hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/t_note_phone.1622312610822.json2021-05-30
> 02:34:04,157 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
> org.apache.flume.sink.hdfs.HDFSDataStream.configure(HDFSDataStream.java:57)]
> Serializer = TEXT, UseRawLocalFileSystem = false2021-05-30 02:34:04,181
> (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO -
> org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:246)] Creating
> hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622313244158.json.tmp2021-05-30
> 02:36:45,047 (hdfs-s1-call-runner-2) [ERROR -
> org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:269)]
> Error while trying to hflushOrSync!2021-05-30 02:36:45,048
> (hdfs-s1-roll-timer-0) [WARN -
> org.apache.flume.sink.hdfs.BucketWriter$CloseHandler.close(BucketWriter.java:348)]
> Closing file:
> hdfs://10.1.53.19:9020/warehouse/ods_flume_json_yunc_transcation/ods_flume_json_thirdparty_db__t_note_phone/datelabel=20210530/__t_note_phonet_note_phone.1622312610822.json.tmp
> failed. Will retry again in 180
> seconds.java.nio.channels.ClosedChannelException at
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:2039)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:2461)
> at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:2395)
> at
> org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:138)
> at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:266)
> at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:134)
> at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:319) at
> org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:316) at
> org.apache.flume.sink.hdfs.BucketWriter$8$1.run(BucketWriter.java:727) at
> org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
> at org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:724) at
> java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
