[
https://issues.apache.org/jira/browse/FLUME-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pan Yuxuan updated FLUME-3054:
------------------------------
Attachment: FLUME-3054.0001.patch
Attach the initial patch.
> hflushOrSync method in HDFS Sink should not treat ClosedChannelException as
> an error
> -------------------------------------------------------------------------------------
>
> Key: FLUME-3054
> URL: https://issues.apache.org/jira/browse/FLUME-3054
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Reporter: Pan Yuxuan
> Fix For: v1.8.0
>
> Attachments: FLUME-3054.0001.patch
>
>
> When use HDFS Sink in multiple threads, we face the error in log as below:
> {code}
> 09 Feb 2017 13:44:14,721 ERROR [hdfs-hsProt6-call-runner-4]
> (org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync:267) - Error
> while trying to hflushOrSync!
> 09 Feb 2017 14:54:48,271 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated:98) -
> Unexpected error while checking replication factor
> java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.flume.sink.hdfs.AbstractHDFSWriter.getNumCurrentReplicas(AbstractHDFSWriter.java:165)
> at
> org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated(AbstractHDFSWriter.java:84)
> at
> org.apache.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java:583)
> at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:518)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:418)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1665)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2151)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.getNumCurrentReplicas(DFSOutputStream.java:2140)
> ... 11 more
> 09 Feb 2017 14:54:48,277 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.HDFSEventSink.process:459) - process failed
> org.apache.flume.auth.SecurityException: Privileged action failed
> at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:49)
> at
> org.apache.flume.auth.KerberosAuthenticator.execute(KerberosAuthenticator.java:63)
> at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1665)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
> at
> org.apache.flume.serialization.BodyTextEventSerializer.write(BodyTextEventSerializer.java:71)
> at
> org.apache.flume.sink.hdfs.HDFSDataStream.append(HDFSDataStream.java:124)
> at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:550)
> at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:547)
> at
> org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1858)
> at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:47)
> ... 6 more
> 09 Feb 2017 14:54:48,280 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver
> event. Exception follows.
> org.apache.flume.EventDeliveryException:
> org.apache.flume.auth.SecurityException: Privileged action failed
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:463)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.flume.auth.SecurityException: Privileged action failed
> at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:49)
> at
> org.apache.flume.auth.KerberosAuthenticator.execute(KerberosAuthenticator.java:63)
> at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ... 1 more
> Caused by: java.nio.channels.ClosedChannelException
> at
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1665)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
> at
> org.apache.flume.serialization.BodyTextEventSerializer.write(BodyTextEventSerializer.java:71)
> at
> org.apache.flume.sink.hdfs.HDFSDataStream.append(HDFSDataStream.java:124)
> at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:550)
> at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:547)
> at
> org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1858)
> at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:47)
> ... 6 more
> {code}
> The error shows that when call hflushOrSync(), the DataOutputStream was
> closed by other threads, so DataOutputStream throws ClosedChannelException.
> But the hdfs sink was not affected by the exception, DataOutputStream just
> throws the ClosedChannelException as a remind. So we should ignore the
> ClosedChannelException instead by logging as an error.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
