[
https://issues.apache.org/jira/browse/STORM-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved STORM-2993.
----------------------------------------
Resolution: Fixed
Fix Version/s: 1.2.2
2.0.0
Thanks [~arunmahadevan],
I merged this into master and branch-1.x. there were merge conflicts, but
appeared to be minor ones, for 1.1.x so I didn't pull it in there. If you want
it pulled back please file another pull request for that.
> Storm HDFS bolt throws ClosedChannelException when Time rotation policy is
> used
> -------------------------------------------------------------------------------
>
> Key: STORM-2993
> URL: https://issues.apache.org/jira/browse/STORM-2993
> Project: Apache Storm
> Issue Type: Bug
> Reporter: Arun Mahadevan
> Assignee: Arun Mahadevan
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.2
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Storm connector throws below error in the worker logs.
>
> 2018-03-12 18:14:58.123 o.a.s.h.c.r.MoveFileAction Timer-3 [INFO] Moving file
> hdfs://ctr-e138-1518143905142-85179-01-000004.hwx.site:8020/tmp/foo/my-bolt-3-0-1520878438104.txt
> to /tmp/dest2/my-bolt-3-0-15 20878438104.txt 2018-03-12 18:14:58.123
> o.a.s.h.c.r.MoveFileAction Timer-0 [INFO] Moving file
> hdfs://ctr-e138-1518143905142-85179-01-000004.hwx.site:8020/tmp/foo/my-bolt-6-0-1520878438104.txt
> to /tmp/dest2/my-bolt-6-0-15 20878438104.txt 2018-03-12 18:14:58.123
> o.a.s.h.c.r.MoveFileAction Timer-1 [INFO] Moving file
> hdfs://ctr-e138-1518143905142-85179-01-000004.hwx.site:8020/tmp/foo/my-bolt-5-0-1520878438104.txt
> to /tmp/dest2/my-bolt-5-0-15 20878438104.txt 2018-03-12 18:14:58.124
> o.a.s.h.c.r.MoveFileAction Timer-2 [INFO] Moving file
> hdfs://ctr-e138-1518143905142-85179-01-000004.hwx.site:8020/tmp/foo/my-bolt-4-0-1520878438104.txt
> to /tmp/dest2/my-bolt-4-0-15 20878438104.txt 2018-03-12 18:14:58.132
> o.a.s.h.b.AbstractHdfsBolt Timer-2 [INFO] File rotation took 28 ms.
> 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt Timer-0 [INFO] File
> rotation took 29 ms. 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt
> Timer-3 [INFO] File rotation took 28 ms. 2018-03-12 18:14:58.132
> o.a.s.h.b.AbstractHdfsBolt Timer-1 [INFO] File rotation took 28 ms.
> 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt
> Thread-12-my-bolt-executor[6 6] [INFO] Tuple failed to write, forcing a flush
> of existing data. 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt
> Thread-8-my-bolt-executor[3 3] [INFO] Tuple failed to write, forcing a flush
> of existing data. 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt
> Thread-16-my-bolt-executor[5 5] [INFO] Tuple failed to write, forcing a flush
> of existing data. 2018-03-12 18:14:58.132 o.a.s.d.executor
> Thread-8-my-bolt-executor[3 3] [ERROR]
> java.nio.channels.ClosedChannelException: null at
> org.apache.hadoop.hdfs.ExceptionLastSeen.throwException4Close(ExceptionLastSeen.java:73)
> ~[stormjar.jar:?] at
> org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:153)
> ~[stormjar.jar:?] at
> org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:105)
> ~[stormjar.jar:?] at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57)
> ~[stormjar.jar:?] at
> java.io.DataOutputStream.write(DataOutputStream.java:107) ~[?:1.8.0_161] at
> java.io.FilterOutputStream.write(FilterOutputStream.java:97) ~[?:1.8.0_161]
> at org.apache.storm.hdfs.common.HDFSWriter.doWrite(HDFSWriter.java:48)
> ~[stormjar.jar:?] at
> org.apache.storm.hdfs.common.AbstractHDFSWriter.write(AbstractHDFSWriter.java:40)
> ~[stormjar.jar:?] at
> org.apache.storm.hdfs.bolt.AbstractHdfsBolt.execute(AbstractHdfsBolt.java:158)
> [stormjar.jar:?] at
> org.apache.storm.daemon.executor$fn__10189$tuple_action_fn__10191.invoke(executor.clj:745)
> [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10108.invoke(executor.clj:473)
> [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at
> org.apache.storm.disruptor$clojure_handler$reify__4115.onEvent(disruptor.clj:41)
> [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:509)
> [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:487)
> [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:74)
> [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at
> org.apache.storm.daemon.executor$fn__10189$fn__10202$fn__10257.invoke(executor.clj:868)
> [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at
> org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484)
> [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at
> clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at
> java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] 2018-03-12 18:14:58.133
> o.a.s.d.executor Thread-16-my-bolt-executor[5 5]
>
> ----
>
> Apparently the Timed rotation policy does not synchronize properly so its
> possible that the HDFS bolt code can attempt to write to a closed writer.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
