Hello,
Recently, we’ve implemented storm 2.0.0 with Kerberos. Before integrating
Kerberos, the logs were fine and no mention of socket output being closed but
since we implemented Kerberos, we’re seeing the following in our logs :
2020-10-21 14:39:53.529 o.a.s.t.t.TIOStreamTransport refresh-connections-timer
[WARN] Error closing output stream.
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116)
~[?:1.8.0_45]
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
~[?:1.8.0_45]
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[?:1.8.0_45]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[?:1.8.0_45]
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
~[?:1.8.0_45]
at
org.apache.storm.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
[storm-shaded-deps-2.0.0.jar:2.0.0]
at org.apache.storm.thrift.transport.TSocket.close(TSocket.java:235)
[storm-shaded-deps-2.0.0.jar:2.0.0]
at
org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:106)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.utils.SupervisorClient.<init>(SupervisorClient.java:45)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.utils.SupervisorClient.getConfiguredClientAs(SupervisorClient.java:73)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.utils.SupervisorClient.getConfiguredClient(SupervisorClient.java:61)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.daemon.worker.WorkerState.getLocalAssignment(WorkerState.java:647)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.daemon.worker.WorkerState.refreshConnections(WorkerState.java:367)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.daemon.worker.WorkerState.refreshConnections(WorkerState.java:356)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.daemon.worker.Worker$$Lambda$61/1922191129.run(Unknown Source)
[storm-client-2.0.0.jar:2.0.0]
at org.apache.storm.StormTimer$1.run(StormTimer.java:110)
[storm-client-2.0.0.jar:2.0.0]
at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:226)
[storm-client-2.0.0.jar:2.0.0]
2020-10-21 14:43:37.827 o.a.s.t.t.TIOStreamTransport heartbeat-timer [WARN]
Error closing output stream.
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116)
~[?:1.8.0_45]
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
~[?:1.8.0_45]
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[?:1.8.0_45]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[?:1.8.0_45]
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
~[?:1.8.0_45]
at
org.apache.storm.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
[storm-shaded-deps-2.0.0.jar:2.0.0]
at org.apache.storm.thrift.transport.TSocket.close(TSocket.java:235)
[storm-shaded-deps-2.0.0.jar:2.0.0]
at
org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:106)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.utils.SupervisorClient.<init>(SupervisorClient.java:45)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.utils.SupervisorClient.getConfiguredClientAs(SupervisorClient.java:73)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.utils.SupervisorClient.getConfiguredClient(SupervisorClient.java:61)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.daemon.worker.Worker.heartbeatToMasterIfLocalbeatFail(Worker.java:429)
[storm-client-2.0.0.jar:2.0.0]
at org.apache.storm.daemon.worker.Worker.doHeartBeat(Worker.java:344)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.daemon.worker.Worker.lambda$loadWorker$1(Worker.java:189)
[storm-client-2.0.0.jar:2.0.0]
at
org.apache.storm.daemon.worker.Worker$$Lambda$30/435297131.run(Unknown Source)
[storm-client-2.0.0.jar:2.0.0]
at org.apache.storm.StormTimer$1.run(StormTimer.java:110)
[storm-client-2.0.0.jar:2.0.0]
at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:226)
[storm-client-2.0.0.jar:2.0.0]
2020-10-21 14:43:37.827 o.a.s.d.w.Worker heartbeat-timer [WARN] Exception when
send heartbeat to local supervisor
I’ve done some searching and tried adjusting the worker heartbeat frequency sec
but that did more harm than good. I’ve followed the trace and the only
configuration that I see that is being pulled in is the
worker.heartbeat.freqs.secs set at 1.
Is this a known issue or has anyone come across this before? Any thoughts on
what might be the issue and if it’s critical? Thanks for your help.
Best,
Peter
This message may contain information that is confidential or privileged. If you
are not the intended recipient, please advise the sender immediately and delete
this message. See
http://www.blackrock.com/corporate/compliance/email-disclaimers for further
information. Please refer to
http://www.blackrock.com/corporate/compliance/privacy-policy for more
information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see
http://www.blackrock.com/corporate/about-us/contacts-locations.
© 2020 BlackRock, Inc. All rights reserved.
