[jira] [Updated] (CASSANDRA-16143) Streaming fails when s SSTable writer finish() exceeds internode_tcp_user_timeout
[ https://issues.apache.org/jira/browse/CASSANDRA-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-16143: --- Fix Version/s: (was: 4.0-beta) 4.0-beta4 Since Version: 4.0-alpha1 Source Control Link: https://github.com/apache/cassandra/commit/c6ea8fabf9f726a58b76ff6e41d21b650941f880 Resolution: Fixed Status: Resolved (was: Ready to Commit) Committed into trunk at c6ea8fabf9f726a58b76ff6e41d21b650941f880 > Streaming fails when s SSTable writer finish() exceeds > internode_tcp_user_timeout > - > > Key: CASSANDRA-16143 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16143 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Jon Meredith >Assignee: Yifan Cai >Priority: Normal > Fix For: 4.0-beta4 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > tl;dr The internode TCP user timeout that provides more responsive detection > of dead nodes for internode message will cause streaming to fail if system > calls to fsync/fdatasync exceed the timeout (default 30s). > To workaround, explicitly set internode_tcp_user_timeout to longer than > fsync/fdatasync, or to zero to revert to the operating system default. > Details: > While bootstrapping a replacement 4.0beta3 node in an existing cluster, > bootstrap streaming repeatedly failed with the streaming follower logging > {code:java} > ERROR 2020-09-10T14:29:34,711 [NettyStreaming-Outbound-1.1.1.1.7000:1] > org.apache.cassandra.streaming.StreamSession:693 - [Stream > #7cb67c00-f3ac-11ea-b940-f7836f164528] Streaming error occurred on session > with peer 1.1.1.1:7000 > org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The channel > this output stream was writing to has been closed >at > org.apache.cassandra.net.AsyncChannelOutputPlus.propagateFailedFlush(AsyncChannelOutputPlus.java:200) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitUntilFlushed(AsyncChannelOutputPlus.java:158) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitForSpace(AsyncChannelOutputPlus.java:140) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.beginFlush(AsyncChannelOutputPlus.java:97) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.lambda$writeToChannel$0(AsyncStreamingOutputPlus.java:142) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.lambda$write$0(CassandraCompressedStreamWriter.java:90) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.writeToChannel(AsyncStreamingOutputPlus.java:138) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.write(CassandraCompressedStreamWriter.java:89) >at > org.apache.cassandra.db.streaming.CassandraOutgoingFile.write(CassandraOutgoingFile.java:180) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage.serialize(OutgoingStreamMessage.java:87) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:45) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:34) >at > org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:40) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:347) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [?:?] >at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > [netty-all-4.1.50.Final.jar:4.1.50.Final] >at java.lang.Thread.run(Thread.java:834) [?:?] >Suppressed: java.nio.channels.ClosedChannelException >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.doFlush(AsyncStreamingOutputPlus.java:78) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.flush(AsyncChannelOutputPlus.java:229) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.close(AsyncChannelOutputPlus.java:248) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:348) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.
[jira] [Updated] (CASSANDRA-16143) Streaming fails when s SSTable writer finish() exceeds internode_tcp_user_timeout
[ https://issues.apache.org/jira/browse/CASSANDRA-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-16143: --- Reviewers: Adam Holmberg, Benjamin Lerer, Berenguer Blasi, David Capwell, Benjamin Lerer (was: Adam Holmberg, Benjamin Lerer, Berenguer Blasi, David Capwell) Adam Holmberg, Benjamin Lerer, Berenguer Blasi, David Capwell, Benjamin Lerer (was: Adam Holmberg, Benjamin Lerer, Berenguer Blasi, David Capwell) Status: Review In Progress (was: Patch Available) > Streaming fails when s SSTable writer finish() exceeds > internode_tcp_user_timeout > - > > Key: CASSANDRA-16143 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16143 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Jon Meredith >Assignee: Yifan Cai >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 3h 40m > Remaining Estimate: 0h > > tl;dr The internode TCP user timeout that provides more responsive detection > of dead nodes for internode message will cause streaming to fail if system > calls to fsync/fdatasync exceed the timeout (default 30s). > To workaround, explicitly set internode_tcp_user_timeout to longer than > fsync/fdatasync, or to zero to revert to the operating system default. > Details: > While bootstrapping a replacement 4.0beta3 node in an existing cluster, > bootstrap streaming repeatedly failed with the streaming follower logging > {code:java} > ERROR 2020-09-10T14:29:34,711 [NettyStreaming-Outbound-1.1.1.1.7000:1] > org.apache.cassandra.streaming.StreamSession:693 - [Stream > #7cb67c00-f3ac-11ea-b940-f7836f164528] Streaming error occurred on session > with peer 1.1.1.1:7000 > org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The channel > this output stream was writing to has been closed >at > org.apache.cassandra.net.AsyncChannelOutputPlus.propagateFailedFlush(AsyncChannelOutputPlus.java:200) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitUntilFlushed(AsyncChannelOutputPlus.java:158) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitForSpace(AsyncChannelOutputPlus.java:140) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.beginFlush(AsyncChannelOutputPlus.java:97) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.lambda$writeToChannel$0(AsyncStreamingOutputPlus.java:142) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.lambda$write$0(CassandraCompressedStreamWriter.java:90) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.writeToChannel(AsyncStreamingOutputPlus.java:138) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.write(CassandraCompressedStreamWriter.java:89) >at > org.apache.cassandra.db.streaming.CassandraOutgoingFile.write(CassandraOutgoingFile.java:180) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage.serialize(OutgoingStreamMessage.java:87) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:45) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:34) >at > org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:40) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:347) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [?:?] >at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > [netty-all-4.1.50.Final.jar:4.1.50.Final] >at java.lang.Thread.run(Thread.java:834) [?:?] >Suppressed: java.nio.channels.ClosedChannelException >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.doFlush(AsyncStreamingOutputPlus.java:78) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.flush(AsyncChannelOutputPlus.java:229) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.close(AsyncChannelOutputPlus.java:248) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:348) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java
[jira] [Updated] (CASSANDRA-16143) Streaming fails when s SSTable writer finish() exceeds internode_tcp_user_timeout
[ https://issues.apache.org/jira/browse/CASSANDRA-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-16143: --- Status: Ready to Commit (was: Review In Progress) > Streaming fails when s SSTable writer finish() exceeds > internode_tcp_user_timeout > - > > Key: CASSANDRA-16143 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16143 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Jon Meredith >Assignee: Yifan Cai >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 3h 40m > Remaining Estimate: 0h > > tl;dr The internode TCP user timeout that provides more responsive detection > of dead nodes for internode message will cause streaming to fail if system > calls to fsync/fdatasync exceed the timeout (default 30s). > To workaround, explicitly set internode_tcp_user_timeout to longer than > fsync/fdatasync, or to zero to revert to the operating system default. > Details: > While bootstrapping a replacement 4.0beta3 node in an existing cluster, > bootstrap streaming repeatedly failed with the streaming follower logging > {code:java} > ERROR 2020-09-10T14:29:34,711 [NettyStreaming-Outbound-1.1.1.1.7000:1] > org.apache.cassandra.streaming.StreamSession:693 - [Stream > #7cb67c00-f3ac-11ea-b940-f7836f164528] Streaming error occurred on session > with peer 1.1.1.1:7000 > org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The channel > this output stream was writing to has been closed >at > org.apache.cassandra.net.AsyncChannelOutputPlus.propagateFailedFlush(AsyncChannelOutputPlus.java:200) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitUntilFlushed(AsyncChannelOutputPlus.java:158) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitForSpace(AsyncChannelOutputPlus.java:140) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.beginFlush(AsyncChannelOutputPlus.java:97) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.lambda$writeToChannel$0(AsyncStreamingOutputPlus.java:142) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.lambda$write$0(CassandraCompressedStreamWriter.java:90) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.writeToChannel(AsyncStreamingOutputPlus.java:138) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.write(CassandraCompressedStreamWriter.java:89) >at > org.apache.cassandra.db.streaming.CassandraOutgoingFile.write(CassandraOutgoingFile.java:180) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage.serialize(OutgoingStreamMessage.java:87) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:45) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:34) >at > org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:40) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:347) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [?:?] >at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > [netty-all-4.1.50.Final.jar:4.1.50.Final] >at java.lang.Thread.run(Thread.java:834) [?:?] >Suppressed: java.nio.channels.ClosedChannelException >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.doFlush(AsyncStreamingOutputPlus.java:78) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.flush(AsyncChannelOutputPlus.java:229) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.close(AsyncChannelOutputPlus.java:248) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:348) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [?:?] >at > io.netty.util.concurrent
[jira] [Updated] (CASSANDRA-16143) Streaming fails when s SSTable writer finish() exceeds internode_tcp_user_timeout
[ https://issues.apache.org/jira/browse/CASSANDRA-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-16143: -- Reviewers: Adam Holmberg, Benjamin Lerer, Berenguer Blasi, David Capwell (was: Adam Holmberg, Benjamin Lerer, Berenguer Blasi) > Streaming fails when s SSTable writer finish() exceeds > internode_tcp_user_timeout > - > > Key: CASSANDRA-16143 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16143 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Jon Meredith >Assignee: Yifan Cai >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 3h 10m > Remaining Estimate: 0h > > tl;dr The internode TCP user timeout that provides more responsive detection > of dead nodes for internode message will cause streaming to fail if system > calls to fsync/fdatasync exceed the timeout (default 30s). > To workaround, explicitly set internode_tcp_user_timeout to longer than > fsync/fdatasync, or to zero to revert to the operating system default. > Details: > While bootstrapping a replacement 4.0beta3 node in an existing cluster, > bootstrap streaming repeatedly failed with the streaming follower logging > {code:java} > ERROR 2020-09-10T14:29:34,711 [NettyStreaming-Outbound-1.1.1.1.7000:1] > org.apache.cassandra.streaming.StreamSession:693 - [Stream > #7cb67c00-f3ac-11ea-b940-f7836f164528] Streaming error occurred on session > with peer 1.1.1.1:7000 > org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The channel > this output stream was writing to has been closed >at > org.apache.cassandra.net.AsyncChannelOutputPlus.propagateFailedFlush(AsyncChannelOutputPlus.java:200) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitUntilFlushed(AsyncChannelOutputPlus.java:158) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitForSpace(AsyncChannelOutputPlus.java:140) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.beginFlush(AsyncChannelOutputPlus.java:97) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.lambda$writeToChannel$0(AsyncStreamingOutputPlus.java:142) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.lambda$write$0(CassandraCompressedStreamWriter.java:90) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.writeToChannel(AsyncStreamingOutputPlus.java:138) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.write(CassandraCompressedStreamWriter.java:89) >at > org.apache.cassandra.db.streaming.CassandraOutgoingFile.write(CassandraOutgoingFile.java:180) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage.serialize(OutgoingStreamMessage.java:87) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:45) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:34) >at > org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:40) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:347) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [?:?] >at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > [netty-all-4.1.50.Final.jar:4.1.50.Final] >at java.lang.Thread.run(Thread.java:834) [?:?] >Suppressed: java.nio.channels.ClosedChannelException >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.doFlush(AsyncStreamingOutputPlus.java:78) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.flush(AsyncChannelOutputPlus.java:229) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.close(AsyncChannelOutputPlus.java:248) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:348) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
[jira] [Updated] (CASSANDRA-16143) Streaming fails when s SSTable writer finish() exceeds internode_tcp_user_timeout
[ https://issues.apache.org/jira/browse/CASSANDRA-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-16143: --- Reviewers: Adam Holmberg, Benjamin Lerer, Berenguer Blasi (was: Adam Holmberg, Berenguer Blasi) > Streaming fails when s SSTable writer finish() exceeds > internode_tcp_user_timeout > - > > Key: CASSANDRA-16143 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16143 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Jon Meredith >Assignee: Yifan Cai >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 1h > Remaining Estimate: 0h > > tl;dr The internode TCP user timeout that provides more responsive detection > of dead nodes for internode message will cause streaming to fail if system > calls to fsync/fdatasync exceed the timeout (default 30s). > To workaround, explicitly set internode_tcp_user_timeout to longer than > fsync/fdatasync, or to zero to revert to the operating system default. > Details: > While bootstrapping a replacement 4.0beta3 node in an existing cluster, > bootstrap streaming repeatedly failed with the streaming follower logging > {code:java} > ERROR 2020-09-10T14:29:34,711 [NettyStreaming-Outbound-1.1.1.1.7000:1] > org.apache.cassandra.streaming.StreamSession:693 - [Stream > #7cb67c00-f3ac-11ea-b940-f7836f164528] Streaming error occurred on session > with peer 1.1.1.1:7000 > org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The channel > this output stream was writing to has been closed >at > org.apache.cassandra.net.AsyncChannelOutputPlus.propagateFailedFlush(AsyncChannelOutputPlus.java:200) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitUntilFlushed(AsyncChannelOutputPlus.java:158) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitForSpace(AsyncChannelOutputPlus.java:140) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.beginFlush(AsyncChannelOutputPlus.java:97) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.lambda$writeToChannel$0(AsyncStreamingOutputPlus.java:142) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.lambda$write$0(CassandraCompressedStreamWriter.java:90) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.writeToChannel(AsyncStreamingOutputPlus.java:138) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.write(CassandraCompressedStreamWriter.java:89) >at > org.apache.cassandra.db.streaming.CassandraOutgoingFile.write(CassandraOutgoingFile.java:180) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage.serialize(OutgoingStreamMessage.java:87) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:45) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:34) >at > org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:40) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:347) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [?:?] >at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > [netty-all-4.1.50.Final.jar:4.1.50.Final] >at java.lang.Thread.run(Thread.java:834) [?:?] >Suppressed: java.nio.channels.ClosedChannelException >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.doFlush(AsyncStreamingOutputPlus.java:78) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.flush(AsyncChannelOutputPlus.java:229) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.close(AsyncChannelOutputPlus.java:248) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:348) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [?:?] >
[jira] [Updated] (CASSANDRA-16143) Streaming fails when s SSTable writer finish() exceeds internode_tcp_user_timeout
[ https://issues.apache.org/jira/browse/CASSANDRA-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-16143: Reviewers: Adam Holmberg, Berenguer Blasi (was: Adam Holmberg) > Streaming fails when s SSTable writer finish() exceeds > internode_tcp_user_timeout > - > > Key: CASSANDRA-16143 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16143 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Jon Meredith >Assignee: Yifan Cai >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 0.5h > Remaining Estimate: 0h > > tl;dr The internode TCP user timeout that provides more responsive detection > of dead nodes for internode message will cause streaming to fail if system > calls to fsync/fdatasync exceed the timeout (default 30s). > To workaround, explicitly set internode_tcp_user_timeout to longer than > fsync/fdatasync, or to zero to revert to the operating system default. > Details: > While bootstrapping a replacement 4.0beta3 node in an existing cluster, > bootstrap streaming repeatedly failed with the streaming follower logging > {code:java} > ERROR 2020-09-10T14:29:34,711 [NettyStreaming-Outbound-1.1.1.1.7000:1] > org.apache.cassandra.streaming.StreamSession:693 - [Stream > #7cb67c00-f3ac-11ea-b940-f7836f164528] Streaming error occurred on session > with peer 1.1.1.1:7000 > org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The channel > this output stream was writing to has been closed >at > org.apache.cassandra.net.AsyncChannelOutputPlus.propagateFailedFlush(AsyncChannelOutputPlus.java:200) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitUntilFlushed(AsyncChannelOutputPlus.java:158) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitForSpace(AsyncChannelOutputPlus.java:140) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.beginFlush(AsyncChannelOutputPlus.java:97) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.lambda$writeToChannel$0(AsyncStreamingOutputPlus.java:142) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.lambda$write$0(CassandraCompressedStreamWriter.java:90) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.writeToChannel(AsyncStreamingOutputPlus.java:138) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.write(CassandraCompressedStreamWriter.java:89) >at > org.apache.cassandra.db.streaming.CassandraOutgoingFile.write(CassandraOutgoingFile.java:180) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage.serialize(OutgoingStreamMessage.java:87) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:45) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:34) >at > org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:40) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:347) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [?:?] >at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > [netty-all-4.1.50.Final.jar:4.1.50.Final] >at java.lang.Thread.run(Thread.java:834) [?:?] >Suppressed: java.nio.channels.ClosedChannelException >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.doFlush(AsyncStreamingOutputPlus.java:78) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.flush(AsyncChannelOutputPlus.java:229) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.close(AsyncChannelOutputPlus.java:248) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:348) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [?:?] >at > io.netty.ut
[jira] [Updated] (CASSANDRA-16143) Streaming fails when s SSTable writer finish() exceeds internode_tcp_user_timeout
[ https://issues.apache.org/jira/browse/CASSANDRA-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Holmberg updated CASSANDRA-16143: -- Reviewers: Adam Holmberg I'm taking a look today as a non-committer reviewer. Meanwhile leaving in "Ready for Review" to invite other reviewers. > Streaming fails when s SSTable writer finish() exceeds > internode_tcp_user_timeout > - > > Key: CASSANDRA-16143 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16143 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Jon Meredith >Assignee: Yifan Cai >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > tl;dr The internode TCP user timeout that provides more responsive detection > of dead nodes for internode message will cause streaming to fail if system > calls to fsync/fdatasync exceed the timeout (default 30s). > To workaround, explicitly set internode_tcp_user_timeout to longer than > fsync/fdatasync, or to zero to revert to the operating system default. > Details: > While bootstrapping a replacement 4.0beta3 node in an existing cluster, > bootstrap streaming repeatedly failed with the streaming follower logging > {code:java} > ERROR 2020-09-10T14:29:34,711 [NettyStreaming-Outbound-1.1.1.1.7000:1] > org.apache.cassandra.streaming.StreamSession:693 - [Stream > #7cb67c00-f3ac-11ea-b940-f7836f164528] Streaming error occurred on session > with peer 1.1.1.1:7000 > org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The channel > this output stream was writing to has been closed >at > org.apache.cassandra.net.AsyncChannelOutputPlus.propagateFailedFlush(AsyncChannelOutputPlus.java:200) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitUntilFlushed(AsyncChannelOutputPlus.java:158) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitForSpace(AsyncChannelOutputPlus.java:140) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.beginFlush(AsyncChannelOutputPlus.java:97) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.lambda$writeToChannel$0(AsyncStreamingOutputPlus.java:142) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.lambda$write$0(CassandraCompressedStreamWriter.java:90) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.writeToChannel(AsyncStreamingOutputPlus.java:138) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.write(CassandraCompressedStreamWriter.java:89) >at > org.apache.cassandra.db.streaming.CassandraOutgoingFile.write(CassandraOutgoingFile.java:180) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage.serialize(OutgoingStreamMessage.java:87) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:45) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:34) >at > org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:40) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:347) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [?:?] >at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > [netty-all-4.1.50.Final.jar:4.1.50.Final] >at java.lang.Thread.run(Thread.java:834) [?:?] >Suppressed: java.nio.channels.ClosedChannelException >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.doFlush(AsyncStreamingOutputPlus.java:78) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.flush(AsyncChannelOutputPlus.java:229) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.close(AsyncChannelOutputPlus.java:248) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:348) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worke
[jira] [Updated] (CASSANDRA-16143) Streaming fails when s SSTable writer finish() exceeds internode_tcp_user_timeout
[ https://issues.apache.org/jira/browse/CASSANDRA-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yifan Cai updated CASSANDRA-16143: -- Test and Documentation Plan: ci, unit test, jvm dtest, adhoc test Status: Patch Available (was: Open) PR: https://github.com/apache/cassandra/pull/824 CI: https://app.circleci.com/pipelines/github/yifan-c/cassandra/164/workflows/a1fee16c-b4cd-4e67-b52f-f2e649ea33da The patch # adds a dedicated TCP user timeout (defaults to 0) for streaming. # collects the processing time for the received incoming stream messages. # logs warning when the processing time exceeds the tcp user timeout. I chose to not add "a separate (unbounded) excecutor service" for FileChannel.force and "periodically call force". In the code base, the incoming stream messages are consumed in a dedicated thread that polls from {{AsyncStreamingInputPlus}}. It does not block the netty event loop. The {{AsyncStreamingInputPlus}} intentionally disables auto read from netty channel to avoid OOM. If we starts an executor service to consume the bytes and FileChannel.force clogs, we end up with keeping more deserialized messages in memory with the risk of OOM. The timeout is actually related to the auto read being disabled. The receiver is blocked at flushing and not able to issue newer channel read. I am also concerned about calling force periodically. According to the java doc of FileChannel.force (quoted below), do so may increase the I/O ops that are not necessary. {quote} Invoking this method may cause an I/O operation to occur even if the channel was only opened for reading. Some operating systems, for example, maintain a last-access time as part of a file's metadata, and this time is updated whenever the file is read. Whether or not this is actually done is system-dependent and is therefore unspecified.{quote} > Streaming fails when s SSTable writer finish() exceeds > internode_tcp_user_timeout > - > > Key: CASSANDRA-16143 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16143 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Jon Meredith >Assignee: Yifan Cai >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > tl;dr The internode TCP user timeout that provides more responsive detection > of dead nodes for internode message will cause streaming to fail if system > calls to fsync/fdatasync exceed the timeout (default 30s). > To workaround, explicitly set internode_tcp_user_timeout to longer than > fsync/fdatasync, or to zero to revert to the operating system default. > Details: > While bootstrapping a replacement 4.0beta3 node in an existing cluster, > bootstrap streaming repeatedly failed with the streaming follower logging > {code:java} > ERROR 2020-09-10T14:29:34,711 [NettyStreaming-Outbound-1.1.1.1.7000:1] > org.apache.cassandra.streaming.StreamSession:693 - [Stream > #7cb67c00-f3ac-11ea-b940-f7836f164528] Streaming error occurred on session > with peer 1.1.1.1:7000 > org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The channel > this output stream was writing to has been closed >at > org.apache.cassandra.net.AsyncChannelOutputPlus.propagateFailedFlush(AsyncChannelOutputPlus.java:200) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitUntilFlushed(AsyncChannelOutputPlus.java:158) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitForSpace(AsyncChannelOutputPlus.java:140) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.beginFlush(AsyncChannelOutputPlus.java:97) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.lambda$writeToChannel$0(AsyncStreamingOutputPlus.java:142) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.lambda$write$0(CassandraCompressedStreamWriter.java:90) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.writeToChannel(AsyncStreamingOutputPlus.java:138) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.write(CassandraCompressedStreamWriter.java:89) >at > org.apache.cassandra.db.streaming.CassandraOutgoingFile.write(CassandraOutgoingFile.java:180) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage.serialize(OutgoingStreamMessage.java:87) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:45) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:34) >at > org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:40) >
[jira] [Updated] (CASSANDRA-16143) Streaming fails when s SSTable writer finish() exceeds internode_tcp_user_timeout
[ https://issues.apache.org/jira/browse/CASSANDRA-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] C. Scott Andreas updated CASSANDRA-16143: - Fix Version/s: (was: 4.0-triage) > Streaming fails when s SSTable writer finish() exceeds > internode_tcp_user_timeout > - > > Key: CASSANDRA-16143 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16143 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Jon Meredith >Assignee: Jon Meredith >Priority: Normal > Fix For: 4.0-beta > > > tl;dr The internode TCP user timeout that provides more responsive detection > of dead nodes for internode message will cause streaming to fail if system > calls to fsync/fdatasync exceed the timeout (default 30s). > To workaround, explicitly set internode_tcp_user_timeout to longer than > fsync/fdatasync, or to zero to revert to the operating system default. > Details: > While bootstrapping a replacement 4.0beta3 node in an existing cluster, > bootstrap streaming repeatedly failed with the streaming follower logging > {code:java} > ERROR 2020-09-10T14:29:34,711 [NettyStreaming-Outbound-1.1.1.1.7000:1] > org.apache.cassandra.streaming.StreamSession:693 - [Stream > #7cb67c00-f3ac-11ea-b940-f7836f164528] Streaming error occurred on session > with peer 1.1.1.1:7000 > org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The channel > this output stream was writing to has been closed >at > org.apache.cassandra.net.AsyncChannelOutputPlus.propagateFailedFlush(AsyncChannelOutputPlus.java:200) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitUntilFlushed(AsyncChannelOutputPlus.java:158) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitForSpace(AsyncChannelOutputPlus.java:140) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.beginFlush(AsyncChannelOutputPlus.java:97) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.lambda$writeToChannel$0(AsyncStreamingOutputPlus.java:142) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.lambda$write$0(CassandraCompressedStreamWriter.java:90) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.writeToChannel(AsyncStreamingOutputPlus.java:138) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.write(CassandraCompressedStreamWriter.java:89) >at > org.apache.cassandra.db.streaming.CassandraOutgoingFile.write(CassandraOutgoingFile.java:180) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage.serialize(OutgoingStreamMessage.java:87) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:45) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:34) >at > org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:40) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:347) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [?:?] >at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > [netty-all-4.1.50.Final.jar:4.1.50.Final] >at java.lang.Thread.run(Thread.java:834) [?:?] >Suppressed: java.nio.channels.ClosedChannelException >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.doFlush(AsyncStreamingOutputPlus.java:78) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.flush(AsyncChannelOutputPlus.java:229) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.close(AsyncChannelOutputPlus.java:248) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:348) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [?:?] >at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) >
[jira] [Updated] (CASSANDRA-16143) Streaming fails when s SSTable writer finish() exceeds internode_tcp_user_timeout
[ https://issues.apache.org/jira/browse/CASSANDRA-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-16143: -- Bug Category: Parent values: Code(13163)Level 1 values: Bug - Unclear Impact(13164) Complexity: Normal Discovered By: Shadow Traffic Fix Version/s: 4.0-beta Severity: Normal Status: Open (was: Triage Needed) > Streaming fails when s SSTable writer finish() exceeds > internode_tcp_user_timeout > - > > Key: CASSANDRA-16143 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16143 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Jon Meredith >Assignee: Jon Meredith >Priority: Normal > Fix For: 4.0-beta > > > tl;dr The internode TCP user timeout that provides more responsive detection > of dead nodes for internode message will cause streaming to fail if system > calls to fsync/fdatasync exceed the timeout (default 30s). > To workaround, explicitly set internode_tcp_user_timeout to longer than > fsync/fdatasync, or to zero to revert to the operating system default. > Details: > While bootstrapping a replacement 4.0beta3 node in an existing cluster, > bootstrap streaming repeatedly failed with the streaming follower logging > {code:java} > ERROR 2020-09-10T14:29:34,711 [NettyStreaming-Outbound-1.1.1.1.7000:1] > org.apache.cassandra.streaming.StreamSession:693 - [Stream > #7cb67c00-f3ac-11ea-b940-f7836f164528] Streaming error occurred on session > with peer 1.1.1.1:7000 > org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The channel > this output stream was writing to has been closed >at > org.apache.cassandra.net.AsyncChannelOutputPlus.propagateFailedFlush(AsyncChannelOutputPlus.java:200) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitUntilFlushed(AsyncChannelOutputPlus.java:158) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.waitForSpace(AsyncChannelOutputPlus.java:140) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.beginFlush(AsyncChannelOutputPlus.java:97) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.lambda$writeToChannel$0(AsyncStreamingOutputPlus.java:142) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.lambda$write$0(CassandraCompressedStreamWriter.java:90) >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.writeToChannel(AsyncStreamingOutputPlus.java:138) >at > org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.write(CassandraCompressedStreamWriter.java:89) >at > org.apache.cassandra.db.streaming.CassandraOutgoingFile.write(CassandraOutgoingFile.java:180) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage.serialize(OutgoingStreamMessage.java:87) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:45) >at > org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:34) >at > org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:40) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:347) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > [?:?] >at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > [netty-all-4.1.50.Final.jar:4.1.50.Final] >at java.lang.Thread.run(Thread.java:834) [?:?] >Suppressed: java.nio.channels.ClosedChannelException >at > org.apache.cassandra.net.AsyncStreamingOutputPlus.doFlush(AsyncStreamingOutputPlus.java:78) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.flush(AsyncChannelOutputPlus.java:229) >at > org.apache.cassandra.net.AsyncChannelOutputPlus.close(AsyncChannelOutputPlus.java:248) >at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:348) >at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] >at java.util.concurrent.FutureTask.run(FutureTask.java:264) > [?:?] >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > [?:?] >at