[jira] [Commented] (CASSANDRA-16143) Streaming fails when s SSTable writer finish() exceeds internode_tcp_user_timeout
[
https://issues.apache.org/jira/browse/CASSANDRA-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245014#comment-17245014
]
Yifan Cai commented on CASSANDRA-16143:
---
Thanks all for the feedback!
New commits were just pushed to address the comments. The patch is ready to be
re-reviewed.
> Streaming fails when s SSTable writer finish() exceeds
> internode_tcp_user_timeout
> -
>
> Key: CASSANDRA-16143
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16143
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Internode
>Reporter: Jon Meredith
>Assignee: Yifan Cai
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> tl;dr The internode TCP user timeout that provides more responsive detection
> of dead nodes for internode message will cause streaming to fail if system
> calls to fsync/fdatasync exceed the timeout (default 30s).
> To workaround, explicitly set internode_tcp_user_timeout to longer than
> fsync/fdatasync, or to zero to revert to the operating system default.
> Details:
> While bootstrapping a replacement 4.0beta3 node in an existing cluster,
> bootstrap streaming repeatedly failed with the streaming follower logging
> {code:java}
> ERROR 2020-09-10T14:29:34,711 [NettyStreaming-Outbound-1.1.1.1.7000:1]
> org.apache.cassandra.streaming.StreamSession:693 - [Stream
> #7cb67c00-f3ac-11ea-b940-f7836f164528] Streaming error occurred on session
> with peer 1.1.1.1:7000
> org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The channel
> this output stream was writing to has been closed
>at
> org.apache.cassandra.net.AsyncChannelOutputPlus.propagateFailedFlush(AsyncChannelOutputPlus.java:200)
>at
> org.apache.cassandra.net.AsyncChannelOutputPlus.waitUntilFlushed(AsyncChannelOutputPlus.java:158)
>at
> org.apache.cassandra.net.AsyncChannelOutputPlus.waitForSpace(AsyncChannelOutputPlus.java:140)
>at
> org.apache.cassandra.net.AsyncChannelOutputPlus.beginFlush(AsyncChannelOutputPlus.java:97)
>at
> org.apache.cassandra.net.AsyncStreamingOutputPlus.lambda$writeToChannel$0(AsyncStreamingOutputPlus.java:142)
>at
> org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.lambda$write$0(CassandraCompressedStreamWriter.java:90)
>at
> org.apache.cassandra.net.AsyncStreamingOutputPlus.writeToChannel(AsyncStreamingOutputPlus.java:138)
>at
> org.apache.cassandra.db.streaming.CassandraCompressedStreamWriter.write(CassandraCompressedStreamWriter.java:89)
>at
> org.apache.cassandra.db.streaming.CassandraOutgoingFile.write(CassandraOutgoingFile.java:180)
>at
> org.apache.cassandra.streaming.messages.OutgoingStreamMessage.serialize(OutgoingStreamMessage.java:87)
>at
> org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:45)
>at
> org.apache.cassandra.streaming.messages.OutgoingStreamMessage$1.serialize(OutgoingStreamMessage.java:34)
>at
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:40)
>at
> org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:347)
>at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
>at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
>at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [?:?]
>at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [?:?]
>at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> [netty-all-4.1.50.Final.jar:4.1.50.Final]
>at java.lang.Thread.run(Thread.java:834) [?:?]
>Suppressed: java.nio.channels.ClosedChannelException
>at
> org.apache.cassandra.net.AsyncStreamingOutputPlus.doFlush(AsyncStreamingOutputPlus.java:78)
>at
> org.apache.cassandra.net.AsyncChannelOutputPlus.flush(AsyncChannelOutputPlus.java:229)
>at
> org.apache.cassandra.net.AsyncChannelOutputPlus.close(AsyncChannelOutputPlus.java:248)
>at
> org.apache.cassandra.streaming.async.NettyStreamingMessageSender$FileStreamTask.run(NettyStreamingMessageSender.java:348)
>at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
>at java.util.concurrent.FutureTask.run(FutureTask.java:264)
> [?:?]
>at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [?:?]
>at
>
[jira] [Created] (CASSANDRA-16314) nodetool cleanup not working
AaronTrazona created CASSANDRA-16314: Summary: nodetool cleanup not working Key: CASSANDRA-16314 URL: https://issues.apache.org/jira/browse/CASSANDRA-16314 Project: Cassandra Issue Type: Bug Reporter: AaronTrazona Attachments: image-2020-12-07-09-23-02-002.png, image-2020-12-07-09-23-33-788.png, image-2020-12-07-09-24-54-453.png, image-2020-12-07-09-26-28-702.png Hi, After setting up the 3 clusters, I want to free up the disk on my first cluster since the previous still there. This is the nodetool status before running the nodetool cleanup !image-2020-12-07-09-23-02-002.png! When I run the nodetool cleanup !image-2020-12-07-09-23-33-788.png! After I run the nodetool cleanup . I check if the node free up the spaces. This is the result !image-2020-12-07-09-24-54-453.png! It's seems that the nodetool cleanup not working well cassandra version and java version !image-2020-12-07-09-26-28-702.png! Thanks. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16079) Improve dtest runtime
[
https://issues.apache.org/jira/browse/CASSANDRA-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244710#comment-17244710
]
Michael Semb Wever edited comment on CASSANDRA-16079 at 12/6/20, 6:19 PM:
--
bq. Did you notice any speed improvements? All runs seem to be more or less the
same when I look.
The objective of this ticket is, from the description: "…optimize dtests in an
attempt to gain back some of that runtime."
If runs looks to have more or less the same performance, given they are
including 13701, then that is a win.
Here's is another run, again showing performance consistent (or possibly
better) than trunk:
https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch-dtest/262/
So… so long as we have made improvements that mostly offset the cost introduced
by 13701 then this ticket I believe should be good to go. That is, this ticket
is a blocker to 13701 getting merged into 4.0, hence this ticket is a blocker
to 4.0. I don't have a problem with identified improvements to further improve
dtest performance being spun out to further tickets (that don't block 4.0).
What are your thoughts [~Bereng], [~aholmber], [~pauloricardomg]?
For me the only remaining question here is if there are other dtests that
should go through the runtime token allocation strategy rather than using the
pre-generated tokens, that is using the cluster env variable:
{{'CASSANDRA_TOKEN_PREGENERATION_DISABLED'}}.
was (Author: michaelsembwever):
bq. Did you notice any speed improvements? All runs seem to be more or less the
same when I look.
The objective of this ticket is, from the description: "…optimize dtests in an
attempt to gain back some of that runtime."
If runs looks to have more or less the same performance, given they are
including 13701, then that is a win.
So… so long as we have made improvements that mostly offset the cost introduced
by 13701 then this ticket I believe should be good to go. That is, this ticket
is a blocker to 13701 getting merged into 4.0, hence this ticket is a blocker
to 4.0. I don't have a problem with identified improvements to further improve
dtest performance being spun out to further tickets (that don't block 4.0).
What are your thoughts [~Bereng], [~aholmber], [~pauloricardomg]?
For me the only remaining question here is if there are other dtests that
should go through the runtime token allocation strategy rather than using the
pre-generated tokens, that is using the cluster env variable:
{{'CASSANDRA_TOKEN_PREGENERATION_DISABLED'}}.
> Improve dtest runtime
> -
>
> Key: CASSANDRA-16079
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16079
> Project: Cassandra
> Issue Type: Improvement
> Components: CI
>Reporter: Adam Holmberg
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Screenshot 2020-09-19 at 12.32.21.png
>
>
> A recent ticket, CASSANDRA-13701, changed the way dtests run, resulting in a
> [30% increase in run
> time|https://www.mail-archive.com/dev@cassandra.apache.org/msg15606.html].
> While that change was accepted, we wanted to spin out a ticket to optimize
> dtests in an attempt to gain back some of that runtime.
> At this time we don't have concrete improvements in mind, so the first order
> of this ticket will be to analyze the state of things currently, and try to
> ascertain some valuable optimizations. Once the problems are understood, we
> will break down subtasks to divide the work.
> Some areas to consider:
> * cluster reuse
> * C* startup optimizations
> * Tests that should be ported to in-JVM dtest or even unit tests
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16079) Improve dtest runtime
[
https://issues.apache.org/jira/browse/CASSANDRA-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244710#comment-17244710
]
Michael Semb Wever edited comment on CASSANDRA-16079 at 12/6/20, 10:45 AM:
---
bq. Did you notice any speed improvements? All runs seem to be more or less the
same when I look.
The objective of this ticket is, from the description: "…optimize dtests in an
attempt to gain back some of that runtime."
If runs looks to have more or less the same performance, given they are
including 13701, then that is a win.
So… so long as we have made improvements that mostly offset the cost introduced
by 13701 that this ticket I believe should be good to go. That is, this ticket
is a blocker to 13701 getting merged into 4.0, and so is a blocker to 4.0. I
don't have a problem with identified improvements to further improve dtest
performance being spun out to further tickets (that don't block 4.0).
What are your thoughts [~Bereng], [~aholmber], [~pauloricardomg]?
For me the only remaining question here is if there are other dtests that
should go through the runtime token allocation strategy rather than using the
pre-generated tokens, that is using the cluster env variable:
{{'CASSANDRA_TOKEN_PREGENERATION_DISABLED'}}.
was (Author: michaelsembwever):
bq. Did you notice any speed improvements? All runs seem to be more or less the
same when I look.
The objective of this ticket is, from the description: "…optimize dtests in an
attempt to gain back some of that runtime."
If runs looks to have more or less the same performance, if they are including
13701, then that is a win.
So… so long as we have made improvements that mostly offset the cost introduced
by 13701 that this ticket I believe should be good to go. That is, this ticket
is a blocker to 13701 getting merged into 4.0, and so is a blocker to 4.0. I
don't have a problem with identified improvements to further improve dtest
performance being spun out to further tickets (that don't block 4.0).
What are your thoughts [~Bereng], [~aholmber], [~pauloricardomg]?
For me the only remaining question here is if there are other dtests that
should go through the runtime token allocation strategy rather than using the
pre-generated tokens, that is using the cluster env variable:
{{'CASSANDRA_TOKEN_PREGENERATION_DISABLED'}}.
> Improve dtest runtime
> -
>
> Key: CASSANDRA-16079
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16079
> Project: Cassandra
> Issue Type: Improvement
> Components: CI
>Reporter: Adam Holmberg
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Screenshot 2020-09-19 at 12.32.21.png
>
>
> A recent ticket, CASSANDRA-13701, changed the way dtests run, resulting in a
> [30% increase in run
> time|https://www.mail-archive.com/dev@cassandra.apache.org/msg15606.html].
> While that change was accepted, we wanted to spin out a ticket to optimize
> dtests in an attempt to gain back some of that runtime.
> At this time we don't have concrete improvements in mind, so the first order
> of this ticket will be to analyze the state of things currently, and try to
> ascertain some valuable optimizations. Once the problems are understood, we
> will break down subtasks to divide the work.
> Some areas to consider:
> * cluster reuse
> * C* startup optimizations
> * Tests that should be ported to in-JVM dtest or even unit tests
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16079) Improve dtest runtime
[
https://issues.apache.org/jira/browse/CASSANDRA-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244710#comment-17244710
]
Michael Semb Wever edited comment on CASSANDRA-16079 at 12/6/20, 10:45 AM:
---
bq. Did you notice any speed improvements? All runs seem to be more or less the
same when I look.
The objective of this ticket is, from the description: "…optimize dtests in an
attempt to gain back some of that runtime."
If runs looks to have more or less the same performance, given they are
including 13701, then that is a win.
So… so long as we have made improvements that mostly offset the cost introduced
by 13701 then this ticket I believe should be good to go. That is, this ticket
is a blocker to 13701 getting merged into 4.0, and so is a blocker to 4.0. I
don't have a problem with identified improvements to further improve dtest
performance being spun out to further tickets (that don't block 4.0).
What are your thoughts [~Bereng], [~aholmber], [~pauloricardomg]?
For me the only remaining question here is if there are other dtests that
should go through the runtime token allocation strategy rather than using the
pre-generated tokens, that is using the cluster env variable:
{{'CASSANDRA_TOKEN_PREGENERATION_DISABLED'}}.
was (Author: michaelsembwever):
bq. Did you notice any speed improvements? All runs seem to be more or less the
same when I look.
The objective of this ticket is, from the description: "…optimize dtests in an
attempt to gain back some of that runtime."
If runs looks to have more or less the same performance, given they are
including 13701, then that is a win.
So… so long as we have made improvements that mostly offset the cost introduced
by 13701 that this ticket I believe should be good to go. That is, this ticket
is a blocker to 13701 getting merged into 4.0, and so is a blocker to 4.0. I
don't have a problem with identified improvements to further improve dtest
performance being spun out to further tickets (that don't block 4.0).
What are your thoughts [~Bereng], [~aholmber], [~pauloricardomg]?
For me the only remaining question here is if there are other dtests that
should go through the runtime token allocation strategy rather than using the
pre-generated tokens, that is using the cluster env variable:
{{'CASSANDRA_TOKEN_PREGENERATION_DISABLED'}}.
> Improve dtest runtime
> -
>
> Key: CASSANDRA-16079
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16079
> Project: Cassandra
> Issue Type: Improvement
> Components: CI
>Reporter: Adam Holmberg
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Screenshot 2020-09-19 at 12.32.21.png
>
>
> A recent ticket, CASSANDRA-13701, changed the way dtests run, resulting in a
> [30% increase in run
> time|https://www.mail-archive.com/dev@cassandra.apache.org/msg15606.html].
> While that change was accepted, we wanted to spin out a ticket to optimize
> dtests in an attempt to gain back some of that runtime.
> At this time we don't have concrete improvements in mind, so the first order
> of this ticket will be to analyze the state of things currently, and try to
> ascertain some valuable optimizations. Once the problems are understood, we
> will break down subtasks to divide the work.
> Some areas to consider:
> * cluster reuse
> * C* startup optimizations
> * Tests that should be ported to in-JVM dtest or even unit tests
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16079) Improve dtest runtime
[
https://issues.apache.org/jira/browse/CASSANDRA-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244710#comment-17244710
]
Michael Semb Wever edited comment on CASSANDRA-16079 at 12/6/20, 10:45 AM:
---
bq. Did you notice any speed improvements? All runs seem to be more or less the
same when I look.
The objective of this ticket is, from the description: "…optimize dtests in an
attempt to gain back some of that runtime."
If runs looks to have more or less the same performance, given they are
including 13701, then that is a win.
So… so long as we have made improvements that mostly offset the cost introduced
by 13701 then this ticket I believe should be good to go. That is, this ticket
is a blocker to 13701 getting merged into 4.0, hence this ticket is a blocker
to 4.0. I don't have a problem with identified improvements to further improve
dtest performance being spun out to further tickets (that don't block 4.0).
What are your thoughts [~Bereng], [~aholmber], [~pauloricardomg]?
For me the only remaining question here is if there are other dtests that
should go through the runtime token allocation strategy rather than using the
pre-generated tokens, that is using the cluster env variable:
{{'CASSANDRA_TOKEN_PREGENERATION_DISABLED'}}.
was (Author: michaelsembwever):
bq. Did you notice any speed improvements? All runs seem to be more or less the
same when I look.
The objective of this ticket is, from the description: "…optimize dtests in an
attempt to gain back some of that runtime."
If runs looks to have more or less the same performance, given they are
including 13701, then that is a win.
So… so long as we have made improvements that mostly offset the cost introduced
by 13701 then this ticket I believe should be good to go. That is, this ticket
is a blocker to 13701 getting merged into 4.0, and so is a blocker to 4.0. I
don't have a problem with identified improvements to further improve dtest
performance being spun out to further tickets (that don't block 4.0).
What are your thoughts [~Bereng], [~aholmber], [~pauloricardomg]?
For me the only remaining question here is if there are other dtests that
should go through the runtime token allocation strategy rather than using the
pre-generated tokens, that is using the cluster env variable:
{{'CASSANDRA_TOKEN_PREGENERATION_DISABLED'}}.
> Improve dtest runtime
> -
>
> Key: CASSANDRA-16079
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16079
> Project: Cassandra
> Issue Type: Improvement
> Components: CI
>Reporter: Adam Holmberg
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Screenshot 2020-09-19 at 12.32.21.png
>
>
> A recent ticket, CASSANDRA-13701, changed the way dtests run, resulting in a
> [30% increase in run
> time|https://www.mail-archive.com/dev@cassandra.apache.org/msg15606.html].
> While that change was accepted, we wanted to spin out a ticket to optimize
> dtests in an attempt to gain back some of that runtime.
> At this time we don't have concrete improvements in mind, so the first order
> of this ticket will be to analyze the state of things currently, and try to
> ascertain some valuable optimizations. Once the problems are understood, we
> will break down subtasks to divide the work.
> Some areas to consider:
> * cluster reuse
> * C* startup optimizations
> * Tests that should be ported to in-JVM dtest or even unit tests
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16079) Improve dtest runtime
[
https://issues.apache.org/jira/browse/CASSANDRA-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244710#comment-17244710
]
Michael Semb Wever commented on CASSANDRA-16079:
bq. Did you notice any speed improvements? All runs seem to be more or less the
same when I look.
The objective of this ticket is, from the description: "…optimize dtests in an
attempt to gain back some of that runtime."
If runs looks to have more or less the same performance, if they are including
13701, then that is a win.
So… so long as we have made improvements that mostly offset the cost introduced
by 13701 that this ticket I believe should be good to go. That is, this ticket
is a blocker to 13701 getting merged into 4.0, and so is a blocker to 4.0. I
don't have a problem with identified improvements to further improve dtest
performance being spun out to further tickets (that don't block 4.0).
What are your thoughts [~Bereng], [~aholmber], [~pauloricardomg]?
For me the only remaining question here is if there are other dtests that
should go through the runtime token allocation strategy rather than using the
pre-generated tokens, that is using the cluster env variable:
{{'CASSANDRA_TOKEN_PREGENERATION_DISABLED'}}.
> Improve dtest runtime
> -
>
> Key: CASSANDRA-16079
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16079
> Project: Cassandra
> Issue Type: Improvement
> Components: CI
>Reporter: Adam Holmberg
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Screenshot 2020-09-19 at 12.32.21.png
>
>
> A recent ticket, CASSANDRA-13701, changed the way dtests run, resulting in a
> [30% increase in run
> time|https://www.mail-archive.com/dev@cassandra.apache.org/msg15606.html].
> While that change was accepted, we wanted to spin out a ticket to optimize
> dtests in an attempt to gain back some of that runtime.
> At this time we don't have concrete improvements in mind, so the first order
> of this ticket will be to analyze the state of things currently, and try to
> ascertain some valuable optimizations. Once the problems are understood, we
> will break down subtasks to divide the work.
> Some areas to consider:
> * cluster reuse
> * C* startup optimizations
> * Tests that should be ported to in-JVM dtest or even unit tests
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
