[jira] [Commented] (CASSANDRA-15900) Close channel and reduce buffer allocation during entire sstable streaming with SSL
[ https://issues.apache.org/jira/browse/CASSANDRA-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152771#comment-17152771 ] ZhaoYang commented on CASSANDRA-15900: -- thanks for the review > Close channel and reduce buffer allocation during entire sstable streaming > with SSL > --- > > Key: CASSANDRA-15900 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15900 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Streaming and Messaging >Reporter: ZhaoYang >Assignee: ZhaoYang >Priority: Normal > Fix For: 4.0-beta > > > CASSANDRA-15740 added the ability to stream entire sstable by loading on-disk > file into user-space off-heap buffer when SSL is enabled, because netty > doesn't support zero-copy with SSL. > But there are two issues: > # file channel is not closed. > # 1mb batch size is used. 1mb exceeds buffer pool's max allocation size, > thus it's all allocated outside the pool and will cause large amount of > allocations. > [Patch|https://github.com/apache/cassandra/pull/651]: > # close file channel when the last batch is loaded into off-heap bytebuffer. > I don't think we need to wait until buffer is flushed by netty. > # reduce the batch to 64kb which is more buffer pool friendly when streaming > entire sstable with SSL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15900) Close channel and reduce buffer allocation during entire sstable streaming with SSL
[ https://issues.apache.org/jira/browse/CASSANDRA-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151008#comment-17151008 ] ZhaoYang commented on CASSANDRA-15900: -- both SimpleReadWriteTest and ImportTest passed locally with JDK11, I don't think they use streaming. > Close channel and reduce buffer allocation during entire sstable streaming > with SSL > --- > > Key: CASSANDRA-15900 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15900 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Streaming and Messaging >Reporter: ZhaoYang >Assignee: ZhaoYang >Priority: Normal > Fix For: 4.0-beta > > > CASSANDRA-15740 added the ability to stream entire sstable by loading on-disk > file into user-space off-heap buffer when SSL is enabled, because netty > doesn't support zero-copy with SSL. > But there are two issues: > # file channel is not closed. > # 1mb batch size is used. 1mb exceeds buffer pool's max allocation size, > thus it's all allocated outside the pool and will cause large amount of > allocations. > [Patch|https://github.com/apache/cassandra/pull/651]: > # close file channel when the last batch is loaded into off-heap bytebuffer. > I don't think we need to wait until buffer is flushed by netty. > # reduce the batch to 64kb which is more buffer pool friendly when streaming > entire sstable with SSL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15900) Close channel and reduce buffer allocation during entire sstable streaming with SSL
[
https://issues.apache.org/jira/browse/CASSANDRA-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149586#comment-17149586
]
Caleb Rackliffe commented on CASSANDRA-15900:
-
I pinged [~bryncooke] about
{{pushed_notifications_test.TestPushedNotifications}}. It's possible
CASSANDRA-15886 actually wasn't a duplicate of CASSANDRA-15677?
[~jasonstack] Unless those last two fail locally or have errors relating to
channel closure, I wouldn't guess they have anything to do with this patch.
However, I'm also not seeing any existing flaky test Jiras for them (closest
are CASSANDRA-15543 and CASSANDRA-15517), so we may need a couple new Jiras :/
> Close channel and reduce buffer allocation during entire sstable streaming
> with SSL
> ---
>
> Key: CASSANDRA-15900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15900
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Streaming and Messaging
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0-beta
>
>
> CASSANDRA-15740 added the ability to stream entire sstable by loading on-disk
> file into user-space off-heap buffer when SSL is enabled, because netty
> doesn't support zero-copy with SSL.
> But there are two issues:
> # file channel is not closed.
> # 1mb batch size is used. 1mb exceeds buffer pool's max allocation size,
> thus it's all allocated outside the pool and will cause large amount of
> allocations.
> [Patch|https://github.com/apache/cassandra/pull/651]:
> # close file channel when the last batch is loaded into off-heap bytebuffer.
> I don't think we need to wait until buffer is flushed by netty.
> # reduce the batch to 64kb which is more buffer pool friendly when streaming
> entire sstable with SSL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15900) Close channel and reduce buffer allocation during entire sstable streaming with SSL
[ https://issues.apache.org/jira/browse/CASSANDRA-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149071#comment-17149071 ] ZhaoYang commented on CASSANDRA-15900: -- rebased and submit another round of ci: [j8|https://circleci.com/workflow-run/cdf55335-c876-450b-8bf9-1d778a2df806] and [j11|https://circleci.com/workflow-run/2080f225-f689-4243-ad67-288bef608640] > Close channel and reduce buffer allocation during entire sstable streaming > with SSL > --- > > Key: CASSANDRA-15900 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15900 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Streaming and Messaging >Reporter: ZhaoYang >Assignee: ZhaoYang >Priority: Normal > Fix For: 4.0-beta > > > CASSANDRA-15740 added the ability to stream entire sstable by loading on-disk > file into user-space off-heap buffer when SSL is enabled, because netty > doesn't support zero-copy with SSL. > But there are two issues: > # file channel is not closed. > # 1mb batch size is used. 1mb exceeds buffer pool's max allocation size, > thus it's all allocated outside the pool and will cause large amount of > allocations. > [Patch|https://github.com/apache/cassandra/pull/651]: > # close file channel when the last batch is loaded into off-heap bytebuffer. > I don't think we need to wait until buffer is flushed by netty. > # reduce the batch to 64kb which is more buffer pool friendly when streaming > entire sstable with SSL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15900) Close channel and reduce buffer allocation during entire sstable streaming with SSL
[
https://issues.apache.org/jira/browse/CASSANDRA-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149066#comment-17149066
]
Caleb Rackliffe commented on CASSANDRA-15900:
-
Let's see...
{{test_restart_node_localhost -
pushed_notifications_test.TestPushedNotifications}} should have been addressed
by CASSANDRA-15677 a few days ago.
{{test_describe - cqlsh_tests.test_cqlsh.TestCqlsh}} and its materialized view
equivalent have a history of flakiness, and don't look directly related to this
patch. (Is there an issue around {{read_repair}} showing up in the table DDL
where it isn't expected?)
> Close channel and reduce buffer allocation during entire sstable streaming
> with SSL
> ---
>
> Key: CASSANDRA-15900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15900
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Streaming and Messaging
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0-beta
>
>
> CASSANDRA-15740 added the ability to stream entire sstable by loading on-disk
> file into user-space off-heap buffer when SSL is enabled, because netty
> doesn't support zero-copy with SSL.
> But there are two issues:
> # file channel is not closed.
> # 1mb batch size is used. 1mb exceeds buffer pool's max allocation size,
> thus it's all allocated outside the pool and will cause large amount of
> allocations.
> [Patch|https://github.com/apache/cassandra/pull/651]:
> # close file channel when the last batch is loaded into off-heap bytebuffer.
> I don't think we need to wait until buffer is flushed by netty.
> # reduce the batch to 64kb which is more buffer pool friendly when streaming
> entire sstable with SSL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15900) Close channel and reduce buffer allocation during entire sstable streaming with SSL
[ https://issues.apache.org/jira/browse/CASSANDRA-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148877#comment-17148877 ] Dinesh Joshi commented on CASSANDRA-15900: -- [~maedhroz] [~jasonstack] looks like there are a few failures. They're likely unrelated but it would be great to double check and make sure. > Close channel and reduce buffer allocation during entire sstable streaming > with SSL > --- > > Key: CASSANDRA-15900 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15900 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Streaming and Messaging >Reporter: ZhaoYang >Assignee: ZhaoYang >Priority: Normal > Fix For: 4.0-beta > > > CASSANDRA-15740 added the ability to stream entire sstable by loading on-disk > file into user-space off-heap buffer when SSL is enabled, because netty > doesn't support zero-copy with SSL. > But there are two issues: > # file channel is not closed. > # 1mb batch size is used. 1mb exceeds buffer pool's max allocation size, > thus it's all allocated outside the pool and will cause large amount of > allocations. > [Patch|https://github.com/apache/cassandra/pull/651]: > # close file channel when the last batch is loaded into off-heap bytebuffer. > I don't think we need to wait until buffer is flushed by netty. > # reduce the batch to 64kb which is more buffer pool friendly when streaming > entire sstable with SSL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15900) Close channel and reduce buffer allocation during entire sstable streaming with SSL
[ https://issues.apache.org/jira/browse/CASSANDRA-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148113#comment-17148113 ] Dinesh Joshi commented on CASSANDRA-15900: -- The patch lgtm too. +1. > Close channel and reduce buffer allocation during entire sstable streaming > with SSL > --- > > Key: CASSANDRA-15900 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15900 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Streaming and Messaging >Reporter: ZhaoYang >Assignee: ZhaoYang >Priority: Normal > Fix For: 4.0-beta > > > CASSANDRA-15740 added the ability to stream entire sstable by loading on-disk > file into user-space off-heap buffer when SSL is enabled, because netty > doesn't support zero-copy with SSL. > But there are two issues: > # file channel is not closed. > # 1mb batch size is used. 1mb exceeds buffer pool's max allocation size, > thus it's all allocated outside the pool and will cause large amount of > allocations. > [Patch|https://github.com/apache/cassandra/pull/651]: > # close file channel when the last batch is loaded into off-heap bytebuffer. > I don't think we need to wait until buffer is flushed by netty. > # reduce the batch to 64kb which is more buffer pool friendly when streaming > entire sstable with SSL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15900) Close channel and reduce buffer allocation during entire sstable streaming with SSL
[ https://issues.apache.org/jira/browse/CASSANDRA-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145740#comment-17145740 ] Caleb Rackliffe commented on CASSANDRA-15900: - +1 I think [~djoshi] might take a quick look as well. > Close channel and reduce buffer allocation during entire sstable streaming > with SSL > --- > > Key: CASSANDRA-15900 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15900 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Streaming and Messaging >Reporter: ZhaoYang >Assignee: ZhaoYang >Priority: Normal > Fix For: 4.0-beta > > > CASSANDRA-15740 added the ability to stream entire sstable by loading on-disk > file into user-space off-heap buffer when SSL is enabled, because netty > doesn't support zero-copy with SSL. > But there are two issues: > # file channel is not closed. > # 1mb batch size is used. 1mb exceeds buffer pool's max allocation size, > thus it's all allocated outside the pool and will cause large amount of > allocations. > [Patch|https://github.com/apache/cassandra/pull/651]: > # close file channel when the last batch is loaded into off-heap bytebuffer. > I don't think we need to wait until buffer is flushed by netty. > # reduce the batch to 64kb which is more buffer pool friendly when streaming > entire sstable with SSL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15900) Close channel and reduce buffer allocation during entire sstable streaming with SSL
[ https://issues.apache.org/jira/browse/CASSANDRA-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145693#comment-17145693 ] ZhaoYang commented on CASSANDRA-15900: -- bq. It might be worthwhile to have a test in AsyncStreamingOutputPlusTest that verifies AsyncStreamingOutputPlus#writeFileToChannel() closes the provided channel. +1 bq. AsyncStreamingOutputPlus#writeFileToChannel(FileChannel, StreamRateLimiter, int) and AsyncStreamingOutputPlus#writeFileToChannelZeroCopy() may be better off at private visibility, given we're treating them as transport-level implementation details. (Perhaps writeFileToChannel would be easier to test at package-private though.) I left them as public and marked "@VisibleForTesting".. bq. The JavaDoc for writeFileToChannel(FileChannel, StreamRateLimiter) is slightly out-of date now, given we've lowered the batch size for the SSL case. (We should make sure to preserve the bit about the method taking ownership of the FileChannel.) +1 > Close channel and reduce buffer allocation during entire sstable streaming > with SSL > --- > > Key: CASSANDRA-15900 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15900 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Streaming and Messaging >Reporter: ZhaoYang >Assignee: ZhaoYang >Priority: Normal > Fix For: 4.0-beta > > > CASSANDRA-15740 added the ability to stream entire sstable by loading on-disk > file into user-space off-heap buffer when SSL is enabled, because netty > doesn't support zero-copy with SSL. > But there are two issues: > # file channel is not closed. > # 1mb batch size is used. 1mb exceeds buffer pool's max allocation size, > thus it's all allocated outside the pool and will cause large amount of > allocations. > [Patch|https://github.com/apache/cassandra/pull/651]: > # close file channel when the last batch is loaded into off-heap bytebuffer. > I don't think we need to wait until buffer is flushed by netty. > # reduce the batch to 64kb which is more buffer pool friendly when streaming > entire sstable with SSL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15900) Close channel and reduce buffer allocation during entire sstable streaming with SSL
[
https://issues.apache.org/jira/browse/CASSANDRA-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144408#comment-17144408
]
Caleb Rackliffe commented on CASSANDRA-15900:
-
[~jasonstack] Made a couple comments inline in the PR where I thought that
would be easier. Then there are a few higher level things:
* It might be worthwhile to have a test in {{AsyncStreamingOutputPlusTest}}
that verifies {{AsyncStreamingOutputPlus#writeFileToChannel()}} closes the
provided channel.
* {{AsyncStreamingOutputPlus#writeFileToChannel(FileChannel, StreamRateLimiter,
int)}} and {{AsyncStreamingOutputPlus#writeFileToChannelZeroCopy()}} may be
better off at `private` visibility, given we're treating them as
transport-level implementation details. (Perhaps {{writeFileToChannel}} would
be easier to test at package-private though.)
* The JavaDoc for {{writeFileToChannel(FileChannel, StreamRateLimiter)}} is
slightly out-of date now, given we've lowered the batch size for the SSL case.
(We should make sure to preserve the bit about the method taking ownership of
the {{FileChannel}}.)
> Close channel and reduce buffer allocation during entire sstable streaming
> with SSL
> ---
>
> Key: CASSANDRA-15900
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15900
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Streaming and Messaging
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0-beta
>
>
> CASSANDRA-15740 added the ability to stream entire sstable by loading on-disk
> file into user-space off-heap buffer when SSL is enabled, because netty
> doesn't support zero-copy with SSL.
> But there are two issues:
> # file channel is not closed.
> # 1mb batch size is used. 1mb exceeds buffer pool's max allocation size,
> thus it's all allocated outside the pool and will cause large amount of
> allocations.
> [Patch|https://github.com/apache/cassandra/pull/651]:
> # close file channel when the last batch is loaded into off-heap bytebuffer.
> I don't think we need to wait until buffer is flushed by netty.
> # reduce the batch to 64kb which is more buffer pool friendly when streaming
> entire sstable with SSL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
