Github user iamaleksey commented on a diff in the pull request:
https://github.com/apache/cassandra/pull/239#discussion_r204531887
--- Diff:
src/java/org/apache/cassandra/db/streaming/CassandraOutgoingFile.java ---
@@ -114,13 +155,51 @@ public void write(StreamSession session,
DataOutputStreamPlus out, int version)
CassandraStreamHeader.serializer.serialize(header, out, version);
out.flush();
- CassandraStreamWriter writer = header.compressionInfo == null ?
- new CassandraStreamWriter(sstable,
header.sections, session) :
- new
CompressedCassandraStreamWriter(sstable, header.sections,
-
header.compressionInfo, session);
+ IStreamWriter writer;
+ if (shouldStreamFullSSTable())
+ {
+ writer = new CassandraBlockStreamWriter(sstable, session,
components);
+ }
+ else
+ {
+ writer = (header.compressionInfo == null) ?
+ new CassandraStreamWriter(sstable, header.sections,
session) :
+ new CompressedCassandraStreamWriter(sstable,
header.sections,
+
header.compressionInfo, session);
+ }
writer.write(out);
}
+ @VisibleForTesting
+ public boolean shouldStreamFullSSTable()
+ {
+ return isFullSSTableTransfersEnabled && isFullyContained;
+ }
+
+ @VisibleForTesting
+ public boolean fullyContainedIn(List<Range<Token>> normalizedRanges,
SSTableReader sstable)
+ {
+ if (normalizedRanges == null)
+ return false;
+
+ RangeOwnHelper rangeOwnHelper = new
RangeOwnHelper(normalizedRanges);
+ try (KeyIterator iter = new KeyIterator(sstable.descriptor,
sstable.metadata()))
+ {
+ while (iter.hasNext())
+ {
+ DecoratedKey key = iter.next();
+ try
+ {
+ rangeOwnHelper.check(key);
+ } catch(RuntimeException e)
--- End diff --
On a more general note, this is potentially quite an expensive thing to do,
especially for big sstables with skinny partitions, and in some cases this will
introduce a performance regression.
The whole optimisation is realistically only useful for bootrstrap, decom,
and rebuild, with LCS (which is still plenty useful and impactful and worth
having). But it wouldn't normally kick in for regular repairs because of the
full-cover requirement, and it won't normally kick in for STCS until
CASSANDRA-10540 (range aware compaction) is implemented. In those cases having
to read through the whole primary index is a perf regression that we shouldn't
allow to happen.
The easiest way to avoid it would be to store sstable's effective token
ranges in sstable metadata in relation to the node's ranges, making this check
essentially free. Otherwise we should probably disable complete sstable
streaming for STCS tables, at least until CASSANDRA-10540 is implemented. That
however wouldn't address the regression to regular streaming, so keeping ranges
in the metadata would be my preferred way to go.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
