[ https://issues.apache.org/jira/browse/CASSANDRA-16325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17656750#comment-17656750 ]
Josh McKenzie commented on CASSANDRA-16325: ------------------------------------------- I'm a little concerned about us hammering those Counters every time we transfer any date: {code:java} // stream each of the required sections of the file for (SSTableReader.PartitionPositionBounds section : sections) { long start = validator == null ? section.lowerPosition : validator.chunkStart(section.lowerPosition); // if the transfer does not start on the valididator's chunk boundary, this is the number of bytes to offset by int transferOffset = (int) (section.lowerPosition - start); if (validator != null) validator.seek(start); // length of the section to read long length = section.upperPosition - start; // tracks write progress long bytesRead = 0; while (bytesRead < length) { int toTransfer = (int) Math.min(bufferSize, length - bytesRead); long lastBytesRead = write(proxy, validator, out, start, transferOffset, toTransfer, bufferSize); start += lastBytesRead; bytesRead += lastBytesRead; progress += (lastBytesRead - transferOffset); session.progress(sstable.descriptor.filenameFor(Component.DATA), ProgressInfo.Direction.OUT, progress, totalSize); fileStreamMetricsListener.onStreamingBytesTransferred(progress); // REVIEWER NOTE: This is where we're going to update the counters on each call transferOffset = 0; }{code} ISTM we might benefit from a simple threshold counter inside {{CassandraStreamWriter}} where it'll update the metrics every N multiples of {{CassandraStreamWriter.DEFAULT_CHUNK_SIZE}} so we don't spam the metrics updates. > Update streaming metrics incrementally > -------------------------------------- > > Key: CASSANDRA-16325 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16325 > Project: Cassandra > Issue Type: Improvement > Components: Observability/Metrics > Reporter: Paulo Motta > Assignee: Isaac Reath > Priority: Normal > Labels: lhf > Fix For: 4.2 > > Time Spent: 10h 10m > Remaining Estimate: 0h > > Currently the inbound and outbound streamed bytes metrics are incremented > after each file is streamed, what doesn't represent the current number of > bytes streamed since it can take a long time for a large file to be streamed. > We should update the metric incrementally as data is streamed. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org