[jira] [Created] (HDFS-16982) Use the right Quantiles Array for Inverse Quantiles snapshot
Ravindra Dingankar created HDFS-16982: - Summary: Use the right Quantiles Array for Inverse Quantiles snapshot Key: HDFS-16982 URL: https://issues.apache.org/jira/browse/HDFS-16982 Project: Hadoop HDFS Issue Type: Bug Components: datanode, metrics Reporter: Ravindra Dingankar Fix For: 3.4.0, 3.3.0 HDFS-16949 introduced InverseQuantiles. However during snapshot for Inverse Quantiles we were still trying to access values from previous snapshot based on the Quantile Array declared in MutableQuantiles. ( Quantile(.50, .050), Quantile(.75, .025), Quantile(.90, .010), Quantile(.95, .005), Quantile(.99, .001) ) For InverseQuantiles we wont have these values ( except for Quantile(.50, .050) ) thus except for 50 Percentile snapshot wont return any value for the remaining quantiles. Fix is to use the correct Quantiles Array to retrieve values during snapshot. The new UTs verify this behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16956) Introduce inverse quantiles for metrics where higher numeric value is better
Ravindra Dingankar created HDFS-16956: - Summary: Introduce inverse quantiles for metrics where higher numeric value is better Key: HDFS-16956 URL: https://issues.apache.org/jira/browse/HDFS-16956 Project: Hadoop HDFS Issue Type: Bug Components: datanode, metrics Affects Versions: 3.3.0, 3.4.0 Reporter: Ravindra Dingankar Currently quantiles are used for latencies, where lower numeric value is better. Hence p90 gives us a value val(p90) such that 90% of our sample set has a value better (lower) than val(p90) However for metrics such as calculating transfer rates (eg : HDFS-16917 ) higher numeric value is better. Thus for such metrics the current quantiles dont work. For these metrics in order for p90 to give a value val(p90) where 90% of the sample set is better (higher) than val(p90) we need to inverse the selection by choosing a value at the (100 - 90)th location instead of the usual 90th position. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16949) Update ReadTransferRate to ReadTransferTimePerByte for effective percentile metrics
Ravindra Dingankar created HDFS-16949: - Summary: Update ReadTransferRate to ReadTransferTimePerByte for effective percentile metrics Key: HDFS-16949 URL: https://issues.apache.org/jira/browse/HDFS-16949 Project: Hadoop HDFS Issue Type: Bug Components: datanode Reporter: Ravindra Dingankar Assignee: Ravindra Dingankar Fix For: 3.4.0, 3.3.0 HDFS-16917 added ReadTransferRate quantiles to calculate the rate which data is read per unit of time. With percentiles the values are sorted in ascending order and hence for the transfer rate p90 gives us the value where 90 percent rates are lower (worse), p99 gives us the value where 99 percent values are lower (worse). Note that value(p90) < p(99) thus p99 is a better transfer rate as compared to p90. However as the percentile increases the value should become worse in order to know how good our system is. Hence instead of calculating the data read transfer rate, we should calculate it's inverse. We will instead calculate the time taken for a byte of data to be read. ( seconds / byte ) After this the p90 value will give us 90 percentage of total values where the time taken is less than value(p90), similarly for p99 and others. Also p(90) < p(99) and here p(99) will become a worse value (taking more time each byte) as compared to p(90) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16917) Add transfer rate quantile metrics for DataNode reads
Ravindra Dingankar created HDFS-16917: - Summary: Add transfer rate quantile metrics for DataNode reads Key: HDFS-16917 URL: https://issues.apache.org/jira/browse/HDFS-16917 Project: Hadoop HDFS Issue Type: Task Components: datanode Reporter: Ravindra Dingankar Currently we have the following metrics for datanode reads. |BytesRead BlocksRead TotalReadTime|Total number of bytes read from DataNode Total number of blocks read from DataNode Total number of milliseconds spent on read operation| We would like to add a new quantile metric calculating the distribution of data transfer rate for datanode reads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
