I downloaded JMeter 5.5 and compared aggregate and HTML reports. Now the results are exactly the same
Thank you very much Felix for fixing it on such short notice. :) [image: image.png] [image: image.png] . On Thu, Jun 3, 2021 at 11:14 PM Felix Schumacher < [email protected]> wrote: > You might find https://bz.apache.org/bugzilla/show_bug.cgi?id=65353 > interesting. > > With the next nightly build or build from trunk, you should be able to use > the new property "backend_metrics_percentile_estimator" with a value of "R_3" > to lessen the difference between both reports. But keep in mind, that the > HTML Report use a sliding window, while the Aggregate Report does not (and > will eventually OOM). > > Felix > > > Am 03.06.21 um 17:46 schrieb Deepak Chaudhari: > > I need to send both the reports to the client and surely the question will > come "Why is there so much difference?" > If there is any way with which we can generate almost similar aggregate > and HTML reports? > > On Thu, Jun 3, 2021 at 9:00 PM Felix Schumacher < > [email protected]> wrote: > >> Without the actual data it is impossible to say, if the reports are >> wrong, or if they follow different ways to calculate a percentile. >> >> The problem here is, that the two reports are using different algorithms >> to calculate the percentiles. We are working with discrete numbers and here >> a percentile is more like a range than a single (correct or wrong) number. >> Say, you have the values [1, 2, 4, 8, 16, 32, 64, 128, 256, 512] want to >> calculate a 90% percentile. For this you take any number that splits the >> (sorted) list into two lists, where one list has all elements, that are >> smaller (or equal) to the number and one list contains all elements that >> are bigger than the number. For this example, any number between 256 and >> 511.99 would be a valid 90% percentile. (that is of course a broad >> description only) >> >> The Aggregate report currently uses a memory intensive way and stores a >> list with all values and picks the smallest number, that fits the above >> description. (This memory hogging behaviour should be fixed and would lead >> to less accurate data and less OOMs or slow JMeter instances) >> >> The HTML report gives you a value that would probably be a good 90% >> percentile, if you would have more data like the one, you already have by >> picking some value between the lower bound and the upper bound of the >> described range (in our example something between 256 and 511.999). >> >> As your report has very few samples (21 for the biggest difference in the >> numbers), the skew between the reported percentiles can look big, but are >> not necessarily wrong. >> >> Note, that if the numbers that both reports tell you are far apart, that >> is probably a sign of having either too few samples, or the samples >> (durations of the samples) are distributed sparsely near the percentile >> ranges. >> >> We have discussed in the past to use the same algorithm for both >> components, so that you get consistent values, but there were always other >> issues, that seemed to be more important. >> >> Felix >> Am 02.06.21 um 15:28 schrieb Deepak Chaudhari: >> >> Hi, >> >> I collected JTL file during a test run and using that I'm generating HTML >> report and aggregate report. When I compare aggregate and HTML reports I >> found that 90th and 95th percentile values are wrong in HTML report. >> I tried to change "jmeter.reportgenerator.statistic_window" value in >> user.properties but still getting wrong values. >> >> *Aggregate report:* >> [image: image.png] >> >> *HTML report:* >> [image: image.png] >> >>
