[GitHub] spark pull request #19268: Incorrect Metric reported in MetricsReporter.scal...

[Taaffy]

GitHub user Taaffy opened a pull request:

    https://github.com/apache/spark/pull/19268

    Incorrect Metric reported in MetricsReporter.scala

    Current implementation for processingRate-total uses wrong metric:
    mistakenly uses inputRowsPerSecond instead of processedRowsPerSecond
    
    ## What changes were proposed in this pull request?
    Adjust processingRate-total from using inputRowsPerSecond to 
processedRowsPerSecond
    
    ## How was this patch tested?
    
    Built spark from source with proposed change and tested output with correct 
parameter. Before change the csv metrics file for inputRate-total and 
processingRate-total displayed the same values due to the error. After changing 
MetricsReporter.scala the processingRate-total csv file displayed the correct 
metric. 
    <img width="963" alt="processed rows per second" 
src="https://user-images.githubusercontent.com/32072374/30554340-82eea12c-9ca4-11e7-8370-8168526ff9a2.png";>
    
    Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Taaffy/spark patch-1

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19268.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19268
    
----
commit afe083ff45313ed07cb95ded4c089ead7d80ecce
Author: Taaffy <32072374+taa...@users.noreply.github.com>
Date:   2017-09-18T16:56:51Z

    Incorrect Metric reported in MetricsReporter.scala 
    
    Current implementation for processingRate-total uses wrong metric:
    mistakenly uses inputRowsPerSecond instead of processedRowsPerSecond

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org