GitHub user tengpeng opened a pull request:
https://github.com/apache/spark/pull/21125
[Spark-24024] Fix poisson deviance calculations in GLM to handle y = 0
## What changes were proposed in this pull request?
It is reported by Spark users that the deviance calculations does not
handle y = 0. Thus, the correct model summary cannot be obtained. The user has
confirmed the the issue is in
override def deviance(y: Double, mu: Double, weight: Double): Double =
{ 2.0 * weight * (y * math.log(y / mu) - (y - mu)) }
when y = 0.
The user also mentioned there are many other places he believe we should
check the same thing. However, no other changes are needed, including Gamma
distribution.
## How was this patch tested?
Add a comparison with R deviance calculation to the existing unit test,
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tengpeng/spark Spark24024GLM
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21125.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21125
----
commit 3c6a4dab973851e385b6c9a2c77e5684ad6171a4
Author: Teng Peng <josephtengpeng@...>
Date: 2018-04-23T02:31:25Z
fix deviance calculation when y = 0
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
