[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-13 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16149 @srowen @sethah Thanks for all the helpful discussions! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-13 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16149 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-13 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/16149 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16149 **[Test build #3495 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3495/consoleFull)** for PR 16149 at commit

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16149 **[Test build #3495 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3495/consoleFull)** for PR 16149 at commit

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-12 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16149 @srowen @sethah One more commit that adds a test case with `weight = 4.7` which will round up to 5 to test the case @sethah described. All tests passed. I'm pretty sure R's rounding is

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-10 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16149 @sethah Would you please review this? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-08 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16149 @sethah @srowen I updated the documentation. I think we have everything needed for this fix. Please merge and close this PR if there is no other issue. Thanks much for all the comments.

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-07 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16149 @sethah @srowen I have added a comment to the weigthCol doc for the Binomial case. I also updated to test the case `weight < 0.5`, i.e., `round(weight) = 0`. All tests passed. --- If

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-07 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/16149 @srowen We can add a note to the doc for `setWeightCol`. We could also use `logInfo` about weights needing to be integer values for `Binomial` family, but that may not be very effective. --- If

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-07 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16149 @srowen @sethah I have cleaned up the change as suggested. Please review and let me know if there is any question. --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-07 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16149 What you have is fine (though I might name it `ylogy` or something). I don't see other places in the code that compute x ln x or something similar, so it's OK to make this a private function. You

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-07 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16149 @srowen @sethah Thanks for the comments. Yes, the major use case is to be able to handle multiple trials (integer weight, real-valued response). Indeed, a better way to do this is through

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16149 **[Test build #3470 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3470/consoleFull)** for PR 16149 at commit

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16149 **[Test build #3470 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3470/consoleFull)** for PR 16149 at commit

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-05 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16149 Jenkins, add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16149: [SPARK-18715][ML]Fix AIC calculations in Binomial GLM

2016-12-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16149 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this