[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-26 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 Merged into master. If there are comments from others, we can address them in follow-up work. Thanks. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-25 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 This looks good to me. cc @jkbradley @srowen @sethah for another pass if you have time. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16344 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16344 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71823/ Test PASSed. ---

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16344 **[Test build #71823 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71823/testReport)** for PR 16344 at commit

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-22 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @yanboliang Thanks so much for your detailed review. Your suggestions make lots of sense and I have included all of them in the new commit. Let me know if there is any other change needed.

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16344 **[Test build #71823 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71823/testReport)** for PR 16344 at commit

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16344 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16344 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71817/ Test PASSed. ---

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16344 **[Test build #71817 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71817/testReport)** for PR 16344 at commit

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16344 **[Test build #71817 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71817/testReport)** for PR 16344 at commit

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16344 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71686/ Test PASSed. ---

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16344 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16344 **[Test build #71686 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71686/testReport)** for PR 16344 at commit

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-19 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 @actuaryzhang This test failure is caused by Jenkins was not stable, you just need to retest if you encounter similar issue. --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16344 **[Test build #71686 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71686/testReport)** for PR 16344 at commit

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-19 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-19 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 Could anybody help me understand what's causing this test to fail? I see several other ML PR failing as well, with the same error message like below: > Error instrumenting

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16344 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71643/ Test FAILed. ---

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16344 **[Test build #71643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71643/testReport)** for PR 16344 at commit

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-18 Thread MLnick
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/16344 jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-18 Thread MLnick
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/16344 jenkins add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-18 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @srowen @yanboliang @felixcheung @jkbradley Could you help kick off the new test please? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-18 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @srowen @yanboliang @felixcheung Could you help kick off the new test please? Seems to be hanging for a day now. Thanks much. --- If your project is set up for it, you can reply to this

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-17 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @yanboliang Finally, the test is done. Is there anything else needed for this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-17 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16344 **[Test build #3536 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3536/testReport)** for PR 16344 at commit

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-17 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 Still not testing... Been in the status "Asked to test" for a few days now. How can we resolve this? Please help kick off the test. Thanks! @yanboliang @felixcheung @srowen @rxin

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16344 **[Test build #3536 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3536/testReport)** for PR 16344 at commit

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-17 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 I can't figure out why I couldn't add you to whitelist and trigger this job. Ping @rxin @srowen Could you help to add @actuaryzhang to whitelist to trigger this job? Thanks. --- If your

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-16 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16344 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-16 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 It's weird that I can't trigger this job as well. Ping @srowen @jkbradley @MLnick, could you add @actuaryzhang to whitelist to trigger this job. Thanks. --- If your project is set up for it,

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-16 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-15 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-15 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 add to whitelist. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-15 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-15 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-14 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16344 **[Test build #3531 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3531/testReport)** for PR 16344 at commit

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16344 **[Test build #3531 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3531/testReport)** for PR 16344 at commit

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-13 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @yanboliang Thanks for the review and comments. I have made a new commit that addressed all your comments. The main change is the new companion object `FamilyAndLink` and factory methods to

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-13 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-13 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 @actuaryzhang The overall architecture looks good now and I left some comments. I did not review the tests carefully, since I think we should trigger Jenkins to run the tests firstly. Thanks.

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-11 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @yanboliang Thanks. Look forward to your feedback. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-11 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 @actuaryzhang Thanks the update, I will have a look before this weekend. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-10 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 Sorry about closing this prematurely. I'm giving it another shot and I think I have an elegant solution to include `linkPower`. The new commit adds the following: 1. It implements the

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-09 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16344 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-06 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @srowen @yanboliang I'm closing this PR since it does not seem to be very clean to integrate into the current GLM setup. I appreciate all the comments and discussions. --- If your

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-06 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @yanboliang Thanks for the feedback. However, I'm not sure why we need to be consistent with R on this one. The usage of 'tweedie' glm almost always uses `link.power = 0, 1, -1`. There is

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-06 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 @actuaryzhang I totally agree that other link functions are rarely used for ```tweedie``` family, but it should be supported correctly for all possible input. I list the reasons why we should

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-05 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @yanboliang Thanks for the detailed review. I have made all changes you suggested except for the part on the new power link function. Yes, the canonical link in the Tweedie in general is `1.0

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-04 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 @actuaryzhang I just came back from vacation, and will take a look in two days. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2017-01-03 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @yanboliang Did you get a chance to take another look at this? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-28 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 @actuaryzhang I'm on travel these days, and will make another pass in a few days. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-27 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @srowen Made a new commit according to your suggestion. Everything looking good now? @yanboliang --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-26 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @srowen @yanboliang Any additional issues regarding this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-23 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @yanboliang Thanks much for the detailed comments. I have addressed all of them in the new commits. Please take another look. @srowen --- If your project is set up for it, you can

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-22 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @srowen Thanks for the comments. Makes lots of sense to move the switch to subclass. I did not know one could override a `val`. In the new commit, I have moved the `defaultLink` and

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-21 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @srowen @yanboliang Thanks much for the feedback. I now have a better understanding of the code and the issue. I have made new commits reflecting your suggestions. The major changes are

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-20 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 @srowen Thanks for the comments. Really helpful. I have made a new commit that addresses the issues you raised: - I think the use of a global family object does not work well for the

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-19 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-19 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-19 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16344 Jenkins, add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-19 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16344 Jenkins, add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16344: [SPARK-18929][ML] Add Tweedie distribution in GLM

2016-12-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16344 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this