[jira] [Commented] (SPARK-18799) Spark SQL expose interface for plug-gable parser extension

[ https://issues.apache.org/jira/browse/SPARK-18799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736118#comment-15736118 ] Jihong MA commented on SPARK-18799: --- are we looking at first quarter of 2017 for Spark 2.2? is now too

[jira] [Commented] (SPARK-18799) Spark SQL expose interface for plug-gable parser extension

[ https://issues.apache.org/jira/browse/SPARK-18799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736100#comment-15736100 ] Jihong MA commented on SPARK-18799: --- DML statement support for instance > Spark SQL expose interface

[jira] [Commented] (SPARK-18799) Spark SQL expose interface for plug-gable parser extension

[ https://issues.apache.org/jira/browse/SPARK-18799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736003#comment-15736003 ] Jihong MA commented on SPARK-18799: --- [~hyukjin.kwon] the intention to remove it at that time is

[jira] [Created] (SPARK-18799) Spark SQL expose interface for plug-gable parser extension

Jihong MA created SPARK-18799: - Summary: Spark SQL expose interface for plug-gable parser extension Key: SPARK-18799 URL: https://issues.apache.org/jira/browse/SPARK-18799 Project: Spark Issue

[jira] [Updated] (SPARK-11720) Handle edge cases when count = 0 or 1 for Stats function

[ https://issues.apache.org/jira/browse/SPARK-11720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-11720: -- Summary: Handle edge cases when count = 0 or 1 for Stats function (was: Return Double.NaN instead of

[jira] [Updated] (SPARK-11720) Handle edge cases when count = 0 or 1 for Stats function

[ https://issues.apache.org/jira/browse/SPARK-11720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-11720: -- Description: update the behavior of stats function when count =0 or 1 to make it in consistent across

[jira] [Created] (SPARK-11720) Return Double.NaN instead of null for Mean and Average when count = 0

Jihong MA created SPARK-11720: - Summary: Return Double.NaN instead of null for Mean and Average when count = 0 Key: SPARK-11720 URL: https://issues.apache.org/jira/browse/SPARK-11720 Project: Spark

[jira] [Updated] (SPARK-11720) Return Double.NaN instead of null for Mean and Average when count = 0

[ https://issues.apache.org/jira/browse/SPARK-11720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-11720: -- Issue Type: Sub-task (was: Improvement) Parent: SPARK-10384 > Return Double.NaN instead of

[jira] [Commented] (SPARK-11720) Return Double.NaN instead of null for Mean and Average when count = 0

[ https://issues.apache.org/jira/browse/SPARK-11720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003558#comment-15003558 ] Jihong MA commented on SPARK-11720: --- [~mengxr] the implementation of average is not a numerical stable

[jira] [Commented] (SPARK-11720) Return Double.NaN instead of null for Mean and Average when count = 0

[ https://issues.apache.org/jira/browse/SPARK-11720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003589#comment-15003589 ] Jihong MA commented on SPARK-11720: --- Also, for mean, we treat Decimal differently vs. other numeric

[jira] [Created] (SPARK-11420) Changing Stddev support with Imperative Aggregate

Jihong MA created SPARK-11420: - Summary: Changing Stddev support with Imperative Aggregate Key: SPARK-11420 URL: https://issues.apache.org/jira/browse/SPARK-11420 Project: Spark Issue Type:

[jira] [Updated] (SPARK-11420) Changing Stddev support with Imperative Aggregate

[ https://issues.apache.org/jira/browse/SPARK-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-11420: -- Issue Type: Sub-task (was: Improvement) Parent: SPARK-10384 > Changing Stddev support with

[jira] [Updated] (SPARK-11420) Updating Stddev support with Imperative Aggregate

[ https://issues.apache.org/jira/browse/SPARK-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-11420: -- Summary: Updating Stddev support with Imperative Aggregate (was: Changing Stddev support with

[jira] [Commented] (SPARK-10646) Bivariate Statistics: Pearson's Chi-Squared goodness of fit test

[ https://issues.apache.org/jira/browse/SPARK-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965298#comment-14965298 ] Jihong MA commented on SPARK-10646: --- [~mengxr] to add chi-squared test support through UDAF framework,

[jira] [Commented] (SPARK-9297) covar_pop and covar_samp aggregate functions

[ https://issues.apache.org/jira/browse/SPARK-9297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965662#comment-14965662 ] Jihong MA commented on SPARK-9297: -- [~viirya] I just noticed your initial PR for Pearson's correlation is

[jira] [Commented] (SPARK-9297) covar_pop and covar_samp aggregate functions

[ https://issues.apache.org/jira/browse/SPARK-9297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964370#comment-14964370 ] Jihong MA commented on SPARK-9297: -- I will work on this and prepare a PR using ImperativeAggregate

[jira] [Commented] (SPARK-10953) Benchmark codegen vs. hand-written code for univariate statistics

[ https://issues.apache.org/jira/browse/SPARK-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958386#comment-14958386 ] Jihong MA commented on SPARK-10953: --- [~mengxr] merged PR9038. below are the avg elapsed time

[jira] [Comment Edited] (SPARK-10953) Benchmark codegen vs. hand-written code for univariate statistics

[ https://issues.apache.org/jira/browse/SPARK-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954447#comment-14954447 ] Jihong MA edited comment on SPARK-10953 at 10/13/15 6:05 AM: - [~mengxr]had a

[jira] [Comment Edited] (SPARK-10953) Benchmark codegen vs. hand-written code for univariate statistics

[ https://issues.apache.org/jira/browse/SPARK-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954447#comment-14954447 ] Jihong MA edited comment on SPARK-10953 at 10/13/15 6:05 AM: - [~mengxr] I had

[jira] [Comment Edited] (SPARK-10953) Benchmark codegen vs. hand-written code for univariate statistics

[ https://issues.apache.org/jira/browse/SPARK-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954447#comment-14954447 ] Jihong MA edited comment on SPARK-10953 at 10/13/15 5:59 AM: - had a quick run

[jira] [Commented] (SPARK-10953) Benchmark codegen vs. hand-written code for univariate statistics

[ https://issues.apache.org/jira/browse/SPARK-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954447#comment-14954447 ] Jihong MA commented on SPARK-10953: --- had a quick run on my laptop, with a stddev implementation based

[jira] [Commented] (SPARK-10953) Benchmark codegen vs. hand-written code for univariate statistics

[ https://issues.apache.org/jira/browse/SPARK-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954281#comment-14954281 ] Jihong MA commented on SPARK-10953: --- [~mengxr]as Yin indicated in the comment, we would like to merge

[jira] [Updated] (SPARK-10641) skewness and kurtosis support

[ https://issues.apache.org/jira/browse/SPARK-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10641: -- Issue Type: Sub-task (was: New Feature) Parent: SPARK-10384 > skewness and kurtosis support >

[jira] [Commented] (SPARK-10953) Benchmark codegen vs. hand-written code for univariate statistics

[ https://issues.apache.org/jira/browse/SPARK-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949797#comment-14949797 ] Jihong MA commented on SPARK-10953: --- we should have a cluster for testing next Monday. we will run the

[jira] [Commented] (SPARK-10953) Benchmark codegen vs. hand-written code for univariate statistics

[ https://issues.apache.org/jira/browse/SPARK-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945532#comment-14945532 ] Jihong MA commented on SPARK-10953: --- [~mengxr] do you mean comparing an implementation which operate

[jira] [Commented] (SPARK-10953) Benchmark codegen vs. hand-written code for univariate statistics

[ https://issues.apache.org/jira/browse/SPARK-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946061#comment-14946061 ] Jihong MA commented on SPARK-10953: --- it should not be too hard to put together an implementation based

[jira] [Updated] (SPARK-10860) Bivariate Statistics: Chi-Squared independence test

[ https://issues.apache.org/jira/browse/SPARK-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10860: -- Description: Pearson's chi-squared independence test > Bivariate Statistics: Chi-Squared independence

[jira] [Created] (SPARK-10862) Univariate Statistics: Adding median support as UDAF

Jihong MA created SPARK-10862: - Summary: Univariate Statistics: Adding median support as UDAF Key: SPARK-10862 URL: https://issues.apache.org/jira/browse/SPARK-10862 Project: Spark Issue Type:

[jira] [Updated] (SPARK-10646) Bivariate Statistics: Pearson's Chi-Squared goodness of fit test

[ https://issues.apache.org/jira/browse/SPARK-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10646: -- Description: Pearson's chi-squared goodness of fit test for observed against the expected

[jira] [Commented] (SPARK-10861) Univariate Statistics: Adding range support as UDAF

[ https://issues.apache.org/jira/browse/SPARK-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14934109#comment-14934109 ] Jihong MA commented on SPARK-10861: --- I will send a PR soon. > Univariate Statistics: Adding range

[jira] [Updated] (SPARK-10861) Univariate Statistics: Adding range support for continuous

[ https://issues.apache.org/jira/browse/SPARK-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10861: -- Description: Range support as UDAF > Univariate Statistics: Adding range support for continuous >

[jira] [Updated] (SPARK-10861) Univariate Statistics: Adding range support as UDAF

[ https://issues.apache.org/jira/browse/SPARK-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10861: -- Description: Range support for continuous (was: Range support as UDAF ) > Univariate Statistics:

[jira] [Updated] (SPARK-10861) Univariate Statistics: Adding range support as UDAF

[ https://issues.apache.org/jira/browse/SPARK-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10861: -- Summary: Univariate Statistics: Adding range support as UDAF (was: Univariate Statistics: Adding

[jira] [Updated] (SPARK-10862) Univariate Statistics: Adding median & quantile support as UDAF

[ https://issues.apache.org/jira/browse/SPARK-10862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10862: -- Summary: Univariate Statistics: Adding median & quantile support as UDAF (was: Univariate Statistics:

[jira] [Updated] (SPARK-10645) Bivariate Statistics: Spearman's Correlation support as UDAF

[ https://issues.apache.org/jira/browse/SPARK-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10645: -- Description: Spearman's rank correlation coefficient :

[jira] [Created] (SPARK-10860) Bivariate Statistics: Chi-Squared independence test

Jihong MA created SPARK-10860: - Summary: Bivariate Statistics: Chi-Squared independence test Key: SPARK-10860 URL: https://issues.apache.org/jira/browse/SPARK-10860 Project: Spark Issue Type:

[jira] [Commented] (SPARK-10860) Bivariate Statistics: Chi-Squared independence test

[ https://issues.apache.org/jira/browse/SPARK-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14934098#comment-14934098 ] Jihong MA commented on SPARK-10860: --- [~josephkb] please assign this JIRA to me. Thanks! > Bivariate

[jira] [Updated] (SPARK-10860) Bivariate Statistics: Chi-Squared independence test

[ https://issues.apache.org/jira/browse/SPARK-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10860: -- Issue Type: Sub-task (was: New Feature) Parent: SPARK-10385 > Bivariate Statistics:

[jira] [Created] (SPARK-10861) Univariate Statistics: Adding range support for continuous

Jihong MA created SPARK-10861: - Summary: Univariate Statistics: Adding range support for continuous Key: SPARK-10861 URL: https://issues.apache.org/jira/browse/SPARK-10861 Project: Spark Issue

[jira] [Updated] (SPARK-10861) Univariate Statistics: Adding range support for continuous

[ https://issues.apache.org/jira/browse/SPARK-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10861: -- Issue Type: Sub-task (was: New Feature) Parent: SPARK-10384 > Univariate Statistics: Adding

[jira] [Updated] (SPARK-10862) Univariate Statistics: Adding median & quantile support as UDAF

[ https://issues.apache.org/jira/browse/SPARK-10862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10862: -- Issue Type: Sub-task (was: New Feature) Parent: SPARK-10384 > Univariate Statistics: Adding

[jira] [Updated] (SPARK-10646) Bivariate Statistics: Pearson's Chi-Squared Test for categorical vs. categorical

[ https://issues.apache.org/jira/browse/SPARK-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10646: -- Description: Pearson's chi-squared goodness of fit test for observed against the expected distribution

[jira] [Commented] (SPARK-10646) Bivariate Statistics: Pearson's Chi-Squared Test for categorical vs. categorical

[ https://issues.apache.org/jira/browse/SPARK-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803378#comment-14803378 ] Jihong MA commented on SPARK-10646: --- [~josephkb] please assign this JIRA to me, I will start working

[jira] [Created] (SPARK-10645) Bivariate Statistics for continuous vs. continuous

Jihong MA created SPARK-10645: - Summary: Bivariate Statistics for continuous vs. continuous Key: SPARK-10645 URL: https://issues.apache.org/jira/browse/SPARK-10645 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-10602) Univariate statistics as UDAFs: single-pass continuous stats

[ https://issues.apache.org/jira/browse/SPARK-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14790816#comment-14790816 ] Jihong MA commented on SPARK-10602: --- I go ahead/ created SPARK-10641, since this JIRA is not listed as

[jira] [Updated] (SPARK-10641) skewness and kurtosis support

[ https://issues.apache.org/jira/browse/SPARK-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10641: -- Issue Type: Sub-task (was: New Feature) Parent: SPARK-10384 > skewness and kurtosis support >

[jira] [Created] (SPARK-10641) skewness and kurtosis support

Jihong MA created SPARK-10641: - Summary: skewness and kurtosis support Key: SPARK-10641 URL: https://issues.apache.org/jira/browse/SPARK-10641 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-10646) Bivariate Statistics: Pearson's Chi-Squared Test for categorical vs. categorical

[ https://issues.apache.org/jira/browse/SPARK-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10646: -- Issue Type: Sub-task (was: New Feature) Parent: SPARK-10385 > Bivariate Statistics: Pearson's

[jira] [Created] (SPARK-10646) Bivariate Statistics: Pearson's Chi-Squared Test for categorical vs. categorical

Jihong MA created SPARK-10646: - Summary: Bivariate Statistics: Pearson's Chi-Squared Test for categorical vs. categorical Key: SPARK-10646 URL: https://issues.apache.org/jira/browse/SPARK-10646 Project:

[jira] [Updated] (SPARK-10645) Bivariate Statistics for continuous vs. continuous

[ https://issues.apache.org/jira/browse/SPARK-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10645: -- Issue Type: Sub-task (was: New Feature) Parent: SPARK-10385 > Bivariate Statistics for

[jira] [Updated] (SPARK-10646) Bivariate Statistics: Pearson's Chi-Squared Test for categorical vs. categorical

[ https://issues.apache.org/jira/browse/SPARK-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10646: -- Description: Pearson's chi-squared goodness of fit test for observed against the expected

[jira] [Updated] (SPARK-10645) Bivariate Statistics for continuous vs. continuous

[ https://issues.apache.org/jira/browse/SPARK-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10645: -- Component/s: SQL ML > Bivariate Statistics for continuous vs. continuous >

[jira] [Updated] (SPARK-10646) Bivariate Statistics: Pearson's Chi-Squared Test for categorical vs. categorical

[ https://issues.apache.org/jira/browse/SPARK-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-10646: -- Component/s: SQL ML > Bivariate Statistics: Pearson's Chi-Squared Test for

[jira] [Commented] (SPARK-8951) support CJK characters in collect()

[ https://issues.apache.org/jira/browse/SPARK-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731140#comment-14731140 ] Jihong MA commented on SPARK-8951: -- This commit cause R style check failure.

[jira] [Commented] (SPARK-8800) Spark SQL Decimal Division operation loss of precision/scale when type is defined as DecimalType.Unlimited

[ https://issues.apache.org/jira/browse/SPARK-8800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627306#comment-14627306 ] Jihong MA commented on SPARK-8800: -- I applied the fix and noticed the same.

[jira] [Commented] (SPARK-8800) Spark SQL Decimal Division operation loss of precision/scale when type is defined as DecimalType.Unlimited

[ https://issues.apache.org/jira/browse/SPARK-8800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627317#comment-14627317 ] Jihong MA commented on SPARK-8800: -- I would like to suggest to revert back the initial

[jira] [Created] (SPARK-8800) Spark SQL Decimal Division operation loss of precision/scale when type is defined as DecimalType.Unlimited

Jihong MA created SPARK-8800: Summary: Spark SQL Decimal Division operation loss of precision/scale when type is defined as DecimalType.Unlimited Key: SPARK-8800 URL: https://issues.apache.org/jira/browse/SPARK-8800

[jira] [Updated] (SPARK-8800) Spark SQL Decimal Division operation loss of precision/scale when type is defined as DecimalType.Unlimited

[ https://issues.apache.org/jira/browse/SPARK-8800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-8800: - Description: According to specification defined in Java doc over BigDecimal :

[jira] [Commented] (SPARK-8800) Spark SQL Decimal Division operation loss of precision/scale when type is defined as DecimalType.Unlimited

[ https://issues.apache.org/jira/browse/SPARK-8800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14612524#comment-14612524 ] Jihong MA commented on SPARK-8800: -- this is an issue noticed after we open up the

[jira] [Commented] (SPARK-8677) Decimal divide operation throws ArithmeticException

[ https://issues.apache.org/jira/browse/SPARK-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14610865#comment-14610865 ] Jihong MA commented on SPARK-8677: -- I am not sure if there is guideline for

[jira] [Commented] (SPARK-8677) Decimal divide operation throws ArithmeticException

[ https://issues.apache.org/jira/browse/SPARK-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14610831#comment-14610831 ] Jihong MA commented on SPARK-8677: -- Thanks for fixing the division problem. but this fix

[jira] [Commented] (SPARK-8359) Spark SQL Decimal type precision loss on multiplication

[ https://issues.apache.org/jira/browse/SPARK-8359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14604041#comment-14604041 ] Jihong MA commented on SPARK-8359: -- This fix is causing issue with divide over

[jira] [Commented] (SPARK-6548) Adding stddev to DataFrame functions

[ https://issues.apache.org/jira/browse/SPARK-6548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562298#comment-14562298 ] Jihong MA commented on SPARK-6548: -- Hi sdfox, I thought you are no longer working on

[jira] [Created] (SPARK-7357) Improving HBaseTest example

Jihong MA created SPARK-7357: Summary: Improving HBaseTest example Key: SPARK-7357 URL: https://issues.apache.org/jira/browse/SPARK-7357 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-7265) Improving documentation for Spark SQL Hive support

Jihong MA created SPARK-7265: Summary: Improving documentation for Spark SQL Hive support Key: SPARK-7265 URL: https://issues.apache.org/jira/browse/SPARK-7265 Project: Spark Issue Type:

[jira] [Updated] (SPARK-7265) Improving documentation for Spark SQL Hive support

[ https://issues.apache.org/jira/browse/SPARK-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jihong MA updated SPARK-7265: - Priority: Minor (was: Trivial) Improving documentation for Spark SQL Hive support