Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1733#discussion_r15981441
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/Statistics.scala
---
@@ -89,4 +91,64 @@ object Statistics {
*/
@Experimental
def corr(x: RDD[Double], y: RDD[Double], method: String): Double =
Correlations.corr(x, y, method)
+
+ /**
+ * :: Experimental ::
+ * Conduct Pearson's chi-squared goodness of fit test of the observed
data against the
+ * expected distribution.
+ *
+ * Note: the two input Vectors need to have the same size.
+ * `observed` cannot contain negative values.
+ * `expected` cannot contain nonpositive values.
+ *
+ * @param observed Vector containing the observed categorical
counts/relative frequencies.
+ * @param expected Vector containing the expected categorical
counts/relative frequencies.
+ * `expected` is rescaled if the `expected` sum differs
from the `observed` sum.
+ * @return ChiSquaredTest object containing the test statistic, degrees
of freedom, p-value,
+ * the method used, and the null hypothesis.
+ */
+ @Experimental
+ def chiSqTest(observed: Vector,
+ expected: Vector): ChiSquaredTestResult =
ChiSquaredTest.chiSquared(observed, expected)
+
+ /**
+ * :: Experimental ::
+ * Conduct Pearson's chi-squared goodness of fit test of the observed
data against the uniform
+ * distribution, with each category having an expected frequency of `1 /
observed.size`.
+ *
+ * Note: `observed` cannot contain negative values.
+ *
+ * @param observed Vector containing the observed categorical
counts/relative frequencies.
+ * @return ChiSquaredTest object containing the test statistic, degrees
of freedom, p-value,
+ * the method used, and the null hypothesis.
+ */
+ @Experimental
+ def chiSqTest(observed: Vector): ChiSquaredTestResult =
ChiSquaredTest.chiSquared(observed)
+
+ /**
+ * :: Experimental ::
+ * Conduct Pearson's independence test on the input contingency matrix,
which cannot contain
+ * negative entries or columns or rows that sum up to 0.
+ *
+ * @param counts The contingency matrix.
+ * @return ChiSquaredTest object containing the test statistic, degrees
of freedom, p-value,
+ * the method used, and the null hypothesis.
+ */
+ @Experimental
+ def chiSqTest(counts: Matrix): ChiSquaredTestResult =
ChiSquaredTest.chiSquaredMatrix(counts)
+
+ /**
+ * :: Experimental ::
+ * Conduct Pearson's independence test for every feature against the
label across the input RDD.
+ * For each feature, the (feature, label) pairs are converted into a
contingency matrix for which
+ * the chi-squared statistic is computed.
+ *
+ * @param data an `RDD[LabeledPoint]` containing the Labeled dataset.
--- End diff --
mention categorical here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
