[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13109 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67813615
--- Diff: R/pkg/R/stats.R ---
@@ -134,9 +129,7 @@ setMethod("freqItems", signature(x = "SparkDataFrame",
cols = "character"),
collect(dataFrame(sct))
})
-#' approxQuantile
-#'
-#' Calculates the approximate quantiles of a numerical column of a
SparkDataFrame.
+#' approxQuantile - Calculates the approximate quantiles of a numerical
column of a SparkDataFrame.
--- End diff --
Fixed, sorry I missed one `approxQuantile`. thanks for testing it out.
I thought completely arbitrary do-not-have-newline would be hard for
someone new to follow, so I added `@description` instead to make it clear in
case of copy/paste and someone added a new empty line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67812909
--- Diff: R/pkg/R/stats.R ---
@@ -134,9 +129,7 @@ setMethod("freqItems", signature(x = "SparkDataFrame",
cols = "character"),
collect(dataFrame(sct))
})
-#' approxQuantile
-#'
-#' Calculates the approximate quantiles of a numerical column of a
SparkDataFrame.
+#' approxQuantile - Calculates the approximate quantiles of a numerical
column of a SparkDataFrame.
--- End diff --
@felixcheung . I found that `@description` and `@details` works for me.
```
#' @description
#' approxQuantile - Calculates the approximate quantiles of a numerical
column of a SparkDataFrame.
#'
#' @details
#' The result o~~
```
If we keep the description, please try the above pair.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67812243
--- Diff: R/pkg/R/stats.R ---
@@ -134,9 +129,7 @@ setMethod("freqItems", signature(x = "SparkDataFrame",
cols = "character"),
collect(dataFrame(sct))
})
-#' approxQuantile
-#'
-#' Calculates the approximate quantiles of a numerical column of a
SparkDataFrame.
+#' approxQuantile - Calculates the approximate quantiles of a numerical
column of a SparkDataFrame.
--- End diff --
Except the above, LGTM!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67812187
--- Diff: R/pkg/R/stats.R ---
@@ -134,9 +129,7 @@ setMethod("freqItems", signature(x = "SparkDataFrame",
cols = "character"),
collect(dataFrame(sct))
})
-#' approxQuantile
-#'
-#' Calculates the approximate quantiles of a numerical column of a
SparkDataFrame.
+#' approxQuantile - Calculates the approximate quantiles of a numerical
column of a SparkDataFrame.
--- End diff --
Oh, @shivaram already mentioned this. +1 for @shivaram 's opinion.
If we delete the details, `@description` is not needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67811795
--- Diff: R/pkg/R/stats.R ---
@@ -134,9 +129,7 @@ setMethod("freqItems", signature(x = "SparkDataFrame",
cols = "character"),
collect(dataFrame(sct))
})
-#' approxQuantile
-#'
-#' Calculates the approximate quantiles of a numerical column of a
SparkDataFrame.
+#' approxQuantile - Calculates the approximate quantiles of a numerical
column of a SparkDataFrame.
--- End diff --
Unfortunately, this line is ignored. We need `@description` here, too.
```
#' @description
#' approxQuantile - Calculates the approximate quantiles of a numerical
column of a SparkDataFrame.
```
After adding that, the description depth will look differently. I mean only
`approxQuantile` has a detail description like the following.
```
crosstab - Computes a pair-wise frequency table of the given columns. Also
known as a contingency table. The number of distinct values for each column
should be less than 1e4. At most 1e6 non-zero pair frequencies will be returned.
freqItems - Finding frequent items for columns, possibly with false
positives. Using the frequent element count algorithm described in
http://dx.doi.org/10.1145/762471.762473, proposed by Karp, Schenker, and
Papadimitriou.
approxQuantile - Calculates the approximate quantiles of a numerical column
of a SparkDataFrame.
The result of this algorithm has the following deterministic bound: If the
SparkDataFrame has N elements and if we request the quantile at probability 'p'
up to error 'err', then the algorithm will return a sample 'x' from the
SparkDataFrame so that the *exact* rank of 'x' is close to (p * N). More
precisely, floor((p - err) * N) <= rank(x) <= ceil((p + err) * N).
This method implements a variation of the Greenwald-Khanna algorithm (with
some speed optimizations). The algorithm was first present in
[[http://dx.doi.org/10.1145/375663.375670 Space-efficient Online Computation of
Quantile Summaries]] by Greenwald and Khanna.
sampleBy - Returns a stratified sample without replacement based on the
fraction given on each stratum.
```
I'm not sure about balancing them by removing `The result of this
algorithm~~~Khanna.`. If you think that is okay, we can keep that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67811551
--- Diff: R/pkg/R/stats.R ---
@@ -134,9 +129,7 @@ setMethod("freqItems", signature(x = "SparkDataFrame",
cols = "character"),
collect(dataFrame(sct))
})
-#' approxQuantile
-#'
-#' Calculates the approximate quantiles of a numerical column of a
SparkDataFrame.
+#' approxQuantile - Calculates the approximate quantiles of a numerical
column of a SparkDataFrame.
#'
--- End diff --
I think we need to delete this line in between the `approxQuantile` line
and the `The result of this algorithm` line. Otherwise the first line doesn't
seem to show up in the rendered doc ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67810438
--- Diff: R/pkg/R/generics.R ---
@@ -430,19 +430,19 @@ setGeneric("coltypes<-", function(x, value) {
standardGeneric("coltypes<-") })
#' @export
setGeneric("columns", function(x) {standardGeneric("columns") })
-#' @rdname statfunctions
+#' @rdname cov
#' @export
setGeneric("cov", function(x, ...) {standardGeneric("cov") })
-#' @rdname statfunctions
+#' @rdname corr
#' @export
setGeneric("corr", function(x, ...) {standardGeneric("corr") })
-#' @rdname statfunctions
+#' @rdname cov
#' @export
setGeneric("covar_samp", function(col1, col2)
{standardGeneric("covar_samp") })
-#' @rdname statfunctions
+#' @rdname cov
--- End diff --
good catch, thx!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67809806
--- Diff: R/pkg/R/generics.R ---
@@ -430,19 +430,19 @@ setGeneric("coltypes<-", function(x, value) {
standardGeneric("coltypes<-") })
#' @export
setGeneric("columns", function(x) {standardGeneric("columns") })
-#' @rdname statfunctions
+#' @rdname cov
#' @export
setGeneric("cov", function(x, ...) {standardGeneric("cov") })
-#' @rdname statfunctions
+#' @rdname corr
#' @export
setGeneric("corr", function(x, ...) {standardGeneric("corr") })
-#' @rdname statfunctions
+#' @rdname cov
#' @export
setGeneric("covar_samp", function(col1, col2)
{standardGeneric("covar_samp") })
-#' @rdname statfunctions
+#' @rdname cov
--- End diff --
If then, here, `cov` -> `covar_pop`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67808208
--- Diff: R/pkg/R/stats.R ---
@@ -19,7 +19,8 @@
setOldClass("jobj")
-#' crosstab
+#' @title SparkDataFrame statistic functions
--- End diff --
Fixed, sorry about that.
`cov`, `corr` shouldn't be there - they are referenced in generic.R these
bugs are also fixed in my other PR #13798 - there are quite a lot of them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67806444
--- Diff: R/pkg/R/stats.R ---
@@ -19,7 +19,8 @@
setOldClass("jobj")
-#' crosstab
+#' @title SparkDataFrame statistic functions
--- End diff --
I don't see the build error - but the page title doesn't come up right. A
screenshot is at
https://www.dropbox.com/s/sc1mrd7upr6t7mp/Screenshot%202016-06-20%2021.25.57.png?dl=0
Also we seem to have some functions like covar_samp, covar_pop that don't
have a description ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67806331
--- Diff: R/pkg/R/stats.R ---
@@ -19,7 +19,8 @@
setOldClass("jobj")
-#' crosstab
+#' @title SparkDataFrame statistic functions
--- End diff --
It's a little bit different from your screenshot (**After**).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67806157
--- Diff: R/pkg/R/stats.R ---
@@ -19,7 +19,8 @@
setOldClass("jobj")
-#' crosstab
+#' @title SparkDataFrame statistic functions
--- End diff --
For me, the generated html file,
`file:///Users/dongjoon/spark/R/pkg/html/statfunctions.html` has the following
title.
```
SparkDataFrame statistic functions crosstab - Computes a pair-wise
frequency table of the given columns Computes a pair-wise frequency table of
the given columns. Also known as a contingency table. The number of distinct
values for each column should be less than 1e4. At most 1e6 non-zero pair
frequencies will be returned.
```
Also, `index.html` shows the above long string for all the stat functions
like `approxQuantile`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67805814
--- Diff: R/pkg/R/stats.R ---
@@ -19,7 +19,8 @@
setOldClass("jobj")
-#' crosstab
+#' @title SparkDataFrame statistic functions
--- End diff --
what's the error you see? this works for me and Jenkins run create docs too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13109: [SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/13109#discussion_r67803750
--- Diff: R/pkg/R/stats.R ---
@@ -19,7 +19,8 @@
setOldClass("jobj")
-#' crosstab
+#' @title SparkDataFrame statistic functions
--- End diff --
Hi, @felixcheung .
When I use `./create-docs.sh`, this breaks the page.
Should I do differently to generate the html page like yours?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
