Github user sun-rui commented on a diff in the pull request: https://github.com/apache/spark/pull/12493#discussion_r60394761 --- Diff: R/pkg/R/DataFrame.R --- @@ -1125,6 +1125,52 @@ setMethod("summarize", agg(x, ...) }) +#' dapply +#' +#' Apply a function to each partition of a DataFrame. +#' +#' @param x A DataFrame +#' @param func A function to be applied to each partition of the DataFrame. +#' @param schema Optional. The schema of the resulting DataFrame after the function is applied. +#' If NULL, the content of the resulting DataFrame is serialized R data. --- End diff -- sure. If schema is not specified, each partition of the result DataFrame will be serialized in R into a single byte array. Such resulting DataFrame can be processed by successive calls to dapply() and collect() on it can be supported. For now collect() on it only returns RAW data, in future PR we can deserialize the RAW data into R objects.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org