HyukjinKwon commented on a change in pull request #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions URL: https://github.com/apache/spark/pull/27433#discussion_r373898606
########## File path: R/pkg/R/functions.R ########## @@ -3281,6 +3322,126 @@ setMethod("row_number", ###################### Collection functions###################### +#' Create o.a.s.sql.expressions.UnresolvedNamedLambdaVariable, +#' convert it to o.s.sql.Column and wrap with R Column. +#' Used by higher order functions. +#' +#' @param ... character of length = 1 +#' if length(...) > 1 then argument is interpreted as a nested +#' Column, for example \code{unresolved_named_lambda_var("a", "b", "c")} +#' yields unresolved \code{a.b.c} +#' @return Column object wrapping JVM UnresolvedNamedLambdaVariable +unresolved_named_lambda_var <- function(...) { + jc <- sparkR.newJObject( + "org.apache.spark.sql.Column", + sparkR.newJObject( + "org.apache.spark.sql.catalyst.expressions.UnresolvedNamedLambdaVariable", + list(...) + ) + ) + column(jc) +} + +#' Create o.a.s.sql.expressions.LambdaFunction corresponding +#' to transformation described by func. +#' Used by higher order functions. +#' +#' @param fun R \code{function} (unary, binary or ternary) +#' that transforms \code{Columns} into a \code{Column} +#' @param expected_nargs numeric a vector of the expected +#' number of arguments. Used for validation +#' @return JVM \code{LambdaFunction} object +create_lambda <- function(fun, expected_nargs) { + as_jexpr <- function(x) callJMethod(x@jc, "expr") + + # Process function arguments + parameters <- formals(fun) + nparameters <- length(parameters) + + stopifnot(nparameters %in% expected_nargs) Review comment: cc @felixcheung, @falaki, @shivaram, this one, I am not sure about it yet (see https://github.com/apache/spark/pull/27406#discussion_r373896752 for more details). Also, given that JVM throws an error as below: ``` : org.apache.spark.sql.AnalysisException: The number of lambda function arguments '2' does not match the number of arguments expected by the higher order function '1'.; ``` if we use `captureJVMException` (see https://github.com/apache/spark/pull/27433/files#r373898428), I assume it will show R-friendly error messages. For a simple checking in R side, I am supportive but I am less sure for such complex checking. I tend to think it's better to have less codes if the error message is already readable at the current moment. We should find a better way to handle the error messages in general (e.g., at [utils.R#L820-L872](https://github.com/apache/spark/blob/master/R/pkg/R/utils.R#L820-L872)). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org