[
https://issues.apache.org/jira/browse/SPARK-56810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated SPARK-56810:
-----------------------------------
Affects Version/s: 4.1.1
3.5.8
4.0.1
(was: 4.1.2)
(was: 3.5.9)
> Fix cleanClosure to skip primitive functions for R 4.4+
> -------------------------------------------------------
>
> Key: SPARK-56810
> URL: https://issues.apache.org/jira/browse/SPARK-56810
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 4.0.1, 3.5.8, 4.2.0, 4.1.1
> Reporter: Kousuke Saruta
> Priority: Major
>
> With R 4.4+, R/run-tests.sh fails.
> {code}
> == Failed
> ======================================================================
> -- 1. Error ('test_rdd.R:49:3'): count by values and keys
> ----------------------
> Error in `h(simpleError(msg, call))`: error in evaluating the argument 'x' in
> selecting a method for function 'collectRDD': (converted from warning)
> setting environment(<primitive function>) is not possible and trying it is
> deprecated
> Backtrace:
> x
> 1. +-SparkR:::countByValue(mods) at test_rdd.R:49:3
> 2. +-SparkR:::countByValue(mods)
> 3. | +-SparkR:::collectRDD(reduceByKey(ones, `+`, getNumPartitionsRDD(x)))
> 4. | +-SparkR:::reduceByKey(ones, `+`, getNumPartitionsRDD(x))
> 5. | \-SparkR:::reduceByKey(ones, `+`, getNumPartitionsRDD(x))
> 6. | +-SparkR:::lapplyPartition(x, reduceVals)
> 7. | \-SparkR:::lapplyPartition(x, reduceVals)
> 8. | +-SparkR:::lapplyPartitionsWithIndex(...)
> 9. | \-SparkR:::lapplyPartitionsWithIndex(...)
> 10. | \-SparkR:::PipelinedRDD(X, FUN)
> 11. | \-methods::new("PipelinedRDD", prev, func, NULL)
> 12. | +-methods::initialize(value, ...)
> 13. | \-SparkR (local) initialize(value, ...)
> 14. | \-SparkR (local) .local(.Object, ...)
> 15. | \-SparkR:::cleanClosure(pipelinedFunc)
> 16. | \-SparkR:::processClosure(...)
> 17. | \-SparkR:::processClosure(...)
> 18. | \-SparkR:::processClosure(...)
> 19. | \-SparkR:::processClosure(...)
> 20. | \-SparkR:::cleanClosure(obj, checkedFuncs)
> 21. | \-SparkR:::processClosure(...)
> 22. | \-SparkR:::processClosure(...)
> 23. | \-SparkR:::processClosure(...)
> 24. | \-SparkR:::cleanClosure(obj,
> checkedFuncs)
> 25. | \-SparkR:::processClosure(...)
> 26. | \-SparkR:::processClosure(...)
> 27. | \-SparkR:::processClosure(...)
> 28. | \-SparkR:::processClosure(...)
> 29. |
> \-SparkR:::processClosure(...)
> 30. |
> \-SparkR:::processClosure(...)
> 31. |
> \-SparkR:::cleanClosure(obj, checkedFuncs)
> 32. +-base::.signalSimpleWarning(...)
> 33. | \-base::withRestarts(...)
> 34. | \-base (local) withOneRestart(expr, restarts[[1L]])
> 35. | \-base (local) doWithOneRestart(return(expr), restart)
> 36. \-base::.handleSimpleError(...)
> 37. \-base (local) h(simpleError(msg, call))
> -- 2. Error ('test_rdd.R:282:3'): maximum() on RDDs
> ----------------------------
> Error in `h(simpleError(msg, call))`: error in evaluating the argument 'x' in
> selecting a method for function 'collectRDD': (converted from warning)
> setting environment(<primitive function>) is not possible and trying it is
> deprecated
> Backtrace:
> x
> 1. +-SparkR:::maximum(rdd) at test_rdd.R:282:3
> 2. +-SparkR:::maximum(rdd)
> 3. | +-SparkR:::reduce(x, max)
> 4. | \-SparkR:::reduce(x, max)
> 5. | +-SparkR:::collectRDD(lapplyPartition(x, reducePartition), flatten =
> FALSE)
> 6. | +-SparkR:::lapplyPartition(x, reducePartition)
> 7. | \-SparkR:::lapplyPartition(x, reducePartition)
> 8. | +-SparkR:::lapplyPartitionsWithIndex(...)
> 9. | \-SparkR:::lapplyPartitionsWithIndex(...)
> 10. | \-SparkR:::PipelinedRDD(X, FUN)
> 11. | \-methods::new("PipelinedRDD", prev, func, NULL)
> 12. | +-methods::initialize(value, ...)
> 13. | \-SparkR (local) initialize(value, ...)
> 14. | \-SparkR (local) .local(.Object, ...)
> 15. | \-SparkR:::cleanClosure(func)
> 16. | \-SparkR:::processClosure(...)
> 17. | \-SparkR:::processClosure(...)
> 18. | \-SparkR:::processClosure(...)
> 19. | \-SparkR:::cleanClosure(obj, checkedFuncs)
> 20. | \-SparkR:::processClosure(...)
> 21. | \-SparkR:::processClosure(...)
> 22. | \-SparkR:::processClosure(...)
> 23. | \-SparkR:::cleanClosure(obj,
> checkedFuncs)
> 24. +-base::.signalSimpleWarning(...)
> 25. | \-base::withRestarts(...)
> 26. | \-base (local) withOneRestart(expr, restarts[[1L]])
> 27. | \-base (local) doWithOneRestart(return(expr), restart)
> 28. \-base::.handleSimpleError(...)
> 29. \-base (local) h(simpleError(msg, call))
> -- 3. Error ('test_rdd.R:287:3'): minimum() on RDDs
> ----------------------------
> Error in `h(simpleError(msg, call))`: error in evaluating the argument 'x' in
> selecting a method for function 'collectRDD': (converted from warning)
> setting environment(<primitive function>) is not possible and trying it is
> deprecated
> Backtrace:
> x
> 1. +-SparkR:::minimum(rdd) at test_rdd.R:287:3
> 2. +-SparkR:::minimum(rdd)
> 3. | +-SparkR:::reduce(x, min)
> 4. | \-SparkR:::reduce(x, min)
> 5. | +-SparkR:::collectRDD(lapplyPartition(x, reducePartition), flatten =
> FALSE)
> 6. | +-SparkR:::lapplyPartition(x, reducePartition)
> 7. | \-SparkR:::lapplyPartition(x, reducePartition)
> 8. | +-SparkR:::lapplyPartitionsWithIndex(...)
> 9. | \-SparkR:::lapplyPartitionsWithIndex(...)
> 10. | \-SparkR:::PipelinedRDD(X, FUN)
> 11. | \-methods::new("PipelinedRDD", prev, func, NULL)
> 12. | +-methods::initialize(value, ...)
> 13. | \-SparkR (local) initialize(value, ...)
> 14. | \-SparkR (local) .local(.Object, ...)
> 15. | \-SparkR:::cleanClosure(func)
> 16. | \-SparkR:::processClosure(...)
> 17. | \-SparkR:::processClosure(...)
> 18. | \-SparkR:::processClosure(...)
> 19. | \-SparkR:::cleanClosure(obj, checkedFuncs)
> 20. | \-SparkR:::processClosure(...)
> 21. | \-SparkR:::processClosure(...)
> 22. | \-SparkR:::processClosure(...)
> 23. | \-SparkR:::cleanClosure(obj,
> checkedFuncs)
> 24. +-base::.signalSimpleWarning(...)
> 25. | \-base::withRestarts(...)
> 26. | \-base (local) withOneRestart(expr, restarts[[1L]])
> 27. | \-base (local) doWithOneRestart(return(expr), restart)
> 28. \-base::.handleSimpleError(...)
> 29. \-base (local) h(simpleError(msg, call))
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
