[jira] [Commented] (SPARK-30629) cleanClosure on recursive call leads to node stack overflow
[ https://issues.apache.org/jira/browse/SPARK-30629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023717#comment-17023717 ] Maciej Szymkiewicz commented on SPARK-30629: Makes sense. I guess if we have to choose between these two, recursive calls will be less common. I think that for now we should disable the test case (I've opened a PR for that), as it is clearly no longer applicable. > cleanClosure on recursive call leads to node stack overflow > --- > > Key: SPARK-30629 > URL: https://issues.apache.org/jira/browse/SPARK-30629 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 3.0.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > This problem surfaced while handling SPARK-22817. In theory there are tests, > which cover that problem, but it seems like they have been dead for some > reason. > Reproducible example > {code:r} > f <- function(x) { > f(x) > } > SparkR:::cleanClosure(f) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30629) cleanClosure on recursive call leads to node stack overflow
[ https://issues.apache.org/jira/browse/SPARK-30629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023715#comment-17023715 ] Hossein Falaki commented on SPARK-30629: Yes, this is a good example. There must be a solution to avoid the old bug and not fail with such a case. I could not find one. > cleanClosure on recursive call leads to node stack overflow > --- > > Key: SPARK-30629 > URL: https://issues.apache.org/jira/browse/SPARK-30629 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 3.0.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > This problem surfaced while handling SPARK-22817. In theory there are tests, > which cover that problem, but it seems like they have been dead for some > reason. > Reproducible example > {code:r} > f <- function(x) { > f(x) > } > SparkR:::cleanClosure(f) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30629) cleanClosure on recursive call leads to node stack overflow
[ https://issues.apache.org/jira/browse/SPARK-30629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023714#comment-17023714 ] Maciej Szymkiewicz commented on SPARK-30629: Although I still have some doubts... This should terminate (at least for some values of {{x}}) {code:r} f <- function(x) { if(x > 0) { f(x - 1) } else { x } } {code} but it still fails with the same exception (build from d5b92b24c41b047c64a4d89cc4061ebf534f0995) {code} > SparkR:::cleanClosure(f) Error: node stack overflow {code} > cleanClosure on recursive call leads to node stack overflow > --- > > Key: SPARK-30629 > URL: https://issues.apache.org/jira/browse/SPARK-30629 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 3.0.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > This problem surfaced while handling SPARK-22817. In theory there are tests, > which cover that problem, but it seems like they have been dead for some > reason. > Reproducible example > {code:r} > f <- function(x) { > f(x) > } > SparkR:::cleanClosure(f) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30629) cleanClosure on recursive call leads to node stack overflow
[ https://issues.apache.org/jira/browse/SPARK-30629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023713#comment-17023713 ] Maciej Szymkiewicz commented on SPARK-30629: Thanks for clarification, I'll handle this in moment. > cleanClosure on recursive call leads to node stack overflow > --- > > Key: SPARK-30629 > URL: https://issues.apache.org/jira/browse/SPARK-30629 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 3.0.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > This problem surfaced while handling SPARK-22817. In theory there are tests, > which cover that problem, but it seems like they have been dead for some > reason. > Reproducible example > {code:r} > f <- function(x) { > f(x) > } > SparkR:::cleanClosure(f) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30629) cleanClosure on recursive call leads to node stack overflow
[ https://issues.apache.org/jira/browse/SPARK-30629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023712#comment-17023712 ] Hossein Falaki commented on SPARK-30629: Oh yes. I think we can disable that specific test. If it ever ran and passed it must have been because of the other bug. Thanks for catching it. > cleanClosure on recursive call leads to node stack overflow > --- > > Key: SPARK-30629 > URL: https://issues.apache.org/jira/browse/SPARK-30629 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 3.0.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > This problem surfaced while handling SPARK-22817. In theory there are tests, > which cover that problem, but it seems like they have been dead for some > reason. > Reproducible example > {code:r} > f <- function(x) { > f(x) > } > SparkR:::cleanClosure(f) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30629) cleanClosure on recursive call leads to node stack overflow
[ https://issues.apache.org/jira/browse/SPARK-30629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023711#comment-17023711 ] Maciej Szymkiewicz commented on SPARK-30629: [~falaki] Fair enough, but the problem is that we actually test for that specific scenario https://github.com/apache/spark/blob/d5b92b24c41b047c64a4d89cc4061ebf534f0995/R/pkg/tests/fulltests/test_utils.R#L92, which cause test failures. > cleanClosure on recursive call leads to node stack overflow > --- > > Key: SPARK-30629 > URL: https://issues.apache.org/jira/browse/SPARK-30629 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 3.0.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > This problem surfaced while handling SPARK-22817. In theory there are tests, > which cover that problem, but it seems like they have been dead for some > reason. > Reproducible example > {code:r} > f <- function(x) { > f(x) > } > SparkR:::cleanClosure(f) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30629) cleanClosure on recursive call leads to node stack overflow
[ https://issues.apache.org/jira/browse/SPARK-30629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023710#comment-17023710 ] Hossein Falaki commented on SPARK-30629: [~zero323] that indefinite recursive call will never execute. So it is better to fail during {{cleanClosure()}} rather than execution on workers. > cleanClosure on recursive call leads to node stack overflow > --- > > Key: SPARK-30629 > URL: https://issues.apache.org/jira/browse/SPARK-30629 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 3.0.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > This problem surfaced while handling SPARK-22817. In theory there are tests, > which cover that problem, but it seems like they have been dead for some > reason. > Reproducible example > {code:r} > f <- function(x) { > f(x) > } > SparkR:::cleanClosure(f) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30629) cleanClosure on recursive call leads to node stack overflow
[ https://issues.apache.org/jira/browse/SPARK-30629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023708#comment-17023708 ] Maciej Szymkiewicz commented on SPARK-30629: CC [~mengxr] [~falaki] > cleanClosure on recursive call leads to node stack overflow > --- > > Key: SPARK-30629 > URL: https://issues.apache.org/jira/browse/SPARK-30629 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 3.0.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > This problem surfaced while handling SPARK-22817. In theory there are tests, > which cover that problem, but it seems like they have been dead for some > reason. > Reproducible example > {code:r} > f <- function(x) { > f(x) > } > newF <- cleanClosure(f) > {code} > Just looking at the {{cleanClosure}} / {{processClosure}} pair, that > function that is being processed is not added to {{checkedFuncs}}. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30629) cleanClosure on recursive call leads to node stack overflow
[ https://issues.apache.org/jira/browse/SPARK-30629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023706#comment-17023706 ] Maciej Szymkiewicz commented on SPARK-30629: OK, so I checked and it looks like the problem has been introduced by SPARK-29777. > cleanClosure on recursive call leads to node stack overflow > --- > > Key: SPARK-30629 > URL: https://issues.apache.org/jira/browse/SPARK-30629 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 3.0.0 >Reporter: Maciej Szymkiewicz >Priority: Major > > This problem surfaced while handling SPARK-22817. In theory there are tests, > which cover that problem, but it seems like they have been dead for some > reason. > Reproducible example > {code:r} > f <- function(x) { > f(x) > } > newF <- cleanClosure(f) > {code} > Just looking at the {{cleanClosure}} / {{processClosure}} pair, that > function that is being processed is not added to {{checkedFuncs}}. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org