Marc Le Bihan created SPARK-58318:
-------------------------------------
Summary: SQL Catalyst 4.2.0 causes a StackOverflow error when it
attempt to optimize of a request it is preparing
Key: SPARK-58318
URL: https://issues.apache.org/jira/browse/SPARK-58318
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Environment: Debian 13
Spark 4.2.0
Java 25
Reporter: Marc Le Bihan
I'm following a String[] of competencies codes, at their indexes :
```log
codesCompetences = \{Arrays$ArrayList@28050} size = 121
0 = "C1005" 1 = "C1010" 2 = "C1015" 3 = "C1020" 4 = "C1025"
5 = "C1030" 6 = "C1035" 7 = "C1040" 8 = "C1045" 9 = "C1505"
10 = "C1510" 11 = "C1515" 12 = "C1520" 13 = "C2000" 14 = "C2005"
15 = "C2010" 16 = "C2015" 17 = "C2020" 18 = "C2025" 19 = "C2030"
20 = "C2035" 21 = "C2040" 22 = "C2045" 23 = "C2050" 24 = "C2055"
25 = "C2060" 26 = "C2065" 27 = "C2070" 28 = "C2075" 29 = "C2080"
30 = "C2085" 31 = "C2090" 32 = "C2095" 33 = "C2096" 34 = "C2500"
35 = "C2505" 36 = "C2510" 37 = "C3000" 38 = "C3005" 39 = "C3010"
40 = "C3015" 41 = "C3020" 42 = "C3025" 43 = "C3030" 44 = "C3035"
45 = "C3040" 46 = "C3045" 47 = "C3050" 48 = "C3055" 49 = "C3060"
50 = "C3500" 51 = "C4000" 52 = "C4005" 53 = "C4010" 54 = "C4500"
55 = "C4505" 56 = "C4510" 57 = "C4515" 58 = "C4520" 59 = "C5000"
60 = "C5005" 61 = "C5010" 62 = "C5015" 63 = "C5020" 64 = "C5025"
65 = "C5030" 66 = "C5035" 67 = "C5040" 68 = "C5045" 69 = "C5050"
70 = "C5500" 71 = "C5505" 72 = "C5510" 73 = "C5515" 74 = "C5520"
75 = "C5525" 76 = "C5530" 77 = "C5535" 78 = "C5540" 79 = "C6000"
80 = "C6005" 81 = "C6010" 82 = "C6015" 83 = "C6020" 84 = "C6025"
85 = "C6030" 86 = "C6035" 87 = "C6040" 88 = "C6045" 89 = "C6055"
90 = "C6060" 91 = "C6065" 92 = "C6070" 93 = "C6075" 94 = "C6500"
95 = "C6505" 96 = "C6510" 97 = "C6515" 98 = "C6520" 99 = "C6525"
100 = "C6530" 101 = "C6535" 102 = "C7000" 103 = "C7500" 104 = "C7505"
105 = "C7510" 106 = "C7515" 107 = "C7520" 108 = "C7525" 109 = "C7530"
110 = "C7535" 111 = "C7540" 112 = "C7541" 113 = "C7545" 114 = "C7546"
115 = "C7550" 116 = "C7555" 117 = "C7560" 118 = "C7565" 119 = "C7570"
120 = "C7575
```
With them, I'm converting values of columns coming from a CSV file:
```java
{color:#0033b3}private
{color}{color:#000000}Dataset{color}<{color:#000000}Row{color}>
extraireCompetencesDelegueesEtInteretCommunautaire({color:#000000}Dataset{color}<{color:#000000}Row{color}>
{color:#000000}dataset{color}, {color:#0033b3}int
{color}{color:#000000}anneeCog{color}) {
{color:#000000}// Ces valeurs sont celles qu'un distinct() montre possibles sur
les colonnes du fichier csv d'entrée.
{color}{color:#000000} {color}{color:#0033b3}final {color}{color:#000000}String
COMPETENCE_DELEGUEE {color}= "délégué";
{color:#0033b3}final {color}{color:#000000}String
COMPETENCE_INTERET_COMMUNAUTAIRE {color}= "avec intérêt communautaire";
{color:#000000}List{color}<{color:#000000}String{color}>
{color:#000000}codesCompetences {color}=
{color:#0033b3}this{color}.{color:#871094}datasetCompetence{color}.codesCompetencesAsList({color:#000000}anneeCog{color});
{color:#0033b3}for{color}({color:#000000}String codeCompetence {color}:
{color:#000000}codesCompetences{color}) {
{color:#000000}Column colCompetence {color}= {color:#0033b3}new
{color}Column({color:#000000}codeCompetence{color});
{color:#000000}// Créer le champ Cxxx_Deleguee et le one hot encoder en "1" ou
"0" selon que la valeur associée au code compétence vaut 'délégué' ou pas.
{color}{color:#000000} {color}{color:#000000}Column competenceDeleguee {color}=
when({color:#000000}colCompetence{color}.equalTo({color:#000000}COMPETENCE_DELEGUEE{color}),
lit("1")).otherwise(lit("0"));
{color:#000000}dataset {color}=
{color:#000000}dataset{color}.withColumn({color:#000000}codeCompetence {color}+
"_Deleguee", {color:#000000}competenceDeleguee{color});
{color:#000000}// Créer le champ Cxxx_InteretCommunautaire et le one hot
encoder en "1" ou "0" selon que la valeur associée au code compétence vaut
'avec intérêt communautaire' ou pas.
{color}{color:#000000} {color}{color:#000000}Column
competenceInteretCommunautaire {color}=
when({color:#000000}colCompetence{color}.equalTo({color:#000000}COMPETENCE_INTERET_COMMUNAUTAIRE{color}),
lit("1")).otherwise(lit("0"));
{color:#000000}dataset {color}=
{color:#000000}dataset{color}.withColumn({color:#000000}codeCompetence {color}+
"_InteretCommunautaire", {color:#000000}competenceInteretCommunautaire{color});
{color:#000000}// one hot encoder la présence de la compétence
{color}{color:#000000} {color}{color:#000000}Column encoding {color}=
when({color:#000000}colCompetence{color}.isin({color:#871094}OUI{color},
{color:#000000}COMPETENCE_DELEGUEE{color},
{color:#000000}COMPETENCE_INTERET_COMMUNAUTAIRE{color}), lit("1"))
.when({color:#000000}colCompetence{color}.equalTo({color:#871094}NON{color}),
lit("0"))
.otherwise({color:#000000}colCompetence{color});
{color:#000000}dataset {color}=
{color:#000000}dataset{color}.withColumn({color:#000000}codeCompetence{color},
{color:#000000}encoding{color});
{color:#871094}LOGGER{color}.info("added: '{color:#0037a6}{}{color}'",
{color:#000000}encoding{color});
}
{color:#0033b3}return {color}{color:#000000}dataset{color};
}
```
This loop adds to the `Dataset` some columns each time,. Among them one is :
```log
added: 'CASE WHEN in(C6525, 'OUI', 'délégué', 'avec intérêt communautaire')
THEN '1' WHEN =(C6525, 'NON')
```
and it's the last log shown before a:
```log
java.lang.StackOverflowError
at java.base/java.lang.ref.Reference.refersToImpl(Reference.java:384)
at java.base/java.lang.ref.Reference.refersTo(Reference.java:375)
at java.base/java.lang.ThreadLocal$ThreadLocalMap.set(ThreadLocal.java:539)
at java.base/java.lang.ThreadLocal.set(ThreadLocal.java:243)
at java.base/java.lang.ThreadLocal.set(ThreadLocal.java:229)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.set(origin.scala:93)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:105)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:495)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$3(TreeNode.scala:500)
at
org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1268)
at
org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1267)
at
org.apache.spark.sql.catalyst.expressions.UnaryExpression.mapChildren(Expression.scala:596)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:500)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$transformExpressionsDownWithPruning$1(QueryPlan.scala:223)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$1(QueryPlan.scala:264)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:107)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:264)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:276)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$4(QueryPlan.scala:282)
at scala.collection.immutable.List.map(List.scala:236)
at scala.collection.immutable.List.map(List.scala:79)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:282)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$5(QueryPlan.scala:287)
at
org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:337)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:287)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsDownWithPruning(QueryPlan.scala:223)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsWithPruning(QueryPlan.scala:194)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressions(QueryPlan.scala:172)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicate(DeduplicateRelations.scala:240)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicateAndRenew(DeduplicateRelations.scala:323)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.org$apache$spark$sql$catalyst$analysis$DeduplicateRelations$$renewDuplicatedRelations(DeduplicateRelations.scala:126)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1(DeduplicateRelations.scala:233)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1$adapted(DeduplicateRelations.scala:232)
at scala.collection.immutable.Vector.foreach(Vector.scala:2125)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicate(DeduplicateRelations.scala:232)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicateAndRenew(DeduplicateRelations.scala:323)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.org$apache$spark$sql$catalyst$analysis$DeduplicateRelations$$renewDuplicatedRelations(DeduplicateRelations.scala:126)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1(DeduplicateRelations.scala:233)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1$adapted(DeduplicateRelations.scala:232)
[...]
```
What I've noticed during my debugs is that:
* The index of the next value, `C6530`, is `100` (based).
* If I do a step over step debugging, without allowing my program to resume
from `C6525`, I can pursue through `C6530`, `C6535`... But as soon as do a
Resume program, the stack overflow happens.
So I guess it's an helper thread that starts to work on the request being
prepared that causes the trouble.
The problem didn't exist with `Spark 4.1.1` I was using before, with `Java 21`
then.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
