[
https://issues.apache.org/jira/browse/SPARK-58318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marc Le Bihan updated SPARK-58318:
----------------------------------
Description:
I'm following a `String[]` of competencies codes, at their indexes :
{code:java}
codesCompetences = (size = 121)
0 = "C1005" 1 = "C1010" 2 = "C1015" 3 = "C1020" 4 = "C1025"
5 = "C1030" 6 = "C1035" 7 = "C1040" 8 = "C1045" 9 = "C1505"
10 = "C1510" 11 = "C1515" 12 = "C1520" 13 = "C2000" 14 = "C2005"
15 = "C2010" 16 = "C2015" 17 = "C2020" 18 = "C2025" 19 = "C2030"
20 = "C2035" 21 = "C2040" 22 = "C2045" 23 = "C2050" 24 = "C2055"
25 = "C2060" 26 = "C2065" 27 = "C2070" 28 = "C2075" 29 = "C2080"
30 = "C2085" 31 = "C2090" 32 = "C2095" 33 = "C2096" 34 = "C2500"
35 = "C2505" 36 = "C2510" 37 = "C3000" 38 = "C3005" 39 = "C3010"
40 = "C3015" 41 = "C3020" 42 = "C3025" 43 = "C3030" 44 = "C3035"
45 = "C3040" 46 = "C3045" 47 = "C3050" 48 = "C3055" 49 = "C3060"
50 = "C3500" 51 = "C4000" 52 = "C4005" 53 = "C4010" 54 = "C4500"
55 = "C4505" 56 = "C4510" 57 = "C4515" 58 = "C4520" 59 = "C5000"
60 = "C5005" 61 = "C5010" 62 = "C5015" 63 = "C5020" 64 = "C5025"
65 = "C5030" 66 = "C5035" 67 = "C5040" 68 = "C5045" 69 = "C5050"
70 = "C5500" 71 = "C5505" 72 = "C5510" 73 = "C5515" 74 = "C5520"
75 = "C5525" 76 = "C5530" 77 = "C5535" 78 = "C5540" 79 = "C6000"
80 = "C6005" 81 = "C6010" 82 = "C6015" 83 = "C6020" 84 = "C6025"
85 = "C6030" 86 = "C6035" 87 = "C6040" 88 = "C6045" 89 = "C6055"
90 = "C6060" 91 = "C6065" 92 = "C6070" 93 = "C6075" 94 = "C6500"
95 = "C6505" 96 = "C6510" 97 = "C6515" 98 = "C6520" 99 = "C6525"
100 = "C6530" 101 = "C6535" 102 = "C7000" 103 = "C7500" 104 = "C7505"
105 = "C7510" 106 = "C7515" 107 = "C7520" 108 = "C7525" 109 = "C7530"
110 = "C7535" 111 = "C7540" 112 = "C7541" 113 = "C7545" 114 = "C7546"
115 = "C7550" 116 = "C7555" 117 = "C7560" 118 = "C7565" 119 = "C7570"
120 = "C7575 {code}
With them, I'm converting values of columns coming from a CSV file:
{code:java}
private Dataset<Row>
extraireCompetencesDelegueesEtInteretCommunautaire(Dataset<Row> dataset, int
anneeCog) {
// Ces valeurs sont celles qu'un distinct() montre possibles sur les
colonnes du fichier csv d'entrée.
final String COMPETENCE_DELEGUEE = "délégué";
final String COMPETENCE_INTERET_COMMUNAUTAIRE = "avec intérêt
communautaire";
List<String> codesCompetences =
this.datasetCompetence.codesCompetencesAsList(anneeCog);
for(String codeCompetence : codesCompetences) {
Column colCompetence = new Column(codeCompetence);
// Créer le champ Cxxx_Deleguee et le one hot encoder en "1" ou "0" selon
que la valeur associée au code compétence vaut 'délégué' ou pas.
Column competenceDeleguee =
when(colCompetence.equalTo(COMPETENCE_DELEGUEE), lit("1")).otherwise(lit("0"));
dataset = dataset.withColumn(codeCompetence + "_Deleguee",
competenceDeleguee);
// Créer le champ Cxxx_InteretCommunautaire et le one hot encoder en "1"
ou "0" selon que la valeur associée au code compétence vaut 'avec intérêt
communautaire' ou pas.
Column competenceInteretCommunautaire =
when(colCompetence.equalTo(COMPETENCE_INTERET_COMMUNAUTAIRE),
lit("1")).otherwise(lit("0"));
dataset = dataset.withColumn(codeCompetence + "_InteretCommunautaire",
competenceInteretCommunautaire);
// one hot encoder la présence de la compétence
Column encoding = when(colCompetence.isin(OUI, COMPETENCE_DELEGUEE,
COMPETENCE_INTERET_COMMUNAUTAIRE), lit("1"))
.when(colCompetence.equalTo(NON), lit("0"))
.otherwise(colCompetence);
dataset = dataset.withColumn(codeCompetence, encoding);
LOGGER.info("added: '{}'", encoding);
}
return dataset;
}
{code}
This loop adds to the `Dataset` some columns each time,. Among them one is :
{code:java}
added: 'CASE WHEN in(C6525, 'OUI', 'délégué', 'avec intérêt communautaire')
THEN '1' WHEN =(C6525, 'NON') {code}
and it's the last log shown before a:
{code:java}
java.lang.StackOverflowError
at java.base/java.lang.ref.Reference.refersToImpl(Reference.java:384)
at java.base/java.lang.ref.Reference.refersTo(Reference.java:375)
at java.base/java.lang.ThreadLocal$ThreadLocalMap.set(ThreadLocal.java:539)
at java.base/java.lang.ThreadLocal.set(ThreadLocal.java:243)
at java.base/java.lang.ThreadLocal.set(ThreadLocal.java:229)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.set(origin.scala:93)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:105)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:495)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$3(TreeNode.scala:500)
at
org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1268)
at
org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1267)
at
org.apache.spark.sql.catalyst.expressions.UnaryExpression.mapChildren(Expression.scala:596)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:500)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$transformExpressionsDownWithPruning$1(QueryPlan.scala:223)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$1(QueryPlan.scala:264)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:107)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:264)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:276)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$4(QueryPlan.scala:282)
at scala.collection.immutable.List.map(List.scala:236)
at scala.collection.immutable.List.map(List.scala:79)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:282)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$5(QueryPlan.scala:287)
at
org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:337)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:287)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsDownWithPruning(QueryPlan.scala:223)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsWithPruning(QueryPlan.scala:194)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressions(QueryPlan.scala:172)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicate(DeduplicateRelations.scala:240)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicateAndRenew(DeduplicateRelations.scala:323)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.org$apache$spark$sql$catalyst$analysis$DeduplicateRelations$$renewDuplicatedRelations(DeduplicateRelations.scala:126)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1(DeduplicateRelations.scala:233)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1$adapted(DeduplicateRelations.scala:232)
at scala.collection.immutable.Vector.foreach(Vector.scala:2125)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicate(DeduplicateRelations.scala:232)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicateAndRenew(DeduplicateRelations.scala:323)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.org$apache$spark$sql$catalyst$analysis$DeduplicateRelations$$renewDuplicatedRelations(DeduplicateRelations.scala:126)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1(DeduplicateRelations.scala:233)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1$adapted(DeduplicateRelations.scala:232)
[...] {code}
What I've noticed during my debugs is that:
* The index of the next value, `C6530`, is `100` (zero-based).
* If I do a step over step debugging, without allowing my program to resume
from `C6525`, I can pursue through `C6530`, `C6535`... But as soon as do a
Resume program, the stack overflow happens.
So I guess it's an helper thread that starts to work on the request being
prepared that causes the trouble.
The problem didn't exist with `Spark 4.1.1` I was using before, with `Java 21`
then.
was:
{code:java}
codesCompetences = (size = 121)
0 = "C1005" 1 = "C1010" 2 = "C1015" 3 = "C1020" 4 = "C1025"
5 = "C1030" 6 = "C1035" 7 = "C1040" 8 = "C1045" 9 = "C1505"
10 = "C1510" 11 = "C1515" 12 = "C1520" 13 = "C2000" 14 = "C2005"
15 = "C2010" 16 = "C2015" 17 = "C2020" 18 = "C2025" 19 = "C2030"
20 = "C2035" 21 = "C2040" 22 = "C2045" 23 = "C2050" 24 = "C2055"
25 = "C2060" 26 = "C2065" 27 = "C2070" 28 = "C2075" 29 = "C2080"
30 = "C2085" 31 = "C2090" 32 = "C2095" 33 = "C2096" 34 = "C2500"
35 = "C2505" 36 = "C2510" 37 = "C3000" 38 = "C3005" 39 = "C3010"
40 = "C3015" 41 = "C3020" 42 = "C3025" 43 = "C3030" 44 = "C3035"
45 = "C3040" 46 = "C3045" 47 = "C3050" 48 = "C3055" 49 = "C3060"
50 = "C3500" 51 = "C4000" 52 = "C4005" 53 = "C4010" 54 = "C4500"
55 = "C4505" 56 = "C4510" 57 = "C4515" 58 = "C4520" 59 = "C5000"
60 = "C5005" 61 = "C5010" 62 = "C5015" 63 = "C5020" 64 = "C5025"
65 = "C5030" 66 = "C5035" 67 = "C5040" 68 = "C5045" 69 = "C5050"
70 = "C5500" 71 = "C5505" 72 = "C5510" 73 = "C5515" 74 = "C5520"
75 = "C5525" 76 = "C5530" 77 = "C5535" 78 = "C5540" 79 = "C6000"
80 = "C6005" 81 = "C6010" 82 = "C6015" 83 = "C6020" 84 = "C6025"
85 = "C6030" 86 = "C6035" 87 = "C6040" 88 = "C6045" 89 = "C6055"
90 = "C6060" 91 = "C6065" 92 = "C6070" 93 = "C6075" 94 = "C6500"
95 = "C6505" 96 = "C6510" 97 = "C6515" 98 = "C6520" 99 = "C6525"
100 = "C6530" 101 = "C6535" 102 = "C7000" 103 = "C7500" 104 = "C7505"
105 = "C7510" 106 = "C7515" 107 = "C7520" 108 = "C7525" 109 = "C7530"
110 = "C7535" 111 = "C7540" 112 = "C7541" 113 = "C7545" 114 = "C7546"
115 = "C7550" 116 = "C7555" 117 = "C7560" 118 = "C7565" 119 = "C7570"
120 = "C7575 {code}
I'm following a `String[]` of competencies codes, at their indexes :
With them, I'm converting values of columns coming from a CSV file:
{code:java}
private Dataset<Row>
extraireCompetencesDelegueesEtInteretCommunautaire(Dataset<Row> dataset, int
anneeCog) {
// Ces valeurs sont celles qu'un distinct() montre possibles sur les
colonnes du fichier csv d'entrée.
final String COMPETENCE_DELEGUEE = "délégué";
final String COMPETENCE_INTERET_COMMUNAUTAIRE = "avec intérêt
communautaire";
List<String> codesCompetences =
this.datasetCompetence.codesCompetencesAsList(anneeCog);
for(String codeCompetence : codesCompetences) {
Column colCompetence = new Column(codeCompetence);
// Créer le champ Cxxx_Deleguee et le one hot encoder en "1" ou "0" selon
que la valeur associée au code compétence vaut 'délégué' ou pas.
Column competenceDeleguee =
when(colCompetence.equalTo(COMPETENCE_DELEGUEE), lit("1")).otherwise(lit("0"));
dataset = dataset.withColumn(codeCompetence + "_Deleguee",
competenceDeleguee);
// Créer le champ Cxxx_InteretCommunautaire et le one hot encoder en "1"
ou "0" selon que la valeur associée au code compétence vaut 'avec intérêt
communautaire' ou pas.
Column competenceInteretCommunautaire =
when(colCompetence.equalTo(COMPETENCE_INTERET_COMMUNAUTAIRE),
lit("1")).otherwise(lit("0"));
dataset = dataset.withColumn(codeCompetence + "_InteretCommunautaire",
competenceInteretCommunautaire);
// one hot encoder la présence de la compétence
Column encoding = when(colCompetence.isin(OUI, COMPETENCE_DELEGUEE,
COMPETENCE_INTERET_COMMUNAUTAIRE), lit("1"))
.when(colCompetence.equalTo(NON), lit("0"))
.otherwise(colCompetence);
dataset = dataset.withColumn(codeCompetence, encoding);
LOGGER.info("added: '{}'", encoding);
}
return dataset;
}
{code}
This loop adds to the `Dataset` some columns each time,. Among them one is :
{code:java}
added: 'CASE WHEN in(C6525, 'OUI', 'délégué', 'avec intérêt communautaire')
THEN '1' WHEN =(C6525, 'NON') {code}
and it's the last log shown before a:
{code:java}
java.lang.StackOverflowError
at java.base/java.lang.ref.Reference.refersToImpl(Reference.java:384)
at java.base/java.lang.ref.Reference.refersTo(Reference.java:375)
at java.base/java.lang.ThreadLocal$ThreadLocalMap.set(ThreadLocal.java:539)
at java.base/java.lang.ThreadLocal.set(ThreadLocal.java:243)
at java.base/java.lang.ThreadLocal.set(ThreadLocal.java:229)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.set(origin.scala:93)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:105)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:495)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$3(TreeNode.scala:500)
at
org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1268)
at
org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1267)
at
org.apache.spark.sql.catalyst.expressions.UnaryExpression.mapChildren(Expression.scala:596)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:500)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$transformExpressionsDownWithPruning$1(QueryPlan.scala:223)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$1(QueryPlan.scala:264)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:107)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:264)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:276)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$4(QueryPlan.scala:282)
at scala.collection.immutable.List.map(List.scala:236)
at scala.collection.immutable.List.map(List.scala:79)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:282)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$5(QueryPlan.scala:287)
at
org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:337)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:287)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsDownWithPruning(QueryPlan.scala:223)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsWithPruning(QueryPlan.scala:194)
at
org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressions(QueryPlan.scala:172)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicate(DeduplicateRelations.scala:240)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicateAndRenew(DeduplicateRelations.scala:323)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.org$apache$spark$sql$catalyst$analysis$DeduplicateRelations$$renewDuplicatedRelations(DeduplicateRelations.scala:126)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1(DeduplicateRelations.scala:233)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1$adapted(DeduplicateRelations.scala:232)
at scala.collection.immutable.Vector.foreach(Vector.scala:2125)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicate(DeduplicateRelations.scala:232)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicateAndRenew(DeduplicateRelations.scala:323)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.org$apache$spark$sql$catalyst$analysis$DeduplicateRelations$$renewDuplicatedRelations(DeduplicateRelations.scala:126)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1(DeduplicateRelations.scala:233)
at
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1$adapted(DeduplicateRelations.scala:232)
[...] {code}
What I've noticed during my debugs is that:
* The index of the next value, `C6530`, is `100` (zero-based).
* If I do a step over step debugging, without allowing my program to resume
from `C6525`, I can pursue through `C6530`, `C6535`... But as soon as do a
Resume program, the stack overflow happens.
So I guess it's an helper thread that starts to work on the request being
prepared that causes the trouble.
The problem didn't exist with `Spark 4.1.1` I was using before, with `Java 21`
then.
> SQL Catalyst 4.2.0 causes a StackOverflow error when it attempt to optimize
> of a request it is preparing
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58318
> URL: https://issues.apache.org/jira/browse/SPARK-58318
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Environment: Debian 13
> Spark 4.2.0
> Java 25
> Reporter: Marc Le Bihan
> Priority: Major
>
> I'm following a `String[]` of competencies codes, at their indexes :
> {code:java}
> codesCompetences = (size = 121)
> 0 = "C1005" 1 = "C1010" 2 = "C1015" 3 = "C1020" 4 = "C1025"
> 5 = "C1030" 6 = "C1035" 7 = "C1040" 8 = "C1045" 9 = "C1505"
> 10 = "C1510" 11 = "C1515" 12 = "C1520" 13 = "C2000" 14 = "C2005"
> 15 = "C2010" 16 = "C2015" 17 = "C2020" 18 = "C2025" 19 = "C2030"
> 20 = "C2035" 21 = "C2040" 22 = "C2045" 23 = "C2050" 24 = "C2055"
> 25 = "C2060" 26 = "C2065" 27 = "C2070" 28 = "C2075" 29 = "C2080"
> 30 = "C2085" 31 = "C2090" 32 = "C2095" 33 = "C2096" 34 = "C2500"
> 35 = "C2505" 36 = "C2510" 37 = "C3000" 38 = "C3005" 39 = "C3010"
> 40 = "C3015" 41 = "C3020" 42 = "C3025" 43 = "C3030" 44 = "C3035"
> 45 = "C3040" 46 = "C3045" 47 = "C3050" 48 = "C3055" 49 = "C3060"
> 50 = "C3500" 51 = "C4000" 52 = "C4005" 53 = "C4010" 54 = "C4500"
> 55 = "C4505" 56 = "C4510" 57 = "C4515" 58 = "C4520" 59 = "C5000"
> 60 = "C5005" 61 = "C5010" 62 = "C5015" 63 = "C5020" 64 = "C5025"
> 65 = "C5030" 66 = "C5035" 67 = "C5040" 68 = "C5045" 69 = "C5050"
> 70 = "C5500" 71 = "C5505" 72 = "C5510" 73 = "C5515" 74 = "C5520"
> 75 = "C5525" 76 = "C5530" 77 = "C5535" 78 = "C5540" 79 = "C6000"
> 80 = "C6005" 81 = "C6010" 82 = "C6015" 83 = "C6020" 84 = "C6025"
> 85 = "C6030" 86 = "C6035" 87 = "C6040" 88 = "C6045" 89 = "C6055"
> 90 = "C6060" 91 = "C6065" 92 = "C6070" 93 = "C6075" 94 = "C6500"
> 95 = "C6505" 96 = "C6510" 97 = "C6515" 98 = "C6520" 99 = "C6525"
> 100 = "C6530" 101 = "C6535" 102 = "C7000" 103 = "C7500" 104 = "C7505"
> 105 = "C7510" 106 = "C7515" 107 = "C7520" 108 = "C7525" 109 = "C7530"
> 110 = "C7535" 111 = "C7540" 112 = "C7541" 113 = "C7545" 114 = "C7546"
> 115 = "C7550" 116 = "C7555" 117 = "C7560" 118 = "C7565" 119 = "C7570"
> 120 = "C7575 {code}
>
> With them, I'm converting values of columns coming from a CSV file:
> {code:java}
> private Dataset<Row>
> extraireCompetencesDelegueesEtInteretCommunautaire(Dataset<Row> dataset, int
> anneeCog) {
> // Ces valeurs sont celles qu'un distinct() montre possibles sur les
> colonnes du fichier csv d'entrée.
> final String COMPETENCE_DELEGUEE = "délégué";
> final String COMPETENCE_INTERET_COMMUNAUTAIRE = "avec intérêt
> communautaire";
> List<String> codesCompetences =
> this.datasetCompetence.codesCompetencesAsList(anneeCog);
> for(String codeCompetence : codesCompetences) {
> Column colCompetence = new Column(codeCompetence);
> // Créer le champ Cxxx_Deleguee et le one hot encoder en "1" ou "0"
> selon que la valeur associée au code compétence vaut 'délégué' ou pas.
> Column competenceDeleguee =
> when(colCompetence.equalTo(COMPETENCE_DELEGUEE),
> lit("1")).otherwise(lit("0"));
> dataset = dataset.withColumn(codeCompetence + "_Deleguee",
> competenceDeleguee);
> // Créer le champ Cxxx_InteretCommunautaire et le one hot encoder en
> "1" ou "0" selon que la valeur associée au code compétence vaut 'avec intérêt
> communautaire' ou pas.
> Column competenceInteretCommunautaire =
> when(colCompetence.equalTo(COMPETENCE_INTERET_COMMUNAUTAIRE),
> lit("1")).otherwise(lit("0"));
> dataset = dataset.withColumn(codeCompetence + "_InteretCommunautaire",
> competenceInteretCommunautaire);
> // one hot encoder la présence de la compétence
> Column encoding = when(colCompetence.isin(OUI, COMPETENCE_DELEGUEE,
> COMPETENCE_INTERET_COMMUNAUTAIRE), lit("1"))
> .when(colCompetence.equalTo(NON), lit("0"))
> .otherwise(colCompetence);
> dataset = dataset.withColumn(codeCompetence, encoding);
> LOGGER.info("added: '{}'", encoding);
> }
> return dataset;
> }
> {code}
> This loop adds to the `Dataset` some columns each time,. Among them one is :
>
> {code:java}
> added: 'CASE WHEN in(C6525, 'OUI', 'délégué', 'avec intérêt communautaire')
> THEN '1' WHEN =(C6525, 'NON') {code}
> and it's the last log shown before a:
>
> {code:java}
> java.lang.StackOverflowError
> at java.base/java.lang.ref.Reference.refersToImpl(Reference.java:384)
> at java.base/java.lang.ref.Reference.refersTo(Reference.java:375)
> at
> java.base/java.lang.ThreadLocal$ThreadLocalMap.set(ThreadLocal.java:539)
> at java.base/java.lang.ThreadLocal.set(ThreadLocal.java:243)
> at java.base/java.lang.ThreadLocal.set(ThreadLocal.java:229)
> at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.set(origin.scala:93)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:105)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:495)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$3(TreeNode.scala:500)
> at
> org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1268)
> at
> org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1267)
> at
> org.apache.spark.sql.catalyst.expressions.UnaryExpression.mapChildren(Expression.scala:596)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:500)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$transformExpressionsDownWithPruning$1(QueryPlan.scala:223)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$1(QueryPlan.scala:264)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:107)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:264)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:276)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$4(QueryPlan.scala:282)
> at scala.collection.immutable.List.map(List.scala:236)
> at scala.collection.immutable.List.map(List.scala:79)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:282)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$5(QueryPlan.scala:287)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:337)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:287)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsDownWithPruning(QueryPlan.scala:223)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsWithPruning(QueryPlan.scala:194)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressions(QueryPlan.scala:172)
> at
> org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicate(DeduplicateRelations.scala:240)
> at
> org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicateAndRenew(DeduplicateRelations.scala:323)
> at
> org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.org$apache$spark$sql$catalyst$analysis$DeduplicateRelations$$renewDuplicatedRelations(DeduplicateRelations.scala:126)
> at
> org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1(DeduplicateRelations.scala:233)
> at
> org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1$adapted(DeduplicateRelations.scala:232)
> at scala.collection.immutable.Vector.foreach(Vector.scala:2125)
> at
> org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicate(DeduplicateRelations.scala:232)
> at
> org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.deduplicateAndRenew(DeduplicateRelations.scala:323)
> at
> org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.org$apache$spark$sql$catalyst$analysis$DeduplicateRelations$$renewDuplicatedRelations(DeduplicateRelations.scala:126)
> at
> org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1(DeduplicateRelations.scala:233)
> at
> org.apache.spark.sql.catalyst.analysis.DeduplicateRelations$.$anonfun$deduplicate$1$adapted(DeduplicateRelations.scala:232)
> [...] {code}
>
> What I've noticed during my debugs is that:
> * The index of the next value, `C6530`, is `100` (zero-based).
>
> * If I do a step over step debugging, without allowing my program to resume
> from `C6525`, I can pursue through `C6530`, `C6535`... But as soon as do a
> Resume program, the stack overflow happens.
>
> So I guess it's an helper thread that starts to work on the request being
> prepared that causes the trouble.
> The problem didn't exist with `Spark 4.1.1` I was using before, with `Java
> 21` then.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
