[ https://issues.apache.org/jira/browse/SPARK-35508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lior Chaga updated SPARK-35508: ------------------------------- Attachment: spark3-image.png > job group and description do not apply on broadcasts > ---------------------------------------------------- > > Key: SPARK-35508 > URL: https://issues.apache.org/jira/browse/SPARK-35508 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.0.0, 3.1.0 > Reporter: Lior Chaga > Priority: Minor > Attachments: spark2-image.png, spark3-image.png > > > Given the following code: > {code:java} > SparkContext context = new SparkContext("local", "test"); > SparkSession session = new SparkSession(context); > List<String> strings = Lists.newArrayList("a", "b", "c"); > List<String> otherString = Lists.newArrayList( "b", "c", "d"); > Dataset<Row> broadcastedDf = session.createDataset(strings, > Encoders.STRING()).toDF(); > Dataset<Row> dataframe = session.createDataset(otherString, > Encoders.STRING()).toDF(); > context.setJobGroup("my group", "my job", false); > dataframe.join(broadcast(broadcastedDf), "value").count(); > {code} > Job group and description do not apply on broadcasted dataframe. > With spark 2.x, broadcast creation is given the same job description as the > query itself. > This seems to be broken with spark 3.x > See attached images > !image-2021-05-25-09-39-36-816.png! > !image-2021-05-25-09-40-12-210.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org