[ https://issues.apache.org/jira/browse/SPARK-24744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533429#comment-16533429 ]
Apache Spark commented on SPARK-24744: -------------------------------------- User 'bjkonglu' has created a pull request for this issue: https://github.com/apache/spark/pull/21718 > Structured Streaming set SparkSession configuration with the value in the > metadata if there is not a option set by user. > --------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-24744 > URL: https://issues.apache.org/jira/browse/SPARK-24744 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 2.3.1 > Reporter: bjkonglu > Priority: Minor > > h3. Background > When I use structured streaming to construct my application, there is > something odd! The application always set option > [spark.sql.shuffle.partitions] to default value [200]. Even though, I set > [spark.sql.shuffle.partitions] to other value by SparkConf or --conf > spark.sql.shuffle.partitions=100, but it doesn't work. The option value is > default value as before. > h3. Analyse > I review the relevant code. The relevant code is in > [org.apache.spark.sql.execution.streaming.OffsetSeqMetadata]. > {code:scala} > /** Set the SparkSession configuration with the values in the metadata */ > def setSessionConf(metadata: OffsetSeqMetadata, sessionConf: > RuntimeConfig): Unit = { > OffsetSeqMetadata.relevantSQLConfs.map(_.key).foreach { confKey => > metadata.conf.get(confKey) match { > case Some(valueInMetadata) => > // Config value exists in the metadata, update the session config > with this value > val optionalValueInSession = sessionConf.getOption(confKey) > if (optionalValueInSession.isDefined && optionalValueInSession.get > != valueInMetadata) { > logWarning(s"Updating the value of conf '$confKey' in current > session from " + > s"'${optionalValueInSession.get}' to '$valueInMetadata'.") > } > sessionConf.set(confKey, valueInMetadata) > case None => > // For backward compatibility, if a config was not recorded in the > offset log, > // then log it, and let the existing conf value in SparkSession > prevail. > logWarning (s"Conf '$confKey' was not found in the offset log, > using existing value") > } > } > } > {code} > In this code, we can find it always set some option in metadata value. But as > user, we want to those option can set by user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org