[ https://issues.apache.org/jira/browse/SPARK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596110#comment-16596110 ]
Nikita Poberezkin commented on SPARK-25102: ------------------------------------------- Hello, [~zi] I've tried this approach: import org.apache.spark.\{SPARK_VERSION} override def getName: String = { SPARK_VERSION } Spark is building with this code. Also I have an idea to add configuration to SQLConf which was already imported to ParquetWriteSupport. Code will look something like this, below is config added to SQLConf: val SPARK_VERSION = buildConf("spark.sql.writerModelName") .doc("Version of Spark which create Parquet file") .stringConf .createWithDefault(SparkContext.getOrCreate().version) And method getName: override def getName: String = { SQLConf.SPARK_VERSION.key } I've created pull request with the first variant. Which one is better (if any of them is valid at all)? > Write Spark version information to Parquet file footers > ------------------------------------------------------- > > Key: SPARK-25102 > URL: https://issues.apache.org/jira/browse/SPARK-25102 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.3.1 > Reporter: Zoltan Ivanfi > Priority: Major > > -PARQUET-352- added support for the "writer.model.name" property in the > Parquet metadata to identify the object model (application) that wrote the > file. > The easiest way to write this property is by overriding getName() of > org.apache.parquet.hadoop.api.WriteSupport. In Spark, this would mean adding > getName() to the > org.apache.spark.sql.execution.datasources.parquet.ParquetWriteSupport class. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org