Repository: spark Updated Branches: refs/heads/master 7a5fd4a91 -> c44eb561e
[SPARK-24768][FOLLOWUP][SQL] Avro migration followup: change artifactId to spark-avro ## What changes were proposed in this pull request? After rethinking on the artifactId, I think it should be `spark-avro` instead of `spark-sql-avro`, which is simpler, and consistent with the previous artifactId. I think we need to change it before Spark 2.4 release. Also a tiny change: use `spark.sessionState.newHadoopConf()` to get the hadoop configuration, thus the related hadoop configurations in SQLConf will come into effect. ## How was this patch tested? Unit test Author: Gengliang Wang <gengliang.w...@databricks.com> Closes #21866 from gengliangwang/avro_followup. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c44eb561 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c44eb561 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c44eb561 Branch: refs/heads/master Commit: c44eb561ec371af0405710d2e9358f9797655145 Parents: 7a5fd4a Author: Gengliang Wang <gengliang.w...@databricks.com> Authored: Wed Jul 25 08:42:45 2018 -0700 Committer: Xiao Li <gatorsm...@gmail.com> Committed: Wed Jul 25 08:42:45 2018 -0700 ---------------------------------------------------------------------- external/avro/pom.xml | 2 +- .../src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/c44eb561/external/avro/pom.xml ---------------------------------------------------------------------- diff --git a/external/avro/pom.xml b/external/avro/pom.xml index ad7df1f..8f118ba 100644 --- a/external/avro/pom.xml +++ b/external/avro/pom.xml @@ -25,7 +25,7 @@ <relativePath>../../pom.xml</relativePath> </parent> - <artifactId>spark-sql-avro_2.11</artifactId> + <artifactId>spark-avro_2.11</artifactId> <properties> <sbt.project.name>avro</sbt.project.name> </properties> http://git-wip-us.apache.org/repos/asf/spark/blob/c44eb561/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala ---------------------------------------------------------------------- diff --git a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala index b043252..c6b3c13 100755 --- a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala +++ b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala @@ -56,7 +56,7 @@ private[avro] class AvroFileFormat extends FileFormat with DataSourceRegister { spark: SparkSession, options: Map[String, String], files: Seq[FileStatus]): Option[StructType] = { - val conf = spark.sparkContext.hadoopConfiguration + val conf = spark.sessionState.newHadoopConf() val parsedOptions = new AvroOptions(options, conf) // Schema evolution is not supported yet. Here we only pick a single random sample file to --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org