This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
     new 9b0971c  [SPARK-35990][BUILD] Remove avro-sbt plugin dependency
9b0971c is described below

commit 9b0971c40ebbc5309f6684edfef3f7e219c9a4b2
Author: Kousuke Saruta <saru...@oss.nttdata.com>
AuthorDate: Fri Jul 2 11:00:59 2021 -0700

    [SPARK-35990][BUILD] Remove avro-sbt plugin dependency
    
    ### What changes were proposed in this pull request?
    
    This PR removes sbt-avro plugin dependency.
    In the current master, Build with SBT depends on the plugin but it seems 
never used.
    Originally, the plugin was introduced for `flume-sink` in SPARK-1729 (#807) 
but `flume-sink` is no longer in Spark repository.
    
    After SBT was upgraded to 1.x in SPARK-21708 (#29286), `avroGenerate` part 
was introduced in `object SQL` in `SparkBuild.scala`.
    It's confusable but I understand `Test / avroGenerate := (Compile / 
avroGenerate).value` is for suppressing sbt-avro for `sql` sub-module.
    In fact, Test/compile will fail if `Test / avroGenerate :=(Compile / 
avroGenerate).value` is commented out.
    
    `sql` sub-module contains `parquet-compat.avpr` and `parquet-compat.avdl` 
but according to `sql/core/src/test/README.md`, they are intended to be handled 
by `gen-avro.sh`.
    
    Also, in terms of Maven build, there seems to be no definition to handle 
`*.avpr` or `*.avdl`.
    
    Based on the above, I think we can remove `sbt-avro`.
    
    ### Why are the changes needed?
    
    If `sbt-avro` is really no longer used, it's confusable that `sbt-avro` 
related configurations are in `SparkBuild.scala`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    GA.
    
    Closes #33190 from sarutak/remove-avro-from-sbt.
    
    Authored-by: Kousuke Saruta <saru...@oss.nttdata.com>
    Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
    (cherry picked from commit 6c4616b2ac75f30db63c49ab91a784285b147b11)
    Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
---
 project/SparkBuild.scala | 7 +------
 project/plugins.sbt      | 3 ---
 2 files changed, 1 insertion(+), 9 deletions(-)

diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index b1d6668..3480f4f 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -737,9 +737,6 @@ object Catalyst {
 }
 
 object SQL {
-
-  import sbtavro.SbtAvro.autoImport._
-
   lazy val settings = Seq(
     (console / initialCommands) :=
       """
@@ -761,10 +758,8 @@ object SQL {
         |import sqlContext.implicits._
         |import sqlContext._
       """.stripMargin,
-    (console / cleanupCommands) := "sc.stop()",
-    Test / avroGenerate := (Compile / avroGenerate).value
+    (console / cleanupCommands) := "sc.stop()"
   )
-
 }
 
 object Hive {
diff --git a/project/plugins.sbt b/project/plugins.sbt
index 8130ed0..e733f86 100644
--- a/project/plugins.sbt
+++ b/project/plugins.sbt
@@ -33,9 +33,6 @@ addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.9.2")
 
 addSbtPlugin("com.eed3si9n" % "sbt-unidoc" % "0.4.3")
 
-addSbtPlugin("com.cavorite" % "sbt-avro" % "2.1.1")
-libraryDependencies += "org.apache.avro" % "avro-compiler" % "1.10.2"
-
 addSbtPlugin("io.spray" % "sbt-revolver" % "0.9.1")
 
 libraryDependencies += "org.ow2.asm"  % "asm" % "9.1"

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

Reply via email to