[GitHub] spark pull request #22977: [BUILD] Bump previousSparkVersion in MimaBuild.sc...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22977#discussion_r232456139 --- Diff: project/MimaExcludes.scala --- @@ -84,7 +84,17 @@ object MimaExcludes { ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.streaming.api.java.JavaPairDStream.flatMapValues"), // [SPARK-25680] SQL execution listener shouldn't happen on execution thread ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.util.ExecutionListenerManager.clone"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.util.ExecutionListenerManager.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.util.ExecutionListenerManager.this"), +// [SPARK-25862][SQL] Remove rangeBetween APIs introduced in SPARK-21608 + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.functions.unboundedFollowing"), + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.functions.unboundedPreceding"), + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.functions.currentRow"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.expressions.Window.rangeBetween"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.expressions.WindowSpec.rangeBetween"), +// [SPARK-23781][CORE] Merge token renewer functionality into HadoopDelegationTokenManager + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkHadoopUtil.nextCredentialRenewalTime"), +// [SPARK-25908][CORE][SQL] Remove old deprecated items in Spark 3 --- End diff -- @cloud-fan really these two lines should move to the top, because the first long chunk of exclusions is for 25908. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22977: [BUILD] Bump previousSparkVersion in MimaBuild.sc...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22977#discussion_r232005093 --- Diff: project/MimaBuild.scala --- @@ -88,7 +88,7 @@ object MimaBuild { def mimaSettings(sparkHome: File, projectRef: ProjectRef) = { val organization = "org.apache.spark" -val previousSparkVersion = "2.2.0" +val previousSparkVersion = "2.4.0" --- End diff -- Thanks, @cloud-fan and @srowen . +1 for another PR for `branch-2.4`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22977: [BUILD] Bump previousSparkVersion in MimaBuild.sc...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22977#discussion_r231935977 --- Diff: project/MimaBuild.scala --- @@ -88,7 +88,7 @@ object MimaBuild { def mimaSettings(sparkHome: File, projectRef: ProjectRef) = { val organization = "org.apache.spark" -val previousSparkVersion = "2.2.0" +val previousSparkVersion = "2.4.0" --- End diff -- I believe @cloud-fan is right here. If master is 3.0, the previous version is 2.4.0 in master. Analogously for branch 2.4. Yes sounds like this should be a step in the release process somewhere. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22977: [BUILD] Bump previousSparkVersion in MimaBuild.sc...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22977#discussion_r231936644 --- Diff: project/MimaExcludes.scala --- @@ -84,7 +84,17 @@ object MimaExcludes { ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.streaming.api.java.JavaPairDStream.flatMapValues"), // [SPARK-25680] SQL execution listener shouldn't happen on execution thread ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.util.ExecutionListenerManager.clone"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.util.ExecutionListenerManager.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.util.ExecutionListenerManager.this"), +// [SPARK-25862][SQL] Remove rangeBetween APIs introduced in SPARK-21608 + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.functions.unboundedFollowing"), + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.functions.unboundedPreceding"), + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.functions.currentRow"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.expressions.Window.rangeBetween"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.expressions.WindowSpec.rangeBetween"), +// [SPARK-23781][CORE] Merge token renewer functionality into HadoopDelegationTokenManager + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkHadoopUtil.nextCredentialRenewalTime"), --- End diff -- It was a `private[spark]` method which has to be implemented as a public method in the Java bytecode. MiMa doesn't know the difference. Yes it can be ignore for sure. So these failures are 'new' because they were introduced after Spark 2.2 (hence not missing since 2.2) then removed, so bumping the previous version causes them to fail now? OK. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22977: [BUILD] Bump previousSparkVersion in MimaBuild.sc...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22977#discussion_r231872904 --- Diff: project/MimaExcludes.scala --- @@ -84,7 +84,17 @@ object MimaExcludes { ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.streaming.api.java.JavaPairDStream.flatMapValues"), // [SPARK-25680] SQL execution listener shouldn't happen on execution thread ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.util.ExecutionListenerManager.clone"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.util.ExecutionListenerManager.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.util.ExecutionListenerManager.this"), +// [SPARK-25862][SQL] Remove rangeBetween APIs introduced in SPARK-21608 + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.functions.unboundedFollowing"), + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.functions.unboundedPreceding"), + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.functions.currentRow"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.expressions.Window.rangeBetween"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.expressions.WindowSpec.rangeBetween"), +// [SPARK-23781][CORE] Merge token renewer functionality into HadoopDelegationTokenManager + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkHadoopUtil.nextCredentialRenewalTime"), --- End diff -- This is actually a private method, I'm not sure why mima tracks it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22977: [BUILD] Bump previousSparkVersion in MimaBuild.sc...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22977#discussion_r231866958 --- Diff: project/MimaBuild.scala --- @@ -88,7 +88,7 @@ object MimaBuild { def mimaSettings(sparkHome: File, projectRef: ProjectRef) = { val organization = "org.apache.spark" -val previousSparkVersion = "2.2.0" +val previousSparkVersion = "2.4.0" --- End diff -- up to my understanding, we should have changed it to 2.3.0 when 2.3.0 was released. Maybe we should send a another PR to branch-2.4 and change it to 2.3.0 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22977: [BUILD] Bump previousSparkVersion in MimaBuild.sc...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22977#discussion_r231860289 --- Diff: project/MimaBuild.scala --- @@ -88,7 +88,7 @@ object MimaBuild { def mimaSettings(sparkHome: File, projectRef: ProjectRef) = { val organization = "org.apache.spark" -val previousSparkVersion = "2.2.0" +val previousSparkVersion = "2.4.0" --- End diff -- According to https://issues.apache.org/jira/browse/SPARK-23070, `2.3.0` seems to be the correct version in this PR. cc @gatorsmile . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22977: [BUILD] Bump previousSparkVersion in MimaBuild.sc...
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/22977 [BUILD] Bump previousSparkVersion in MimaBuild.scala to be 2.4.0 ## What changes were proposed in this pull request? Since Spark 2.4.0 is already in maven repo, we can Bump previousSparkVersion in MimaBuild.scala to be 2.4.0. Note that, seems we forgot to do it for 2.4.0, so this PR also updates MimaExcludes.scala ## How was this patch tested? N/A You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloud-fan/spark mima Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22977.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22977 commit a0357782e75967d309dc5229b3c36a0c295f2956 Author: Wenchen Fan Date: 2018-11-08T11:34:31Z Bump previousSparkVersion in MimaBuild.scala to be 2.2.0 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org