[ https://issues.apache.org/jira/browse/SPARK-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280013#comment-15280013 ]
Nick Pentreath edited comment on SPARK-13959 at 5/11/16 11:56 AM: ------------------------------------------------------------------ All {{ML}} potential errors are related to {{DataFrame}} -> {{Dataset}} change: {code} [info] spark-mllib: found 4 potential binary incompatibilities while checking against org.apache.spark:spark-mllib_2.11:1.6.0 (filtered 151) [error] * method transform(org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame in class org.apache.spark.ml.UnaryTransformer's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.sql.Dataset instead of (org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame [error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.UnaryTransformer.transform") [error] * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.DecisionTreeClassificationModel in class org.apache.spark.ml.classification.DecisionTreeClassifier's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.DecisionTreeClassificationModel [error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.classification.DecisionTreeClassifier.train") [error] * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.LogisticRegressionModel in class org.apache.spark.ml.classification.LogisticRegression's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.LogisticRegressionModel [error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.classification.LogisticRegression.train") [error] * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.regression.DecisionTreeRegressionModel in class org.apache.spark.ml.regression.DecisionTreeRegressor's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.regression.DecisionTreeRegressionModel [error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.regression.DecisionTreeRegressor.train") {code} It's weird that the above errors for {{DecisionTreeClassifier.train}}, {{DecisionTreeRegressor.train}} and {{LogisticRegression.train}} refer to a return type of {{PredictionModel}} in the "current version". But in any case I verified the return types have not changed for those methods was (Author: mlnick): All {{ML}} potential errors are related to {{DataFrame}} -> {{Dataset}} change: {code} [info] spark-mllib: found 4 potential binary incompatibilities while checking against org.apache.spark:spark-mllib_2.11:1.6.0 (filtered 151) [error] * method transform(org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame in class org.apache.spark.ml.UnaryTransformer's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.sql.Dataset instead of (org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame [error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.UnaryTransformer.transform") [error] * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.DecisionTreeClassificationModel in class org.apache.spark.ml.classification.DecisionTreeClassifier's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.DecisionTreeClassificationModel [error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.classification.DecisionTreeClassifier.train") [error] * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.LogisticRegressionModel in class org.apache.spark.ml.classification.LogisticRegression's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.classification.LogisticRegressionModel [error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.classification.LogisticRegression.train") [error] * method train(org.apache.spark.sql.DataFrame)org.apache.spark.ml.regression.DecisionTreeRegressionModel in class org.apache.spark.ml.regression.DecisionTreeRegressor's type is different in current version, where it is (org.apache.spark.sql.Dataset)org.apache.spark.ml.PredictionModel instead of (org.apache.spark.sql.DataFrame)org.apache.spark.ml.regression.DecisionTreeRegressionModel [error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.regression.DecisionTreeRegressor.train") {code} It's weird that the above errors for {{DecisionTreeClassifier.train}}, {{DecisionTreeRegressor.train}} and {{LogisticRegression.train}} refer to {{PredictionModel}} in the "current version". But in any case I verified the return types have not changed for those methods > Audit MiMa excludes added in SPARK-13948 to make sure none are unintended > incompatibilities > ------------------------------------------------------------------------------------------- > > Key: SPARK-13959 > URL: https://issues.apache.org/jira/browse/SPARK-13959 > Project: Spark > Issue Type: Task > Components: Build > Reporter: Josh Rosen > Priority: Critical > > The patch for SPARK-13948 added a number of MiMa excludes for cases which > were missed due to our old way of programatically generating excludes in > GenerateMIMAIgnore. > Before Spark 2.0, we need to audit the additional excludes that I added to > make sure that none represent unintentional incompatibilities which should be > fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org