Dong Wang created SPARK-29824: --------------------------------- Summary: Missing persist on trainDataset in ml.classification.GBTClassifier.train() Key: SPARK-29824 URL: https://issues.apache.org/jira/browse/SPARK-29824 Project: Spark Issue Type: Sub-task Components: ML Affects Versions: 2.4.3 Reporter: Dong Wang
The rdd trainDataset in ml.classification.GBTClassifier.train() is used by an action first and other actions in GradientBoostedTrees.run/runWithValidation, but it is not persisted, which will cause recomputation on trainDataset. {code:scala} override protected def train( dataset: Dataset[_]): GBTClassificationModel = instrumented { instr => val categoricalFeatures: Map[Int, Int] = MetadataUtils.getCategoricalFeatures(dataset.schema($(featuresCol))) ... val numFeatures = trainDataset.first().features.size // first use trainDataset ... // trainDataset will be used by other actions in run methods. val (baseLearners, learnerWeights) = if (withValidation) { GradientBoostedTrees.runWithValidation(trainDataset, validationDataset, boostingStrategy, $(seed), $(featureSubsetStrategy)) } else { GradientBoostedTrees.run(trainDataset, boostingStrategy, $(seed), $(featureSubsetStrategy)) } {code} This issue is reported by our tool CacheCheck, which is used to dynamically detecting persist()/unpersist() api misuses. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org