[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15415 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r103068598
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation. Note null values in the feature column are
ignored during fit().
+ *
+ * @see http://en.wikipedia.org/wiki/Association_rule_learning;>
+ * Association rule learning
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r103068619
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation. Note null values in the feature column are
ignored during fit().
+ *
+ * @see http://en.wikipedia.org/wiki/Association_rule_learning;>
+ * Association rule learning
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r103050909
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation. Note null values in the feature column are
ignored during fit().
+ *
+ * @see http://en.wikipedia.org/wiki/Association_rule_learning;>
+ * Association rule learning
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102860331
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation.
+ *
+ * @see http://en.wikipedia.org/wiki/Association_rule_learning;>
+ * Association rule learning (Wikipedia)
+ */
+@Since("2.2.0")
+@Experimental
+class
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102856964
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation.
+ *
+ * @see http://en.wikipedia.org/wiki/Association_rule_learning;>
+ * Association rule learning (Wikipedia)
+ */
+@Since("2.2.0")
+@Experimental
+class
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102856168
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation.
+ *
+ * @see http://en.wikipedia.org/wiki/Association_rule_learning;>
+ * Association rule learning (Wikipedia)
+ */
+@Since("2.2.0")
+@Experimental
+class
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102855117
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation.
+ *
+ * @see http://en.wikipedia.org/wiki/Association_rule_learning;>
+ * Association rule learning (Wikipedia)
+ */
+@Since("2.2.0")
+@Experimental
+class
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102845724
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation.
+ *
+ * @see http://en.wikipedia.org/wiki/Association_rule_learning;>
+ * Association rule learning (Wikipedia)
+ */
+@Since("2.2.0")
+@Experimental
+class
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102845588
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation.
+ *
+ * @see http://en.wikipedia.org/wiki/Association_rule_learning;>
+ * Association rule learning (Wikipedia)
+ */
+@Since("2.2.0")
+@Experimental
+class
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102840184
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation.
+ *
+ * @see http://en.wikipedia.org/wiki/Association_rule_learning;>
+ * Association rule learning (Wikipedia)
+ */
+@Since("2.2.0")
+@Experimental
+class
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102840479
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation.
+ *
+ * @see http://en.wikipedia.org/wiki/Association_rule_learning;>
+ * Association rule learning (Wikipedia)
+ */
+@Since("2.2.0")
+@Experimental
+class
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102792088
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation.
+ *
+ * @see http://en.wikipedia.org/wiki/Association_rule_learning;>
+ * Association rule learning (Wikipedia)
+ */
+@Since("2.2.0")
+@Experimental
+class
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102792306
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation.
+ *
+ * @see http://en.wikipedia.org/wiki/Association_rule_learning;>
+ * Association rule learning (Wikipedia)
+ */
+@Since("2.2.0")
+@Experimental
+class
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102792331
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of a frequent pattern",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets. The algorithm
is described in
+ * http://dx.doi.org/10.1145/1454008.1454027;>Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation. PFP distributes computation in such a way that each
worker executes an
+ * independent group of mining tasks. The FP-Growth algorithm is described
in
+ * http://dx.doi.org/10.1145/335191.335372;>Han et al., Mining
frequent patterns without
+ * candidate generation.
+ *
--- End diff --
Here or elsewhere, comment that null featuresCol values are ignored during
fit() and are treated as empty sets during transform().
---
If your
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102647844
--- Diff: mllib/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala
---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.ml.fpm
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.ml.util.DefaultReadWriteTest
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+class FPGrowthSuite extends SparkFunSuite with MLlibTestSparkContext with
DefaultReadWriteTest {
+
+ @transient var dataset: Dataset[_] = _
+
+ override def beforeAll(): Unit = {
+super.beforeAll()
+dataset = FPGrowthSuite.getFPGrowthData(spark)
+ }
+
+ test("FPGrowth fit and transform with different data types") {
+Array(IntegerType, StringType, ShortType, LongType, ByteType).foreach
{ dt =>
+ val intData = dataset.withColumn("features",
col("features").cast(ArrayType(dt)))
+ val model = new FPGrowth().setMinSupport(0.5).fit(intData)
+ val generatedRules = model.setMinConfidence(0.5).getAssociationRules
+ val expectedRules = spark.createDataFrame(Seq(
+(Array("2"), Array("1"), 1.0),
+(Array("1"), Array("2"), 0.75)
+ )).toDF("antecedent", "consequent", "confidence")
+.withColumn("antecedent", col("antecedent").cast(ArrayType(dt)))
+.withColumn("consequent", col("consequent").cast(ArrayType(dt)))
+ assert(expectedRules.sort("antecedent").rdd.collect().sameElements(
+generatedRules.sort("antecedent").rdd.collect()))
+
+ val transformed = model.transform(intData)
+ val expectedTransformed = spark.createDataFrame(Seq(
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "3"), Array(2))
+ )).toDF("id", "features", "prediction")
+.withColumn("features", col("features").cast(ArrayType(dt)))
+.withColumn("prediction", col("prediction").cast(ArrayType(dt)))
+ assert(expectedTransformed.sort("id").rdd.collect().sameElements(
+transformed.sort("id").rdd.collect()))
+}
+ }
+
+ test("FPGrowth getFreqItems") {
+val model = new FPGrowth().setMinSupport(0.7).fit(dataset)
+val expectedFreq = spark.createDataFrame(Seq(
+ (Array("1"), 4L),
+ (Array("2"), 3L),
+ (Array("1", "2"), 3L),
+ (Array("2", "1"), 3L)
--- End diff --
As the sequence of the frequent items is not guaranteed, a little hack..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102646065
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102598489
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102598758
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102535175
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102594866
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102535113
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
--- End diff --
"of the frequent pattern" -> "of a frequent pattern"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102599019
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102595600
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102535118
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
--- End diff --
Don't state default value in built-in Param doc
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102535156
--- Diff: mllib/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala
---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.ml.fpm
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.ml.util.DefaultReadWriteTest
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+class FPGrowthSuite extends SparkFunSuite with MLlibTestSparkContext with
DefaultReadWriteTest {
+
+ @transient var dataset: Dataset[_] = _
+
+ override def beforeAll(): Unit = {
+super.beforeAll()
+dataset = FPGrowthSuite.getFPGrowthData(spark)
+ }
+
+ test("FPGrowth fit and transform with different data types") {
+Array(IntegerType, StringType, ShortType, LongType, ByteType).foreach
{ dt =>
+ val intData = dataset.withColumn("features",
col("features").cast(ArrayType(dt)))
+ val model = new FPGrowth().setMinSupport(0.5).fit(intData)
+ val generatedRules = model.setMinConfidence(0.5).getAssociationRules
+ val expectedRules = spark.createDataFrame(Seq(
+(Array("2"), Array("1"), 1.0),
+(Array("1"), Array("2"), 0.75)
+ )).toDF("antecedent", "consequent", "confidence")
+.withColumn("antecedent", col("antecedent").cast(ArrayType(dt)))
+.withColumn("consequent", col("consequent").cast(ArrayType(dt)))
+ assert(expectedRules.sort("antecedent").rdd.collect().sameElements(
+generatedRules.sort("antecedent").rdd.collect()))
+
+ val transformed = model.transform(intData)
+ val expectedTransformed = spark.createDataFrame(Seq(
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "3"), Array(2))
+ )).toDF("id", "features", "prediction")
+.withColumn("features", col("features").cast(ArrayType(dt)))
+.withColumn("prediction", col("prediction").cast(ArrayType(dt)))
+ assert(expectedTransformed.sort("id").rdd.collect().sameElements(
+transformed.sort("id").rdd.collect()))
+}
+ }
+
+ test("FPGrowth getFreqItems") {
+val model = new FPGrowth().setMinSupport(0.7).fit(dataset)
+val expectedFreq = spark.createDataFrame(Seq(
+ (Array("1"), 4L),
+ (Array("2"), 3L),
+ (Array("1", "2"), 3L),
+ (Array("2", "1"), 3L)
+)).toDF("items", "freqExp")
+val freqItems = model.getFreqItemsets
+
+val checkDF = freqItems.join(expectedFreq, "items")
+assert(checkDF.count() == 3 && checkDF.filter(col("freq") ===
col("freqExp")).count() == 3)
+ }
+
+ test("FPGrowth getFreqItems with Null") {
--- End diff --
Btw, I could imagine us wanting to change this later. If we're
recommending items a user could add to their basket, then we might want to
suggest the most frequent item rather than nothing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102598955
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102598078
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102598022
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102599552
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102535137
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102535131
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102598249
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102535190
--- Diff: mllib/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala
---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.ml.fpm
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.ml.util.DefaultReadWriteTest
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+class FPGrowthSuite extends SparkFunSuite with MLlibTestSparkContext with
DefaultReadWriteTest {
+
+ @transient var dataset: Dataset[_] = _
+
+ override def beforeAll(): Unit = {
+super.beforeAll()
+dataset = FPGrowthSuite.getFPGrowthData(spark)
+ }
+
+ test("FPGrowth fit and transform with different data types") {
+Array(IntegerType, StringType, ShortType, LongType, ByteType).foreach
{ dt =>
+ val intData = dataset.withColumn("features",
col("features").cast(ArrayType(dt)))
+ val model = new FPGrowth().setMinSupport(0.5).fit(intData)
+ val generatedRules = model.setMinConfidence(0.5).getAssociationRules
+ val expectedRules = spark.createDataFrame(Seq(
+(Array("2"), Array("1"), 1.0),
+(Array("1"), Array("2"), 0.75)
+ )).toDF("antecedent", "consequent", "confidence")
+.withColumn("antecedent", col("antecedent").cast(ArrayType(dt)))
+.withColumn("consequent", col("consequent").cast(ArrayType(dt)))
+ assert(expectedRules.sort("antecedent").rdd.collect().sameElements(
+generatedRules.sort("antecedent").rdd.collect()))
+
+ val transformed = model.transform(intData)
+ val expectedTransformed = spark.createDataFrame(Seq(
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "3"), Array(2))
+ )).toDF("id", "features", "prediction")
+.withColumn("features", col("features").cast(ArrayType(dt)))
+.withColumn("prediction", col("prediction").cast(ArrayType(dt)))
+ assert(expectedTransformed.sort("id").rdd.collect().sameElements(
--- End diff --
I'd collect and convert toSet, rather than calling RDD.sort, since it is a
bit faster.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102598358
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102599281
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102535103
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
--- End diff --
Could you please go ahead and copy the relevant text and links from the
Scaladoc string for mllib.fpm.FPGrowth?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102535151
--- Diff: mllib/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala
---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.ml.fpm
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.ml.util.DefaultReadWriteTest
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+class FPGrowthSuite extends SparkFunSuite with MLlibTestSparkContext with
DefaultReadWriteTest {
+
+ @transient var dataset: Dataset[_] = _
+
+ override def beforeAll(): Unit = {
+super.beforeAll()
+dataset = FPGrowthSuite.getFPGrowthData(spark)
+ }
+
+ test("FPGrowth fit and transform with different data types") {
+Array(IntegerType, StringType, ShortType, LongType, ByteType).foreach
{ dt =>
+ val intData = dataset.withColumn("features",
col("features").cast(ArrayType(dt)))
+ val model = new FPGrowth().setMinSupport(0.5).fit(intData)
+ val generatedRules = model.setMinConfidence(0.5).getAssociationRules
+ val expectedRules = spark.createDataFrame(Seq(
+(Array("2"), Array("1"), 1.0),
+(Array("1"), Array("2"), 0.75)
+ )).toDF("antecedent", "consequent", "confidence")
+.withColumn("antecedent", col("antecedent").cast(ArrayType(dt)))
+.withColumn("consequent", col("consequent").cast(ArrayType(dt)))
+ assert(expectedRules.sort("antecedent").rdd.collect().sameElements(
+generatedRules.sort("antecedent").rdd.collect()))
+
+ val transformed = model.transform(intData)
+ val expectedTransformed = spark.createDataFrame(Seq(
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "3"), Array(2))
+ )).toDF("id", "features", "prediction")
+.withColumn("features", col("features").cast(ArrayType(dt)))
+.withColumn("prediction", col("prediction").cast(ArrayType(dt)))
+ assert(expectedTransformed.sort("id").rdd.collect().sameElements(
+transformed.sort("id").rdd.collect()))
+}
+ }
+
+ test("FPGrowth getFreqItems") {
+val model = new FPGrowth().setMinSupport(0.7).fit(dataset)
+val expectedFreq = spark.createDataFrame(Seq(
+ (Array("1"), 4L),
+ (Array("2"), 3L),
+ (Array("1", "2"), 3L),
+ (Array("2", "1"), 3L)
--- End diff --
This is a duplicate, right? It explains the weird check for count() = 3
below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102505813
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+ FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions (>=1) used by parallel FP-growth. By default the
param is not set, and
+ * partition number of the input dataset is used.
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * Minimal confidence for generating Association Rule.
+ * Note that minConfidence has no effect during fitting.
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ @Since("2.2.0")
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102273256
--- Diff: mllib/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala
---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.ml.fpm
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.ml.util.DefaultReadWriteTest
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+class FPGrowthSuite extends SparkFunSuite with MLlibTestSparkContext with
DefaultReadWriteTest {
+
+ @transient var dataset: Dataset[_] = _
+
+ override def beforeAll(): Unit = {
+super.beforeAll()
+dataset = FPGrowthSuite.getFPGrowthData(spark)
+ }
+
+ test("FPGrowth fit and transform with different data types") {
+Array(IntegerType, StringType, ShortType, LongType, ByteType).foreach
{ dt =>
+ val intData = dataset.withColumn("features",
col("features").cast(ArrayType(dt)))
--- End diff --
rename intData -> data
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r102273266
--- Diff: mllib/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala
---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.ml.fpm
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.ml.util.DefaultReadWriteTest
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+class FPGrowthSuite extends SparkFunSuite with MLlibTestSparkContext with
DefaultReadWriteTest {
+
+ @transient var dataset: Dataset[_] = _
+
+ override def beforeAll(): Unit = {
+super.beforeAll()
+dataset = FPGrowthSuite.getFPGrowthData(spark)
+ }
+
+ test("FPGrowth fit and transform with different data types") {
+Array(IntegerType, StringType, ShortType, LongType, ByteType).foreach
{ dt =>
+ val intData = dataset.withColumn("features",
col("features").cast(ArrayType(dt)))
+ val model = new FPGrowth().setMinSupport(0.5).fit(intData)
+ val generatedRules = model.setMinConfidence(0.5).getAssociationRules
+ val expectedRules = spark.createDataFrame(Seq(
+(Array("2"), Array("1"), 1.0),
+(Array("1"), Array("2"), 0.75)
+ )).toDF("antecedent", "consequent", "confidence")
+.withColumn("antecedent", col("antecedent").cast(ArrayType(dt)))
+.withColumn("consequent", col("consequent").cast(ArrayType(dt)))
+ assert(expectedRules.sort("antecedent").rdd.collect().sameElements(
+generatedRules.sort("antecedent").rdd.collect()))
+
+ val transformed = model.transform(intData)
+ val expectedTransformed = spark.createDataFrame(Seq(
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "2"), Array.emptyIntArray),
+(0, Array("1", "3"), Array(2))
+ )).toDF("id", "features", "prediction")
+.withColumn("features", col("features").cast(ArrayType(dt)))
+.withColumn("prediction", col("prediction").cast(ArrayType(dt)))
+ assert(expectedTransformed.sort("id").rdd.collect().sameElements(
+transformed.sort("id").rdd.collect()))
+}
+ }
+
+ test("FPGrowth getFreqItems") {
+val model = new FPGrowth().setMinSupport(0.7).fit(dataset)
+val expectedFreq = spark.createDataFrame(Seq(
+ (Array("1"), 4L),
+ (Array("2"), 3L),
+ (Array("1", "2"), 3L),
+ (Array("2", "1"), 3L)
+)).toDF("items", "freqExp")
+val freqItems = model.getFreqItemsets
+
+val checkDF = freqItems.join(expectedFreq, "items")
+assert(checkDF.count() == 3 && checkDF.filter(col("freq") ===
col("freqExp")).count() == 3)
+ }
+
+ test("FPGrowth getFreqItems with Null") {
--- End diff --
In FPGrowth, document that null values are treated as empty sequences.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r101939121
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * minimal confidence for generating Association Rule
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group expertSetParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam
+ * minConfidence has not effect during fitting.
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r101698347
--- Diff: mllib/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala
---
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.ml.fpm
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.ml.util.DefaultReadWriteTest
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+class FPGrowthSuite extends SparkFunSuite with MLlibTestSparkContext with
DefaultReadWriteTest {
+
+ @transient var dataset: Dataset[_] = _
+
+ override def beforeAll(): Unit = {
+super.beforeAll()
+dataset = FPGrowthSuite.getFPGrowthData(spark)
+ }
+
+ test("FPGrowth fit and transform with different data types") {
+Array(IntegerType, StringType, ShortType, LongType, ByteType).foreach
{ dt =>
+ val intData = dataset.withColumn("features",
col("features").cast(ArrayType(dt)))
+ val model = new FPGrowth().setMinSupport(0.8).fit(intData)
+ val generatedRules = model.setMinConfidence(0.8).getAssociationRules
+ val expectedRules = spark.createDataFrame(Seq(
+(Array("2"), Array("1"), 1.0),
+(Array("1"), Array("2"), 1.0)
+ )).toDF("antecedent", "consequent", "confidence")
+.withColumn("antecedent", col("antecedent").cast(ArrayType(dt)))
+.withColumn("consequent", col("consequent").cast(ArrayType(dt)))
+
+ assert(expectedRules.sort("antecedent").rdd.collect().sameElements(
+generatedRules.sort("antecedent").rdd.collect()))
+ val transformed = model.transform(intData)
+ assert(transformed.count() == 3)
+}
+ }
+
+ test("FPGrowth getFreqItems") {
+val model = new FPGrowth().setMinSupport(0.8).fit(dataset)
+val expectedFreq = spark.createDataFrame(Seq(
+ (Array("1"), 3L),
+ (Array("2"), 3L),
+ (Array("1", "2"), 3L)
+)).toDF("items", "freq")
+val freqItems = model.getFreqItemsets
+assert(freqItems.sort("items").rdd.collect()
+ .sameElements(expectedFreq.sort("items").rdd.collect()))
+ }
+
+ test("FPGrowth get Association Rules") {
+val model = new FPGrowth().setMinSupport(0.8).fit(dataset)
+val expectedRules = spark.createDataFrame(Seq(
+ (Array("2"), Array("1"), 1.0),
+ (Array("1"), Array("2"), 1.0)
+)).toDF("antecedent", "consequent", "confidence")
+val associationRules = model.getAssociationRules
+
+assert(associationRules.sort("antecedent").rdd.collect()
+ .sameElements(expectedRules.sort("antecedent").rdd.collect()))
+ }
+
+ test("FPGrowth parameter check") {
+val fpGrowth = new FPGrowth().setMinSupport(0.4567)
+val model = fpGrowth.fit(dataset)
+ .setMinConfidence(0.5678)
+assert(fpGrowth.getMinSupport === 0.4567)
+assert(model.getMinConfidence === 0.5678)
+ }
+
+ test("read/write") {
+def checkModelData(model: FPGrowthModel, model2: FPGrowthModel): Unit
= {
+ assert(model.freqItemsets.sort("items").collect() ===
+model2.freqItemsets.sort("items").collect())
+}
+val fPGrowth = new FPGrowth()
+testEstimatorAndModelReadWrite(
+ fPGrowth, dataset, FPGrowthSuite.allParamSettings, checkModelData)
+ }
+
+}
+
+object FPGrowthSuite {
+
+ def getFPGrowthData(spark: SparkSession): DataFrame = {
+spark.createDataFrame(Seq(
+ (0, Array("1", "2", "3", "5")),
+ (0, Array("1", "2", "3", "6")),
+ (0, Array("1", "2", "7"))
+)).toDF("id", "features")
+ }
+
+ /**
+ * Mapping from all Params to valid settings which differ from the
defaults.
+ * This is
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r101698262
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+FPGrowth => MLlibFPGrowth}
--- End diff --
style: indent with 2 spaces
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r101698270
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
--- End diff --
Since annotation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r101698275
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
--- End diff --
Document what it means when this is not set and that this must be >= 1.
Also say this is not set by default.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r101698338
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * minimal confidence for generating Association Rule
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group expertSetParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam
+ * minConfidence has not effect during
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r101698323
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * minimal confidence for generating Association Rule
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group expertSetParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam
+ * minConfidence has not effect during
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r101698331
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * minimal confidence for generating Association Rule
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group expertSetParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam
+ * minConfidence has not effect during
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r101698335
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * minimal confidence for generating Association Rule
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group expertSetParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam
+ * minConfidence has not effect during
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r101698340
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * minimal confidence for generating Association Rule
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group expertSetParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam
+ * minConfidence has not effect during
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r101698315
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * minimal confidence for generating Association Rule
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group expertSetParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam
+ * minConfidence has not effect during
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r101698326
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * minimal confidence for generating Association Rule
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group expertSetParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam
+ * minConfidence has not effect during
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r101698328
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules,
+FPGrowth => MLlibFPGrowth}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql._
+import org.apache.spark.sql.types._
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+val inputType = schema($(featuresCol)).dataType
+require(inputType.isInstanceOf[ArrayType],
+ s"The input column must be ArrayType, but got $inputType.")
+SchemaUtils.appendColumn(schema, $(predictionCol),
schema($(featuresCol)).dataType)
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group expertParam
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group expertGetParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+ /**
+ * minimal confidence for generating Association Rule
+ * Default: 0.8
+ * @group param
+ */
+ @Since("2.2.0")
+ val minConfidence: DoubleParam = new DoubleParam(this, "minConfidence",
+"minimal confidence for generating Association Rule (Default: 0.8)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minConfidence -> 0.8)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinConfidence: Double = $(minConfidence)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("fpgrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group expertSetParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam
+ * minConfidence has not effect during
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98959530
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group param
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("FPGrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol,
value)
+
+ def fit(dataset: Dataset[_]): FPGrowthModel = {
--- End diff --
override
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98959548
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group param
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("FPGrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol,
value)
+
+ def fit(dataset: Dataset[_]): FPGrowthModel = {
+val data = dataset.select($(featuresCol)).rdd.map(r =>
r.getSeq[String](0).toArray)
+val parentModel = new
MLlibFPGrowth().setMinSupport($(minSupport)).run(data)
+copyValues(new FPGrowthModel(uid, parentModel))
+ }
+
+ @Since("2.2.0")
+ override def transformSchema(schema: StructType): StructType = {
+validateAndTransformSchema(schema)
+ }
+
+ override def
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98959536
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group param
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("FPGrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol,
value)
+
+ def fit(dataset: Dataset[_]): FPGrowthModel = {
+val data = dataset.select($(featuresCol)).rdd.map(r =>
r.getSeq[String](0).toArray)
+val parentModel = new
MLlibFPGrowth().setMinSupport($(minSupport)).run(data)
+copyValues(new FPGrowthModel(uid, parentModel))
--- End diff --
setParent
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98959414
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.HasPredictionCol
+import org.apache.spark.ml.util.{DefaultParamsReader, DefaultParamsWriter,
_}
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+
+/**
+ * Common params for AssociationRules and AssociationRulesModel
+ */
+private[fpm] trait AssociationRulesParam extends Params with
HasPredictionCol{
+
+ /**
+ * Param for items column name. Items must be array of Strings.
+ * Default: "items"
+ *
+ * @group param
+ */
+ final val itemsCol: Param[String] = new Param[String](this, "itemsCol",
"column name in the" +
+" DataFrame containing the items")
+
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getItemsCol: String = $(itemsCol)
+ setDefault(itemsCol -> "items")
+}
+
+@Since("2.2.0")
+object AssociationRules extends DefaultParamsReadable[AssociationRules] {
+
+ @Since("2.2.0")
+ override def load(path: String): AssociationRules = super.load(path)
+}
+
+/**
+ * :: Experimental ::
+ *
+ * Generates association rules from frequent itemsets DataFrame("items",
"freq"). This method only
+ * generates association rules which have a single item as the consequent.
+ */
+@Since("2.2.0")
+@Experimental
+class AssociationRules(override val uid: String)
+ extends Estimator[AssociationRulesModel] with AssociationRulesParam with
HasPredictionCol {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("AssociationRules"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setItemsCol(value: String): this.type = set(itemsCol, value)
+
+ /**
+ * Param for frequency column name. Data type should be Long.
+ * Default: "freq"
+ *
+ * @group param
+ */
+ final val freqCol: Param[String] = new Param[String](this, "freqCol",
"frequency column name")
+
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getFreqCol: String = $(freqCol)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFreqCol(value: String): this.type = set(freqCol, value)
+ setDefault(freqCol -> "freq")
+
+ /**
+ * Param for minimum confidence, range [0.0, 1.0].
+*
+* @group param
+ */
+ final val minConfidence: DoubleParam = new DoubleParam(this,
"minConfidence", "min confidence",
+ParamValidators.inRange(0.0, 1.0))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getMinConfidence: Double = $(minConfidence)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinConfidence(value: Double): this.type = set(minConfidence,
value)
+ setDefault(minConfidence -> 0.8)
+
+ /**
+ * Computes the association rules with confidence above
[[minConfidence]].
+ *
+ * @return a DataFrame("antecedent", "consequent", "confidence")
containing the association
+ * rules.
+ */
+ @Since("2.2.0")
+ override def fit(dataset: Dataset[_]): AssociationRulesModel = {
--- End diff --
Definitely document the input schema since it is non-standard. The
description here can be short, with more details in the class-level Scala doc.
---
If your
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98959585
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.HasPredictionCol
+import org.apache.spark.ml.util.{DefaultParamsReader, DefaultParamsWriter,
_}
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+
+/**
+ * Common params for AssociationRules and AssociationRulesModel
+ */
+private[fpm] trait AssociationRulesParam extends Params with
HasPredictionCol{
+
+ /**
+ * Param for items column name. Items must be array of Strings.
+ * Default: "items"
+ *
+ * @group param
+ */
+ final val itemsCol: Param[String] = new Param[String](this, "itemsCol",
"column name in the" +
+" DataFrame containing the items")
+
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getItemsCol: String = $(itemsCol)
+ setDefault(itemsCol -> "items")
+}
+
+@Since("2.2.0")
+object AssociationRules extends DefaultParamsReadable[AssociationRules] {
+
+ @Since("2.2.0")
+ override def load(path: String): AssociationRules = super.load(path)
+}
+
+/**
+ * :: Experimental ::
+ *
+ * Generates association rules from frequent itemsets DataFrame("items",
"freq"). This method only
+ * generates association rules which have a single item as the consequent.
+ */
+@Since("2.2.0")
+@Experimental
+class AssociationRules(override val uid: String)
+ extends Estimator[AssociationRulesModel] with AssociationRulesParam with
HasPredictionCol {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("AssociationRules"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setItemsCol(value: String): this.type = set(itemsCol, value)
+
+ /**
+ * Param for frequency column name. Data type should be Long.
+ * Default: "freq"
+ *
+ * @group param
+ */
+ final val freqCol: Param[String] = new Param[String](this, "freqCol",
"frequency column name")
+
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getFreqCol: String = $(freqCol)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFreqCol(value: String): this.type = set(freqCol, value)
+ setDefault(freqCol -> "freq")
+
+ /**
+ * Param for minimum confidence, range [0.0, 1.0].
+*
+* @group param
+ */
+ final val minConfidence: DoubleParam = new DoubleParam(this,
"minConfidence", "min confidence",
+ParamValidators.inRange(0.0, 1.0))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getMinConfidence: Double = $(minConfidence)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinConfidence(value: Double): this.type = set(minConfidence,
value)
+ setDefault(minConfidence -> 0.8)
+
+ /**
+ * Computes the association rules with confidence above
[[minConfidence]].
+ *
+ * @return a DataFrame("antecedent", "consequent", "confidence")
containing the association
--- End diff --
returns a model, not DataFrame
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98959506
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.HasPredictionCol
+import org.apache.spark.ml.util.{DefaultParamsReader, DefaultParamsWriter,
_}
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+
+/**
+ * Common params for AssociationRules and AssociationRulesModel
+ */
+private[fpm] trait AssociationRulesParam extends Params with
HasPredictionCol{
+
+ /**
+ * Param for items column name. Items must be array of Strings.
+ * Default: "items"
+ *
+ * @group param
+ */
+ final val itemsCol: Param[String] = new Param[String](this, "itemsCol",
"column name in the" +
+" DataFrame containing the items")
+
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getItemsCol: String = $(itemsCol)
+ setDefault(itemsCol -> "items")
+}
+
+@Since("2.2.0")
+object AssociationRules extends DefaultParamsReadable[AssociationRules] {
+
+ @Since("2.2.0")
+ override def load(path: String): AssociationRules = super.load(path)
+}
+
+/**
+ * :: Experimental ::
+ *
+ * Generates association rules from frequent itemsets DataFrame("items",
"freq"). This method only
+ * generates association rules which have a single item as the consequent.
+ */
+@Since("2.2.0")
+@Experimental
+class AssociationRules(override val uid: String)
+ extends Estimator[AssociationRulesModel] with AssociationRulesParam with
HasPredictionCol {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("AssociationRules"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setItemsCol(value: String): this.type = set(itemsCol, value)
+
+ /**
+ * Param for frequency column name. Data type should be Long.
+ * Default: "freq"
+ *
+ * @group param
+ */
+ final val freqCol: Param[String] = new Param[String](this, "freqCol",
"frequency column name")
+
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getFreqCol: String = $(freqCol)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFreqCol(value: String): this.type = set(freqCol, value)
+ setDefault(freqCol -> "freq")
+
+ /**
+ * Param for minimum confidence, range [0.0, 1.0].
+*
+* @group param
+ */
+ final val minConfidence: DoubleParam = new DoubleParam(this,
"minConfidence", "min confidence",
+ParamValidators.inRange(0.0, 1.0))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getMinConfidence: Double = $(minConfidence)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinConfidence(value: Double): this.type = set(minConfidence,
value)
+ setDefault(minConfidence -> 0.8)
+
+ /**
+ * Computes the association rules with confidence above
[[minConfidence]].
+ *
+ * @return a DataFrame("antecedent", "consequent", "confidence")
containing the association
+ * rules.
+ */
+ @Since("2.2.0")
+ override def fit(dataset: Dataset[_]): AssociationRulesModel = {
+val freqItemSetRdd = dataset.select($(itemsCol), $(freqCol)).rdd
+ .map(row => new FreqItemset(row.getSeq[String](0).toArray,
row.getLong(1)))
+
+val spark =
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98959524
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group param
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("FPGrowth"))
--- End diff --
Use lowercase name to match other algs: "fpgrowth"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98959499
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.HasPredictionCol
+import org.apache.spark.ml.util.{DefaultParamsReader, DefaultParamsWriter,
_}
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+
+/**
+ * Common params for AssociationRules and AssociationRulesModel
+ */
+private[fpm] trait AssociationRulesParam extends Params with
HasPredictionCol{
+
+ /**
+ * Param for items column name. Items must be array of Strings.
+ * Default: "items"
+ *
+ * @group param
+ */
+ final val itemsCol: Param[String] = new Param[String](this, "itemsCol",
"column name in the" +
+" DataFrame containing the items")
+
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getItemsCol: String = $(itemsCol)
+ setDefault(itemsCol -> "items")
+}
+
+@Since("2.2.0")
+object AssociationRules extends DefaultParamsReadable[AssociationRules] {
+
+ @Since("2.2.0")
+ override def load(path: String): AssociationRules = super.load(path)
+}
+
+/**
+ * :: Experimental ::
+ *
+ * Generates association rules from frequent itemsets DataFrame("items",
"freq"). This method only
+ * generates association rules which have a single item as the consequent.
+ */
+@Since("2.2.0")
+@Experimental
+class AssociationRules(override val uid: String)
+ extends Estimator[AssociationRulesModel] with AssociationRulesParam with
HasPredictionCol {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("AssociationRules"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setItemsCol(value: String): this.type = set(itemsCol, value)
+
+ /**
+ * Param for frequency column name. Data type should be Long.
+ * Default: "freq"
+ *
+ * @group param
+ */
+ final val freqCol: Param[String] = new Param[String](this, "freqCol",
"frequency column name")
+
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getFreqCol: String = $(freqCol)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFreqCol(value: String): this.type = set(freqCol, value)
+ setDefault(freqCol -> "freq")
+
+ /**
+ * Param for minimum confidence, range [0.0, 1.0].
+*
+* @group param
+ */
+ final val minConfidence: DoubleParam = new DoubleParam(this,
"minConfidence", "min confidence",
+ParamValidators.inRange(0.0, 1.0))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getMinConfidence: Double = $(minConfidence)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinConfidence(value: Double): this.type = set(minConfidence,
value)
+ setDefault(minConfidence -> 0.8)
+
+ /**
+ * Computes the association rules with confidence above
[[minConfidence]].
+ *
+ * @return a DataFrame("antecedent", "consequent", "confidence")
containing the association
+ * rules.
+ */
+ @Since("2.2.0")
+ override def fit(dataset: Dataset[_]): AssociationRulesModel = {
+val freqItemSetRdd = dataset.select($(itemsCol), $(freqCol)).rdd
+ .map(row => new FreqItemset(row.getSeq[String](0).toArray,
row.getLong(1)))
+
+val spark =
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98959519
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group param
--- End diff --
Make this an expertParam
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98959540
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group param
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("FPGrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol,
value)
+
+ def fit(dataset: Dataset[_]): FPGrowthModel = {
+val data = dataset.select($(featuresCol)).rdd.map(r =>
r.getSeq[String](0).toArray)
+val parentModel = new
MLlibFPGrowth().setMinSupport($(minSupport)).run(data)
+copyValues(new FPGrowthModel(uid, parentModel))
+ }
+
+ @Since("2.2.0")
+ override def transformSchema(schema: StructType): StructType = {
+validateAndTransformSchema(schema)
+ }
+
+ override def
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98959496
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala ---
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.HasPredictionCol
+import org.apache.spark.ml.util.{DefaultParamsReader, DefaultParamsWriter,
_}
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+
+/**
+ * Common params for AssociationRules and AssociationRulesModel
+ */
+private[fpm] trait AssociationRulesParam extends Params with
HasPredictionCol{
+
+ /**
+ * Param for items column name. Items must be array of Strings.
+ * Default: "items"
+ *
+ * @group param
+ */
+ final val itemsCol: Param[String] = new Param[String](this, "itemsCol",
"column name in the" +
+" DataFrame containing the items")
+
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getItemsCol: String = $(itemsCol)
+ setDefault(itemsCol -> "items")
+}
+
+@Since("2.2.0")
+object AssociationRules extends DefaultParamsReadable[AssociationRules] {
+
+ @Since("2.2.0")
+ override def load(path: String): AssociationRules = super.load(path)
+}
+
+/**
+ * :: Experimental ::
+ *
+ * Generates association rules from frequent itemsets DataFrame("items",
"freq"). This method only
+ * generates association rules which have a single item as the consequent.
+ */
+@Since("2.2.0")
+@Experimental
+class AssociationRules(override val uid: String)
+ extends Estimator[AssociationRulesModel] with AssociationRulesParam with
HasPredictionCol {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("AssociationRules"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setItemsCol(value: String): this.type = set(itemsCol, value)
+
+ /**
+ * Param for frequency column name. Data type should be Long.
+ * Default: "freq"
+ *
+ * @group param
+ */
+ final val freqCol: Param[String] = new Param[String](this, "freqCol",
"frequency column name")
+
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getFreqCol: String = $(freqCol)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFreqCol(value: String): this.type = set(freqCol, value)
+ setDefault(freqCol -> "freq")
+
+ /**
+ * Param for minimum confidence, range [0.0, 1.0].
+*
+* @group param
+ */
+ final val minConfidence: DoubleParam = new DoubleParam(this,
"minConfidence", "min confidence",
+ParamValidators.inRange(0.0, 1.0))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ final def getMinConfidence: Double = $(minConfidence)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinConfidence(value: Double): this.type = set(minConfidence,
value)
+ setDefault(minConfidence -> 0.8)
+
+ /**
+ * Computes the association rules with confidence above
[[minConfidence]].
+ *
+ * @return a DataFrame("antecedent", "consequent", "confidence")
containing the association
+ * rules.
+ */
+ @Since("2.2.0")
+ override def fit(dataset: Dataset[_]): AssociationRulesModel = {
+val freqItemSetRdd = dataset.select($(itemsCol), $(freqCol)).rdd
+ .map(row => new FreqItemset(row.getSeq[String](0).toArray,
row.getLong(1)))
+
+val spark =
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98959562
--- Diff: mllib/src/test/scala/org/apache/spark/ml/fpm/FPGrowthSuite.scala
---
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.ml.fpm
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.ml.util.DefaultReadWriteTest
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
+
+class FPGrowthSuite extends SparkFunSuite with MLlibTestSparkContext with
DefaultReadWriteTest {
+
+ @transient var dataset: Dataset[_] = _
+
+ override def beforeAll(): Unit = {
+super.beforeAll()
+
+dataset = FPGrowthSuite.getFPGrowthData(spark)
+ }
+
+ test("FPGrowth fit and transform") {
+val model = new FPGrowth().setMinSupport(0.8).fit(dataset)
+val generatedRules = model.setMinConfidence(0.8).getAssociationRules
+val expectedRules = spark.createDataFrame(Seq(
+ (Array("2"), Array("1"), 1.0),
+ (Array("1"), Array("2"), 1.0)
+)).toDF("antecedent", "consequent", "confidence")
+
+assert(expectedRules.sort("antecedent").rdd.collect().sameElements(
+ generatedRules.sort("antecedent").rdd.collect()))
+val transformed = model.transform(dataset)
+assert(transformed.count() == 3)
+ }
+
+ test("FPGrowth getFreqItems") {
+val model = new FPGrowth().setMinSupport(0.8).fit(dataset)
+val expectedFreq = spark.createDataFrame(Seq(
+ (Array("1"), 3L),
+ (Array("2"), 3L),
+ (Array("1", "2"), 3L)
+)).toDF("items", "freq")
+val freqItems = model.getFreqItems
+assert(freqItems.sort("items").rdd.collect()
+ .sameElements(expectedFreq.sort("items").rdd.collect()))
+ }
+
+ test("FPGrowth get Association Rules") {
+val model = new FPGrowth().setMinSupport(0.8).fit(dataset)
+val expectedRules = spark.createDataFrame(Seq(
+ (Array("2"), Array("1"), 1.0),
+ (Array("1"), Array("2"), 1.0)
+)).toDF("antecedent", "consequent", "confidence")
+val associationRules = model.getAssociationRules
+
+assert(associationRules.sort("antecedent").rdd.collect()
+ .sameElements(expectedRules.sort("antecedent").rdd.collect()))
+ }
+
+ test("FPGrowth parameter check") {
+val fpGrowth = new FPGrowth().setMinSupport(0.4567)
+val model = fpGrowth.fit(dataset)
+ .setMinConfidence(0.5678)
+assert(fpGrowth.getMinSupport === 0.4567)
+assert(model.getMinConfidence === 0.5678)
+ }
+
+ test("read/write") {
+def checkModelData(model: FPGrowthModel, model2: FPGrowthModel): Unit
= {
+ assert(model.getAssociationRules.collect() ===
+model2.getAssociationRules.collect())
+}
+val fPGrowth = new FPGrowth()
+testEstimatorAndModelReadWrite(
+ fPGrowth, dataset, FPGrowthSuite.allParamSettings, checkModelData)
+ }
+
+}
+
+object FPGrowthSuite {
+
+ def getFPGrowthData(spark: SparkSession): DataFrame = {
+spark.createDataFrame(Seq(
+ (0, Array("1", "2", "3", "5")),
+ (0, Array("1", "2", "3", "6")),
+ (0, Array("1", "2", "7"))
+)).toDF("id", "features")
+ }
+
+ /**
+ * Mapping from all Params to valid settings which differ from the
defaults.
+ * This is useful for tests which need to exercise all Params, such as
save/load.
+ * This excludes input columns to simplify some tests.
+ */
+ val allParamSettings: Map[String, Any] = Map(
+"minSupport" -> 0.3
--- End diff --
This should include numPartitions and predictionCol
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98959556
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group param
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("FPGrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol,
value)
+
+ def fit(dataset: Dataset[_]): FPGrowthModel = {
+val data = dataset.select($(featuresCol)).rdd.map(r =>
r.getSeq[String](0).toArray)
+val parentModel = new
MLlibFPGrowth().setMinSupport($(minSupport)).run(data)
+copyValues(new FPGrowthModel(uid, parentModel))
+ }
+
+ @Since("2.2.0")
+ override def transformSchema(schema: StructType): StructType = {
+validateAndTransformSchema(schema)
+ }
+
+ override def
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r98836498
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group param
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("FPGrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol,
value)
+
+ def fit(dataset: Dataset[_]): FPGrowthModel = {
+val data = dataset.select($(featuresCol)).rdd.map(r =>
r.getSeq[String](0).toArray)
+val parentModel = new
MLlibFPGrowth().setMinSupport($(minSupport)).run(data)
+copyValues(new FPGrowthModel(uid, parentModel))
+ }
+
+ @Since("2.2.0")
+ override def transformSchema(schema: StructType): StructType = {
+validateAndTransformSchema(schema)
+ }
+
+ override def
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user aray commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r97168170
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group param
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("FPGrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol,
value)
+
+ def fit(dataset: Dataset[_]): FPGrowthModel = {
+val data = dataset.select($(featuresCol)).rdd.map(r =>
r.getSeq[String](0).toArray)
+val parentModel = new
MLlibFPGrowth().setMinSupport($(minSupport)).run(data)
+copyValues(new FPGrowthModel(uid, parentModel))
+ }
+
+ @Since("2.2.0")
+ override def transformSchema(schema: StructType): StructType = {
+validateAndTransformSchema(schema)
+ }
+
+ override def
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user aray commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r97162464
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group param
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("FPGrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol,
value)
+
+ def fit(dataset: Dataset[_]): FPGrowthModel = {
+val data = dataset.select($(featuresCol)).rdd.map(r =>
r.getSeq[String](0).toArray)
+val parentModel = new
MLlibFPGrowth().setMinSupport($(minSupport)).run(data)
+copyValues(new FPGrowthModel(uid, parentModel))
+ }
+
+ @Since("2.2.0")
+ override def transformSchema(schema: StructType): StructType = {
+validateAndTransformSchema(schema)
+ }
+
+ override def
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user aray commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r97168311
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group param
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("FPGrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol,
value)
+
+ def fit(dataset: Dataset[_]): FPGrowthModel = {
+val data = dataset.select($(featuresCol)).rdd.map(r =>
r.getSeq[String](0).toArray)
+val parentModel = new
MLlibFPGrowth().setMinSupport($(minSupport)).run(data)
+copyValues(new FPGrowthModel(uid, parentModel))
+ }
+
+ @Since("2.2.0")
+ override def transformSchema(schema: StructType): StructType = {
+validateAndTransformSchema(schema)
+ }
+
+ override def
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user aray commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r97166816
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * Minimal support level of the frequent pattern. [0.0, 1.0]. Any
pattern that appears
+ * more than (minSupport * size-of-the-dataset) times will be output
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)",
+ParamValidators.inRange(0.0, 1.0))
+ setDefault(minSupport -> 0.3)
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+ /**
+ * Number of partitions used by parallel FP-growth
+ * @group param
+ */
+ @Since("2.2.0")
+ val numPartitions: IntParam = new IntParam(this, "numPartitions",
+"Number of partitions used by parallel FP-growth",
ParamValidators.gtEq[Int](1))
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getNumPartitions: Int = $(numPartitions)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("FPGrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setNumPartitions(value: Int): this.type = set(numPartitions, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol,
value)
+
+ def fit(dataset: Dataset[_]): FPGrowthModel = {
+val data = dataset.select($(featuresCol)).rdd.map(r =>
r.getSeq[String](0).toArray)
+val parentModel = new
MLlibFPGrowth().setMinSupport($(minSupport)).run(data)
+copyValues(new FPGrowthModel(uid, parentModel))
+ }
+
+ @Since("2.2.0")
+ override def transformSchema(schema: StructType): StructType = {
+validateAndTransformSchema(schema)
+ }
+
+ override def
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r96804046
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.{DoubleParam, ParamMap, Params}
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * the minimal support level of the frequent pattern
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)")
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("FPGrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+ setDefault(minSupport -> 0.3)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setPredictionCol(value: String): this.type = set(predictionCol,
value)
+
+ def fit(dataset: Dataset[_]): FPGrowthModel = {
+val data = dataset.select($(featuresCol)).rdd.map(r =>
r.getSeq[String](0).toArray)
+val parentModel = new
MLlibFPGrowth().setMinSupport($(minSupport)).run(data)
+copyValues(new FPGrowthModel(uid, parentModel))
+ }
+
+ @Since("2.2.0")
+ override def transformSchema(schema: StructType): StructType = {
+validateAndTransformSchema(schema)
+ }
+
+ override def copy(extra: ParamMap): FPGrowth = defaultCopy(extra)
+}
+
+
+@Since("2.2.0")
+object FPGrowth extends DefaultParamsReadable[FPGrowth] {
+
+ @Since("2.2.0")
+ override def load(path: String): FPGrowth = super.load(path)
+}
+
+/**
+ * :: Experimental ::
+ * Model fitted by FPGrowth.
+ *
+ * @param parentModel a model trained by spark.mllib.fpm.FPGrowth
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowthModel private[ml] (
+@Since("2.2.0") override val uid: String,
+private val parentModel: MLlibFPGrowthModel[_])
+ extends Model[FPGrowthModel] with FPGrowthParams with
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r96803812
--- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala ---
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.{DoubleParam, ParamMap, Params}
+import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
+import org.apache.spark.ml.util._
+import org.apache.spark.mllib.fpm.{FPGrowth => MLlibFPGrowth,
FPGrowthModel => MLlibFPGrowthModel}
+import org.apache.spark.sql.{DataFrame, _}
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.{ArrayType, StringType, StructType}
+
+/**
+ * Common params for FPGrowth and FPGrowthModel
+ */
+private[fpm] trait FPGrowthParams extends Params with HasFeaturesCol with
HasPredictionCol {
+
+ /**
+ * Validates and transforms the input schema.
+ * @param schema input schema
+ * @return output schema
+ */
+ protected def validateAndTransformSchema(schema: StructType): StructType
= {
+SchemaUtils.checkColumnType(schema, $(featuresCol), new
ArrayType(StringType, false))
+SchemaUtils.appendColumn(schema, $(predictionCol), new
ArrayType(StringType, false))
+ }
+
+ /**
+ * the minimal support level of the frequent pattern
+ * Default: 0.3
+ * @group param
+ */
+ @Since("2.2.0")
+ val minSupport: DoubleParam = new DoubleParam(this, "minSupport",
+"the minimal support level of the frequent pattern (Default: 0.3)")
+
+ /** @group getParam */
+ @Since("2.2.0")
+ def getMinSupport: Double = $(minSupport)
+
+}
+
+/**
+ * :: Experimental ::
+ * A parallel FP-growth algorithm to mine frequent itemsets.
+ *
+ * @see [[http://dx.doi.org/10.1145/1454008.1454027 Li et al., PFP:
Parallel FP-Growth for Query
+ * Recommendation]]
+ */
+@Since("2.2.0")
+@Experimental
+class FPGrowth @Since("2.2.0") (
+@Since("2.2.0") override val uid: String)
+ extends Estimator[FPGrowthModel] with FPGrowthParams with
DefaultParamsWritable {
+
+ @Since("2.2.0")
+ def this() = this(Identifiable.randomUID("FPGrowth"))
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setMinSupport(value: Double): this.type = set(minSupport, value)
+ setDefault(minSupport -> 0.3)
+
+ /** @group setParam */
+ @Since("2.2.0")
+ def setFeaturesCol(value: String): this.type = set(featuresCol, value)
--- End diff --
Thanks. Let's collect more feedback about it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15415: [SPARK-14503][ML] spark.ml API for FPGrowth
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/15415#discussion_r96802011
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/fpm/AssociationRules.scala ---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.fpm
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.ml.param.{DoubleParam, Param, ParamMap, Params}
+import org.apache.spark.ml.util.Identifiable
+import org.apache.spark.mllib.fpm.{AssociationRules =>
MLlibAssociationRules}
+import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
+import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
+
+/**
+ * :: Experimental ::
+ *
+ * Generates association rules from frequent itemsets ("items", "freq").
This method only generates
+ * association rules which have a single item as the consequent.
+ */
+@Since("2.1.0")
+@Experimental
+class AssociationRules(override val uid: String) extends Params {
--- End diff --
`freqItemsets` and `rules` does not have a one-to-one mapping relation and
will probably violates the primitives of Transformer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
