[jira] [Commented] (SPARK-6239) Spark MLlib fpm#FPGrowth minSupport should use long instead
[ https://issues.apache.org/jira/browse/SPARK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354450#comment-14354450 ] Littlestar commented on SPARK-6239: --- I just want to set minCount=2 for test 10*1*1 records. FPGrowthModel model = new FPGrowth() .setMinSupport(2/(10*1.0*1.0)) .setNumPartitions(500) .run(maps); I think use minCount is better than minSupport. > Spark MLlib fpm#FPGrowth minSupport should use long instead > --- > > Key: SPARK-6239 > URL: https://issues.apache.org/jira/browse/SPARK-6239 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.3.0 >Reporter: Littlestar >Priority: Minor > > Spark MLlib fpm#FPGrowth minSupport should use long instead > == > val minCount = math.ceil(minSupport * count).toLong > because: > 1. [count]numbers of datasets is not kown before read. > 2. [minSupport ]double precision. > from mahout#FPGrowthDriver.java > addOption("minSupport", "s", "(Optional) The minimum number of times a > co-occurrence must be present." > + " Default Value: 3", "3"); > I just want to set minCount=2 for test. > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6239) Spark MLlib fpm#FPGrowth minSupport should use long instead
[ https://issues.apache.org/jira/browse/SPARK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354829#comment-14354829 ] Littlestar commented on SPARK-6239: --- When use FPGrowthModel, the numbers of input records is unkown before read. I think change the meaning of minSupport, or add setMinCount If I want to set minCount=2, I must use.setMinSupport(1.99/(rdd.count())), because of double's precision. val minCount = math.ceil(minSupport * count).toLong math.ceil > Spark MLlib fpm#FPGrowth minSupport should use long instead > --- > > Key: SPARK-6239 > URL: https://issues.apache.org/jira/browse/SPARK-6239 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.3.0 >Reporter: Littlestar >Priority: Minor > > Spark MLlib fpm#FPGrowth minSupport should use long instead > == > val minCount = math.ceil(minSupport * count).toLong > because: > 1. [count]numbers of datasets is not kown before read. > 2. [minSupport ]double precision. > from mahout#FPGrowthDriver.java > addOption("minSupport", "s", "(Optional) The minimum number of times a > co-occurrence must be present." > + " Default Value: 3", "3"); > I just want to set minCount=2 for test. > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6239) Spark MLlib fpm#FPGrowth minSupport should use long instead
[ https://issues.apache.org/jira/browse/SPARK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354837#comment-14354837 ] Sean Owen commented on SPARK-6239: -- Conversely, you have to know the size of the input if you wish to express support as a fraction, and the only available param is an absolute value. You do know the input size via {{count()}} and can do just as you say. I think you could argue for this parameter either way but the established API chooses to use a fraction, and changing it just exchanges a little convenience in one use case for another. > Spark MLlib fpm#FPGrowth minSupport should use long instead > --- > > Key: SPARK-6239 > URL: https://issues.apache.org/jira/browse/SPARK-6239 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.3.0 >Reporter: Littlestar >Priority: Minor > > Spark MLlib fpm#FPGrowth minSupport should use long instead > == > val minCount = math.ceil(minSupport * count).toLong > because: > 1. [count]numbers of datasets is not kown before read. > 2. [minSupport ]double precision. > from mahout#FPGrowthDriver.java > addOption("minSupport", "s", "(Optional) The minimum number of times a > co-occurrence must be present." > + " Default Value: 3", "3"); > I just want to set minCount=2 for test. > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6239) Spark MLlib fpm#FPGrowth minSupport should use long instead
[ https://issues.apache.org/jira/browse/SPARK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386924#comment-14386924 ] Tomasz Bartczak commented on SPARK-6239: I also stumbled upon this little inconvenience in the API. My point in the discussion is that 1. FPGrowth internally is using a Long value internally, aka minCount (see https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/fpm/FPGrowth.scala#L120 and https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/fpm/FPGrowth.scala#L146) - so it is more performant to specify that directly without doing a count. 2. a good API can be used in multiple use cases. This PR https://github.com/apache/spark/pull/5246 adds 'minCount' to be specified while keeping the existing API untouched. Why would that be a bad idea to include such an option then? > Spark MLlib fpm#FPGrowth minSupport should use long instead > --- > > Key: SPARK-6239 > URL: https://issues.apache.org/jira/browse/SPARK-6239 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.3.0 >Reporter: Littlestar >Priority: Minor > > Spark MLlib fpm#FPGrowth minSupport should use long instead > == > val minCount = math.ceil(minSupport * count).toLong > because: > 1. [count]numbers of datasets is not kown before read. > 2. [minSupport ]double precision. > from mahout#FPGrowthDriver.java > addOption("minSupport", "s", "(Optional) The minimum number of times a > co-occurrence must be present." > + " Default Value: 3", "3"); > I just want to set minCount=2 for test. > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6239) Spark MLlib fpm#FPGrowth minSupport should use long instead
[ https://issues.apache.org/jira/browse/SPARK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386933#comment-14386933 ] Sean Owen commented on SPARK-6239: -- Just the little API overhead for littler gain IMHO. I'm open to other committer opinions but just didn't seem worth changing. I would imagine a relative value is more usually useful. > Spark MLlib fpm#FPGrowth minSupport should use long instead > --- > > Key: SPARK-6239 > URL: https://issues.apache.org/jira/browse/SPARK-6239 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.3.0 >Reporter: Littlestar >Priority: Minor > > Spark MLlib fpm#FPGrowth minSupport should use long instead > == > val minCount = math.ceil(minSupport * count).toLong > because: > 1. [count]numbers of datasets is not kown before read. > 2. [minSupport ]double precision. > from mahout#FPGrowthDriver.java > addOption("minSupport", "s", "(Optional) The minimum number of times a > co-occurrence must be present." > + " Default Value: 3", "3"); > I just want to set minCount=2 for test. > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6239) Spark MLlib fpm#FPGrowth minSupport should use long instead
[ https://issues.apache.org/jira/browse/SPARK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387748#comment-14387748 ] Littlestar commented on SPARK-6239: --- >>If I want to set minCount=2, I must use.setMinSupport(1.99/(rdd.count())), >>because of double's precision. How to reopen this PR and mark relation to pull/5246, thanks. > Spark MLlib fpm#FPGrowth minSupport should use long instead > --- > > Key: SPARK-6239 > URL: https://issues.apache.org/jira/browse/SPARK-6239 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.3.0 >Reporter: Littlestar >Priority: Minor > > Spark MLlib fpm#FPGrowth minSupport should use long instead > == > val minCount = math.ceil(minSupport * count).toLong > because: > 1. [count]numbers of datasets is not kown before read. > 2. [minSupport ]double precision. > from mahout#FPGrowthDriver.java > addOption("minSupport", "s", "(Optional) The minimum number of times a > co-occurrence must be present." > + " Default Value: 3", "3"); > I just want to set minCount=2 for test. > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6239) Spark MLlib fpm#FPGrowth minSupport should use long instead
[ https://issues.apache.org/jira/browse/SPARK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14394247#comment-14394247 ] Apache Spark commented on SPARK-6239: - User 'kretes' has created a pull request for this issue: https://github.com/apache/spark/pull/5246 > Spark MLlib fpm#FPGrowth minSupport should use long instead > --- > > Key: SPARK-6239 > URL: https://issues.apache.org/jira/browse/SPARK-6239 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.3.0 >Reporter: Littlestar >Priority: Minor > > Spark MLlib fpm#FPGrowth minSupport should use long instead > == > val minCount = math.ceil(minSupport * count).toLong > because: > 1. [count]numbers of datasets is not kown before read. > 2. [minSupport ]double precision. > from mahout#FPGrowthDriver.java > addOption("minSupport", "s", "(Optional) The minimum number of times a > co-occurrence must be present." > + " Default Value: 3", "3"); > I just want to set minCount=2 for test. > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org