[jira] [Commented] (SPARK-11069) Add RegexTokenizer option to convert to lowercase
[ https://issues.apache.org/jira/browse/SPARK-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074864#comment-16074864 ] Levente Torok commented on SPARK-11069: --- PySpark interface doesn't have this function implemented, this is why I am mislead. > Add RegexTokenizer option to convert to lowercase > - > > Key: SPARK-11069 > URL: https://issues.apache.org/jira/browse/SPARK-11069 > Project: Spark > Issue Type: New Feature > Components: ML >Reporter: Joseph K. Bradley >Assignee: yuhao yang >Priority: Minor > Fix For: 1.6.0 > > > Tokenizer converts strings to lowercase automatically, but RegexTokenizer > does not. It would be nice to add an option to RegexTokenizer to convert to > lowercase. Proposal: > * call the Boolean Param "toLowercase" > * set default to false (so behavior does not change) > *Q*: Should conversion to lowercase happen before or after regex matching? > * Before: This is simpler. > * After: This gives the user full control since they can have the regex treat > upper/lower case differently. > --> I'd vote for conversion before matching. If a user needs full control, > they can convert to lowercase manually. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11069) Add RegexTokenizer option to convert to lowercase
[
https://issues.apache.org/jira/browse/SPARK-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16073987#comment-16073987
]
yuhao yang commented on SPARK-11069:
[~levente.torok.ge] use
val regexTokenizer = new RegexTokenizer()
*{color:red} .setToLowercase(false){color}*
> Add RegexTokenizer option to convert to lowercase
> -
>
> Key: SPARK-11069
> URL: https://issues.apache.org/jira/browse/SPARK-11069
> Project: Spark
> Issue Type: New Feature
> Components: ML
>Reporter: Joseph K. Bradley
>Assignee: yuhao yang
>Priority: Minor
> Fix For: 1.6.0
>
>
> Tokenizer converts strings to lowercase automatically, but RegexTokenizer
> does not. It would be nice to add an option to RegexTokenizer to convert to
> lowercase. Proposal:
> * call the Boolean Param "toLowercase"
> * set default to false (so behavior does not change)
> *Q*: Should conversion to lowercase happen before or after regex matching?
> * Before: This is simpler.
> * After: This gives the user full control since they can have the regex treat
> upper/lower case differently.
> --> I'd vote for conversion before matching. If a user needs full control,
> they can convert to lowercase manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11069) Add RegexTokenizer option to convert to lowercase
[ https://issues.apache.org/jira/browse/SPARK-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16073508#comment-16073508 ] Levente Torok commented on SPARK-11069: --- With this modification, in v1.6.x, there is no way to tokenize w/o. So this modification sucks. So if "toLowercase" option is not implemented, as it is now, it is still better to have no conversion at all, since one can convert before using it if he/she wants, but one cannot use w/o conversion if he/she doesn't want. > Add RegexTokenizer option to convert to lowercase > - > > Key: SPARK-11069 > URL: https://issues.apache.org/jira/browse/SPARK-11069 > Project: Spark > Issue Type: New Feature > Components: ML >Reporter: Joseph K. Bradley >Assignee: yuhao yang >Priority: Minor > Fix For: 1.6.0 > > > Tokenizer converts strings to lowercase automatically, but RegexTokenizer > does not. It would be nice to add an option to RegexTokenizer to convert to > lowercase. Proposal: > * call the Boolean Param "toLowercase" > * set default to false (so behavior does not change) > *Q*: Should conversion to lowercase happen before or after regex matching? > * Before: This is simpler. > * After: This gives the user full control since they can have the regex treat > upper/lower case differently. > --> I'd vote for conversion before matching. If a user needs full control, > they can convert to lowercase manually. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11069) Add RegexTokenizer option to convert to lowercase
[ https://issues.apache.org/jira/browse/SPARK-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954536#comment-14954536 ] Apache Spark commented on SPARK-11069: -- User 'hhbyyh' has created a pull request for this issue: https://github.com/apache/spark/pull/9092 > Add RegexTokenizer option to convert to lowercase > - > > Key: SPARK-11069 > URL: https://issues.apache.org/jira/browse/SPARK-11069 > Project: Spark > Issue Type: New Feature > Components: ML >Reporter: Joseph K. Bradley >Priority: Minor > > Tokenizer converts strings to lowercase automatically, but RegexTokenizer > does not. It would be nice to add an option to RegexTokenizer to convert to > lowercase. Proposal: > * call the Boolean Param "toLowercase" > * set default to false (so behavior does not change) > *Q*: Should conversion to lowercase happen before or after regex matching? > * Before: This is simpler. > * After: This gives the user full control since they can have the regex treat > upper/lower case differently. > --> I'd vote for conversion before matching. If a user needs full control, > they can convert to lowercase manually. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11069) Add RegexTokenizer option to convert to lowercase
[ https://issues.apache.org/jira/browse/SPARK-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954185#comment-14954185 ] yuhao yang commented on SPARK-11069: I'll try to do it and test with several cases. Updates will be posted here if anything unexpected found. > Add RegexTokenizer option to convert to lowercase > - > > Key: SPARK-11069 > URL: https://issues.apache.org/jira/browse/SPARK-11069 > Project: Spark > Issue Type: New Feature > Components: ML >Reporter: Joseph K. Bradley >Priority: Minor > > Tokenizer converts strings to lowercase automatically, but RegexTokenizer > does not. It would be nice to add an option to RegexTokenizer to convert to > lowercase. Proposal: > * call the Boolean Param "toLowercase" > * set default to false (so behavior does not change) > *Q*: Should conversion to lowercase happen before or after regex matching? > * Before: This is simpler. > * After: This gives the user full control since they can have the regex treat > upper/lower case differently. > --> I'd vote for conversion before matching. If a user needs full control, > they can convert to lowercase manually. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
