[
https://issues.apache.org/jira/browse/SPARK-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073987#comment-16073987
]
yuhao yang edited comment on SPARK-11069 at 7/4/17 6:31 PM:
------------------------------------------------------------
[~levente.torok.ge] toLowerCase is set to true by default from 1.6+ to be
consistent with Tokenizer and accommodate the general user scenarios. The
change of behavior was documented in the release notes of 1.6.
https://spark.apache.org/releases/spark-release-1-6-0.html
val regexTokenizer = new RegexTokenizer()
*{color:red} .setToLowercase(false){color}*
was (Author: yuhaoyan):
[~levente.torok.ge] use
val regexTokenizer = new RegexTokenizer()
*{color:red} .setToLowercase(false){color}*
> Add RegexTokenizer option to convert to lowercase
> -------------------------------------------------
>
> Key: SPARK-11069
> URL: https://issues.apache.org/jira/browse/SPARK-11069
> Project: Spark
> Issue Type: New Feature
> Components: ML
> Reporter: Joseph K. Bradley
> Assignee: yuhao yang
> Priority: Minor
> Fix For: 1.6.0
>
>
> Tokenizer converts strings to lowercase automatically, but RegexTokenizer
> does not. It would be nice to add an option to RegexTokenizer to convert to
> lowercase. Proposal:
> * call the Boolean Param "toLowercase"
> * set default to false (so behavior does not change)
> *Q*: Should conversion to lowercase happen before or after regex matching?
> * Before: This is simpler.
> * After: This gives the user full control since they can have the regex treat
> upper/lower case differently.
> --> I'd vote for conversion before matching. If a user needs full control,
> they can convert to lowercase manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
