Github user MLnick commented on the pull request: https://github.com/apache/spark/pull/1671#issuecomment-51371929 Fair enough. Standard approach for same token appearing multiple times in a training example is additive, so it would act the same as HashingTF computation for text (and as a one-hot style encoder for categorical features, and just normally for real features). To deal with collisions, typically take signed features by either (1) apply two hashes, one of which determines sign (cf VW), or (2) use just one signed hash function and take sign of hash value as sign of feature (cf scikit-learn). The idea being that collisions cancel each other out. On Wed, Aug 6, 2014 at 7:01 PM, Xiangrui Meng <notificati...@github.com> wrote: > @MLnick <https://github.com/MLnick> FeatureHasher might be too general. > One thing it is not clear from its name is how to resolve conflicts: same > word appears more than once, or different words having the same hash value. > We can add FeatureHasher (or called HashingVectorizer) later. I think it > might be useful to keep HashingTF as a special case. > > â > Reply to this email directly or view it on GitHub > <https://github.com/apache/spark/pull/1671#issuecomment-51364477>. >
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org