2012/5/25, Philipp Singer <[email protected]>:
> Is it possible to easly include stemming to text feature extraction in
> scikit-learn?
>
> I know that nltk has an implementation of the Porter stemmer, but I do
> not want to change my whole
> text feature extraction process to nltl if possible. Would be nice if I
> could include that somehow easyly.
If you use the latest scikit-learn, v0.11, then you can pass a
callable to the vectorizer constructor called "analyzer". This is
passed the raw text and should perform tokenization and other
processing steps, e.g.
stemmer = nltk.stem.PorterStemmer()
def stemming_analyzer(text):
return [stemmer.stem(w) for w in nltk.word_tokenize(text.lower())
if re.match(r"\W", w)]
vectorizer = TfidfVectorizer(analyzer=stemming_analyzer)
See
http://scikit-learn.org/stable/modules/feature_extraction.html#customizing-the-vectorizer-classes
and the docstring for CountVectorizer or TfidfVectorizer for details.
HTH,
Lars
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
