This looks like a very nice approach for getting rid of the goo. I often advocate using words/phrases/ngrams that are highly predicted by the domain name as an alternative for removing boilerplate. That has the advantage that it doesn't require training text. In the case of wiki-pedia, this is not so useful because everything is in the same domain. The domain predictor trick will only work if the feature you are using for the input is not very content based. Thus, this can fail for small domain-focused sites or if you use a content laden URL for the task.
On Fri, Nov 13, 2009 at 10:36 AM, Ken Krugler <[email protected]>wrote: > Hi all, > > Another issue came up, about cleaning the text. > > One interested user suggested using nCleaner (see > http://www.lrec-conf.org/proceedings/lrec2008/pdf/885_paper.pdf) as a way > of tossing boilerplate text that skews text frequency data. > > Any thoughts on this? > > Thanks, > > -- Ken > > > On Nov 3, 2009, at 5:43am, Grant Ingersoll wrote: > > Might be of interest to all you Mahouts out there... >> http://bixolabs.com/datasets/public-terabyte-dataset-project/ >> >> Would be cool to get this converted over to our vector format so that we >> can cluster, etc. >> > > -------------------------------------------- > Ken Krugler > +1 530-210-6378 > http://bixolabs.com > e l a s t i c w e b m i n i n g > > > > > -- Ted Dunning, CTO DeepDyve
