Drew, Did you pick your whitelist using the LLR score? What is the kind of over-representation you're trying to prune out? DF will certainly help you remove "too common" bigrams, but that's not what you're looking for, is it?
-jake On Feb 16, 2010 8:29 AM, "Drew Farris" <[email protected]> wrote: I have a collection of about 800k bigrams from a corpus of 3.7m documents that I'm in the process of working with. I'm looking to determine an appropriate subset of these to use both as features for both an ML and an IR application. Specifically I'm considering white-listing a subset of these to use as features when building a classifier and separately as terms when building an index and doing query parsing. As a part of the earlier collocation discussion Ted mentioned that tests for over-representation could be used to identify dubious members of such a set. Does anyone have any pointers to discussions of how such a test could be implemented? Thanks, Drew
