indeed, frequency usage is collection and use case dependant... Not directly your case, but the idea is the same.
We used this information in spell/typo-variations context to boost/penalize similarity, by dividing terms into a couple of freq based segments. Take an example: Maria - Very High Freq Marina - Very High Freq Mraia - Very Low Freq similarity(Maria, Marina) is by string distance measures very high, practically the same as (Maria, Mraia) but the likelihood that you mistyped Mraia is an order of magnitude higher than if you hit VHF-VHF pair. Point being, frequency hides a lot of semantics, and how you tune it, as Martin said, does not really matter, if it works. We also never found theory that formalize this, but it was logical, and it worked in practice. What you said, makes sense to me, especially for very big collections (or specialized domains with limited vocabulary...) the bigger the collection, the bigger "garbage density" in VLF domain (above certain size of the collection). If "vocabulary" in your collection is somehow limited, there is a size limit where most of new terms (VLF) are "crapterms". One could try to estimate how "saturated" a collection is... cheers, eks On Wed, Apr 13, 2011 at 9:36 PM, Marvin Humphrey <mar...@rectangular.com> wrote: > On Wed, Apr 13, 2011 at 01:01:09AM +0400, Earwin Burrfoot wrote: >> Excuse me for somewhat of an offtopic, but have anybody ever seen/used >> -subj- ? >> Something that looks like like http://dl.dropbox.com/u/920413/IDFplusplus.png >> Traditional log(N/x) tail, but when nearing zero freq, instead of >> going to +inf you do a nice round bump (with controlled >> height/location/sharpness) and drop down to -inf (or zero). > > I haven't used that technique, nor can I quote academic literature blessing > it. Nevertheless, what you're doing makes sense makes sense to me. > >> Rationale is that - most good, discriminating terms are found in at >> least a certain percentage of your documents, but there are lots of >> mostly unique crapterms, which at some collection sizes stop being >> strictly unique and with IDF's help explode your scores. > > So you've designed a heuristic that allows you to filter a certain kind of > noise. It sounds a lot like how people tune length normalization to adapt to > their document collections. Many tuning techniques are corpus-specific. > Whatever works, works! > > Marvin Humphrey > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
