We've been thinking about running some kind of a classifier against each book to select books with a high percentage of dirty OCR for some kind of special processing. Haven't quite figured out a multilingual feature set yet other than the punctuation/alphanumeric and character block ideas mentioned above.
I'm not sure I understand your suggestion. Since real word hapax legomenons are generally pretty common (maybe 40-60% of unique words) wouldn't using them as the "no" set provide mixed signals to the classifier? Tom Walter Underwood-2 wrote: > > > Hmm, how about a classifier? Common words are the "yes" training set, > hapax legomenons are the "no" set, and n-grams are the features. > > But why isn't the OCR program already doing this? > > wunder > > > > > > -- View this message in context: http://old.nabble.com/Cleaning-up-dirty-OCR-tp27840753p27871444.html Sent from the Solr - User mailing list archive at Nabble.com.
