Hi to all, I've an algortihm thats given below, can anybody help me implementing it. Any sort of suggestion will be appreciated. I've finished removing stop words, calculating term frequencies with Lucene. The rest of the part is not quite clear. I'm working only on the English part.
3.2.3 Extracting Representative Words After preprocessing the page, the system retains the most characteristic parts of the contents. Association rules are used to collect Chinese and English phrases as representative words. The approach can be outlined as follows: - Scan the documents to retrieve each Chinese and English word. Then, each individual Chinese word and English word can be distinguished. - ‧ Discard all stopwords, i.e., neither Chinese nor English word. Stopwords are high frequency words that are found in all texts and that do not contribute to the distinctive meaning of the text, for example prepositions and articles such as "in", "on" and "the". - ‧ Calculate the frequency of occurrence for each remaining word. - ‧ If the frequency of a word is less than two, it is discarded. Next, the association rule is used to compute the confidence value as follows. . Confidence (A ⇒B) = P(A∩B) / P(A) If P(A) is 1 and the co-occurrence, P(A∩B), is also 1, then the resulting confidence value must be 1. Obviously, it must be greater than the threshold Clearly, P(A) cannot be 0 as this will result in a division-by-zero [13]. - ‧ Assign the frequencies of the remaining words to each word in the document. For all the words in the document that already have been eliminated a frequency of 0 is assigned. - ‧ After all the inverse term frequencies of the document are identified and terms with a frequency less than 2 are discarded, the document is scanned again to calculate the confidence value of the cooccurrence. The word is not incorporated into the current document's phrase array if: 1) the confidence value is below the threshold, 2) the same word has already been assigned to document's phrase array, or 3) the subclass word is already assigned to the current document's phrase array. For example, assume the current document's phrase array includes the phrase "operating system". If the system subsequently processes the related phrase "operating", then the latter phrase is not added to the current document's phrase array. The steps needed to compute the confidence value of each word are as follows. (1) Fetch the previous two words until either term frequency is not equal to 0. (2) Calculate the confidence value of this term and the next word. This step is discussed further in a later section. (3) When the confidence value is less than the threshold (0.7), the system looks up whether the value of computing words is two. If it is, then skip and ignore it; otherwise, we will save this phrase, excluding the current word, into the phrase array and continue scanning the next word. A check is performed for each word to determine whether it exists in the phrase array or not. (4) If the confidence value of the phrase is larger than the threshold, the next word is fetched. If its term frequency is 0, then save the current phrase and the frequency; otherwise, steps 2 to 4 are repeated until the end of the document is reached. Thanx in Advance, -- சாய் Hari