J, On Mon, Dec 14, 2015 at 11:32 PM, J Rao <[email protected]> wrote:
> Right, the system would need some way to deal with words not in its > vocabulary (which I assume would always be limited initially). I think the > standard practice is to replace all such words with a unknown word token, > or better yet try to infer its meaning based on the words around it. > My present plan is to just add in new words on the fly, on the theory that the first new words will probably be more common than later new words. Not perfect, but good enough. Steve > > On 12/15/2015 1:19 PM, Steve Richfield wrote: > >> On Mon, Dec 14, 2015 at 5:39 PM, J Rao <[email protected] <mailto: >> [email protected]>> wrote: >> >> Is there a reason we couldn't just measure the frequency using a >> big corpus? >> >> >> Yea. To do that you must process the big corpus, which requires this >> information in advance to make the processing go faster. In short, this >> creates a sort of chicken-or-egg problem. Also, the frequencies CHANGE with >> time as interests wax and wane. >> >> Remember - it takes MANY occurrences of a word to establish its frequency >> with any accuracy. >> >> Note that the Wikipedia article I mentioned processed the entirety of >> Wikipedia. You might notice the little dashes at the bottom of the red >> line. Those come from words that occur just once in all of Wikipedia - >> probably just spelling errors. >> >> For really rare words, like neologies, there probably aren't enough >> occurrences on the entire Internet to establish frequency from observation. >> >> Fortunately, all **I** need to be able to do is compare the frequencies >> of short lists of words with the frequencies of other short lists of words, >> which hopefully won't be particularly sensitive to the effects I have been >> discussing. Even if there is an "error" in such a comparison, it would be >> between nearly equally occurring lists, so there would be little lost, >> other than a few milliseconds of computer time. >> >> /Steve/ >> // >> >> =============== >> >> On 12/15/2015 3:33 AM, Steve Richfield wrote: >> >> Hi, >> >> Just to make sure we are starting on the same page, see the >> Wikipedia article about Zipf's law at: >> >> https://en.wikipedia.org/wiki/Zipf's_law >> <https://en.wikipedia.org/wiki/Zipf%27s_law> >> <https://en.wikipedia.org/wiki/Zipf%27s_law> >> >> In summary, this provides a formula to convert word ranking >> into approximate frequency of occurrence, which is VERY useful >> in identifying least frequently used words to trigger >> processing, etc. >> >> Whatever formula someone might consider should sum to 1.0 over >> an infinite list of ranked words, as each word in a text >> appears SOMEWHERE in a ranking. However in reality, the story >> is more complex. >> >> Looking at words in Wikipedia, frequency goes as 0.07/N (which >> does NOT converge for an infinite list of words) out to 10,000 >> or so, and then drops off considerably more rapidly so that >> the millionth-ranked word is nearly 2 orders of magnitude less >> frequent than it would if the linear relationship had >> continued. Apparently no one has (yet) done the math to fit >> this to SOMETHING that converges to a total frequency of 1.0. >> >> I just HATE non-converging series. >> >> Note that a simple formula that fits the ENTIRE Wikipedia >> curve can be had by simply substituting the formula 700/(N^2) >> for N>10^4 >> >> OK, so where does the magic 10,000 come from? THAT appears to >> be our basic vocabulary, beyond which various subgroups add >> their own specialized vocabularies, explaining the rapid >> drop-off after 10,000 words. A corpus other than Wikipedia >> that is an amalgamation of many disparate subjects would >> doubtless have a very different "curve" out beyond 10,000. It >> looks to me like the 3,000 word basic vocabulary picked the >> wrong number - they should have gone for 10,000 words. >> >> This seems to also say a lot about language granularity - how >> finely we presume the construction of our universe to be. For >> those who think we are in some sort of simulation, this might >> say something about the precision of such a simulation, etc. >> >> This seems to also say a lot about how much would be needed by >> an AI/AGI text "understanding" system - "understanding" >> somewhere beyond 10^4 words to be broadly useful. >> >> Anyway - I saw some wisdom in these numbers, along with some >> mathematical shortfalls in the associated formulas that >> someone needs to be turn into equations that sum to 1.0 >> >> Thoughts? >> >> /Steve/ >> *AGI* | Archives >> <https://www.listbox.com/member/archive/303/=now> >> <https://www.listbox.com/member/archive/rss/303/26346070-1cd82ca6 >> > >> | Modify <https://www.listbox.com/member/?&;> Your >> Subscription [Powered by Listbox] <http://www.listbox.com> >> >> >> >> >> >> ------------------------------------------- >> AGI >> Archives: https://www.listbox.com/member/archive/303/=now >> RSS Feed: >> https://www.listbox.com/member/archive/rss/303/10443978-6f4c28ac >> Modify Your Subscription: https://www.listbox.com/member/?&; >> Powered by Listbox: http://www.listbox.com >> >> >> >> >> -- >> Full employment can be had with the stoke of a pen. Simply institute a >> six hour workday. That will easily create enough new jobs to bring back >> full employment. >> >> *AGI* | Archives <https://www.listbox.com/member/archive/303/=now> < >> https://www.listbox.com/member/archive/rss/303/26346070-1cd82ca6> | >> Modify <https://www.listbox.com/member/?&;> Your Subscription [Powered >> by Listbox] <http://www.listbox.com> >> >> > > > > ------------------------------------------- > AGI > Archives: https://www.listbox.com/member/archive/303/=now > RSS Feed: https://www.listbox.com/member/archive/rss/303/10443978-6f4c28ac > Modify Your Subscription: > https://www.listbox.com/member/?&; > Powered by Listbox: http://www.listbox.com > -- Full employment can be had with the stoke of a pen. Simply institute a six hour workday. That will easily create enough new jobs to bring back full employment. ------------------------------------------- AGI Archives: https://www.listbox.com/member/archive/303/=now RSS Feed: https://www.listbox.com/member/archive/rss/303/21088071-f452e424 Modify Your Subscription: https://www.listbox.com/member/?member_id=21088071&id_secret=21088071-58d57657 Powered by Listbox: http://www.listbox.com
