Hi Matthew, As this is pretty much my field, and at the risk of leading the list slightly off-topic, I thought I'd add some comments. Hopefully they might be of interest to others who are dealing with mapping too.
> The problem of scanning a piece of text and locating references to named > objects is often called entity recognition or extraction. There are other > steps required which transform the piece of text into some semantic > format. So the text > 'I live in London' > can be scanned for the string 'London' which is a mention of a place > with a specific lat/lon. Yes, but 'I live in' is, of course, not always followed by a place name. Checking the Bank of English (a 450 million word database of genuine examples of world English, based at Birmingham University in England), we find the phrase "I live in" occurs 984 times. The most common words after 'in' are (in order): a, the, london, hope, an, and, new, north, my, this, is, fear, such, i, one, leicester, it, what, england, west, east, south, britain To complicate matters still further, In 13 cases, 'London' is separated from 'in' by another word: I live in (North|South|East|Tooting,) London Is 'North London' the same place as 'London' and, if it is, why is 'New York' not the same place as 'York'? (ie. We have to rely on the computer realising that 'North' is an adjective, as it normally is, but conversely 'New' is part of a multi-word unit making up the Proper Noun, and not assuming its normal role of adjective.) > Many attempts at this problem in general only go half way and locate > mentions via patterns in text (e.g. the word 'in' followed by a capitalized > word suggests that the capitalized word is a location of some sort). What > we here are interested in is going all the way to the semantics (in this > case the lat lat and probably the unique identifier mentioned later by Barry). Personally, I would worry about the idea of using capitalised words as a basis for this. What we probably ought to be doing is using more complex word patterns to find places. We'd want any process we came up with to work in any language, including for example German, where all nouns, not just proper nouns, are capitalised. In speech we can't capitalise letters, and so it must be possible to uncover place names without relying on this, although this causes interesting issues with cases like "I live in Hope." (attested in the Bank of English), where 'Hope' is actually a place. (The computer would have to read forward, finding the references to 'village' etc. to be able to interpret 'Hope' as a place, just as we would in our minds if somebody said this phrase to us.) To take this issue a step further, I found this example in the Bank of English: "That's a pretty good description of San Francisco, actually. In fact, I live in what is probably one of the ultimate tourist destinations. Our beaches are lousy and there are lots of drawbacks, but on some levels San Francisco represents an unfulfilled dream..." Here, the place is obviously "San Francisco", but the place name is well separated from the node phrase "I live in". We really need to find ways to get computers to analyse this just as we do mentally, so that we could substitue "San Francisco" with anything, and have the computer realise it must be a place name, whether or not it is in a list of known places. As you suggest in talking about the tsunami, unknown places are just as important as known ones! On an unrelated note entirely, the members of the list might not yet have seen my prototype at http://www.domsmith.co.uk/bbcnews The aim was to produce a window that you can leave on your desktop and view news stories one by one. It has been implemented in XHTML+SMIL which sadly means it is only Internet Explorer 6 compatible at the moment, but was never really intended for anything more than to demonstrate what adding the SMIL timing elements to XHTML can do. Enjoy! Best wishes, Dominic. -- From: Dominic Smith MPhil postgraduate student in Corpus Linguistics, University of Birmingham. http://www.domsmith.co.uk
