The problem of scanning a piece of text and locating references to named objects is often called entity recognition or extraction. There are other steps required which transform the piece of text into some semantic format. So the text
'I live in London' can be scanned for the string 'London' which is a mention of a place with a specific lat/lon. Many attempts at this problem in general only go half way and locate mentions via patterns in text (e.g. the word 'in' followed by a capitalized word suggests that the capitalized word is a location of some sort). What we here are interested in is going all the way to the semantics (in this case the lat lat and probably the unique identifier mentioned later by Barry). As we want to go this far, I'm interested in using this database of names in an integrated manner with the scanning process. The approx 3MM non us locations (plus about 200k us locations) can easily be stored in memory in an manner efficient for lookup. However, there are lots of interesting/fun problems. For example, there are places with names like 'live', 'but', 'i', etc. Part of my motivation for this is to be able to not only catch mentions of common places like 'Paris' or 'California', but also unknown places like those affected by the tsunami in asia (I got into this due to working on blog analysis concerning the tsunami found here : tsunami.blogpulse.com). On 5/27/05, Duncan Barclay <[EMAIL PROTECTED]> wrote: > Hi, > > Is there any chance you could elaborate further on the idea about serving > as a "knowledge resource for location understanding in text"? If my > interpretation is correct, you are considering basically storing all the > data and retrieving bits of it when requested, which seems like a brilliant > idea, but could have some problems (730mb of data there, which could be a > problem technically). Assuming that is what you are saying, they it would > be an extremely useful resource and would work well with many of the > projects being developed on backstage.bbc.co.uk. > > Good luck with it, > Duncan > > > Matthew Hurst wrote: > I'm curious about the availability of this data. I have been using what I > guess is siimlar data found from starting out at > > http://geonames.usgs.gov/ > > and to save you all the navigation (though it is worth it as there is lots > of > data here), grab the file from > > http://earth-info.nga.mil/gns/html/cntry_files.html > > (i.e. the single compressed file) > > which as far as I can tell is still freely available and what is more, > maintained. It is some effort to deal with as there are many locations > with multiple names (you need to test for equality of coordinates to > determine that Tokyo is another name for Edo for example). > > It has 5MM lines in it, of which about 3MM are populated places (PPL > designations). > > Is this basically the same data? > > I've been working on an in memory representation of this data with the goal > of being able to serve as a knowledge resource for location understanding > in text as well as for the general task of looking up places near to some > position. > > Anyway - if anyone has clarification of the 'freeness' of this data, > I'd be interested > to hear about it. > > Matt Hurst > Senior Research Scientist > Intelliseek, Inc, > BlogPulse: www.blogpulse.com > > On 5/25/05, David Tattersall <[EMAIL PROTECTED]> wrote: > > > > Hi, > > This looks like a very good find - a possible future idea could be getting > bbc's forecasts and placing the appropriate weather symbols on a google map > (for those who can't quite get the new 3d maps!) > > If the bbc's RSS feed doesn't include lat/long (although the weather web > site does so perhaps the database isn't needed) then this set of figures > could come in useful. > > ps - new here :) I love the idea behind this. Keep up the good work. > > > ________________________________ > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Kosso > Sent: 25 May 2005 4:09 > To: [email protected] > Subject: Re: [backstage] World Coordinates - Finally Available. > > > i'll get this lot into a MySqldb later;) > > good find!! > > > > > On 5/25/05, Therion Ware <[EMAIL PROTECTED]> wrote: > > > Hi > > - Finally there. The database of all world cities can be found at: > > http://video.s-vid.com/~admin/geog/world.zip > > The file is in CSV format, of the form: > > iName cName CityName AdminDistrict LatLong > Africa / Middle East Tanzania Musasa Tanzania > > (General) > > > 003:21:00S031:33:00E > > About 24Mb. Unzips to about 193Mb. > > And includes the United States, which after much looking around, I > found - would that it were so easy to loose in real life...! > > There's 2,410,343 entries, and it's only population centres. There's > another 2 million or so sea mounts, geographical features and so on > omitted. If you really need these, let me know, and I'll see. > > I was thinking of setting up a MySQL backend that allows an HTTP > request where you specify a lat and long, and get back the nearest > place. If that'd be of use let me know here. > > As I originally said, I got the data free from the US geographical > survey back when they didn't charge for these things. What's amusing > is that Vietnam constitutes a unique region (Col 1). Presumably that > reflects the spirit of the them times to us today through inertia! > > Oh - another thing: projecting these coordinates on to a graphic map > is *not* a trivial thing for lots of different reasons. Spherical > coordinates in relation to your map image and all that... > > Anyway, have fun, and sniff, if you use them, think about crediting me > as in www.video2cd.co.uk as the source seeing as these days it'd cost > you money to get them! > > Best, > TW > > > -- > www.video2cd.co.uk <--- blatant self publicity > and unsubtle effort to flog stuff... > > > > > > > > > >
