Re: [Talk-ca] GeoBase Import - Where we stand now
Hi all, I took a look at the wiki, great chart! (i now need to sit down and analyse it a little, as i think i can clean up the pages a little.) On the canada page we have all the discussion -post the announcement. User: in7ane said he was able to import some data (i think it was in nova scotia, that was feb 07) created an import file called 'GML to OSM0.1.3a.zip' -i'll try to get in touch with that user, as the help is needed. Also user: dshpak knows how to make that python work & user: mungwell also can help. Hopefully Dale can also help, when his exams are done. (esp. regarding geogratis stuff) I also have others like gpsdave & gps teaching company who might be able to help too. So ya, there is help out there, but as we all know, its work on our free time; so if we can hold out for a bit -? (i'll post links to the users pages, as the relevent info now needs to be moved anyway) cheers, On 12/7/08, Sam Vekemans <[EMAIL PROTECTED]> wrote: > Cool, > I think the next step is to update the wiki (im a little behind now) :) > -cause we explored probably the best options available, and now know > some order to tasks :) > > I think the rest of the team IS around, but probably just busy with > school/work, and dont check the list regularly. > > Cheers, > Sam > ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] GeoBase Import - Where we stand now
Cool, I think the next step is to update the wiki (im a little behind now) :) -cause we explored probably the best options available, and now know some order to tasks :) I think the rest of the team IS around, but probably just busy with school/work, and dont check the list regularly. Cheers, Sam ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] GeoBase Import - Where we stand now
On Sun, 7 Dec 2008, Steve Singer wrote: My previous numbers were wrong, they excluded more data than they included. Looking at planet_osm_roads isn't correct, planet_osm_line with highway<>null seems to be what should have been using. > It looks like 245 of the 250k (4 digit) NTS tiles have at least 1 'road' in > OSM (see note 1). Make that 284 NTS tiles with at least one road > The Tofino BC is part of the 'Port Alberni' 250k tile 092F. It has 103 > roads and is probably a good divider for discussion. > > 8 Tiles have more than 1000 roads in OSM > (TORONTO,OTTAWA,QUEBEC,MONTREAL,KITCHENER,DETROIT,VANCOUVER) Actually 33 titles have more than 1000, 5 tiles have close to 10,000+. CALGARY and EDMONTON and BUFFALO should have been in the top 8. > > 40 tiles have more than 100 roads in OSM. About 100 tiles have 100 or more roads. > 105 tiles have 20 or more roads. Make that 180. I'm not sure what any of this means to how we want to proceed. Steve ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] GeoBase Import - Where we stand now
Hi all, I tried very hard to understand where you are going unfortunately I cannot. Did I miss a location where the method, scripts are explained ? Do you need help ? Did the method accept (if it exist) ? What is the plan ? Who are the participants ? I would like to help but at this stage how I can. Michel 2008/12/7 Sam Vekemans <[EMAIL PROTECTED]> > Thanks Steve for the breakdown :) > > I need to announce a couple points: > 1 Calgary sits smack in the middle of 4 tiles, and same with other > cities, so that estimate 8, needs to be taken in as 4 of the '40' > estimate. > And winnipeg could be also like that too. > > 2 (imo) when Dale started producing the tiles, he started with the 8 > or so 'popular' ones, (in our case, we can save these for later > -last), and made a call out requesting people ask for particular tiles > to work with. > As people started working, calls came in announcing errors. So with > some fixups (in the script) the next ones were better. > > 3 the technical process of converting, seems to be an automated, but > time intensive 1 person job, they will have the tiles and converted > files all on their computer, they will also know what tiles have been > loaded. > 4 how we all can help is in the manual messaging of the data, and > reporting all 'bugs' in the script. > > By starting with the ones that users are prepared to take 'ownership' > of the manual fixing, then once those tiles are 'imported & fixed' the > rest should be clear sailing. > > I thing that sounds like a plan. Don't you? > > Sam Vekemans > Across Canada Trails > > On 12/7/08, Steve Singer <[EMAIL PROTECTED]> wrote: > > On Sat, 6 Dec 2008, Steve Singer wrote: > > > >> Do we have a handle on: > >> > > > > To attempt to answer my own question. > > > >> a) How many tiles have 0 OSM roads (but do have at least 1 road in the > >> geobase nrn) > > > > It looks like 245 of the 250k (4 digit) NTS tiles have at least 1 'road' > in > > OSM (see note 1). > > > > I do not yet know how many NTS tiles have at least 1 road in geobase (I > > don't have all the data downloaded). > > > > > >> b) How many tiles have OSM coverage similar to Tofino BC (or any your > >> other > >> examples from above) , and how much manual effort does it take someone > in > >> JOSM to manually fix up the duplicate ways following a OSM import. > >> c) How many tiles (that have roads) are left over? > >> > > > > The Tofino BC is part of the 'Port Alberni' 250k tile 092F. It has 103 > > roads and is probably a good divider for discussion. > > > > 8 Tiles have more than 1000 roads in OSM > > (TORONTO,OTTAWA,QUEBEC,MONTREAL,KITCHENER,DETROIT,VANCOUVER) > > > > 40 tiles have more than 100 roads in OSM. > > > > 105 tiles have 20 or more roads. > > > > Breaking the analysis into the smaller 50k tiles shows that only 50 of > the > > smaller tiles have more than 100 roads. > > > > > > > > I think Dale pointed out that the NRN is segmented by province not NTS > tile. > > This is true but a province to too large of an area to deal with as a > > single 'work unit'. The NTS tiling system seems like a reasonable way of > > breaking things up (geobase distributes the larger datasets like the > hydro > > network by nts tile id). > > > > Notes: > > 1. 'road' means anything that the osm2pgsql program adds to the '_roads' > > table and has a non-null highway attribute. > > 2. This is based on recent (downloaded Dec06/08) canada.osm snapshot. > > > > Steve > > > > > > > > ___ > Talk-ca mailing list > Talk-ca@openstreetmap.org > http://lists.openstreetmap.org/listinfo/talk-ca > ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] GeoBase Import - Where we stand now
Thanks Steve for the breakdown :) I need to announce a couple points: 1 Calgary sits smack in the middle of 4 tiles, and same with other cities, so that estimate 8, needs to be taken in as 4 of the '40' estimate. And winnipeg could be also like that too. 2 (imo) when Dale started producing the tiles, he started with the 8 or so 'popular' ones, (in our case, we can save these for later -last), and made a call out requesting people ask for particular tiles to work with. As people started working, calls came in announcing errors. So with some fixups (in the script) the next ones were better. 3 the technical process of converting, seems to be an automated, but time intensive 1 person job, they will have the tiles and converted files all on their computer, they will also know what tiles have been loaded. 4 how we all can help is in the manual messaging of the data, and reporting all 'bugs' in the script. By starting with the ones that users are prepared to take 'ownership' of the manual fixing, then once those tiles are 'imported & fixed' the rest should be clear sailing. I thing that sounds like a plan. Don't you? Sam Vekemans Across Canada Trails On 12/7/08, Steve Singer <[EMAIL PROTECTED]> wrote: > On Sat, 6 Dec 2008, Steve Singer wrote: > >> Do we have a handle on: >> > > To attempt to answer my own question. > >> a) How many tiles have 0 OSM roads (but do have at least 1 road in the >> geobase nrn) > > It looks like 245 of the 250k (4 digit) NTS tiles have at least 1 'road' in > OSM (see note 1). > > I do not yet know how many NTS tiles have at least 1 road in geobase (I > don't have all the data downloaded). > > >> b) How many tiles have OSM coverage similar to Tofino BC (or any your >> other >> examples from above) , and how much manual effort does it take someone in >> JOSM to manually fix up the duplicate ways following a OSM import. >> c) How many tiles (that have roads) are left over? >> > > The Tofino BC is part of the 'Port Alberni' 250k tile 092F. It has 103 > roads and is probably a good divider for discussion. > > 8 Tiles have more than 1000 roads in OSM > (TORONTO,OTTAWA,QUEBEC,MONTREAL,KITCHENER,DETROIT,VANCOUVER) > > 40 tiles have more than 100 roads in OSM. > > 105 tiles have 20 or more roads. > > Breaking the analysis into the smaller 50k tiles shows that only 50 of the > smaller tiles have more than 100 roads. > > > > I think Dale pointed out that the NRN is segmented by province not NTS tile. > This is true but a province to too large of an area to deal with as a > single 'work unit'. The NTS tiling system seems like a reasonable way of > breaking things up (geobase distributes the larger datasets like the hydro > network by nts tile id). > > Notes: > 1. 'road' means anything that the osm2pgsql program adds to the '_roads' > table and has a non-null highway attribute. > 2. This is based on recent (downloaded Dec06/08) canada.osm snapshot. > > Steve > > > ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] GeoBase Import - Where we stand now
On Sat, 6 Dec 2008, Steve Singer wrote: > Do we have a handle on: > To attempt to answer my own question. > a) How many tiles have 0 OSM roads (but do have at least 1 road in the > geobase nrn) It looks like 245 of the 250k (4 digit) NTS tiles have at least 1 'road' in OSM (see note 1). I do not yet know how many NTS tiles have at least 1 road in geobase (I don't have all the data downloaded). > b) How many tiles have OSM coverage similar to Tofino BC (or any your other > examples from above) , and how much manual effort does it take someone in > JOSM to manually fix up the duplicate ways following a OSM import. > c) How many tiles (that have roads) are left over? > The Tofino BC is part of the 'Port Alberni' 250k tile 092F. It has 103 roads and is probably a good divider for discussion. 8 Tiles have more than 1000 roads in OSM (TORONTO,OTTAWA,QUEBEC,MONTREAL,KITCHENER,DETROIT,VANCOUVER) 40 tiles have more than 100 roads in OSM. 105 tiles have 20 or more roads. Breaking the analysis into the smaller 50k tiles shows that only 50 of the smaller tiles have more than 100 roads. I think Dale pointed out that the NRN is segmented by province not NTS tile. This is true but a province to too large of an area to deal with as a single 'work unit'. The NTS tiling system seems like a reasonable way of breaking things up (geobase distributes the larger datasets like the hydro network by nts tile id). Notes: 1. 'road' means anything that the osm2pgsql program adds to the '_roads' table and has a non-null highway attribute. 2. This is based on recent (downloaded Dec06/08) canada.osm snapshot. Steve ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] GeoBase Import - Where we stand now
Hi all, Thanks dale, so perhaps i need to ask the OpenStreetMap Foundation this question then. Does the foundation agree that the license of geogratis.ca is compatable with that of the openstreetmap foundation? If so, this would then open the door to a whole lot more data. Thanks, Sam Vekemans Across Canada Trails On 12/6/08, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > There seems to be some general confusion around on exactly what is > part of the Geobase dataset. (someone can correct me if I'm the one > confused). > > The Geobase road network is split by province, not NTS grid. All this > talk about grids. > > Next, most of the data on my maps comes not from Geobase, but from the > Geogratis database (specifically the NTDB). Only the road network > comes from geobase (not sure how clear this already was to everyone... > some of the talk in the previous e-mails has been implying otherwise...) > > From the outside looking in (I don't do much with OSM stuff), I have > a few questions. > > How complete is the OSM dataset currently? (How many km of roads in > Canada might be a good judge). If not very, then I'd say basically > wipe the whole road network and use Geobase as a start, and then if > possible transfer any additional parameters (street addresses/street > names/speed limits etc) from the OSM database. (I know this might not > be popular, as a lot of people have put a lot of work in to the data > that is there, but the Geobase dataset is pretty darned complete, so I > imagine you'd be overall a lot further ahead (close to 100% coverage) > doing this than trying to integrate the datasets. Updates/additions > could then be handled *much* easier (by continuing to associate the > NID with the data). Addition of new roads with no NID could still be > done, and later duplicates caused by syncing with the geobase dataset > (as new releases become available) would likely be quickly handled > manually as these would be in areas that contributors are interested > in, and hence caught quickly. > > > Anyways, back to the grind stone. Three major final exams next week, > so I better get to it! > > Dale > > > Quoting Steve Singer <[EMAIL PROTECTED]>: > >> On Sat, 6 Dec 2008, Michel Gilbert wrote: >> >>> I need some clarification about the parallel database (PostGIS). I am >>> sure I >>> see the whole picture of the Geobase Import process. Who is going to host >>> the change tracking PostGIS database ? How are we going to access it ? I >>> prefer a process model that uses only osm database. I don't see how we >>> are >>> going to maintain the PostGis database for users that access Potlatch or >>> JOSM. May be I have not understood the concept yet. >> >> I was thinking that the spatial database would only be used to generate >> the mappings between the OSM and geobase nodes. Once a OSM node was >> deemed to be a geobase node the live OSM database would be updated to >> contain the geobase_nid tag. You wouldn't keep the PostGIS database >> around, after the initial mapping process has done you will be able to >> get the 'mappings' from the OSM database by examining everything with a >> geobase_nid tag. >> >> PostGIS was only an example platform for generating automated mappings, >> you could probably do this in a JOSM plugin (that reads geobase files) >> and has geometric routines for determine if two lines a essentially the >> same. >> >> >>> Did we figure out the development platform ? The basic script to map the >>> road classes are written to work with JOSM ? or could it be something >>> else. >>> I work with FME (Safe Software) a canadian company in BC. It is a great >>> ETL >>> (Extract,Transform,Load) application. I know it is a commercial product >>> but >>> it could help for the task we have. FME read osm and PostGIS formats and >>> offers many tools to find change detections and manipulate attributes and >>> geometry. FME has a large community, may be even among osm. It could be >>> interesting to find out. >> >> This is still open. A lot depends on how much manual intervention is >> required in the process. If the process can be automated then I think >> we'd want scripts that can be run in a batch fashion (so not a JOSM >> plugin) but if each tile is going to need a fair amount of manual >> massaging then a JOSM plugin is more appealing. >> >> We also have to consider what people are willing and able to develop >> in. I wouldn't exclude FME on the basis that it is commercial it does >> restrict the pool of developers that could help with any scripting (and >> might have implications on where/how the import gets run) >>> >>> Michel >>> > > > > ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] GeoBase Import - Where we stand now
On Sat, Dec 6, 2008 at 5:15 PM, Michel Gilbert <[EMAIL PROTECTED]> wrote: > > However when i have done some tests to import Geobase > data into osm I did not use intermidiate storage. Here's what i have done: > > Read osm data (bounding box) > Read Geobase data in shape file (same bounding box) > Reproject in LCC (it is easier to manipulate data in x,y coordinates) > Compare both sources to identify Geobase segments that will be import > Tag Geobase segment as osm ways > Integrate the geobase to osm ways: > > snap geobase ways to osm ways (add nodes to existing osm way) > > Reproject to geographic coordinates > Generate osm nodes: > > existing osm nodes > new nodes added to existing osm way (geobase snapped) > new nodes for geobase segments > > Generate osm ways to include new osm nodes > Generate new osm ways that came from geobase > Generate osm file > Then use JOSM to upload the update file The question I have, is this: Is the import of Geobase data going to be a process that average OSM participant is going to be able to take part in? I would like to be able to participate, but have no clue how to do any of the above listed steps. Talk of starting in one single area and only proceeding once that tile is done would seem to be counterproductive. It is unlikely that participants from Newfoundland would be excited about helping map Tofino, nor would it be expected that participants from BC would still be working with the same enthusiasm once the import tile reached Torbay. Also trying to get multiple users all working on the same area concurrently can lead to some overlap and duplicated work. Allowing people to work on areas that are of the most interest to them is how OSM is being built currently, and it's working fairly well. If you let people work in the areas that they are interested in, they are more apt to do the work. If there's going to be any type of ongoing synchronization between Geobase, and OSM, we are going to need to keep Geobase ID tags intact. Checking for duplicate ways can be worked around by avoiding the issue during the initial import, but if there are Geobase updates that are to be imported in the future, there will have to be a way of determining if the existing nodes and ways have changed or not, and that will have to be using existing ID references. James VE6SRV ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] GeoBase Import - Where we stand now
2008/12/6 Steve Singer <[EMAIL PROTECTED]> > On Sat, 6 Dec 2008, Michel Gilbert wrote: > > I need some clarification about the parallel database (PostGIS). I am sure >> I >> see the whole picture of the Geobase Import process. Who is going to host >> the change tracking PostGIS database ? How are we going to access it ? I >> prefer a process model that uses only osm database. I don't see how we are >> going to maintain the PostGis database for users that access Potlatch or >> JOSM. May be I have not understood the concept yet. >> > > I was thinking that the spatial database would only be used to generate the > mappings between the OSM and geobase nodes. Once a OSM node was deemed to > be a geobase node the live OSM database would be updated to contain the > geobase_nid tag. You wouldn't keep the PostGIS database around, after the > initial mapping process has done you will be able to get the 'mappings' from > the OSM database by examining everything with a geobase_nid tag. > > PostGIS was only an example platform for generating automated mappings, you > could probably do this in a JOSM plugin (that reads geobase files) and has > geometric routines for determine if two lines a essentially the same. > Ok. I do not mind to work with spatial database such as PostGIS if it is needed for the import. However when i have done some tests to import Geobase data into osm I did not use intermidiate storage. Here's what i have done: 1. Read osm data (bounding box) 2. Read Geobase data in shape file (same bounding box) 3. Reproject in LCC (it is easier to manipulate data in x,y coordinates) 4. Compare both sources to identify Geobase segments that will be import 5. Tag Geobase segment as osm ways 6. Integrate the geobase to osm ways: 1. snap geobase ways to osm ways (add nodes to existing osm way) 7. Reproject to geographic coordinates 8. Generate osm nodes: 1. existing osm nodes 2. new nodes added to existing osm way (geobase snapped) 3. new nodes for geobase segments 9. Generate osm ways to include new osm nodes 10. Generate new osm ways that came from geobase 11. Generate osm file 12. Then use JOSM to upload the update file > > > Did we figure out the development platform ? The basic script to map the >> road classes are written to work with JOSM ? or could it be something >> else. >> I work with FME (Safe Software) a canadian company in BC. It is a great >> ETL >> (Extract,Transform,Load) application. I know it is a commercial product >> but >> it could help for the task we have. FME read osm and PostGIS formats and >> offers many tools to find change detections and manipulate attributes and >> geometry. FME has a large community, may be even among osm. It could be >> interesting to find out. >> > > This is still open. A lot depends on how much manual intervention is > required in the process. If the process can be automated then I think we'd > want scripts that can be run in a batch fashion (so not a JOSM plugin) but > if each tile is going to need a fair amount of manual massaging then a JOSM > plugin is more appealing. > > We also have to consider what people are willing and able to develop in. I > wouldn't exclude FME on the basis that it is commercial it does restrict the > pool of developers that could help with any scripting (and might have > implications on where/how the import gets run) Good. I am familiar fme. If i participate in the import I'll try to make generic fme workbenches so they'll be sharable among participants. > > >> Michel >> >> > ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] GeoBase Import - Where we stand now
There seems to be some general confusion around on exactly what is part of the Geobase dataset. (someone can correct me if I'm the one confused). The Geobase road network is split by province, not NTS grid. All this talk about grids. Next, most of the data on my maps comes not from Geobase, but from the Geogratis database (specifically the NTDB). Only the road network comes from geobase (not sure how clear this already was to everyone... some of the talk in the previous e-mails has been implying otherwise...) From the outside looking in (I don't do much with OSM stuff), I have a few questions. How complete is the OSM dataset currently? (How many km of roads in Canada might be a good judge). If not very, then I'd say basically wipe the whole road network and use Geobase as a start, and then if possible transfer any additional parameters (street addresses/street names/speed limits etc) from the OSM database. (I know this might not be popular, as a lot of people have put a lot of work in to the data that is there, but the Geobase dataset is pretty darned complete, so I imagine you'd be overall a lot further ahead (close to 100% coverage) doing this than trying to integrate the datasets. Updates/additions could then be handled *much* easier (by continuing to associate the NID with the data). Addition of new roads with no NID could still be done, and later duplicates caused by syncing with the geobase dataset (as new releases become available) would likely be quickly handled manually as these would be in areas that contributors are interested in, and hence caught quickly. Anyways, back to the grind stone. Three major final exams next week, so I better get to it! Dale Quoting Steve Singer <[EMAIL PROTECTED]>: > On Sat, 6 Dec 2008, Michel Gilbert wrote: > >> I need some clarification about the parallel database (PostGIS). I am sure I >> see the whole picture of the Geobase Import process. Who is going to host >> the change tracking PostGIS database ? How are we going to access it ? I >> prefer a process model that uses only osm database. I don't see how we are >> going to maintain the PostGis database for users that access Potlatch or >> JOSM. May be I have not understood the concept yet. > > I was thinking that the spatial database would only be used to generate > the mappings between the OSM and geobase nodes. Once a OSM node was > deemed to be a geobase node the live OSM database would be updated to > contain the geobase_nid tag. You wouldn't keep the PostGIS database > around, after the initial mapping process has done you will be able to > get the 'mappings' from the OSM database by examining everything with a > geobase_nid tag. > > PostGIS was only an example platform for generating automated mappings, > you could probably do this in a JOSM plugin (that reads geobase files) > and has geometric routines for determine if two lines a essentially the > same. > > >> Did we figure out the development platform ? The basic script to map the >> road classes are written to work with JOSM ? or could it be something else. >> I work with FME (Safe Software) a canadian company in BC. It is a great ETL >> (Extract,Transform,Load) application. I know it is a commercial product but >> it could help for the task we have. FME read osm and PostGIS formats and >> offers many tools to find change detections and manipulate attributes and >> geometry. FME has a large community, may be even among osm. It could be >> interesting to find out. > > This is still open. A lot depends on how much manual intervention is > required in the process. If the process can be automated then I think > we'd want scripts that can be run in a batch fashion (so not a JOSM > plugin) but if each tile is going to need a fair amount of manual > massaging then a JOSM plugin is more appealing. > > We also have to consider what people are willing and able to develop > in. I wouldn't exclude FME on the basis that it is commercial it does > restrict the pool of developers that could help with any scripting (and > might have implications on where/how the import gets run) >> >> Michel >> ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] GeoBase Import - Where we stand now
On Sat, 6 Dec 2008, Michel Gilbert wrote: > I need some clarification about the parallel database (PostGIS). I am sure I > see the whole picture of the Geobase Import process. Who is going to host > the change tracking PostGIS database ? How are we going to access it ? I > prefer a process model that uses only osm database. I don't see how we are > going to maintain the PostGis database for users that access Potlatch or > JOSM. May be I have not understood the concept yet. I was thinking that the spatial database would only be used to generate the mappings between the OSM and geobase nodes. Once a OSM node was deemed to be a geobase node the live OSM database would be updated to contain the geobase_nid tag. You wouldn't keep the PostGIS database around, after the initial mapping process has done you will be able to get the 'mappings' from the OSM database by examining everything with a geobase_nid tag. PostGIS was only an example platform for generating automated mappings, you could probably do this in a JOSM plugin (that reads geobase files) and has geometric routines for determine if two lines a essentially the same. > Did we figure out the development platform ? The basic script to map the > road classes are written to work with JOSM ? or could it be something else. > I work with FME (Safe Software) a canadian company in BC. It is a great ETL > (Extract,Transform,Load) application. I know it is a commercial product but > it could help for the task we have. FME read osm and PostGIS formats and > offers many tools to find change detections and manipulate attributes and > geometry. FME has a large community, may be even among osm. It could be > interesting to find out. This is still open. A lot depends on how much manual intervention is required in the process. If the process can be automated then I think we'd want scripts that can be run in a batch fashion (so not a JOSM plugin) but if each tile is going to need a fair amount of manual massaging then a JOSM plugin is more appealing. We also have to consider what people are willing and able to develop in. I wouldn't exclude FME on the basis that it is commercial it does restrict the pool of developers that could help with any scripting (and might have implications on where/how the import gets run) > > Michel > ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] GeoBase Import - Where we stand now
On Sat, 6 Dec 2008, Sam Vekemans wrote: > IF the prime purpose IS to complete the road network across Canada, then in > doing so this will cause issues. This is because, technically speaking. > The GeoBase road database IS ALREADY the complete road network across > Canada. Period. With future updates, this database, will ALWAYS be the > complete road and official road network for Canada. It's a federally funded > database, with the goal of being the complete network. > So .. no matter how we twist and turn it, it will still be complete. I think the purpose is closer to what you describe below (an editiable) but I am not sure I agree that the geobase data is always 'complete'. When new roads are built it can take time for those roads to get into geobase. During that time the geobase data is 'incomplete'. Also remember that different parts of the country are in different stages of 'completeness' the geobase site talks about this. > > Soo... However, if the prime purpose is to create a totally free and > editable map of the world, we must then treat GeoBase with the same level of > indifference that we do to user:Acrosscanadatrails. Just so happens that > there will be a few users who are importing the data, and have spent a real > long time gathering tracks and points, and converting the points into ways, > and labeling them, and is ready to start bringing in the data. > I agree data imported from geobase shouldn't be given any sort of special standing above what a normal user might enter (users are free to add any special mapping tags to the nodes as well). > > I propose that we start with tile 092F04 (Tofino, BC) as thats what i did, > and probably needs the basic script to get the road class right. > Or maybe 082E14 (Kelowna, BC) > or maybe the 030M04 (Hamilton, Stoney Creek, Ontario) but there is alot of > OSM coverage, so i think it would make sense to work with less OSM coverage > areas 1st... so to tweak the import script 1st. and to see what it's like > for fixing GeoBase data to conform with OSM standards. > ... or even 001K10 & 001K11? (south of Saint Johns NFLD) and importing all > the kinds of data available for it? Your referring to an 'import' script in the above. What import script, is this Jason's python script in the OSM svn repo, a modified version of that script, some other script or something that still needs to be written? Looking at the wiki I see three approaches we've been discussing 1. An approach using a spatial database to try and generate automatic mappings between OSM and geobase nodes 2. Import tile by tile excluding OSM areas with lost of data. The person that is manually importing each tile would be responsible for eliminating (or merging) the duplicate streets. 3. Use a JOSM plugin to display geobase data as a second layer and have people manually trace the geobase nodes into the OSM layer. Your referring to proceeding with (2) or am I misunderstanding. Do we have a handle on: a) How many tiles have 0 OSM roads (but do have at least 1 road in the geobase nrn) b) How many tiles have OSM coverage similar to Tofino BC (or any your other examples from above) , and how much manual effort does it take someone in JOSM to manually fix up the duplicate ways following a OSM import. c) How many tiles (that have roads) are left over? Tiles in group (a) can be proceed once we've agreed on a mapping of geobase to osm tags. I think it will be easier to obtain concensus to start moving forward with group (a) while people are still thinking over solutions to merging duplicates. The answer to the questions I pose in (b) might tell us if it is worthwhile pursing an automated mapping approach and if an approach requiring tile by tile human intervention is feasible. I suspect the major population centres will all be in group (c). > > Cheers, > Sam Vekemans > Across Canada Trails > Steve ___ Talk-ca mailing list Talk-ca@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-ca
Re: [Talk-ca] GeoBase Import - Where we stand now
2008/12/6 Sam Vekemans <[EMAIL PROTECTED]> > Ok, cool :) Lots of great discussion points.Again, I'll try > to summarize with this info, and hopefully setting somethings as a > foundation. (in story form) > > Basic questions: > > What do we want to accomplish with the OpenStreetMap project? > and > What do we want to accomplish from Geobase? > > Here's IMO plus adding in everyone else's views (to the best of my > ability) > > By adding features that we are not likely to be able add the map ourselves, > we can import GeoBase data. > > IF the prime purpose IS to complete the road network across Canada, then > in doing so this will cause issues. This is because, technically > speaking. The GeoBase road database IS ALREADY the complete road network > across Canada. Period. With future updates, this database, will ALWAYS be > the complete road and official road network for Canada. It's a federally > funded database, with the goal of being the complete network. > So .. no matter how we twist and turn it, it will still be complete. > > Soo... However, if the prime purpose is to create a totally free and > editable map of the world, we must then treat GeoBase with the same level of > indifference that we do to user:Acrosscanadatrails. Just so happens that > there will be a few users who are importing the data, and have spent a real > long time gathering tracks and points, and converting the points into ways, > and labeling them, and is ready to start bringing in the data. > > Just as user:Acrosscanadatrails did, when he started; He didn't know how > this whole thing worked, and so he started adding in roads. .. and just had > it as highway=secondary, and highway=road for those ways that he didn't now. > ... then he found out that having it as a secondary road, didn't quite match > what similar roads where in other towns, he changed it to > highway=residential, then someone else came along and changed the name from > James St. to James Street and HWY 4 to highway 4. > > So the point is; Is that we can go ahead and start the import, working at > it 1 tile at a time. And just learn as we go along. (use the chart and the > talk list to keep the discussion going) > > (i use User:Acrosscanadatrails as the example for me importing geobase > data) > > So for the areas where User:Acrosscanadatrails did alot of gps tracks and > ways, and converted them to ways and points, and has it all sitting waiting > for the import. Sadly :( the work he did was for nothing. :( as he found > out that a whole mapping team was just in the area and did it all already. > So user:Acrosscanadatrails, doesn't need to complain about it, as after > all, the GPS Tracks he did, DID get imported to the back-end database. ... > and all his points and ways he made got put aside and into the PostGIS > database, as after-all, he did take time to write the house numbers down. > But these numbers just need to be converted to a point along side the road > in the center of where the house is, or real close or converted to a square, > where the 1 side is right on the road. but anyways.. after > all no renderer's can read his hand writing from his note paper (BTW, I do > have bad hand writing :) ..) but he was smart enough to keep track of his > own numbers, so these numbers are listed with each point and way he made. > > > So after a while, when user:Acrosscanadatrails started to import roads; he > called them 'freeways' instead of 'moterways'. ... perhaps because > he's stubborn? and doesn't want to change his ways? ... > well fortunately Python script came along to help him translate and speak > OSM language. So the next tile import does the conversion already. > > Fortunatly, user:Acrosscanadatrails still keeps the cross reference number > he originally started with. So now user:Bikecanada comes along and imports > more data. and imports it slightly different, and tags it differently. .. > but thats ok, because other users will come along and fix it to how it > should be. > > So now user:Acrosscanadatrails went around a year later and added in more > roads (not thinking if any thing has been added) and goes around and just > takes note of all the new subdivisions that came up. Well, he'll just to > what he did before... upload his traces to the database, and upload his > points and ways he made to the PostGIS database. But this time, the script > which converts his previous imports is tweaked so it's much much smarter. > ... However, because he knows where his reference points where, he can only > add on extra features that would not be added on otherwise. So... he needs > to only add in the house numbers, and not add in the actual road. (or he can > add in the road class/surface, but only if it hasn't been done) As he knows > that there are other users who are around, and have already drawn in > cycleways, schools, parks in detail. as well as the local roads. > > All of the roads that have been drawn in by hand, are well within t