[Talk-us] python validation tools for OSM data?
Hey guys, I wonder if anyone knows of some python code to perform validations on OSM data. I'm hoping for some functionality like JOSM's validator tools but in python. I'm particularly interested in overlapping ways validation test. Thanks, Brian ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] parcel data next steps
On Thu, Feb 21, 2013 at 9:04 PM, Brian May b...@mapwise.com wrote: On 2/21/2013 7:27 PM, Brian Cavagnolo wrote: Hey guys, In a previous thread on parcel data, some people expressed interest in participating in creating some sort of open repository for parcel data. I was imagining a conference call or something to discuss next steps, but I think we can advance with email. I'm imagining that it makes sense to separate the data gathering process from the data standardization/import process. Regarding the data gathering, the main objective is to gather recent raw data, licensing terms, and meta data from jurisdictions in whatever form they make it available, organize it in a dumb directory structure. I was just going to set up an FTP (read-write) and HTTP (read-only) server to get this going. Are there any recommendations/opinions on a longer-term approach here? Custom webapp? Off-the-shelf webapp? Somebody mentioned a git repository. Regarding standardization/import, I was planning on setting up an empty instance of the rails port as a test bed. Then participating users could point JOSM and other tools at this alternative rails port to examine, edit, and import parcel data. We could also provide planet-style dumps and mapnik tiles. The idea is that we would have a safe place to screw up and learn. Does this sound like a reasonable direction? Oh, and I found this fantastic paper that some parcel data people (Abt Associates, Fairview Industries, Smart Data Strategies) recently put together for HUD [1] that examines many of the issues that they faced building a parcel database. Timely. Ciao, Brian [1] http://nationalcad.org/download/the-feasibility-of-developing-a-national-parcel-database-county-data-records-project-final-report/ ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us Hi Brian, I am interested in collaborating on this. So here's some thoughts: From my perspective (and I think others as mentioned in other email threads), the main thrust of utilizing parcels is a source of addresses based on parcel centroids where address points or buildings with addresses are not available. In addition, several people have mentioned they utilize parcels as an overlay to assist with digitizing. The current consensus is that parcels should not be imported as a whole. The current _majority_ is that parcels should not be imported ;-) I also think we need a little bit more sophisticated Data Catalog than a google spreadsheet. We need to capture more information and a spreadsheet gets a bit unwieldy after so may columns. I've got a prototype that I am working on getting out in the wild soon, but basically its a web form that people register to use and the info sits in a database. I'm with you on this. I think we can get going with Ian's existing spreadsheet, but I think we're going to eventually want a longer form capability. There seems to be some open source open data portal options out there (e.g., https://github.com/azavea/Open-Data-Catalog/). Also, Nancy von Meyer, one of the authors of that paper I posted a link to, (and as you mentioned, a long time advocate for a national parcel database) advised me of this inventory of parcel data that she and others have built up: http://www.bhgis.org/inventory/ ...of course it's read-only. But it's a good resource to browse around. And we could probably arrange more back-end access. A by-product of the effort to identify, catalog, gather raw data, etc. would be having a central location for storing raw parcel data that is not readily downloadable. For example, someone happens to have a copy of X county parcel data that they had to send a check for $25 to acquire, they received it on CD, and they would like to donate it to a central repository. This is assuming there are no restrictions on the data. It sounds like you're willing and able to donate disk and bandwidth to support this effort. I don't see a need to make a copy of data that is already sitting on the web. Good point about not duplicating the data that is readily available on the web. But one thing I've run into in the few cases that I've investigated is that explicit terms are often unavailable on those websites. So some outreach is required to confirm the terms before redistributing the data. And of course policies can change. So there's something to be said for saving off a copy of some data to a place where it is clearly guaranteed to be OSM compatible. As far as standardization/import and the rails server - I think this is not the right path to take. As mentioned above, we shouldn't be wholesale importing parcels. Now you could do some standardization of parcel data for example to render polygons by land use codes and show single family, multi-family, commercial, government, etc land use types as an overlay layer
Re: [Talk-us] parcel data next steps
On Mon, Feb 25, 2013 at 12:39 PM, SteveCoast st...@asklater.com wrote: Majority of what exactly? I think it's tough to put much credence in a couple of people on this mailing list vs. anyone who added data this month as statistically valid. Fair enough. I'll downgrade my statement from majority to concerns have been expressed. Ciao, Brian On Feb 25, 2013, at 12:17 PM, Brian Cavagnolo bcavagn...@gmail.com wrote: On Thu, Feb 21, 2013 at 9:04 PM, Brian May b...@mapwise.com wrote: On 2/21/2013 7:27 PM, Brian Cavagnolo wrote: Hey guys, In a previous thread on parcel data, some people expressed interest in participating in creating some sort of open repository for parcel data. I was imagining a conference call or something to discuss next steps, but I think we can advance with email. I'm imagining that it makes sense to separate the data gathering process from the data standardization/import process. Regarding the data gathering, the main objective is to gather recent raw data, licensing terms, and meta data from jurisdictions in whatever form they make it available, organize it in a dumb directory structure. I was just going to set up an FTP (read-write) and HTTP (read-only) server to get this going. Are there any recommendations/opinions on a longer-term approach here? Custom webapp? Off-the-shelf webapp? Somebody mentioned a git repository. Regarding standardization/import, I was planning on setting up an empty instance of the rails port as a test bed. Then participating users could point JOSM and other tools at this alternative rails port to examine, edit, and import parcel data. We could also provide planet-style dumps and mapnik tiles. The idea is that we would have a safe place to screw up and learn. Does this sound like a reasonable direction? Oh, and I found this fantastic paper that some parcel data people (Abt Associates, Fairview Industries, Smart Data Strategies) recently put together for HUD [1] that examines many of the issues that they faced building a parcel database. Timely. Ciao, Brian [1] http://nationalcad.org/download/the-feasibility-of-developing-a-national-parcel-database-county-data-records-project-final-report/ ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us Hi Brian, I am interested in collaborating on this. So here's some thoughts: From my perspective (and I think others as mentioned in other email threads), the main thrust of utilizing parcels is a source of addresses based on parcel centroids where address points or buildings with addresses are not available. In addition, several people have mentioned they utilize parcels as an overlay to assist with digitizing. The current consensus is that parcels should not be imported as a whole. The current _majority_ is that parcels should not be imported ;-) I also think we need a little bit more sophisticated Data Catalog than a google spreadsheet. We need to capture more information and a spreadsheet gets a bit unwieldy after so may columns. I've got a prototype that I am working on getting out in the wild soon, but basically its a web form that people register to use and the info sits in a database. I'm with you on this. I think we can get going with Ian's existing spreadsheet, but I think we're going to eventually want a longer form capability. There seems to be some open source open data portal options out there (e.g., https://github.com/azavea/Open-Data-Catalog/). Also, Nancy von Meyer, one of the authors of that paper I posted a link to, (and as you mentioned, a long time advocate for a national parcel database) advised me of this inventory of parcel data that she and others have built up: http://www.bhgis.org/inventory/ ...of course it's read-only. But it's a good resource to browse around. And we could probably arrange more back-end access. A by-product of the effort to identify, catalog, gather raw data, etc. would be having a central location for storing raw parcel data that is not readily downloadable. For example, someone happens to have a copy of X county parcel data that they had to send a check for $25 to acquire, they received it on CD, and they would like to donate it to a central repository. This is assuming there are no restrictions on the data. It sounds like you're willing and able to donate disk and bandwidth to support this effort. I don't see a need to make a copy of data that is already sitting on the web. Good point about not duplicating the data that is readily available on the web. But one thing I've run into in the few cases that I've investigated is that explicit terms are often unavailable on those websites. So some outreach is required to confirm the terms before redistributing the data. And of course policies can change. So there's
[Talk-us] parcel data next steps
Hey guys, In a previous thread on parcel data, some people expressed interest in participating in creating some sort of open repository for parcel data. I was imagining a conference call or something to discuss next steps, but I think we can advance with email. I'm imagining that it makes sense to separate the data gathering process from the data standardization/import process. Regarding the data gathering, the main objective is to gather recent raw data, licensing terms, and meta data from jurisdictions in whatever form they make it available, organize it in a dumb directory structure. I was just going to set up an FTP (read-write) and HTTP (read-only) server to get this going. Are there any recommendations/opinions on a longer-term approach here? Custom webapp? Off-the-shelf webapp? Somebody mentioned a git repository. Regarding standardization/import, I was planning on setting up an empty instance of the rails port as a test bed. Then participating users could point JOSM and other tools at this alternative rails port to examine, edit, and import parcel data. We could also provide planet-style dumps and mapnik tiles. The idea is that we would have a safe place to screw up and learn. Does this sound like a reasonable direction? Oh, and I found this fantastic paper that some parcel data people (Abt Associates, Fairview Industries, Smart Data Strategies) recently put together for HUD [1] that examines many of the issues that they faced building a parcel database. Timely. Ciao, Brian [1] http://nationalcad.org/download/the-feasibility-of-developing-a-national-parcel-database-county-data-records-project-final-report/ ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] parcel boundaries and associated data in OSM
Hey guys, Thanks for the fantastic feedback. Instead of responding in detail, I will attempt to organize these thoughts on the Parcel wiki page. In any case, I think it is safe to say that there is no strong consensus for uploading parcel data. That said, it sounds like there is strong consensus that some incarnation of an open parcel map maintained with OSM compatibility in mind would be useful. We (the UAL at Berkeley) do have resources to contribute to this, including hardware, bandwidth, and some grad student labor. Some folks on this thread mentioned some ideas about possible next steps. I'm open to setting up a conference call and maybe going from idea to strategy. Let me know off-list if you're interested in participating and we'll try to find a good time. Old Topo Depot and any other Bay Area locals: I live in SF and am commonly in Berkeley. Let's grab a beer or cup of coffee sometime soonish. Alternatively, is there a regular OSM users group meeting or something? Ciao, Brian On Fri, Feb 15, 2013 at 10:36 AM, Serge Wroclawski emac...@gmail.com wrote: On Thu, Feb 14, 2013 at 3:29 PM, Brian Cavagnolo bcavagn...@gmail.com wrote: We really want a nationwide consolidated, standard parcel database to build upon. This idea is [obviously] inspired by OSM. And my immediate thought was, Fun! Let's add parcel data to OSM! How do we do that? We've started an Import Committee to help with such questions. I need to schedule the next meeting, but I invite you to join us and help shape the conversation. I'm facilitating the committee but the opinions I'll express below are my own. This inquiry has of course led to numerous more detailed questions, the most fundamental one, of course, being: Is parcel data welcome in OSM? As you found out, this is a complex question that will depend on who you ask. I've spent some time reading through the mailing list history. In addition to gaining an appreciation for some of the issues regarding the management of parcel data, I promptly learned that this is a controversial question. For each claim that a consensus exists against parcel data in OSM, a parcel data advocate seems to emerge. This leads to debate, which seems to focus on a specific set of issues that I have posed as specific questions below. I've also dusted off and enriched the wiki page and associated talk page on the matter (http://wiki.openstreetmap.org/wiki/Parcel). My hope is that people can respond to these questions and we can reach a clear consensus on {whether,what sort of,conditions under which} parcel data is welcome. And of course feel free to bring up any issues that are not represented in this list. Finally, even if you believe that parcel data does not belong in OSM, but that a nationwide open consolidated parcel database would be useful (and possible:) I'm super interested in this perspective. I am of this view. Furthermore, I think that projects should Free datasets intermixing this way, just as we do with topo data. Is parcel data useful to OSM? This is actually a three part question. 1. Is the data useful? 2. Is the data useful to OSM users? 3. Does the data belong in the OSM core dataset? In my opinion, the answer to question 1 is yes. The question to two and three are more subtle. I think the data is of use to the OSM project, but does not belong in the OSM dataset. What I mean by that is that we have tools (renderers, geocoders, routers, etc.) which may want to use parcel data. I think that such tools should be able to. But I think the data belongs alongside the OSM dataset, rather than part of it. So to make this clear: I think the data is useful, but would be more useful to OSM users if it's not part of the OSM crowdsourced dataset. Can parcel data possibly be kept up to date? Parcel data (with very few exceptions) can't be manually surveyed by amatuer mappers. Therefore it doesn't benefit from the OSM process of survey, refinement, survey to provide additional detail and over-time accuracy. Put in plain English How can a regular person, with no additional information, survey the area to find mistakes in the survey data? - the answer is that for the most part, they can't. Parcel data is determined by a central authority. So then if we had it in OSM's core crowdsourced database, we would need a synchronization process. This is something many of us have wanted, and worked on, for several years, and not come up with a solution for. But if we had the data as a database that could be integrated by tools, then the data could be optionally rendered, used for geocoding, used for routing, etc. in just the same way as OSM data, and OSM users would get the benefit of current, up to date parcel data. That would be a real win. Does parcel data meet the on the ground verifiability criteria? I don't see how, but I'm open to being shown that I'm wrong. Can tools be adapted to accommodate parcel data
[Talk-us] parcel boundaries and associated data in OSM
Hey guys, In my research group (the Urban Analytics Lab at Berkeley's Department of City and Regional Planning), we use parcel data for land-use projection, accessibility, and visualization. For example, over the past couple of years we worked with regional government agencies here in the Bay Area to put together a parcel-level urbansim (http://www.urbansim.org) land-use model for regional planning purposes. We've also developed a prototype 3D visualization tool (http://www.urbansim.org/Documentation/UrbanVision) to visualize parcel data, and published on using OSM data for accessibility calculations [0]. If you poke around the Internet for references to our director Prof. Paul Waddell you'll get the idea. We really want a nationwide consolidated, standard parcel database to build upon. Such products are available from numerous proprietary data vendors who make it their business to routinely gather and consolidate data from local government agencies around the country. Of course these are often expensive and have restrictions on redistribution. Our federal government has been trying for sometime to create a nationwide public domain parcel database [1][2][3], but this has not happened. Many states have managed to consolidate parcel data (e.g., Massachusetts, Montana, New Jersey). This is very helpful, but notable work is required to adapt tools or research from one state to another. And our state along with many others has no such offering. As a result, parcel data users for whom proprietary sources are too restrictive or expensive go about manually gathering the data from county agencies. If the application doesn't span county lines, and if the county is open with their data, this may not be a problem. But these two conditions are often not both met, driving a more intensive data gathering effort. Such efforts are often duplicated for different projects. We believe that this landscape of use and parcel data availability represents an opportunity to form a parcel data community concerned with building and maintaining an open nationwide (global?) consolidated parcel database. This idea is [obviously] inspired by OSM. And my immediate thought was, Fun! Let's add parcel data to OSM! How do we do that? This inquiry has of course led to numerous more detailed questions, the most fundamental one, of course, being: Is parcel data welcome in OSM? I've spent some time reading through the mailing list history. In addition to gaining an appreciation for some of the issues regarding the management of parcel data, I promptly learned that this is a controversial question. For each claim that a consensus exists against parcel data in OSM, a parcel data advocate seems to emerge. This leads to debate, which seems to focus on a specific set of issues that I have posed as specific questions below. I've also dusted off and enriched the wiki page and associated talk page on the matter (http://wiki.openstreetmap.org/wiki/Parcel). My hope is that people can respond to these questions and we can reach a clear consensus on {whether,what sort of,conditions under which} parcel data is welcome. And of course feel free to bring up any issues that are not represented in this list. Finally, even if you believe that parcel data does not belong in OSM, but that a nationwide open consolidated parcel database would be useful (and possible:) I'm super interested in this perspective. Is parcel data useful to OSM? Can parcel data possibly be kept up to date? Does parcel data meet the on the ground verifiability criteria? Can tools be adapted to accommodate parcel data density? Ciao, Brian [0] http://onlinepubs.trb.org/onlinepubs/conferences/2012/4thITM/Papers-A/0117-62.pdf [1] http://books.nap.edu/openbook.php?isbn=NI000560 [2] http://www.nap.edu/catalog.php?record_id=11978 [3] http://www.fas.org/sgp/crs/misc/R40717.pdf ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] parcel boundaries and associated data in OSM
Ha. Totally missed this very recent discussion on the matter: http://lists.openstreetmap.org/pipermail/talk-us/2012-December/010017.html Nonetheless, feel free to respond. I'm going to go do some more homework ;-) Ciao, Brian On Thu, Feb 14, 2013 at 12:29 PM, Brian Cavagnolo bcavagn...@gmail.com wrote: Hey guys, In my research group (the Urban Analytics Lab at Berkeley's Department of City and Regional Planning), we use parcel data for land-use projection, accessibility, and visualization. For example, over the past couple of years we worked with regional government agencies here in the Bay Area to put together a parcel-level urbansim (http://www.urbansim.org) land-use model for regional planning purposes. We've also developed a prototype 3D visualization tool (http://www.urbansim.org/Documentation/UrbanVision) to visualize parcel data, and published on using OSM data for accessibility calculations [0]. If you poke around the Internet for references to our director Prof. Paul Waddell you'll get the idea. We really want a nationwide consolidated, standard parcel database to build upon. Such products are available from numerous proprietary data vendors who make it their business to routinely gather and consolidate data from local government agencies around the country. Of course these are often expensive and have restrictions on redistribution. Our federal government has been trying for sometime to create a nationwide public domain parcel database [1][2][3], but this has not happened. Many states have managed to consolidate parcel data (e.g., Massachusetts, Montana, New Jersey). This is very helpful, but notable work is required to adapt tools or research from one state to another. And our state along with many others has no such offering. As a result, parcel data users for whom proprietary sources are too restrictive or expensive go about manually gathering the data from county agencies. If the application doesn't span county lines, and if the county is open with their data, this may not be a problem. But these two conditions are often not both met, driving a more intensive data gathering effort. Such efforts are often duplicated for different projects. We believe that this landscape of use and parcel data availability represents an opportunity to form a parcel data community concerned with building and maintaining an open nationwide (global?) consolidated parcel database. This idea is [obviously] inspired by OSM. And my immediate thought was, Fun! Let's add parcel data to OSM! How do we do that? This inquiry has of course led to numerous more detailed questions, the most fundamental one, of course, being: Is parcel data welcome in OSM? I've spent some time reading through the mailing list history. In addition to gaining an appreciation for some of the issues regarding the management of parcel data, I promptly learned that this is a controversial question. For each claim that a consensus exists against parcel data in OSM, a parcel data advocate seems to emerge. This leads to debate, which seems to focus on a specific set of issues that I have posed as specific questions below. I've also dusted off and enriched the wiki page and associated talk page on the matter (http://wiki.openstreetmap.org/wiki/Parcel). My hope is that people can respond to these questions and we can reach a clear consensus on {whether,what sort of,conditions under which} parcel data is welcome. And of course feel free to bring up any issues that are not represented in this list. Finally, even if you believe that parcel data does not belong in OSM, but that a nationwide open consolidated parcel database would be useful (and possible:) I'm super interested in this perspective. Is parcel data useful to OSM? Can parcel data possibly be kept up to date? Does parcel data meet the on the ground verifiability criteria? Can tools be adapted to accommodate parcel data density? Ciao, Brian [0] http://onlinepubs.trb.org/onlinepubs/conferences/2012/4thITM/Papers-A/0117-62.pdf [1] http://books.nap.edu/openbook.php?isbn=NI000560 [2] http://www.nap.edu/catalog.php?record_id=11978 [3] http://www.fas.org/sgp/crs/misc/R40717.pdf ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us