[Talk-us] python validation tools for OSM data?
Hey guys, I wonder if anyone knows of some python code to perform validations on OSM data. I'm hoping for some functionality like JOSM's validator tools but in python. I'm particularly interested in overlapping ways validation test. Thanks, Brian ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] parcel data next steps
On Thu, Feb 21, 2013 at 9:04 PM, Brian May b...@mapwise.com wrote: On 2/21/2013 7:27 PM, Brian Cavagnolo wrote: Hey guys, In a previous thread on parcel data, some people expressed interest in participating in creating some sort of open repository for parcel data. I was imagining a conference call or something to discuss next steps, but I think we can advance with email. I'm imagining that it makes sense to separate the data gathering process from the data standardization/import process. Regarding the data gathering, the main objective is to gather recent raw data, licensing terms, and meta data from jurisdictions in whatever form they make it available, organize it in a dumb directory structure. I was just going to set up an FTP (read-write) and HTTP (read-only) server to get this going. Are there any recommendations/opinions on a longer-term approach here? Custom webapp? Off-the-shelf webapp? Somebody mentioned a git repository. Regarding standardization/import, I was planning on setting up an empty instance of the rails port as a test bed. Then participating users could point JOSM and other tools at this alternative rails port to examine, edit, and import parcel data. We could also provide planet-style dumps and mapnik tiles. The idea is that we would have a safe place to screw up and learn. Does this sound like a reasonable direction? Oh, and I found this fantastic paper that some parcel data people (Abt Associates, Fairview Industries, Smart Data Strategies) recently put together for HUD [1] that examines many of the issues that they faced building a parcel database. Timely. Ciao, Brian [1] http://nationalcad.org/download/the-feasibility-of-developing-a-national-parcel-database-county-data-records-project-final-report/ ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us Hi Brian, I am interested in collaborating on this. So here's some thoughts: From my perspective (and I think others as mentioned in other email threads), the main thrust of utilizing parcels is a source of addresses based on parcel centroids where address points or buildings with addresses are not available. In addition, several people have mentioned they utilize parcels as an overlay to assist with digitizing. The current consensus is that parcels should not be imported as a whole. The current _majority_ is that parcels should not be imported ;-) I also think we need a little bit more sophisticated Data Catalog than a google spreadsheet. We need to capture more information and a spreadsheet gets a bit unwieldy after so may columns. I've got a prototype that I am working on getting out in the wild soon, but basically its a web form that people register to use and the info sits in a database. I'm with you on this. I think we can get going with Ian's existing spreadsheet, but I think we're going to eventually want a longer form capability. There seems to be some open source open data portal options out there (e.g., https://github.com/azavea/Open-Data-Catalog/). Also, Nancy von Meyer, one of the authors of that paper I posted a link to, (and as you mentioned, a long time advocate for a national parcel database) advised me of this inventory of parcel data that she and others have built up: http://www.bhgis.org/inventory/ ...of course it's read-only. But it's a good resource to browse around. And we could probably arrange more back-end access. A by-product of the effort to identify, catalog, gather raw data, etc. would be having a central location for storing raw parcel data that is not readily downloadable. For example, someone happens to have a copy of X county parcel data that they had to send a check for $25 to acquire, they received it on CD, and they would like to donate it to a central repository. This is assuming there are no restrictions on the data. It sounds like you're willing and able to donate disk and bandwidth to support this effort. I don't see a need to make a copy of data that is already sitting on the web. Good point about not duplicating the data that is readily available on the web. But one thing I've run into in the few cases that I've investigated is that explicit terms are often unavailable on those websites. So some outreach is required to confirm the terms before redistributing the data. And of course policies can change. So there's something to be said for saving off a copy of some data to a place where it is clearly guaranteed to be OSM compatible. As far as standardization/import and the rails server - I think this is not the right path to take. As mentioned above, we shouldn't be wholesale importing parcels. Now you could do some standardization of parcel data for example to render polygons by land use codes and show single family, multi-family, commercial, government, etc land use types as an overlay layer
Re: [Talk-us] parcel data next steps
On Mon, Feb 25, 2013 at 12:39 PM, SteveCoast st...@asklater.com wrote: Majority of what exactly? I think it's tough to put much credence in a couple of people on this mailing list vs. anyone who added data this month as statistically valid. Fair enough. I'll downgrade my statement from majority to concerns have been expressed. Ciao, Brian On Feb 25, 2013, at 12:17 PM, Brian Cavagnolo bcavagn...@gmail.com wrote: On Thu, Feb 21, 2013 at 9:04 PM, Brian May b...@mapwise.com wrote: On 2/21/2013 7:27 PM, Brian Cavagnolo wrote: Hey guys, In a previous thread on parcel data, some people expressed interest in participating in creating some sort of open repository for parcel data. I was imagining a conference call or something to discuss next steps, but I think we can advance with email. I'm imagining that it makes sense to separate the data gathering process from the data standardization/import process. Regarding the data gathering, the main objective is to gather recent raw data, licensing terms, and meta data from jurisdictions in whatever form they make it available, organize it in a dumb directory structure. I was just going to set up an FTP (read-write) and HTTP (read-only) server to get this going. Are there any recommendations/opinions on a longer-term approach here? Custom webapp? Off-the-shelf webapp? Somebody mentioned a git repository. Regarding standardization/import, I was planning on setting up an empty instance of the rails port as a test bed. Then participating users could point JOSM and other tools at this alternative rails port to examine, edit, and import parcel data. We could also provide planet-style dumps and mapnik tiles. The idea is that we would have a safe place to screw up and learn. Does this sound like a reasonable direction? Oh, and I found this fantastic paper that some parcel data people (Abt Associates, Fairview Industries, Smart Data Strategies) recently put together for HUD [1] that examines many of the issues that they faced building a parcel database. Timely. Ciao, Brian [1] http://nationalcad.org/download/the-feasibility-of-developing-a-national-parcel-database-county-data-records-project-final-report/ ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us Hi Brian, I am interested in collaborating on this. So here's some thoughts: From my perspective (and I think others as mentioned in other email threads), the main thrust of utilizing parcels is a source of addresses based on parcel centroids where address points or buildings with addresses are not available. In addition, several people have mentioned they utilize parcels as an overlay to assist with digitizing. The current consensus is that parcels should not be imported as a whole. The current _majority_ is that parcels should not be imported ;-) I also think we need a little bit more sophisticated Data Catalog than a google spreadsheet. We need to capture more information and a spreadsheet gets a bit unwieldy after so may columns. I've got a prototype that I am working on getting out in the wild soon, but basically its a web form that people register to use and the info sits in a database. I'm with you on this. I think we can get going with Ian's existing spreadsheet, but I think we're going to eventually want a longer form capability. There seems to be some open source open data portal options out there (e.g., https://github.com/azavea/Open-Data-Catalog/). Also, Nancy von Meyer, one of the authors of that paper I posted a link to, (and as you mentioned, a long time advocate for a national parcel database) advised me of this inventory of parcel data that she and others have built up: http://www.bhgis.org/inventory/ ...of course it's read-only. But it's a good resource to browse around. And we could probably arrange more back-end access. A by-product of the effort to identify, catalog, gather raw data, etc. would be having a central location for storing raw parcel data that is not readily downloadable. For example, someone happens to have a copy of X county parcel data that they had to send a check for $25 to acquire, they received it on CD, and they would like to donate it to a central repository. This is assuming there are no restrictions on the data. It sounds like you're willing and able to donate disk and bandwidth to support this effort. I don't see a need to make a copy of data that is already sitting on the web. Good point about not duplicating the data that is readily available on the web. But one thing I've run into in the few cases that I've investigated is that explicit terms are often unavailable on those websites. So some outreach is required to confirm the terms before redistributing the data. And of course policies can change. So there's
[Talk-us] parcel data next steps
Hey guys, In a previous thread on parcel data, some people expressed interest in participating in creating some sort of open repository for parcel data. I was imagining a conference call or something to discuss next steps, but I think we can advance with email. I'm imagining that it makes sense to separate the data gathering process from the data standardization/import process. Regarding the data gathering, the main objective is to gather recent raw data, licensing terms, and meta data from jurisdictions in whatever form they make it available, organize it in a dumb directory structure. I was just going to set up an FTP (read-write) and HTTP (read-only) server to get this going. Are there any recommendations/opinions on a longer-term approach here? Custom webapp? Off-the-shelf webapp? Somebody mentioned a git repository. Regarding standardization/import, I was planning on setting up an empty instance of the rails port as a test bed. Then participating users could point JOSM and other tools at this alternative rails port to examine, edit, and import parcel data. We could also provide planet-style dumps and mapnik tiles. The idea is that we would have a safe place to screw up and learn. Does this sound like a reasonable direction? Oh, and I found this fantastic paper that some parcel data people (Abt Associates, Fairview Industries, Smart Data Strategies) recently put together for HUD [1] that examines many of the issues that they faced building a parcel database. Timely. Ciao, Brian [1] http://nationalcad.org/download/the-feasibility-of-developing-a-national-parcel-database-county-data-records-project-final-report/ ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] parcel boundaries and associated data in OSM
Hey guys,
Thanks for the fantastic feedback. Instead of responding in detail, I
will attempt to organize these thoughts on the Parcel wiki page. In
any case, I think it is safe to say that there is no strong consensus
for uploading parcel data. That said, it sounds like there is strong
consensus that some incarnation of an open parcel map maintained
with OSM compatibility in mind would be useful. We (the UAL at
Berkeley) do have resources to contribute to this, including hardware,
bandwidth, and some grad student labor. Some folks on this thread
mentioned some ideas about possible next steps. I'm open to setting
up a conference call and maybe going from idea to strategy. Let me
know off-list if you're interested in participating and we'll try to
find a good time.
Old Topo Depot and any other Bay Area locals: I live in SF and am
commonly in Berkeley. Let's grab a beer or cup of coffee sometime
soonish. Alternatively, is there a regular OSM users group meeting or
something?
Ciao,
Brian
On Fri, Feb 15, 2013 at 10:36 AM, Serge Wroclawski emac...@gmail.com wrote:
On Thu, Feb 14, 2013 at 3:29 PM, Brian Cavagnolo bcavagn...@gmail.com wrote:
We really want a nationwide consolidated, standard parcel database to
build upon.
This idea is [obviously] inspired by OSM. And my immediate thought
was, Fun! Let's add parcel data to OSM! How do we do that?
We've started an Import Committee to help with such questions. I need
to schedule the next meeting, but I invite you to join us and help
shape the conversation. I'm facilitating the committee but the
opinions I'll express below are my own.
This
inquiry has of course led to numerous more detailed questions, the
most fundamental one, of course, being: Is parcel data welcome in OSM?
As you found out, this is a complex question that will depend on who you ask.
I've spent some time reading through the mailing list history. In
addition to gaining an appreciation for some of the issues regarding
the management of parcel data, I promptly learned that this is a
controversial question. For each claim that a consensus exists
against parcel data in OSM, a parcel data advocate seems to emerge.
This leads to debate, which seems to focus on a specific set of issues
that I have posed as specific questions below. I've also dusted off
and enriched the wiki page and associated talk page on the matter
(http://wiki.openstreetmap.org/wiki/Parcel). My hope is that people
can respond to these questions and we can reach a clear consensus on
{whether,what sort of,conditions under which} parcel data is welcome.
And of course feel free to bring up any issues that are not
represented in this list.
Finally, even if you believe that parcel
data does not belong in OSM, but that a nationwide open consolidated
parcel database would be useful (and possible:) I'm super interested
in this perspective.
I am of this view. Furthermore, I think that projects should Free
datasets intermixing this way, just as we do with topo data.
Is parcel data useful to OSM?
This is actually a three part question.
1. Is the data useful?
2. Is the data useful to OSM users?
3. Does the data belong in the OSM core dataset?
In my opinion, the answer to question 1 is yes. The question to two
and three are more subtle. I think the data is of use to the OSM
project, but does not belong in the OSM dataset. What I mean by that
is that we have tools (renderers, geocoders, routers, etc.) which may
want to use parcel data. I think that such tools should be able to.
But I think the data belongs alongside the OSM dataset, rather than
part of it.
So to make this clear: I think the data is useful, but would be more
useful to OSM users if it's not part of the OSM crowdsourced dataset.
Can parcel data possibly be kept up to date?
Parcel data (with very few exceptions) can't be manually surveyed by
amatuer mappers. Therefore it doesn't benefit from the OSM process of
survey, refinement, survey to provide additional detail and over-time
accuracy. Put in plain English How can a regular person, with no
additional information, survey the area to find mistakes in the survey
data? - the answer is that for the most part, they can't. Parcel data
is determined by a central authority.
So then if we had it in OSM's core crowdsourced database, we would
need a synchronization process. This is something many of us have
wanted, and worked on, for several years, and not come up with a
solution for.
But if we had the data as a database that could be integrated by
tools, then the data could be optionally rendered, used for geocoding,
used for routing, etc. in just the same way as OSM data, and OSM users
would get the benefit of current, up to date parcel data. That would
be a real win.
Does parcel data meet the on the ground verifiability criteria?
I don't see how, but I'm open to being shown that I'm wrong.
Can tools be adapted to accommodate parcel data
[Talk-us] parcel boundaries and associated data in OSM
Hey guys,
In my research group (the Urban Analytics Lab at Berkeley's Department
of City and Regional Planning), we use parcel data for land-use
projection, accessibility, and visualization. For example, over the
past couple of years we worked with regional government agencies here
in the Bay Area to put together a parcel-level urbansim
(http://www.urbansim.org) land-use model for regional planning
purposes. We've also developed a prototype 3D visualization tool
(http://www.urbansim.org/Documentation/UrbanVision) to visualize
parcel data, and published on using OSM data for accessibility
calculations [0]. If you poke around the Internet for references to
our director Prof. Paul Waddell you'll get the idea.
We really want a nationwide consolidated, standard parcel database to
build upon. Such products are available from numerous proprietary
data vendors who make it their business to routinely gather and
consolidate data from local government agencies around the country.
Of course these are often expensive and have restrictions on
redistribution. Our federal government has been trying for sometime
to create a nationwide public domain parcel database [1][2][3], but
this has not happened. Many states have managed to consolidate parcel
data (e.g., Massachusetts, Montana, New Jersey). This is very
helpful, but notable work is required to adapt tools or research from
one state to another. And our state along with many others has no
such offering. As a result, parcel data users for whom proprietary
sources are too restrictive or expensive go about manually gathering
the data from county agencies. If the application doesn't span county
lines, and if the county is open with their data, this may not be a
problem. But these two conditions are often not both met, driving a
more intensive data gathering effort. Such efforts are often
duplicated for different projects. We believe that this landscape of
use and parcel data availability represents an opportunity to form a
parcel data community concerned with building and maintaining an open
nationwide (global?) consolidated parcel database.
This idea is [obviously] inspired by OSM. And my immediate thought
was, Fun! Let's add parcel data to OSM! How do we do that? This
inquiry has of course led to numerous more detailed questions, the
most fundamental one, of course, being: Is parcel data welcome in OSM?
I've spent some time reading through the mailing list history. In
addition to gaining an appreciation for some of the issues regarding
the management of parcel data, I promptly learned that this is a
controversial question. For each claim that a consensus exists
against parcel data in OSM, a parcel data advocate seems to emerge.
This leads to debate, which seems to focus on a specific set of issues
that I have posed as specific questions below. I've also dusted off
and enriched the wiki page and associated talk page on the matter
(http://wiki.openstreetmap.org/wiki/Parcel). My hope is that people
can respond to these questions and we can reach a clear consensus on
{whether,what sort of,conditions under which} parcel data is welcome.
And of course feel free to bring up any issues that are not
represented in this list. Finally, even if you believe that parcel
data does not belong in OSM, but that a nationwide open consolidated
parcel database would be useful (and possible:) I'm super interested
in this perspective.
Is parcel data useful to OSM?
Can parcel data possibly be kept up to date?
Does parcel data meet the on the ground verifiability criteria?
Can tools be adapted to accommodate parcel data density?
Ciao,
Brian
[0]
http://onlinepubs.trb.org/onlinepubs/conferences/2012/4thITM/Papers-A/0117-62.pdf
[1] http://books.nap.edu/openbook.php?isbn=NI000560
[2] http://www.nap.edu/catalog.php?record_id=11978
[3] http://www.fas.org/sgp/crs/misc/R40717.pdf
___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] parcel boundaries and associated data in OSM
Ha. Totally missed this very recent discussion on the matter:
http://lists.openstreetmap.org/pipermail/talk-us/2012-December/010017.html
Nonetheless, feel free to respond. I'm going to go do some more homework ;-)
Ciao,
Brian
On Thu, Feb 14, 2013 at 12:29 PM, Brian Cavagnolo bcavagn...@gmail.com wrote:
Hey guys,
In my research group (the Urban Analytics Lab at Berkeley's Department
of City and Regional Planning), we use parcel data for land-use
projection, accessibility, and visualization. For example, over the
past couple of years we worked with regional government agencies here
in the Bay Area to put together a parcel-level urbansim
(http://www.urbansim.org) land-use model for regional planning
purposes. We've also developed a prototype 3D visualization tool
(http://www.urbansim.org/Documentation/UrbanVision) to visualize
parcel data, and published on using OSM data for accessibility
calculations [0]. If you poke around the Internet for references to
our director Prof. Paul Waddell you'll get the idea.
We really want a nationwide consolidated, standard parcel database to
build upon. Such products are available from numerous proprietary
data vendors who make it their business to routinely gather and
consolidate data from local government agencies around the country.
Of course these are often expensive and have restrictions on
redistribution. Our federal government has been trying for sometime
to create a nationwide public domain parcel database [1][2][3], but
this has not happened. Many states have managed to consolidate parcel
data (e.g., Massachusetts, Montana, New Jersey). This is very
helpful, but notable work is required to adapt tools or research from
one state to another. And our state along with many others has no
such offering. As a result, parcel data users for whom proprietary
sources are too restrictive or expensive go about manually gathering
the data from county agencies. If the application doesn't span county
lines, and if the county is open with their data, this may not be a
problem. But these two conditions are often not both met, driving a
more intensive data gathering effort. Such efforts are often
duplicated for different projects. We believe that this landscape of
use and parcel data availability represents an opportunity to form a
parcel data community concerned with building and maintaining an open
nationwide (global?) consolidated parcel database.
This idea is [obviously] inspired by OSM. And my immediate thought
was, Fun! Let's add parcel data to OSM! How do we do that? This
inquiry has of course led to numerous more detailed questions, the
most fundamental one, of course, being: Is parcel data welcome in OSM?
I've spent some time reading through the mailing list history. In
addition to gaining an appreciation for some of the issues regarding
the management of parcel data, I promptly learned that this is a
controversial question. For each claim that a consensus exists
against parcel data in OSM, a parcel data advocate seems to emerge.
This leads to debate, which seems to focus on a specific set of issues
that I have posed as specific questions below. I've also dusted off
and enriched the wiki page and associated talk page on the matter
(http://wiki.openstreetmap.org/wiki/Parcel). My hope is that people
can respond to these questions and we can reach a clear consensus on
{whether,what sort of,conditions under which} parcel data is welcome.
And of course feel free to bring up any issues that are not
represented in this list. Finally, even if you believe that parcel
data does not belong in OSM, but that a nationwide open consolidated
parcel database would be useful (and possible:) I'm super interested
in this perspective.
Is parcel data useful to OSM?
Can parcel data possibly be kept up to date?
Does parcel data meet the on the ground verifiability criteria?
Can tools be adapted to accommodate parcel data density?
Ciao,
Brian
[0]
http://onlinepubs.trb.org/onlinepubs/conferences/2012/4thITM/Papers-A/0117-62.pdf
[1] http://books.nap.edu/openbook.php?isbn=NI000560
[2] http://www.nap.edu/catalog.php?record_id=11978
[3] http://www.fas.org/sgp/crs/misc/R40717.pdf
___
Talk-us mailing list
Talk-us@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-us
