Re: [Talk-ca] [OSM Sherbrooke discussion] Re: Geobase / Canvec import workshop in Sherbrooke?

[Sam Vekemans]

Cool,
thanks for the clarification. And yuppee!
 You understood!
That would be great if you could maintain that 'portal' batch script :)

just say the programs i need to install and the buttons to press,
...so i can sit back & watch the batch do its 'magic'.


The technical editing (scripting) just needs to be done by 1 person
(at a time) but we all know that :)

fyi, im now working 4 night shifts a week (mainly because my part of
the conversion script is just about done) :) ... And i can start
planning for my trip Across Canada next year! :-)


actually it'll be in the next few days, but im giving it longer,
so i can 'cleanup' my own script. (sams_babel-to-english.exe')

cheers,
Sam

On 11/7/09, Frank Steggink  wrote:
> Hi Sam,
>
> Waiting for Nov. 21st isn't a problem. The idea I had to discuss the
> conversion (both Geobase and Canvec) is still fresh, and we haven't set
> a date yet, nor settled on the details. Since I have a bit of experience
> with the Geobase conversion (with the new method to copy over data), and
> especially with the cleaning up part, I want to share it. That way
> others might benefit from it, and help us getting it all done. It would
> also be a good occasion to discuss / demonstrate the Canvec import for
> our region (roughly NTS tiles 021E and 021L). Michel has already thought
> about this more.
>
> Regarding the conversion script: which script are you talking about?
> Earlier this week I mentioned I wanted to rewrite your batch files
> (which call Ian's shp-to-osm) to Python, so they can be used on other
> OS's, and hopefully they'll also be easier to maintain. That's all. I
> haven't given other aspects much thought, before my e-mail this morning.
> I won't be available as the maintainer for such a script for a number of
> reasons, so I'll only do the "port" of your batch files. Hopefully this
> is clear now.
>
> By the way, instead of working on the Python-version of your batch
> files, I imported some Geobase data. Just one sheet left, and I'll put
> it on hold for a while (although that remains to be seen ;) ).
>
> Frank
>
> Sam Vekemans wrote:
>> Hi,
>> Would you be able to hold out until nov 21st?
>>
>> At that time, myself will have the shp-to-osm java script out of beta,
>> and available to 'build-from' using python/java FME tools.
>>
>> Frank has offered to be the main person to maintain the conversion
>> script once im done with it.
>>
>> And Yes, i agree, the actual act of importing the .osm files (made by
>> the scripts) can be done with a 'couch mapping party'
>>
>> Cheers,
>> Sam
>>
>>
>> On 11/7/09, Michel Gilbert  wrote:
>>
>>> Hi Frank, Yves and all,
>>>
>>> I like the idea to have a mapping party in Sherbrooke that may attract
>>> people of other regions. The mapping party, considering we are in
>>> November,
>>> could concentrate on importing CanVec data for OSM. So, a Mapping Party
>>> with
>>> no field work, only desktop work in front of a fireplace.  We would need
>>> a
>>> place in Sherbrooke with good wireless connections, good coffees and a
>>> room
>>> where we can talk and help each other. I think that "La Brulerie" on
>>> Wellington meets all the requirements.
>>>
>>> In order to make the event possible we will need experts that could
>>> explain
>>> importing tools (python, java, fme) and editing tools (josm, potlatch).
>>> Any
>>> volunteers ? Also, we need available data to import. My preference would
>>> be
>>> to have data that include all the CanVec feature classes. At my
>>> knowledge, I
>>> don't think such data can be generated rigth now. There are tools in
>>> development to generate such data but are not ready for importation. We
>>> (Daniel and I) do plan to work on a FME CanVec Import tool this winter
>>> but
>>> the tool won't be available in short terms.
>>>
>>> If we have Canvec OSM file ready to import, I will participate to
>>> organize
>>> the "Sherbrooke Couch Mapping Party". Here are some ideas for the agenda:
>>>- Explain translation between CanVec features and OSM tags
>>>- Training on Importing tools
>>>- Explain OSM xml file
>>>- Training on JOSM
>>>- Importing session. Each participant take a region and import data.
>>>
>>> Let me know if anyone would like to participate at such event. We will
>>> keep
>>> on discussing to put in place the missing pieces.
>>>
>>> Cheers,
>>>
>>> Michel
>>>
>


-- 
Twitter: @Acrosscanada
Blog:  http://Acrosscanadatrails.blogspot.com
Facebook: http://www.facebook.com/sam.vekemans
OpenStreetMap IRC: http://irc.openstreetmap.org
@Acrosscanadatrails

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [OSM Sherbrooke discussion] Re: Geobase / Canvec import workshop in Sherbrooke?

[Frank Steggink]

Hi Sam,

Waiting for Nov. 21st isn't a problem. The idea I had to discuss the 
conversion (both Geobase and Canvec) is still fresh, and we haven't set 
a date yet, nor settled on the details. Since I have a bit of experience 
with the Geobase conversion (with the new method to copy over data), and 
especially with the cleaning up part, I want to share it. That way 
others might benefit from it, and help us getting it all done. It would 
also be a good occasion to discuss / demonstrate the Canvec import for 
our region (roughly NTS tiles 021E and 021L). Michel has already thought 
about this more.

Regarding the conversion script: which script are you talking about? 
Earlier this week I mentioned I wanted to rewrite your batch files 
(which call Ian's shp-to-osm) to Python, so they can be used on other 
OS's, and hopefully they'll also be easier to maintain. That's all. I 
haven't given other aspects much thought, before my e-mail this morning. 
I won't be available as the maintainer for such a script for a number of 
reasons, so I'll only do the "port" of your batch files. Hopefully this 
is clear now.

By the way, instead of working on the Python-version of your batch 
files, I imported some Geobase data. Just one sheet left, and I'll put 
it on hold for a while (although that remains to be seen ;) ).

Frank

Sam Vekemans wrote:
> Hi,
> Would you be able to hold out until nov 21st?
>
> At that time, myself will have the shp-to-osm java script out of beta,
> and available to 'build-from' using python/java FME tools.
>
> Frank has offered to be the main person to maintain the conversion
> script once im done with it.
>
> And Yes, i agree, the actual act of importing the .osm files (made by
> the scripts) can be done with a 'couch mapping party'
>
> Cheers,
> Sam
>
>
> On 11/7/09, Michel Gilbert  wrote:
>   
>> Hi Frank, Yves and all,
>>
>> I like the idea to have a mapping party in Sherbrooke that may attract
>> people of other regions. The mapping party, considering we are in November,
>> could concentrate on importing CanVec data for OSM. So, a Mapping Party with
>> no field work, only desktop work in front of a fireplace.  We would need a
>> place in Sherbrooke with good wireless connections, good coffees and a room
>> where we can talk and help each other. I think that "La Brulerie" on
>> Wellington meets all the requirements.
>>
>> In order to make the event possible we will need experts that could explain
>> importing tools (python, java, fme) and editing tools (josm, potlatch).  Any
>> volunteers ? Also, we need available data to import. My preference would be
>> to have data that include all the CanVec feature classes. At my knowledge, I
>> don't think such data can be generated rigth now. There are tools in
>> development to generate such data but are not ready for importation. We
>> (Daniel and I) do plan to work on a FME CanVec Import tool this winter but
>> the tool won't be available in short terms.
>>
>> If we have Canvec OSM file ready to import, I will participate to organize
>> the "Sherbrooke Couch Mapping Party". Here are some ideas for the agenda:
>>- Explain translation between CanVec features and OSM tags
>>- Training on Importing tools
>>- Explain OSM xml file
>>- Training on JOSM
>>- Importing session. Each participant take a region and import data.
>>
>> Let me know if anyone would like to participate at such event. We will keep
>> on discussing to put in place the missing pieces.
>>
>> Cheers,
>>
>> Michel
>> 

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] canvec-to-osm 0.9.6 now available

[Sam Vekemans]

Hi ya,

OK, so your all aware, a few bugs im working on and will be done by nov 21.

-i'll fix the 'named_features.oso to be correct
-i'll remove ALL 'inner' tags
-i'll tidy the rules.txt files so they are shorter
-i'll re-work the readme.txt file so its clear as mud.

Thanks everyone for your enthuasm & patience

btw, i'd recommend that the folks who know how, finish the geobase2osm
process, (for eastern Canada & other provinces) BEFORE the 'CanVec
Train' roles all over the countryside :-)

cheers,
Sam

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [OSM Sherbrooke discussion] Re: Geobase / Canvec import workshop in Sherbrooke?

[Sam Vekemans]

Hi,
Would you be able to hold out until nov 21st?

At that time, myself will have the shp-to-osm java script out of beta,
and available to 'build-from' using python/java FME tools.

Frank has offered to be the main person to maintain the conversion
script once im done with it.

And Yes, i agree, the actual act of importing the .osm files (made by
the scripts) can be done with a 'couch mapping party'

Cheers,
Sam


On 11/7/09, Michel Gilbert  wrote:
> Hi Frank, Yves and all,
>
> I like the idea to have a mapping party in Sherbrooke that may attract
> people of other regions. The mapping party, considering we are in November,
> could concentrate on importing CanVec data for OSM. So, a Mapping Party with
> no field work, only desktop work in front of a fireplace.  We would need a
> place in Sherbrooke with good wireless connections, good coffees and a room
> where we can talk and help each other. I think that "La Brulerie" on
> Wellington meets all the requirements.
>
> In order to make the event possible we will need experts that could explain
> importing tools (python, java, fme) and editing tools (josm, potlatch).  Any
> volunteers ? Also, we need available data to import. My preference would be
> to have data that include all the CanVec feature classes. At my knowledge, I
> don't think such data can be generated rigth now. There are tools in
> development to generate such data but are not ready for importation. We
> (Daniel and I) do plan to work on a FME CanVec Import tool this winter but
> the tool won't be available in short terms.
>
> If we have Canvec OSM file ready to import, I will participate to organize
> the "Sherbrooke Couch Mapping Party". Here are some ideas for the agenda:
>- Explain translation between CanVec features and OSM tags
>- Training on Importing tools
>- Explain OSM xml file
>- Training on JOSM
>- Importing session. Each participant take a region and import data.
>
> Let me know if anyone would like to participate at such event. We will keep
> on discussing to put in place the missing pieces.
>
> Cheers,
>
> Michel
>
>
>
> 2009/11/6 Frank Steggink 
>
>> Yves Moisan wrote:
>>
>>> Hi Frank,
>>>
>>> Sorry for top posting, but I just want the ball rolling on the OSM
>>> Sherbrooke list.  Michel was thinking of a Sherbrooke OSM meeting end
>>> November.  I think the training you are proposing could be part of that
>>> meeting, although it is probably a bit technical for the average mapper.
>>> Michel, what do you think ?
>>>
>>> Yves
>>>
>>>
>>>
 Hi guys,

 I just had an idea, but because I don't know if it is a good one, I'll
  share it with you. Would it be a good idea to organize a workshop to
  import Geobase and Canvec data?

 Today I was contacted by Oxalin, who has been editing in Granby  lately,
 and he mentioned that he and a friend are willing to help with  the
 import
 as well (at least for St-Jean-sur-Richelieu). Perhaps there  are also
 some
 people from Montreal interested. There is a large  concentration of
 users in
 Sherbrooke, and because it is somewhat in  between Montreal and Quebec,
 I
 think it would be the most obvious  location.

 Regarding the workshop: I could explain how to import the roads, and
  edit / make connections with JOSM. We could also discuss the import of
  Canvec and NHN (hydro) data, because both Daniel and I are interested
 in
 it. Maybe Michel can explain how he imported the data through FME,
 although
 I'm not sure if this is relevant, because not everyone has  access to
 it.

 About NHN import: I think that users from the Montreal area should  also
 pick up the gauntlet. There is a whole lot of data to be  imported. I
 also
 have no idea about the current data quality, but I  expect this to be a
 huge
 undertaking. (And what I've done so far pales  in comparison with that.)
 Anyways, they'll also benefit from OSM :)

 If you think this would be a good idea, what would be the term to
  organize such an event? A couple of hours, together with a small
 mapping
 party (proposal: gathering and entering addresses) should be  enough.
 And of
 course a dinner at the end of the day :) For me it  won't be a problem
 to
 come from Quebec, but I would prefer doing this  before the winter
 really
 sets in. The winter is too cold and snowy to  venture outdoors anyways,
 so
 what would be better than to spend our  time importing data in our
 beloved
 Open Street Map?

 Cheers,

 Frank

>>> Hi Yves,
>>
>> What kind of meeting are you considering? Is it a mapping party, or just
>> an
>> event planned in an evening. In the latter case it won't be possible for
>> me
>> to attend, because of the distance. When the idea is to hold a mapping
>> party
>> (on a day in the weekend), I think it can be combined very well. For
>> example, we

Re: [Talk-ca] [OSM Sherbrooke discussion] Re: Geobase / Canvec import workshop in Sherbrooke?

[Michel Gilbert]

Hi Frank, Yves and all,

I like the idea to have a mapping party in Sherbrooke that may attract
people of other regions. The mapping party, considering we are in November,
could concentrate on importing CanVec data for OSM. So, a Mapping Party with
no field work, only desktop work in front of a fireplace.  We would need a
place in Sherbrooke with good wireless connections, good coffees and a room
where we can talk and help each other. I think that "La Brulerie" on
Wellington meets all the requirements.

In order to make the event possible we will need experts that could explain
importing tools (python, java, fme) and editing tools (josm, potlatch).  Any
volunteers ? Also, we need available data to import. My preference would be
to have data that include all the CanVec feature classes. At my knowledge, I
don't think such data can be generated rigth now. There are tools in
development to generate such data but are not ready for importation. We
(Daniel and I) do plan to work on a FME CanVec Import tool this winter but
the tool won't be available in short terms.

If we have Canvec OSM file ready to import, I will participate to organize
the "Sherbrooke Couch Mapping Party". Here are some ideas for the agenda:
   - Explain translation between CanVec features and OSM tags
   - Training on Importing tools
   - Explain OSM xml file
   - Training on JOSM
   - Importing session. Each participant take a region and import data.

Let me know if anyone would like to participate at such event. We will keep
on discussing to put in place the missing pieces.

Cheers,

Michel



2009/11/6 Frank Steggink 

> Yves Moisan wrote:
>
>> Hi Frank,
>>
>> Sorry for top posting, but I just want the ball rolling on the OSM
>> Sherbrooke list.  Michel was thinking of a Sherbrooke OSM meeting end
>> November.  I think the training you are proposing could be part of that
>> meeting, although it is probably a bit technical for the average mapper.
>> Michel, what do you think ?
>>
>> Yves
>>
>>
>>
>>> Hi guys,
>>>
>>> I just had an idea, but because I don't know if it is a good one, I'll
>>>  share it with you. Would it be a good idea to organize a workshop to
>>>  import Geobase and Canvec data?
>>>
>>> Today I was contacted by Oxalin, who has been editing in Granby  lately,
>>> and he mentioned that he and a friend are willing to help with  the import
>>> as well (at least for St-Jean-sur-Richelieu). Perhaps there  are also some
>>> people from Montreal interested. There is a large  concentration of users in
>>> Sherbrooke, and because it is somewhat in  between Montreal and Quebec, I
>>> think it would be the most obvious  location.
>>>
>>> Regarding the workshop: I could explain how to import the roads, and
>>>  edit / make connections with JOSM. We could also discuss the import of
>>>  Canvec and NHN (hydro) data, because both Daniel and I are interested  in
>>> it. Maybe Michel can explain how he imported the data through FME,  although
>>> I'm not sure if this is relevant, because not everyone has  access to it.
>>>
>>> About NHN import: I think that users from the Montreal area should  also
>>> pick up the gauntlet. There is a whole lot of data to be  imported. I also
>>> have no idea about the current data quality, but I  expect this to be a huge
>>> undertaking. (And what I've done so far pales  in comparison with that.)
>>> Anyways, they'll also benefit from OSM :)
>>>
>>> If you think this would be a good idea, what would be the term to
>>>  organize such an event? A couple of hours, together with a small  mapping
>>> party (proposal: gathering and entering addresses) should be  enough. And of
>>> course a dinner at the end of the day :) For me it  won't be a problem to
>>> come from Quebec, but I would prefer doing this  before the winter really
>>> sets in. The winter is too cold and snowy to  venture outdoors anyways, so
>>> what would be better than to spend our  time importing data in our beloved
>>> Open Street Map?
>>>
>>> Cheers,
>>>
>>> Frank
>>>
>> Hi Yves,
>
> What kind of meeting are you considering? Is it a mapping party, or just an
> event planned in an evening. In the latter case it won't be possible for me
> to attend, because of the distance. When the idea is to hold a mapping party
> (on a day in the weekend), I think it can be combined very well. For
> example, we could start an hour earlier in the morning, and I can explain a
> number of things about the import. The mapping party itself could start
> afterwards, so then people who do not care about the import, don't have to
> hear all of this stuff. I only mentioned "a couple of hours" because I was
> not aware you were thinking about organizing something yourself, and someone
> from farther away would not come if there was not a mapping party or
> something else going on.
>
> Right now the import of Geobase and Canvec data in Quebec is gaining
> momentum, so this is an excellent occasion to get more people jumping in :)
> Daniel already mentioned that he and

Re: [Talk-ca] [Imports] canvec-to-osm 0.9.6 now available

[James A. Treacy]

On Sat, Nov 07, 2009 at 11:10:59AM -0500, Frank Steggink wrote:
> This actually means that the NTS tiles are too big to handle all at
> once. Maybe we should decide to use smaller working areas, by
> splitting up the NTS tiles in 4 x 4 subtiles. A disadvantage is that
> features will have to be split up (if this can be done
> automatically), or they have to be duplicated in the subtiles.

Is there a reason that features crossing a tile boundary need to
be split? If not, then features that cross the right or top edge
can be left out and they will be included in the adjacent tile.
There are some exceptional cases, for example features that cross an
entire tile, but there are ways of dealing with them if this is ever
implemented.

-- 
James Treacy
tre...@debian.org

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] canvec-to-osm 0.9.6 now available

[Ian Dees]

On Sat, Nov 7, 2009 at 9:59 AM, James A. Treacy  wrote:

> On Sat, Nov 07, 2009 at 09:14:58AM -0600, Ian Dees wrote:
> > Tags appears on the inner ways because you have an "inner" rule in the
> > rules.txt file. If you remove those lines from the rules.txt, you should
> end
> > up with no tags on inner ways of multipolygons. If this doesn't happen,
> then
> > it's a bug and I will fix it.
>
> When you say 'no tags' do you really mean zero?
>
> What about when (as in our case) the source data requires attribution
> tags. Is it ok to duplicate those on the inner way? Without them it is
> not clear where the data came from.
>

I was just pointing out why there are inner tags when you run shp-to-osm.
Whether or not you want those tags to be there is another question.
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] canvec-to-osm 0.9.6 now available

[Ian Dees]

On Sat, Nov 7, 2009 at 9:48 AM, James A. Treacy  wrote:

>
> There are cases where the same way exists in different data sets but
> with different uses. A good example is that islands in the waterbody
> data are often repeated in the wooded data (if the island happens to
> be wooded, which happens a lot). Importing one set after another leads
> to duplicated ways. This case is time consuming to fix because you
> have to reconcile two ways and a relation. This same issue arises with
> wetlands. Probably others too but these are the ones I've encountered
> most often.
>
>
This is the desired effect. Technically there should be one multipolygon
defining the waterway (or lake) and another multipolygon for the wooded area
(with separate ways? or maybe the wooded area's outer way should be the
lake's inner way?).

Keep in mind that removing duplicate nodes/ways in all cases is bad ... in
some cases there actually are two features at the exact same point.
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] canvec-to-osm 0.9.6 now available

[Ian Dees]

On Sat, Nov 7, 2009 at 8:59 AM, Frank Steggink wrote:

> Sam, can you give some additional clarification what your intentions are?
> I'm afraid I'm not following them well. When you mentioning removing
> duplicate nodes and relations, it looks as if you intend to create a script
> which does some post-processing. Is that correct? I haven't started anything
> in that area. (I actually still need to start with the Python version of
> your batch script, but I'm going to work on that today.)
>

Are these duplicate nodes/relations being created by the converter or are
they in the source data?


> Now we're talking on this: in shp-to-osm (Java) tags are now put properly
> on the multipolygon relationship. They also still appear on the inner
> polygons (mentioned to Ian already), but that should be fixed.
>

Tags appears on the inner ways because you have an "inner" rule in the
rules.txt file. If you remove those lines from the rules.txt, you should end
up with no tags on inner ways of multipolygons. If this doesn't happen, then
it's a bug and I will fix it.


> Since shp-to-osm is called for one feature type at a time, there are some
> new challenges when multiple feature types are involved. I guess you've been
> thinking about that already. Duplicate nodes will become an issue when you
> have for example a residential area with an adjacent wooded area (assuming
> that the boundaries are matching exactly). It will be difficult to deal with
> this. I'm not sure if it would be technically possible to adjust shp-to-osm
> for that, but the result will be that the files will become huge. They
> already have to be split up for certain feature types, and I don't think it
> is possible to use the same set of IDs over multiple output files.
>

I do have a "relationificator" plugin started for shp-to-osm that will
attempt to solve this problem by converting exactly-overlapping edges into
relations and delete duplicate primitives. If there's a strong need for it I
can continue to work on it.


> From what I understand about the upload process (and someone please correct
> me if this isn't right), the OSM server will return new ID numbers for any
> nodes, ways, and relationships uploaded. In the OSM files generated by Ian,
> and also when you're editing in JOSM yourself, temporary IDs are assigned.
> They have a negative value, which indicates that these objects don't exist
> on the server. So, this means that, after you have uploaded file00.osm, and
> you open file01.osm, JOSM or the server do no longer remember to what
> objects any IDs are _referring_ to, if those objects are not _defined_ in
> the same file.


> The same issue is going on with multipolygon relationships, where a part of
> the ways are reused. This can only happen if everything is defined in the
> same file. And such a file will be way too large to upload safely to the
> server. Recently I noticed that if you want to create/update/delete about
> 10k objects the server is going to "act difficult".


I normally upload my NHD changesets with 40k-50k changes in each upload
without problem. It takes an hour or so to apply (depending on server load),
but it works without error.


> Regarding relationships, and reuse of the geometry: I think that we have
> not only to remove duplicate nodes, but also split up ways, otherwise the
> JOSM validator will complain about overlapping ways. A way can be used in
> multiple relationships.
>

This would also happen with the relationificator.


> A third thing which might need to be resolved are map features which cross
> the boundary of the NTS tiles. Do we want to merge them? If these features
> have the same Geobase metadata (ID, etc.), then it shouldn't be a big
> problem, otherwise we need to decide whether we prefer to keep the metadata,
> or if we want to have merged features.
>
> All of this means we can't do anything to clean up the data. Sure we can,
> but this can only be done after an initial upload to the server. That way we
> can still apply any logic to deal with duplicate nodes, reuse of features in
> multiple relationships, and merging features. The script will have to work
> live on the server: download an area, do the cleanup, and upload. In such
> case I think it would be the safest (and required!) that the script only
> does the download and the cleanup, and that a human verifies the result
> before upload. If we're implementing such cleanup, it needs to be executed
> as soon as possible after the upload, because sometimes users are very quick
> to make changes to freshly uploaded data.
>

It's relatively dangerous to upload these "dirty" OSM files to the server
and the apply changes later. If someone makes a change to a single node in
your data, then you suddenly can't make that automated change.
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] canvec-to-osm 0.9.6 now available

[Ian Dees]

On Nov 6, 2009, at 9:28 PM, Sam Vekemans  
 wrote:

>
> This python script is yet to be made 'canvec2osm.py' its open to
> anyone to make, and i recommend that who ever does, its ONLY 1 person
> who is in charge of maintaining the script.

Why are you creating another shp to osm converter? Is there something  
the existing tools don't do? I thought you were using shp-to-osm? What  
changed? 

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] canvec-to-osm 0.9.6 now available

[Ian Dees]

On Sat, Nov 7, 2009 at 9:56 AM, Frank Steggink wrote:

> I do have a "relationificator" plugin started for shp-to-osm that will
>> attempt to solve this problem by converting exactly-overlapping edges into
>> relations and delete duplicate primitives. If there's a strong need for it I
>> can continue to work on it.
>>
>
> That sounds very interesting. I guess this only works within the single
> shape file which is being converted, correct? What is the behavior if a file
> has to be split up, because it becomes too large?


As with the shp-to-osm "glomming" plugin, the entire shapefile will be read
into memory, the plugin will be run on it, and then the split-up OSM files
will be written out. Thus, you should not have to worry about primitives
crossing file boundaries.

I normally upload my NHD changesets with 40k-50k changes in each upload
> without problem. It takes an hour or so to apply (depending on server load),
> but it works without error.
>
What program are you using for the upload? Is it bulk-upload, JOSM, or
> something else? I'm using JOSM, because I had problems with bulk-upload. If
> there is something better and more robust (the upload with JOSM fails when
> there are about 10k changes), I would certainly give it a try.


I use JOSM -- I check "upload as one changeset" and check "close changeset
after upload". I usually try to do this when the load on the database and
ruby workers are low.
___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Talk-us] US SOTM

[Anthony]

For the newbies like me, apparently SOTM stands for "state of the map".

On Wed, Nov 4, 2009 at 3:17 PM, SteveC  wrote:
> Dear all
>
> On the US chapter call this week we briefly discussed doing a SOTM in
> the USA a couple of weeks before or after the main SOTM.
>
> If you'd like to be involved with this, the first call will be at
>
>        Monday Nov 9th
>        5:30PM PST/8:30PM EST (immediately after the US chapter call)
>
>        +1 218-486-3891 x 224699644
>
>        Agenda: http://docs.google.com/Doc?id=dkbvxqx_7g83v5kjm
>        (please feel free to suggest topics)
>
> Yours &c.
>
> Steve
>
> ___
> Talk-us mailing list
> talk...@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/talk-us
>

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] canvec-to-osm 0.9.6 now available

[Frank Steggink]

Ian Dees wrote:
>
> I normally upload my NHD changesets with 40k-50k changes in each
> upload without problem. It takes an hour or so to apply (depending
> on server load), but it works without error.
>
> What program are you using for the upload? Is it bulk-upload,
> JOSM, or something else? I'm using JOSM, because I had problems
> with bulk-upload. If there is something better and more robust
> (the upload with JOSM fails when there are about 10k changes), I
> would certainly give it a try.
>
>
> I use JOSM -- I check "upload as one changeset" and check "close 
> changeset after upload". I usually try to do this when the load on the 
> database and ruby workers are low.
OK thanks. I've seen this option, and I've only tried it once. This was 
with a small upload only. I did not think of trying it with a large 
changeset.

Regarding the "inner" rules: I've just checked it, and this is working 
as you mentioned. Great :) I also tried the -t option, with comparison 
on file level, and any ways used in a relation are being kept. As far as 
I'm concerned, this version of shp-to-osm is good to go, if we're not 
going to deal with any of the additional issues we discussed this morning.

Great job Ian!

Frank

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] canvec-to-osm 0.9.6 now available

[Frank Steggink]

Hi James,

James A. Treacy wrote:
> Frank,
> Some more issues to add to the list.
>
> There are cases where the same way exists in different data sets but
> with different uses. A good example is that islands in the waterbody
> data are often repeated in the wooded data (if the island happens to
> be wooded, which happens a lot). Importing one set after another leads
> to duplicated ways. This case is time consuming to fix because you
> have to reconcile two ways and a relation. This same issue arises with
> wetlands. Probably others too but these are the ones I've encountered
> most often.
>   
Exactly, but it could be done automatically, provided that nobody else 
has made changes on the server already.
> It would be wonderful if this was handled automatically, but as you
> stated merging the different data sets for a more sophisticated
> analysis of the data would lead to massive files.
>   
Yes. I think that this is most problematic for the initial upload. When 
the data is downloaded afterwards, and changed (automatically or 
manually), the new changeset will be much smaller. The original nodes 
don't need to be reuploaded. I think they will be manageable, but we 
need to check how this really turns out.
> Another issue is how the data is partitioned. It would be really
> helpful if they were divided by geographical area. Currently there
> are a number of files sparsely populated with features (040P07 has 23
> files for Wooded_area). If you are working on one area you need to
> import all the files which defeats the purpose of splitting them in
> the first place.
>   
This actually means that the NTS tiles are too big to handle all at 
once. Maybe we should decide to use smaller working areas, by splitting 
up the NTS tiles in 4 x 4 subtiles. A disadvantage is that features will 
have to be split up (if this can be done automatically), or they have to 
be duplicated in the subtiles.
> Please don't suggest importing the current files one at a time as you
> can't download that large an area at once from openstreetmap. One
> could do a mass upload of each file but I thought that was frowned
> upon. Additionally, it means uploading without running the validator
> (resulting in duplicated nodes & ways).
>   
True, there is also the 50k nodes limit when downloading data. I'm 
already running into this when downloading the Quebec City area 
(021L14), and I haven't uploaded any single Canvec file there yet!
Eventually, all Canvec data will be uploaded one day, so this problem 
can't be avoided. (Unless OSM removes the 50k node limit.)

Frank


___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] canvec-to-osm 0.9.6 now available

[Frank Steggink]

Hi Ian,

Ian Dees wrote:
> On Sat, Nov 7, 2009 at 8:59 AM, Frank Steggink  > wrote:
>
> Sam, can you give some additional clarification what your
> intentions are? I'm afraid I'm not following them well. When you
> mentioning removing duplicate nodes and relations, it looks as if
> you intend to create a script which does some post-processing. Is
> that correct? I haven't started anything in that area. (I actually
> still need to start with the Python version of your batch script,
> but I'm going to work on that today.)
>
>
> Are these duplicate nodes/relations being created by the converter or 
> are they in the source data?
I'm talking mostly from my experience with Geobase. When roads are 
imported, where there is already data from an adjacent tiles, nodes get 
duplicated. This also happens when I copy over Geobase data multiple times.

I'm not sure how this works out for the Canvec data, because there are 
already multiple files. However, if features in multiple files are 
touching each other, their nodes will be identified as duplicates. This 
is also true for adjacent NTS tiles. The problem with shp-to-osm which I 
found earlier this week has already been fixed :)
>  Now we're talking on this: in shp-to-osm (Java) tags are now put 
> properly on the multipolygon relationship. They also still appear on 
> the inner polygons (mentioned to Ian already), but that should be fixed.
>
> Tags appears on the inner ways because you have an "inner" rule in the 
> rules.txt file. If you remove those lines from the rules.txt, you 
> should end up with no tags on inner ways of multipolygons. If this 
> doesn't happen, then it's a bug and I will fix it.
OK, that could explain why this is happening. I'll look at it, and share 
my observations.
>  
>
> Since shp-to-osm is called for one feature type at a time, there
> are some new challenges when multiple feature types are involved.
> I guess you've been thinking about that already. Duplicate nodes
> will become an issue when you have for example a residential area
> with an adjacent wooded area (assuming that the boundaries are
> matching exactly). It will be difficult to deal with this. I'm not
> sure if it would be technically possible to adjust shp-to-osm for
> that, but the result will be that the files will become huge. They
> already have to be split up for certain feature types, and I don't
> think it is possible to use the same set of IDs over multiple
> output files.
>
>
> I do have a "relationificator" plugin started for shp-to-osm that will 
> attempt to solve this problem by converting exactly-overlapping edges 
> into relations and delete duplicate primitives. If there's a strong 
> need for it I can continue to work on it.
That sounds very interesting. I guess this only works within the single 
shape file which is being converted, correct? What is the behavior if a 
file has to be split up, because it becomes too large?
>  
>
> >From what I understand about the upload process (and someone
> please correct me if this isn't right), the OSM server will return
> new ID numbers for any nodes, ways, and relationships uploaded. In
> the OSM files generated by Ian, and also when you're editing in
> JOSM yourself, temporary IDs are assigned. They have a negative
> value, which indicates that these objects don't exist on the
> server. So, this means that, after you have uploaded file00.osm,
> and you open file01.osm, JOSM or the server do no longer remember
> to what objects any IDs are _referring_ to, if those objects are
> not _defined_ in the same file. 
>
>
> The same issue is going on with multipolygon relationships, where
> a part of the ways are reused. This can only happen if everything
> is defined in the same file. And such a file will be way too large
> to upload safely to the server. Recently I noticed that if you
> want to create/update/delete about 10k objects the server is going
> to "act difficult". 
>
>
> I normally upload my NHD changesets with 40k-50k changes in each 
> upload without problem. It takes an hour or so to apply (depending on 
> server load), but it works without error.
What program are you using for the upload? Is it bulk-upload, JOSM, or 
something else? I'm using JOSM, because I had problems with bulk-upload. 
If there is something better and more robust (the upload with JOSM fails 
when there are about 10k changes), I would certainly give it a try.
>  
>
> Regarding relationships, and reuse of the geometry: I think that
> we have not only to remove duplicate nodes, but also split up
> ways, otherwise the JOSM validator will complain about overlapping
> ways. A way can be used in multiple relationships.
>
>
> This would also happen with the relationificator.
>  
>
> A third thing which might need to be resolved are map features
> which cross

Re: [Talk-ca] [Imports] canvec-to-osm 0.9.6 now available

[James A. Treacy]

On Sat, Nov 07, 2009 at 09:14:58AM -0600, Ian Dees wrote:
> Tags appears on the inner ways because you have an "inner" rule in the
> rules.txt file. If you remove those lines from the rules.txt, you should end
> up with no tags on inner ways of multipolygons. If this doesn't happen, then
> it's a bug and I will fix it.

When you say 'no tags' do you really mean zero?

What about when (as in our case) the source data requires attribution
tags. Is it ok to duplicate those on the inner way? Without them it is
not clear where the data came from.

-- 
James Treacy
tre...@debian.org

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] canvec-to-osm 0.9.6 now available

[James A. Treacy]

Frank,
Some more issues to add to the list.

There are cases where the same way exists in different data sets but
with different uses. A good example is that islands in the waterbody
data are often repeated in the wooded data (if the island happens to
be wooded, which happens a lot). Importing one set after another leads
to duplicated ways. This case is time consuming to fix because you
have to reconcile two ways and a relation. This same issue arises with
wetlands. Probably others too but these are the ones I've encountered
most often.

It would be wonderful if this was handled automatically, but as you
stated merging the different data sets for a more sophisticated
analysis of the data would lead to massive files.

Another issue is how the data is partitioned. It would be really
helpful if they were divided by geographical area. Currently there
are a number of files sparsely populated with features (040P07 has 23
files for Wooded_area). If you are working on one area you need to
import all the files which defeats the purpose of splitting them in
the first place.

Please don't suggest importing the current files one at a time as you
can't download that large an area at once from openstreetmap. One
could do a mass upload of each file but I thought that was frowned
upon. Additionally, it means uploading without running the validator
(resulting in duplicated nodes & ways).

-- 
James Treacy
tre...@debian.org

___
Talk-ca mailing list
Talk-ca@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk-ca

Re: [Talk-ca] [Imports] canvec-to-osm 0.9.6 now available

[Frank Steggink]

Sam Vekemans wrote:
> Im not making a new script, the folks who understand python, are
> making a script that can solve the 'duplicate intersecting nodes' and
> 'inner/outer relation' problem.
>
> (Yes you might have fixed it, but "i've reached the crux of my
> technical ability")
>
> Sure, they can use java, if that works for them.
>
> Im just isolating my canvec-to-osm script to handle the 80 map
> features that i know converts correctly. The other 10 can be dealt
> with using another option.
>
> Since i dont know how to program (accept in basic DOS) im not that much help.
>
> Others who are experienced in python & java are welcome to take the lead.
>
> Frank already started to help out maptastically :)
>
> Sam
>
> On 11/6/09, Ian Dees  wrote:
>   
>> On Nov 6, 2009, at 9:28 PM, Sam Vekemans
>>  wrote:
>>
>> 
>>> This python script is yet to be made 'canvec2osm.py' its open to
>>> anyone to make, and i recommend that who ever does, its ONLY 1 person
>>> who is in charge of maintaining the script.
>>>   
>> Why are you creating another shp to osm converter? Is there something
>> the existing tools don't do? I thought you were using shp-to-osm? What
>> changed?
>> 

Hi Sam,

The idea what I mentioned is to convert your batch files to Python. The 
main reason for that is that this way people using Linux or other OSs 
can also perform this conversion locally. I know you want to convert all 
of Canada yourself, but that is still a lot of work. Many hands make 
light work. The only thing we should be concerned about is that we're 
all using the same rule file. This is the most vital part in ensuring 
consistency across the country.

To Ian: shp-to-osm won't disappear in the process. Sam's batch files are 
still calling it. It would be pointless to create a second version. 
There is already a Python script with the same name, but it was written 
specifically for the MassGIS import.

Sam, can you give some additional clarification what your intentions 
are? I'm afraid I'm not following them well. When you mentioning 
removing duplicate nodes and relations, it looks as if you intend to 
create a script which does some post-processing. Is that correct? I 
haven't started anything in that area. (I actually still need to start 
with the Python version of your batch script, but I'm going to work on 
that today.)

Now we're talking on this: in shp-to-osm (Java) tags are now put 
properly on the multipolygon relationship. They also still appear on the 
inner polygons (mentioned to Ian already), but that should be fixed.

Since shp-to-osm is called for one feature type at a time, there are 
some new challenges when multiple feature types are involved. I guess 
you've been thinking about that already. Duplicate nodes will become an 
issue when you have for example a residential area with an adjacent 
wooded area (assuming that the boundaries are matching exactly). It will 
be difficult to deal with this. I'm not sure if it would be technically 
possible to adjust shp-to-osm for that, but the result will be that the 
files will become huge. They already have to be split up for certain 
feature types, and I don't think it is possible to use the same set of 
IDs over multiple output files.

 From what I understand about the upload process (and someone please 
correct me if this isn't right), the OSM server will return new ID 
numbers for any nodes, ways, and relationships uploaded. In the OSM 
files generated by Ian, and also when you're editing in JOSM yourself, 
temporary IDs are assigned. They have a negative value, which indicates 
that these objects don't exist on the server. So, this means that, after 
you have uploaded file00.osm, and you open file01.osm, JOSM or the 
server do no longer remember to what objects any IDs are _referring_ to, 
if those objects are not _defined_ in the same file.

The same issue is going on with multipolygon relationships, where a part 
of the ways are reused. This can only happen if everything is defined in 
the same file. And such a file will be way too large to upload safely to 
the server. Recently I noticed that if you want to create/update/delete 
about 10k objects the server is going to "act difficult". Regarding 
relationships, and reuse of the geometry: I think that we have not only 
to remove duplicate nodes, but also split up ways, otherwise the JOSM 
validator will complain about overlapping ways. A way can be used in 
multiple relationships.

A third thing which might need to be resolved are map features which 
cross the boundary of the NTS tiles. Do we want to merge them? If these 
features have the same Geobase metadata (ID, etc.), then it shouldn't be 
a big problem, otherwise we need to decide whether we prefer to keep the 
metadata, or if we want to have merged features.

All of this means we can't do anything to clean up the data. Sure we 
can, but this can only be done after an initial upload to the server. 
That way we can still apply any logic to deal wi