Re: [OSM-dev] setting up mirror

[Kevin Mcintyre]

Hey Andrew - Thanks for this message.

I'm looking to have a closer to home files that would also be available
publicly.  Specifically I have 1.6T disk on rackspace for hosting OSM files
but the reality is this is not enough to host all the files.  I'll probably
start with latest planet and extracts used by our devs and see where that
gets me.

Thanks all for the replies, am new to OSM...so please forgive ambiguous
questions and my terrible grammar.

Kevin

On Wed, Nov 18, 2015 at 10:44 AM, Andrew Byrd  wrote:

> Hello,
>
> I think we should clarify some terms here. The word “mirror” usually
> refers to a secondary copy of some very large or very heavily used
> resource, generally at a different physical location or network link than
> the primary “official” copy. Its purpose is to reduce load on the main
> server and avoid sending huge amounts of data across the world by keeping a
> recent duplicate of the main resource nearer to the final consumers.
>
> When you say “planet osm mirror” people might assume you mean "a copy of
> planet.pbf closer to home”. Creating this kind of mirror is just that
> simple: you’d place a recent copy of planet.pbf on a web or file server of
> some kind, and have people within your organization or geographic area
> fetch planet.pbf from your server instead of the main one (generally to
> reduce traffic crossing out of your local network or traveling long
> distances). Planet.pbf is updated regularly, so you’d probably schedule
> some scripts to keep your mirrored copy in sync by overwriting it once in a
> while or applying OSM diffs.
>
> I am a bit confused by the comments saying there is only one planet
> mirror. There are eight of them listed at
> http://wiki.openstreetmap.org/wiki/Planet.osm and I’m sure there are many
> more undocumented mirrors around the world within organizations that use
> OSM data.
>
> If you only want to work on one geographic region, it’s also possible to
> download a PBF extract of that region and keep it up to date with tools
> like osmupdate or osmosis. This would be much less bandwidth and resource
> intensive than cloning the whole planet.
> http://wiki.openstreetmap.org/wiki/Osmupdate
>
> There is another notion which could also arguably be called “mirroring”
> but is commonly referred to as “replication”: creating a local database
> with a similar schema to the main OSM database and applying diffs to
> maintain a completely up-to-date copy of that main database. This allows
> extensive indexing and thus searching for items within OSM by location, ID,
> tags etc. http://wiki.openstreetmap.org/wiki/Osmosis/Replication
>
> Special-purpose OSM replication databases exist, which you could use
> instead of a general-purpose relational database depending on your use
> case. Overpass is one such project. Instead of using the public overpass
> API server, you can also install Overpass locally to mirror the whole main
> OSM database and keep your mirror in sync:
> http://wiki.openstreetmap.org/wiki/Overpass_API/Installation
>
> I’m also working on a replication system called Vanilla Extract that is
> geared toward performing large, fast geographic extracts. It’s still under
> development but stable enough for us to use it on a daily basis. Our use
> case is routing, so it currently only indexes ways and the nodes referenced
> by them. No free-floating POI nodes (though we’ll eventually add that), and
> no author metadata.
>
> The original version (in C) loads a PBF at a single point in time and does
> no updates.
> https://github.com/conveyal/vanilla-extract
>
> The newer version (in Java using MapDB) is capable of replicating
> planet.osm with a one-line command and performs minutely updates, with the
> option of even pulling the OSM data in over the network without a local
> copy of planet.pbf. However, it’s still a bit slow on large extracts. We’re
> currently migrating it to MapDB2 and expect a performance improvement.
>
> https://github.com/conveyal/osm-lib/blob/master/src/main/java/com/conveyal/osmlib/VanillaExtract.java
>
> Andrew
>
> On 18 Nov 2015, at 18:46, Kevin Mcintyre  wrote:
>
> There's some good information here -
> http://download.geofabrik.de/technical.html one companies process at
> least.
>
> Appreciate the response, it's a tricky search because the results are
> generally information on mirrors not how to become a mirror.
>
>
>
> On Wed, Nov 18, 2015 at 9:21 AM, Tom Hughes  wrote:
>
>> On 18/11/15 17:02, Kevin Mcintyre wrote:
>>
>> Hello - I'm seeking information on setting up a planet osm mirror.
>>>
>>
>> I think the reason you're not getting replies is that you're asking for
>> something that doesn't really exist - there isn't any general
>> infrastructure for mirrors in existence.
>>
>> We have one mirror that we redirect most downloads to, and that's it.
>>
>> Tom
>>
>> --
>> Tom Hughes (t...@compton.nu)
>> http://compton.nu/
>>
>
> 

Re: [OSM-dev] setting up mirror

[Kevin Mcintyre]

There's some good information here -
http://download.geofabrik.de/technical.html one companies process at least.

Appreciate the response, it's a tricky search because the results are
generally information on mirrors not how to become a mirror.



On Wed, Nov 18, 2015 at 9:21 AM, Tom Hughes  wrote:

> On 18/11/15 17:02, Kevin Mcintyre wrote:
>
> Hello - I'm seeking information on setting up a planet osm mirror.
>>
>
> I think the reason you're not getting replies is that you're asking for
> something that doesn't really exist - there isn't any general
> infrastructure for mirrors in existence.
>
> We have one mirror that we redirect most downloads to, and that's it.
>
> Tom
>
> --
> Tom Hughes (t...@compton.nu)
> http://compton.nu/
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] setting up mirror

[Tom Hughes]

On 18/11/15 17:02, Kevin Mcintyre wrote:


Hello - I'm seeking information on setting up a planet osm mirror.


I think the reason you're not getting replies is that you're asking for 
something that doesn't really exist - there isn't any general 
infrastructure for mirrors in existence.


We have one mirror that we redirect most downloads to, and that's it.

Tom

--
Tom Hughes (t...@compton.nu)
http://compton.nu/

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] setting up mirror

[Andrew Byrd]

Hello,

I think we should clarify some terms here. The word “mirror” usually refers to 
a secondary copy of some very large or very heavily used resource, generally at 
a different physical location or network link than the primary “official” copy. 
Its purpose is to reduce load on the main server and avoid sending huge amounts 
of data across the world by keeping a recent duplicate of the main resource 
nearer to the final consumers.

When you say “planet osm mirror” people might assume you mean "a copy of 
planet.pbf closer to home”. Creating this kind of mirror is just that simple: 
you’d place a recent copy of planet.pbf on a web or file server of some kind, 
and have people within your organization or geographic area fetch planet.pbf 
from your server instead of the main one (generally to reduce traffic crossing 
out of your local network or traveling long distances). Planet.pbf is updated 
regularly, so you’d probably schedule some scripts to keep your mirrored copy 
in sync by overwriting it once in a while or applying OSM diffs.

I am a bit confused by the comments saying there is only one planet mirror. 
There are eight of them listed at http://wiki.openstreetmap.org/wiki/Planet.osm 
 and I’m sure there are many 
more undocumented mirrors around the world within organizations that use OSM 
data.

If you only want to work on one geographic region, it’s also possible to 
download a PBF extract of that region and keep it up to date with tools like 
osmupdate or osmosis. This would be much less bandwidth and resource intensive 
than cloning the whole planet.
http://wiki.openstreetmap.org/wiki/Osmupdate 


There is another notion which could also arguably be called “mirroring” but is 
commonly referred to as “replication”: creating a local database with a similar 
schema to the main OSM database and applying diffs to maintain a completely 
up-to-date copy of that main database. This allows extensive indexing and thus 
searching for items within OSM by location, ID, tags etc. 
http://wiki.openstreetmap.org/wiki/Osmosis/Replication 


Special-purpose OSM replication databases exist, which you could use instead of 
a general-purpose relational database depending on your use case. Overpass is 
one such project. Instead of using the public overpass API server, you can also 
install Overpass locally to mirror the whole main OSM database and keep your 
mirror in sync: http://wiki.openstreetmap.org/wiki/Overpass_API/Installation 


I’m also working on a replication system called Vanilla Extract that is geared 
toward performing large, fast geographic extracts. It’s still under development 
but stable enough for us to use it on a daily basis. Our use case is routing, 
so it currently only indexes ways and the nodes referenced by them. No 
free-floating POI nodes (though we’ll eventually add that), and no author 
metadata.

The original version (in C) loads a PBF at a single point in time and does no 
updates. 
https://github.com/conveyal/vanilla-extract 


The newer version (in Java using MapDB) is capable of replicating planet.osm 
with a one-line command and performs minutely updates, with the option of even 
pulling the OSM data in over the network without a local copy of planet.pbf. 
However, it’s still a bit slow on large extracts. We’re currently migrating it 
to MapDB2 and expect a performance improvement.
https://github.com/conveyal/osm-lib/blob/master/src/main/java/com/conveyal/osmlib/VanillaExtract.java
 


Andrew

> On 18 Nov 2015, at 18:46, Kevin Mcintyre  wrote:
> 
> There's some good information here - 
> http://download.geofabrik.de/technical.html 
>  one companies process at least.
> 
> Appreciate the response, it's a tricky search because the results are 
> generally information on mirrors not how to become a mirror.
> 
> 
> 
> On Wed, Nov 18, 2015 at 9:21 AM, Tom Hughes  > wrote:
> On 18/11/15 17:02, Kevin Mcintyre wrote:
> 
> Hello - I'm seeking information on setting up a planet osm mirror.
> 
> I think the reason you're not getting replies is that you're asking for 
> something that doesn't really exist - there isn't any general infrastructure 
> for mirrors in existence.
> 
> We have one mirror that we redirect most downloads to, and that's it.
> 
> Tom
> 
> -- 
> Tom Hughes (t...@compton.nu )
> http://compton.nu/ 
> 
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev