Re: [OSM-dev] planet.osm.org is verry slow

[Grant Slater]

Hi Wolfram and others,

On Sat, 25 Jan 2020 at 10:32, Wolfram Schneider  wrote:
>
> I wanted to download the latest planet from planet.osm.org and the
> download rate is  less than 450KByte/s. In the past I got
> 30-60Mbyte/s. It takes now more than 36 hours to download the planet
> (was 15min). All mirrors are outdated, you have to wait until Sat
> morning to see a new planet.osm.pbf file.
>

I am part of the OpenStreetMap Operations team and Operations Working Group.

I am the person who temporarily instigated the download rate limit.
https://github.com/openstreetmap/chef/commit/1647548590878bfd153b8b8c74e73eda0b8a2cd3

Some background...

www.openstreetmap.org + API, planet and many other services primarily
run in our Amsterdam data center (Equinix AM6).
See servers here: https://hardware.openstreetmap.org/#equinix-am6
Our uplink from these servers to the internet is a 1Gbps link to Cogent.
Bandwidth usage:
https://munin.osm.org/openstreetmap.org/switch1.openstreetmap.org/if_bytes/if_49.html

Due to recent increases in outbound bandwidth usage (partially due to
switch to BBR congestion control
https://github.com/openstreetmap/chef/commit/67c05c6c4b484a6cdac9473b4e6951e5c76c42a3),
 we started to exceed 1Gbps... Once we exceed the limit, packets get
dropped.
We initially did not understand what was causing the packets to be
dropped. We suspected a flaw with switches and/or GBIC modules and/or
fibre link.

The dropped packet caused DNS failures and reliability issues with the
API, which were reported in many channels: email, Slack osm-us, irc
etc.

The planet downloads are the main reason the upstream link is being
swamped. 60Mbyte/s alone is nearly 50% of our available upstream
capacity. Normally around 30 to 60 simultaneous requests.

I've started to prep for upgrading our upstream link to 10Gbps. But
we've had ongoings with Cogent peering and would prefer to switch to
using a new provider upstream.
https://twitter.com/OSM_Tech/status/1220276733359919104

The 20GB+ planet .osm.bz2 files are automatically redirected to the
https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/
mirror
But unfortunately due to flaw in my  mirror update script the redirect
was now working for PBF files. Pull requests welcome:
https://github.com/openstreetmap/chef/blob/master/cookbooks/planet/templates/default/planet-mirror-redirect-update.erb
I have manually fixed the PBF file mirroring now (for current files).

Once we have fixed the upstream link issue, I will remove the download
rate limits.
For the moment, please use a mirror for full speed downloads.

The minutely, hourly and daily diff files will only take a second or 2
extra to download with the current late limits.

Kind regards,

Grant

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM tile server issue

[Grant Slater]

Hi Rodolfo,

Reply per snippet...

On Thu, 30 May 2019 at 19:38, Justmapzz Rodolfo  wrote:
>
> Here is another victim of your usage policy.
> I am the main developer of Justmapzz, which uses GMAP.NET for map viewing.
> Since 2 days our users can no longer view OSM, because you blocked that 
> library for everybody.
> We have always used GMAP.NET  "as is", without diving into the code.
> Now that Grant Slater indicated a portion of the code that defines a 
> user-agent, I had a close look at that detail and I must admit, that this 
> definition of the user-agent gives room for doubt, but mainly because it 
> makes it more difficult for you to develop server behaviour, based on the 
> user-agent info.
> So I changed the definition of the user-agent to a fixed string complying 
> with the directions for the syntax of user-agents and checked the response of 
> your tile server to discover that nothing has changed, still error 403. 
> Apparently you use more filter conditions.

For the benefit of others, here is the semi-random generated
User-Agent the GMAP.NET library creates:
https://github.com/radioman/greatmaps/blob/master/GMap.NET.Core/GMap.NET.MapProviders/GMapProvider.cs#L333-L336

And the GMAP.net's developer's response:
https://github.com/radioman/greatmaps/issues/132

> But wait a minute! A user agent was not designed for access control, but 
> mainly for indicating the nature of a request initiator (=the software) to 
> optimize the web response (an exception to this is the detection and blocking 
> of bots).

Yes, please review the "Technical Usage Requirements" here:
https://operations.osmfoundation.org/policies/tiles/ and once you are
in compliance the tiles will work for you again.
The service is free. We are not paid for you using it. I am a
volunteer. The service is primarily for OpenStreetMap mappers and
hobbyists doing cool and novel things.

> That is why the vast majority of service providers use license keys, or 
> whatever they call them.
> Then every web request must include the license key in order to gain access 
> to the requested information.
> This system also makes it easy to check the usage level or block an 
> individual license for any reason, without affecting other users.

Should we implement it them? Are you willing to develop the supporting
stack for us?
Who will maintain it?

> In this case, it would be much easier to detect the users that cause capacity 
> problems and block only the problematic users, instead of "punishing" all 
> users of a certain app, based on the software libraries, that app uses.
> Have you ever considered the introduction of a license key system?
> In the mean time, what must we do to comply with your usage policy, other 
> than changing the user-agent, which appeared to be insufficient?
> Thanks for any help.

See above.

Kind regards,

Grant
Part of the OpenStreetMap sysadmin team.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] 403 forbidden

[Grant Slater]

Hi Vijaya,

If you have changed the User-Agent to comply with the usage policy and
still being blocked then you are likely in violation for faking
another request header.
Please review the Technical Requirement section here:
https://operations.osmfoundation.org/policies/tiles/

Kind regards,

Grant

On Wed, 29 May 2019 at 11:43, Vijaya Nand  wrote:
>
> DeadSureMobile/1.0.0.1
>
> This is my user agent. Please do let this approved so that I can use it. I am 
> using gmap.net.
>
> Thanks, please confirm so that I can test.
>
>
>
>
> On Wed, 29 May 2019, 16:07 Grant Slater,  wrote:
>>
>> Hi Vijaya,
>>
>> I would suggest using a user-agent with name and version at least eg:
>> org.github.app-name/1.0.0
>> Do make sure you read and follow:
>> https://operations.osmfoundation.org/policies/tiles/
>>
>> Here is why gmap.net got blocked and will remain blocked:
>> https://github.com/radioman/greatmaps/blob/799f3a38a65350ff00c21acf9b88c96848bb3880/GMap.NET.Core/GMap.NET.MapProviders/GMapProvider.cs#L329-L336
>>
>> Kind regards,
>>
>> Grant
>>
>>
>> On Wed, 29 May 2019 at 11:04, Vijaya Nand  wrote:
>> >
>> > Hi Grant.
>> >
>> >
>> > Please suggest me what user agent should I pass. My application name is 
>> > DeadsureMobile.
>> >
>> > Please suggest.
>> >
>> > It's Very urgent for me.
>> >
>> > Thanks,
>> > Vijaya
>> >
>> > On Wed, 29 May 2019, 15:27 Grant Slater,  
>> > wrote:
>> >>
>> >> Hi Vijaya,
>> >>
>> >> We have recent had to crack down on users who are violating our usage 
>> >> policy:
>> >> https://operations.osmfoundation.org/policies/tiles/
>> >>
>> >> Please check what User-Agent your app uses or if you fake any browser 
>> >> headers.
>> >>
>> >> The tile.openstreetmap.org CDN is at breaking point.
>> >> We now frequently push more than 3Tb per second of traffic, with over
>> >> 22000 requests per second/
>> >> We are reaching the limit of what our current stack/config can handle.
>> >>
>> >> Kind regards,
>> >>
>> >> Grant
>> >> Part of the OpenStreetMap sysadmin team
>> >>
>> >> On Wed, 29 May 2019 at 10:26, Vijaya Nand  
>> >> wrote:
>> >> >
>> >> > Dear sir,
>> >> >
>> >> > I am using a gmap.net control to use OSM but suddent it started giving 
>> >> > 403 error. Please suggest.
>> >> >
>> >> > Thanks,
>> >> > Vijaya
>> >> > ___
>> >> > dev mailing list
>> >> > dev@openstreetmap.org
>> >> > https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] 403 forbidden

[Grant Slater]

Hi Vijaya,

I would suggest using a user-agent with name and version at least eg:
org.github.app-name/1.0.0
Do make sure you read and follow:
https://operations.osmfoundation.org/policies/tiles/

Here is why gmap.net got blocked and will remain blocked:
https://github.com/radioman/greatmaps/blob/799f3a38a65350ff00c21acf9b88c96848bb3880/GMap.NET.Core/GMap.NET.MapProviders/GMapProvider.cs#L329-L336

Kind regards,

Grant


On Wed, 29 May 2019 at 11:04, Vijaya Nand  wrote:
>
> Hi Grant.
>
>
> Please suggest me what user agent should I pass. My application name is 
> DeadsureMobile.
>
> Please suggest.
>
> It's Very urgent for me.
>
> Thanks,
> Vijaya
>
> On Wed, 29 May 2019, 15:27 Grant Slater,  wrote:
>>
>> Hi Vijaya,
>>
>> We have recent had to crack down on users who are violating our usage policy:
>> https://operations.osmfoundation.org/policies/tiles/
>>
>> Please check what User-Agent your app uses or if you fake any browser 
>> headers.
>>
>> The tile.openstreetmap.org CDN is at breaking point.
>> We now frequently push more than 3Tb per second of traffic, with over
>> 22000 requests per second/
>> We are reaching the limit of what our current stack/config can handle.
>>
>> Kind regards,
>>
>> Grant
>> Part of the OpenStreetMap sysadmin team
>>
>> On Wed, 29 May 2019 at 10:26, Vijaya Nand  wrote:
>> >
>> > Dear sir,
>> >
>> > I am using a gmap.net control to use OSM but suddent it started giving 403 
>> > error. Please suggest.
>> >
>> > Thanks,
>> > Vijaya
>> > ___
>> > dev mailing list
>> > dev@openstreetmap.org
>> > https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] 403 forbidden

[Grant Slater]

Hi Vijaya,

We have recent had to crack down on users who are violating our usage policy:
https://operations.osmfoundation.org/policies/tiles/

Please check what User-Agent your app uses or if you fake any browser headers.

The tile.openstreetmap.org CDN is at breaking point.
We now frequently push more than 3Tb per second of traffic, with over
22000 requests per second/
We are reaching the limit of what our current stack/config can handle.

Kind regards,

Grant
Part of the OpenStreetMap sysadmin team

On Wed, 29 May 2019 at 10:26, Vijaya Nand  wrote:
>
> Dear sir,
>
> I am using a gmap.net control to use OSM but suddent it started giving 403 
> error. Please suggest.
>
> Thanks,
> Vijaya
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Shutting down opestreetmap-mirror, can hand it off if anyone cares

[Grant Slater]

Thank you Ævar.

And thank you for the push all the years back towards git.

I think turning off the mirroring is a good idea... Most of us already
push directly to the git repos.

Kind regards,

Grant
On Mon, 26 Nov 2018 at 12:20, Ævar Arnfjörð Bjarmason  wrote:
>
> Some of the repositories at https://github.com/openstreetmap, I think
> https://github.com/openstreetmap/josm is the most popular, are
> mirrored from Git or SVN via scripts hosted here:
> https://github.com/openstreetmap/openstreetmap-mirror
>
> I set this up in 2010 and have been running it ever since. I'm not
> active in OpenStreetMap these days, and the server it's been running
> on will be decommissioned in the next few days. It's done via the
> https://github.com/openstreetmap-mirror user.
>
> If anyone's interested in continuing this service I'd be happy to hand
> off that GitHub user to someone (not that a new one couldn't be
> created). The scripts / crontabs are already in the GitHub repo above.
> They currently take around 2GB of storage.
>
> The problem is not that I don't have a server to run this on (I have
> one). I'd need to set this up again, and don't want to spend time on
> doing that or maintaining a service that I myself no longer use.
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Any document on Chef for installing OSM

[Grant Slater]

Hi,

What bit of OpenStreetMap's infrastructure are you interested in running?

The chef repo we manage is used for managing all the servers and
software we used for running OpenStreetMap.org and related sites.

Kind regards,

Grant

On Fri, 14 Sep 2018 at 14:06, Thang Luu  wrote:
>
> Dear Sir/Madam,
>
>
>
> I am trying to install OSM using Chef now.
>
>
>
> Do you have any document on Chef.
>
>
>
> Thank you very much
>
>
>
> Best regards,
>
>
>
>
>
>
>
> Sent from Mail for Windows 10
>
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] New OSM Data Export: Deleted User Account IDs

[Grant Slater]

Announcing a new OpenStreetMap data export...

https://planet.openstreetmap.org/users_deleted/

The export contains the list of historical OpenStreetMap User Account
IDs where the User Account has been deleted.

The most common reason why a user account is deleted is because the
user account was being used exclusively for spamming purposes (diary,
user details etc).

Sometimes User Accounts are also deleted for legal reasons (GDPR etc).

Kind regards,

Grant
Part of OpenStreetMap Sysadmin Team.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Tileserver DNS problems

[Grant Slater]

Hi,

Yes. There was an issue with the [a-f].a.ns.openstreetmap.org Geo DNS
servers we used for tile.openstreetmap.org

The fixed was pushed a few hours ago and the issue should now have
been resolved.

Kind regards,

Grant


On 11 May 2018 at 18:47, Jorge Gustavo Rocha  wrote:
> Hi Stephan,
>
> I have no problems querying tile servers DNS. Every query is less than
> 100ms, wuth or without cache. Maybe the problem has gone.
>
> Example:
>
> dig @a.ns.bytemark.co.uk tile.geo.openstreetmap.org.
>
> ; <<>> DiG 9.11.3-1ubuntu1-Ubuntu <<>> @a.ns.bytemark.co.uk
> tile.geo.openstreetmap.org.
> ; (2 servers found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31012
> ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 6, ADDITIONAL: 6
> ;; WARNING: recursion requested but not available
>
> ;; QUESTION SECTION:
> ;tile.geo.openstreetmap.org.IN  A
>
> ;; AUTHORITY SECTION:
> geo.openstreetmap.org.  259200  IN  NS  a.ns.openstreetmap.org.
> geo.openstreetmap.org.  259200  IN  NS  b.ns.openstreetmap.org.
> geo.openstreetmap.org.  259200  IN  NS  c.ns.openstreetmap.org.
> geo.openstreetmap.org.  259200  IN  NS  d.ns.openstreetmap.org.
> geo.openstreetmap.org.  259200  IN  NS  e.ns.openstreetmap.org.
> geo.openstreetmap.org.  259200  IN  NS  f.ns.openstreetmap.org.
>
> ;; ADDITIONAL SECTION:
> a.ns.openstreetmap.org. 259200  IN  A   212.110.172.32
> b.ns.openstreetmap.org. 259200  IN  A   185.73.44.30
> c.ns.openstreetmap.org. 259200  IN  A   31.169.50.10
> d.ns.openstreetmap.org. 259200  IN  A   64.62.205.202
> e.ns.openstreetmap.org. 259200  IN  A   144.76.70.77
> f.ns.openstreetmap.org. 259200  IN  A   140.211.167.105
>
> ;; Query time: 60 msec
> ;; SERVER: 2001:41c8:2::3#53(2001:41c8:2::3)
> ;; WHEN: Fri May 11 18:44:54 WEST 2018
> ;; MSG SIZE  rcvd: 239
>
> Regards,
>
> Jorge
>
> Às 18:33 de 11/05/2018, Stephan Bösch-Plepelits escreveu:
>> Hi!
>>
>> Apparently there are some DNS problems with the tile server for the default
>> tiles.
>>
>> For a.tile.openstreetmap.org I get 'host not found'.
>>
>> The DNS servers for openstreetmap.org are A, B and C.NS.BYTEMARK.CO.UK
>>
>>  8< 
>> # nslookup
>>> server A.NS.BYTEMARK.CO.UK
>> Default server: A.NS.BYTEMARK.CO.UK
>> Address: 80.68.80.26#53
>>> a.tile.openstreetmap.org
>> Server: A.NS.BYTEMARK.CO.UK
>> Address:80.68.80.26#53
>>
>> a.tile.openstreetmap.orgcanonical name =
>> tile.geo.openstreetmap.org.
>>> tile.geo.openstreetmap.org
>> Server: A.NS.BYTEMARK.CO.UK
>> Address:80.68.80.26#53
>>
>> Non-authoritative answer:
>> *** Can't find tile.geo.openstreetmap.org: No answer
>>  8< 
>>
>> Also other users report, that they don't get tiles ...
>>
>> Please fix!
>>
>> greetings,
>>   Stephan
>>
>>
>>
>> ___
>> dev mailing list
>> dev@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/dev
>>
>
> --
> Geomaster, Lda
> Avenida Barros e Soares, 423
> Nogueira
> 4715-214 Braga
> NIF 510906109
> Tel 253 680 223
> Tm  910 333 888
> Email j...@geomaster.pt
> Site geomaster.pt
> GPS 41.53322,-8.41929
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] iD 2.0.0 always start in whole world view

[Grant Slater]

On 17 November 2016 at 10:39, joost schouppe  wrote:
> Another thing I noticed is that the "best imagery" is no longer loaded by
> default. Is there already an Issue for that?
> (example: in northern Belgium AGIV is marked as best and used to be loaded
> by default)
>

Yip, also a known issue: https://github.com/openstreetmap/iD/issues/3586

It will be fixed once iD v2.0.1 is released.

/ Grant

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Machine Learning Imagery

[Grant Slater]

Hi OSM Dev,

If anyone needs imagery for OSM related machine learning, please feel
free to use any of the following layers...

Please DO use an HTTP user-agent, be sensible and do not abuse the services.

* http://aerial.openstreetmap.org.za/
* http://os.openstreetmap.org/
* http://hampshire.aerial.openstreetmap.org.uk/ (RGB + False Colour IR)
* http://surrey.aerial.openstreetmap.org.uk/
* http://agri.openstreetmap.org/

I setup above the imagery. If you need anything specific, please let
me know. Please share any results ;-)

More details about the above layers in this ticket:
https://github.com/trailbehind/DeepOSM/issues/47

If beneficial I can likely add SRTM or Aster Digital Elevation Model
tiles too. (Likely 16-bit palette grayscale, ideal for computer
processing)

Kind regards,

Grant

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Fwd: TCP Fast Open Experiment

[Grant Slater]

On 2 June 2016 at 18:00, Jeffrey Ollie  wrote:
> On Thu, Jun 2, 2016 at 11:57 AM, Jeffrey Ollie  wrote:
>>
>>
>> On Thu, Jun 2, 2016 at 11:50 AM, Tom Hughes  wrote:
>>>
>>> On 02/06/16 17:47, Jeffrey Ollie wrote:


 HTTP/2 support became available in the stable release of nginx in April
 when 1.10.0 was released. It's been available in the development
 versions for quite a while.
>>>
>>>
>>> Well that machine is running 1.10.1 so I guess it should be available.
>>
>>
>> It's not enabled by default, you need to make sure that nginx has the
>> http2 module compiled in and then there's a small configuration change to
>> make:
>>
>> https://nginx.org/en/docs/http/ngx_http_v2_module.html
>
>
> And most importantly, HTTP/2 basically requires TLS so it's not going to
> help much right now. Personally I'd like to see OpenStreetMap go TLS-only
> but that's a bigger discussion.
>

This experiment is only for HTTP without TLS for the moment. TFO and
HTTP/2 are a create combo and I plan to test them soon. We are using
Ubuntu 16.04 which ships with openssl 1.0.2 and supports
Application-Layer Protocol Negotiation which is required for HTTP/2.

I too would like to go TLS only, but we have a long tail of software
and versions we'd like to support. We also have mappers dispersed all
over with world (YAY!), but some have slow or unreliable connections.
TLS has traditionally suffered from extra round trips and has seemed
to suffer more when connections have some packet loss.

Kind regards,

Grant

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Fwd: TCP Fast Open Experiment

[Grant Slater]

Hi OSM Dev,

I have experimentally enabled TCP Fast Open (TFO) on the imagery site:
http://os.openstreetmap.org/

I am interested in any feedback... Does the site not work from any
connections, operating system or software? Improved performance on
high latency slow connections?

TFO is a "new" way to speed up establishing TCP connections.
https://en.wikipedia.org/wiki/TCP_Fast_Open

I'd like to see TFO rolled out to more services in OSM over time.
Depending how things go, next up would be the tile.openstreetmap.org caches.

Kind regards,

Grant

Part of the OSM ops team

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] oauth api errors?

[Grant Slater]

Hi,

Solve in a discussion on #osm-dev via http://irc.openstreetmap.org the
issue was Christian updated his python2 to python3 causing an issue
with a oauth library he used.

Kind regards,

Grant


On 24 May 2016 at 10:59, Christian Ledermann
 wrote:
>  since a few days I cannot connect to osm via the Oauthapi any more
>   500 Server Error: Internal Server Error for url:
> http://www.openstreetmap.org/oauth/request_token
>  same for api06.dev
>   500 Server Error: Internal Server Error for url:
> http://api06.dev.openstreetmap.org/oauth/request_token
>   were there any changes recently?
>
> --
> Best Regards,
>
> Christian Ledermann
>
> Newark-on-Trent - UK
> Mobile : +44 7474997517
>
> https://uk.linkedin.com/in/christianledermann
> https://github.com/cleder/
>
>
> <*)))>{
>
> If you save the living environment, the biodiversity that we have left,
> you will also automatically save the physical environment, too. But If
> you only save the physical environment, you will ultimately lose both.
>
> 1) Don’t drive species to extinction
>
> 2) Don’t destroy a habitat that species rely on.
>
> 3) Don’t change the climate in ways that will result in the above.
>
> }<(((*>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] No Access to wiki.openstreetmap.org

[Grant Slater]

Hi,

I've been able to put up a read-only copy of wiki.openstreetmap.org
up. I used the Friday 13th May 2016 morning backup, so some content
may be missing for the weekend.

The wiki will return properly on Monday with all content.

Kind regards,

Grant


On 13 May 2016 at 22:23, Mikael Nordfeldth  wrote:
> On 2016-05-13 23:16, Pierre Béland wrote:
>> cannot access Wiki pages. My navigator reports no access.
>
> IRC channel topic says:
> "OSM wiki down for the weekend (server move)"
>
> irc://irc.oftc.net/#osm
>
> --
> Mikael Nordfeldth
> https://blog.mmn-o.se/
> XMPP/mail: m...@hethane.se
> OpenPGP Fingerprint: AE68 9813 0B7C FCE3 B2FA 727B C7CE 635B B52E
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] API read-only for next 45mins

[Grant Slater]

Hi All,

The mapping API is now back to normal. Happy mapping all.

Kind regards,

Grant


On 4 May 2016 at 16:33, Grant Slater  wrote:
> Hi All,
>
> Sorry for the extremely short notice...
>
> The OpenStreetMap map API will be read-only for the next 45mins.
> Any unsaved changes can be saved at the end of the of the 45mins
> maintenance window.
>
> Maintenance will end at: 4:15pm GMT / UTC on 4th May 2016.
>
> Please see https://twitter.com/OSM_Tech for updates.
>
> Kind regards,
>
> Grant

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] API read-only for next 45mins

[Grant Slater]

Hi All,

Sorry for the extremely short notice...

The OpenStreetMap map API will be read-only for the next 45mins.
Any unsaved changes can be saved at the end of the of the 45mins
maintenance window.

Maintenance will end at: 4:15pm GMT / UTC on 4th May 2016.

Please see https://twitter.com/OSM_Tech for updates.

Kind regards,

Grant

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Upcoming Planned outage - 9th May 2016

[Grant Slater]

Hi All,

On the 9th of May 2016 the primary OpenStreetMap database will be
read-only for important maintenance. Editing the map will not be
possible during the maintenance.

Full announcement is available here:
https://blog.openstreetmap.org/2016/05/02/planned-server-outage-9th-may/

The Operations Working Group are in the process of planning some
hardware moves as a result of maintenance at our primary datacentre.
The plans are subject to change, but these moves are expected to take
place on Monday 9th May (a week from today), and this will mean a
period of read only operation lasting for up to 24 hours (into
Tuesday).

The read-only period will mean the OpenStreetMap website and map is
still available, but no map editing will be possible during this time.

The https://wiki.openstreetmap.org/wiki/2016_May_server_maintenance
wiki page has some more technical details, and will be updated with
more precise timing information as the plans are finalised in the
coming days.

We’re putting out this announcement to give some advanced notice, but
we appreciate some people will be wanting to know more about the exact
timings of the read-only period. Please have patience. The Operations
Working Group are busy making plans, and will aim to firm things up in
the coming days.

Kind regards,

Grant

on behalf of the OSM operations team

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Disruption of OpenStreetMap Access

[Grant Slater]

Hi All,

All services are now back and running.

If you are still experiencing problems, please report them in #osm-dev
on http://irc.openstreetmap.org

Kind regards,

Grant
Part of the OpenStreetMap operations team.


On 8 December 2015 at 11:48, Grant Slater  wrote:
> Hi All,
>
> OpenStreetMap.org is currently suffering from a partial outage due to
> a major denial of service attack against the UK Academic Network.
>
> The outage mostly affects those outside of the UK. An outage like this
> is extremely rare.
>
> The operation team will implement a workaround if the problems persist.
>
> Apologies for the inconvenience.
>
> Updates will be posted here: https://twitter.com/osm_tech
>
> Kind regards,
>
> Grant
> Part of the OpenStreetMap operations team.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Disruption of OpenStreetMap Access

[Grant Slater]

Hi All,

OpenStreetMap.org is currently suffering from a partial outage due to
a major denial of service attack against the UK Academic Network.

The outage mostly affects those outside of the UK. An outage like this
is extremely rare.

The operation team will implement a workaround if the problems persist.

Apologies for the inconvenience.

Updates will be posted here: https://twitter.com/osm_tech

Kind regards,

Grant
Part of the OpenStreetMap operations team.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Important Maintenance shortly

[Grant Slater]

Hi All,

OpenStreetMap website + API will be read-only midday to 1pm (UTC/GMT)
later today for important hardware maintenance. Apologies for the
short notice.

Mappers will not be able to save map updates during this maintenance window.

For status updates during the maintenance window please visit
http://irc.openstreetmap.org in the #osm-dev channel.

Kind regards,
Grant
Part of the OpenStreetMap operations team.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Changeset without bbox

[Grant Slater]

Hi Michael,

Changesets which only modify relations do not have a bbox.

Kind regards,
Grant

On 17 May 2015 at 16:44, Michael Larsen  wrote:
> Hi,
>
> I recently observed a changeset which does not have a bbox (the 'min_lat',
> 'max_lat' etc. tags).   The changeset re-orderes members of a relation, so it
> is not a typical changeset and maybe this is the expected behavior since no
> nodes are directly involved?
>
> The changeset is here:
> https://www.openstreetmap.org/api/0.6/changeset/31217983
>
> The web-map-view from the URL below also does not show the bbox nor does it
> zoom to the area for the changeset. This indicates to me that the bbox was
> expected by the web-map-view backend...?
>
> https://www.openstreetmap.org/changeset/31217983
>
> /MichaelVL
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] HTTPS support for osmchanges url

[Grant Slater]

Hi Ramesh,

https:// has now been enabled on planet.openstreetmap.org.

https://git.openstreetmap.org/chef.git/commitdiff/5f2de3518b17b1fb18294649f6c238e289ca7835

Kind regards,
Grant


On 7 April 2015 at 08:26, J Ramesh Kumar  wrote:
> Hi,
>
> Since the OSM updates diff support only http, I cannot update the OSM
> database in my production environment due to security issue. It is
> vulnerable to MITM(Man-In-The-Middle) attack. When I tried https url, it
> redirects to http url.
>
> https://planet.openstreetmap.org/replication/day/000/000/
>
> Is there any specific reason for not having https support ?
>
> It would be great, if we give support https. And also we may add md5check
> sum details in the xx_state.txt files or a separate file.
>
> I am ready to take this work and contribute it. Please guide me that where
> can I start this work ?
>
>
> Thanks,
>
> Ramesh
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Monitoring mod_tile/renderd uptime and alert

[Grant Slater]

Hi Jason,

monit, upstart, runnit, daemontools (http://cr.yp.to/daemontools.html)
are a few.

Kind regards,
Grant

On 20 March 2015 at 13:59, Jason Lee  wrote:
> Hi,
> We use mod_tile with renderd back end for a dynamic mapping service but
> every now and again (not very often), renderd fails due to some unexpected
> request and the map service remains unavailable until I log into the server
> and start up renderd again.
>
> Is there a way to monitor renderd to check it is still running and then I
> can set up an alert or even script it to auto restart when such an outage
> happens again?
>
> Jason
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Partial Website + API outage

[Grant Slater]

On 9 Mar 2015 17:38, "Grant Slater"  wrote:
>
> Hi All,
>
> OpenStreetMap is currently offline due to a network problem at the
> data centre which hosts us.
>
> Status updates here: https://twitter.com/OSM_Tech/
>
> Alternative: #osm-dev on http://irc.openstreetmap.org/
>
> We are trying to get everything back up ASAP, but currently do not yet
> have an ETA.
>

All servers are now back to normal.

Kind regards,
Grant
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Partial Website + API outage

[Grant Slater]

Hi All,

OpenStreetMap is currently offline due to a network problem at the
data centre which hosts us.

Status updates here: https://twitter.com/OSM_Tech/

Alternative: #osm-dev on http://irc.openstreetmap.org/

We are trying to get everything back up ASAP, but currently do not yet
have an ETA.

Kind regards,
Grant
Part of the OSM sysadmin team

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Blue Planet

[Grant Slater]

Hi All,

Yes it is a bug. It has been reverted and will likely take upto a day
or 2 to fully revert.

Kind regards,

Grant
Part of the OSM sysadmin team.


On 6 February 2015 at 22:15, Imre Samu  wrote:
> Status ...
>   https://github.com/gravitystorm/openstreetmap-carto/issues/1294
>
>
> 2015-02-06 22:31 GMT+01:00 John F. Eldredge :
>>
>> On February 6, 2015 3:23:36 PM CST, Michael Maier
>>  wrote:
>>>
>>> Hello,
>>>
>>> Today, the Earth is completely blue in OpenStreetMap¹ (at least in the
>>> openstreetmap-carto on osm.org) - All the oceans drained into the land.
>>>
>>> [1] https://wiki.openstreetmap.org/wiki/File:Blue_planet.png
>>>
>>> I guess the coastlines are broken?
>>>
>>> Greetings,
>>> Michael
>>
>>
>> Since this seems to have hit the entire planet at one time, it probably
>> indicates a bug in the rendering logic. It doesn't seem likely that all of
>> the coastlines would develop gaps simultaneously.
>>
>> --
>> John F. Eldredge -- j...@jfeldredge.com
>> "Darkness cannot drive out darkness: only light can do that. Hate cannot
>> drive out hate: only love can do that." -- Martin Luther King, Jr.
>>
>> ___
>> dev mailing list
>> dev@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/dev
>>
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [OSM-talk] Release openstreetmap-carto v2.26.0

[Grant Slater]

Great work Matthijs & Andy.
Appreciate the dedication.

Kind regards,
Grant

On 2 January 2015 at 17:18, Matthijs Melissen  wrote:
> Dear all,
>
> Today, v2.26.0 of the openstreetmap-carto stylesheet has been
> released. It will be rolled out to the openstreetmap.org servers soon.
>
> Changes include:
>
> * Buildings are now rendered in a much lighter colour
> * Airport labels are rendered in a different colour
> * The tag natural=mud is rendered from z10 instead of z13
> * City names no longer hide country and state names on low zoom levels
> * The tag natural=saddle is now rendered
> * Areas tagged as tourism=attraction are no longer rendered in a special way
> * Various bug fixes
>
> For a full list of commits, see
> https://github.com/gravitystorm/openstreetmap-carto/compare/v2.25.0...v2.26.0.
>
> As always, we welcome any bug reports at
> https://github.com/gravitystorm/openstreetmap-carto/issues.
>
> -- Matthijs
>
> ___
> talk mailing list
> t...@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Unplanned OSM Outage

[Grant Slater]

Hi All,

Quick confirmation, all services are up again and back to normal.
The network issues where resolved around 01:30am GMT/UTC.

Kind regards,
Grant
Part of the OpenStreetMap sysadmin team


On 13 December 2014 at 20:31, Grant Slater  wrote:
> Hi All,
>
> OpenStreetMap is currently offline due to a major network outage at
> the institution hosting of primary services.
> Website, DB, API, wiki offline.
>
> The OpenStreetMap operations team are currently investigating and will
> return the services as soon as possible.
> For further updates best to check: https://twitter.com/OSM_Tech
>
> This is the first such outage in 3 years.
>
> Kind regards,
> Grant
> Part of the OpenStreetMap sysadmin team

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Unplanned OSM Outage

[Grant Slater]

Hi All,

OpenStreetMap is currently offline due to a major network outage at
the institution hosting of primary services.
Website, DB, API, wiki offline.

The OpenStreetMap operations team are currently investigating and will
return the services as soon as possible.
For further updates best to check: https://twitter.com/OSM_Tech

This is the first such outage in 3 years.

Kind regards,
Grant
Part of the OpenStreetMap sysadmin team

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OSM Operations Challenge: Tile CDN QoS

[Grant Slater]

Hi All,

I am part of the OpenStreetMap sysadmin team...

I am involved with running of the tile.openstreetmap.org CDN.

Currently we have 2 perpetually overloaded rendering servers (orm & yevaud)
The render servers are fronted by a collection of caching servers
distributed all over the globe. See:
http://dns.openstreetmap.org/tile.openstreetmap.org.html
Clients are directed to the cache servers by "Geo" DNS. (PowerDNS geo backend)
We use squid 2.7 (ancient) for caching with squid delay pool (Per IP
token bucket) to slow down mass-downlowders / abusers who would
normally degrade the service for everyone.

Squid delays pools is basic token bucket implementation... Each client
IP is allocated a bucket, the client's download rate drains the
bucket... bucket is topped by at a slow rate. Once bucket is drained a
client IP cannot exceed the top-up rate.

I am working on a rewrite of the CDN caching layer to use modern
varnish cache (4.0).

I have also been looking at moving the token bucket abuse management
to linux QoS.

I'm working toward something like:

1) varnish + libvmod-vsthrottle (
https://github.com/daghf/libvmod-vsthrottle ) triggers a log event for
a client with an excessive request rate.
2) log monitor fires off tc to switch client to a rate limited tc qdisc.
3) After x minutes log monitor resets client back to default qdisc.

Basic tc script example: https://gist.github.com/Firefishy/9464cfa4f6b8bec2644a

Anyone with deep knowledge of varnish and/or linux QoS?
Any suggestions? Tips?

Happy to discuss here on list or on IRC Firefishy in #osm-dev (oftc)
http://irc.openstreetmap.org/

Kind regards,

Grant

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] New Notes Dump

[Grant Slater]

Hi All,

OpenStreetMap now produces a dump of all* notes:

http://planet.openstreetmap.org/notes/

The dump currently runs daily at 3am GMT/UTC and takes around 10mins to produce.

Thank you to Ian Dees who wrote the notes dump code, Jochen Topf, Ilya
Zverev & others who suggested improvement during the alpha phase.

*: Where all is Open + Closed notes, excludes notes that have
specifically been hidden. Notes are normally only hidden by an admin
if they are obvious spam.

Kind regards,

Grant
Part of the OSM sysadmin team

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] NOTICE: Mail Disruption & overnight outage

[Grant Slater]

Hi All,

OpenStreetMap email (outgoing and incoming) will be disrupted for up
to the next 24 hours.

OpenStreetMap.org signup confirmation emails, Wiki emails,
OpenStreetMap Message emails (outgoing & incoming), mailing list post
(incoming & outgoing) and a few other email types may be significantly
delayed during this period.

Also...
For a few hours tonight (00:00 midnight to 05:00am GMT+1) a few
services will be unavailable:
mailing lists, help.openstreetmap.org, svn.openstreetmap.org and
trac.openstreetmap.org
It may take until 20:00 GMT+1 for everyone to be able access these
services again. (Old DNS records expiry)

Sorry for the disruption.

Kind regards,
Grant Slater
On behalf of the OpenStreetMap sysadmin team

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Full notes export (alpha)

[Grant Slater]

Few items have been fixed:
https://github.com/iandees/planet-notes-dump/issues?page=1&state=closed

And a new dump created:
http://grant.dev.openstreetmap.org/tmp/planet-notes-dump-testing/

Anything else? ;-)

/ Grant


On 7 July 2014 17:44, Grant Slater  wrote:
> Hi OSM Devs,
>
> A test OSM notes dump has been created:
> http://grant.dev.openstreetmap.org/tmp/planet-notes-dump-testing/
>
> The dump contains all notes (open/closed) + all comments. The XML
> format will likely change during this alpha phase. Once finalised the
> weekly dumps/exports will be made available on
> http://planet.openstreetmap.org/
>
> Feedback / comments / patches:
> https://github.com/iandees/planet-notes-dump
>
> Many thanks to Ian for writing the code.
>
> Kind regards
> Grant
> Part of the OSM sysadmin team.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Full notes export (alpha)

[Grant Slater]

Hi OSM Devs,

A test OSM notes dump has been created:
http://grant.dev.openstreetmap.org/tmp/planet-notes-dump-testing/

The dump contains all notes (open/closed) + all comments. The XML
format will likely change during this alpha phase. Once finalised the
weekly dumps/exports will be made available on
http://planet.openstreetmap.org/

Feedback / comments / patches:
https://github.com/iandees/planet-notes-dump

Many thanks to Ian for writing the code.

Kind regards
Grant
Part of the OSM sysadmin team.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] NOTICE: Upcoming Maintenance - Some services will be down

[Grant Slater]

On Saturday 5th of July 2014 between 09:00 and 19:00 (GMT / UTC) we
are moving our servers hosted by University College London to another
data center.

The following services WILL be affected:

* Search (nominatim.openstreetmap.org) will be unavailable. [1]
* Slower map updates / Reduced tile rendering capacity. (Yevaud outage)
* OSM Foundation websites and blog.openstreetmap.org will be unavailable.
* Taginfo (taginfo.openstreetmap.org) will be unavailable.
* Development Server (errol) will be unavailable.
* Some imagery services will be unavailable. (GPX Render, OS
Streetview, OOC, AGRI, CD:NGI aerial)

Other OpenStreetMap provided services should not be affected - all of
the following are expected to function normally:

* www.openstreetmap.org web site WILL allow edits as per normal (iD or
Potlatch).
* API will allow map editing (using iD, JOSM, Merkaartor etc.)
* Forum
* trac (bug-tracker)
* help.openstreetmap.org
* tile serving ("View The Map" & "Export")
* Wiki
* mailing lists
* subversion and git (source code repositories)
* donate.openstreetmap.org

Technical: We are moving all the servers listed here
http://wiki.openstreetmap.org/wiki/Servers#UCL_-_In_Use to a new UCL
data center. The current building is being closed soon for
refurbishment. The new data center has better server racks, power
feeds, cooling and faster networking.

[1] Searches through the website will still work - we will redirect
them to another nominatim instance temporarily.

Sincerely
Grant Slater
On behalf of the OpenStreetMap sysadmin team.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osm.org not included in ssl-cert

[Grant Slater]

Hi Martin,

The certificate is only valid for openstreetmap.org not the short name.
use https://openstreetmap.org

Very few certificate providers seem to allow mixing wildcard
certificates with multiple domains: eg: *.openstreetmap.org +
*.osm.org + openstreetmap.org + osm.org. None seems to offer this at a
reasonable cost (time or money).

Kind regards,
 Grant
 Part of Sysadmin team

On 13 May 2014 11:01, Martin Koppenhoefer  wrote:
> If you go to https://osm.org you get a security warning because this site is
> not included in the cert:
>
> it says something like:
> certificate only valid for *.openstreetmap.org, openstreetmap.org
> (errorcode: ssl_error_bad_cert_domain)
>
> Cheers,
> Martin
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Heartbleed

[Grant Slater]

On 10 April 2014 20:53, Andrew Hain  wrote:
> What if anything has been the effect of the Heartbleed bug on the OSM family
> of websites and what do people need to do in response?
>

For the primary openstreetmap.org sites:

OpenSSL was patched on all systems by early Tuesday morning [GMT]. The
sysadmin team also re-issued new site certificates for both the
*.openstreetmap.org & *.tile.openstreetmap.org certificates on
Wednesday. I am still checking if the certificate authority has as
expected revoked the previous certificates.

The probability is extremely low that any user data was compromised,
but I would not discourage anyone from changing their password if they
are concerned.

Kind regards,
 Grant
 Part of OpenStreetMap sysadmin team

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Taginfo now supports SSL

[Grant Slater]

Hi OSM Dev,

taginfo.openstreetmap.org now supports SSL.

https://taginfo.openstreetmap.org/

Taginfo was migrated to OSM hardware and went live this morning:
http://blog.jochentopf.com/2014-02-18-taginfo-on-osmf-server.html

Hardware:
http://wiki.openstreetmap.org/wiki/Servers/grindtooth

The setup is managed by chef, the cookbook used is available here:
http://git.openstreetmap.org/chef.git/tree/HEAD:/cookbooks/taginfo

Thank you Tom and Jochen.

Regards
 Grant
 Part of OpenStreetMap sysadmin team

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] HTTPS / SSL enabled on tile.openstreetmap.org

[Grant Slater]

Hi OSM Dev,

tile.openstreetmap.org is now SSL enabled.

We are not using SSL on openstreetmap.org yet, so consider the service
as a beta. ;-)

NOTE:
The certificate is "*.tile.openstreetmap.org" so please make sure to
use the a.tile.openstreetmap.org, b.tile.openstreetmap.org,
c.tile.openstreetmap.org aliases.

Perfect Forward Secrecy enabled. The protocol suite *does* allow IE6 on XP.

Qualys SSL Labs report:
https://www.ssllabs.com/ssltest/analyze.html?d=a.tile.openstreetmap.org&s=192.163.219.40

Regards
 Grant
 Part of OSM sysadmin team

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] NOTICE: Upcoming Maintenance / Downtime

[Grant Slater]

On Wednesday 27th of November 2013 between 17:30 and 22:00 (GMT / UTC)
the primary database server will be unavailable due to maintenance.

I apologise for the short notice.

The following services WILL be affected:
* www.openstreetmap.org web site WILL NOT allow edits (iD or Potlatch). [1]
* API will NOT allow map editing (using iD, JOSM, Merkaartor etc.),
but will remain available as read-only. [2]

Other OpenStreetMap provided services should not be affected - all of
the following are expected to function normally:
* Forum
* trac (bug-tracker)
* help.openstreetmap.org
* tile serving ("View The Map" & "Export")
* Wiki
* Nominatim (search)
* mailing lists
* subversion and git (source code repositories)
* donate.openstreetmap.org

Technical: Database servers ramoth & katla hardware maintenance.
Upgrade of web frontends spike-01, spike-02 & spike-03 with HP DL360
G6 (Xeon 56xx) hardware.

1: Maps will still be viewable on the openstreetmap.org homepage and
on other people's websites.
2: The sysadmin team will try as far as possible to keep the API
available in read-only mode, but the API may be briefly unavailable.

Sincerely
  Grant Slater
  On behalf of the OpenStreetMap sysadmin team.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] HTTPS on Nominatim

[Grant Slater]

On 13 November 2013 17:56, François Lacombe
 wrote:
>
> I was wondering if it's possible to send HTTPS requests to Nominatim service?
> https://nominatim.openstreetmap.org/search doesn't answer.
>

Currently HTTPS is not offered for nominatim.openstreetmap.org
Nominatim serves over 100 requests per second and we do not yet have
the hardware resources to offer HTTPS.

Technical:
While easy enough to enable, the server is already heavily CPU loaded.
The server uses Xeon 56xx CPUs with the AES instruction set but the
SSL connection setup / key-exchange is still expensive. The nature of
requests on nominatim is that most requests will start a new
connection setup. We are planning to get another server for nominatim
in the next few months. (redundancy / easier maintenance / spread
load)

Regards
 Grant
 Part of OSM sysadmin team.

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Matching live bus position to route?

[Grant Slater]

Hi OSM-Dev,

Dev challenge...

I have a near live feed of bus positions for around 130 buses and
(soon) all the passenger bus routes. Routes are relations in OSM.

Does anyone have experience or firm suggestions on how best to match a
live feed of bus positions to a set route?

Feed Data:
 * BusID (no direct match to route)
 * Timestamp
 * Position
 * Travel Direction (Degrees)

I also have 2 weeks worth of historical data.

Typically:
* Bus A will drive from depot to start of route 1 and loop on route 1
all day... maybe route 4 tomorrow.
* Bus B as above but does route 2 and then route 3...
* Bus C will be deployed on any route as required.

All routes may share a few short segments.

Kind regards
 Grant

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] New tile rendering (Live Worldwide)

[Grant Slater]

Hi OpenStreetMappers,

The default OpenStreetMap.org "standard" map was switched across to a
new rendering server setup over the last weekend.

In addition to new hardware, the rendering server also uses the new
“openstreetmap-carto” stylesheet. This is a complete re-write of the
old XML stylesheet to use CartoCSS, making it easier for our
cartographers to work with. The style is designed to look as similar
as possible to the old XML stylesheet.

Andy Allan presented a great talk at State of the Map US conference
describing the reasons for re-writing the stylesheet:
http://stateofthemap.us/saturday.html#schedule/saturday/putting-the-carto-into-openstreetmap-cartography
Andy will present a follow-up at State of the Map next month.
http://2013.stateofthemap.org/

The "openstreetmap-carto" stylesheet is maintained here:
https://github.com/gravitystorm/openstreetmap-carto
The "openstreetmap-carto" is a good base for creating custom styles,
and should be much easier to work with. If you want to help improve
the style, or add new features, please fork it and contribute pull
requests!

Please support OSM’s server hardware fundraising drive:
http://donate.openstreetmap.org/server2013/

Kind regards
 Grant Slater
 Part of the OSM Sysadmin Team

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Planet.osm delayed

[Grant Slater]

Unfortunately due to a badly timed single disk failure in secondary DB
server (smaug RAID6) there will be no planet.osm for this last week.
The disk failure unexpectedly caused the controller to lockup. Still
investigating.

29/05/2013 planet should come out as per normal.

/ Grant

On 24 May 2013 00:46, Grant Slater  wrote:
> Dev,
>
> This week's planet.osm has been delayed due to hardware issues
> effecting our slave database server.
>
> The slave server is catching up to our live instance again, once done
> I will run the ~24 hour planetdump process
> Syncing process can kind of be watched here:
> http://munin.openstreetmap.org/openstreetmap/smaug.openstreetmap/df.html
>
> Expected ETA for this week's planet.osm is late afternoon of Saturday
> or Sunday. (UTC/GMT)
>
> Support of the hardware funding donation drive is greatly appreciated:
> http://donate.openstreetmap.org/server2013/
>
> Regards
>  Grant
>  Part of OSM sysadmin team

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Planet.osm delayed

[Grant Slater]

Dev,

This week's planet.osm has been delayed due to hardware issues
effecting our slave database server.

The slave server is catching up to our live instance again, once done
I will run the ~24 hour planetdump process
Syncing process can kind of be watched here:
http://munin.openstreetmap.org/openstreetmap/smaug.openstreetmap/df.html

Expected ETA for this week's planet.osm is late afternoon of Saturday
or Sunday. (UTC/GMT)

Support of the hardware funding donation drive is greatly appreciated:
http://donate.openstreetmap.org/server2013/

Regards
 Grant
 Part of OSM sysadmin team

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] SotM ads for front page

[Grant Slater]

On 16 May 2013 13:05, Richard Weait  wrote:
>
> Sorry that you didn't get an ad on the front page then.  Obviously you would 
> now, because the policy has changed.

SoTM-EU was 2011.

If there was a "policy" change it was in 2012:

"Add a banner for SOTM US"
http://git.openstreetmap.org/rails.git/commitdiff/5f0c590c32565ffb807699df02c7ce8392513210

We also briefly ran a SOTM Scotland banner in 2012:
http://git.openstreetmap.org/rails.git/commitdiff/749caaff73596e284c39ce7d0a840f8a0a50687d

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] planet-130422.osm.bz2 is out. (semi experimental)

[Grant Slater]

Dev,

Pascal Neis spotted that the latest changeset file is missing a
significant number of the changeset tags.
I have regenerated a corrected changesets-130422.osm.bz2 to fix the issue.

The replacement file will have synced to mirrors within a few hours.
Do compare md5sum against
http://planet.openstreetmap.org/planet/2013/changesets-130422.osm.bz2.md5.

/ Grant

On 23 April 2013 12:06, Grant Slater  wrote:
> OSM Dev,
>
> planet-130422.osm.bz2 is out. PBF is generating and will be out shortly.
>
> This is the first planet.osm file generated against our read slave
> database. (smaug)
> There is a very low probability of consistency issues, please report
> if you find breakage.
>
> Generation time was 21 hours 9 mins. (from >72 hours)
>
> If all checks out, I will return to the normal Wednesday date release
> from next week.
>
> Regards
>  Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Notes Branch

[Grant Slater]

On 24 April 2013 19:57, Mitja Kleider  wrote:
> On 04/24/2013 08:48 PM, Tom Hughes wrote:
>> Don't know where you got three months from - less than twelve hours
>> would be an accurate number.
>
> Because this issue is three months old:
> 
> Guess that was on a testing server, sorry for the confusion.
>
> Great news, btw! Thanks Tom, Kai and whoever else was involved.
>

Notes hasn't been widely announced yet because there are a few last
minute bugs (now mostly fixed) and getting the API documentation onto
the wiki.
Communications working group are working on an announcement.

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] planet-130422.osm.bz2 is out. (semi experimental)

[Grant Slater]

OSM Dev,

planet-130422.osm.bz2 is out. PBF is generating and will be out shortly.

This is the first planet.osm file generated against our read slave
database. (smaug)
There is a very low probability of consistency issues, please report
if you find breakage.

Generation time was 21 hours 9 mins. (from >72 hours)

If all checks out, I will return to the normal Wednesday date release
from next week.

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] planet-130417.osm.bz2 is out

[Grant Slater]

OSM Dev,

After a few missed planet.osm exports... planet-130417.osm.bz2 is finally out.

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] First full GPX Planet has just landed

[Grant Slater]

Dev,

First full GPX planet has just landed.

Accompanying blog post:
http://blog.osmfoundation.org/2013/04/12/bulk-gpx-track-data/

848059 GPX files (260GB) many with route information where permissions allow.

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] No planet.osm dump this week

[Grant Slater]

OSM Dev,

Due to high priority PostgreSQL security updates and data centre power
issues this week's planet.osm file has been skipped.

The next planet file should be planet-130409.osm.bz2 should be available on
12th April 2013.

Regards
 Grant
 Part of OSM sysadmin team.
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Planet.osm out early: planet-130304.osm

[Grant Slater]

OSM Dev,

planet-130304.osm.bz2 is out now. http://planet.openstreetmap.org/
pbf is currently generating and will be out in an hour or 3.

Note: This planet.osm was started on a Monday so as not to conflict
with the planned maintenance tomorrow:
http://blog.osmfoundation.org/2013/03/02/scheduled-maintenance-08-march-2013/

Start time is important for diffs:
 command: bzcat planet-130304.osm.bz2|head -n 2|tail -n 1|awk -F \"
'{print $12}'
 output: 2013-03-04T14:40:16Z

Using that timestamp the follow-on diff file can be found using
http://toolserver.org/~mazder/replicate-sequences/ or I believe
osmupdate

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Data loss in current planet file

[Grant Slater]

On 14 February 2013 11:30, Grant Slater  wrote:
> On 14 February 2013 11:13, NopMap  wrote:
>>
>> Hi!
>>
>> In the German forum, a Mapper was investigating missing data which he had
>> mapped. Tracing it back he found that the current planet file was 8 GB
>> smaller than the previous version.
>>
>> This can be easily confirmed on http://planet.openstreetmap.org/planet/2013/
>> planet-130213.osm.bz2 14-Feb-2013 02:11   18G
>> planet-130206.osm.bz2 09-Feb-2013 16:11   26G
>>
>> What is wrong with the current planetfile?
>> Is this problem known/already fixed?
>>
>
> Known now. I have removed the broken planet.osm export. The planetdump
> code failed due to integer overflow. Will be fixed up shortly.
> Next planet.osm will likely be next week now.
>

planet-130214.osm.bz2 is now out.
Next planet.osm should be out on Friday.

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Data loss in current planet file

[Grant Slater]

On 14 February 2013 11:13, NopMap  wrote:
>
> Hi!
>
> In the German forum, a Mapper was investigating missing data which he had
> mapped. Tracing it back he found that the current planet file was 8 GB
> smaller than the previous version.
>
> This can be easily confirmed on http://planet.openstreetmap.org/planet/2013/
> planet-130213.osm.bz2 14-Feb-2013 02:11   18G
> planet-130206.osm.bz2 09-Feb-2013 16:11   26G
>
> What is wrong with the current planetfile?
> Is this problem known/already fixed?
>

Known now. I have removed the broken planet.osm export. The planetdump
code failed due to integer overflow. Will be fixed up shortly.
Next planet.osm will likely be next week now.

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Static maps from “pafciu17” no longer work?

[Grant Slater]

I've restored it for the moment, but I really want to see it replaced
with something better. With a few scrapers calling complex maps it
easily overloads the dev server:
http://wiki.osm.org/wiki/Servers/errol

The code is here: https://code.google.com/p/gsoc-os-static-maps-api/
but suspect it is a little out of date.
It was abandoned around 2009/2010.

/ Grant

On 9 February 2013 01:47, Ian Dees  wrote:
> Grant turned it off today because it causes a very spikey and high load on
> errol (the dev server).
>
> The code could probably be cleaned up and hosted elsewhere once it was
> hardened against scrapers and high load.
>
>
> On Fri, Feb 8, 2013 at 7:39 PM, Michal Migurski  wrote:
>>
>> Hi,
>>
>> I'm curious why this no longer returns static maps:
>> http://pafciu17.dev.openstreetmap.org/
>>
>> According to the linked wiki page, it should still be up:
>> http://wiki.openstreetmap.org/wiki/Static_map_images
>>
>> -mike.
>>
>> 
>> michal migurski- contact info and pgp key:
>> sf/cahttp://mike.teczno.com/contact.html
>>
>>
>>
>>
>>
>> ___
>> dev mailing list
>> dev@openstreetmap.org
>> http://lists.openstreetmap.org/listinfo/dev
>
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Planet file date unclear at planet.osm.org

[Grant Slater]

On 11 January 2013 13:57, Toby Murray  wrote:
> Right now that happens to be the same as UTC but when London
> enters daylight savings time it will be an hour off.
>

All openstreetmap.org servers managed by the OSM sysadmin team now run
UTC (GMT+0) year round. No DST.

/ Grant
Part of OSM sysadmin team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Planet.osm delayed

[Grant Slater]

Dev,

planet-121219.osm.bz2 dump process failed and the files have been
removed from http://planet.osm.org/

planet-121221.osm.bz2 will be available late Sunday or early Monday.

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Minute Diffs

[Grant Slater]

On 12 September 2012 09:22, Roland Olbricht  wrote:
> Dear all,
>
> the final step of the ODbL transistion has apparently started. The minute 
> diffs are suspended.
>

I am currently re-organising http://planet.openstreetmap.org/ and hope
to have the new ODbL diffs available shortly.

/ Grant
Part of OSM sysadmin team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Mirror Planet.openstreetmap.org?

[Grant Slater]

planet.openstreetmap.org Mirror-er,

There are major changes coming to the directory structure / file
layout of the *planet.osm* files on planet.openstreetmap.org later
this week.

If you use rsync to mirror the site, please turn on rsync
"--hard-links" if possible.

You will save a SIGNIFICANT amount of bandwidth (and time) if you
rsync with the -H or --hard-links option. This option is not turned on
by default (even if using -a)

I have already informed heanet, up.ac.za and gwdg.de but I do not have
contacts for the other mirrors listed on
http://wiki.openstreetmap.org/wiki/Planet.osm#Worldwide_data

Regards
 Grant
 Part of OpenStreetMap sysadmin team

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Random server errors on API (custom install)

[Grant Slater]

On 30 August 2012 04:41, Jan Schejbal  wrote:
> Hi,
> when calling the API on a custom server to get some data like this:
>   http://example.com:3000/api/0.6/map?bbox=8.19,49.83,9.208,50.373
> I sometimes get a result, and sometimes I just get a server error (code
> 500) with the following message:
>  NameError: uninitialized constant XMLEncoding::UTF_8
>
> However, it seems that hard-reloading the map page
> (http://example.com:3000/ in this case) "fixes" the error for some time.
> Restarting the server seems to bring the issue back. Waiting a few hours
> seems to do too.
>
> Does anyone have any ideas what can cause this, or how I can help debug
> this issue? I could imagine it to have something to do with some XML
> library not being properly loaded.
>

According to this entry:
http://wiki.openstreetmap.org/wiki/Talk:The_Rails_Port#Two_problems_I_encountered...
It is likely due to a missing (or broken) Gem install.

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Older change files before sept 27 2011

[Grant Slater]

On 6 August 2012 22:22, Mike  Dupont  wrote:
> On Mon, Aug 6, 2012 at 8:26 PM, Paul Norman  wrote:
>> planet.openstreetmap.org
>
> I cannot access this any more. as I said I am being blocked or something.
>

There are no IP address, user-agent* or other blocks at
http://planet.openstreetmap.org/
You are most definitely not blocked by us.

*: Note the robots.txt: http://planet.osm.org/robots.txt
The generic disallow is due to many idiotic crawler bots attempting to
slurp all our tasty gz / bz2 xml. robots.txt has been unchanged since
2010.

Regards
 Grant
 Part of OSM sysadmin team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] NOTICE: Diffs: Hourly + Daily REWOUND to 11-July-2012

[Grant Slater]

On 23 July 2012 06:17, Lynn W. Deffenbaugh (Mr)  wrote:
>
> On 7/23/2012 1:13 AM, Grant Slater wrote:
>>
>> New rewind...
>> This is the first rewind of the hourly and daily diffs.
>> Hourly: sequence: 2626 -> 2357
>> Daily: sequence: 109 -> 98
>>
>> There was a rewind of the minutely diffs for a short period on the
>> 11th July 2012. Sequence 141373 -> 141272:
>
>
> Ah, now I understand my mistake.  Missed the detail of Hourly and Daily, but
> anyone that did the Minutely rollback back on the 11th is ok through this
> one, right?
>

Yes. Minutely are unaffected.

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] NOTICE: Diffs: Hourly + Daily REWOUND to 11-July-2012

[Grant Slater]

On 23 July 2012 05:54, Lynn W. Deffenbaugh (Mr)  wrote:
> Is this a new rewind, or a very delayed notice of the rewind that occurred
> way back then?
>

New rewind...
This is the first rewind of the hourly and daily diffs.
Hourly: sequence: 2626 -> 2357
Daily: sequence: 109 -> 98

There was a rewind of the minutely diffs for a short period on the
11th July 2012. Sequence 141373 -> 141272:
http://planet.osm.org/redaction-period/minute-replicate/000/141/272.state.txt
and diff generation stopped. The corrected minutely sequence 141273
was only released at Jul 12 11:59:42 UTC.

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] NOTICE: Diffs: Hourly + Daily REWOUND to 11-July-2012

[Grant Slater]

Dev,

Hourly + Daily http://planet.osm.org/redaction-period/ diffs have been
rewound due to an error on the 11-July-2012.

The hourly diffs after sequence 2357 file:
http://planet.osm.org/redaction-period/hour-replicate/000/002/357.state.txt
were incomplete.
Likewise for the daily diffs after sequence 98 file:
http://planet.osm.org/redaction-period/day-replicate/000/000/098.state.txt
were incomplete

To fix:
* If you use the hourly diffs: stop osmosis, replace your state.txt
with http://planet.osm.org/redaction-period/hour-replicate/000/002/357.state.txt
and restart osmosis.
* If you use the daily diffs: stop osmosis, replace your state.txt
with http://planet.osm.org/redaction-period/day-replicate/000/000/098.state.txt
and restart osmosis.

Extra:
If you were following the daily diffs and consumed sequence 2358
"Timestamp: 2012-07-11T15\:00\:00Z" between 11th July 15:00 GMT/UTC
and 11th July 17:30 GMT/UTC, then you will unfortunately have some
invalid data. I have an untested 'patch' diff available, contact me
offlist or see below.

If you need help, likely best to ask in #osm-dev on OFTC.
http://irc.openstreetmap.org/

Regards
 Grant
 Part of OpenStreetMap sysadmin team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Suspend Imports / Bulk edits / Bots

[Grant Slater]

Dear All,

Summary: Please stop Imports, Automated Edits, Bulk edits & Bots until
the redaction process has ended.

The redaction bot has just started running. Until the redaction
process has been completed please suspend all imports and bulk edit
operations. Imports or bots which ignore this request may be subject
to an immediate temporary account block to ease up load on the API and
database servers.

Normal editing operations should be unaffected although I would
recommend saving often.

If you have any questions or a pressing need for such an edit please
contact the sysadmins in advance on IRC. See #osm-dev on OFTC via
http://irc.openstreetmap.org/

Grant
on behalf of OpenStreetMap Sysadmins

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Deprecating non-standard tile.osm.org URLs

[Grant Slater]

OpenStreetMap Devs,

After the 30th June 2012 a number of non-standard
tile.openstreetmap.org URLs will no longer function.

The only valid URL format is:
http://tile.openstreetmap.org/{z}/{x}/{y}.png

Valid host aliases are: a.tile.openstreetmap.org,
b.tile.openstreetmap.org, c.tile.openstreetmap.org, tile.osm.org,
a.tile.osm.org, b.tile.osm.org and c.tile.osm.org

Common non-standard URLs:
http://tile.openstreetmap.org//{z}/{x}/{y}.png
http://tile.openstreetmap.org/mapnik/{z}/{x}/{y}.png
http://tile.openstreetmap.org/mapnik_tiles/{z}/{x}/{y}.png
 ...and many more
All non-standard tile URLs will return 404 Not Found from the 30th June onwards.

The vast majority of websites and apps are using the valid URL format,
however you may wish to check that you are using the correct URL to
access tiles.

Please be aware that tile.openstreetmap.org has a usage policy:
http://wiki.openstreetmap.org/wiki/Tile_usage_policy
Mass scraping of tiles for offline usage or similar is NOT permitted
on this tile server.

Technical: The invalid URLs are poisoning our caches. Storage
re-writing is impractical due to the request rate. (>1000 tiles per
second)

Sincerely
Grant
Part of OpenStreetMap sysadmin team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Nominatim's reverse geocoding statistics

[Grant Slater]

On 2 May 2012 09:43, Derick Rethans  wrote:
> On Wed, 2 May 2012, Charles DESNEUF wrote:
>
>> And if you have an idea about what parameters (Php, Postgres, Apache ...)
>> can we play on to improve the results I'll be glad to ear it.
>
> On the PHP side. Make sure you have APC installed, and running the
> latest version of PHP (5.4.1). It uses a lot less memory, and is about
> 25-40% faster than 5.3.
>

Further... make sure you are running PHP as an apache module (aka
mod_php)* and not PHP via cgi.

*: It would also be interesting to see benchmarks for nominatim via PHP+fastcgi.

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Archive.org Tile Serving

[Grant Slater]

On 25 April 2012 12:18, Stefan de Konink  wrote:
> Hi,
>
> Last night Archive.org has set up a collection to facilitate tile serving
> from Archive.org. We can experiment on what the best structure of this would
> be. Anyone with interest in this subject please contact me.
>

Sure. I think there is interest here. Maybe share here on dev@?

On other matters...
tile.osm.org uses GeoDNS and we are interested in getting additional
servers. (USA, additional server in Western Europe, Eastern Europe and
Eurasia would be good)
Current setup here: http://dns.openstreetmap.org/tile.openstreetmap.org.html

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] NOTICE: Upcoming Maintenance - Read Only

[Grant Slater]

Between Sunday 1st of April 2012 (no joke) 8:00am (GMT / UTC) and the
morning of the 4th of April 2012 the primary database server will be
switched to read-only mode so that we can bring the new server
(ramoth) onstream, bought thanks to your donations.

Additional schedule details are available here:
http://blog.osmfoundation.org/2012/03/27/service-schedule-march-april-2012/

When service is resumed, we will begin the license change as a
background process, redacting any data whose authors have not
consented to the new terms. If you have not yet agreed or declined
please log in and do so before the downtime begins. Login issues?
Assistance is available in #osm (irc.oftc.net) on http://irc.osm.org/

The following services WILL be affected:
* www.openstreetmap.org web site will not allow edits (Potlatch or
Potlatch 2). [1]
* Write API (POST / PUT) and map database editing (using JOSM,
Merkaartor etc.) will be unavailable.
* planet.openstreetmap.org will be available but no new diffs will be
generated until the license change is complete.

Other services will NOT be affected - all of the following are
expected to function normally:
* tile serving ("View The Map" & "Export")
* wiki.osm.org
* forum.osm.org (will allow logins)
* trac.osm.org (bug-tracker, logins allowed)
* help.osm.org (will allow logins)
* Nominatim.osm.org (search)
* mailing lists - lists.osm.org
* subversion ( svn.openstreetmap.org ) and git.openstreetmap.org
(source code repositories)
* donate.openstreetmap.org

Technical: pg_dump (smaug) to pg_restore -j x (ramoth). Upgrading from
PostgreSQL 8.4 to PostgreSQL 9.1

1: Maps will still be viewable on the openstreetmap.org homepage and
on other people's websites.

Sincerely
Grant Slater
On behalf of the OpenStreetMap sysadmin team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] NOTICE: Upcoming Emergency Maintenance / Downtime

[Grant Slater]

On 13 March 2012 21:27, Stefan de Konink  wrote:
>
> At this time you do wonder... one faulty motherbord... why not a nice slave
> server.
>

We have a machine ready to be a slave, but are not yet ready to enable
it. We are planning this for mid/late April timeframe.

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] NOTICE: Upcoming Emergency Maintenance / Downtime

[Grant Slater]

On Tuesday 20th of March 2012 between 13:45 and 16:15 (GMT / UTC) the
primary database server will unavailable due to emergency maintenance.

The following services WILL be affected:
* www.openstreetmap.org web site will not allow user login or edits
(Potlatch). [1]
* API and map database editing (using JOSM, Merkaartor etc.) will be
unavailable.
* planet.openstreetmap.org will be available but no new diffs will be
generated during the outage.
* Forum (no logins)
* trac (bug-tracker, no logins)
* help.openstreetmap.org (no logins)

Other services will NOT be affected - all of the following are
expected to function normally:
* tile serving (“View The Map” & “Export”)
* Wiki
* Nominatim (search)
* mailing lists
* subversion and git (source code repositories)
* donate.openstreetmap.org

Technical: Database Server Smaug: Replacing faulty motherboard.
Supplier Engineer Onsite. We have contingency hardware available.

1: Maps will still be viewable on the openstreetmap.org homepage and
on other people's websites.

Sincerely
Grant Slater
On behalf of the OpenStreetMap sysadmin team

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] NOTICE: Upcoming Maintenance / Downtime

[Grant Slater]

On Friday 3rd of February 2012 between 11:30 and 13:00 (GMT / UTC) the
primary database server will unavailable due to planned maintenance.

The following services WILL be affected:
* www.openstreetmap.org web site will not allow user login or edits
(Potlatch). [1]
* API and map database editing (using JOSM, Merkaartor etc.) will be
unavailable.
* planet.openstreetmap.org will be available but no new diffs will be
generated during the outage.
* Forum (no logins) [ http://wiki.openstreetmap.org ]
* trac (bug-tracker) [ http://wiki.openstreetmap.org ]
* help.openstreetmap.org (no logins)

Other services will NOT be affected - all of the following are
expected to function normally:
* tile serving (“View The Map” & “Export”)
* Wiki http://wiki.openstreetmap.org
* Nominatim (search)
* mailing lists
* subversion and git (source code repositories)
* donate.openstreetmap.org

Technical: Server Smaug: BIOS Update (to enable PCI-E 256 byte
payload), RAID Controller PCI-E slot re-location and potential rack
move. Server ramoth is being racked.

1: Maps will still be viewable on the openstreetmap.org homepage and
on other people's websites.

Sincerely
 Grant Slater
 On behalf of the OpenStreetMap sysadmin team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Script for Installing Mapnik2

[Grant Slater]

On 2 November 2011 15:37, Parveen Arora  wrote:
> Hi All,
>
> I was searching for the script to install Mapnik2, Is there any available?
> If Yes, Please let me know.

On Linux? Distro?

On Ubuntu:
sudo add-apt-repository ppa:mapnik/nightly-trunk
sudo apt-get update
sudo apt-get install libmapnik2 libmapnik2-dev mapnik2-utils

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] CORS headers on OSM tiles (for WebGL use)

[Grant Slater]

On 17 September 2011 22:46, klo...@gmail.com  wrote:
> Dear OSM developers,
>
> Would it be possible to add the CORS HTTP header: 
> Access-Control-Allow-Origin: *
> to the OSM tileserver when a tile is requested?
>

Enabled. Cached tiles will take a few days to expire.

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] NOTICE: Upcoming Maintenance Work (Partial Outage)

[Grant Slater]

Quick Reminder: This weekend we have a partial outage...
Further details: http://wiki.openstreetmap.org/wiki/Power_Maintenance_Q3_2011

http://stateofthemap.org/ will be available this weekend, including
tickets / registration.

On 28 June 2011 12:21, Grant Slater  wrote:
> Some secondary OpenStreetMap services will be unavailable during the
> following dates, due to power maintenance work at our server hosting:
>
> - 2nd and 3rd July 2011. (Saturday & Sunday)
> - 30th and 31st July 2011. (Saturday & Sunday)
> - 3rd and 4th September 2011. (Saturday & Sunday)
> - and 1 weekday (to be confirmed) during August 2011.
>
> On each of these weekends, the servers will be shut down at 7:30 UTC,
> and will be started again after works are completed, which will be
> Sunday 16:30 UTC at the latest, but may be earlier. We do not yet have
> a precise date for the August maintenance.
>
> The following services will be unavailable during these periods:
> * XAPI / jXAPI
> * dev.openstreetmap.org server (errol)
> * rendering of new map tiles
> * osmfoundation.org and stateofthemap.org web sites
>
> As a temporary replacement for XAPI/jXAPI, consider using third-party
> services such as that provided by MapQuest:
> http://developer.mapquest.com/web/products/open/xapi
>
> Additional information will be posted to the following page:
> http://wiki.openstreetmap.org/wiki/Power_Maintenance_Q3_2011
>
> The following services are *not* affected:
> * www.openstreetmap.org web site
> * API and Editing (*phew*)
> * planet.openstreetmap.org and diffs.
> * tile serving for existing tiles (details below)
> * Wiki
> * Nominatim (search)
> * mailing lists
> * subversion and git (source code repositories)
> * trac (bug-tracker)
> * donate.openstreetmap.org
>
> Maps will still be viewable on the openstreetmap.org homepage and on
> other people's websites. We’ll be serving tiles from a back-up tile
> server. However rendering engines will be de-activated, meaning that
> new rendering of map updates will not take place during the
> maintenance period, some requests for tiles will fail where no cached
> copy is available, and tile response times may be slower than normal.
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] problem with slippy map wiki extension

[Grant Slater]

2011/8/3 M∡rtin Koppenhoefer :
> the slippy map media wiki extension currently doesn't work:
>
> http://wiki.openstreetmap.org/wiki/Slippy_Map_MediaWiki_Extension
>

I have added a very horrible hack to make it work again for the moment
with MediaWiki 1.17.0
If anyone cares to do it properly, the code is here:
http://svn.openstreetmap.org/applications/utils/wiki-extensions/SlippyMap-local/
also see: http://www.mediawiki.org/wiki/Extension:SlippyMap

Harry Wood and myself have been discussing moving across to the
http://www.mediawiki.org/wiki/Extension:Maps

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] How to Increase the Limit of Inodes

[Grant Slater]

On 5 August 2011 10:29, Parveen Arora  wrote:

> I am using ubuntu 10.04 and ext4 file system.
> Now I want to know that how this problem can be Solved or how to
> increase the number of inodes or how openstreetmap manages it for all
> the world.
>

Are you using metatiles? tile.osm.org (yevaud) do not separately store
individual tiles on the filesystem.
You cannot increase the number of available inodes on an existing
filesystem. An ugly hack would be to create a suitable sized loopback
filesystem and specify type "news" when creating fs. eg: "mkfs.ext4 -T
news"

yevaud
--
FilesystemInodes   IUsed   IFree IUse% Mounted on
/dev/sdb 597562544 38961127 5586014177% /tiles
/dev/sdc 585937408 6512370 5794250382% /tiles-low
--

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Partial outage this weekend

[Grant Slater]

Devs,

The dev server (errol) will not be available this weekend due to power
maintenance work at hosts.

Further details and list of servers involved here:
http://wiki.openstreetmap.org/wiki/Power_Maintenance_Q3_2011

www.osm.org and API are NOT affected.

/ Grant
Part of Sysadmin Team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Minute-Diff

[Grant Slater]

On 11 July 2011 19:39, Roland Olbricht  wrote:
> Hello,
>
> does anybody know what has happened to
> http://planet.openstreetmap.org/minute-replicate/ ?
> It apparently stopped half an hour ago.
>

Seem ok to me now. Likely just a temporary hiccup.

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] NOTICE: Upcoming Maintenance Work (Partial Outage)

[Grant Slater]

Some secondary OpenStreetMap services will be unavailable during the
following dates, due to power maintenance work at our server hosting:

- 2nd and 3rd July 2011. (Saturday & Sunday)
- 30th and 31st July 2011. (Saturday & Sunday)
- 3rd and 4th September 2011. (Saturday & Sunday)
- and 1 weekday (to be confirmed) during August 2011.

On each of these weekends, the servers will be shut down at 7:30 UTC,
and will be started again after works are completed, which will be
Sunday 16:30 UTC at the latest, but may be earlier. We do not yet have
a precise date for the August maintenance.

The following services will be unavailable during these periods:
* XAPI / jXAPI
* dev.openstreetmap.org server (errol)
* rendering of new map tiles
* osmfoundation.org and stateofthemap.org web sites

As a temporary replacement for XAPI/jXAPI, consider using third-party
services such as that provided by MapQuest:
http://developer.mapquest.com/web/products/open/xapi

Additional information will be posted to the following page:
http://wiki.openstreetmap.org/wiki/Power_Maintenance_Q3_2011

The following services are *not* affected:
* www.openstreetmap.org web site
* API and Editing (*phew*)
* planet.openstreetmap.org and diffs.
* tile serving for existing tiles (details below)
* Wiki
* Nominatim (search)
* mailing lists
* subversion and git (source code repositories)
* trac (bug-tracker)
* donate.openstreetmap.org

Maps will still be viewable on the openstreetmap.org homepage and on
other people's websites. We’ll be serving tiles from a back-up tile
server. However rendering engines will be de-activated, meaning that
new rendering of map updates will not take place during the
maintenance period, some requests for tiles will fail where no cached
copy is available, and tile response times may be slower than normal.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] NOTICE: Scheduled Maintenance - 23rd June 2011

[Grant Slater]

Quick update. The maintenance has been completed. All services are up
and running again.
API + www.openstreetmap.org is running a little slower than normal due
to reduced capacity, but will return to normal late Friday afternoon
(GMT).

/ Grant

On 9 June 2011 21:17, Grant Slater  wrote:
> Please copy this to local lists as appropriate.
>
> Thursday 7:30am (23rd June 2011 GMT/UTC+0) the API and map editing on
> www.openstreetmap.org will be unavailable. The maintenance period is
> expected to last for 12 hours.
>
> The following services will be unavailable during the maintenance period:
> API, editing features of www.openstreetmap.org and
> planet.openstreetmap.org including replication diffs.
>
> The wiki, mailing lists and help.openstreetmap.org will be unaffected.
>
> Technical: Some of the core servers are being re-located to another 
> data-centre.
>
> Additional information will be posted to the following link closer to the 
> time:
> http://wiki.openstreetmap.org/wiki/Servers/June_2011_Maintenance
>
> / Grant
> Part of the OSM sysadmin team.
>

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Planet Export running earlier this week

[Grant Slater]

Dev,

Planet dump run is starting earlier this week to avoid a conflict with
the server maintenance on Thursday. [1]

Normal time: Wednesday 1:11am (British Summer Time)
This week time: Tuesday 0:11am (British Summer Time)

The planet run will likely return to normal schedule next week.

PS: We're also aiming to have a full history dump out in a few days.

1: http://wiki.openstreetmap.org/wiki/Servers/June_2011_Maintenance

/ Grant
Part of OSM Sysadmin Team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] NOTICE: Scheduled Maintenance - 23rd June 2011

[Grant Slater]

Please copy this to local lists as appropriate.

Thursday 7:30am (23rd June 2011 GMT/UTC+0) the API and map editing on
www.openstreetmap.org will be unavailable. The maintenance period is
expected to last for 12 hours.

The following services will be unavailable during the maintenance period:
API, editing features of www.openstreetmap.org and
planet.openstreetmap.org including replication diffs.

The wiki, mailing lists and help.openstreetmap.org will be unaffected.

Technical: Some of the core servers are being re-located to another data-centre.

Additional information will be posted to the following link closer to the time:
http://wiki.openstreetmap.org/wiki/Servers/June_2011_Maintenance

/ Grant
Part of the OSM sysadmin team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] CSS for , on wiki

[Grant Slater]

On 9 June 2011 16:45, Josh Doe  wrote:
> You can see it on the wiki, but be warned that I know almost nothing
> about CSS, so I'm probably doing everything The Wrong Way:
> http://wiki.openstreetmap.org/wiki/User:Joshdoe/vector.css
>
> div.mw-geshi {
>    background-color: #F9F9F9;
>    border: 1px dashed #2F6FAB;
>    color: black;
>    line-height: 1.1em;
>    padding: 1em;
>    font: 1em/1.2em monospace;
>    font-size: 1.2em;
> }
>
> pre {
>    font-size: 1.2em;
> }
>
> -Josh
>

Done. CSS files are heavily cached and may take awhile to refresh.
Normal route is to request via: http://wiki.openstreetmap.org/wiki/Talk:Wiki

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] planet.openstreetmap.org, pbf, latest-link

[Grant Slater]

On 8 June 2011 15:38, Sebastian Kürten  wrote:
> Hi,
>
> planet.openstreetmap.org offers experimental support for pbf-files here:
> http://planet.openstreetmap.org/pbf-experimental/
>
> what about inserting a symlink to the latest file like this, as it is
> done for the .osm.bz2 files already?
> http://planet.openstreetmap.org/pbf-experimental/planet-latest.osm.pbf
> http://planet.openstreetmap.org/pbf-experimental/planet-latest.osm.pbf.md5
>
> would be really helpful in using access to the latest files from
> within scripts!
>

Sure. I'll set it up.

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Planet File Delayed

[Grant Slater]

Devs,

This week's weekly planet file has been delayed by around 40 hours.

planet-110602.osm.bz2 (instead of 110601) should be out sometime
around Saturday 4th Jun 2011 13:00 (UTC).

Diffs are unaffected.

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Paging the tile server admins

[Grant Slater]

On 6 May 2011 13:52, menion  wrote:
>
> 3) I checked header and you were probably right. It was part of old code
> from my previous Locify application.
>  So I changet User-agent to
>    Mozilla/5.0 (Linux; U; Android; en-us)
>  and added Referer to header
>    http://locus.asamm.cz/;version:X
>

Please set a User-Agent which lists the App.
May I suggest something like: "Locus/version (Linux; U; Android; en-us)"

Please don't abuse the referer.

Regards
 Grant
 Sysadmin Team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Something wrong with the renderer?

[Grant Slater]

On 5 May 2011 18:05, Igor Brejc  wrote:
> On Thu, May 5, 2011 at 6:33 PM, M∡rtin Koppenhoefer 
> wrote:
>>
>> 2011/5/5 Grant Slater :
>>
>> What about changing the tile urls frequently? This could be automated
>> and synchronized with the start page and would stop those apps quite
>> fast (at least until they start making the urls in their app dynamic
>> and synchronize as well).
>
>
> That would punish everyone, even those apps that adhere to the policy. I see
> no reason to prevent access to an app which just mimics the browser and
> shows a limited map area, uses throttling and caches those tiles for more
> than a week.
> Besides, this would not really help OSM spread its word. I'm sure there are
> better ways of dealing with offenders.

Yup, you capture my feelings exactly.

I also think something like this should be added to the policy: App
must display: 'Help improve this map: www.osm.org' with a clickable
link. Translations acceptable.

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Something wrong with the renderer?

[Grant Slater]

On 5 May 2011 16:15, Manuel Reimer  wrote:
> Hello,
>
> currently it often takes days until I see a change, I uploaded, in the
> official Mapnik rendered map.
>
> In the past, rendering was much faster. Often changes were rendered within
> minutes. Is something wrong with the renderer?
>

Something wrong: no... But the server is completely overloaded yes.

Apps which scape masses of tiles for offline usage is becoming a big
problem. Some of these apps fake their User-Agent and other headers.
I've politely asked the heaviest of these commercial apps to move to
other tile servers as per our tile usage policy:
http://wiki.openstreetmap.org/wiki/Tile_Usage_Policy

Google Maps has been sending legal letters to tile scraping apps for
being in violation of their terms of service. eg: gmapcatcher, MOBAC,
NaviComputer, Locus Free and likely others, the apps have been
switching to using tile.openstreetmap.org as the default tile layer.

The scrapers cause problems because they normally attempt to download
all zoom layers available. Zoom 0 to 18. Only 6.69% of z16 has ever
been viewed (and therefore rendered). z17 2.50% and z18 0.90% [1].
When a request comes in for one of these layers the server normally
has to specially renders a copy, the server is only powerful enough to
render a few tiles per second. It is this rendering of tiles for
scrapers that is blocking up the render queue from serving fresh tiles
to mappers.

I've employed some techniques in an attempt to limit the scaper's
download rate while keeping things responsive to mappers. Token bucket
rate limiting [2]. There are also some upgrades in the pipeline.

1: http://wiki.openstreetmap.org/wiki/Tile_Disk_Usage (In total only
1.79% of the theoretical number of tiles have ever been viewed.)
2: http://en.wikipedia.org/wiki/Token_bucket

Regards
 Grant
 Part of the sysadmin team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] rails_port problem

[Grant Slater]

On 22 April 2011 10:23, Kirill Bestoujev  wrote:
> Hi everybody!
>
> I have a problem with rails port.
>
> Ubuntu 10.04, x64.
> I installed the rails port according to the wiki, uploaded the data into it.
> Activated the users.\
>
> All looks fine, I can download data with Josm, but I fail to upload
> anything.
>
> In josm I get Internal server error 500.
> In rails log I have
>

Try downgrade your version of libxml-ruby to version 1.1.3
There has been a report of >=2 causing an issue.

Let us know if you come right.

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Paging the tile server admins

[Grant Slater]

On 27 April 2011 14:18, Dermot McNally  wrote:
> On 27 April 2011 14:08, Jaak Laineste  wrote:
>
>> Good application =
>> a) is www.openstreetmap.org OR
>> b) generates less than 1000 tile requests per day
>
> I accept the spirit of the suggestion, but 1000 tiles is probably a
> bit conservative. There is a level at which (IMHO) we would prefer low
> volume applications of whatever sort to use OSM instead of Google etc.
> even if that comes at the cost of tile accesses on OSM servers. I
> suppose my rule of thumb would be that the human eyes looking at the
> third-party application would be no greedier than had we had those
> same eyes on the main OSM site.
>
> In such cases, assuming correct attribution, such sites are providing
> us with welcome publicity and we need not be mean with our tiles.
>

Exactly.
The iPhoneTracker app ( http://petewarden.github.com/iPhoneTracker/ )
which came out last week tripled our usual traffic to around 300Mb/s.
Well over 1000 tiles per second.

The closing paragraph of this interview with the developers gave me
the warm fuzzies:
http://radar.oreilly.com/2011/04/iphone-tracking-followup.html

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Paging the tile server admins

[Grant Slater]

On 27 April 2011 12:23, Ramas  wrote:
>
> This QOS limit is real problem for applications which doesn't works like
> bulk downloaders but does many requests to tile server.
> Maybe we could do some excludes for "good" applications (like proxy caches)
> and rought limits for "bad" applications?
>

Can you expand on where the problem is being experienced?
The "QOS" only kicks in if a large number of uncached tiles are
download in a short period of time.

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Paging the tile server admins

[Grant Slater]

Hi NopMap

On 27 April 2011 09:29, NopMap  wrote:
>
> Hi!
>
> Paging the admins of the OSM tile servers. I'd like to get in touch about
> the topic of mass downloaders. It appears that some problems first become
> visible on small tile servers like my own, but hit the main tile servers a
> while later, so it might be a good thing to share experiences. E.g. MOABC
> was an issue for my server in december and had to be blocked on the main
> servers about three months later.
>

Yes, the MOBAC app is becoming a serious problem for the tile.osm.org
server. Earlier versions fake a Firefox user-agent. I have had to
reduce max tile QOS limit downward in an attempt to control the
situation. Allowing mass tile scraping is not sustainable.
A lively thread when I restricted the Faked user-agent:
http://sourceforge.net/projects/mobac/forums/forum/861096/topic/4423122

tile.osm.org traffic has recently peaked around 300Mb/s.

> A few days ago a new mass downloading application has appeared on my server,
> already causing tile requests in the order of millions with up to 60
> tiles for a single client. The user agent is "Profile/MIDP-2.0
> Configuration/CLDC-1.1" which indicates some sort of handy software, but a
> search in the net brought up no clues as to what exactly this might be. Do
> you have any suggestions?
>

I too am perplexed by the MIDP user-agent. I believe the user-agent is
from Java Mobile Edition. Some? BlackBerry's prefix their user-agent
eg: "BlackBerry9800/6.0.0.246 Profile/MIDP-2.0 Configuration/CLDC-1.1".

Basic tile.osm.org user-agent stats:
http://grant.dev.openstreetmap.org/tmp/logs/yevaud-user-agents-16042011.txt.bz2

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Changes to PHP on Dev Server (errol)

[Grant Slater]

Devs on Errol,

Within the last hour I have changed how PHP operates on the Dev Server
(Errol). PHP should now be much quicker.
Please let me know if you have any problems.

Details...
I've changed PHP from using setuid php-cgi to per user fastcgi PHP
instances using php-fpm + mod-fastcgi-handler. I have also enable php
apc and switched to apache event mpm.

Regards
 Grant
 Part of sysadmin team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Warning. Brief dev server outage this evening

[Grant Slater]

Hi Devs,

The dev server "errol" will have a brief outage this evening around
5:45pm GMT for hardware maintenance.

Regards
 Grant
 Part of sysadmin team.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Fwd: [OpenStreetMap] ODbL

[Grant Slater]

On 10 January 2011 11:41, John Smith  wrote:
> What's it going to take to stop people spamming me about the ODBL?
>

People contacting you is exceptional AFAIK. The issue is you have made
global trivial tag changes which effects local mappers in most
regions.

> The current CTs conflict with data I've edited or derived so I can't
> agree to the license change even if I wanted to.
>

Other than NearMap any others?

Regards
 Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] error when compiling osm2pgsql

[Grant Slater]

On 20 December 2010 15:23, Peter  wrote:
> Hi all,
>
> I get a error in compiling osm2pgsql
> http://pastebin.com/gSF0ca5X
> What do I do now?
>

Install package: libtool

/ Grant

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] API Dev server down?

[Grant Slater]

Server has been suffering from some extremely high IO load recently, I
will look when I get home. / Grant

On 12/8/10, Stuart Johnston  wrote:
> The API Dev server appears to be down.   Anyone know what is going on?
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>

-- 
Sent from my mobile device

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev