Thanks mmd: That was exactly what I tried to summarize after some observations:
0. The typical delay (augmented diff id) compared to current time is 1 to 4 minutes. 1. The id is always increasing but sometimes an id (or more) is/are left out between to consecutive deliveries. 2. The time interval for a new diff can be more than a minute (sometimes hours in blackouts). 3. The interval delivering new diffs can be less than a minute (esp. also when previous left out's are handed in later). So a client polls augmented_diff_status and a. waits to download diffs if there's no new id number from augmented_diff_status, b. or it fetches diffs up to and including to the augmented_diff_status id So there may be still missing diff files when fetching up (which means that an internal sequence id is not enough; there is also a need for a list of missing id's)? And the client can poll every minute (sometimes longer if client is busy or diff downloading takes time) without hitting load-limitations? :Stefan P.S. I'm at SoTM too this week! 2016-09-18 14:58 GMT+02:00 mmd <[email protected]>: > Hi, > > Am 18.09.2016 um 14:41 schrieb Michael Larsen: >> Hi, >> >> Last time I tried consuming augmented diffs on a minutely basis, I hit the >> load-limitations which meant that I could not consume augmented diffs for >> some >> time afterwards, i.e. this will lead to black holes in your history. > > the value returned by augmented_diff_status corresponds to current > database timestamp (as number of minutes since the license change). It > does not need to increase one-by-one(!), e.g. the database may process > several minutely diffs in one go, due to some backlog. If you always > download the number returned by augmented_diff_stauts, you will indeed > get some holes! > > That situation can be avoided, if you keep your own internal sequence > id, and fetch augmented diffs up to and including to the value returned > by augmented_diff_status. > > If augmented_diff_status does not return any value (due to overload), > just wait some time and try again. The same applies to downloading > augmented_diffs: you may get HTTP 429 or HTTP 504 in case of overload, > or if you exceed your quota (see /api/status for details). In that case, > don't increase your internal sequence id yet, but try downloading the > same augmented diff again. > >> >> Also, using timestamp start/end to fetch diffs for a given timestamp (like >> avachi) is problematic with some changesets that stay open for > 1 hours >> (this >> happens quite ofter). The live service running on osm.expandable.dk use the >> API as described previously to get the augmented diff for a changeset. If >> there where a better way I'm all ears! >> > > I hope the situation will improve a bit once the database has moved to > version 0.7.53 and a different compression. If you're at SotM next week, > you could maybe ask Roland for a current status. > > -- > > _______________________________________________ dev mailing list [email protected] https://lists.openstreetmap.org/listinfo/dev
