Re: [OSM-dev] Minute Diffs broken?

On Nov 6, 2011 8:45 AM, "Tom Hughes" wrote: > > On 06/11/11 13:35, Jeremy Adams wrote: It looks like the minute replicate diffs have stopped at 1071883. http://planet.openstreetmap.org/minute-replicate/001/057/ Could someone please check? >>> >>> >>> Yes they will down until a

Re: [OSM-dev] Minute Diffs broken?

On 06/11/11 13:35, Jeremy Adams wrote: It looks like the minute replicate diffs have stopped at 1071883. http://planet.openstreetmap.org/minute-replicate/001/057/ Could someone please check? Yes they will down until a new osmosis build can be deployed. Is that something that's happening the n

Re: [OSM-dev] Minute Diffs broken?

>> It looks like the minute replicate diffs have stopped at 1071883. >> http://planet.openstreetmap.org/minute-replicate/001/057/ >> Could someone please check? > > Yes they will down until a new osmosis build can be deployed. > > Tom > > -- > Tom Hughes (t...@compton.nu) > http://compton.nu/ Is t

Re: [OSM-dev] Minute Diffs broken?

On 06/11/11 12:35, Roland Olbricht wrote: It looks like the minute replicate diffs have stopped at 1071883. http://planet.openstreetmap.org/minute-replicate/001/057/ Could someone please check? Yes they will down until a new osmosis build can be deployed. Tom -- Tom Hughes (t...@compton.nu)

[OSM-dev] Minute Diffs broken?

Hello everybody, It looks like the minute replicate diffs have stopped at 1071883. http://planet.openstreetmap.org/minute-replicate/001/057/ Could someone please check? Cheers, Roland ___ dev mailing list dev@openstreetmap.org http://lists.openstreetm

Re: [OSM-dev] Minute Diffs Broken

Brett Henderson a écrit : > Etienne Chové wrote: >> I think there is a bug in .state file on timestamp line... there are >> useless backslashes. >> > They are Java properties files and presumably a ':' character is being > escaped with a '\' character. Apologies but I can't do much about it

Re: [OSM-dev] Minute Diffs Broken

On Tue, May 5, 2009 at 2:30 AM, Steve Singer wrote: > What you want is the timestamp that the change was committed at, not the > timestamp it was inserted at.  However there is no way to get this with > postgresql. I keep seeing this. See: http://www.postgresql.org/docs/8.3/static/functions-date

Re: [OSM-dev] Minute Diffs Broken

Etienne Chové wrote: > Matt Amos a écrit : > >> we hope so. there are (currently experimental) "replication" diffs >> here http://planet.openstreetmap.org/minute-replicate/ in a three-deep >> numeric directory structure. these diffs are produced from the >> transactions in the database, so each

Re: [OSM-dev] Minute Diffs Broken

Matt Amos a écrit : > we hope so. there are (currently experimental) "replication" diffs > here http://planet.openstreetmap.org/minute-replicate/ in a three-deep > numeric directory structure. these diffs are produced from the > transactions in the database, so each doesn't strictly cover a minute

Re: [OSM-dev] Minute Diffs Broken

On Tue, Oct 6, 2009 at 3:03 PM, Florian Lohoff wrote: > > Hi, > > On Tue, May 05, 2009 at 08:32:44AM +1000, Brett Henderson wrote: >> Hi Everybody, >> >> Unfortunately the minute diffs appear to be regularly missing data.  In >> the last 8 hours at least 3 changesets have been missed.  The ones I'

Re: [OSM-dev] Minute Diffs Broken

Hi, On Tue, May 05, 2009 at 08:32:44AM +1000, Brett Henderson wrote: > Hi Everybody, > > Unfortunately the minute diffs appear to be regularly missing data. In > the last 8 hours at least 3 changesets have been missed. The ones I've > noticed are 1076325, 1076998, 1077469. These have been d

Re: [OSM-dev] Minute Diffs Broken

Karl Newman wrote: > Okay, so I'm 5 days behind on my email (although scanning through more > recent subjects doesn't show any new activity about this), and maybe > this problem has been solved already. Apologies if so, but I want to > capture my thoughts while they're fresh. Please take this wi

Re: [OSM-dev] Minute Diffs Broken

On Tue, May 5, 2009 at 3:43 PM, Brett Henderson wrote: > Matt Amos wrote: > > the XML document is parsed incrementally to save memory, rather than > > for its behaviour, but it appears that rails, lighty and fastcgi all > > support streaming input. i am unsure if they all work together, but > > t

Re: [OSM-dev] Minute Diffs Broken

On Wed, 6 May 2009, Brett Henderson wrote: > On second thoughts, a queueing mechanism may not be appropriate. A queue > would be great if the queue contained all the data needed for replication but > that isn't likely to be the case. You could A. Have a trigger put the entire row contents in

Re: [OSM-dev] Minute Diffs Broken

On Tue, May 5, 2009 at 6:16 PM, Brett Henderson wrote: > Ian Dees wrote: > >> Forgive me for injecting into this conversation part-way through, but >> would it make sense to offer an HTTP stream of the complete contents of all >> changesets as they are closed and applied to the database? >> > How

Re: [OSM-dev] Minute Diffs Broken

On Tue, May 5, 2009 at 10:53 PM, Frederik Ramm wrote: > I was trying to say that you don't have to look them up. You get the > member ids/types of the new relation, remove from that list the member > ids/types of the old relation, and then look up only the remaining > members, if any. implemented

Re: [OSM-dev] Minute Diffs Broken

Greg Troxel wrote: > Brett Henderson writes: > > > Given the use of pgsql transactions, osmosis won't see data from > uncommitted transactions. So I really meant "changes in the database, > subject to the notion that uncommitted transactions won't be visible." > Hehe, terminology strikes ag

Re: [OSM-dev] Minute Diffs Broken

Ian Dees wrote: > On Tue, May 5, 2009 at 5:54 PM, Brett Henderson > wrote: > > The second problem is that I don't always want all records in the > queue. I'd still like to be able to break up records into time > intervals rather than grabbing everything availa

Re: [OSM-dev] Minute Diffs Broken

Brett Henderson writes: >> Given the way the world is, it seems like the minute diffs really should >> be looking for new transactions, not new changesets. I can see >> Frederik's point of only exporting closed changesets, but for that to >> really make sense I think the main database has to is

Re: [OSM-dev] Minute Diffs Broken

On Tue, May 5, 2009 at 5:54 PM, Brett Henderson wrote: > > The second problem is that I don't always want all records in the > queue. I'd still like to be able to break up records into time > intervals rather than grabbing everything available in the order it was > logged. However this mightn't

Re: [OSM-dev] Minute Diffs Broken

On Tue, May 5, 2009 at 11:50 PM, Stefan de Konink wrote: > Richard Fairhurst wrote: >> Shaun McDonald wrote: >>> Relations take a long time to complete because of the >>> check (preconditions_ok?) to make sure that each node, >>> way or relation that is in it is valid. >> >> Can we do something ab

Re: [OSM-dev] Minute Diffs Broken

Matt Amos wrote: > isn't that what frederik said? It is :D I didn't see that one :) GMTA ;) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Minute Diffs Broken

Brett Henderson wrote: > Steve Singer wrote: >> On Tue, 5 May 2009, Brett Henderson wrote: >> >>> That does look interesting. I'd hope to use that outside the main >>> database though. My thoughts were to use triggers to populate short >>> term flag tables which a single threaded process would

Re: [OSM-dev] Minute Diffs Broken

Richard Fairhurst wrote: > Shaun McDonald wrote: >> Relations take a long time to complete because of the >> check (preconditions_ok?) to make sure that each node, >> way or relation that is in it is valid. > > Can we do something about this? If you are stuck with [insert your database of choic

Re: [OSM-dev] Minute Diffs Broken

Matt Amos wrote: > the XML document is parsed incrementally to save memory, rather than > for its behaviour, but it appears that rails, lighty and fastcgi all > support streaming input. i am unsure if they all work together, but > the rails docs suggest that it does. > > streaming input isn't nearl

Re: [OSM-dev] Minute Diffs Broken

Hi, Tom Hughes wrote: >> Perhaps this could be done internally: User uploads new relation; >> Rails goes through the elements of that new upload, and checks whether >> these elements were already in the previous version. If yes, no >> further check is required (because the previous relation is

Re: [OSM-dev] Minute Diffs Broken

Greg Troxel wrote: > My aim all along has been to provide people with up to date data. The > nice thing about the minute changesets is that they let you have an > offline database that exactly matches the API as of 6 minutes ago. I'd > completely agree with you if the API only released

Re: [OSM-dev] Minute Diffs Broken

Frederik Ramm wrote: > Perhaps this could be done internally: User uploads new relation; Rails > goes through the elements of that new upload, and checks whether these > elements were already in the previous version. If yes, no further check > is required (because the previous relation is assum

Re: [OSM-dev] Minute Diffs Broken

Hi, Richard Fairhurst wrote: > My personal preference would be to have discrete operations for "add > member(s) to relation" and "delete member(s) from relation" (with the usual > version number checks, of course). That way you only need to run the check > on the members that are being added/delet

Re: [OSM-dev] Minute Diffs Broken

Shaun McDonald wrote: > Relations take a long time to complete because of the > check (preconditions_ok?) to make sure that each node, > way or relation that is in it is valid. Can we do something about this? A few weeks further on and, from my Potlatch point of view, this is the only really s

Re: [OSM-dev] Minute Diffs Broken

On 5 May 2009, at 12:02, Tom Hughes wrote: > Brett Henderson wrote: > >> I've created a report at: >> http://planet.openstreetmap.org/minute/audit/report-2009-05-05_11-23.txt >> >> Some of the impacted changesets are: >> 1081602 > > OK. That one (which has 13225 nodes and 1810 ways) took 566.4609

Re: [OSM-dev] Minute Diffs Broken

On Tue, May 5, 2009 at 1:45 PM, Greg Troxel wrote: > I think we have > >  uploads == db transactions (perhaps "microchangesets" of "changeset > fragments"??) > >  changesets == (some group of uploads, with a common id and comment) > >  minute diffs == (some collection of uploads) your summary is

Re: [OSM-dev] Minute Diffs Broken

My aim all along has been to provide people with up to date data. The nice thing about the minute changesets is that they let you have an offline database that exactly matches the API as of 6 minutes ago. I'd completely agree with you if the API only released data once the changese

Re: [OSM-dev] Minute Diffs Broken

On Tue, May 5, 2009 at 12:50 PM, Frederik Ramm wrote: > Matt Amos wrote: >>> But being able to make a bunch of edits and process them in one >>> transaction is new with 0.6 and not something that users actually >>> demanded to have - it is something we thought could come in handy. >> >> doesn't it

Re: [OSM-dev] Minute Diffs Broken

Brett Henderson wrote: > Tom Hughes wrote: >> Brett Henderson wrote: >> >>> I've created a report at: >>> http://planet.openstreetmap.org/minute/audit/report-2009-05-05_11-23.txt >>> >>> >>> Some of the impacted changesets are: >>> 1081602 >> >> OK. That one (which has 13225 nodes and 1810 ways) t

Re: [OSM-dev] Minute Diffs Broken

Hi, Matt Amos wrote: >> But being able to make a bunch of edits and process them in one >> transaction is new with 0.6 and not something that users actually >> demanded to have - it is something we thought could come in handy. > > doesn't it make the client coding much simpler to have an > all-or

Re: [OSM-dev] Minute Diffs Broken

Matt Amos wrote: As for JOSM, I have already heard complaints, and people have asked how they can go back to the old non-changeset upload, because they explicitly *want* as much as possible of their changeset to upload, and are pissed off when the whole changeset fails because it contains a chang

Re: [OSM-dev] Minute Diffs Broken

Tom Hughes wrote: > Brett Henderson wrote: > >> I've created a report at: >> http://planet.openstreetmap.org/minute/audit/report-2009-05-05_11-23.txt >> >> Some of the impacted changesets are: >> 1081602 > > OK. That one (which has 13225 nodes and 1810 ways) took 566.46090 > seconds to process, wh

Re: [OSM-dev] Minute Diffs Broken

On Tue, May 5, 2009 at 11:38 AM, Frederik Ramm wrote: > Hi, > > Tom Hughes wrote: >>> Of course there's always the possibility to simply drop the >>> transaction around the diff upload. I don't think many people actually >>> *rely* on it, but it was one of the features we thought were nice to >>>

Re: [OSM-dev] Minute Diffs Broken

Brett Henderson wrote: > I've created a report at: > http://planet.openstreetmap.org/minute/audit/report-2009-05-05_11-23.txt > > Some of the impacted changesets are: > 1081602 OK. That one (which has 13225 nodes and 1810 ways) took 566.46090 seconds to process, which is just short of 10 minute

Re: [OSM-dev] Minute Diffs Broken

On Tue, May 5, 2009 at 8:34 AM, Tom Hughes wrote: > Steve Singer wrote: > >> I'm not familiar with the rails API code, but I want to make sure that the >> nodes.timestamp column your querying isn't being populated with the >> postgresql now() function but instead with some time that rails computes

Re: [OSM-dev] Minute Diffs Broken

Hi, Tom Hughes wrote: >> Of course there's always the possibility to simply drop the >> transaction around the diff upload. I don't think many people actually >> *rely* on it, but it was one of the features we thought were nice to >> have... > > Er no there isn't. That is absolutely positively

Re: [OSM-dev] Minute Diffs Broken

Tom Hughes wrote: > Brett Henderson wrote: > I have to say I don't fully understand what the issue you're seeing is > as I only skimmed the vast amount of discussion that went on overnight. > > Are you really seeing a five minute delay? Or just a delay that > happens to be over a five minute boun

Re: [OSM-dev] Minute Diffs Broken

Frederik Ramm wrote: > Of course there's always the possibility to simply drop the transaction > around the diff upload. I don't think many people actually *rely* on it, > but it was one of the features we thought were nice to have... Er no there isn't. That is absolutely positively definitely

Re: [OSM-dev] Minute Diffs Broken

Hi, Brett Henderson wrote: >> We're only just getting things stable again and I have no desire to >> start fiddling with things just yet - we need time to let what we have >> bed in properly. > I understand your concerns. I wouldn't have even mentioned it if I had > valid alternatives. At th

Re: [OSM-dev] Minute Diffs Broken

Brett Henderson wrote: > If you're referring to multi-mastered clustered databases then that is a > whole different problem that shouldn't be confused with what I'm trying > to achieve. I'm simply trying to provide a way for people to access > regular updates in a read-only fashion where data

Re: [OSM-dev] Minute Diffs Broken

Hi, Brett Henderson wrote: > My aim all along has been to provide people with up to date data. The > nice thing about the minute changesets is that they let you have an > offline database that exactly matches the API as of 6 minutes ago. I'd > completely agree with you if the API only release

Re: [OSM-dev] Minute Diffs Broken

Tom Hughes wrote: > Brett Henderson wrote: > >> That does look interesting. I'd hope to use that outside the main >> database though. My thoughts were to use triggers to populate short >> term flag tables which a single threaded process would read, use as >> keys to select modified data into a

Re: [OSM-dev] Minute Diffs Broken

Brett Henderson wrote: >>> Also, there's a potential flaw with this approach. Lets say I >>> create node 100 with version 1 in changeset 10 in Potlatch and leave >>> my changeset open. You then come along with JOSM and edit node 100 >>> creating version 2 within changeset 11 and close your cha

Re: [OSM-dev] Minute Diffs Broken

Frederik Ramm wrote: > Hi, > > Brett Henderson wrote: >> I'm fairly uncomfortable with this approach. It could be very >> confusing. But I'm prepared to be swayed, it is certainly simple :-) > > As I tried to explain, I don't really find it confusing; I actually > *like* the idea of changesets

Re: [OSM-dev] Minute Diffs Broken

Brett Henderson wrote: > That does look interesting. I'd hope to use that outside the main > database though. My thoughts were to use triggers to populate short > term flag tables which a single threaded process would read, use as keys > to select modified data into an offline database, then

Re: [OSM-dev] Minute Diffs Broken

Steve Singer wrote: > I'm not familiar with the rails API code, but I want to make sure that the > nodes.timestamp column your querying isn't being populated with the > postgresql now() function but instead with some time that rails computes. > (The now() function returns the time when your tra

Re: [OSM-dev] Minute Diffs Broken

Hi, Brett Henderson wrote: > I'm fairly uncomfortable with this approach. It could be very > confusing. But I'm prepared to be swayed, it is certainly simple :-) As I tried to explain, I don't really find it confusing; I actually *like* the idea of changesets not being propagated until they a

Re: [OSM-dev] Minute Diffs Broken

Steve Singer wrote: > On Tue, 5 May 2009, Brett Henderson wrote: > >> That does look interesting. I'd hope to use that outside the main >> database though. My thoughts were to use triggers to populate short >> term flag tables which a single threaded process would read, use as >> keys to selec

Re: [OSM-dev] Minute Diffs Broken

On Tue, 5 May 2009, Brett Henderson wrote: > That does look interesting. I'd hope to use that outside the main database > though. My thoughts were to use triggers to populate short term flag tables > which a single threaded process would read, use as keys to select modified > data into an off

Re: [OSM-dev] Minute Diffs Broken

Steve Singer wrote: > > What you want is the timestamp that the change was committed at, not > the timestamp it was inserted at. However there is no way to get this > with postgresql. > > The options that come to mind are > > A) Modify the rails code to insert a row in a transaction table with a

Re: [OSM-dev] Minute Diffs Broken

On Mon, 4 May 2009, Greg Troxel wrote: > With select by time, it would still be buggy. But if the select was > "all changesets > X" where X was the highest changeset in the previous > select, it would work, because there would have to be a total ordering > of transactions (at least as far as anyo

Re: [OSM-dev] Minute Diffs Broken

On Mon, 4 May 2009, Greg Troxel wrote: > > openstreetmap-...@scd.debian.net writes: > >> Would this work? >> >> How about the situation: >> >> Changeset A creates a node >> >> Changeset B uses the node in a way >> >> Changeset B closes >> >> (Later) Changeset A closes > > Transactions are intended

Re: [OSM-dev] Minute Diffs Broken

Frederik Ramm wrote: > I don't see why these systems should show data from un-closes > changesets. The way I like to think of changesets, this might even be > misleading - think of someone deleting a road because he wants to > re-draw it from a better GPX trace. I would of course do both inside

Re: [OSM-dev] Minute Diffs Broken

Sorry, I was assuming that a changeset and a database transaction were the same thing. If not, we need a sequence number on additions to the history table, and use that for knowing what's fresh. pgpgVUHBSeHce.pgp Description: PGP signature ___ dev ma

Re: [OSM-dev] Minute Diffs Broken

On Tue, 5 May 2009, Brett Henderson wrote: > The way osmosis identifies changed records is by query the history table > for entities with a timestamp within a time interval. The time interval > will be an hour long for hourly diffs, a minute long for minute diffs. > > For example, the node query

Re: [OSM-dev] Minute Diffs Broken

Frederik Ramm writes: > Hi, > > Greg Troxel wrote: >> So obviously we aren't running "SET TRANSACTION ISOLATION LEVEL >> SERIALIZABLE", since that would kill performance and make things harder, >> but it would solve this :-) > > How so? The problem seems to be too much transaction isolation, not

Re: [OSM-dev] Minute Diffs Broken

Greg Troxel wrote: Frederik Ramm writes: 3. Make a semantic change to the way we handle diffs: Let the diff for interval X not be "all changes with timestamp within X" but instead "all changes that happened in a changeset that was closed within X". Changesets not being atomic should pose

Re: [OSM-dev] Minute Diffs Broken

openstreetmap-...@scd.debian.net writes: > Would this work? > > How about the situation: > > Changeset A creates a node > > Changeset B uses the node in a way > > Changeset B closes > > (Later) Changeset A closes Transactions are intended to avoid this. It may be that the changeset B transactio

Re: [OSM-dev] Minute Diffs Broken

Hi, Greg Troxel wrote: > So obviously we aren't running "SET TRANSACTION ISOLATION LEVEL > SERIALIZABLE", since that would kill performance and make things harder, > but it would solve this :-) How so? The problem seems to be too much transaction isolation, not too little. If we were operating o

Re: [OSM-dev] Minute Diffs Broken

In article <49ff86a3.8060...@remote.org> frede...@remote.org wrote: >Thinking about possible solutions: .. >3. Make a semantic change to the way we handle diffs: Let the diff for >interval X not be "all changes with timestamp within X" but instead "all >changes that happened in a changeset that

Re: [OSM-dev] Minute Diffs Broken

Hi, Brett Henderson wrote: >> 3. Make a semantic change to the way we handle diffs: Let the diff for >> interval X not be "all changes with timestamp within X" but instead >> "all changes that happened in a changeset that was closed within X". [...] > I think this would introduce far too larg

Re: [OSM-dev] Minute Diffs Broken

Frederik Ramm writes: > 3. Make a semantic change to the way we handle diffs: Let the diff for > interval X not be "all changes with timestamp within X" but instead "all > changes that happened in a changeset that was closed within X". > Changesets not being atomic should pose no problem for

Re: [OSM-dev] Minute Diffs Broken

Frederik Ramm wrote: > Hi, > > Brett Henderson wrote: >> I'm not reading any of the changeset table data so the behaviour of >> the closed_at field doesn't affect osmosis. The changeset table is >> effectively useless to osmosis processing because changesets aren't >> atomic. > > Thinking about

Re: [OSM-dev] Minute Diffs Broken

Hi, Brett Henderson wrote: > I'm not reading any of the changeset table data so the behaviour of the > closed_at field doesn't affect osmosis. The changeset table is > effectively useless to osmosis processing because changesets aren't > atomic. Thinking about possible solutions: 1. When upd

Re: [OSM-dev] Minute Diffs Broken

Hi, Brett Henderson wrote: > Presumably this is because the large changesets > sometimes take longer than 5 minutes to process (seems awfully slow but > that's what I'm seeing) So we're talking about diff uploads then (because only they are done in a transaction). I was about to say this could

Re: [OSM-dev] Minute Diffs Broken

Brett Henderson wrote: > Frederik Ramm wrote: >> Hi, >> >> Brett Henderson wrote: >>> Unfortunately the minute diffs appear to be regularly missing data. >>> In the last 8 hours at least 3 changesets have been missed. The >>> ones I've noticed are 1076325, 1076998, 1077469. These have been >>

Re: [OSM-dev] Minute Diffs Broken

Frederik Ramm wrote: > Hi, > > Brett Henderson wrote: >> Unfortunately the minute diffs appear to be regularly missing data. >> In the last 8 hours at least 3 changesets have been missed. The ones >> I've noticed are 1076325, 1076998, 1077469. These have been detected >> by comparing the norm

Re: [OSM-dev] Minute Diffs Broken

Hi, Brett Henderson wrote: > Unfortunately the minute diffs appear to be regularly missing data. In > the last 8 hours at least 3 changesets have been missed. The ones I've > noticed are 1076325, 1076998, 1077469. These have been detected by > comparing the normal minute diffs against anothe

[OSM-dev] Minute Diffs Broken

Hi Everybody, Unfortunately the minute diffs appear to be regularly missing data. In the last 8 hours at least 3 changesets have been missed. The ones I've noticed are 1076325, 1076998, 1077469. These have been detected by comparing the normal minute diffs against another minute diff process