Re: [OSM-talk] OSM very old data

[Tom Ka]

From my experience, all this weird things are caused by missing
history, so I would guess the export may be fine and current API
queries are incomplete thus showing incorrect information sometimes.

Bye

po 16. 12. 2019 v 12:33 odesílatel Martin Koppenhoefer
 napsal:
>
> Maybe the following is a result of the history not being available for older 
> times, in the v6-planet-060403.osm there are objects that seem they shouldn't 
> be there? E.g. node 393303 according to the current API has its first version 
> from Aug, 1 2007 (according to JOSM) or 31/07/2007 according to the website 
> (not sure which of them is showing UTC and which local time, neither are 
> telling it explicitly, or if it has to do which time is displayed: when the 
> changeset was created or closed).
> Anyway, this node shouldn't be in a dump from 2006 (supposing that the fact 
> there were redactions e.g. for the license change, would still be visible, 
> and that ids have not been reassigned even in these old days)?
>
> There are more nodes like this, e.g. 589954 
> https://www.openstreetmap.org/node/589954/history  (this one shows that 
> redactions are accounted for, because you can see that there is a version 1 
> and 2 which cannot be shown any more).
>
> 196651 https://www.openstreetmap.org/node/196651/history
> This last one indicates that something weird is going on with the API (maybe 
> due to missing history) because this is a deletion in version 1 and no 
> redactions are shown for this object.
>
> Cheers
> Martin

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Martin Koppenhoefer]

Maybe the following is a result of the history not being available for
older times, in the v6-planet-060403.osm there are objects that seem they
shouldn't be there? E.g. node 393303 according to the current API has its
first version from Aug, 1 2007 (according to JOSM) or 31/07/2007 according
to the website (not sure which of them is showing UTC and which local time,
neither are telling it explicitly, or if it has to do which time is
displayed: when the changeset was created or closed).
Anyway, this node shouldn't be in a dump from 2006 (supposing that the fact
there were redactions e.g. for the license change, would still be visible,
and that ids have not been reassigned even in these old days)?

There are more nodes like this, e.g. 589954
https://www.openstreetmap.org/node/589954/history  (this one shows that
redactions are accounted for, because you can see that there is a version 1
and 2 which cannot be shown any more).

196651 https://www.openstreetmap.org/node/196651/history
This last one indicates that something weird is going on with the API
(maybe due to missing history) because this is a deletion in version 1 and
no redactions are shown for this object.

Cheers
Martin
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Tom Ka]

Just few final notes (mainly for list archive):

- oldest usable .osm file is from 2006-04-03. Those old files have
plenty of nodes and ways (segments), but very few tagged, so only
geometry is there.
(https://planet.openstreetmap.org/cc-by-sa/planet-060403-fromARobinson.osm.7z)
- The oldest object in CZ (one of my goals) with proof of history is
from 2005-08-15: https://www.openstreetmap.org/node/172533/history
(lowest id is 172510 from the same edit - can be seen in 2006-08-18
data, but it's not in .osc files)
- There is no complete history before 2006-04-03. There are change
files (.osc) per day starting from 2005-04-17 but without initial .osm
to start with, they are of limited use (but contain timestamp and some
user info, used to identify node 172533).
(https://planet.openstreetmap.org/cc-by-sa/history/2005/0417-0418.osc.gz)
- I tried to have a look on fosm.org (those old data are the same),
but seems they do not have old history either.

Thanks for all the support here.
tom.k

čt 12. 12. 2019 v 14:29 odesílatel Tom Ka  napsal:
>
> Hi,
>
> successfully processed all data from cc-by-sa folder up to 2006-04-03.
> All results in v0.6 are available here:
>
> https://osm.fit.vutbr.cz/extracts/
>
> Bye tom.k
>
> čt 5. 12. 2019 v 13:03 odesílatel Tom Ka  napsal:
> >
> > Just to note - most of these old (v0.3) data have errors in UTF-8
> > encoding in some tag values (mainly name).
> >
> > Bye tom.k
> >
> > so 30. 11. 2019 v 13:09 odesílatel Tom Ka  napsal:
> > >
> > > Final v0.6 extracts for planet are available here:
> > >
> > > http://osm.fit.vutbr.cz/extracts/
> > >
> > > Will prepare some info about the process for others.
> > >
> > > Thanks to all, who helped.
> > >
> > > pá 29. 11. 2019 v 11:05 odesílatel Frederik Ramm  
> > > napsal:
> > > >
> > > > Hi,
> > > >
> > > > On 29.11.19 10:43, Tom Ka wrote:
> > > > > What it
> > > > > the meaning - deleted segments, so should I ignore those ways?
> > > >
> > > > I guess so.
> > > >
> > > > > 2) from 061205 the size increases, so I guess it contain some
> > > > > reasonable data, but for older (planet-061128.osm.bz2 -
> > > > > planet-060818.osm.bz2  ) there is size jump, which is strange. Is
> > > > > there any reason for this?
> > > >
> > > > Possibly https://wiki.openstreetmap.org/wiki/Old_TIGER_Import_2005/2006
> > > >
> > > > Bye
> > > > Frederik
> > > >
> > > > --
> > > > Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
> > > >
> > > > ___
> > > > talk mailing list
> > > > talk@openstreetmap.org
> > > > https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Tom Ka]

čt 12. 12. 2019 v 15:31 odesílatel Martin Koppenhoefer
 napsal:
>
> Very interesting! The timestamps are not valid (quite often there is a time 
> tag, which seems to be the GPG-logger time) and there is no user information 
> though. Also a lot of way ids are negative and all objects are in version 1. 
> Are these known limitations?
> I have only looked at v6-planet-060403.osm.bz2 so far

Timestamps have to be added where missing for following processing to
work (I use 2000-01-01 so you can see, what's artificial timestamp).
User info is missing in source v0.3 as well (but was not necessary for
processing to v0.6 so I did not add any dumb value). Version
information was not modified by me, and negative way ids are from the
04to05.pl script (segments to ways). Will publish the worksflow in a
diary soon, so anyone can check/repeat it.

Bye tom.k

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Martin Koppenhoefer]

Very interesting! The timestamps are not valid (quite often there is a time
tag, which seems to be the GPG-logger time) and there is no user
information though. Also a lot of way ids are negative and all objects are
in version 1. Are these known limitations?
I have only looked at v6-planet-060403.osm.bz2 so far

Cheers,
Martin
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Tom Ka]

Hi,

successfully processed all data from cc-by-sa folder up to 2006-04-03.
All results in v0.6 are available here:

https://osm.fit.vutbr.cz/extracts/

Bye tom.k

čt 5. 12. 2019 v 13:03 odesílatel Tom Ka  napsal:
>
> Just to note - most of these old (v0.3) data have errors in UTF-8
> encoding in some tag values (mainly name).
>
> Bye tom.k
>
> so 30. 11. 2019 v 13:09 odesílatel Tom Ka  napsal:
> >
> > Final v0.6 extracts for planet are available here:
> >
> > http://osm.fit.vutbr.cz/extracts/
> >
> > Will prepare some info about the process for others.
> >
> > Thanks to all, who helped.
> >
> > pá 29. 11. 2019 v 11:05 odesílatel Frederik Ramm  
> > napsal:
> > >
> > > Hi,
> > >
> > > On 29.11.19 10:43, Tom Ka wrote:
> > > > What it
> > > > the meaning - deleted segments, so should I ignore those ways?
> > >
> > > I guess so.
> > >
> > > > 2) from 061205 the size increases, so I guess it contain some
> > > > reasonable data, but for older (planet-061128.osm.bz2 -
> > > > planet-060818.osm.bz2  ) there is size jump, which is strange. Is
> > > > there any reason for this?
> > >
> > > Possibly https://wiki.openstreetmap.org/wiki/Old_TIGER_Import_2005/2006
> > >
> > > Bye
> > > Frederik
> > >
> > > --
> > > Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
> > >
> > > ___
> > > talk mailing list
> > > talk@openstreetmap.org
> > > https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Tom Ka]

Just to note - most of these old (v0.3) data have errors in UTF-8
encoding in some tag values (mainly name).

Bye tom.k

so 30. 11. 2019 v 13:09 odesílatel Tom Ka  napsal:
>
> Final v0.6 extracts for planet are available here:
>
> http://osm.fit.vutbr.cz/extracts/
>
> Will prepare some info about the process for others.
>
> Thanks to all, who helped.
>
> pá 29. 11. 2019 v 11:05 odesílatel Frederik Ramm  napsal:
> >
> > Hi,
> >
> > On 29.11.19 10:43, Tom Ka wrote:
> > > What it
> > > the meaning - deleted segments, so should I ignore those ways?
> >
> > I guess so.
> >
> > > 2) from 061205 the size increases, so I guess it contain some
> > > reasonable data, but for older (planet-061128.osm.bz2 -
> > > planet-060818.osm.bz2  ) there is size jump, which is strange. Is
> > > there any reason for this?
> >
> > Possibly https://wiki.openstreetmap.org/wiki/Old_TIGER_Import_2005/2006
> >
> > Bye
> > Frederik
> >
> > --
> > Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
> >
> > ___
> > talk mailing list
> > talk@openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Tom Ka]

Final v0.6 extracts for planet are available here:

http://osm.fit.vutbr.cz/extracts/

Will prepare some info about the process for others.

Thanks to all, who helped.

pá 29. 11. 2019 v 11:05 odesílatel Frederik Ramm  napsal:
>
> Hi,
>
> On 29.11.19 10:43, Tom Ka wrote:
> > What it
> > the meaning - deleted segments, so should I ignore those ways?
>
> I guess so.
>
> > 2) from 061205 the size increases, so I guess it contain some
> > reasonable data, but for older (planet-061128.osm.bz2 -
> > planet-060818.osm.bz2  ) there is size jump, which is strange. Is
> > there any reason for this?
>
> Possibly https://wiki.openstreetmap.org/wiki/Old_TIGER_Import_2005/2006
>
> Bye
> Frederik
>
> --
> Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Frederik Ramm]

Hi,

On 29.11.19 10:43, Tom Ka wrote:
> What it
> the meaning - deleted segments, so should I ignore those ways?

I guess so.

> 2) from 061205 the size increases, so I guess it contain some
> reasonable data, but for older (planet-061128.osm.bz2 -
> planet-060818.osm.bz2  ) there is size jump, which is strange. Is
> there any reason for this?

Possibly https://wiki.openstreetmap.org/wiki/Old_TIGER_Import_2005/2006

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Maarten Deen]

On 2019-11-29 10:43, Tom Ka wrote:


https://kasparkovi.net/osm/
(top letf selection for older maps)


No answer to your questions, but cool to see the map grow like that!

Maarten

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Tom Ka]

Hi, I tried to convert the old 0.3 data and have some questions if
anybody remembers those days:
(all data are from https://planet.openstreetmap.org/cc-by-sa/ )

1) planet-061205.osm contains (other have similar too):











the 04to05.pl end with:




.

which is not valid for osmosis0-0.35 to further process it. What it
the meaning - deleted segments, so should I ignore those ways?

2) from 061205 the size increases, so I guess it contain some
reasonable data, but for older (planet-061128.osm.bz2 -
planet-060818.osm.bz2  ) there is size jump, which is strange. Is
there any reason for this?

I must admit, this OSM archeology is a sort of fun :-)

I already have tiles for old CZ maps up to 2007.10.10, have to prepare
some better UI, can have a look here:

https://kasparkovi.net/osm/
(top letf selection for older maps)

Bye tom.k

po 28. 10. 2019 v 10:07 odesílatel Tom Ka  napsal:
>
> Thank you, I was thinking about similar way too.
>
> Best tkk.
>
> so 26. 10. 2019 v 23:22 odesílatel Frederik Ramm  napsal:
> >
> > Hi,
> >
> > On 25.10.19 16:18, Tom Ka wrote:
> > > OK, one question that remains unanswered will be: what was the first
> > > object in Czech republic :-)
> >
> > Nodes are generally numbered in ascending order, and have been from the
> > start. Since anything that can be mapped either is a node or depends on
> > a node, it should be possible to find the node with the lowest node id
> > in the Czech Republic.
> >
> > I'll offer https://www.openstreetmap.org/node/172508 - the web site says
> > "Version #1 · Changeset #209315 - edited Tue 06 Feb 2007". At the same
> > time this node is already present in the file called
> > "planet-060501-FromLA2.osm.bz2" which purports to be from May 2006.
> > Subtract 8 from the node ID and you get a node where the API claims it
> > was first edited in August 2005, so something is a bit fishy here with
> > regards to the exact timestamps. Nonetheless, 172508 seems to be the
> > lowest node ID in the country. Incidentally it is still the lowest node
> > ID in the country today, but it is also the lowest node ID in that old
> > planet file.
> >
> > Bye
> > Frederik
> >
> > --
> > Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
> >
> > ___
> > talk mailing list
> > talk@openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Tom Ka]

Thank you, I was thinking about similar way too.

Best tkk.

so 26. 10. 2019 v 23:22 odesílatel Frederik Ramm  napsal:
>
> Hi,
>
> On 25.10.19 16:18, Tom Ka wrote:
> > OK, one question that remains unanswered will be: what was the first
> > object in Czech republic :-)
>
> Nodes are generally numbered in ascending order, and have been from the
> start. Since anything that can be mapped either is a node or depends on
> a node, it should be possible to find the node with the lowest node id
> in the Czech Republic.
>
> I'll offer https://www.openstreetmap.org/node/172508 - the web site says
> "Version #1 · Changeset #209315 - edited Tue 06 Feb 2007". At the same
> time this node is already present in the file called
> "planet-060501-FromLA2.osm.bz2" which purports to be from May 2006.
> Subtract 8 from the node ID and you get a node where the API claims it
> was first edited in August 2005, so something is a bit fishy here with
> regards to the exact timestamps. Nonetheless, 172508 seems to be the
> lowest node ID in the country. Incidentally it is still the lowest node
> ID in the country today, but it is also the lowest node ID in that old
> planet file.
>
> Bye
> Frederik
>
> --
> Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Frederik Ramm]

Hi,

On 25.10.19 16:18, Tom Ka wrote:
> OK, one question that remains unanswered will be: what was the first
> object in Czech republic :-)

Nodes are generally numbered in ascending order, and have been from the
start. Since anything that can be mapped either is a node or depends on
a node, it should be possible to find the node with the lowest node id
in the Czech Republic.

I'll offer https://www.openstreetmap.org/node/172508 - the web site says
"Version #1 · Changeset #209315 - edited Tue 06 Feb 2007". At the same
time this node is already present in the file called
"planet-060501-FromLA2.osm.bz2" which purports to be from May 2006.
Subtract 8 from the node ID and you get a node where the API claims it
was first edited in August 2005, so something is a bit fishy here with
regards to the exact timestamps. Nonetheless, 172508 seems to be the
lowest node ID in the country. Incidentally it is still the lowest node
ID in the country today, but it is also the lowest node ID in that old
planet file.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Tom Ka]

pá 25. 10. 2019 v 15:08 odesílatel Frederik Ramm  napsal:
> On 25.10.19 09:52, Tom Ka wrote:
> > - any recommendation for tool/app/etc to process v0.3 data (up to
> > 01.05.2006 on planet/cc-by-sa) (and convert them to v0.6)?
>
> There's a perl script "04to05.pl" in SVN, this works for 0.3 data as well.

Perfect, will try. For archive, it can be found here:
https://svn.openstreetmap.org/applications/utils/conv05/

> > - is history before 01.05.2006 (easily) available (other archive or
> > local source) (Europe or CZ would be enough)?
>
> We didn't have history files at the time and those were times when we
> still used segments, so in today's history files these early times
> cannot be represented (at least not for ways). I am not aware of any
> older files. But the amount of .cz data in that old planet file is so
> small (7251 nodes and 7453 extremely short ways) that it probably
> wouldn't make a lot of sense to go back further.

OK, one question that remains unanswered will be: what was the first
object in Czech republic :-)

Thanks tkk.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Frederik Ramm]

Hi,

On 25.10.19 09:52, Tom Ka wrote:
> - any recommendation for tool/app/etc to process v0.3 data (up to
> 01.05.2006 on planet/cc-by-sa) (and convert them to v0.6)?

There's a perl script "04to05.pl" in SVN, this works for 0.3 data as well.

> - is history before 01.05.2006 (easily) available (other archive or
> local source) (Europe or CZ would be enough)?

We didn't have history files at the time and those were times when we
still used segments, so in today's history files these early times
cannot be represented (at least not for ways). I am not aware of any
older files. But the amount of .cz data in that old planet file is so
small (7251 nodes and 7453 extremely short ways) that it probably
wouldn't make a lot of sense to go back further.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Tom Ka]

Most dumps/diffs are newer then 2007, the only older dirs are in
https://planet.osm.org/cc-by-sa/history/, but those are diffs (not
full dumps) and all in 2004 and lot in 2005 are empty, which is weird.
Being diffs I would need starting dump to update to use them anyway.

Bye tkk

pá 25. 10. 2019 v 13:48 odesílatel Mateusz Konieczny
 napsal:
>
>
>
>
> 25 Oct 2019, 09:52 by tomas.kaspa...@gmail.com:
>
> - is history before 01.05.2006 (easily) available (other archive or
> local source) (Europe or CZ would be enough)?
>
> https://planet.osm.org/cc-by-sa/ ?
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] OSM very old data

[Mateusz Konieczny]

25 Oct 2019, 09:52 by tomas.kaspa...@gmail.com:

> - is history before 01.05.2006 (easily) available (other archive or
> local source) (Europe or CZ would be enough)?
>
https://planet.osm.org/cc-by-sa/  ?

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] OSM very old data

[Tom Ka]

Hello,

I'm trying to create map of Czech Republic with older versions of OSM
data. No problem to proceed to 21.04.2009 (first v0.6 data) and with
older (0.35 + --migrate) osmosis to 10.10.2007 (first v0.5 data). All
data are from https://planet.openstreetmap.org/cc-by-sa/. I got in
troubles when I want to go further in history:

- any recommendation for tool/app/etc to process v0.3 data (up to
01.05.2006 on planet/cc-by-sa) (and convert them to v0.6)?
- is history before 01.05.2006 (easily) available (other archive or
local source) (Europe or CZ would be enough)?

Thank you in advance
tkk.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk