Re: [OSM-dev] Import error using osm2pgsql in slim mode
Hi Jon,
I just tried your patch but now I always get the following error when doing
an import:
node_changed_mark failed: ERROR: prepared statement "node_changed_mark"
does not exist
Any ideas?
Thanks,
Jason
On Fri, Jan 22, 2010 at 8:48 PM, Jason Beverage wrote:
> Hi John,
>
> Thanks for the patch, I will give it a test this weekend! I think it would
> be a decent option on the command line if it works out well.
>
> Jason
>
>
> On Fri, Jan 22, 2010 at 6:38 PM, Jon Burgess
> wrote:
>
>> > > I'm working with osm2pgsql (latest trunk version) to import OSM into
>> > a
>> > > PostgreSQL/PostGIS database. The issue I'm having is that if I use
>> > > the --slim option I'll occasionally get an error similar to:
>> > >
>> > > Going over pending relations
>> > > COPY_END for COPY osm_rels FROM STDIN;
>> > > failed: ERROR: duplicate key value violates unique constraint
>> > > "osm_rels_pkey"
>> > > CONTEXT: COPY osm_rels, line 1207: "2841320323
>> > >
>> >
>> {42198453,20559277,20559273,20558263,20494565,40702583,19846737,19826462,20582455,20585..."
>> > >
>> > > If I don't use slim mode, it seems to always work. Right now I'm
>> > just
>> > > testing with state snapshots from http://downloads.cloudmade.com/,
>> > but
>> > > I'd like to be able to import the whole OSM planet database and I'd
>> > > really like to be able to use slim mode so I can do incremental
>> > > updates and reduce the memory requirements for importing.
>> >
>> > If I remember correctly this error occurs when you try to import two
>> > data sets which contain some overlapping data. In your case it appears
>> > that relation ID 284132 appears in both the data extracts. Only the
>> > slim
>> > mode keeps a copy of this node/way/relation data and triggers this
>> > error.
>>
>> You could try the attached patch to osm2pgsql. This makes it treat all
>> new data as a modify operation which should avoid problems caused by
>> duplicate data. Unfortunately this makes the data import process
>> significantly slower so I won't be applying this change to the trunk
>> code.
>>
>> I will think about adding this as a new command line option and try to
>> figure out if there is something that can be done to reduce the
>> performance penalty.
>>
>>Jon
>>
>>
>>
>>
>
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
Hi John,
Thanks for the patch, I will give it a test this weekend! I think it would
be a decent option on the command line if it works out well.
Jason
On Fri, Jan 22, 2010 at 6:38 PM, Jon Burgess wrote:
> > > I'm working with osm2pgsql (latest trunk version) to import OSM into
> > a
> > > PostgreSQL/PostGIS database. The issue I'm having is that if I use
> > > the --slim option I'll occasionally get an error similar to:
> > >
> > > Going over pending relations
> > > COPY_END for COPY osm_rels FROM STDIN;
> > > failed: ERROR: duplicate key value violates unique constraint
> > > "osm_rels_pkey"
> > > CONTEXT: COPY osm_rels, line 1207: "2841320323
> > >
> >
> {42198453,20559277,20559273,20558263,20494565,40702583,19846737,19826462,20582455,20585..."
> > >
> > > If I don't use slim mode, it seems to always work. Right now I'm
> > just
> > > testing with state snapshots from http://downloads.cloudmade.com/,
> > but
> > > I'd like to be able to import the whole OSM planet database and I'd
> > > really like to be able to use slim mode so I can do incremental
> > > updates and reduce the memory requirements for importing.
> >
> > If I remember correctly this error occurs when you try to import two
> > data sets which contain some overlapping data. In your case it appears
> > that relation ID 284132 appears in both the data extracts. Only the
> > slim
> > mode keeps a copy of this node/way/relation data and triggers this
> > error.
>
> You could try the attached patch to osm2pgsql. This makes it treat all
> new data as a modify operation which should avoid problems caused by
> duplicate data. Unfortunately this makes the data import process
> significantly slower so I won't be applying this change to the trunk
> code.
>
> I will think about adding this as a new command line option and try to
> figure out if there is something that can be done to reduce the
> performance penalty.
>
>Jon
>
>
>
>
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
> > I'm working with osm2pgsql (latest trunk version) to import OSM into
> a
> > PostgreSQL/PostGIS database. The issue I'm having is that if I use
> > the --slim option I'll occasionally get an error similar to:
> >
> > Going over pending relations
> > COPY_END for COPY osm_rels FROM STDIN;
> > failed: ERROR: duplicate key value violates unique constraint
> > "osm_rels_pkey"
> > CONTEXT: COPY osm_rels, line 1207: "2841320323
> >
> {42198453,20559277,20559273,20558263,20494565,40702583,19846737,19826462,20582455,20585..."
> >
> > If I don't use slim mode, it seems to always work. Right now I'm
> just
> > testing with state snapshots from http://downloads.cloudmade.com/,
> but
> > I'd like to be able to import the whole OSM planet database and I'd
> > really like to be able to use slim mode so I can do incremental
> > updates and reduce the memory requirements for importing.
>
> If I remember correctly this error occurs when you try to import two
> data sets which contain some overlapping data. In your case it appears
> that relation ID 284132 appears in both the data extracts. Only the
> slim
> mode keeps a copy of this node/way/relation data and triggers this
> error.
You could try the attached patch to osm2pgsql. This makes it treat all
new data as a modify operation which should avoid problems caused by
duplicate data. Unfortunately this makes the data import process
significantly slower so I won't be applying this change to the trunk
code.
I will think about adding this as a new command line option and try to
figure out if there is something that can be done to reduce the
performance penalty.
Jon
Index: osm2pgsql.c
===
--- osm2pgsql.c (revision 19148)
+++ osm2pgsql.c (working copy)
@@ -137,7 +137,7 @@
actions_t new_action = ACTION_NONE;
xmlChar *action = xmlTextReaderGetAttribute( reader, BAD_CAST "action" );
if( action == NULL )
-new_action = ACTION_CREATE;
+new_action = ACTION_MODIFY;
else if( strcmp((char *)action, "modify") == 0 )
new_action = ACTION_MODIFY;
else if( strcmp((char *)action, "delete") == 0 )
@@ -160,7 +160,7 @@
if (xmlStrEqual(name, BAD_CAST "osm"))
{
filetype = FILETYPE_OSM;
-action = ACTION_CREATE;
+action = ACTION_MODIFY;
}
else if (xmlStrEqual(name, BAD_CAST "osmChange"))
{
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
On Jan 21, 2010, at 1:19 PM, Frederik Ramm wrote: > It is fixed now. > > Bye > Frederik Thanks Frederik! Dane ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
Hi, Brett Henderson wrote: > I was wondering how you fixed it. That's neat. Do you need the > --sort-change-0.6? You probably do, but I don't know what order you end > up with using --apply-change with full history diffs. I tried it without, and it complained - for some reason the newer version came before the older version in the file. > Perhaps I should create a --read-zero task. An empty osm file is often > handy. It would avoid having to create empty.osm files like that. True. Or maybe "--read-inline data=''" ;-) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
On Fri, Jan 22, 2010 at 8:54 AM, Frederik Ramm wrote: > Hi, > > > Brett Henderson wrote: > >> If you use the latest SVN version it will detect if multiple versions of >> an entity exist and throw an error in the --apply-change task. I'll try to >> do another official release of Osmosis over the next day or two with this >> change included. >> > > Just for laughs - and wonder if you can come up with something better? - > this is what I did to remove the duplicate objects from an existing OSM file > named "faulty.osm": > > echo "" > empty.osm > > osmosis --rx faulty.osm --rx empty.osm --dc --sort-change-0.6 --simc --wxc > good.osc > > osmosis --rxc good.osc --rx empty.osm --ac --wx good.osm > > (The two osmosis steps could be written as one but that would probably make > it more confusing.) I was wondering how you fixed it. That's neat. Do you need the --sort-change-0.6? You probably do, but I don't know what order you end up with using --apply-change with full history diffs. Perhaps I should create a --read-zero task. An empty osm file is often handy. It would avoid having to create empty.osm files like that. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
Hi, Brett Henderson wrote: > If you use the latest SVN version it will detect if multiple versions of > an entity exist and throw an error in the --apply-change task. I'll try > to do another official release of Osmosis over the next day or two with > this change included. Just for laughs - and wonder if you can come up with something better? - this is what I did to remove the duplicate objects from an existing OSM file named "faulty.osm": echo "" > empty.osm osmosis --rx faulty.osm --rx empty.osm --dc --sort-change-0.6 --simc --wxc good.osc osmosis --rxc good.osc --rx empty.osm --ac --wx good.osm (The two osmosis steps could be written as one but that would probably make it more confusing.) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
On Fri, Jan 22, 2010 at 8:19 AM, Frederik Ramm wrote: > Hi, > > Jon Burgess wrote: > > It seems there are two versions of a way in the haiti data: > > My bad. I forgot a --simc switch when I moved from Haiti only to > Haiti+DomRep this afternoon and this broke things. > > It is fixed now. > If you use the latest SVN version it will detect if multiple versions of an entity exist and throw an error in the --apply-change task. I'll try to do another official release of Osmosis over the next day or two with this change included. This has caught a number of people out now, and it's very non-obvious. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
Hi, Jon Burgess wrote: > It seems there are two versions of a way in the haiti data: My bad. I forgot a --simc switch when I moved from Haiti only to Haiti+DomRep this afternoon and this broke things. It is fixed now. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
On Thu, 2010-01-21 at 11:29 -0800, Dane Springmeyer wrote: > On Jan 21, 2010, at 5:51 AM, Mike Warren wrote: > > > Jon Burgess writes: > > > >> If I remember correctly this error occurs when you try to import two > >> data sets which contain some overlapping data. In your case it > >> appears > >> that relation ID 284132 appears in both the data extracts. Only the > >> slim > >> mode keeps a copy of this node/way/relation data and triggers this > >> error. > > > > FWIW, I had a similar error (duplicate node) importing just the > > alberta.osm.bz2 CloudMade extract from December 25th; non-slim mode > > works fine. > > > > Same error here as well with the haiti extracts from geofabrik today. It seems there are two versions of a way in the haiti data: $ bzip2 -dc 2010-01-21-19-30.osm.bz2 | grep 28418855 I don't know why they are both there, it maybe a bug in the export process. One idea which was proposed for osm2pgsql was to record and use the version field when applying updates. I'm not sure if this would help in this case, we don't expect to find multiple versions when importing a single file. Jon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
2010/1/21 Jon Burgess :
> On Thu, 2010-01-21 at 11:13 -0500, Jason Beverage wrote:
>> I'm working with osm2pgsql (latest trunk version) to import OSM into a
>> PostgreSQL/PostGIS database. The issue I'm having is that if I use
>> the --slim option I'll occasionally get an error similar to:
>>
>> Going over pending relations
>> COPY_END for COPY osm_rels FROM STDIN;
>> failed: ERROR: duplicate key value violates unique constraint
>> "osm_rels_pkey"
>> CONTEXT: COPY osm_rels, line 1207: "284132 0 323
>> {42198453,20559277,20559273,20558263,20494565,40702583,19846737,19826462,20582455,20585..."
>>
>> If I don't use slim mode, it seems to always work. Right now I'm just
>> testing with state snapshots from http://downloads.cloudmade.com/, but
>> I'd like to be able to import the whole OSM planet database and I'd
>> really like to be able to use slim mode so I can do incremental
>> updates and reduce the memory requirements for importing.
>
> If I remember correctly this error occurs when you try to import two
> data sets which contain some overlapping data. In your case it appears
> that relation ID 284132 appears in both the data extracts. Only the slim
> mode keeps a copy of this node/way/relation data and triggers this
> error.
>
> It is not clear what the code should do in this case. It is possible
> that some of the data from the original object might have been removed
> when the extract was created because it fell outside the bounding box so
> neither of the extracts have the complete information. This issue is
> probably easier to understand if you consider what might happen to the
> way representing a road crossing the border between two countries.
>
> The only obvious solution is to either manually fix up the overlapping
> data or obtain an extract with all the data you want in a single file.
Or, merge the way JOSM merges freshly downloaded data with your
existing layer. This would assume the extracts are created with the
option that keeps the full ways /relations if only one node is inside
the area, like the map API call does.
Cheers,
Andrew
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
On Jan 21, 2010, at 5:51 AM, Mike Warren wrote: > Jon Burgess writes: > >> If I remember correctly this error occurs when you try to import two >> data sets which contain some overlapping data. In your case it >> appears >> that relation ID 284132 appears in both the data extracts. Only the >> slim >> mode keeps a copy of this node/way/relation data and triggers this >> error. > > FWIW, I had a similar error (duplicate node) importing just the > alberta.osm.bz2 CloudMade extract from December 25th; non-slim mode > works fine. > Same error here as well with the haiti extracts from geofabrik today. Dane ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
Jon Burgess writes: > If I remember correctly this error occurs when you try to import two > data sets which contain some overlapping data. In your case it appears > that relation ID 284132 appears in both the data extracts. Only the slim > mode keeps a copy of this node/way/relation data and triggers this > error. FWIW, I had a similar error (duplicate node) importing just the alberta.osm.bz2 CloudMade extract from December 25th; non-slim mode works fine. -- mike warren m...@mike-warren.com + http://www.mike-warren.com ATDT +1.403.764.6686 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Import error using osm2pgsql in slim mode
On Thu, 2010-01-21 at 11:13 -0500, Jason Beverage wrote:
> Hi all,
>
> I'm working with osm2pgsql (latest trunk version) to import OSM into a
> PostgreSQL/PostGIS database. The issue I'm having is that if I use
> the --slim option I'll occasionally get an error similar to:
>
> Going over pending relations
> COPY_END for COPY osm_rels FROM STDIN;
> failed: ERROR: duplicate key value violates unique constraint
> "osm_rels_pkey"
> CONTEXT: COPY osm_rels, line 1207: "2841320323
> {42198453,20559277,20559273,20558263,20494565,40702583,19846737,19826462,20582455,20585..."
>
> If I don't use slim mode, it seems to always work. Right now I'm just
> testing with state snapshots from http://downloads.cloudmade.com/, but
> I'd like to be able to import the whole OSM planet database and I'd
> really like to be able to use slim mode so I can do incremental
> updates and reduce the memory requirements for importing.
If I remember correctly this error occurs when you try to import two
data sets which contain some overlapping data. In your case it appears
that relation ID 284132 appears in both the data extracts. Only the slim
mode keeps a copy of this node/way/relation data and triggers this
error.
It is not clear what the code should do in this case. It is possible
that some of the data from the original object might have been removed
when the extract was created because it fell outside the bounding box so
neither of the extracts have the complete information. This issue is
probably easier to understand if you consider what might happen to the
way representing a road crossing the border between two countries.
The only obvious solution is to either manually fix up the overlapping
data or obtain an extract with all the data you want in a single file.
> Any help is greatly appreciated.
>
> Thanks!
>
> Jason
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
