Re: [Talk-us] Cloudmade california.osm: dups or osmosis bug?
Hi Mike,
good catch, I forgot to add the ...Factory.java file. Hopefully I
haven't forgotten any more files.
Cheers
On 22 May 2010 04:08, Mike N. wrote:
> Hi andrzej,
>
> I tried to apply the patch, but I get an undefined symbol when trying to
> build 'FlattenFilterFactory'. Am I missing part of the patch or is there
> another setting to create the Factory?
>
> Thanks,
>
> Mike Nice
>
> --
> From: "andrzej zaborowski"
> Sent: Friday, May 21, 2010 2:40 PM
> To: "David Carmean"
> Cc: ; "OSM US Talk"
>
> Subject: Re: [Talk-us] Cloudmade california.osm: dups or osmosis bug?
>
>> Hi,
>>
>> On 21 May 2010 01:54, David Carmean wrote:
>>>
>>> Is it the dataset or osmosis that is giving me a single 2-node duplicate
>>> for each postGIS table created by osmosis?
>>
>> Not sure if it's related, but I noticed that many of the CloudMade
>> extracts can't be processed using osmosis (even though they're
>> generated with osmosis), except for the operations that don't try to
>> validate the order of elements. It seems they often contain
>> consecutive elements with the same id, but possibly different version.
>> I have a little patch (attached) to add a new filter (--ff) that
>> "flattens" the file. It does the same operation on sorted entity
>> streams as --simplify-change does on change streams, unfortunately
>> "--simplify" is taken by a different operation now. I decided it was
>> faster to modify osmosis than to download the planet and make my own
>> un-broken extracts.
>>
>> Cheers
>> (I'm not subscribed to osmosis-dev)
>>
>
>
>
>> ___
>> Talk-us mailing list
>> Talk-us@openstreetmap.org
>> http://lists.openstreetmap.org/listinfo/talk-us
>>
>
Index: src/org/openstreetmap/osmosis/core/filter/v0_6/FlattenFilter.java
===
--- src/org/openstreetmap/osmosis/core/filter/v0_6/FlattenFilter.java (revision 0)
+++ src/org/openstreetmap/osmosis/core/filter/v0_6/FlattenFilter.java (revision 0)
@@ -0,0 +1,91 @@
+/* This software is released into the Public Domain.
+ * See copying.txt for details. */
+package org.openstreetmap.osmosis.core.filter.v0_6;
+
+import org.openstreetmap.osmosis.core.container.v0_6.BoundContainer;
+import org.openstreetmap.osmosis.core.container.v0_6.EntityContainer;
+import org.openstreetmap.osmosis.core.container.v0_6.EntityProcessor;
+import org.openstreetmap.osmosis.core.domain.v0_6.Entity;
+import org.openstreetmap.osmosis.core.task.v0_6.Sink;
+import org.openstreetmap.osmosis.core.task.v0_6.SinkSource;
+
+
+/**
+ * Flatten / simplify a sorted entity stream.
+ * (similar to --simplify-change)
+ */
+public class FlattenFilter implements SinkSource {
+ private Sink sink;
+ private EntityContainer previous_container;
+
+ /**
+ * Creates a new instance.
+ */
+ public FlattenFilter() {
+ }
+
+ /**
+ * Process a node, way or relation.
+ *
+ * @param current_container
+ *The entity container to be processed.
+ */
+ public void process(EntityContainer current_container) {
+ if (previous_container == null) {
+ previous_container = current_container;
+ return;
+ }
+
+ Entity current = current_container.getEntity();
+ Entity previous = previous_container.getEntity();
+
+ if (current.getId() != previous.getId() ||
+current.getClass() != previous.getClass()) {
+ sink.process(previous_container);
+ previous_container = current_container;
+ return;
+ }
+
+ if (current.getVersion() > previous.getVersion())
+ previous_container = current_container;
+ }
+
+ /**
+ * Process the bound.
+ *
+ * @param boundContainer
+ *The bound to be processed.
+ */
+ public void process(BoundContainer boundContainer) {
+ /* By default, pass it on unchanged */
+ sink.process(boundContainer);
+ }
+
+ /**
+ * {...@inheritdoc}
+ */
+ public void complete() {
+ /*
+ * If we've stored entities temporarily, we now need to
+ * forward the stored ones to the output.
+ */
+ if (previous_container != null)
+ sink.process(previous_container);
+
+ sink.complete();
+ }
+
+ /**
+ * {...@inheritdoc}
+ */
+ public void release() {
+ sink.release();
+ }
+
+ /**
+ * {...@inheritdoc}
+ */
+ public void setSink(Sink sink) {
+ this.sink = sink;
+ }
+}
Index: src/org/openstreetmap/osmosis/core/filter/v0_6/FlattenFilterFactory.java
===
--- src/org/openstreetmap/osmosis/core/filter/v0_6/FlattenFilterFactory.java (revision 0)
+++ src/org/openstreetmap/osmosis/core/filter/v0_6/FlattenFilterFac
Re: [Talk-us] Cloudmade california.osm: dups or osmosis bug?
Thanks for the patch - I'm also running into random dupes on CloudMade extracts for my state. > Not sure if it's related, but I noticed that many of the CloudMade > extracts can't be processed using osmosis (even though they're > generated with osmosis), except for the operations that don't try to > validate the order of elements. It seems they often contain > consecutive elements with the same id, but possibly different version. > I have a little patch (attached) to add a new filter (--ff) that > "flattens" the file. It does the same operation on sorted entity > streams as --simplify-change does on change streams, unfortunately > "--simplify" is taken by a different operation now. I decided it was > faster to modify osmosis than to download the planet and make my own > un-broken extracts. ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
Re: [Talk-us] Cloudmade california.osm: dups or osmosis bug?
Hi,
On 21 May 2010 01:54, David Carmean wrote:
> Is it the dataset or osmosis that is giving me a single 2-node duplicate
> for each postGIS table created by osmosis?
Not sure if it's related, but I noticed that many of the CloudMade
extracts can't be processed using osmosis (even though they're
generated with osmosis), except for the operations that don't try to
validate the order of elements. It seems they often contain
consecutive elements with the same id, but possibly different version.
I have a little patch (attached) to add a new filter (--ff) that
"flattens" the file. It does the same operation on sorted entity
streams as --simplify-change does on change streams, unfortunately
"--simplify" is taken by a different operation now. I decided it was
faster to modify osmosis than to download the planet and make my own
un-broken extracts.
Cheers
(I'm not subscribed to osmosis-dev)
Index: src/org/openstreetmap/osmosis/core/filter/v0_6/FlattenFilter.java
===
--- src/org/openstreetmap/osmosis/core/filter/v0_6/FlattenFilter.java (revision 0)
+++ src/org/openstreetmap/osmosis/core/filter/v0_6/FlattenFilter.java (revision 0)
@@ -0,0 +1,91 @@
+/* This software is released into the Public Domain.
+ * See copying.txt for details. */
+package org.openstreetmap.osmosis.core.filter.v0_6;
+
+import org.openstreetmap.osmosis.core.container.v0_6.BoundContainer;
+import org.openstreetmap.osmosis.core.container.v0_6.EntityContainer;
+import org.openstreetmap.osmosis.core.container.v0_6.EntityProcessor;
+import org.openstreetmap.osmosis.core.domain.v0_6.Entity;
+import org.openstreetmap.osmosis.core.task.v0_6.Sink;
+import org.openstreetmap.osmosis.core.task.v0_6.SinkSource;
+
+
+/**
+ * Flatten / simplify a sorted entity stream.
+ * (similar to --simplify-change)
+ */
+public class FlattenFilter implements SinkSource {
+ private Sink sink;
+ private EntityContainer previous_container;
+
+ /**
+ * Creates a new instance.
+ */
+ public FlattenFilter() {
+ }
+
+ /**
+ * Process a node, way or relation.
+ *
+ * @param current_container
+ *The entity container to be processed.
+ */
+ public void process(EntityContainer current_container) {
+ if (previous_container == null) {
+ previous_container = current_container;
+ return;
+ }
+
+ Entity current = current_container.getEntity();
+ Entity previous = previous_container.getEntity();
+
+ if (current.getId() != previous.getId() ||
+current.getClass() != previous.getClass()) {
+ sink.process(previous_container);
+ previous_container = current_container;
+ return;
+ }
+
+ if (current.getVersion() > previous.getVersion())
+ previous_container = current_container;
+ }
+
+ /**
+ * Process the bound.
+ *
+ * @param boundContainer
+ *The bound to be processed.
+ */
+ public void process(BoundContainer boundContainer) {
+ /* By default, pass it on unchanged */
+ sink.process(boundContainer);
+ }
+
+ /**
+ * {...@inheritdoc}
+ */
+ public void complete() {
+ /*
+ * If we've stored entities temporarily, we now need to
+ * forward the stored ones to the output.
+ */
+ if (previous_container != null)
+ sink.process(previous_container);
+
+ sink.complete();
+ }
+
+ /**
+ * {...@inheritdoc}
+ */
+ public void release() {
+ sink.release();
+ }
+
+ /**
+ * {...@inheritdoc}
+ */
+ public void setSink(Sink sink) {
+ this.sink = sink;
+ }
+}
Index: src/org/openstreetmap/osmosis/core/TaskRegistrar.java
===
--- src/org/openstreetmap/osmosis/core/TaskRegistrar.java (revision 21311)
+++ src/org/openstreetmap/osmosis/core/TaskRegistrar.java (working copy)
@@ -45,6 +45,7 @@
import org.openstreetmap.osmosis.core.filter.v0_6.UsedNodeFilterFactory;
import org.openstreetmap.osmosis.core.filter.v0_6.WayKeyFilterFactory;
import org.openstreetmap.osmosis.core.filter.v0_6.WayKeyValueFilterFactory;
+import org.openstreetmap.osmosis.core.filter.v0_6.FlattenFilterFactory;
import org.openstreetmap.osmosis.core.merge.v0_6.ChangeAppenderFactory;
import org.openstreetmap.osmosis.core.merge.v0_6.ChangeMergerFactory;
import org.openstreetmap.osmosis.core.merge.v0_6.ChangeSimplifierFactory;
@@ -153,8 +154,10 @@
factoryRegister.register("read-xml", new XmlReaderFactory());
factoryRegister.register("fast-read-xml", new FastXmlReaderFactory());
factoryRegister.register("rx", new XmlReaderFactory());
-factoryRegister.register("read-xml-change", new XmlChangeReaderFactory());
-factoryRegister.register("upload-xml-change", new XmlChangeUploaderFactory());
+ factoryRegister.register("flatten", new FlattenFilterFactory());
+ factoryRegister.register("ff", new FlattenFilterFactory());
+ factoryRegister.register("read-xml-change", new XmlChangeReaderFactory());
+ factoryRegister.register("upload-xml-change", new XmlChangeUploaderFactory());
factoryRegister.register("rxc", new Xml
[Talk-us] Cloudmade california.osm: dups or osmosis bug?
Is it the dataset or osmosis that is giving me a single 2-node duplicate for each postGIS table created by osmosis? Or PEBCAK? ___ Talk-us mailing list Talk-us@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-us
