Re: [osmosis-dev] Patch: ReplicationLagReader

[Peter Körner]

Am 27.07.2010 10:38, schrieb Kai Krueger:

What a coincidence. I also wrote a munin plugin to monitor minutely-diff
replication lags on Sunday, not knowing anything about your effort.
It was no big effort - one evening to learn about osmosis' task 
structure and to write the code and another evening to do all the 
required cleanups and comments.


The patch is in svn right now.


It is simple shell script, that reads a local state.txt file of the
rendering db and compares it to current time, (assuming that the system
clock is correctly synchronized).

Yes I also used a shell-script:


But in order to depoly it with osmosis I saw the need to include this 
functionality directly into osmosis.



It would also be easy to include the state.txt on planet.osm.org too, to
see how much of the lag comes from the local osm2pgsql import and how
much of the lag comes from osmosis generating the diff files in the
first place (which partly seems to be the bigger factor at the moment)
That's a nice idea! I'll extend the task and the munin plugin to print 
out the server lag as well, so they could be added on top of each other 
in the munin graph. You're showing the lag in minutes, I was using 
seconds now. What do you think is better for visualizing?


Peter

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Patch: ReplicationLagReader

[Kai Krueger]

Peter Körner wrote:
>I also added a munin plugin that plots the replication lag into a >graph.
>On some systems the plugin needs some extra time (upper munin-node
>timeout) because of javas slow startup.

What a coincidence. I also wrote a munin plugin to monitor minutely-diff 
replication lags on Sunday, not knowing anything about your effort.


It is simple shell script, that reads a local state.txt file of the 
rendering db and compares it to current time, (assuming that the system 
clock is correctly synchronized).


Jon Burgess subsequently fixed up the code and deployed it on yevaud 
(the OSM tile server). 
http://munin.openstreetmap.org/openstreetmap/yevaud.openstreetmap/replication_delay2.html


It would also be easy to include the state.txt on planet.osm.org too, to 
see how much of the lag comes from the local osm2pgsql import and how 
much of the lag comes from osmosis generating the diff files in the 
first place (which partly seems to be the bigger factor at the moment)


Kai




___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Patch: ReplicationLagReader

[Brett Henderson]

Yep, go ahead and commit it.

Keep an eye on the continuous integration server to ensure the build
successfully completes:
http://dev.openstreetmap.de:23457/hudson/job/osmosis-SNAPSHOT-ant/

You can trigger a build manually once you've checked in, although it will
eventually start automatically (it polls SVN regularly).

Brett

On Tue, Jul 27, 2010 at 6:56 AM, Peter Körner wrote:

> Am 26.07.2010 21:15, schrieb Peter Körner:
>
>  I fixed all this and attached a new version of the patch.
>>
>
> Oops there was a bug in the munin script. Anyway - If it's ok for you I'd
> suggest to commit the changes myself (I already got a svn account) so that
> it's later clear who did what changes.
>
> Peter
>
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Patch: ReplicationLagReader

[Peter Körner]

Am 26.07.2010 21:15, schrieb Peter Körner:

I fixed all this and attached a new version of the patch.


Oops there was a bug in the munin script. Anyway - If it's ok for you 
I'd suggest to commit the changes myself (I already got a svn account) 
so that it's later clear who did what changes.


Peter

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Patch: ReplicationLagReader

[Peter Körner]

Am 26.07.2010 13:05, schrieb Brett Henderson:

I'm happy to apply it, just needs a few cleanups first.

* If you run "ant clean all" from the replication directory you'll
  notice it fails at the checkstyle step (You may need to run "ant
  publish" from the root directory first).  Some code formatting
  needs to be cleaned up to fix the checkstyle errors.
* Kinda picky, but while you're fixing the checkstyle errors, might
  be worth fixing some typos in the comments (locale->local,
  beeing->being, etc).
* One comment states "If local state isn't available we need to copy
  server state to be the initial local state then exit.".  That
  comment has been copied from my code but it's not true for your
  implementation.  That threw me off course when reading it for the
  first time.
* The munin config files can't live in the source tree.  They
  definitely won't be included in the final package if they're
  there, they'll only get embedded inside the
  osmosis-replication.jar file which isn't much help.  They'll need
  to live in the "package" project somewhere.  There's already a
  "script" directory which gets bundled with the release, would a
  "munin" directory in there be appropriate?
I fixed all this and attached a new version of the patch. Thank you for 
reviewing it. If you still see typos I'm sorry about it - i don't have 
eclipse and thus no automatic spell checking - and I'm no native 
speaker, so sorry for them.



* And finally, once it's been committed you'll need to document it
  on the 0.36 release wiki page
  http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.36
  including references to the munin files.

That shall not be the problem.

Peter
Index: package/script/munin/osm-replication-lag
===
--- package/script/munin/osm-replication-lag(Revision 0)
+++ package/script/munin/osm-replication-lag(Revision 0)
@@ -0,0 +1,35 @@
+#!/bin/sh
+# -*- sh -*-
+
+# load the munin plugin helper
+. $MUNIN_LIBDIR/plugins/plugin.sh
+
+# if no workingDirectory has been configures
+if [ ! $workingDirectory ]; then
+   # exit with an error
+   echo "no workingDirectory configured" >&2
+   exit 1
+fi
+
+# path to osmosis binary
+[ $osmosis ]  || osmosis="osmosis"
+
+# configuration section
+if [ "$1" = "config" ]; then
+
+   echo 'graph_title OSM PostGIS Database Replag'
+   echo 'graph_args --base 1000'
+   echo 'graph_vlabel seconds behind main database'
+   echo 'graph_category osm'
+   
+   echo 'lag.label replication lag'
+   echo 'lag.draw LINE'
+   
+   exit 0
+fi
+
+# invoke osmosis to calculate the replication lag
+lag=$($osmosis --read-replication-lag workingDirectory="$workingDirectory" 
2>/dev/null)
+echo "lag.value=$lag"
+
+exit 0
Index: package/script/munin/osm-replication.conf
===
--- package/script/munin/osm-replication.conf   (Revision 0)
+++ package/script/munin/osm-replication.conf   (Revision 0)
@@ -0,0 +1,15 @@
+[osm*]
+
+# the osmosis invocation may take some time
+timeout 60
+
+# the system user that has access to the working directory, if it'S different
+# from "munin"
+#user osm
+
+# path to the osmosis binary. if not set, osmosis is assumed to be in $PATH
+#env.osmosis /opt/osmosis/bin/osmosis
+
+# working directory of the osmosis replication.
+# this must be set to make the munin plugin work
+#env.workingDirectory /path/to/state.txt
Index: package/script/munin/README
===
--- package/script/munin/README (Revision 0)
+++ package/script/munin/README (Revision 0)
@@ -0,0 +1,7 @@
+to activate the munin plugins
+ - copy "osm-replication-lag" to "/usr/share/munin/plugins"
+ - make "/usr/share/munin/plugins/osm-replication-lag" executable
+ - symlink "/usr/share/munin/plugins/osm-replication-lag" to 
"/etc/munin/plugins"
+ - copy "osm-replication.conf" to "/etc/munin/plugin-conf.d"
+ - edit "/etc/munin/plugin-conf.d/osm-replication.conf" and set the 
workingDirectory
+ - restart the munin-node
Index: 
replication/src/org/openstreetmap/osmosis/replication/ReplicationPluginLoader.java
===
--- 
replication/src/org/openstreetmap/osmosis/replication/ReplicationPluginLoader.java
  (Revision 22443)
+++ 
replication/src/org/openstreetmap/osmosis/replication/ReplicationPluginLoader.java
  (Arbeitskopie)
@@ -12,6 +12,7 @@
 import 
org.openstreetmap.osmosis.replication.v0_6.ReplicationDownloaderInitializerFactory;
 import org.openstreetmap.osmosis.replication.v0_6.ReplicationFileMergerFactory;
 import 
org.openstreetmap.osmosis.replication.v0_6.ReplicationFileMergerInitializerFactory;
+import org.openstreetmap.osmosis.replication.v0_6.ReplicationLagReaderFactory;
 
 
 /**
@@ -43,6 +44,9 @@
  

Re: [osmosis-dev] Patch: ReplicationLagReader

[Brett Henderson]

On Mon, Jul 26, 2010 at 7:40 PM, Peter Körner wrote:

> Am 26.07.2010 10:01, schrieb Brett Henderson:
>
> > Hi Peter,
> >
> > You seem to have left my name as the author in some of the source
> > code.
> > Presumably that should be your name ;-)
> Yes, I think so ^^
>
>
> > Can you let me know what the purpose of this is?  Okay, I understand
> > that this is for "replication lag", but what is it comparing?  On
> > first read it appears to be comparing the latest server timestamp
> > from the local timestamp, but I can't see the local timestamp ever
> > being set.  Is that correct?
> It assumes a regular updated database using the rri-task. It compares the
> timstamp of he current state-file in the updating environment against the
> timestamp of the server's state-file and calculates the replication lag,
> that's the time the local database is behind the main database.
>

Ah, okay.  Now I get it.

I'm happy to apply it, just needs a few cleanups first.

   - If you run "ant clean all" from the replication directory you'll notice
   it fails at the checkstyle step (You may need to run "ant publish" from the
   root directory first).  Some code formatting needs to be cleaned up to fix
   the checkstyle errors.
   - Kinda picky, but while you're fixing the checkstyle errors, might be
   worth fixing some typos in the comments (locale->local, beeing->being, etc).
   - One comment states "If local state isn't available we need to copy
   server state to be the initial local state then exit.".  That comment has
   been copied from my code but it's not true for your implementation.  That
   threw me off course when reading it for the first time.
   - The munin config files can't live in the source tree.  They definitely
   won't be included in the final package if they're there, they'll only get
   embedded inside the osmosis-replication.jar file which isn't much help.
   They'll need to live in the "package" project somewhere.  There's already a
   "script" directory which gets bundled with the release, would a "munin"
   directory in there be appropriate?
   - And finally, once it's been committed you'll need to document it on the
   0.36 release wiki page
   http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.36 including
   references to the munin files.


Cheers,
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Patch: ReplicationLagReader

[Peter Körner]

Am 26.07.2010 10:01, schrieb Brett Henderson:
> Hi Peter,
>
> You seem to have left my name as the author in some of the source
> code.
> Presumably that should be your name ;-)
Yes, I think so ^^

> Can you let me know what the purpose of this is?  Okay, I understand
> that this is for "replication lag", but what is it comparing?  On
> first read it appears to be comparing the latest server timestamp
> from the local timestamp, but I can't see the local timestamp ever
> being set.  Is that correct?
It assumes a regular updated database using the rri-task. It compares 
the timstamp of he current state-file in the updating environment 
against the timestamp of the server's state-file and calculates the 
replication lag, that's the time the local database is behind the main 
database.


Peter

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] Patch: ReplicationLagReader

[Brett Henderson]

Hi Peter,

You seem to have left my name as the author in some of the source code.
Presumably that should be your name ;-)

Can you let me know what the purpose of this is?  Okay, I understand that
this is for "replication lag", but what is it comparing?  On first read it
appears to be comparing the latest server timestamp from the local
timestamp, but I can't see the local timestamp ever being set.  Is that
correct?

Brett

On Sun, Jul 25, 2010 at 10:34 AM, Peter Körner wrote:

> Hi
>
> To learn a bit about the internal structure of osmosis I developed a new
> task in the "replication" section. It's basicly the same as the little
> replag script [1] I use on my servers.
>
> I also added a munin plugin that plots the replication lag into a graph. On
> some systems the plugin needs some extra time (upper munin-node timeout)
> because of javas slow startup.
>
> I don't know if this task should be comitted to svn or if it's too simple
> (its task can also be fulfilled by that simple shell-script). I also don't
> know where to place the muinin plugin in the source tree and how to make it
> installed into the right place when using debuild to make a debian package.
>
> Please find the patch attached.
>
> Peter
>
> [1] 
>
> ___
> osmosis-dev mailing list
> osmosis-dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/osmosis-dev
>
>
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] Patch: ReplicationLagReader

[Peter Körner]

Hi

To learn a bit about the internal structure of osmosis I developed a new 
task in the "replication" section. It's basicly the same as the little 
replag script [1] I use on my servers.


I also added a munin plugin that plots the replication lag into a graph. 
On some systems the plugin needs some extra time (upper munin-node 
timeout) because of javas slow startup.


I don't know if this task should be comitted to svn or if it's too 
simple (its task can also be fulfilled by that simple shell-script). I 
also don't know where to place the muinin plugin in the source tree and 
how to make it installed into the right place when using debuild to make 
a debian package.


Please find the patch attached.

Peter

[1] 
Index: 
replication/src/org/openstreetmap/osmosis/replication/ReplicationPluginLoader.java
===
--- 
replication/src/org/openstreetmap/osmosis/replication/ReplicationPluginLoader.java
  (Revision 22433)
+++ 
replication/src/org/openstreetmap/osmosis/replication/ReplicationPluginLoader.java
  (Arbeitskopie)
@@ -12,8 +12,8 @@
 import 
org.openstreetmap.osmosis.replication.v0_6.ReplicationDownloaderInitializerFactory;
 import org.openstreetmap.osmosis.replication.v0_6.ReplicationFileMergerFactory;
 import 
org.openstreetmap.osmosis.replication.v0_6.ReplicationFileMergerInitializerFactory;
+import org.openstreetmap.osmosis.replication.v0_6.ReplicationLagReaderFactory;
 
-
 /**
  * The plugin loader for the replication tasks.
  * 
@@ -43,6 +43,9 @@
factoryMap.put("merge-replication-files-init", new 
ReplicationFileMergerInitializerFactory());
factoryMap.put("mrfi", new 
ReplicationFileMergerInitializerFactory());

+   factoryMap.put("read-replication-lag", new 
ReplicationLagReaderFactory());
+   factoryMap.put("rrl", new ReplicationLagReaderFactory());
+   
factoryMap.put("read-change-interval-0.6", new 
IntervalDownloaderFactory());
factoryMap.put("read-change-interval-init-0.6", new 
IntervalDownloaderInitializerFactory());
factoryMap.put("read-replication-interval-0.6", new 
ReplicationDownloaderFactory());
Index: replication/src/org/openstreetmap/osmosis/replication/munin/sample-config
===
--- replication/src/org/openstreetmap/osmosis/replication/munin/sample-config   
(Revision 0)
+++ replication/src/org/openstreetmap/osmosis/replication/munin/sample-config   
(Revision 0)
@@ -0,0 +1,3 @@
+[osm*]
+env.osmosis /opt/osmosis/bin/osmosis
+env.workingDirectory /home/osm/data/diffs/
Index: 
replication/src/org/openstreetmap/osmosis/replication/munin/osm-replication-lag
===
--- 
replication/src/org/openstreetmap/osmosis/replication/munin/osm-replication-lag 
(Revision 0)
+++ 
replication/src/org/openstreetmap/osmosis/replication/munin/osm-replication-lag 
(Revision 0)
@@ -0,0 +1,36 @@
+#!/bin/sh
+# -*- sh -*-
+
+. $MUNIN_LIBDIR/plugins/plugin.sh
+
+if [ ! $workingDirectory ]; then
+   echo "no workingDirectory configured" >&2
+   exit 1
+fi
+
+[ $osmosis ]  || osmosis="/opt/osmosis/bin/osmosis"
+
+
+
+if [ "$1" = "autoconf" ]; then
+   echo yes 
+   exit 0
+fi
+
+if [ "$1" = "config" ]; then
+
+   echo 'graph_title OSM PostGIS Database Replag'
+   echo 'graph_args --base 1000'
+   echo 'graph_vlabel seconds behind main database'
+   echo 'graph_category osm'
+   
+   echo 'lag.label replication lag'
+   echo 'lag.draw LINE'
+   
+   exit 0
+fi
+
+lag=$($osmosis --read-replication-lag workingDirectory="$workingDirectory" 
2>/dev/null)
+echo "lag=$lag"
+
+exit 0

Eigenschaftsänderungen: 
replication/src/org/openstreetmap/osmosis/replication/munin/osm-replication-lag
___
Hinzugefügt: svn:executable
   + *

Index: 
replication/src/org/openstreetmap/osmosis/replication/v0_6/ReplicationLagReaderFactory.java
===
--- 
replication/src/org/openstreetmap/osmosis/replication/v0_6/ReplicationLagReaderFactory.java
 (Revision 0)
+++ 
replication/src/org/openstreetmap/osmosis/replication/v0_6/ReplicationLagReaderFactory.java
 (Revision 0)
@@ -0,0 +1,55 @@
+// This software is released into the Public Domain.  See copying.txt for 
details.
+package org.openstreetmap.osmosis.replication.v0_6;
+
+import java.io.File;
+
+import org.openstreetmap.osmosis.core.pipeline.common.RunnableTaskManager;
+import org.openstreetmap.osmosis.core.pipeline.common.TaskConfiguration;
+import org.openstreetmap.osmosis.core.pipeline.common.TaskManager;
+import org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactory;
+
+
+/**
+ * The task manager factory for a repl