[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 --- Comment #16 from Yuvi Panda --- I'm currently reading through the rsyslog manual, should be able to set something up soon :) -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846
Ori Livneh changed:
What|Removed |Added
CC||o...@wikimedia.org
--- Comment #15 from Ori Livneh ---
(In reply to comment #14)
> There is of course no python module for this
Bzzt! http://docs.python.org/2/library/logging.handlers.html#datagramhandler
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
import logging
import logging.handlers
class ToolLabsHandler(logging.handlers.DatagramHandler):
def makePickle(self, record):
return 's {0.name} 1 {0.msg}'.format(record).encode('utf-8',
errors='replace')
if __name__ == '__main__':
# If invoked as standalone, perform self-test.
log = logging.getLogger('my_tool_name')
log.setLevel(logging.INFO)
handler = ToolLabsHandler('localhost', 64386)
log.addHandler(handler)
log.info('Hello, world.')
--
You are receiving this mail because:
You are on the CC list for the bug.
___
Wikibugs-l mailing list
Wikibugs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846
--- Comment #14 from Peter Bena ---
I hit enter before I wrote it...
* Regarding your second question about python:
There is of course no python module for this, but if I knew python it would be
matter of 2 minutes to make some (which is not a case of rsyslog)
Just create a python function that does some equivalent of this
public static void Log(string text, string toolname)
{
// Change SendUDP / SendTCP to some python fuction which allows sending
over UDP or TCP
// SendUDP ( host, port, text)
SendUDP( "tools-syslog", 64386, "s " + toolname + " 1 " + text);
}
That is all
--
You are receiving this mail because:
You are on the CC list for the bug.
___
Wikibugs-l mailing list
Wikibugs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 --- Comment #13 from Peter Bena --- there /is/ rsyslog installed, it may not be configured to work well enough because IMHO single instance of rsyslog doesn't even support such a complex configuration that it would log the local syslog (the one in /var/log) to local fs and remote syslog from certain kind of services to network storage - I think we would need to spawn multiple rsyslog daemons for that. If you know how to do that - do that! Bots project is a testing environment for this kind of stuff, you can have root there if you don't have it already and there is equivalent of this on bots-syslog (identical server as tools-syslog) Once you finish configuration of rsyslog, then preferably either insert it to puppet class (syslog.pp in modules/toollabs/manifests) or just document it and I will do it. This bug is about central logging service - which we apparently have now. I believe that wast majority of tool operators will prefer something really simple over stuff that is complicated - this daemon allows you to define things like if the logs should be written in a txt / xml / html file how they should be formatted per line, define own formatting on demand or define own sections for logfiles on demand per tool. This isn't possible in rsyslog without changing configuration files. Rsyslog is a good tool for gathering system (server / kernel / daemon) logs for person with root on server, but it is lacking features bot operators might use. However I have nothing against having multiple options for people, if you like rsyslog, or whatever else, why not... Regard -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 --- Comment #12 from Tim Landscheidt --- (In reply to comment #11) > > > Rsyslogd doesn't seem to be enough for this, a separate service needs to > > > be > > > created > > Why? What's missing? > I forgot to mention that rsyslog was installed there as well, so feel free to > use that if you prefer. It is missing basically everything I would use as a > tool operator and it's incredibly complicated. But you can of course use it What's complicated about rsyslog? What's missing there that on the other hand a simple TCP sink has?! :-) > > > Is this running syslog or a similar service? I hope it isn't a custom C# > > > server > > > running there with no monitoring or logging of its own... > > > /me eyes petan suspiciously > > I hope it has libraries in C#, PHP, Perl, Python, C, Java, C++, Tcl, and > > whatever else people use on Tools. > It is listening on UDP and TCP ports, I don't know what libraries you need? > There is provided a very simple 1 line example in shell script which is > likely > easy to convert to any language I know. It is far simpler than rsyslog or > scribe, but nearly same powerful. Why do you think it is running with no > monitoring? As you know, there are lots of logging frameworks where you can just plug in modules for rsyslog or other remote daemons (or files or printers or ...). Your daemon implements a new protocol, so we need new plugins for them. For example, how do I configure Python's logging to log to your daemon? -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 --- Comment #11 from Peter Bena --- (In reply to comment #10) > (In reply to comment #7) > > Rsyslogd doesn't seem to be enough for this, a separate service needs to be > > created > > Why? What's missing? > I forgot to mention that rsyslog was installed there as well, so feel free to use that if you prefer. It is missing basically everything I would use as a tool operator and it's incredibly complicated. But you can of course use it > (In reply to comment #9) > > Is this running syslog or a similar service? I hope it isn't a custom C# > > server > > running there with no monitoring or logging of its own... > > > /me eyes petan suspiciously > > I hope it has libraries in C#, PHP, Perl, Python, C, Java, C++, Tcl, and > whatever else people use on Tools. It is listening on UDP and TCP ports, I don't know what libraries you need? There is provided a very simple 1 line example in shell script which is likely easy to convert to any language I know. It is far simpler than rsyslog or scribe, but nearly same powerful. Why do you think it is running with no monitoring? -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 --- Comment #10 from Tim Landscheidt --- (In reply to comment #7) > Rsyslogd doesn't seem to be enough for this, a separate service needs to be > created Why? What's missing? (In reply to comment #9) > Is this running syslog or a similar service? I hope it isn't a custom C# > server > running there with no monitoring or logging of its own... > /me eyes petan suspiciously I hope it has libraries in C#, PHP, Perl, Python, C, Java, C++, Tcl, and whatever else people use on Tools. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 --- Comment #9 from Yuvi Panda --- Is this running syslog or a similar service? I hope it isn't a custom C# server running there with no monitoring or logging of its own... /me eyes petan suspiciously -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 Peter Bena changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #8 from Peter Bena --- http://tools.wmflabs.org/logs/ o/ -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 --- Comment #7 from Peter Bena --- Rsyslogd doesn't seem to be enough for this, a separate service needs to be created -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 --- Comment #6 from Peter Bena --- yes production is using the tool written by tim starling, (udplogger) anyway I think instead of creating a tool which listen on port which changes, we should create a logging server dedicated to this task with permanent IP and port -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 --- Comment #5 from Yuvi Panda --- Also, does the production cluster use any such service? -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 --- Comment #4 from Yuvi Panda --- Yup. Also seeing how current tools do logging - if they do it at all - would be helpful I think. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 Peter Bena changed: What|Removed |Added Priority|Unprioritized |Low CC||benap...@gmail.com Severity|normal |enhancement --- Comment #3 from Peter Bena --- yes this would be nice, maybe we should first discuss which solution is best and then implement it -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 --- Comment #2 from Yuvi Panda --- Twitter also uses Facebook's Scribe server. There is also http://www.rsyslog.com/ which can be used with syslog for aggregation. I don't think we should write our own. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48846] Provide a central logging service for tools
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 --- Comment #1 from Yuvi Panda --- We don't need to even go as far as Scribe - even syslog can be used for this if necessary. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
