Re: logging via SocketHandler and TCPserver
Rob Wolfe wrote: Larry Bates <[EMAIL PROTECTED]> writes: Can multiple applications send SocketHandler logging records to the same socket server on the same port simultaneously? Of course they can. Server can accept requests from many clients. You have used `SocketServer.ThreadingTCPServer`. That server for example handles every request in a different thread. If so, then I understand your answer completely and will go in that direction. I guess I was trying to not use up bandwidth/CPU cycles on the applications that weren't being actively monitored by just not having the socket server connected to them. I think you may be a 'little to close' to the (excellent) application you have written to understand the steep learning curve that I see. You know the saying, "Brain surgery is easy to a brain surgeon". I should point out that I'm no newbie. I've used PIL, ReportLab, BeautifulSoup, Mechanize, Win32 extensions, ElementTree and a whole host of other modules with less difficulty. Please don't take this as anything more than an observation on my part. From what I see, you have written (and generously donated) an extremely powerful library and it is greatly appreciated. It is most likely just me. Well, I know what you mean. I had troubles to understand that library either. Logging docs are quite good as a reference, but there is lack of a good introductory tutorial. Now when I know `logging` I can't live without it, but the start wasn't easy. I shouldn't say it here, but I found helpful docs to the similar library in Java. ;) http://logging.apache.org/log4j/1.2/manual.html There are many differences (e.g. Appenders vs Handlers), but the rule is the same. As far as the book is concerned, I guess I'd purchase the only copy ;-). "The Art of Logging" - four volumes. ;-) Regards, Rob Five volumes if you add in "Logging within Frameworks (Django, TurboGears, and Pylons" ;-). Now on to UnitTesting (I use a home-grown method), Twisted, and other things I can't seem to find time to get my brain around... -Larry -- http://mail.python.org/mailman/listinfo/python-list
Re: logging via SocketHandler and TCPserver
Larry Bates <[EMAIL PROTECTED]> writes: > Can multiple applications send SocketHandler logging records to the > same socket server on the same port simultaneously? Of course they can. Server can accept requests from many clients. You have used `SocketServer.ThreadingTCPServer`. That server for example handles every request in a different thread. > If so, then I understand your answer completely and will go > in that direction. I guess I was trying to not use up > bandwidth/CPU cycles on the applications that weren't being actively > monitored by just not having the socket server connected to them. > > I think you may be a 'little to close' to the (excellent) application > you have written to understand the steep learning curve that I see. > You know the saying, "Brain surgery is easy to a brain surgeon". I > should point out that I'm no newbie. I've used PIL, ReportLab, > BeautifulSoup, Mechanize, Win32 extensions, ElementTree and a whole > host of other modules with less difficulty. Please don't take this as > anything more than an observation on my part. From what I see, you > have written (and generously donated) an extremely powerful library > and it is greatly appreciated. It is most likely just me. Well, I know what you mean. I had troubles to understand that library either. Logging docs are quite good as a reference, but there is lack of a good introductory tutorial. Now when I know `logging` I can't live without it, but the start wasn't easy. I shouldn't say it here, but I found helpful docs to the similar library in Java. ;) http://logging.apache.org/log4j/1.2/manual.html There are many differences (e.g. Appenders vs Handlers), but the rule is the same. > As far as the book is concerned, I guess I'd purchase the only copy ;-). "The Art of Logging" - four volumes. ;-) Regards, Rob -- http://mail.python.org/mailman/listinfo/python-list
Re: logging via SocketHandler and TCPserver
Vinay Sajip wrote: On Jul 14, 11:16 pm, Larry Bates <[EMAIL PROTECTED]> wrote: Vinay, Thanks for your detailed explanation, but IMHO your suggested solution is almost the opposite (right idea wrong direction) of what I'm looking for. Hypothetical setup: application1 - SocketHandlerloggingturned on application2 - SocketHandlerloggingturned on application3 - SocketHandlerloggingturned on . . . applicationN monitoring app - ThreadingTCPServer target that Allows user to connect to ANY running application and to view the real-time log messages. After monitoring, it should be able to disconnect and connect to another application... The clients (application1..N) never stop sending and don't ever send anything short to disconnect themselves from the monitoring application. The disconnect/ reconnect is done at the monitoring app end based on input from the user. I think each one would beloggingto a different port, but I never really quite got that far. Perhaps there is some other way that I'm missing. It seems like this is something that might be able to be "generalized" into a robust monitoring application for an arbitrary number of asynchronously running applications. Each application's real-time log might be reached by clicking on a tab, menu, etc. Here's how I see it: the socket server listens, and receives logging events from numerous applications in real time. If the server is configured to monitor app1, then it quietly discards (or doesn't show in the UI) all events from other apps - it only collects/shows events from app1. When you click on a tab/menu/whatever to switch to monitoring app2, then this information is used to tell the server to discard (or not show) events from all apps except app2. You would of course need to ensure the communication between UI thread and server thread were done in a thread-safe manner. As to the difficulty, I might just have a mental block but two modules that I've had a real hard time getting my mind wrapped around is yourLoggingand Twisted. They both have a multitude of options/ways to be used and not very many working examples to work from. Might be a chance to make some money on a book. If it was full of examples, I'd purchase it. But there are numerous examples in the logging docs - the script you quoted as having put together after several hours of Googling is pretty much the same as the (working) version in the logging docs! If you have reviewed the logging docs and find them lacking examples, please provide more detail about the kind of examples you think are missing. And I can't speak for Twisted, but it does a lot more than logging - and I don't think there's enough complexity in Python logging to warrant a paid-for book. (No doubt people will tell me if they disagree!) Best regards, Vinay Sajip Can multiple applications send SocketHandler logging records to the same socket server on the same port simultaneously? If so, then I understand your answer completely and will go in that direction. I guess I was trying to not use up bandwidth/CPU cycles on the applications that weren't being actively monitored by just not having the socket server connected to them. I think you may be a 'little to close' to the (excellent) application you have written to understand the steep learning curve that I see. You know the saying, "Brain surgery is easy to a brain surgeon". I should point out that I'm no newbie. I've used PIL, ReportLab, BeautifulSoup, Mechanize, Win32 extensions, ElementTree and a whole host of other modules with less difficulty. Please don't take this as anything more than an observation on my part. From what I see, you have written (and generously donated) an extremely powerful library and it is greatly appreciated. It is most likely just me. As far as the book is concerned, I guess I'd purchase the only copy ;-). I do appreciate your help very much. Regards, Larry -- http://mail.python.org/mailman/listinfo/python-list
Re: logging via SocketHandler and TCPserver
On Jul 14, 11:16 pm, Larry Bates <[EMAIL PROTECTED]> wrote: > Vinay, > > Thanks for your detailed explanation, but IMHO your suggested solution is > almost > the opposite (right idea wrong direction) of what I'm looking for. > Hypothetical > setup: > > application1 - SocketHandlerloggingturned on > application2 - SocketHandlerloggingturned on > application3 - SocketHandlerloggingturned on > . > . > . > applicationN > > monitoring app - ThreadingTCPServer target that Allows user to connect to ANY > running application and to view the real-time log messages. After monitoring, > it should be able to disconnect and connect to another application... The > clients (application1..N) never stop sending and don't ever send anything > short > to disconnect themselves from the monitoring application. The disconnect/ > reconnect is done at the monitoring app end based on input from the user. I > think each one would beloggingto a different port, but I never really quite > got that far. Perhaps there is some other way that I'm missing. It seems > like > this is something that might be able to be "generalized" into a robust > monitoring application for an arbitrary number of asynchronously running > applications. > > Each application's real-time log might be reached by clicking on a tab, menu, > etc. Here's how I see it: the socket server listens, and receives logging events from numerous applications in real time. If the server is configured to monitor app1, then it quietly discards (or doesn't show in the UI) all events from other apps - it only collects/shows events from app1. When you click on a tab/menu/whatever to switch to monitoring app2, then this information is used to tell the server to discard (or not show) events from all apps except app2. You would of course need to ensure the communication between UI thread and server thread were done in a thread-safe manner. > As to the difficulty, I might just have a mental block but two modules that > I've > had a real hard time getting my mind wrapped around is yourLoggingand Twisted. > They both have a multitude of options/ways to be used and not very many > working > examples to work from. Might be a chance to make some money on a book. If it > was full of examples, I'd purchase it. > But there are numerous examples in the logging docs - the script you quoted as having put together after several hours of Googling is pretty much the same as the (working) version in the logging docs! If you have reviewed the logging docs and find them lacking examples, please provide more detail about the kind of examples you think are missing. And I can't speak for Twisted, but it does a lot more than logging - and I don't think there's enough complexity in Python logging to warrant a paid-for book. (No doubt people will tell me if they disagree!) Best regards, Vinay Sajip -- http://mail.python.org/mailman/listinfo/python-list
Re: logging via SocketHandler and TCPserver
Vinay Sajip wrote:
On Jul 13, 9:25 pm, Larry Bates <[EMAIL PROTECTED]> wrote:
Every time I look at theloggingmodule (up until now) I've given up and
continue to use my home-grown logger that I've been using for years. I'm not
giving up this time ;-)
I find that I REALLY need to be able to monitor LOTS of running
programs/processes and thought it would be nice to have them use
SocketHandlerloggingand then I would write TCPServer to accept the log messages
for
real-time monitoring. I Googled (is that now a verb?) for several hours and
came up with some code that I've turned in to something that works, but I can't
figure out how to disconnect the server once it is connected The goal is to be
able to start TCPServer, monitor the log messages sent via SocketHandler logger,
disconnect, and move to the next application. Eventually I would like to wrap a
GUI around all of this for monitoring a complex web application.
Everything works, it just appears that I get into the while loop in
LogRecordStreamHandler.handle and it never breaks out (until I kill the client).
I can't seem to do anything with the LogRecordSocketReceiver.abort attribute to
make it quit.
I'm sure it is something simple (stupid?), but I've spent about 4 hours and I'm
not getting anywhere.
Thanks in advance for any assistance.
Regards,
Larry
Below is my code:
import sys
import time
importlogging
if sys.argv[1] == 'client':
importlogging.config
logging.config.fileConfig("logging.conf")
#create logger
logger =logging.getLogger("VESconsole")
while 1:
logger.debug("debug message")
logger.info("info message")
logger.warn("warn message")
logger.error("error message")
logger.critical("critical message")
time.sleep(2)
elif sys.argv[1] == 'server':
import cPickle
importlogging.handlers
import SocketServer
import struct
import signal
class LogRecordStreamHandler(SocketServer.StreamRequestHandler):
"""Handler for a streamingloggingrequest.
This basically logs the record using whateverloggingpolicy is
configured locally.
"""
def handle(self):
"""
Handle multiple requests - each expected to be a 4-byte length,
followed by the LogRecord in pickle format. Logs the record
according to whatever policy is configured locally.
"""
while 1:
chunk = self.connection.recv(4)
if len(chunk) < 4:
break
slen = struct.unpack(">L", chunk)[0]
chunk = self.connection.recv(slen)
while len(chunk) < slen:
chunk = chunk + self.connection.recv(slen - len(chunk))
obj = self.unPickle(chunk)
record =logging.makeLogRecord(obj)
self.handleLogRecord(record)
def unPickle(self, data):
return cPickle.loads(data)
def handleLogRecord(self, record):
t = time.strftime('%a, %d %b %y %H:%M:%S',
time.localtime(record.created))
print "%s %s" % (t, record.getMessage())
class LogRecordSocketReceiver(SocketServer.ThreadingTCPServer):
"""simple TCP socket-basedloggingreceiver suitable for testing.
"""
allow_reuse_address = 1
def __init__(self, host='localhost',
port=logging.handlers.DEFAULT_TCP_LOGGING_PORT,
handler=LogRecordStreamHandler):
SocketServer.ThreadingTCPServer.__init__(self,
(host, port),
handler)
self.abort = 0
self.timeout = 1
self.logname = None
def serve_until_stopped(self):
import select
abort = 0
while not abort:
rd, wr, ex = select.select([self.socket.fileno()],
[], [],
self.timeout)
if rd:
self.handle_request()
abort = self.abort
print "serve_until_stopped exiting"
#
# Start ThreadingTCPServer instance to accept SocketHandler log
# messages from client.
#
tcpserver = LogRecordSocketReceiver()
print "Starting ThreadingTCPServer..."
tcpserver.serve_until_stopped()
'''
#-logging.conf-
[loggers]
keys=root
[handlers]
keys=socketHandler
[formatters]
keys=simpleFormatter
[logger_root]
level=DEBUG
handlers=socketHandler
[handler_socketHandler]
class=handlers.SocketHandler
level=DEBUG
args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT)
host=localhost
port=DEFAULT_TCP_LOGGING_PORT
[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(mess
Re: logging via SocketHandler and TCPserver
On Jul 13, 9:25 pm, Larry Bates <[EMAIL PROTECTED]> wrote:
> Every time I look at theloggingmodule (up until now) I've given up and
> continue to use my home-grown logger that I've been using for years. I'm not
> giving up this time ;-)
>
> I find that I REALLY need to be able to monitor LOTS of running
> programs/processes and thought it would be nice to have them use
> SocketHandlerloggingand then I would write TCPServer to accept the log
> messages for
> real-time monitoring. I Googled (is that now a verb?) for several hours and
> came up with some code that I've turned in to something that works, but I
> can't
> figure out how to disconnect the server once it is connected The goal is to
> be
> able to start TCPServer, monitor the log messages sent via SocketHandler
> logger,
> disconnect, and move to the next application. Eventually I would like to
> wrap a
> GUI around all of this for monitoring a complex web application.
>
> Everything works, it just appears that I get into the while loop in
> LogRecordStreamHandler.handle and it never breaks out (until I kill the
> client).
> I can't seem to do anything with the LogRecordSocketReceiver.abort attribute
> to
> make it quit.
>
> I'm sure it is something simple (stupid?), but I've spent about 4 hours and
> I'm
> not getting anywhere.
>
> Thanks in advance for any assistance.
>
> Regards,
> Larry
>
> Below is my code:
>
> import sys
> import time
> importlogging
>
> if sys.argv[1] == 'client':
> importlogging.config
>
> logging.config.fileConfig("logging.conf")
>
> #create logger
> logger =logging.getLogger("VESconsole")
>
> while 1:
> logger.debug("debug message")
> logger.info("info message")
> logger.warn("warn message")
> logger.error("error message")
> logger.critical("critical message")
> time.sleep(2)
>
> elif sys.argv[1] == 'server':
> import cPickle
> importlogging.handlers
> import SocketServer
> import struct
> import signal
>
> class LogRecordStreamHandler(SocketServer.StreamRequestHandler):
> """Handler for a streamingloggingrequest.
>
> This basically logs the record using whateverloggingpolicy is
> configured locally.
> """
>
> def handle(self):
> """
> Handle multiple requests - each expected to be a 4-byte length,
> followed by the LogRecord in pickle format. Logs the record
> according to whatever policy is configured locally.
> """
> while 1:
> chunk = self.connection.recv(4)
> if len(chunk) < 4:
> break
>
> slen = struct.unpack(">L", chunk)[0]
> chunk = self.connection.recv(slen)
> while len(chunk) < slen:
> chunk = chunk + self.connection.recv(slen - len(chunk))
>
> obj = self.unPickle(chunk)
> record =logging.makeLogRecord(obj)
> self.handleLogRecord(record)
>
> def unPickle(self, data):
> return cPickle.loads(data)
>
> def handleLogRecord(self, record):
> t = time.strftime('%a, %d %b %y %H:%M:%S',
>time.localtime(record.created))
>
> print "%s %s" % (t, record.getMessage())
>
> class LogRecordSocketReceiver(SocketServer.ThreadingTCPServer):
> """simple TCP socket-basedloggingreceiver suitable for testing.
> """
>
> allow_reuse_address = 1
>
> def __init__(self, host='localhost',
> port=logging.handlers.DEFAULT_TCP_LOGGING_PORT,
> handler=LogRecordStreamHandler):
>
> SocketServer.ThreadingTCPServer.__init__(self,
> (host, port),
> handler)
> self.abort = 0
> self.timeout = 1
> self.logname = None
>
> def serve_until_stopped(self):
> import select
> abort = 0
> while not abort:
> rd, wr, ex = select.select([self.socket.fileno()],
> [], [],
> self.timeout)
> if rd:
> self.handle_request()
>
> abort = self.abort
>
> print "serve_until_stopped exiting"
>
> #
> # Start ThreadingTCPServer instance to accept SocketHandler log
> # messages from client.
> #
> tcpserver = LogRecordSocketReceiver()
> print "Starting ThreadingTCPServer..."
> tcpserver.serve_until_stopped()
>
> '''
> #-logging.conf-
> [loggers]
> keys=root
>
> [handlers]
> keys=socketHandler
>
> [formatters]
> keys=simpleFormatter
>
> [logger_root]
> level=DEBUG
> handlers=soc
logging via SocketHandler and TCPserver
Every time I look at the logging module (up until now) I've given up and
continue to use my home-grown logger that I've been using for years. I'm not
giving up this time ;-)
I find that I REALLY need to be able to monitor LOTS of running
programs/processes and thought it would be nice to have them use SocketHandler
logging and then I would write TCPServer to accept the log messages for
real-time monitoring. I Googled (is that now a verb?) for several hours and
came up with some code that I've turned in to something that works, but I can't
figure out how to disconnect the server once it is connected The goal is to be
able to start TCPServer, monitor the log messages sent via SocketHandler logger,
disconnect, and move to the next application. Eventually I would like to wrap a
GUI around all of this for monitoring a complex web application.
Everything works, it just appears that I get into the while loop in
LogRecordStreamHandler.handle and it never breaks out (until I kill the client).
I can't seem to do anything with the LogRecordSocketReceiver.abort attribute to
make it quit.
I'm sure it is something simple (stupid?), but I've spent about 4 hours and I'm
not getting anywhere.
Thanks in advance for any assistance.
Regards,
Larry
Below is my code:
import sys
import time
import logging
if sys.argv[1] == 'client':
import logging.config
logging.config.fileConfig("logging.conf")
#create logger
logger = logging.getLogger("VESconsole")
while 1:
logger.debug("debug message")
logger.info("info message")
logger.warn("warn message")
logger.error("error message")
logger.critical("critical message")
time.sleep(2)
elif sys.argv[1] == 'server':
import cPickle
import logging.handlers
import SocketServer
import struct
import signal
class LogRecordStreamHandler(SocketServer.StreamRequestHandler):
"""Handler for a streaming logging request.
This basically logs the record using whatever logging policy is
configured locally.
"""
def handle(self):
"""
Handle multiple requests - each expected to be a 4-byte length,
followed by the LogRecord in pickle format. Logs the record
according to whatever policy is configured locally.
"""
while 1:
chunk = self.connection.recv(4)
if len(chunk) < 4:
break
slen = struct.unpack(">L", chunk)[0]
chunk = self.connection.recv(slen)
while len(chunk) < slen:
chunk = chunk + self.connection.recv(slen - len(chunk))
obj = self.unPickle(chunk)
record = logging.makeLogRecord(obj)
self.handleLogRecord(record)
def unPickle(self, data):
return cPickle.loads(data)
def handleLogRecord(self, record):
t = time.strftime('%a, %d %b %y %H:%M:%S',
time.localtime(record.created))
print "%s %s" % (t, record.getMessage())
class LogRecordSocketReceiver(SocketServer.ThreadingTCPServer):
"""simple TCP socket-based logging receiver suitable for testing.
"""
allow_reuse_address = 1
def __init__(self, host='localhost',
port=logging.handlers.DEFAULT_TCP_LOGGING_PORT,
handler=LogRecordStreamHandler):
SocketServer.ThreadingTCPServer.__init__(self,
(host, port),
handler)
self.abort = 0
self.timeout = 1
self.logname = None
def serve_until_stopped(self):
import select
abort = 0
while not abort:
rd, wr, ex = select.select([self.socket.fileno()],
[], [],
self.timeout)
if rd:
self.handle_request()
abort = self.abort
print "serve_until_stopped exiting"
#
# Start ThreadingTCPServer instance to accept SocketHandler log
# messages from client.
#
tcpserver = LogRecordSocketReceiver()
print "Starting ThreadingTCPServer..."
tcpserver.serve_until_stopped()
'''
#-logging.conf-
[loggers]
keys=root
[handlers]
keys=socketHandler
[formatters]
keys=simpleFormatter
[logger_root]
level=DEBUG
handlers=socketHandler
[handler_socketHandler]
class=handlers.SocketHandler
level=DEBUG
args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT)
host=localhost
port=DEFAULT_TCP_LOGGING_PORT
[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=
'''
--
http://mail.python.org/mailman/listinfo/python-list
