Using Log4j, its possible to write a database or nosql appender [1] if we want to provide a remote access to the logs. I have used database appender before and it is asynchronous logging without any overhead.
Thanks Raminder 1. http://logging.apache.org/log4j/2.x/manual/appenders.html On Aug 28, 2014, at 11:13 AM, Schwartz, Terri <[email protected]> wrote: > For cipres logging we use a single file, rolled over daily, and include > thread-id, and when relevant, task-id in each entry using log4j's nested > diagnostic context. It's usually easy enough to grep for just the task I'm > interested in. > > Terri > ________________________________________ > From: Miller, Mark [[email protected]] > Sent: Thursday, August 28, 2014 8:02 AM > To: [email protected] > Subject: RE: Experiment log messages to different log files > > Re max files, here is what I found: > > It depends on the file system. ext3 suppport ~32000 subdirectories (not > files!) in a given directory, with ext4 it's 64000 by default. xfs has no > limit to my knowledge. You should consider not putting too many files in a > single directory. Most software doesn't handle that well (e.g. mc will be > slow, many gui tools will be unusable). It's better to create a hierarchy of > nested folders and distribute the files in them using some algorithm (hash of > file name or content or any other method which will distribute the files > equally). That's what many mature programs (e.g. squid) do. > > > -----Original Message----- > From: Marlon Pierce [mailto:[email protected]] > Sent: Thursday, August 28, 2014 7:54 AM > To: Airavata Dev > Subject: Experiment log messages to different log files > > I'm wondering if it would be a good idea to create a separate log file for > each experiment. We current direct everything to airavata.log and > airavata-server.out. As a consequence, log entries for different submissions > get interleaved, which makes tracing a particular experiment's life cycle > difficult. > > Note there is a 32,000 file limit per directory in Linux by default (unless > my knowledge is obsolete), so we'd need to take this into account. > > Alternatively, we could do a better job of labeling the entries in > airavata.log so that it was clear which experiment is associated with the > entry. > > Other suggestions? > > Marlon >
