From what I could see you plan to edit the rolling file appender so it
can take a custom action on some events (roll over when already rolled
over files is > X) at which point files are pushed to another system.
Rolling file appender would continue to do what it does natively. I
think a better design would be to enhance the appender in a generic way
so it can have a RollingFileListeners and tell these objects about
important events and pass on config and current state to them. The
listeners can then do further processing, there might be a need to tell
the rolling appender of action taken like files were moved or deleted.
The appender does not need to know where the files are gone (to hadoop
or just zipped for long term storage) but it might help with its
internal state being more consistent with changes to the folder.
On 3/29/2013 9:48 AM, Pranav Bhole wrote:
Hello to All,
As per our discussion I am attaching the "Blueprint of Simple,
Efficient and Reliable HadoopAppender in Log4j". I would love see the
suggestions.
Thank you
On Fri, Mar 22, 2013 at 8:42 PM, Gary Gregory <[email protected]
<mailto:[email protected]>> wrote:
An AppScale appender might be interesting:
http://appscale.cs.ucsb.edu/datastores.html
Gary
On Fri, Mar 22, 2013 at 4:34 PM, Christian Grobmeier
<[email protected] <mailto:[email protected]>> wrote:
+1
I would love to support a GSOC student, and if your more concrete
proposal meets some interest here I am willing to actually help.
That said, while Apache Flume is great, its maybe a bit "too
much". I
already have had some thoughts on some kind of a server which
utilizes
receivers to send data to $x. Less features than Flume, but
easy to
setup. Not sure if that has some value.
Pranav, please let us hear more of your ideas.
On Fri, Mar 22, 2013 at 4:40 PM, Ralph Goers
<[email protected]
<mailto:[email protected]>> wrote:
> The Flume Appender leverages Apache Flume to route data into
various places.
> The primary sponsor of Flume is Cloudera, so naturally Flume
supports
> writing data into Hadoop. In addition, my employer is using
Flume to write
> data into Cassandra. That said, we would welcome
contributions and if you
> can provide more details on how you would implement your
idea I'd love to
> see them. Perhaps you can create a page on the logging wiki
with your
> proposal.
>
> Ralph
>
> On Mar 22, 2013, at 2:04 AM, Pranav Bhole wrote:
>
> Hello to all,
> This is Pranav Bhole, I am Master student at The
University of Texas
> at Dallas. My research interest is Big Data. I haven been
using Log4j
> extensively as core since 5-6 years in my academic and
professional work.
> Recently an idea came up in my mind by facing some of the
difficulties in
> managing the TeraBytes of Log files. I would like to
implement this idea as
> plug in or functionality in the existing log4j appender
module as student of
> Google Summer of Code 2013.
>
> Short description of the idea:
> Server appends the bulk of log files and in the most cases
server lacks with
> the storage space for these logs files and also computing on
such bulk of
> file is costly for the server. With the consideration of
this problem, idea
> proposes to write a module which could be able to move these
files into
> Public (S3 of AWS, Azure) or private cloud (Hadoop) on the
rolling basis
> based on the configuration file. To resolve the computing
layer objective,
> the idea proposes the Big Data Query generator based on the
logging format
> used. Such Big Data Queries will include MapReduce, PIG etc.
Administrator
> would be able to run these BigData queries generated by
Log4j to track the
> keywords in the logs like error number, TimeStamp or any
other arbitrary
> string.
>
> I would like to appreciate to all of you for reading this
idea. I would
> really love to get involved in Log4j development team with
your support and
> suggestion on this idea.
>
> Thank you very much.
>
> --
> Pranav Bhole
> Student of MS in Computer Science for Fall 2012,
> University of Texas at Dallas
> http://www.linkedin.com/in/pranavbhole
> Cell Phone No: 972-978-6108 <tel:972-978-6108>.
>
>
--
http://www.grobmeier.de
https://www.timeandbill.de
---------------------------------------------------------------------
To unsubscribe, e-mail:
[email protected]
<mailto:[email protected]>
For additional commands, e-mail:
[email protected]
<mailto:[email protected]>
--
E-Mail: [email protected] <mailto:[email protected]> |
[email protected] <mailto:[email protected]>
JUnit in Action, 2nd Ed: http://bit.ly/ECvg0
Spring Batch in Action: http://bit.ly/bqpbCK
Blog: http://garygregory.wordpress.com
<http://garygregory.wordpress.com/>
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory
--
Pranav Bhole
Student of MS in Computer Science for Fall 2012,
University of Texas at Dallas
http://www.linkedin.com/in/pranavbhole
Cell Phone No: 972-978-6108.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
