well that’s the idea. have log4j rotate files, keep it in a separate directory and a script or an agent take care of compression and shipping.
On 17 Nov 2017, 12:07 AM +0530, Matt Sicker <[email protected]>, wrote: > As for an agent, that'd be a separate daemon to watch your log files and > compress them. Due to file handle issues with Java, I don't think said > agent could handle file rotations and such without some convoluted staging > area. > > On 16 November 2017 at 12:34, Matt Sicker <[email protected]> wrote: > > > I don't think Java supports process forking (not even sure if Windows > > does), but you can execute and control processes at least. > > > > On 16 November 2017 at 12:28, Ralph Goers <[email protected] > > wrote: > > > > > Sure, why not? > > > > > > Ralph > > > > > > > On Nov 16, 2017, at 11:24 AM, Chandra <chandra.tungathurthi@rwth-aac > > > hen.de> wrote: > > > > > > > > with-in log4j? > > > > > > > > On 16 Nov 2017, 11:46 PM +0530, Ralph Goers > > > > <[email protected]>, > > > wrote: > > > > > I could see forking a process instead of spawning a thread to perform > > > the compression as a viable approach. > > > > > > > > > > Ralph > > > > > > > > > > > On Nov 16, 2017, at 10:01 AM, Chandra <chandra.tungathurthi@rwth-aac > > > hen.de> wrote: > > > > > > > > > > > > thanks for info matt, was planning to use this on some other project > > > in mind ( might pick your brain on that later ;) ) > > > > > > > > > > > > separating it out the compression to external process isn’t > > > necessarily a bad idea, but having non-reliable scripts is. As it so > > > happened many times before. I’d rather depend on my app than an external > > > process. > > > > > > which is why I was looking for an “agent” . > > > > > > > > > > > > thanks! > > > > > > chandra > > > > > > > > > > > > > > > > > > On 16 Nov 2017, 9:57 PM +0530, Matt Sicker <[email protected]>, > > > > > > wrote: > > > > > > > I brought up Snappy only because I used their off-heap API > > > > > > > recently. > > > Snappy > > > > > > > is more about real time compression rather than size (I think > > > > > > > snappy > > > files > > > > > > > tend to be larger than gzip files, but take less resources to > > > compress and > > > > > > > decompress). The idea here is to offer support via libraries using > > > native > > > > > > > implementations that can work with direct byte buffers, mmap'd > > > files, or > > > > > > > even just a file name. > > > > > > > > > > > > > > With that in mind, is it so bad to offer the ability to execute an > > > external > > > > > > > process to compress the file? > > > > > > > > > > > > > > On 16 November 2017 at 09:59, Chandra > > > > > > > <chandra.tungathurthi@rwth-aac > > > hen.de > > > > > > > wrote: > > > > > > > > > > > > > > > > What if compression worked off-heap > > > > > > > > off-heap compression sounds interesting. Let me check if I can > > > > > > > > find > > > any. > > > > > > > > > > > > > > > > Snappy’s compression is a different altogether, I am not > > > > > > > > necessarily > > > > > > > > looking for a different compression formats, as I’d have add > > > support for it > > > > > > > > in downstream. Standard bz2 , gzip would work in-fact. > > > > > > > > > > > > > > > > > > > > > > > > Ideally a reliable agent something like `FileBeat` would be > > > > > > > > great > > > for this > > > > > > > > situation. :-/ > > > > > > > > > > > > > > > > Best, > > > > > > > > Chandra > > > > > > > > > > > > > > > > > > > > > > > > On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <[email protected]>, > > > wrote: > > > > > > > > > What if compression worked off-heap like with some of the > > > > > > > > > native > > > > > > > > > implementations of codecs? I'm thinking of this one < > > > > > > > > > https://github.com/xerial/snappy-java> for example. > > > > > > > > > > > > > > > > > > On 15 November 2017 at 23:41, Chandra > > > > > > > > > <chandra.tungathurthi@rwth- > > > > > > > > aachen.de > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Guys, > > > > > > > > > > > > > > > > > > > > I need some input on how this handle situation: > > > > > > > > > > > > > > > > > > > > we are on HP/LL setting where the incoming requests are > > > > > > > > > > processed > > > and > > > > > > > > > > logged ( buffered, of course with “async logging”). There > > > > > > > > > > are some > > > > > > > > > > situations where due to spikes in the volume of requests, > > > > > > > > > > the > > > > > > > > compression > > > > > > > > > > on rolling creates memory starvation. > > > > > > > > > > > > > > > > > > > > Now, a straight forward fix would to remove it from the > > > > > > > > > > “context” > > > of > > > > > > > > jvm > > > > > > > > > > and use a script to monitor and compress it timely. as > > > low-hanging the > > > > > > > > > > solution may be, I am skeptical of this solution as the > > > > > > > > > > script > > > _may_ > > > > > > > > fail > > > > > > > > > > leading to disk starvation(!) > > > > > > > > > > > > > > > > > > > > I am looking for an alternate and _full-proof_ solution for > > > > > > > > > > this > > > > > > > > > > situation. Any thoughts, suggestions would be useful and > > > appreciated. > > > > > > > > > > > > > > > > > > > > thanks! > > > > > > > > > > Chandra > > > > > > > > > > > > > > > > > > > > On 16 Nov 2017, 10:01 AM +0530, Ralph Goers < > > > > > > > > [email protected]>, > > > > > > > > > > wrote: > > > > > > > > > > > Unless someone objects I plan to start the release > > > > > > > > > > > process for > > > Log4j > > > > > > > > > > 2.10 tomorrow. I had wanted to include a fix for > > > > > > > > > > LOG4J2-2106 but > > > the > > > > > > > > > > problem only occurs in rare situations and I am not sure > > > > > > > > > > how to > > > fix it > > > > > > > > yet. > > > > > > > > > > There are a lot of other issues that deserve looking at but > > > nothing I > > > > > > > > can > > > > > > > > > > see that warrants waiting on. > > > > > > > > > > > > > > > > > > > > > > Ralph > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Matt Sicker <[email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Matt Sicker <[email protected] > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Matt Sicker <[email protected] > > > > > > -- > Matt Sicker <[email protected]
