As for an agent, that'd be a separate daemon to watch your log files and compress them. Due to file handle issues with Java, I don't think said agent could handle file rotations and such without some convoluted staging area.
On 16 November 2017 at 12:34, Matt Sicker <[email protected]> wrote: > I don't think Java supports process forking (not even sure if Windows > does), but you can execute and control processes at least. > > On 16 November 2017 at 12:28, Ralph Goers <[email protected]> > wrote: > >> Sure, why not? >> >> Ralph >> >> > On Nov 16, 2017, at 11:24 AM, Chandra <chandra.tungathurthi@rwth-aac >> hen.de> wrote: >> > >> > with-in log4j? >> > >> > On 16 Nov 2017, 11:46 PM +0530, Ralph Goers <[email protected]>, >> wrote: >> >> I could see forking a process instead of spawning a thread to perform >> the compression as a viable approach. >> >> >> >> Ralph >> >> >> >>> On Nov 16, 2017, at 10:01 AM, Chandra <chandra.tungathurthi@rwth-aac >> hen.de> wrote: >> >>> >> >>> thanks for info matt, was planning to use this on some other project >> in mind ( might pick your brain on that later ;) ) >> >>> >> >>> separating it out the compression to external process isn’t >> necessarily a bad idea, but having non-reliable scripts is. As it so >> happened many times before. I’d rather depend on my app than an external >> process. >> >>> which is why I was looking for an “agent” . >> >>> >> >>> thanks! >> >>> chandra >> >>> >> >>> >> >>> On 16 Nov 2017, 9:57 PM +0530, Matt Sicker <[email protected]>, wrote: >> >>>> I brought up Snappy only because I used their off-heap API recently. >> Snappy >> >>>> is more about real time compression rather than size (I think snappy >> files >> >>>> tend to be larger than gzip files, but take less resources to >> compress and >> >>>> decompress). The idea here is to offer support via libraries using >> native >> >>>> implementations that can work with direct byte buffers, mmap'd >> files, or >> >>>> even just a file name. >> >>>> >> >>>> With that in mind, is it so bad to offer the ability to execute an >> external >> >>>> process to compress the file? >> >>>> >> >>>> On 16 November 2017 at 09:59, Chandra <chandra.tungathurthi@rwth-aac >> hen.de >> >>>> wrote: >> >>>> >> >>>>>> What if compression worked off-heap >> >>>>> off-heap compression sounds interesting. Let me check if I can find >> any. >> >>>>> >> >>>>> Snappy’s compression is a different altogether, I am not necessarily >> >>>>> looking for a different compression formats, as I’d have add >> support for it >> >>>>> in downstream. Standard bz2 , gzip would work in-fact. >> >>>>> >> >>>>> >> >>>>> Ideally a reliable agent something like `FileBeat` would be great >> for this >> >>>>> situation. :-/ >> >>>>> >> >>>>> Best, >> >>>>> Chandra >> >>>>> >> >>>>> >> >>>>> On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <[email protected]>, >> wrote: >> >>>>>> What if compression worked off-heap like with some of the native >> >>>>>> implementations of codecs? I'm thinking of this one < >> >>>>>> https://github.com/xerial/snappy-java> for example. >> >>>>>> >> >>>>>> On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth- >> >>>>> aachen.de >> >>>>>> wrote: >> >>>>>> >> >>>>>>> Guys, >> >>>>>>> >> >>>>>>> I need some input on how this handle situation: >> >>>>>>> >> >>>>>>> we are on HP/LL setting where the incoming requests are processed >> and >> >>>>>>> logged ( buffered, of course with “async logging”). There are some >> >>>>>>> situations where due to spikes in the volume of requests, the >> >>>>> compression >> >>>>>>> on rolling creates memory starvation. >> >>>>>>> >> >>>>>>> Now, a straight forward fix would to remove it from the “context” >> of >> >>>>> jvm >> >>>>>>> and use a script to monitor and compress it timely. as >> low-hanging the >> >>>>>>> solution may be, I am skeptical of this solution as the script >> _may_ >> >>>>> fail >> >>>>>>> leading to disk starvation(!) >> >>>>>>> >> >>>>>>> I am looking for an alternate and _full-proof_ solution for this >> >>>>>>> situation. Any thoughts, suggestions would be useful and >> appreciated. >> >>>>>>> >> >>>>>>> thanks! >> >>>>>>> Chandra >> >>>>>>> >> >>>>>>> On 16 Nov 2017, 10:01 AM +0530, Ralph Goers < >> >>>>> [email protected]>, >> >>>>>>> wrote: >> >>>>>>>> Unless someone objects I plan to start the release process for >> Log4j >> >>>>>>> 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but >> the >> >>>>>>> problem only occurs in rare situations and I am not sure how to >> fix it >> >>>>> yet. >> >>>>>>> There are a lot of other issues that deserve looking at but >> nothing I >> >>>>> can >> >>>>>>> see that warrants waiting on. >> >>>>>>>> >> >>>>>>>> Ralph >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> Matt Sicker <[email protected] >> >>>>> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Matt Sicker <[email protected] >> >> >> >> >> >> >> > > > -- > Matt Sicker <[email protected]> > -- Matt Sicker <[email protected]>
