I could see forking a process instead of spawning a thread to perform the compression as a viable approach.
Ralph > On Nov 16, 2017, at 10:01 AM, Chandra <[email protected]> > wrote: > > thanks for info matt, was planning to use this on some other project in mind > ( might pick your brain on that later ;) ) > > separating it out the compression to external process isn’t necessarily a bad > idea, but having non-reliable scripts is. As it so happened many times > before. I’d rather depend on my app than an external process. > which is why I was looking for an “agent” . > > thanks! > chandra > > > On 16 Nov 2017, 9:57 PM +0530, Matt Sicker <[email protected]>, wrote: >> I brought up Snappy only because I used their off-heap API recently. Snappy >> is more about real time compression rather than size (I think snappy files >> tend to be larger than gzip files, but take less resources to compress and >> decompress). The idea here is to offer support via libraries using native >> implementations that can work with direct byte buffers, mmap'd files, or >> even just a file name. >> >> With that in mind, is it so bad to offer the ability to execute an external >> process to compress the file? >> >> On 16 November 2017 at 09:59, Chandra <[email protected] >> wrote: >> >>>> What if compression worked off-heap >>> off-heap compression sounds interesting. Let me check if I can find any. >>> >>> Snappy’s compression is a different altogether, I am not necessarily >>> looking for a different compression formats, as I’d have add support for it >>> in downstream. Standard bz2 , gzip would work in-fact. >>> >>> >>> Ideally a reliable agent something like `FileBeat` would be great for this >>> situation. :-/ >>> >>> Best, >>> Chandra >>> >>> >>> On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <[email protected]>, wrote: >>>> What if compression worked off-heap like with some of the native >>>> implementations of codecs? I'm thinking of this one < >>>> https://github.com/xerial/snappy-java> for example. >>>> >>>> On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth- >>> aachen.de >>>> wrote: >>>> >>>>> Guys, >>>>> >>>>> I need some input on how this handle situation: >>>>> >>>>> we are on HP/LL setting where the incoming requests are processed and >>>>> logged ( buffered, of course with “async logging”). There are some >>>>> situations where due to spikes in the volume of requests, the >>> compression >>>>> on rolling creates memory starvation. >>>>> >>>>> Now, a straight forward fix would to remove it from the “context” of >>> jvm >>>>> and use a script to monitor and compress it timely. as low-hanging the >>>>> solution may be, I am skeptical of this solution as the script _may_ >>> fail >>>>> leading to disk starvation(!) >>>>> >>>>> I am looking for an alternate and _full-proof_ solution for this >>>>> situation. Any thoughts, suggestions would be useful and appreciated. >>>>> >>>>> thanks! >>>>> Chandra >>>>> >>>>> On 16 Nov 2017, 10:01 AM +0530, Ralph Goers < >>> [email protected]>, >>>>> wrote: >>>>>> Unless someone objects I plan to start the release process for Log4j >>>>> 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but the >>>>> problem only occurs in rare situations and I am not sure how to fix it >>> yet. >>>>> There are a lot of other issues that deserve looking at but nothing I >>> can >>>>> see that warrants waiting on. >>>>>> >>>>>> Ralph >>>>> >>>> >>>> >>>> >>>> -- >>>> Matt Sicker <[email protected] >>> >> >> >> >> -- >> Matt Sicker <[email protected]
