wow - that's a lot of data! when you say 60 or 90% reduction, are you
talking about the sizes compared to the uncompressed data? When I made the
decision to use gzip it was largely driven by the Compress::gzip library
that came with perl and because preliminary tests showed close to 90%.
are you talking centrally storing raw or plot data? do you need to store
everything? I've seen close to 1/2 the data is for processes alone and by
using --tworaw you can at least generate separate process data files and
that could give you the choice for storing one or both types. and of
course tab files container much (but not all) the data in the raw files and
take a lot less storage as well. though they lack the capability for
playback.
something I've toyed with by never had the time to do, but maybe you'd like
to raise your hand ;), is to provide the ability to play back plot data.
then you could centrally store it and also have the ability to play it
back if you wanted to drill down. it actually wouldn't be that hard to do,
basically reading in the plot file and loading all the data into the right
variables before the print routines are called. I'd certainly be happy to
point you (or anyone else) to the places that would need to change.
-mark
On Mon, Jul 6, 2015 at 10:54 PM, Aaron Knister <[email protected]>
wrote:
> Hi Everyone,
>
> I'm interested in using collectl to monitor ~3500 nodes. I've used it
> extensively in the past but not at this scale. Central collection of the
> collectl data files is proving to be a small feat-- I'm looking at 136GB of
> data per day. I was toying with the possibility of using bzip2 to compress
> the data files and I got about a 60% reduction in size.. Another test using
> lzma gave me a 90% reduction in file size. I'm pretty sure bzip2 and lzma
> require larger amounts of CPU time which might make them undesirable for
> real-time compression.
>
> Does anyone have thoughts on how to add support for bzip2 or lzma? Is
> there even interest in having these features in collectl?
>
> Thanks!
>
> -Aaron
>
>
>
> ------------------------------------------------------------------------------
> Don't Limit Your Business. Reach for the Cloud.
> GigeNET's Cloud Solutions provide you with the tools and support that
> you need to offload your IT needs and focus on growing your business.
> Configured For All Businesses. Start Your Cloud Today.
> https://www.gigenetcloud.com/
> _______________________________________________
> Collectl-interest mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/collectl-interest
>
>
------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Collectl-interest mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/collectl-interest
