This is a great post Matt! Fluentbit, Fluentd, Logstash, and Filebeat are the main tools used for log forwarding. While they all have some amount of plugability none of the are as flexible as Flume. In addition, as I have mentioned before, none of them provide guaranteed delivery so I would never recommend them for forwarding audit logs.
I have also previously explained my use case for using Flume, which is for forwarding Call Detail Records that start off as records in the Radius protocol [1] across data centers, which also requires guaranteed delivery. I wouldn’t be able to use any of those other tools to do that without significant modification. I am all for supporting standards. If you can outline what you are proposing on a Confluence page I would wholeheartedly support it. As you probably know, I started work on separating out things that I don’t consider to be “core” to Flume into separate repos. That work is only half completed. I would suggest that you consider whether what you are proposing also be in its own repo. As it is, the CI for Flume fails because the generated logs are exceeding the available disk space. In addition, the build takes a long time. Also, I have never really been a big fan of the configuration mechanism Flume uses. I was able to somewhat bypass it by implementing support for Spring Boot, but it would be great if the Log4j Plugin system could somehow be used to simplify configuring Flume for those who don’t want to use Spring boot. I know that is right up your alley. Ralph 1. https://networkradius.com/doc/current/introduction/RADIUS.html > On Nov 29, 2023, at 5:32 PM, Matt Sicker <m...@musigma.org> wrote: > > One of the main reasons why I supported Flume joining this PMC was that I > noticed it has significant overlap with projects in the observability space > despite not being advertised as such. For example, the project FluentBit is > extremely similar to Flume, but its main purpose is for collecting, > processing, forwarding, etc., logs, metrics, and traces (i.e., observability > data). FluentBit is not the only thing in this space, though it seems to be > fairly popular. These sorts of tools are used for ultimately publishing > observability data to one or more observability tools like Prometheus, > Splunk, Jaeger, Grafana, etc., and with a unified collector and processor, it > becomes possible to publish all your observability data into one tool rather > than three or more disparate tools (and the added operational costs of > storing tons of duplicated log data from three or more methods of generating > log data). > > A project at the CNCF, OpenTelemetry, has become the sort of de facto > standard for interoperability in this space. In particular, they’ve published > the OTLP specification <https://opentelemetry.io/docs/specs/otlp/> for > general telemetry data delivery and the OpenTelemetry specification > <https://opentelemetry.io/docs/specs/otel/> for various common APIs. While > I’m still researching in this space, I think it would be useful for Flume to > integrate with some of these APIs and SDKs (while other parts might be more > relevant in our logging libraries instead). There is also the Open Agent > Management Protocol <https://opentelemetry.io/docs/specs/opamp/> which is > still in beta status that might also be relevant here (and potentially > relevant in the logging libraries). > > Supporting common standards for our projects seems like a useful thing to do, > and despite the popularity of some existing solutions there, I believe there > is plenty of space for us to contribute. I also think that this can provide > opportunity for the various components in this PMC to interoperate as these > specs are fairly language neutral with some sample versions in many different > languages.
