A key feature for me is guaranteed delivery, which is why is sometimes I use JMS with Log4j.
Gary On Wed, Nov 29, 2023, 11:54 PM Ralph Goers <ralph.go...@dslextreme.com> wrote: > This is a great post Matt! > > Fluentbit, Fluentd, Logstash, and Filebeat are the main tools used for log > forwarding. While they all have some amount of plugability none of the are > as flexible as Flume. In addition, as I have mentioned before, none of them > provide guaranteed delivery so I would never recommend them for forwarding > audit logs. > > I have also previously explained my use case for using Flume, which is for > forwarding Call Detail Records that start off as records in the Radius > protocol [1] across data centers, which also requires guaranteed delivery. > I wouldn’t be able to use any of those other tools to do that without > significant modification. > > I am all for supporting standards. If you can outline what you are > proposing on a Confluence page I would wholeheartedly support it. > > As you probably know, I started work on separating out things that I don’t > consider to be “core” to Flume into separate repos. That work is only half > completed. I would suggest that you consider whether what you are proposing > also be in its own repo. As it is, the CI for Flume fails because the > generated logs are exceeding the available disk space. In addition, the > build takes a long time. > > Also, I have never really been a big fan of the configuration mechanism > Flume uses. I was able to somewhat bypass it by implementing support for > Spring Boot, but it would be great if the Log4j Plugin system could somehow > be used to simplify configuring Flume for those who don’t want to use > Spring boot. I know that is right up your alley. > > Ralph > > > 1. https://networkradius.com/doc/current/introduction/RADIUS.html > > > On Nov 29, 2023, at 5:32 PM, Matt Sicker <m...@musigma.org> wrote: > > > > One of the main reasons why I supported Flume joining this PMC was that > I noticed it has significant overlap with projects in the observability > space despite not being advertised as such. For example, the project > FluentBit is extremely similar to Flume, but its main purpose is for > collecting, processing, forwarding, etc., logs, metrics, and traces (i.e., > observability data). FluentBit is not the only thing in this space, though > it seems to be fairly popular. These sorts of tools are used for ultimately > publishing observability data to one or more observability tools like > Prometheus, Splunk, Jaeger, Grafana, etc., and with a unified collector and > processor, it becomes possible to publish all your observability data into > one tool rather than three or more disparate tools (and the added > operational costs of storing tons of duplicated log data from three or more > methods of generating log data). > > > > A project at the CNCF, OpenTelemetry, has become the sort of de facto > standard for interoperability in this space. In particular, they’ve > published the OTLP specification < > https://opentelemetry.io/docs/specs/otlp/> for general telemetry data > delivery and the OpenTelemetry specification < > https://opentelemetry.io/docs/specs/otel/> for various common APIs. While > I’m still researching in this space, I think it would be useful for Flume > to integrate with some of these APIs and SDKs (while other parts might be > more relevant in our logging libraries instead). There is also the Open > Agent Management Protocol <https://opentelemetry.io/docs/specs/opamp/> > which is still in beta status that might also be relevant here (and > potentially relevant in the logging libraries). > > > > Supporting common standards for our projects seems like a useful thing > to do, and despite the popularity of some existing solutions there, I > believe there is plenty of space for us to contribute. I also think that > this can provide opportunity for the various components in this PMC to > interoperate as these specs are fairly language neutral with some sample > versions in many different languages. > > >
