On Thu, Nov 30, 2023, at 18:04, Matt Sicker wrote:
> Oh yes, I’d still love to see how we can adapt the Log4j plugin system
> here! And yeah, this would likely make sense as its own repository.
> I’ll make a proposal about the OTel stuff before working on any code.
Please do. I would also like to learn more about this stuff and maybe we can
make Chainsaw to receive Flume messages or OTel things too!
> —
> Matt Sicker
>
>> On Nov 29, 2023, at 22:54, Ralph Goers <ralph.go...@dslextreme.com> wrote:
>>
>> This is a great post Matt!
>>
>> Fluentbit, Fluentd, Logstash, and Filebeat are the main tools used for log
>> forwarding. While they all have some amount of plugability none of the are
>> as flexible as Flume. In addition, as I have mentioned before, none of them
>> provide guaranteed delivery so I would never recommend them for forwarding
>> audit logs.
>>
>> I have also previously explained my use case for using Flume, which is for
>> forwarding Call Detail Records that start off as records in the Radius
>> protocol [1] across data centers, which also requires guaranteed delivery. I
>> wouldn’t be able to use any of those other tools to do that without
>> significant modification.
>>
>> I am all for supporting standards. If you can outline what you are proposing
>> on a Confluence page I would wholeheartedly support it.
>>
>> As you probably know, I started work on separating out things that I don’t
>> consider to be “core” to Flume into separate repos. That work is only half
>> completed. I would suggest that you consider whether what you are proposing
>> also be in its own repo. As it is, the CI for Flume fails because the
>> generated logs are exceeding the available disk space. In addition, the
>> build takes a long time.
>>
>> Also, I have never really been a big fan of the configuration mechanism
>> Flume uses. I was able to somewhat bypass it by implementing support for
>> Spring Boot, but it would be great if the Log4j Plugin system could somehow
>> be used to simplify configuring Flume for those who don’t want to use Spring
>> boot. I know that is right up your alley.
>>
>> Ralph
>>
>>
>> 1. https://networkradius.com/doc/current/introduction/RADIUS.html
>>
>>> On Nov 29, 2023, at 5:32 PM, Matt Sicker <m...@musigma.org> wrote:
>>>
>>> One of the main reasons why I supported Flume joining this PMC was that I
>>> noticed it has significant overlap with projects in the observability space
>>> despite not being advertised as such. For example, the project FluentBit is
>>> extremely similar to Flume, but its main purpose is for collecting,
>>> processing, forwarding, etc., logs, metrics, and traces (i.e.,
>>> observability data). FluentBit is not the only thing in this space, though
>>> it seems to be fairly popular. These sorts of tools are used for ultimately
>>> publishing observability data to one or more observability tools like
>>> Prometheus, Splunk, Jaeger, Grafana, etc., and with a unified collector and
>>> processor, it becomes possible to publish all your observability data into
>>> one tool rather than three or more disparate tools (and the added
>>> operational costs of storing tons of duplicated log data from three or more
>>> methods of generating log data).
>>>
>>> A project at the CNCF, OpenTelemetry, has become the sort of de facto
>>> standard for interoperability in this space. In particular, they’ve
>>> published the OTLP specification
>>> <https://opentelemetry.io/docs/specs/otlp/> for general telemetry data
>>> delivery and the OpenTelemetry specification
>>> <https://opentelemetry.io/docs/specs/otel/> for various common APIs. While
>>> I’m still researching in this space, I think it would be useful for Flume
>>> to integrate with some of these APIs and SDKs (while other parts might be
>>> more relevant in our logging libraries instead). There is also the Open
>>> Agent Management Protocol <https://opentelemetry.io/docs/specs/opamp/>
>>> which is still in beta status that might also be relevant here (and
>>> potentially relevant in the logging libraries).
>>>
>>> Supporting common standards for our projects seems like a useful thing to
>>> do, and despite the popularity of some existing solutions there, I believe
>>> there is plenty of space for us to contribute. I also think that this can
>>> provide opportunity for the various components in this PMC to interoperate
>>> as these specs are fairly language neutral with some sample versions in
>>> many different languages.
>>
>>
