One of the main reasons why I supported Flume joining this PMC was that I noticed it has significant overlap with projects in the observability space despite not being advertised as such. For example, the project FluentBit is extremely similar to Flume, but its main purpose is for collecting, processing, forwarding, etc., logs, metrics, and traces (i.e., observability data). FluentBit is not the only thing in this space, though it seems to be fairly popular. These sorts of tools are used for ultimately publishing observability data to one or more observability tools like Prometheus, Splunk, Jaeger, Grafana, etc., and with a unified collector and processor, it becomes possible to publish all your observability data into one tool rather than three or more disparate tools (and the added operational costs of storing tons of duplicated log data from three or more methods of generating log data).
A project at the CNCF, OpenTelemetry, has become the sort of de facto standard for interoperability in this space. In particular, they’ve published the OTLP specification <https://opentelemetry.io/docs/specs/otlp/> for general telemetry data delivery and the OpenTelemetry specification <https://opentelemetry.io/docs/specs/otel/> for various common APIs. While I’m still researching in this space, I think it would be useful for Flume to integrate with some of these APIs and SDKs (while other parts might be more relevant in our logging libraries instead). There is also the Open Agent Management Protocol <https://opentelemetry.io/docs/specs/opamp/> which is still in beta status that might also be relevant here (and potentially relevant in the logging libraries). Supporting common standards for our projects seems like a useful thing to do, and despite the popularity of some existing solutions there, I believe there is plenty of space for us to contribute. I also think that this can provide opportunity for the various components in this PMC to interoperate as these specs are fairly language neutral with some sample versions in many different languages.
