do you konw if the current fluentd supports kafka 3 (my version is 3.2.1)?
fluentd dry-run outputs:
2022-08-12 10:47:24 +0800 [info]: gem
'fluent-plugin-calyptia-monitoring' version '0.1.3'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-elasticsearch'
version '5.2.3'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-flowcounter-simple'
version '0.1.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-kafka' version '0.18.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-metrics-cmetrics'
version '0.1.2'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-opensearch' version
'1.0.7'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-prometheus' version
'2.0.3'
2022-08-12 10:47:24 +0800 [info]: gem
'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-record-modifier'
version '2.1.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-rewrite-tag-filter'
version '2.4.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-s3' version '1.7.1'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-systemd' version
'1.0.5'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-td' version '1.2.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-webhdfs' version
'1.5.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluentd' version '1.15.1'
2022-08-12 10:47:24 +0800 [info]: starting fluentd-1.15.1 as dry run
mode ruby="2.7.6"
2022-08-12 10:47:24 +0800 [info]: brokers has been set: ["127.0.0.1:9092"]
my pipeline is: apache2 weblogs -> fluentd -> kafka -> spark
But the issue I met is I can't write the output to the correct topic
from fluentd, the messages always fall into the default topic, not the
one I specified.
The plugin I am using is:
https://github.com/fluent/fluent-plugin-kafka
which is built-in by td-agent (the latest version as well).
Thanks.