Basapuram Kumar created AMBARI-26617:
----------------------------------------
Summary: Add KAFKA_CLIENT component to deploy Kafka client
configuration on non-broker hosts
Key: AMBARI-26617
URL: https://issues.apache.org/jira/browse/AMBARI-26617
Project: Ambari
Issue Type: New Feature
Components: ambari-server
Affects Versions: 2.7.9, 3.0.0
Reporter: Basapuram Kumar
h3. Summary
Ambari’s Kafka service currently defines only the {{KAFKA_BROKER}} component.
Client configuration files ({{{}kafka-env.sh{}}}, {{{}log4j.properties{}}},
{{{}tools-log4j.properties{}}}, and Kerberos JAAS files) are generated only
when the broker is installed and configured on a host.
This makes it difficult to use Kafka client tools (for example
{{{}kafka-topics.sh{}}}, {{{}kafka-console-producer.sh{}}},
{{{}kafka-console-consumer.sh{}}}) on hosts that do not run a Kafka broker,
especially in Kerberos-enabled clusters where JAAS and environment settings are
required.
This feature adds a new {{KAFKA_CLIENT}} CLIENT component to the Kafka service
so Ambari can install the Kafka client package and push client configuration to
any host in the cluster.
h3. Problem
* Kafka client configs are tied to broker install/configure lifecycle.
* Client-only hosts do not receive required files under
{{{}/etc/kafka/conf{}}}.
* In secure clusters, client tools fail because {{kafka-env.sh}} and JAAS
configs are missing.
* Operators must manually copy or maintain client configs outside Ambari.
h3. Proposed Solution
Add a {{KAFKA_CLIENT}} component to the Kafka service stack definition with:
# New component: {{KAFKA_CLIENT}} (category: {{{}CLIENT{}}}, cardinality:
{{{}1+{}}})
# Install/configure script: {{kafka_client.py}} following the existing Ambari
CLIENT pattern (similar to {{{}HIVE_CLIENT{}}}, {{{}HDFS_CLIENT{}}})
# Generated configs on client hosts:
** {{kafka-env.sh}}
** {{log4j.properties}}
** {{tools-log4j.properties}}
** {{kafka_client_jaas.conf}} (when Kerberos/SASL is enabled)
# Kerberos support:
** Add {{KAFKA_CLIENT}} identity in {{kerberos.json}} referencing broker
principal/keytab
** Update {{kafka_client_jaas_conf.xml}} to use keytab-based login
({{{}useKeyTab{}}}, {{{}principal{}}}) instead of ticket cache
** On client-only hosts, also create {{kafka_jaas.conf}} when absent (since
{{kafka-env.sh}} references it and broker configure does not run there)
# Stack packaging:
** Register {{KAFKA_CLIENT}} in {{stack_packages.json}} with {{kafka-broker}}
stack-select package mapping
# Config export support:
** Add client-related params ({{{}kafka_broker_hosts{}}},
{{{}kafka_broker_port{}}}) for exported client config files
h3. Scope
* Target stack: {{BIGTOP/3.2.0}} (base Kafka service definition)
h3. Benefits
* Enables Ambari-managed Kafka client deployment on non-broker nodes
* Improves Kerberos cluster usability for Kafka CLI/tools
* Aligns Kafka with existing Ambari CLIENT component patterns
* Reduces manual config management and operational errors
h3. Acceptance Criteria
* {{KAFKA_CLIENT}} appears as assignable component in Ambari UI for Kafka
service
* Install/Configure on a host without {{KAFKA_BROKER}} succeeds
* Client config files are created under {{/etc/kafka/conf}}
* In Kerberos-enabled cluster, client tools can authenticate using generated
JAAS/env files
* Stack upgrade pre-restart for client component selects correct package
version
* No regression for existing {{KAFKA_BROKER}} install/configure/start/stop
behavior
h3. Test Plan
* Setup a 4-node cluster with 3-kafka brokers and 1-kafka client node.
* {{Verify KAFKA_CLIENT}} Installations on client node.
* Verify configuration files "/etc/kafka/conf/" at client node.
* perform kafka CLI operations from client like kafka console topics list,
producer, consumer related operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
