[ https://issues.apache.org/jira/browse/KAFKA-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Bono updated KAFKA-16708: ------------------------------- Description: First of all, thank you all for adding the official Kafka Docker image (I know it is a big responsibility and adds to the team workload). I am migrating from {{wurstmeister/kafka}} to the official {{apache/kafka}} image. My advertised port is not static and was relying on [the PORT_COMMAND feature|https://github.com/wurstmeister/kafka-docker/commit/c66375fc3b94e98dbecd603c5d2b44c06e927e88] in the {{wurstmeister/kafka}} image to determine the port programatically. This would let me define a docker-compose as follows: {code:java} services: kafka: image: apache/kafka:latest hostname: kafka ports: - "9092" volumes: - '/var/run/docker.sock.raw:/var/run/docker.sock' environment: KAFKA_NODE_ID: 1 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT_DOCKER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT' KAFKA_LISTENERS: 'CONTROLLER://kafka:29093,PLAINTEXT_DOCKER://kafka:29092,PLAINTEXT_HOST://0.0.0.0:9092' KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT_DOCKER://kafka:29092,PLAINTEXT_HOST://localhost:_{PORT_COMMAND}' KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_PROCESS_ROLES: 'broker,controller' KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093' KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT_DOCKER' KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' PORT_COMMAND: "docker ps | egrep 'kafka' | cut -d: -f 3 | cut -d- -f 1"{code} Notice how the "ports" are dynamically mapped (i.e. not *"port:port"* syntax) - the actual port will *not* be "9092". Do you have a suggestion for an alternative approach on how to obtain a non-static port for advertised listeners? If not, would adding conditional support for this feature be welcomed? I am aware of the complication/concern of this request as it is Docker in Docker for non-root users (described [here|https://jonfriesen.ca/articles/docker-in-docker-non-root-user/]) and as such we could make it inactive by default and users would have to opt-in explicitly. I have created a [rough WIP|https://github.com/onobc/kafka/commit/6556c4adbf08155b89c9804c2c5d1a988f8371f2] that illustrates the concept (there is no conditionality in it currently). Note that the container is not run as {*}root{*}, but rather the *appuser* is added to whatever group that own the docker.sock (which on my machine is root). P.S. * This is my first time filing an issue w/ Kafka so if I missed anything please let me know and I am glad to add whatever other info, etc.. * I am not sure what "Component" this should be under (the other Kafka Docker related issues had differing values here) was: First of all, thank you all for adding the official Kafka Docker image (I know it is a big responsibility and adds to the team workload). I am migrating from {{wurstmeister/kafka}} to the official {{apache/kafka}} image. My advertised port is not static and was relying on [the PORT_COMMAND feature|https://github.com/wurstmeister/kafka-docker/commit/c66375fc3b94e98dbecd603c5d2b44c06e927e88] in the {{wurstmeister/kafka}} image to determine the port programatically. This would let me define a docker-compose as follows: {{}} {code:java} services: kafka: image: apache/kafka:latest hostname: kafka ports: - "9092" volumes: - '/var/run/docker.sock.raw:/var/run/docker.sock' environment: KAFKA_NODE_ID: 1 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT_DOCKER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT' KAFKA_LISTENERS: 'CONTROLLER://kafka:29093,PLAINTEXT_DOCKER://kafka:29092,PLAINTEXT_HOST://0.0.0.0:9092' KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT_DOCKER://kafka:29092,PLAINTEXT_HOST://localhost:_{PORT_COMMAND}' KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_PROCESS_ROLES: 'broker,controller' KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093' KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT_DOCKER' KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' PORT_COMMAND: "docker ps | egrep 'kafka' | cut -d: -f 3 | cut -d- -f 1"{code} {{}} Notice how the "ports" are dynamically mapped (i.e. not *"port:port"* syntax) - the actual port will *not* be "9092". Do you have a suggestion for an alternative approach on how to obtain a non-static port for advertised listeners? If not, would adding conditional support for this feature be welcomed? I am aware of the complication/concern of this request as it is Docker in Docker for non-root users (described [here|https://jonfriesen.ca/articles/docker-in-docker-non-root-user/]) and as such we could make it inactive by default and users would have to opt-in explicitly. I have created a [rough WIP|https://github.com/onobc/kafka/commit/6556c4adbf08155b89c9804c2c5d1a988f8371f2] that illustrates the concept (there is no conditionality in it currently). Note that the container is not run as {*}root{*}, but rather the *appuser* is added to whatever group that own the docker.sock (which on my machine is root). P.S. This is my first time filing an issue w/ Kafka so if I missed anything please let me know and I am glad to add whatever other info, etc.. > Allow dynamic port for advertised listeners in Docker image > ----------------------------------------------------------- > > Key: KAFKA-16708 > URL: https://issues.apache.org/jira/browse/KAFKA-16708 > Project: Kafka > Issue Type: Improvement > Reporter: Chris Bono > Priority: Major > > First of all, thank you all for adding the official Kafka Docker image (I > know it is a big responsibility and adds to the team workload). > I am migrating from {{wurstmeister/kafka}} to the official {{apache/kafka}} > image. > My advertised port is not static and was relying on [the PORT_COMMAND > feature|https://github.com/wurstmeister/kafka-docker/commit/c66375fc3b94e98dbecd603c5d2b44c06e927e88] > in the {{wurstmeister/kafka}} image to determine the port programatically. > This would let me define a docker-compose as follows: > {code:java} > services: > kafka: > image: apache/kafka:latest > hostname: kafka > ports: > - "9092" > volumes: > - '/var/run/docker.sock.raw:/var/run/docker.sock' > environment: > KAFKA_NODE_ID: 1 > KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: > 'CONTROLLER:PLAINTEXT,PLAINTEXT_DOCKER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT' > KAFKA_LISTENERS: > 'CONTROLLER://kafka:29093,PLAINTEXT_DOCKER://kafka:29092,PLAINTEXT_HOST://0.0.0.0:9092' > KAFKA_ADVERTISED_LISTENERS: > 'PLAINTEXT_DOCKER://kafka:29092,PLAINTEXT_HOST://localhost:_{PORT_COMMAND}' > KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 > KAFKA_PROCESS_ROLES: 'broker,controller' > KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093' > KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT_DOCKER' > KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' > PORT_COMMAND: "docker ps | egrep 'kafka' | cut -d: -f 3 | cut -d- -f > 1"{code} > Notice how the "ports" are dynamically mapped (i.e. not *"port:port"* syntax) > - the actual port will *not* be "9092". > Do you have a suggestion for an alternative approach on how to obtain a > non-static port for advertised listeners? If not, would adding conditional > support for this feature be welcomed? > I am aware of the complication/concern of this request as it is Docker in > Docker for non-root users (described > [here|https://jonfriesen.ca/articles/docker-in-docker-non-root-user/]) and as > such we could make it inactive by default and users would have to opt-in > explicitly. > I have created a [rough > WIP|https://github.com/onobc/kafka/commit/6556c4adbf08155b89c9804c2c5d1a988f8371f2] > that illustrates the concept (there is no conditionality in it currently). > Note that the container is not run as {*}root{*}, but rather the *appuser* is > added to whatever group that own the docker.sock (which on my machine is > root). > > P.S. > * This is my first time filing an issue w/ Kafka so if I missed anything > please let me know and I am glad to add whatever other info, etc.. > * I am not sure what "Component" this should be under (the other Kafka > Docker related issues had differing values here) > > -- This message was sent by Atlassian Jira (v8.20.10#820010)