Thanks for the update. Regards JB
> Le 29 juin 2020 à 08:29, Gerald Kallas <catsh...@mailbox.org> a écrit : > > Hi all, > > I was finally able to get the Kafka connection running with some kafka server > related configuration. > > For information, I'm using the docker image from > https://hub.docker.com/r/bitnami/kafka/ > > This is creating 2 container (Zookeeper and Kafka). I've changed the > docker-compose.yml as following with the lines > > - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,EXTERNAL://:9093 > - > KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://:9092,EXTERNAL://localhost:9093 > - > KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT > > (see full yml below). This allows an internal connection from camel-kafka > inside the docker network as well as an external communication. > > Best > - Gerald > > > version: '2' > > services: > zookeeper: > container_name: zookeeper > image: 'docker.io/bitnami/zookeeper:3-debian-10' > ports: > - '2181:2181' > volumes: > - 'zookeeper_data:/bitnami' > environment: > - ALLOW_ANONYMOUS_LOGIN=yes > kafka: > container_name: kafka > image: 'docker.io/bitnami/kafka:2-debian-10' > ports: > - '9092:9092' > - '9093:9093' > volumes: > - 'kafka_data:/bitnami' > environment: > - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 > - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,EXTERNAL://:9093 > - > KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://:9092,EXTERNAL://localhost:9093 > - > KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT > - ALLOW_PLAINTEXT_LISTENER=yes > depends_on: > - zookeeper > > volumes: > zookeeper_data: > driver: local > kafka_data: > driver: local > > networks: > default: > external: > name: casisp > > >> Jean-Baptiste Onofre <j...@nanthrax.net> hat am 29.06.2020 08:14 geschrieben: >> >> >> Hi, >> >> Don’t you have another trace in the log, like missing Avro packages >> (ClassNotFoundException) in the Kafka bundle client ? >> >> I saw an issue with pulsar bundle missing Avro import (it used avro.* import >> whereas "new" Avro release uses org.apache.avro.*). Maybe Avro optional >> import is required in your case (even if Kafka-clients doesn’t define the >> dependency explicitly). >> >> Regards >> JB >> >>> Le 27 juin 2020 à 09:45, Gerald Kallas <catsh...@mailbox.org> a écrit : >>> >>> Dear all, >>> >>> I've setup Camel 3.4.0 and want to consume from a Kafka topic. My route >>> looks like >>> >>> <blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0";> >>> <camelContext xmlns="http://camel.apache.org/schema/blueprint"; >>> streamCache="true"> >>> <route> >>> <from >>> uri="kafka:casisp?brokers=ec2-3-124-33-3.eu-central-1.compute.amazonaws.com:9092" >>> /> >>> <log message="body: ${in.body}" /> >>> </route> >>> </camelContext> >>> </blueprint> >>> >>> The log shows the connection error as below. >>> >>> A connection from Kafka CLI or another tool works without problems. >>> >>> Any hints are appreciated. >>> >>> - Gerald >>> >>> >>> Log file excerpt >>> >>> 2020-06-27T07:37:29,692 | INFO | Blueprint Event Dispatcher: 1 | >>> KafkaConsumer | 120 - org.apache.camel.camel-kafka - >>> 3.4.0 | Starting Kafka consumer on topic: casisp with breakOnFirstError: >>> false >>> 2020-06-27T07:37:29,710 | INFO | Blueprint Event Dispatcher: 1 | >>> ConsumerConfig | 195 - >>> org.apache.servicemix.bundles.kafka-clients - 2.4.1.1 | ConsumerConfig >>> values: >>> allow.auto.create.topics = true >>> auto.commit.interval.ms = 5000 >>> auto.offset.reset = latest >>> bootstrap.servers = >>> [ec2-3-124-33-3.eu-central-1.compute.amazonaws.com:9092] >>> check.crcs = true >>> client.dns.lookup = default >>> client.id = >>> client.rack = >>> connections.max.idle.ms = 540000 >>> default.api.timeout.ms = 60000 >>> enable.auto.commit = true >>> exclude.internal.topics = true >>> fetch.max.bytes = 52428800 >>> fetch.max.wait.ms = 500 >>> fetch.min.bytes = 1 >>> group.id = 689cc1d7-fcf7-4565-9056-ddd7218d261f >>> group.instance.id = null >>> heartbeat.interval.ms = 3000 >>> interceptor.classes = [] >>> internal.leave.group.on.close = true >>> isolation.level = read_uncommitted >>> key.deserializer = class >>> org.apache.kafka.common.serialization.StringDeserializer >>> max.partition.fetch.bytes = 1048576 >>> max.poll.interval.ms = 300000 >>> max.poll.records = 500 >>> metadata.max.age.ms = 300000 >>> metric.reporters = [] >>> metrics.num.samples = 2 >>> metrics.recording.level = INFO >>> metrics.sample.window.ms = 30000 >>> partition.assignment.strategy = >>> [org.apache.kafka.clients.consumer.RangeAssignor] >>> receive.buffer.bytes = 65536 >>> reconnect.backoff.max.ms = 1000 >>> reconnect.backoff.ms = 50 >>> request.timeout.ms = 40000 >>> retry.backoff.ms = 100 >>> sasl.client.callback.handler.class = null >>> sasl.jaas.config = null >>> sasl.kerberos.kinit.cmd = /usr/bin/kinit >>> sasl.kerberos.min.time.before.relogin = 60000 >>> sasl.kerberos.service.name = null >>> sasl.kerberos.ticket.renew.jitter = 0.05 >>> sasl.kerberos.ticket.renew.window.factor = 0.8 >>> sasl.login.callback.handler.class = null >>> sasl.login.class = null >>> sasl.login.refresh.buffer.seconds = 300 >>> sasl.login.refresh.min.period.seconds = 60 >>> sasl.login.refresh.window.factor = 0.8 >>> sasl.login.refresh.window.jitter = 0.05 >>> sasl.mechanism = GSSAPI >>> security.protocol = PLAINTEXT >>> security.providers = null >>> send.buffer.bytes = 131072 >>> session.timeout.ms = 10000 >>> ssl.cipher.suites = null >>> ssl.enabled.protocols = [TLSv1.2] >>> ssl.endpoint.identification.algorithm = https >>> ssl.key.password = null >>> ssl.keymanager.algorithm = SunX509 >>> ssl.keystore.location = null >>> ssl.keystore.password = null >>> ssl.keystore.type = JKS >>> ssl.protocol = TLSv1.2 >>> ssl.provider = null >>> ssl.secure.random.implementation = null >>> ssl.trustmanager.algorithm = PKIX >>> ssl.truststore.location = null >>> ssl.truststore.password = null >>> ssl.truststore.type = JKS >>> value.deserializer = class >>> org.apache.kafka.common.serialization.StringDeserializer >>> >>> 2020-06-27T07:37:29,884 | WARN | Blueprint Event Dispatcher: 1 | >>> ConsumerConfig | 195 - >>> org.apache.servicemix.bundles.kafka-clients - 2.4.1.1 | The configuration >>> 'specific.avro.reader' was supplied but isn't a known config. >>> 2020-06-27T07:37:29,884 | INFO | Blueprint Event Dispatcher: 1 | >>> AppInfoParser | 195 - >>> org.apache.servicemix.bundles.kafka-clients - 2.4.1.1 | Kafka version: 2.4.1 >>> 2020-06-27T07:37:29,885 | INFO | Blueprint Event Dispatcher: 1 | >>> AppInfoParser | 195 - >>> org.apache.servicemix.bundles.kafka-clients - 2.4.1.1 | Kafka commitId: >>> c57222ae8cd7866b >>> 2020-06-27T07:37:29,885 | INFO | Blueprint Event Dispatcher: 1 | >>> AppInfoParser | 195 - >>> org.apache.servicemix.bundles.kafka-clients - 2.4.1.1 | Kafka startTimeMs: >>> 1593243449884 >>> 2020-06-27T07:37:29,888 | INFO | Blueprint Event Dispatcher: 1 | >>> InternalRouteStartupManager | 95 - org.apache.camel.camel-base - 3.4.0 >>> | Route: route9 started and consuming from: kafka://casisp >>> 2020-06-27T07:37:29,890 | INFO | Blueprint Event Dispatcher: 1 | >>> AbstractCamelContext | 95 - org.apache.camel.camel-base - 3.4.0 >>> | Total 1 routes, of which 1 are started >>> 2020-06-27T07:37:29,892 | INFO | Blueprint Event Dispatcher: 1 | >>> AbstractCamelContext | 95 - org.apache.camel.camel-base - 3.4.0 >>> | Apache Camel 3.4.0 (camel-9) started in 0.374 seconds >>> 2020-06-27T07:37:29,893 | INFO | Camel (camel-9) thread #21 - >>> KafkaConsumer[casisp] | KafkaConsumer | 120 - >>> org.apache.camel.camel-kafka - 3.4.0 | Subscribing casisp-Thread 0 to topic >>> casisp >>> 2020-06-27T07:37:29,900 | INFO | Camel (camel-9) thread #21 - >>> KafkaConsumer[casisp] | KafkaConsumer | 195 - >>> org.apache.servicemix.bundles.kafka-clients - 2.4.1.1 | [Consumer >>> clientId=consumer-689cc1d7-fcf7-4565-9056-ddd7218d261f-9, >>> groupId=689cc1d7-fcf7-4565-9056-ddd7218d261f] Subscribed to topic(s): casisp >>> 2020-06-27T07:37:29,918 | INFO | >>> fileinstall-/opt/apache-karaf-4.2.9/deploy | fileinstall >>> | 10 - org.apache.felix.fileinstall - 3.6.6 | Started bundle: >>> blueprint:file:/opt/apache-karaf-4.2.9/deploy/isp.route.TSTISP001.xml >>> 2020-06-27T07:37:30,096 | INFO | Camel (camel-9) thread #21 - >>> KafkaConsumer[casisp] | Metadata | 195 - >>> org.apache.servicemix.bundles.kafka-clients - 2.4.1.1 | [Consumer >>> clientId=consumer-689cc1d7-fcf7-4565-9056-ddd7218d261f-9, >>> groupId=689cc1d7-fcf7-4565-9056-ddd7218d261f] Cluster ID: >>> NIuHmfEiSOGS7Bm56dOngg >>> 2020-06-27T07:37:30,097 | INFO | Camel (camel-9) thread #21 - >>> KafkaConsumer[casisp] | AbstractCoordinator | 195 - >>> org.apache.servicemix.bundles.kafka-clients - 2.4.1.1 | [Consumer >>> clientId=consumer-689cc1d7-fcf7-4565-9056-ddd7218d261f-9, >>> groupId=689cc1d7-fcf7-4565-9056-ddd7218d261f] Discovered group coordinator >>> localhost:9092 (id: 2147482646 rack: null) >>> 2020-06-27T07:37:30,098 | INFO | Camel (camel-9) thread #21 - >>> KafkaConsumer[casisp] | AbstractCoordinator | 195 - >>> org.apache.servicemix.bundles.kafka-clients - 2.4.1.1 | [Consumer >>> clientId=consumer-689cc1d7-fcf7-4565-9056-ddd7218d261f-9, >>> groupId=689cc1d7-fcf7-4565-9056-ddd7218d261f] (Re-)joining group >>> 2020-06-27T07:37:30,100 | WARN | Camel (camel-9) thread #21 - >>> KafkaConsumer[casisp] | NetworkClient | 195 - >>> org.apache.servicemix.bundles.kafka-clients - 2.4.1.1 | [Consumer >>> clientId=consumer-689cc1d7-fcf7-4565-9056-ddd7218d261f-9, >>> groupId=689cc1d7-fcf7-4565-9056-ddd7218d261f] Connection to node 2147482646 >>> (localhost/127.0.0.1:9092) could not be established. Broker may not be >>> available. >>> 2020-06-27T07:37:30,100 | INFO | Camel (camel-9) thread #21 - >>> KafkaConsumer[casisp] | AbstractCoordinator | 195 - >>> org.apache.servicemix.bundles.kafka-clients - 2.4.1.1 | [Consumer >>> clientId=consumer-689cc1d7-fcf7-4565-9056-ddd7218d261f-9, >>> groupId=689cc1d7-fcf7-4565-9056-ddd7218d261f] Group coordinator >>> localhost:9092 (id: 2147482646 rack: null) is unavailable or invalid, will >>> attempt rediscovery >>> 2020-06-27T07:37:30,202 | WARN | Camel (camel-9) thread #21 - >>> KafkaConsumer[casisp] | NetworkClient | 195 - >>> org.apache.servicemix.bundles.kafka-clients - 2.4.1.1 | [Consumer >>> clientId=consumer-689cc1d7-fcf7-4565-9056-ddd7218d261f-9, >>> groupId=689cc1d7-fcf7-4565-9056-ddd7218d261f] Connection to node 1001 >>> (localhost/127.0.0.1:9092) could not be established. Broker may not be >>> available.
