Hi everyone, New user to Kafka, and am psyched to get it working in our environment! I'm on 0.11.0.0, and using spring-kafka 2.0.0.rc1. I'm struggling to get things working in our unit/integration tests.
I don't want to overload everyone with the context of the issues, but this is the stacktrace I'm seeing (from the tail end of the Maven build): https://gist.github.com/emetsger/05ef47ce15ca54b2735a65627992a2d0 The full build output is here: https://goo.gl/pwM8Lw It appears that the AdminClient times out trying to create the topic for the unit test: 13:54:40.964 [-client-thread | adminclient-1] DEBUG [ KafkaAdminClient$Call] - Call(callName=createTopics, deadlineMs=1507053280963) timed out at 1507053280964 after 1 attempt(s) java.lang.Exception: TimeoutException: Timed out waiting for a node assignment. at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:475) at org.apache.kafka.clients.admin.KafkaAdminClient$TimeoutProcessor.handleTimeouts(KafkaAdminClient.java:591) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.timeoutNewCalls(KafkaAdminClient.java:663) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:908) at java.lang.Thread.run(Thread.java:748) The unit test uses the maven-buildhelper-plugin to reserve a port for the Kakfa broker, in this case, port 55383. If you look at the gist, I've added the properties used by the AdminClient and the Broker, and it seems that they are configured properly (that is, the AC is configured to talk to 127.0.0.1:55383, which is where the broker is listening). Another relevant note is that I'm using spring-kafka, and Spring is managing all the beans related to Kafka: the producer, consumer, and broker. The broker is being started using the @EmbeddedKafka annotation, which will create an embedded instance of Zookeeper, and an EmbeddedKafka instance: @TestPropertySource(locations = { "classpath:/rmapcore.properties", "classpath:/kafka-broker.properties" }) @EmbeddedKafka(topics = { "rmap-event-topic" }, brokerProperties = { "log.dir=${kafka.broker.logs-dir}", "port=${kafka.broker.port}" }) Kafka should be automatically creating the topic (`rmap-event-topic` in this case), and it looks like the AdminClient tries, but ultimately fails, timing out trying to connect to the broker: 15:21:34.977 [ main] DEBUG [KafkaAdminClient$AdminClientRu] - adminclient-1: queueing Call(callName=createTopics, deadlineMs=1507058614976) with a timeout 120000 ms from now. ... 15:21:35.047 [-client-thread | adminclient-1] TRACE [ NetworkClient] - Found least loaded node 127.0.0.1:58230 (id: -1 rack: null) 15:21:35.047 [-client-thread | adminclient-1] DEBUG [NetworkClient$DefaultMetadataU] - Initialize connection to node -1 for sending metadata request 15:21:35.047 [-client-thread | adminclient-1] DEBUG [ NetworkClient] - Initiating connection to node -1 at 127.0.0.1:58230. 15:21:35.048 [-client-thread | adminclient-1] DEBUG [ Metrics] - Added sensor with name node--1.bytes-sent 15:21:35.048 [-client-thread | adminclient-1] DEBUG [ Metrics] - Added sensor with name node--1.bytes-received 15:21:35.049 [-client-thread | adminclient-1] DEBUG [ Metrics] - Added sensor with name node--1.latency 15:21:35.055 [-client-thread | adminclient-1] DEBUG [ Selector] - Connection with /127.0.0.1 disconnected java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50) at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:95) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:359) at org.apache.kafka.common.network.Selector.poll(Selector.java:326) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:432) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:927) at java.lang.Thread.run(Thread.java:748) Polling continues, and ultimately times out: 15:23:34.981 [-client-thread | adminclient-1] DEBUG [ KafkaAdminClient$Call] - Call(callName=createTopics, deadlineMs=1507058614976) timed out at 1507058614981 after 1 attempt(s) java.lang.Exception: TimeoutException: Timed out waiting for a node assignment. at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:475) at org.apache.kafka.clients.admin.KafkaAdminClient$TimeoutProcessor.handleTimeouts(KafkaAdminClient.java:591) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.timeoutNewCalls(KafkaAdminClient.java:663) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:908) at java.lang.Thread.run(Thread.java:748) Any help or advice is appreciated! Thanks, Elliot Feel free to follow the steps to reproduce locally, or just look at the full build output here https://goo.gl/pwM8Lw 1. Because I'm using Spring release candidates in the build, you'll need to add two repositories to your ~/.m2/settings.xml: <repositories> <repository> <id>spring-libs-release</id> <name>Spring Releases</name> <url>https://repo.spring.io/libs-release</url> <snapshots> <enabled>false</enabled> </snapshots> </repository> <repository> <id>spring-libs-snap</id> <name>Spring Snapshots</name> <url>https://repo.spring.io/libs-snapshot</url> <snapshots> <enabled>true</enabled> </snapshots> </repository> </repositories> 2. Check out https://github.com/emetsger/spring-kafka, branch property-placeholder-resolution. Run `./gradlew clean install` from inside the spring-kafka directory. - this is the same as spring-kafka 2.0.0.rc1, but allows for the use of property placeholders in the `brokerProperties` attribute of @EmbeddedKafka (PR pending) 3. Check out https://github.com/emetsger/rmap.git, branch kakfa-adminclient-timeout in a new directory. 4. Run mvn clean install -Dcheckstyle.skip -Djavadoc.skip -DskipTests from the RMap directory. The build will fail at the "Solr indexing" (rmap-indexing-solr) module. That's OK, ignore it. This is just to populate your local Maven repository with the necessary artifacts (and `rmap-indexing-solr` isn't necessary for reproducing this issue). 5. Now that all the dependencies are installed, cd into the `core` directory and run an example unit test to replicate the problem: `mvn test -Dlogback.configurationFile=src/test/resources/logback-test.xml -Dtest=ORMapEventCreationTest`
