Ok, I've made some progress but I'm seeing crashes now. I can easily populate a set number of messages into a topic. No issues, and offsets are properly set.
My consume code with 0.11-M1: count = 0 val consumerSettings = ConsumerSettings(as, new ByteArrayDeserializer, new StringDeserializer, Set("lowercaseStrings")) .withBootstrapServers("192.168.99.100:9092") .withGroupId("group1") .withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest") val graph = RunnableGraph.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[akka.NotUsed] => import GraphDSL.Implicits._ type In = CommittableMessage[Array[Byte],String] val src = Consumer.committableSource(consumerSettings.withClientId("client1")) val commit = Flow[In].map{ msg => msg.committableOffset.commit() } val show = Flow[In].map{ i => count += 1 if(i.value=="done") println(s"Time ($count): "+(System.currentTimeMillis() - now)) i } src ~> show ~> commit ~> Sink.ignore ClosedShape }) now = System.currentTimeMillis() graph.run() Ok, timings aside for a moment--I still have concerns there, but let's focus on the crashing. I restart Kafka each run so there's no left-over messages/offsets from previous run. I'm using a single topic with just one partition for this test. If I run this for a small number of messages pre-populated, maybe up to 10000, it works. If I run it with 100,000 I get this error: [ReactiveKafka-akka.kafka.default-dispatcher-11] ERROR akka.kafka.internal.CommittableConsumerStage$$anon$1 - Error in poll java.lang.IllegalArgumentException: requirement failed at scala.Predef$.require(Predef.scala:207) at akka.kafka.internal.ConsumerStageLogic.handleResult$1(ConsumerStage.scala:372) at akka.kafka.internal.ConsumerStageLogic.poll(ConsumerStage.scala:384) at akka.kafka.internal.CommittableConsumerStage$$anon$1.poll(ConsumerStage.scala:145) at akka.kafka.internal.CommittableConsumerStage$$anon$1.commitAsync(ConsumerStage.scala:257) at akka.kafka.internal.CommittableConsumerStage$$anon$1.akka$kafka$internal$CommittableConsumerStage$$anon$$commitSingleInternal(ConsumerStage.scala:223) at akka.kafka.internal.CommittableConsumerStage$$anon$1$$anonfun$6.apply(ConsumerStage.scala:131) at akka.kafka.internal.CommittableConsumerStage$$anon$1$$anonfun$6.apply(ConsumerStage.scala:130) at akka.stream.impl.fusing.GraphInterpreter.runAsyncInput(GraphInterpreter.scala:569) at akka.stream.impl.fusing.GraphInterpreterShell.receive(ActorGraphInterpreter.scala:417) at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:593) at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:604) at akka.actor.Actor$class.aroundReceive(Actor.scala:482) at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:519) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.