Re: Structured streaming flatMapGroupWithState results out of order messages when reading from Kafka

dentical problem. Here’s what I came up with: > > https://github.com/ubiquibit-inc/sensor-failure > > > On Tue, Apr 9, 2019 at 04:37 Akila Wajirasena > wrote: > >> Hi >> >> I have a Kafka topic which is already loaded with data. I use a stateful >> structu

Structured streaming flatMapGroupWithState results out of order messages when reading from Kafka

Hi I have a Kafka topic which is already loaded with data. I use a stateful structured streaming pipeline using flatMapGroupWithState to consume the data in kafka in a streaming manner. However when I set shuffle partition count > 1 I get some out of order messages in to each of my GroupState. I

Zookeeper and Spark deployment for standby master

Hi, Is it necessary to deploy Apache Zookeeper and Spark in separate set of hosts when using standby masters? What will be the consequences of running Zookeeper quorum in the same physical machines that runs the Spark cluster.