On Wed, May 4, 2016 at 12:57 PM, amir bahmanyari <[email protected]> wrote:
> The root cause ended up to be the Kafka version in my lab. > Kafka server must be version 9.0+ for the KafkaIO call in the Beam app > code to populate the PCollection object . > Please let us know if there is something KafkaIO could do for you to have caught this earlier. I think older Kafka versions (especially 0.8.x) are pretty widely used and many unsuspecting users will be affected. I will check if there is a good way to detect this early in Kafka consumer. > I know Unbounded is the ultimate way to achieve a true real-time streaming. > Given that Unbounded is not available at the moment, is there a work > around that makes every single record available immediately to the app? > Can use use Google dataflow runner? It fully supports UnboundedSource in KafkaIO. Raghu.
