[GitHub] beam pull request #4034: [BEAM-2257] Ensure Kafka sink serializers are set.

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/4034 [BEAM-2257] Ensure Kafka sink serializers are set. Ensure that Kafka serializers are set. Please merge this after #3969 is merged. You can merge this pull request into a Git repository

[GitHub] beam pull request #4009: [BEAM-2720] Update kafka client version to 0.11.0.1

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/4009 [BEAM-2720] Update kafka client version to 0.11.0.1 This was supposed to be in earlier PR #3612, but it slipped through. +R: @kennknowles, @iemejia You can merge this pull request

[GitHub] beam pull request #4005: Add missing license header in ProducerSpEL.java.

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/4005 Add missing license header in ProducerSpEL.java. This is a quick follow up to BEAM-2720 to add missing Apache license header. +R: @kennknowles, thanks for pointing out. You can merge

[GitHub] beam pull request #3985: [BEAM-2979] Fix a race condition in getWatermark() ...

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/3985 [BEAM-2979] Fix a race condition in getWatermark() in KafkaIO. Two fixes : - Don't set curRecord to null before updating. If user deserializers throw, ok to keep curRecord pointing to old one

[GitHub] beam pull request #3758: [BEAM-2703] Fix min_timestamp used for KafkaIO wate...

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/3758 [BEAM-2703] Fix min_timestamp used for KafkaIO watermark. Use correct -ve infinity timestamp in KafkaIO as suggested in description of [BEAM-2703].(https://issues.apache.org/jira/browse/BEAM-2703

[GitHub] beam pull request #3612: Kafka exactly-once sink.

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/3612 Kafka exactly-once sink. Implementation of an exactly-once sink for Kafka, making use of transactions added in Kafka 0.11. This requires exact-once semantics for runners similar to Dataflow

[GitHub] beam pull request #3514: [BEAM-2534] Cherry-pick #3461 into 2.1.0.

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/3514 [BEAM-2534] Cherry-pick #3461 into 2.1.0. Cherry-pick #3461 : Handle offset gaps in Kafka messages. R: @jbonofre CC: @kennknowles. You can merge this pull request into a Git repository

[GitHub] beam pull request #3513: [BEAM-2551] Cherrypick #3492 to 2.1.0

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/3513 [BEAM-2551] Cherrypick #3492 to 2.1.0 Cherrypick #3492 to 2.1.0 +R @kennknowles. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rangadi/beam

[GitHub] beam pull request #3492: [BEAM-2551] KafkaIO reader blocks indefinitely in c...

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/3492 [BEAM-2551] KafkaIO reader blocks indefinitely in case of network issues If the KafaIO source reader on the worker can't reach the server, Kafka consumer blocks forever inside UnboundedReader.start

[GitHub] beam pull request #3461: [BEAM-2534] Handle offset gaps in Kafka messages.

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/3461 [BEAM-2534] Handle offset gaps in Kafka messages. KafkaIO logged a warning when there is a gap in offstes for messages. Kafka also supports log compaction for topics some of the messages

[GitHub] beam pull request #2986: [BEAM-2221] KafkaIO API clean up.

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/2986 [BEAM-2221] KafkaIO API clean up. - Removed withKeyCoder() and withValueCoder() methods. - Their meaning changed when KafkaIO added support for Deserializers. The coders can

[GitHub] beam pull request #2541: [BEAM-1978] Avoid repackaging bigtable classes in d...

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/2541 [BEAM-1978] Avoid repackaging bigtable classes in dataflow runner. Dataflow bundled jar does not need to repackage bigtable classes. This repackaging was probably never required in Apache Beam

[GitHub] beam pull request #2369: [BEAM-1837] Fix NPE in KafkaIO writer.

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/2369 [BEAM-1837] Fix NPE in KafkaIO writer. `KafkaIO.writer()...values()` does not require user to set key coder since the key always null. Validation passes, but it results in an NPE at runtime when

[GitHub] beam pull request #2364: [BEAM-1830] KafkaIO : Add withTopic() api that take...

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/2364 [BEAM-1830] KafkaIO : Add withTopic() api that takes single topic. Overwhelming uses of KafkaIO consume just one topic. It would be nice to have `withTopic(topic)` rather than always requiring

[GitHub] beam pull request #2009: [BEAM-958] Improve desired number of splits in Data...

GitHub user rangadi opened a pull request: https://github.com/apache/beam/pull/2009 [BEAM-958] Improve desired number of splits in Dataflow. Set desired number of splits for unbounded sources to 4 x (max)workers. I am not sure of the rationale for previous values, especially