KafkaIOIT can also use testcontainers to create a fake Kafka service [1].
In theory we could do something similar to test PubSubIO by using the
provided emulator [2], but currently we only test against the production
pubsub service. There's a useful TestPubsub [3] rule which can be used to
create a
> High-level: ensure you have gradle cache enabled so only the first build
is slow. If you encounter nondeterministic or noncached targets upstream of
the module you are editing, that's worth discussing and probably fixing.
I do have caching enabled (I do not rebuild non-gcp targets every time)
>
I'd like to add tests verifying that Beam is compatible with both Avro 1.8 and
1.9 similar to what has been done to Hadoop and Kafka.
Probably all Avro dependencies would have to be changed from compile to
provided - won't it be problematic for users? They will be broken after the
update unless
Hi Jithin,
Updating Beam to 2.25.0 and using "withIsDeleteFn" works as expected.
Thanks a lot for the detailed answer!
Best,
Arif
On Tue, Dec 1, 2020 at 11:30 AM JITHIN SUKUMAR
wrote:
> Hi Arif Alili,
>
> Deleting documents using ElasticsearchIO [1] was included in Apache Beam
> since version
High-level: ensure you have gradle cache enabled so only the first build is
slow. If you encounter nondeterministic or noncached targets upstream of
the module you are editing, that's worth discussing and probably fixing.
That's going to be a separate issue from wanting to build a single part of
t
Hi Arif Alili,
Deleting documents using ElasticsearchIO [1] was included in Apache Beam
since version 2.25.0 [2]. You can check out the javadoc [3] or some example
implementations [4].
References:
[1]: https://issues.apache.org/jira/browse/BEAM-5757
[2]: https://beam.apache.org/blog/beam-2.25.0/
Hi all,
I am writing to Elasticsearch using Beam (Google Dataflow) class
ElasticsearchIO. Creating indexes and writing document goes well, however,
I am struggling to find a way to delete Elasticsearch documents, looking at
this documentation
https://beam.apache.org/releases/javadoc/2.5.0/org/apac
