Hi,
yes, that is correct. The failure mapper is there to cause a failover event for
which we can then check i) that exactly-once or at-least-once is not violated,
depending on the expected semantics and ii) that the restore works at all ;-).
You might be able to reuse org.apache.flink.streaming
Hi Stefan,
Thanks for the info. So if I understand correctly, the pipeline you had in
mind is:
Consumer -> Map -> Producer
What do you expect as outcome of the mapper failure? That no records are
lost but some possibly duplicated in the sink?
Regarding the abstraction, I will see what I can do
Hi,
I was also just planning to work on it before Stephan contacted Thomas to ask
about this test.
Thomas, you are right about the structure, the test should also go into the
`run-nightly-tests.sh`. What I was planning to do is a simple job that consists
of a Kinesis consumer, a mapper that fa
Hi Thomas,
the community is really interested in adding an end-to-end test for the
Kinesis connector (producer as well as consumer). Thus, it would be really
helpful if you could contribute your work you've already done.
Using Kinesalite sounds good to me and you're right and that we assume that
Hi Thomas,
I think Stefan Richter is also working on the Kinesis end-to-end test, and
seems to be planning to implement it against a real Kinesis service instead
of Kinesalite.
Perhaps efforts should be synced here.
Cheers,
Gordon
On Thu, Nov 8, 2018 at 1:38 PM Thomas Weise wrote:
> Hi,
>
> I
Hi,
I'm planning to add an end-to-end test for the Kinesis consumer. We have
done something similar at Lyft, using Kinesalite, which can be run as
Docker container.
I see that some tests already make use of Docker, so we can assume it to be
present in the target environment(s)?
I also found the
