the FLIP-238 to be accepted?
>
> -Original Message-
> From: Qingsheng Ren
> Sent: jueves, 9 de junio de 2022 12:16
> To: Sanabria, Carlos
> Cc: user
> Subject: Re: [External] Re: Source vs SourceFunction and testing
>
> Hi Carlos,
>
> FLIP-238 [1] is proposing
?
-Original Message-
From: Qingsheng Ren
Sent: jueves, 9 de junio de 2022 12:16
To: Sanabria, Carlos
Cc: user
Subject: Re: [External] Re: Source vs SourceFunction and testing
Hi Carlos,
FLIP-238 [1] is proposing a FLIP-27-based data generator source and I think
this is what you are looking
t is planned to implement the FromElementsSource we'd rather prefer to wait
> for it.
>
> Thanks!
> Carlos
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/testing/#junit-rule-miniclusterwithclientresource
>
> -----Original Mes
2022 12:10
To: Piotr Domagalski
Cc: user@flink.apache.org
Subject: [External] Re: Source vs SourceFunction and testing
This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links
and attachments.
Glad to see you have resolved the issue!
If you want to learn more about the S
Glad to see you have resolved the issue!
If you want to learn more about the Source API, the Flink document [1] has a
detailed description about it. The original proposal FLIP-27 [2] is also a good
reference.
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/
Thank you Qingsheng, this context helps a lot!
And once again thank you all for being such a helpful community!
P.S. I actually struggled for a bit trying to understand why my refactored
solution which accepts DataStream<> wouldn't work ("no operators defined in
the streaming topology"). Turns ou
Hi Piotr,
I’d like to share my understanding about this. Source and SourceFunction are
both interfaces to data sources. SourceFunction was designed and introduced
earlier and as the project evolved, many shortcomings emerged. Therefore, the
community re-designed the source interface and introdu
Hi Piotr,
Yes, that should work (using DataStream as the common result of both
source creation options)
— Ken
> On May 24, 2022, at 12:19 PM, Piotr Domagalski wrote:
>
> Hi Ken,
>
> Thanks Ken. I guess the problem I had was, as a complete newbie to Flink,
> navigating the type system and be
Hi Ken,
Thanks Ken. I guess the problem I had was, as a complete newbie to Flink,
navigating the type system and being still confused about differences
between Source, SourceFunction, DataStream, DataStreamOperator, etc.
I think the DataStream<> type is what I'm looking for? That is, then I can
u
Hi Piotr,
The way I handle this is via a workflow class that uses a builder approach to
specifying inputs, outputs, and any other configuration settings.
The inputs are typically DataStream.
This way I can separate out the Kafka inputs, and use testing sources that give
me very precise control
Depending on the kind of testing you're hoping to do you may want to
look into https://github.com/mguenther/kafka-junit. For example,
you're looking for some job level smoke tests that just answer the
question "Is everything wired up correctly?" Personally, I like how
this approach doesn't require
Hi,
I'm wondering: what ithe recommended way to structure the job which one
would like to test later on with `MiniCluster`.
I've looked at the flink-training repository examples [1] and they tend to
expose the main job as a class that accepts a `SourceFunction` and a
`SinkFunction`, which make se
12 matches
Mail list logo
