+1.
I would suggest to take a step even further and see what users really need
to test/try/play with table API and Flink SQL. Besides this one, here're
some more sources and sinks that I have developed or used previously to
facilitate building Flink table/SQL pipelines.
1. random input data source
- should generate random data at a specified rate according to schema
- purposes
- test Flink pipeline and data can end up in external storage
correctly
- stress test Flink sink as well as tuning up external storage
2. print data sink
- should print data in row format in console
- purposes
- make it easier to test Flink SQL job e2e in IDE
- test Flink pipeline and ensure output data format/value is
correct
3. no output data sink
- just swallow output data without doing anything
- purpose
- evaluate and tune performance of Flink source and the whole
pipeline. Users' don't need to worry about sink back pressure
These may be taken into consideration all together as an effort to lower
the threshold of running Flink SQL/table API, and facilitate users' daily
work.
Cheers,
Bowen
On Thu, Mar 19, 2020 at 10:32 PM Jingsong Li <jingsongl...@gmail.com> wrote:
> Hi all,
>
> I heard some users complain that table is difficult to test. Now with SQL
> client, users are more and more inclined to use it to test rather than
> program.
> The most common example is Kafka source. If users need to test their SQL
> output and checkpoint, they need to:
>
> - 1.Launch a Kafka standalone, create a Kafka topic .
> - 2.Write a program, mock input records, and produce records to Kafka
> topic.
> - 3.Then test in Flink.
>
> The step 1 and 2 are annoying, although this test is E2E.
>
> Then I found StatefulSequenceSource, it is very good because it has deal
> with checkpoint things, so it is very good to checkpoint mechanism.Usually,
> users are turned on checkpoint in production.
>
> With computed columns, user are easy to create a sequence source DDL same
> to Kafka DDL. Then they can test inside Flink, don't need launch other
> things.
>
> Have you consider this? What do you think?
>
> CC: @Aljoscha Krettek <aljos...@apache.org> the author
> of StatefulSequenceSource.
>
> Best,
> Jingsong Lee
>
