Hi Timo,
Thanks for your answers, no problem with the delay, I was in vacation
too last week :)
My comments are inline
Best,
Etienne
On 07/07/2021 16:48, Timo Walther wrote:
Hi Etienne,
sorry for the late reply due to my vacation last week.
Regarding: "support of aggregations in batch mode for DataStream API
[...] is there a plan to solve it before the actual drop of DataSet API"
Just to clarify it again: we will not drop the DataSet API any time
soon. So users will have enough time to update their pipelines. There
are a couple of features missing to fully switch to DataStream API in
batch mode. Thanks for opening an issue, this is helpful for us to
gradually remove those barriers. They don't need to have a "Blocker"
priority in JIRA for now.
Ok I thought the drop was sooner, no problem then.
But aggregations is a good example where we should discuss if it would
be easier to simply switch to Table API for that. Table API has a lot
of aggregation optimizations and can work on binary data. Also joins
should be easier in Table API. DataStream API can be a very low-level
API in the near future and most use cases (esp. the batch ones) should
be possible in Table API.
Yes sure. As a matter of fact, my point was to use low level DataStream
API in a benchmark to compare with Table API but I guess it is not a
common user behavior.
Regarding: "Is it needed to port these Avro enhancements to new
DataStream connectors (add a new equivalent of
ParquetColumnarRowInputFormat but for Avro)"
We should definitely not loose functionality. The same functionality
should be present in the new connectors. The questions is rather
whether we need to offer a DataStream API connector or if a Table API
connector would be nicer to use (also nicely integrated with catalogs).
So a user can use a simple CREATE TABLE statement to configure the
connector; an easier abstraction is almost not possible. With
`tableEnv.toDataStream(table)` you can then continue in DataStream API
if there is still a need for it.
Yes I agree, there is no easier connector setup than CREATE TABLE, and
with tableEnv.toDataStream(table) if one would want to stay with
DataStream (in my bench for example) it is still possible. And by the
way the doc for parquet (1) for example only mentions using Table API
connector.
So I guess the table connector would be nicer for the user indeed. But
if we use tableEnv.toDataStream(table), we would be able to produce only
usual types like Row, Tuples or Pojos, we still need to add Avro support
right ?
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/formats/parquet/
Regarding: "there are parquet bugs still open on deprecated parquet
connector"
Yes, bugs should still be fixed in 1.13.
OK
Regarading: "I've been doing TPCDS benchmarks with Flink lately"
Great to hear that :-)
And congrats again on blink performances !
Did you also see the recent discussion? A TPC-DS benchmark can further
be improved by providing statistics. Maybe this is helpful to you:
https://lists.apache.org/thread.html/ra383c23f230ab8e7fa16ec64b4f277c267d6358d55cc8a0edc77bb63%40%3Cuser.flink.apache.org%3E
No, I missed that thread, thanks for the pointer, I'll read it and
comment if I have something to add.
I will prepare a blog post shortly.
Good to hear :)
Regards,
Timo
On 06.07.21 15:05, Etienne Chauchot wrote:
Hi all,
Any comments ?
cheers,
Etienne
On 25/06/2021 15:09, Etienne Chauchot wrote:
Hi everyone,
@Timo, my comments are inline for steps 2, 4 and 5, please tell me
what you think.
Best
Etienne
On 23/06/2021 15:27, Chesnay Schepler wrote:
If we want to publicize this plan more shouldn't we have a rough
timeline for when 2.0 is on the table?
On 6/23/2021 2:44 PM, Stephan Ewen wrote:
Thanks for writing this up, this also reflects my understanding.
I think a blog post would be nice, ideally with an explicit call for
feedback so we learn about user concerns.
A blog post has a lot more reach than an ML thread.
Best,
Stephan
On Wed, Jun 23, 2021 at 12:23 PM Timo Walther <twal...@apache.org>
wrote:
Hi everyone,
I'm sending this email to make sure everyone is on the same page
about
slowly deprecating the DataSet API.
There have been a few thoughts mentioned in presentations, offline
discussions, and JIRA issues. However, I have observed that there
are
still some concerns or different opinions on what steps are
necessary to
implement this change.
Let me summarize some of the steps and assumpations and let's have a
discussion about it:
Step 1: Introduce a batch mode for Table API (FLIP-32)
[DONE in 1.9]
Step 2: Introduce a batch mode for DataStream API (FLIP-134)
[DONE in 1.12]
I've been using DataSet API and I tested migrating to DataStream +
batch mode.
I opened this (1) ticket regarding the support of aggregations in
batch mode for DataStream API. It seems that join operation (at
least) does not work in batch mode even though I managed to
implement a join using low level KeyedCoProcessFunction (thanks
Seth, for the pointer !).
=> Should it be considered a blocker ? Is there a plan to solve it
before the actual drop of DataSet API ? Maybe in step 6 ?
[1] https://issues.apache.org/jira/browse/FLINK-22587
Step 3: Soft deprecate DataSet API (FLIP-131)
[DONE in 1.12]
We updated the documentation recently to make this deprecation
even more
visible. There is a dedicated `(Legacy)` label right next to the
menu
item now.
We won't deprecate concrete classes of the API with a @Deprecated
annotation to avoid extensive warnings in logs until then.
Step 4: Drop the legacy SQL connectors and formats (FLINK-14437)
[DONE in 1.14]
We dropped code for ORC, Parque, and HBase formats that were only
used
by DataSet API users. The removed classes had no documentation
and were
not annotated with one of our API stability annotations.
The old functionality should be available through the new sources
and
sinks for Table API and DataStream API. If not, we should bring them
into a shape that they can be a full replacement.
DataSet users are encouraged to either upgrade the API or use Flink
1.13. Users can either just stay at Flink 1.13 or copy only the
format's
code to a newer Flink version. We aim to keep the core interfaces
(i.e.
InputFormat and OutputFormat) stable until the next major version.
We will maintain/allow important contributions to dropped
connectors in
1.13. So 1.13 could be considered as kind of a DataSet API LTS
release.
I added several bug fixes and enhancements (avro support, automatic
schema etc...) to parquet DataSet connector. After discussing with
Jingsong and Arvid, we agreed to merge them to 1.13 in accordance to
the fact that 1.13 is a LTS release receiving maintenance changes as
you mentioned here.
=> Is it needed to port these Avro enhancements to new DataStream
connectors (add a new equivalent of ParquetColumnarRowInputFormat
but for Avro) ? IMHO opinion it is an important feature that the
users will need. So, if I understand the plan correctly, we have
until the release of 2.0 to implement it, right ?
=> Also there are parquet bugs still open on deprecated parquet
connector: https://issues.apache.org/jira/browse/FLINK-21520,
https://issues.apache.org/jira/browse/FLINK-21468, I think that the
same applies, we should fix them on 1.13 right ?
Step 5: Drop the legacy SQL planner (FLINK-14437)
[DONE in 1.14]
This included dropping support of DataSet API with SQL.
That is a major point ! I've been doing TPCDS benchmarks with Flink
lately by coding query3 with a DataSet pipeline, a DataStream
pipeline and a SQL pipeline. What I can tell is that when I migrated
from the legacy SQL planer to blink SQL planner, I got 2 major
improvements:
1. around 25% gain in run times on 1TB input dataset (even if memory
conf was slightly different between runs of the 2 planners)
2. global order support: with legacy planer based on DataSet, only
local partition ordering was supported. As a consequence, a SQL
query with an ORDER BY clause actually produced wrong results. With
blink planner based on DataStream, global order is supported and now
the query results are correct !
=> congrats to everyone involved in these big SQL improvements !
Step 6: Connect both Table and DataStream API in batch mode
(FLINK-20897)
[PLANNED in 1.14]
Step 7: Reach feature parity of Table API/DataStream API with
DataSet API
[PLANNED for 1.14++]
We need to identify blockers when migrating from DataSet API to
Table
API/DataStream API. Here we need to estabilish a good feedback
pipeline
to include DataSet users in the roadmap planning.
Step 7: Drop the Gelly library
No concrete plan yet. Latest would be the next major Flink
version aka
Flink 2.0.
Step 8: Drop DataSet API
Planned for the next major Flink version aka Flink 2.0.
Please let me know if this matches your thoughts. We can also
convert
this into a blog post or mention it in the next release notes.
Regards,
Timo
