Maybe to give some background about Dawid's latest email:
Kurt raised some good points regarding the conversion of data types at
the boundaries of the API and SPI. After that, Dawid and I had a long
discussion of how users can define those boundaries in a nicer way. The
outcome of this discussion was similar to Blink's current distinction
between InternalTypes and ExternalTypes. I updated the document with a
improved structure of DataTypes (for users, API, SPI with conversion
information) and LogicalTypes (used internally and close to standard SQL
types).
Thanks for the feedback so far,
Timo
Am 28.03.19 um 11:18 schrieb Dawid Wysakowicz:
Another big +1 from my side. Thank you Timo for preparing the document!
I really look forward for this to have a standardized way of type
handling. This should solve loads of problems. I really like the
separation of logical type from its physical representation, I think we
should aim to introduce that and keep it separated.
Best,
Dawid
On 28/03/2019 08:51, Kurt Young wrote:
Big +1 to this! I left some comments in google doc.
Best,
Kurt
On Wed, Mar 27, 2019 at 11:32 PM Timo Walther <[email protected]> wrote:
Hi everyone,
some of you might have already read FLIP-32 [1] where we've described an
approximate roadmap of how to handle the big Blink SQL contribution and
how we can make the Table & SQL API equally important to the existing
DataStream API.
As mentioned there (Advance the API and Unblock New Features, Item 1),
the rework of the Table/SQL type system is a crucial step for unblocking
future contributions. In particular, Flink's current type system has
many shortcomings which make an integration with other systems (such as
Hive), DDL statements, and a unified API for Java/Scala difficult. We
propose a new type system that is closer to the SQL standard, integrates
better with other SQL vendors, and solves most of the type-related
issues we had in the past.
The design document for FLIP-37 can be found here:
https://docs.google.com/document/d/1a9HUb6OaBIoj9IRfbILcMFPrOL7ALeZ3rVI66dvA2_U/edit?usp=sharing
I'm looking forward to your feedback.
Thanks,
Timo
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
