Thanks Jamie for your thoughtful inputs!
Regarding the necessity of supporting updates, I'd like to add a scenario
where the base table for the 'view' created by the user does not entirely
come from a database(under the relational model).
For example, in a specific scenario: personalized user
>
> In the SQL standard regarding the definition of View, there are the
> following restrictions:
1. Partitioned view is not supported.
I agree that partitioned "views" don't really make sense since there is no
storage to partition. However, a bit of a search reveals that partitioned
FLIP-282 [1] has also introduced Update & Delete API for modifying table.
1.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
Best,
Ron
Ron liu 于2024年4月12日周五 19:49写道：
> Hi, jgrier
>
> Thanks for your insightful input.
>
> First of all, very much agree with you that
Hi, jgrier
Thanks for your insightful input.
First of all, very much agree with you that it is a right direction that we
should strive towards making Flink SQL more user-friendly, including
simplifying the job execution parameters, execution modes, data processing
pipeline definitions and
Sorry for coming very late to this thread. I have not contributed much to
Flink publicly for quite some time but I have been involved with Flink,
daily, for years now and I'm keenly interested in where we take Flink SQL
going forward.
Thanks for the proposal!! I think it's definitely a step in
Hi all,
I thought I'd cast my vote as well to give extra data:
1. Materialized Table
2. Materialized View (generally speaking I am not too concerned with
using a View here, but since there are concerns around updating a view I
put it second)
I think what is suggested in this FLIP is
I have been following up on the discussion, it's a great FLIP to further
unify stream and batch ETL pipelines. Thanks for the proposal!
Here is my ranking:
1. Materialized Table -> "The table materializes the results of a query
that you specify", this can reflect what we want and doesn't
Thanks for the proposal. I like the FLIP.
My ranking:
1. Refresh(ing) / Live Table -> easy to understand and implies the dynamic
characteristic
2. Derived Table -> easy to understand.
3. Materialized Table -> sounds like just a table with physical data stored
somewhere.
4. Materialized View
Thanks Ron and Timo for your proposal!
Here is my ranking:
1. Derived table -> extend the persistent semantics of derived table in SQL
standard, with a strong association with query, and has industry
precedents
such as Google Looker.
2. Live Table -> an alternative for 'dynamic table'
Hi, Dev
My rankings are:
1. Derived Table
2. Materialized Table
3. Live Table
4. Materialized View
Best,
Ron
Ron liu 于2024年4月9日周二 20:07写道：
> Hi, Dev
>
> After several rounds of discussion, there is currently no consensus on the
> name of the new concept. Timo has proposed that we decide
Hi, Dev
After several rounds of discussion, there is currently no consensus on the
name of the new concept. Timo has proposed that we decide the name through
a vote. This is a good solution when there is no clear preference, so we
will adopt this approach.
Regarding the name of the new concept,
t;>>>>>>>>>>>>>>> suggestions
>> >>>>>>>>>>>>>>>>>>>>> in the thread ) and address the holistic
>> >>> changes in a separate
>> >>>>>>>>&g
gt; >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>&g
Z-j73zexZBXu$
[3]
https://urldefense.com/v3/__https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j735ghqpxk$
Best
Yun Tang
____________________
From: Lincoln Lee <
lincol
Hi, Dev
This is a summary letter. After several rounds of discussion, there is a
strong consensus about the FLIP proposal and the issues it aims to address.
The current point of disagreement is the naming of the new concept. I have
summarized the candidates as follows:
1. Derived Table (Inspired
;
> > > > > > > > > > > > > >> > > >>>>>> This explanation makes sense to me.
> But the docs also say "A
> > > > > > > > > > > >> > > >>>> continuous
> > > > > > &g
Hello everybody!
Thanks for the FLIP as it looks amazing (and I think the prove is this deep
discussion it is provoking :))
I have a couple of comments to add to this:
Even though I get the reason why you rejected MATERIALIZED VIEW, I still like
it a lot, and I would like to provide pointers
> >> allow
> >> > > >>>> data
> >> > > >>>>>> modifications. This way we are only extending the standard
> and
> >> > don't
> >> > > >>>>>> introduce ne
gt;> > > >>>>>>
>> > > >>>>>> It's good that we called the concept STATEMENT SET. This
>> allows us
>> > > to
>> > > >>>>>> defined CREATE TABLE within. Even if it might look a bit
>>
> > > >>>>>>
> > > >>>>>> On 21.03.24 04:14, Feng Jin wrote:
> > > >>>>>>> Hi Ron and Lincoln
> > > >>>>>>>
> > > >>>>>>> Thanks for driving this discussion
t; > >>>> wonder
> > >>>>>> if
> > >>>>>>> we can provide a better way to help users rebuild it without
> > >>>> affecting
> > >>>>>>> downstream OLAP queries.
> > >>>&g
is mean that if a system needs to
> >>>> adapt
> >>>>> to
> >>>>>>> Dynamic Tables, it also needs to store Flink's job information in
> >>>> the
> >>>>>>> corresponding system?
> >>>>&g
he+DataStream+API
[2] https://docs.snowflake.com/en/user-guide/dynamic-tables-about
[3]
https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh
Best
Yun Tang
____________
From: Lincoln Lee
Sent: Thursday, March 14, 2024 14:35
To: dev@flink.apache.org
Subject:
3A+Batch+execution+for+the+DataStream+API
[2] https://docs.snowflake.com/en/user-guide/dynamic-tables-about
[3]
https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh
Best
Yun Tang
____________
From: Lincoln Lee
Sent: Thursday, March 14, 2024 14:35
To: dev@fl
the
>> > > >> end-to-end freshness of multiple
>> > > >> cascaded dynamic tables, he can manually split them for now. Of
>> > > >> course, how to let multiple cascaded
>> > > >> or dependent dynamic tables complet
, combined with the
> > > >> catalog and lineage, theoretically,
> > > >> users can also finish the cascading refresh.
> > > >>
> > > >>
> > > >> Best,
> > > >> Lincoln Lee
> > > >>
> > >
> > >>>
> > >>> From my point of view, instead of the work of unifying streaming and
> > >> batch
> > >>> in DataStream API [1], this FLIP actually could make users benefit
> from
> > >> one
> > >>> engine
t;>> dynamic tables [2], we still lack an incremental refresh mode to make
> the
> >>> ETL near real-time with a much cheaper computation cost. However, I
> think
> >>> this could be done under the current design by introducing another
> >> ref
+DataStream+API
[2] https://docs.snowflake.com/en/user-guide/dynamic-tables-about
[3] https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh
Best
Yun Tang
From: Lincoln Lee
Sent: Thursday, March 14, 2024 14:35
To: dev@flink.apache.org
Subject: Re: [DISCU
cal refreshes, we should consider the lineage
> in
> > the catalog or somewhere else.
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-134%3A+Batch+execution+for+the+DataStream+API
> > [2] https://docs.snowflake.com/en/user-guide/dynamic-tables-
esh
>
> Best
> Yun Tang
>
>
> ________
> From: Lincoln Lee
> Sent: Thursday, March 14, 2024 14:35
> To: dev@flink.apache.org
> Subject: Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for
> Simplifying Data Pipelines
>
> Hi Ji
arch 14, 2024 14:35
To: dev@flink.apache.org
Subject: Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying
Data Pipelines
Hi Jing,
Thanks for your attention to this flip! I'll try to answer the following
questions.
> 1. How to define query of dynamic table?
> Use flink sql
Hi Jing,
Thanks for your attention to this flip! I'll try to answer the following
questions.
> 1. How to define query of dynamic table?
> Use flink sql or introducing new syntax?
> If use flink sql, how to handle the difference in SQL between streaming
and
> batch processing?
> For example, a
Hi, Timo
Sorry for later response, thanks for your feedback.
Regarding your questions:
> Flink has introduced the concept of Dynamic Tables many years ago. How
does the term "Dynamic Table" fit into Flink's regular tables and also
how does it relate to Table API?
> I fear that adding the
Hi, Lincoln & Ron,
Thanks for the proposal.
I agree with the question raised by Timo.
Besides, I have some other questions.
1. How to define query of dynamic table?
Use flink sql or introducing new syntax?
If use flink sql, how to handle the difference in SQL between streaming and
batch
Hi Lincoln & Ron,
thanks for proposing this FLIP. I think a design similar to what you
propose has been in the heads of many people, however, I'm wondering how
this will fit into the bigger picture.
I haven't deeply reviewed the FLIP yet, but would like to ask some
initial questions:
Hi, Dev
Lincoln Lee and I would like to start a discussion about FLIP-435:
Introduce a New Dynamic Table for Simplifying Data Pipelines.
This FLIP is designed to simplify the development of data processing
pipelines. With Dynamic Tables with uniform SQL statements and
freshness, users can
37 matches
Mail list logo
