Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements
Hi, Jark, thanks for explantations.
if there are no other questions, i will try to start a VOTE to address
current things in this FLIP.
We may open another dedicated FLIP to discuss INFORMATION_SCHEMA.
hi, guys, WDYT?
Best Regards,
Ran Tao
Jark Wu 于2023年2月25日周六 22:57写道：
> Hi Sergey,
>
> I think INFORMATION_SCHEMA is a very interesting idea, and I hope we can
> support it. However, it doesn't conflict with the idea of auxiliary
> statements. I can see different benefits of them. The information schema
> provides powerful and flexible capabilities but needs to learn the complex
> entity relationship[1]. The auxiliary SQL statements are super handy and
> can resolve most problems, but they offer limited features.
>
> I can see almost all the mature systems support both of them. I think it
> also makes sense to support both of them in Flink. Considering they
> are orthogonal and information schema requires more complex design and
> discussion, it deserves a separate FLIP. Sergey, are you willing to
> contribute this FLIP?
>
> Best,
> Jark
>
> [1]:
>
> https://docs.databricks.com/sql/language-manual/sql-ref-information-schema.html
>
>
> On Fri, 24 Feb 2023 at 22:43, Ran Tao wrote:
>
> > Thanks John.
> >
> > It seems that most people prefer the information_schema implementation.
> > information_schema does have more benefits (however, the show operation
> is
> > also an option and supplement).
> > Otherwise, the sql syntax and keywords may be changed frequently.
> > Of course, it will be more complicated than the extension of the show
> > operation.
> > It is necessary to design various tables in information_schema, which
> may
> > take a period of effort.
> >
> > I will try to design the information_schema and integrate it with flink.
> > This may be a relatively big feature for me. I hope you guys can give
> > comments and opinions later.
> > Thank you all.
> >
> > Best Regards,
> > Ran Tao
> >
> >
> > John Roesler 于2023年2月24日周五 21:53写道：
> >
> > > Hello Ran,
> > >
> > > Thanks for the FLIP!
> > >
> > > Do you mind if we revisit the topic of doing this by adding an
> > Information
> > > schema? The SHOW approach requires modifying the parser/language for
> > every
> > > gap we identify. On the flip side, an Information schema is much easier
> > to
> > > discover and remember how to use, and the ability to run queries on it
> is
> > > quite valuable for admins. It’s also better for Flink maintainers,
> > because
> > > the information tables’ schemas can be evolved over time just like
> > regular
> > > tables, whereas every change to a SHOW statement would be a breaking
> > > change.
> > >
> > > I understand that it may seem like a big effort, but we’re proposing
> > quite
> > > a big extension to the space of SHOW statement, so it seems appropriate
> > to
> > > take the opportunity and migrate to a better framework rather than
> > > incrementally building on (and tying us even more firmly to) the
> existing
> > > approach.
> > >
> > > Thanks for your consideration,
> > > John
> > >
> > > On Fri, Feb 24, 2023, at 05:58, Sergey Nuyanzin wrote:
> > > > thanks for explanation
> > > >
> > > >>But it's not clear to me what exactly
> > > >> you want to display? Is it the name of the plugin?
> > > >
> > > > I was thinking about name, type (source/sink) and may be version (not
> > > sure
> > > > if it's possible right now)
> > > >
> > > > On Fri, Feb 24, 2023 at 12:46 PM Ran Tao
> > wrote:
> > > >
> > > >> Hi, Sergey. thanks. first step we can support filtering for show
> > > operations
> > > >> in this FLIP try to align other engines.
> > > >> If we want to support describe other objects, of course we need to
> > > design
> > > >> how to get these metadatas or informations and printAsStyle. (So it
> > > maybe
> > > >> need another FLIP for more details).
> > > >>
> > > >> > Does it make sense to add support for connectors e.g. show
> > > >> > {sink|source|all} connectors?
> > > >> I think we can support it, currently flink do support some
> operations
> > > only
> > > >> for flink itself such as showJobs. But it's not clear to me what
> > exactly
> > > >> you want to display? Is it the name of the plugin?
> > > >> Just Like:
> > > >> Kafka
> > > >> Hudi
> > > >> Files
> > > >>
> > > >> Best Regards,
> > > >> Ran Tao
> > > >>
> > > >>
> > > >> Sergey Nuyanzin 于2023年2月24日周五 19:11写道：
> > > >>
> > > >> > Thanks for driving the FLIP
> > > >> >
> > > >> > I have a couple of questions
> > > >> > Am I right that INFORMATION_SCHEMA mentioned by Timo[1] is out of
> > > scope
> > > >> of
> > > >> > this FLIP?
> > > >> > I noticed there are some support of it in
> > > Spark[2]/Hive[3]/Snowflake[4]
> > > >> and
> > > >> > others
> > > >> >
> > > >> > Does it make sense to add support for connectors e.g. show
> > > >> > {sink|source|all} connectors?
> > > >> >
> > > >> > [1]
> > https://lists.apache.org/thread/2g108qlfwbhb56wnoc5qj0yq29dvq1vv
> > > >> > [2] https://issues.apache.org/jira/browse/SPARK-16452
> > > >> > [3]
[jira] [Created] (FLINK-31225) rocksdb max open file can lead to oom
xiaogang zhou created FLINK-31225: - Summary: rocksdb max open file can lead to oom Key: FLINK-31225 URL: https://issues.apache.org/jira/browse/FLINK-31225 Project: Flink Issue Type: Improvement Components: Runtime / State Backends Affects Versions: 1.16.1 Reporter: xiaogang zhou -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements
Hi Sergey,
I think INFORMATION_SCHEMA is a very interesting idea, and I hope we can
support it. However, it doesn't conflict with the idea of auxiliary
statements. I can see different benefits of them. The information schema
provides powerful and flexible capabilities but needs to learn the complex
entity relationship[1]. The auxiliary SQL statements are super handy and
can resolve most problems, but they offer limited features.
I can see almost all the mature systems support both of them. I think it
also makes sense to support both of them in Flink. Considering they
are orthogonal and information schema requires more complex design and
discussion, it deserves a separate FLIP. Sergey, are you willing to
contribute this FLIP?
Best,
Jark
[1]:
https://docs.databricks.com/sql/language-manual/sql-ref-information-schema.html
On Fri, 24 Feb 2023 at 22:43, Ran Tao wrote:
> Thanks John.
>
> It seems that most people prefer the information_schema implementation.
> information_schema does have more benefits (however, the show operation is
> also an option and supplement).
> Otherwise, the sql syntax and keywords may be changed frequently.
> Of course, it will be more complicated than the extension of the show
> operation.
> It is necessary to design various tables in information_schema, which may
> take a period of effort.
>
> I will try to design the information_schema and integrate it with flink.
> This may be a relatively big feature for me. I hope you guys can give
> comments and opinions later.
> Thank you all.
>
> Best Regards,
> Ran Tao
>
>
> John Roesler 于2023年2月24日周五 21:53写道：
>
> > Hello Ran,
> >
> > Thanks for the FLIP!
> >
> > Do you mind if we revisit the topic of doing this by adding an
> Information
> > schema? The SHOW approach requires modifying the parser/language for
> every
> > gap we identify. On the flip side, an Information schema is much easier
> to
> > discover and remember how to use, and the ability to run queries on it is
> > quite valuable for admins. It’s also better for Flink maintainers,
> because
> > the information tables’ schemas can be evolved over time just like
> regular
> > tables, whereas every change to a SHOW statement would be a breaking
> > change.
> >
> > I understand that it may seem like a big effort, but we’re proposing
> quite
> > a big extension to the space of SHOW statement, so it seems appropriate
> to
> > take the opportunity and migrate to a better framework rather than
> > incrementally building on (and tying us even more firmly to) the existing
> > approach.
> >
> > Thanks for your consideration,
> > John
> >
> > On Fri, Feb 24, 2023, at 05:58, Sergey Nuyanzin wrote:
> > > thanks for explanation
> > >
> > >>But it's not clear to me what exactly
> > >> you want to display? Is it the name of the plugin?
> > >
> > > I was thinking about name, type (source/sink) and may be version (not
> > sure
> > > if it's possible right now)
> > >
> > > On Fri, Feb 24, 2023 at 12:46 PM Ran Tao
> wrote:
> > >
> > >> Hi, Sergey. thanks. first step we can support filtering for show
> > operations
> > >> in this FLIP try to align other engines.
> > >> If we want to support describe other objects, of course we need to
> > design
> > >> how to get these metadatas or informations and printAsStyle. (So it
> > maybe
> > >> need another FLIP for more details).
> > >>
> > >> > Does it make sense to add support for connectors e.g. show
> > >> > {sink|source|all} connectors?
> > >> I think we can support it, currently flink do support some operations
> > only
> > >> for flink itself such as showJobs. But it's not clear to me what
> exactly
> > >> you want to display? Is it the name of the plugin?
> > >> Just Like:
> > >> Kafka
> > >> Hudi
> > >> Files
> > >>
> > >> Best Regards,
> > >> Ran Tao
> > >>
> > >>
> > >> Sergey Nuyanzin 于2023年2月24日周五 19:11写道：
> > >>
> > >> > Thanks for driving the FLIP
> > >> >
> > >> > I have a couple of questions
> > >> > Am I right that INFORMATION_SCHEMA mentioned by Timo[1] is out of
> > scope
> > >> of
> > >> > this FLIP?
> > >> > I noticed there are some support of it in
> > Spark[2]/Hive[3]/Snowflake[4]
> > >> and
> > >> > others
> > >> >
> > >> > Does it make sense to add support for connectors e.g. show
> > >> > {sink|source|all} connectors?
> > >> >
> > >> > [1]
> https://lists.apache.org/thread/2g108qlfwbhb56wnoc5qj0yq29dvq1vv
> > >> > [2] https://issues.apache.org/jira/browse/SPARK-16452
> > >> > [3] https://issues.apache.org/jira/browse/HIVE-1010
> > >> > [4] https://docs.snowflake.com/en/sql-reference/info-schema
> > >> >
> > >> >
> > >> > On Fri, Feb 24, 2023 at 4:19 AM Jark Wu wrote:
> > >> >
> > >> > > Hi Jing,
> > >> > >
> > >> > > > we'd better reduce the dependency chain and follow the Law of
> > >> > > Demeter(LoD, clean code).
> > >> > > > Adding a new method in TableEnvironment is therefore better than
> > >> > calling
> > >> > > an API chain
> > >> > >
> > >> > > I think I don't fully agree that LoD is a good practice.
>
[jira] [Created] (FLINK-31224) Add metrics for flink table store
Jun Zhang created FLINK-31224: - Summary: Add metrics for flink table store Key: FLINK-31224 URL: https://issues.apache.org/jira/browse/FLINK-31224 Project: Flink Issue Type: Improvement Components: Table Store Affects Versions: table-store-0.3.1 Reporter: Jun Zhang Fix For: table-store-0.4.0 Add metrics for flink table store -- This message was sent by Atlassian Jira (v8.20.10#820010)
