[jira] [Created] (FLINK-33459) Introduce the new source that supports the same functionality as the original JDBC input format
RocMarshal created FLINK-33459: -- Summary: Introduce the new source that supports the same functionality as the original JDBC input format Key: FLINK-33459 URL: https://issues.apache.org/jira/browse/FLINK-33459 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33460) Support more authentication connection types such as the secret.
RocMarshal created FLINK-33460: -- Summary: Support more authentication connection types such as the secret. Key: FLINK-33460 URL: https://issues.apache.org/jira/browse/FLINK-33460 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33461) Support stream related semantics for the new jdbc source
RocMarshal created FLINK-33461: -- Summary: Support stream related semantics for the new jdbc source Key: FLINK-33461 URL: https://issues.apache.org/jira/browse/FLINK-33461 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33462) Sort out the document page about the new Jdbc source.
RocMarshal created FLINK-33462: -- Summary: Sort out the document page about the new Jdbc source. Key: FLINK-33462 URL: https://issues.apache.org/jira/browse/FLINK-33462 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33463) Support the implementation of dynamic source tables based on the new source
RocMarshal created FLINK-33463: -- Summary: Support the implementation of dynamic source tables based on the new source Key: FLINK-33463 URL: https://issues.apache.org/jira/browse/FLINK-33463 Project: Flink Issue Type: Sub-task Components: Table SQL / JDBC Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: How do I source debug the scala in the flink table planner?
Hi David since you've mentioned FLINK-33365 I'm curious whether you really need to use SQL client? Couldn't you go same way like it's done for tests like apache.flink.connector.jdbc.table.JdbcDynamicTableSourceITCase and just use TableEnvironment something like env = StreamExecutionEnvironment.getExecutionEnvironment(); tEnv = StreamTableEnvironment.create(env); tEnv.executeSql(); and then you don't have this issue with SQL client you've mentioned above On Fri, Nov 3, 2023 at 2:16 PM David Radley wrote: > Hi Sergey, > Thanks Sergey. Yes I tried that. > > I notice > https://nightlies.apache.org/flink/flink-docs-master/docs/concepts/flink-architecture/ > which shows the planner running in the Flink application, which in my case > is the SQL client. This makes sense as the debug statements to stdout show > in the sql client console. > > I amended the sql-client script to add in extra jvm options including the > debug port. I start the SQL client and can see it starting up the debug > port – in my case 9000. > > I go into Intellij which has the Scala plugin and attach to the process – > which I can see. But the Scala breakpoints do not spring. I wonder if it > relates to the planner being loaded in a different class loader. > > Kind regards, David. > > > > > > From: Sergey Nuyanzin > Date: Thursday, 2 November 2023 at 19:58 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: How do I source debug the scala in the flink table > planner? > Hi David > > thanks on working on it > > May be I don't fully understand your issue however if you are using > IntellijIdea and scala plugin (at least in readme it is mentioned that it > is recommended) > then you should be able to download corresponding sources(at least for > table-planner) set breakpoints both in java and scala and debug... > Or what is the issue with this? > > On Thu, Nov 2, 2023 at 6:59 PM David Radley > wrote: > > > Hi, > > I am working on issue https://issues.apache.org/jira/browse/FLINK-33365 > > which has been marked as critical and a blocker for the next release of > the > > jdbc connector. I can recreate an issue locally using code I built from > > source, so I can add in println’s which are coming out – but this is slow > > and tedious. > > > > Ideally I would like to be able to source debug the scala in the flink > > table planner; any advice would be fab? > > > > Kind regards, David. > > > > Unless otherwise stated above: > > > > IBM United Kingdom Limited > > Registered in England and Wales with number 741598 > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU > > > > > -- > Best regards, > Sergey > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU > -- Best regards, Sergey
[DISCUSS] Release Flink 1.16.3
Hi all, I would like to discuss creating a new 1.16 patch release (1.16.3). The last 1.16 release is over five months old, and since then, 50 tickets have been closed [1], of which 10 are blocker/critical [2]. Some of them are quite important, such as FLINK-32296 [3], FLINK-32548 [4] and FLINK-33010[5]. In addition to this, FLINK-33149 [6] is important to bump snappy-java to 1.1.10.4. Although FLINK-33149 is unresolved, it was done in 1.16.3. I am not aware of any unresolved blockers and there are no in-progress tickets [7]. Please let me know if there are any issues you'd like to be included in this release but still not merged. Since 1.18.0 has been released, I'd suggest that we vote to make 1.16.3 the final bugix release of 1.16, looking forward to any feedback from you. Background info could be found at [8], and thanks Jing for the information. If the community agrees to create this new patch release, I could volunteer as the release manager. Since there will be another flink-1.17.2 release request during the same time, I will work with Yun and Yu since many issues will be fixed in both releases. [1] https://issues.apache.org/jira/browse/FLINK-32231?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.16.3%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC [2] https://issues.apache.org/jira/browse/FLINK-32231?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.16.3%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC [3] https://issues.apache.org/jira/browse/FLINK-32296 [4] https://issues.apache.org/jira/browse/FLINK-32548 [5] https://issues.apache.org/jira/browse/FLINK-33010 [6] https://issues.apache.org/jira/browse/FLINK-33149 [7] https://issues.apache.org/jira/projects/FLINK/versions/12353259 [8] https://lists.apache.org/thread/szq23kr3rlkm80rw7k9n95js5vqpsnbv Best, Rui
[DISCUSS] Release 1.17.2
Hi all, I would like to discuss creating a new 1.17 patch release (1.17.2). The last 1.17 release is near half a year old, and since then, 79 tickets have been closed [1], of which 15 are blocker/critical [2]. Some of them are quite important, such as FLINK-32758 [3], FLINK-32296 [4], FLINK-32548 [5] and FLINK-33010[6]. In addition to this, FLINK-33149 [7] is important to bump snappy-java to 1.1.10.4. Although FLINK-33149 is unresolved, it was done in 1.17.2. I am not aware of any unresolved blockers and there are no in-progress tickets [8]. Please let me know if there are any issues you'd like to be included in this release but still not merged. If the community agrees to create this new patch release, I could volunteer as the release manager with Yu Chen. Since there will be another flink-1.16.3 release request during the same time, we will work with Rui Fan since many issues will be fixed in both releases. [1] https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.2%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC [2] https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.2%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC [3] https://issues.apache.org/jira/browse/FLINK-32758 [4] https://issues.apache.org/jira/browse/FLINK-32296 [5] https://issues.apache.org/jira/browse/FLINK-32548 [6] https://issues.apache.org/jira/browse/FLINK-33010 [7] https://issues.apache.org/jira/browse/FLINK-33149 [8] https://issues.apache.org/jira/projects/FLINK/versions/12353260 Best Yun Tang
Re: [DISCUSS] FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects
Hi Jing, Thank you for your feedback on the proposal. I appreciate your input and understand your concerns. Regarding your questions, the complex Java class that will be deprecated is org.apache.flink.api.common.restartstrategy.RestartStrategies and and its internal classes, which include: - RestartStrategyConfiguration - FallbackRestartStrategyConfiguration - FailureRateRestartStrategyConfiguration - ExponentialDelayRestartStrategyConfiguration - FixedDelayRestartStrategyConfiguration - NoRestartStrategyConfiguration And based on your suggestion, I have restructured the content of the FLIP to clearly indicate which entire classes will be deprecated and which methods and fields will be deprecated. Please take a look at the updated version. Best regards, Junrui Jing Ge 于2023年11月5日周日 00:31写道： > Hi Junrui, > > Thanks for bringing the proposal to our attention. It looks overall great. > > I am just a little bit confused with the content of Flip with the following > questions: > > 1. How many complex java classes will be deprecated? > 2. After reading the section title "Deprecate following classes, fields and > methods", I thought that e.g. > org.apache.flink.streaming.api.environment.StreamExecutionEnvironment will > be deprecated. I would suggest restructure the content and point out > clearly which entire classes will be deprecated and which methods and > fields and not the class itself will be deprecated instead of mixing them > up. WDYT? > > Best regards, > Jing > > > On Fri, Nov 3, 2023 at 8:59 AM Wencong Liu wrote: > > > Thanks Junrui for your effort! > > > > Making all configuration code paths lead to ConfigOption is a more > > standardized > > approach to configuring Flink applications. > > > > +1 for this proposal. > > > > Best, > > Wencong Liu > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2023-11-02 10:10:14, "Junrui Lee" wrote: > > >Hi devs, > > > > > >I would like to start a discussion on FLIP-381: Deprecate configuration > > >getters/setters that return/set complex Java objects[1]. > > > > > >Currently, the job configuration in FLINK is spread out across different > > >components, which leads to inconsistencies and confusion. To address > this > > >issue, it is necessary to migrate non-ConfigOption complex Java objects > to > > >use ConfigOption and adopt a single Configuration object to host all the > > >configuration. > > >However, there is a significant blocker in implementing this solution. > > >These complex Java objects in StreamExecutionEnvironment, > > CheckpointConfig, > > >and ExecutionConfig have already been exposed through the public API, > > >making it challenging to modify the existing implementation. > > > > > >Therefore, I propose to deprecate these Java objects and their > > >corresponding getter/setter interfaces, ultimately removing them in > > >FLINK-2.0. > > > > > >Your feedback and thoughts on this proposal are highly appreciated. > > > > > >Best regards, > > >Junrui Lee > > > > > >[1] > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278464992 > > >
[DISCUSS] Release flink-shaded 16.2
Hi, all, I would like to discuss creating a new 16.2 patch release for flink-shaded[1]. In our ARM environment, we recently encountered a critical bug (FLINK-33417[2]) while using flink-shaded 16.1 in flink 1.17. The issue has been discussed in the discussion[3]. Fortunately, we could resolve this bug by updating the netty version. To address this issue comprehensively, we need to release a new minor version for flink-shaded. As a result, Flink 1.17 should update its dependent flink-shaded version accordingly. However, there is no need for Flink 1.18+ to update the flink-shaded version since these newer versions already depend on flink-shaded 17.0+ and have updated their netty version, thereby eliminating the existence of this bug. Considering the low probability of this issue occurring, I do not recommend considering it as a blocker for the Flink 1.17 release. Furthermore, the external connector has been decoupled from flink-shaded in FLINK-33190[4], and the compatibility of the newer netty version 4.1.91 has been verified in the new Flink 1.18+(which is dependent on flink-shaded 17.0 and it is dependent on netty 4.1.91). Hence, we believe this update will not cause any compatibility issues. If there are no objections, we would greatly appreciate it if a committer could volunteer as the release manager. Thank you. [1] https://github.com/apache/flink-shaded [2] https://issues.apache.org/jira/browse/FLINK-33417 [3] https://lists.apache.org/thread/y1c8545bcsx2836d9pgfdzj65knvw7kb [4] https://issues.apache.org/jira/browse/FLINK-33190 Best, Yuxin
