date:20231031

Re: [VOTE] Release flink-connector-aws, v4.2.0 release candidate #1

2023-10-31 Thread mystic lama

+1 (non-binding)

- validated shasum
- verified build
   - Java 8   - build good and all test cases pass
   - Java 11 - build good and all test cases pass

Observations: got test failures with Java 17, something to look for in
future

On Tue, 31 Oct 2023 at 08:42, Danny Cranmer  wrote:

> Hi everyone,
>
> Please review and vote on release candidate #1 for the version 4.2.0, as
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2],
> which are signed with the key with fingerprint
> 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag v4.2.0-rc1 [5],
> * website pull request listing the new release [6].
> * A link to the CI run on the release tag [7]
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Danny
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353011
> [2]
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-4.2.0-rc1
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1665/
> [5] https://github.com/apache/flink-connector-aws/releases/tag/v4.2.0-rc1
> [6] https://github.com/apache/flink-web/pull/693
> [7] https://github.com/apache/flink-connector-aws/actions/runs/6707962074
>

Re: Pointers to computational models of Flink CEP

2023-10-31 Thread David Anderson

The implementation of Flink CEP was largely based on Efficient Pattern
Matching over Event Streams by Jagrati Agrawal, Yanlei Diao, Daniel
Gyllstrom, and Neil Immerman from UMass Amherst [1].

[1] https://people.cs.umass.edu/~yanlei/publications/sase-sigmod08.pdf

Cheers,
David

On Tue, Oct 31, 2023 at 10:04 AM Vishwas Kalani 
wrote:

> Hey,
> I am a computer science student working on a project related to flink cep.
> I want to understand the theoretical foundations of flink cep. Could I get
> some pointers to the computational models used by Flink CEP and how do
> various elements of flink CEP function.
> Thanking you in advance
>

[jira] [Created] (FLINK-33422) Add restore tests for Calc node

2023-10-31 Thread Bonnie Varghese (Jira)

Bonnie Varghese created FLINK-33422:
---

 Summary: Add restore tests for Calc node
 Key: FLINK-33422
 URL: https://issues.apache.org/jira/browse/FLINK-33422
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Bonnie Varghese






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33421) Implement ExecNode Restore Tests

2023-10-31 Thread Bonnie Varghese (Jira)

Bonnie Varghese created FLINK-33421:
---

 Summary: Implement ExecNode Restore Tests
 Key: FLINK-33421
 URL: https://issues.apache.org/jira/browse/FLINK-33421
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Reporter: Bonnie Varghese


Implement Restore Tests for various exec nodes to improve coverage

Related JIRA: https://issues.apache.org/jira/browse/FLINK-33375



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[ANNOUNCE] Apache Flink Kafka Connectors 3.0.1 released

2023-10-31 Thread Tzu-Li (Gordon) Tai

The Apache Flink community is very happy to announce the release of Apache
Flink Kafka Connectors 3.0.1. This release is compatible with the Apache
Flink 1.17.x and 1.18.x release series.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352910

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Gordon

[jira] [Created] (FLINK-33420) Run e2e test fails Intermetiently with ClientCoordinationHandler : Unhandled exception

2023-10-31 Thread Samrat Deb (Jira)

Samrat Deb created FLINK-33420:
--

 Summary: Run e2e test fails Intermetiently with 
ClientCoordinationHandler : Unhandled exception
 Key: FLINK-33420
 URL: https://issues.apache.org/jira/browse/FLINK-33420
 Project: Flink
  Issue Type: Bug
Reporter: Samrat Deb


 

```
Oct 31 08:49:37 2023-10-31 08:49:33,348 ERROR 
org.apache.flink.runtime.rest.handler.job.coordination.ClientCoordinationHandler
 [] - Unhandled exception.
Oct 31 08:49:37 org.apache.flink.runtime.messages.FlinkJobNotFoundException: 
Could not find Flink job (8528fbf0d50c0f038653f6815d56f6fd)
Oct 31 08:49:37 at 
org.apache.flink.runtime.dispatcher.Dispatcher.getJobMasterGateway(Dispatcher.java:1450)
 ~[flink-dist-1.18-SNAPSHOT.jar:1.18-SNAPSHOT]
Oct 31 08:49:37 at 
org.apache.flink.runtime.dispatcher.Dispatcher.performOperationOnJobMasterGateway(Dispatcher.java:1465)
 ~[flink-dist-1.18-SNAPSHOT.jar:1.18-SNAPSHOT]
Oct 31 08:49:37 at 
org.apache.flink.runtime.dispatcher.Dispatcher.deliverCoordinationRequestToCoordinator(Dispatcher.java:1088)
 ~[flink-dist-1.18-SNAPSHOT.jar:1.18-SNAPSHOT]
Oct 31 08:49:37 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method) ~[?:1.8.0_382] 
```

log Link : 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54212=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=0f3adb59-eefa-51c6-2858-3654d9e0749d=ae4f8708-9994-57d3-c2d7-b892156e7812



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Pointers to computational models of Flink CEP

2023-10-31 Thread Vishwas Kalani

Hey,
I am a computer science student working on a project related to flink cep.
I want to understand the theoretical foundations of flink cep. Could I get
some pointers to the computational models used by Flink CEP and how do
various elements of flink CEP function.
Thanking you in advance

Re: [DISCUSS] AWS Connectors v4.2.0 release + 1.18 support

2023-10-31 Thread Danny Cranmer

There is no plan for the other connectors yet. I can pickup a few, will
start a separate thread tomorrow if someone doesn't beat me to it.

Danny.

On Tue, 31 Oct 2023, 04:00 Leonard Xu,  wrote:

> +1, thanks Dany for driving this.
>
> One related question, do we have plan to find some volunteers to release
> rest external connectors for 1.18 support?
>
> Best,
> Leonard
>
>
> > 2023年10月31日 上午12:17，Tzu-Li (Gordon) Tai  写道：
> >
> > +1
> >
> > On Mon, Oct 30, 2023 at 9:00 AM Danny Cranmer 
> > wrote:
> >
> >> Hey,
> >>
> >>> Did you mean skip 4.1.1, since 4.1.0 has already been released?
> >>
> >> I meant skip "4.1.0-1.18" since we could release this with the existing
> >> source. We will additionally skip 4.1.1 and jump to 4.2.0 since this
> >> version has features it should be a minor version rather than a patch
> [1].
> >>
> >>> Does this imply that the 4.1.x series will be reserved for Flink 1.17,
> >> and the 4.2.x series will correspond to Flink 1.18?
> >>
> >> 4.1.x will receive bug fixes for Flink 1.17.
> >> 4.2.x will receive bug fixes and features for Flink 1.17 and 1.18.
> >>
> >> Thanks,
> >> Danny
> >>
> >> [1] https://semver.org/
> >>
> >>
> >> On Mon, Oct 30, 2023 at 3:47 PM Samrat Deb 
> wrote:
> >>
> >>> Hi Danny ,
> >>>
> >>> Thank you for driving it.
> >>>
> >>> +1 (non binding )
> >>>
> >>>
>  I am proposing we skip 4.1.0 for Flink 1.18 and go
> >>> straight to 4.2.0.
> >>>
> >>> Does this imply that the 4.1.x series will be reserved for Flink 1.17,
> >> and
> >>> the 4.2.x series will correspond to Flink 1.18?
> >>>
> >>> Bests,
> >>> Samrat
> >>>
> >>>
> >>> On Mon, Oct 30, 2023 at 7:32 PM Jing Ge 
> >>> wrote:
> >>>
>  Hi Danny,
> 
>  +1 Thanks for driving it. Did you mean skip 4.1.1, since 4.1.0 has
> >>> already
>  been released?
> 
>  Best regards,
>  Jing
> 
>  On Mon, Oct 30, 2023 at 11:49 AM Danny Cranmer <
> >> dannycran...@apache.org>
>  wrote:
> 
> > Hello all,
> >
> > I would like to start the discussion to release Apache Flink AWS
>  connectors
> > v4.2.0. We released v4.1.0 over six months ago on 2023-04-03. Since
> >>> then
>  we
> > have resolved 23 issues [1]. Additionally now Flink 1.18 is live we
> >>> need
>  to
> > add support for this. I am proposing we skip 4.1.0 for Flink 1.18 and
> >>> go
> > straight to 4.2.0. The CI is stable [2].
> >
> > I volunteer myself as the release manager.
> >
> > Thanks,
> > Danny
> >
> > [1]
> >
> >
> 
> >>>
> >>
> https://issues.apache.org/jira/browse/FLINK-33021?jql=statusCategory%20%3D%20done%20AND%20project%20%3D%2012315522%20AND%20fixVersion%20%3D%2012353011%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC
> > [2] https://github.com/apache/flink-connector-aws/actions
> >
> 
> >>>
> >>
>
>

[VOTE] Release flink-connector-aws, v4.2.0 release candidate #1

2023-10-31 Thread Danny Cranmer

Hi everyone,

Please review and vote on release candidate #1 for the version 4.2.0, as
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which are signed with the key with fingerprint
0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag v4.2.0-rc1 [5],
* website pull request listing the new release [6].
* A link to the CI run on the release tag [7]

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Danny

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353011
[2]
https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-4.2.0-rc1
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1665/
[5] https://github.com/apache/flink-connector-aws/releases/tag/v4.2.0-rc1
[6] https://github.com/apache/flink-web/pull/693
[7] https://github.com/apache/flink-connector-aws/actions/runs/6707962074

[jira] [Created] (FLINK-33419) Port PROCTIME/ROWTIME functions to the new inference stack

2023-10-31 Thread Dawid Wysakowicz (Jira)

Dawid Wysakowicz created FLINK-33419:


 Summary: Port PROCTIME/ROWTIME functions to the new inference stack
 Key: FLINK-33419
 URL: https://issues.apache.org/jira/browse/FLINK-33419
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.19.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33418) SqlGatewayE2ECase failed due to ConnectException

2023-10-31 Thread Matthias Pohl (Jira)

Matthias Pohl created FLINK-33418:
-

 Summary: SqlGatewayE2ECase failed due to ConnectException
 Key: FLINK-33418
 URL: https://issues.apache.org/jira/browse/FLINK-33418
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Client, Tests
Reporter: Matthias Pohl


The container couldn't be started in [this 
build|https://github.com/XComp/flink/actions/runs/6696839844/job/18195926497#step:15:11765]:
{code}
Error: 20:18:40 20:18:40.111 [ERROR] Tests run: 1, Failures: 0, Errors: 1, 
Skipped: 0, Time elapsed: 110.789 s <<< FAILURE! - in 
org.apache.flink.table.gateway.SqlGatewayE2ECase
Error: 20:18:40 20:18:40.111 [ERROR] 
org.apache.flink.table.gateway.SqlGatewayE2ECase  Time elapsed: 110.789 s  <<< 
ERROR!
Oct 30 20:18:40 org.testcontainers.containers.ContainerLaunchException: 
Container startup failed for image prestodb/hdp2.6-hive:10
Oct 30 20:18:40 at 
org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:349)
Oct 30 20:18:40 at 
org.apache.flink.table.gateway.containers.HiveContainer.doStart(HiveContainer.java:69)
Oct 30 20:18:40 at 
org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
Oct 30 20:18:40 at 
org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1131)
Oct 30 20:18:40 at 
org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:28)
Oct 30 20:18:40 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
Oct 30 20:18:40 at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
Oct 30 20:18:40 at 
org.junit.runners.ParentRunner.run(ParentRunner.java:413)
Oct 30 20:18:40 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
Oct 30 20:18:40 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
Oct 30 20:18:40 at 
org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
Oct 30 20:18:40 at 
org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
Oct 30 20:18:40 at 
org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
Oct 30 20:18:40 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
Oct 30 20:18:40 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
Oct 30 20:18:40 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
Oct 30 20:18:40 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
Oct 30 20:18:40 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
Oct 30 20:18:40 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
Oct 30 20:18:40 at 
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
Oct 30 20:18:40 at 
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
Oct 30 20:18:40 at 
org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
Oct 30 20:18:40 at 
org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
Oct 30 20:18:40 at 
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
Oct 30 20:18:40 at 
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
Oct 30 20:18:40 at 
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:128)
Oct 30 20:18:40 at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
Oct 30 20:18:40 at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
Oct 30 20:18:40 at 
org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
Oct 30 20:18:40 at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Oct 30 20:18:40 Caused by: org.rnorth.ducttape.RetryCountExceededException: 
Retry limit hit with exception
Oct 30 20:18:40 at 
org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
Oct 30 20:18:40 at 
org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
Oct 30 20:18:40 ... 29 more
Oct 30 20:18:40 Caused by: 
org.testcontainers.containers.ContainerLaunchException: Could not create/start 
container
Oct 30 20:18:40 at

Re: Request to release flink 1.6.3

2023-10-31 Thread Rui Fan

Thanks Vikas for the ask!

Hi devs,

Is anyone willing to pick up the release of 1.16.3 and 1.17.2 with me? If
so, I can volunteer to release one of the versions. If no one picks it up
for more than three days, I volunteer to release two versions. After it’s
determined, the official discussion can be started.

Looking forward to other committer or  PMC join this release!


Hi Matthias,

Thank you for sorting out these 2 lists. 1.16.3 may be the final version of
1.16 series, it makes sense to sort out all left over issues.

Also, some of release processes need PMC  permission, would you mind
helping it? Thanks~

Best,
Rui

On Tue, 31 Oct 2023 at 21:42, vikas patil  wrote:

> Hello Rui,
>
> Do we need more votes for this or are we good to go with the release of
> 1.6.3 ? Please let me know. Thanks.
>
> -Vikas
>
> On Tue, Oct 24, 2023 at 9:27 AM Rui Fan <1996fan...@gmail.com> wrote:
>
> > Hi Vikas,
> >
> > Thanks for your feedback!
> >
> > Do you mean flink 1.16.3 instead of 1.6.3?
> >
> > The 1.16.2 and 1.17.1 were released on 2023-05-25,
> > it’s been 5 months. And the flink community has fixed
> > many bugs in the past 5 months. Usually, there is a
> > fix(minor) version every three or four months, so I propose
> > to release 1.16.3 and 1.17.2 now.
> >
> > If the community agrees to create this new patch release, I could
> > volunteer as the release manager for one of the 1.16.3 or 1.17.2.
> >
> > Looking forward to feedback from the community, thank you
> >
> > [1]
> >
> >
> https://flink.apache.org/2023/05/25/apache-flink-1.16.2-release-announcement/
> > [2]
> >
> >
> https://flink.apache.org/2023/05/25/apache-flink-1.17.1-release-announcement/
> >
> > Best,
> > Rui
> >
> > On Tue, Oct 24, 2023 at 9:50 PM vikas patil 
> > wrote:
> >
> > > Hello All,
> > >
> > > Facing this FLINK-28185 <
> > https://issues.apache.org/jira/browse/FLINK-28185
> > > >
> > > issue for one of the flink jobs. We are running flink version 1.6.1 but
> > it
> > > looks like the backport 
> to
> > > 1.6
> > > was never released as 1.6.3. The latest that was released is 1.6.2
> > > . By any chance we can
> get
> > > the 1.6.3 released ?
> > >
> > > Also we use the official flink docker  >
> > > image. Not sure if that needs to be updated as well manually. Thanks.
> > >
> > > -Vikas
> > >
> >
>

Re: Request to release flink 1.6.3

2023-10-31 Thread vikas patil

Hello Rui,

Do we need more votes for this or are we good to go with the release of
1.6.3 ? Please let me know. Thanks.

-Vikas

On Tue, Oct 24, 2023 at 9:27 AM Rui Fan <1996fan...@gmail.com> wrote:

> Hi Vikas,
>
> Thanks for your feedback!
>
> Do you mean flink 1.16.3 instead of 1.6.3?
>
> The 1.16.2 and 1.17.1 were released on 2023-05-25,
> it’s been 5 months. And the flink community has fixed
> many bugs in the past 5 months. Usually, there is a
> fix(minor) version every three or four months, so I propose
> to release 1.16.3 and 1.17.2 now.
>
> If the community agrees to create this new patch release, I could
> volunteer as the release manager for one of the 1.16.3 or 1.17.2.
>
> Looking forward to feedback from the community, thank you
>
> [1]
>
> https://flink.apache.org/2023/05/25/apache-flink-1.16.2-release-announcement/
> [2]
>
> https://flink.apache.org/2023/05/25/apache-flink-1.17.1-release-announcement/
>
> Best,
> Rui
>
> On Tue, Oct 24, 2023 at 9:50 PM vikas patil 
> wrote:
>
> > Hello All,
> >
> > Facing this FLINK-28185 <
> https://issues.apache.org/jira/browse/FLINK-28185
> > >
> > issue for one of the flink jobs. We are running flink version 1.6.1 but
> it
> > looks like the backport  to
> > 1.6
> > was never released as 1.6.3. The latest that was released is 1.6.2
> > . By any chance we can get
> > the 1.6.3 released ?
> >
> > Also we use the official flink docker 
> > image. Not sure if that needs to be updated as well manually. Thanks.
> >
> > -Vikas
> >
>

Re: [DISCUSS] FLIP-379: Dynamic source parallelism inference for batch jobs

2023-10-31 Thread Zhu Zhu

Thanks for opening the FLIP and kicking off this discussion, Xia!
The proposed changes make up an important missing part of the dynamic
parallelism inference of adaptive batch scheduler.

Besides that, it is also one good step towards supporting dynamic
parallelism inference for streaming sources, e.g. allowing Kafka
sources to determine its parallelism automatically based on the
number of partitions.

+1 for the proposal.

Thanks,
Zhu

Xia Sun  于2023年10月31日周二 16:01写道：

> Hi everyone,
> I would like to start a discussion on FLIP-379: Dynamic source parallelism
> inference for batch jobs[1].
>
> In general, there are three main ways to set source parallelism for batch
> jobs:
> (1) User-defined source parallelism.
> (2) Connector static parallelism inference.
> (3) Dynamic parallelism inference.
>
> Compared to manually setting parallelism, automatic parallelism inference
> is easier to use and can better adapt to varying data volumes each day.
> However, static parallelism inference cannot leverage runtime information,
> resulting in inaccurate parallelism inference. Therefore, for batch jobs,
> dynamic parallelism inference is the most ideal, but currently, the support
> for adaptive batch scheduler is not very comprehensive.
>
> Therefore, we aim to introduce a general interface that enables the
> adaptive batch scheduler to dynamically infer the source parallelism at
> runtime. Please refer to the FLIP[1] document for more details about the
> proposed design and implementation.
>
> I also thank Zhu Zhu and LiJie Wang for their suggestions during the
> pre-discussion.
> Looking forward to your feedback and suggestions, thanks.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-379%3A+Dynamic+source+parallelism+inference+for+batch+jobs
>
> Best regards,
> Xia
>

Re: [DISCUSS] FLIP-376: Add DISTRIBUTED BY clause

2023-10-31 Thread Timo Walther

Hi Jark,

here are the checks I had in mind so far. But we can also discuss this
during the implementation in the PRs. Most of the tasks are very similar
to PARTITIONED BY which is also a characteristic of a sink.

1) Check that DISTRIBUTED BY columns reference physical columns and at
least 1. In DefaultSchemaResolver like we do for PARTITIONED BY.
2) Check that if DISTRIBUTED is defined the sink implements
SupportsBucketing. In DynamicSinkUtils like we do for metadata columns.

Currently, for sources we would only check for semantical correctness
(1) but not more. Like we do for PARTITIONED BY.

Do you have more checks in mind? Of course, during implementation I will
make sure that all derived utils will properly work; including CREATE
TABLE LIKE.

Regards,
Timo

On 31.10.23 07:22, Jark Wu wrote:

Hi Timo,

Thank you for the update. The FLIP looks good to me now.
I only have one more question.

What does Flink check and throw exceptions for the bucketing?
For example, do we check interfaces when executing create/alter
DDL and when used as a source?

Best,
Jark

On Tue, 31 Oct 2023 at 00:25, Timo Walther wrote:

Hi Jing,

> Have you considered using BUCKET BY directly?

Which vendor uses this syntax? Most vendors that I checked call this
concept "distribution".

In any case, the "BY" is optional, so certain DDL statements would
declare it like "BUCKET INTO 6 BUCKETS"? And following the PARTITIONED,
we should use the passive voice.

> Did you mean users can use their own algorithm? How to do it?

"own algorithm" only refers to deciding between a list of partitioning
strategies (namely hash and range partitioning) if the connector offers
more than one.

Regards,
Timo

On 30.10.23 12:39, Jing Ge wrote:

Hi Timo,

The FLIP looks great! Thanks for bringing it to our attention! In order

make sure we are on the same page, I would ask some questions:

1. DISTRIBUTED BY reminds me DISTRIBUTE BY from Hive like Benchao

mentioned

which is used to distribute rows amond reducers, i.e. focusing on the
shuffle during the computation. The FLIP is focusing more on storage, if

am not mistaken. Have you considered using BUCKET BY directly?

2. According to the FLIP: " CREATE TABLE MyTable (uid BIGINT, name

STRING)

DISTRIBUTED BY HASH(uid) INTO 6 BUCKETS

- For advanced users, the algorithm can be defined explicitly.
- Currently, either HASH() or RANGE().

"
Did you mean users can use their own algorithm? How to do it?

Best regards,
Jing

On Mon, Oct 30, 2023 at 11:13 AM Timo Walther

wrote:

Let me reply to the feedback from Yunfan:

> Distribute by in DML is also supported by Hive

I see DISTRIBUTED BY and DISTRIBUTE BY as two separate discussions. This
discussion is about DDL. For DDL, we have more freedom as every vendor
has custom syntax for CREATE TABLE clauses. Furthermore, this is tightly
connector to the connector implementation, not the engine. However, for
DML we need to watch out for standard compliance and introduce changes
with high caution.

How a LookupTableSource interprets the DISTRIBUTED BY is
connector-dependent in my opinion. In general this FLIP is a sink
ability, but we could have a follow FLIP that helps in distributing load
of lookup joins.

> to avoid data skew problem

I understand the use case and that it is important to solve it
eventually. Maybe a solution might be to introduce helper Polymorphic
Table Functions [1] in the future instead of new syntax.

[1]

https://www.ifis.uni-luebeck.de/~moeller/Lectures/WS-19-20/NDBDM/12-Literature/Michels-SQL-2016.pdf

Let me reply to the feedback from Benchao:

> Do you think it's useful to add some extensibility for the hash
strategy

The hash strategy is fully determined by the connector, not the Flink
SQL engine. We are not using Flink's hash strategy in any way. If the
hash strategy for the regular Flink file system connector should be
changed, this should be expressed via config option. Otherwise we should
offer a dedicated `hive-filesystem` or `spark-filesystem` connector.

Regards,
Timo

On 30.10.23 10:44, Timo Walther wrote:

Hi Jark,

my intention was to avoid too complex syntax in the first version. In
the past years, we could enable use cases also without this clause, so
we should be careful with overloading it with too functionality in the
first version. We can still iterate on it later, the interfaces are
flexible enough to support more in the future.

I agree that maybe an explicit HASH and RANGE doesn't harm. Also making
the bucket number optional.

I updated the FLIP accordingly. Now the SupportsBucketing interface
declares more methods that help in validation and proving helpful error
messages to users.

Let me know what you think.

Regards,
Timo

On 27.10.23 10:20, Jark Wu wrote:

Hi Timo,

Thanks for starting this discussion. I really like it!
The FLIP is already in good shape, I only have some minor comments.

1. Could we also support HASH and RANGE distribution kind on the

[jira] [Created] (FLINK-33417) Update netty version to 4.1.83 for flink-shaded

2023-10-31 Thread Yuxin Tan (Jira)

Yuxin Tan created FLINK-33417:
-

 Summary: Update netty version to 4.1.83 for flink-shaded
 Key: FLINK-33417
 URL: https://issues.apache.org/jira/browse/FLINK-33417
 Project: Flink
  Issue Type: Bug
  Components: BuildSystem / Shaded
Reporter: Yuxin Tan


In our ARM environment, we encounter a compile error when
using Flink 1.17.

Flink 1.17 depends on flink-shaded 16.1, which uses netty 4.1.82.
However, flink-shaded 16.1 fails to compile in the ARM
environment. As a result, we are unable to compile Flink 1.17
due to this issue.

We have tested compiling flink-shaded using netty 4.1.83 or
a later version in ARM env, and it can compile successfully.

Taking into consideration the previous discussions regarding
compatibility and the dependency of external connectors on
this version, I propose addressing the bug by only updating
flink-shaded's netty to a minor version (e.g., 4.1.83) rather than
backporting FLINK-32032. 

To implement the update, maybe a new release of flink-shaded 16.2 needs to be 
released.

The discussion details is at 
https://lists.apache.org/thread/y1c8545bcsx2836d9pgfdzj65knvw7kb.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33416) FineGrainedSlotManagerTest failed with fatal error

2023-10-31 Thread Matthias Pohl (Jira)

Matthias Pohl created FLINK-33416:
-

 Summary: FineGrainedSlotManagerTest failed with fatal error
 Key: FLINK-33416
 URL: https://issues.apache.org/jira/browse/FLINK-33416
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Reporter: Matthias Pohl


In FLINK-33245, we reported an error of the 
{{ZooKeeperLeaderElectionConnectionHandlingTest}} failure due to a fatal error. 
The corresponding build is [this 
one|https://github.com/XComp/flink/actions/runs/6472726326/job/17575765131].

But the stacktrace indicates that it's actually {{FineGrainedSlotManagerTest}} 
which ran before the ZK-related test:
{code}
Test 
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest.testSlotAllocationAccordingToStrategyResult[testSlotAllocationAccordingToStrategyResult()]
 successfully run.

19:30:11,463 [   pool-752-thread-1] ERROR 
org.apache.flink.util.FatalExitExceptionHandler  [] - FATAL: Thread 
'pool-752-thread-1' produced an uncaught exception. Stopping the process...
java.util.concurrent.CompletionException: 
java.util.concurrent.RejectedExecutionException: Task 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1201ef67[Not
 completed, task = 
java.util.concurrent.Executors$RunnableAdapter@1ea6ccfa[Wrapped task = 
java.util.concurrent.CompletableFuture$UniHandle@36f84d94]] rejected from 
java.util.concurrent.ScheduledThreadPoolExecutor@4642c78d[Shutting down, pool 
size = 1, active threads = 1, queued tasks = 1, completed tasks = 194]
at 
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
 ~[?:?]
at 
java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:951)
 ~[?:?]
at 
java.util.concurrent.CompletableFuture.handleAsync(CompletableFuture.java:2276) 
~[?:?]
at 
org.apache.flink.runtime.resourcemanager.slotmanager.DefaultSlotStatusSyncer.allocateSlot(DefaultSlotStatusSyncer.java:138)
 ~[classes/:?]
at 
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager.allocateSlotsAccordingTo(FineGrainedSlotManager.java:722)
 ~[classes/:?]
at 
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager.checkResourceRequirements(FineGrainedSlotManager.java:645)
 ~[classes/:?]
at 
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager.lambda$checkResourceRequirementsWithDelay$12(FineGrainedSlotManager.java:603)
 ~[classes/:?]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
 [?:?]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
[?:?]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.util.concurrent.RejectedExecutionException: Task 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1201ef67[Not
 completed, task = 
java.util.concurrent.Executors$RunnableAdapter@1ea6ccfa[Wrapped task = 
java.util.concurrent.CompletableFuture$UniHandle@36f84d94]] rejected from 
java.util.concurrent.ScheduledThreadPoolExecutor@4642c78d[Shutting down, pool 
size = 1, active threads = 1, queued tasks = 1, completed tasks = 194]
at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
 ~[?:?]
at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) 
~[?:?]
at 
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
 ~[?:?]
at 
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
 ~[?:?]
at 
java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:705)
 ~[?:?]
at 
java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:687)
 ~[?:?]
at 
java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:949)
 ~[?:?]
... 11 more
[...]
{code}

I leave this issue open for documentation purposes for now. ...in case the 
issue pops up again.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33415) HAJobRunOnHadoopS3FileSystemITCase failed due to NoSuchMethodError

2023-10-31 Thread Matthias Pohl (Jira)

Matthias Pohl created FLINK-33415:
-

 Summary: HAJobRunOnHadoopS3FileSystemITCase failed due to 
NoSuchMethodError
 Key: FLINK-33415
 URL: https://issues.apache.org/jira/browse/FLINK-33415
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / FileSystem
Reporter: Matthias Pohl


{code:java}
Error: 23:16:52 23:16:52.433 [ERROR] Tests run: 1, Failures: 0, Errors: 1, 
Skipped: 0, Time elapsed: 6.271 s <<< FAILURE! - in 
org.apache.flink.fs.s3hadoop.HAJobRunOnHadoopS3FileSystemITCase
37033Error: 23:16:52 23:16:52.433 [ERROR] 
org.apache.flink.fs.s3hadoop.HAJobRunOnHadoopS3FileSystemITCase  Time elapsed: 
6.271 s  <<< ERROR!
37034Oct 10 23:16:52 java.lang.NoSuchMethodError: 'void 
org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism(java.lang.String)'
37035Oct 10 23:16:52at 
org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:84)
37036Oct 10 23:16:52at 
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:315)
37037Oct 10 23:16:52at 
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:300)
37038Oct 10 23:16:52at 
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:575)
37039Oct 10 23:16:52at 
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:474)
37040Oct 10 23:16:52at 
org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:129)
37041Oct 10 23:16:52at 
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:508)
37042Oct 10 23:16:52at 
org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
37043Oct 10 23:16:52at 
org.apache.flink.core.fs.Path.getFileSystem(Path.java:279)
37044Oct 10 23:16:52at 
org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:99)
37045Oct 10 23:16:52at 
org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:86)
37046Oct 10 23:16:52at 
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createZooKeeperHaServices(HighAvailabilityServicesUtils.java:89)
37047Oct 10 23:16:52at 
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createAvailableOrEmbeddedServices(HighAvailabilityServicesUtils.java:69)
37048Oct 10 23:16:52at 
org.apache.flink.runtime.minicluster.MiniCluster$RegularHighAvailabilityServicesFactory.createHAServices(MiniCluster.java:1530)
37049Oct 10 23:16:52at 
org.apache.flink.runtime.minicluster.MiniCluster.createHighAvailabilityServices(MiniCluster.java:617)
37050Oct 10 23:16:52at 
org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:438)
37051Oct 10 23:16:52at 
org.apache.flink.runtime.testutils.MiniClusterResource.startMiniCluster(MiniClusterResource.java:246)
37052Oct 10 23:16:52at 
org.apache.flink.runtime.testutils.MiniClusterResource.before(MiniClusterResource.java:110)
37053Oct 10 23:16:52at 
org.apache.flink.runtime.testutils.InternalMiniClusterExtension.beforeAll(InternalMiniClusterExtension.java:72)
37054Oct 10 23:16:52at 
org.apache.flink.test.junit5.MiniClusterExtension.beforeAll(MiniClusterExtension.java:231)
[...] {code}
[Run 
#14|https://github.com/XComp/flink/actions/runs/6472816505/job/17575963787#step:11:37035]
 in the {{finegrained_resourcemanagement}} stage (see FLINK-33245)

[Run 
#11|https://github.com/XComp/flink/actions/runs/6471147857/job/17571310183#step:11:41740]
 in the {{finegrained_resourcemanagement}} stage (see FLINK-33245)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33414) MiniClusterITCase.testHandleStreamingJobsWhenNotEnoughSlot fails due to unexpected TimeoutException

2023-10-31 Thread Matthias Pohl (Jira)

Matthias Pohl created FLINK-33414:
-

 Summary: 
MiniClusterITCase.testHandleStreamingJobsWhenNotEnoughSlot fails due to 
unexpected TimeoutException
 Key: FLINK-33414
 URL: https://issues.apache.org/jira/browse/FLINK-33414
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Matthias Pohl


We see this test instability in [this 
build|https://github.com/XComp/flink/actions/runs/6695266358/job/18192039035#step:12:9253].
{code:java}
Error: 17:04:52 17:04:52.042 [ERROR] Failures: 
9252Error: 17:04:52 17:04:52.042 [ERROR]   
MiniClusterITCase.testHandleStreamingJobsWhenNotEnoughSlot:120 
9253Oct 30 17:04:52 Expecting a throwable with root cause being an instance of:
9254Oct 30 17:04:52   
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException
9255Oct 30 17:04:52 but was an instance of:
9256Oct 30 17:04:52   java.util.concurrent.TimeoutException: Timeout has 
occurred: 100 ms
9257Oct 30 17:04:52 at 
org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:86)
9258Oct 30 17:04:52 at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
9259Oct 30 17:04:52 at 
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
9260Oct 30 17:04:52 ...(27 remaining lines not displayed - this can be 
changed with Assertions.setMaxStackTraceElementsDisplayed) {code}
The same error occurred in the [finegrained_resourcemanager stage of this 
build|https://github.com/XComp/flink/actions/runs/6468655160/job/17563927249#step:11:26516]
 (as reported in FLINK-33245



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33413) Bump Avro in AWS Connectors to address CVE-2023-39410

2023-10-31 Thread Danny Cranmer (Jira)

Danny Cranmer created FLINK-33413:
-

 Summary: Bump Avro in AWS Connectors to address CVE-2023-39410
 Key: FLINK-33413
 URL: https://issues.apache.org/jira/browse/FLINK-33413
 Project: Flink
  Issue Type: Bug
  Components: Connectors / AWS
Affects Versions: aws-connector-4.1.0
Reporter: Danny Cranmer
 Fix For: aws-connector-4.2.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Kubernetes Operator 1.7.0 release planning

2023-10-31 Thread Rui Fan

Thanks Gyula for driving this release!

I'd like to check with you and community, could we
postpone the code freeze by a week?

I'm developing the FLINK-33099[1], and the prod code is done.
I need some time to develop the tests. I hope this feature is included in
1.7.0 for two main reasons:

1. We have completed the decoupling of the autoscaler and
kubernetes-operator in 1.7.0. During the decoupling period, we modified
a large number of autoscaler-related interfaces. The standalone autoscaler
is an autoscaler process that can run independently. It can help us confirm
whether the new interface is reasonable.
2. 1.18.0 was recently released, standalone autoscaler allows more users to
play autoscaler and in-place rescale.

I have created a draft PR[2] for FLINK-33099, it just includes prod code.
I have run it manually, it works well. And I will try my best to finish all
unit tests before Friday, but must finish all unit tests before next Monday
at the latest.

WDYT?

I'm deeply sorry for the request to postpone the release.

[1] https://issues.apache.org/jira/browse/FLINK-33099
[2] https://github.com/apache/flink-kubernetes-operator/pull/698

Best,
Rui

On Tue, Oct 31, 2023 at 4:10 PM Samrat Deb  wrote:

> Thank you Gyula
>
> (+1 non-binding) in support of you taking on the role of release manager.
>
> > I think this is reasonable as I am not aware of any big features / bug
> fixes being worked on right now. Given the size of the changes related to
> the autoscaler module refactor we should try to focus the remaining time on
> testing.
>
> I completely agree with you. Since the changes are quite extensive, it's
> crucial to allocate more time for thorough testing and verification.
>
> Regarding working with you for the release, I might not have the necessary
> privileges for that.
>
> However, I'd be more than willing to assist with testing the changes,
> validating various features, and checking for any potential regressions in
> the flink-kubernetes-operator.
> Just let me know how I can support the testing efforts.
>
> Bests,
> Samrat
>
>
> On Tue, 31 Oct 2023 at 12:59 AM, Gyula Fóra  wrote:
>
> > Hi all!
> >
> > I would like to kick off the release planning for the operator 1.7.0
> > release. We have added quite a lot of new functionality over the last few
> > weeks and I think the operator is in a good state to kick this off.
> >
> > Based on the original release schedule we had Nov 1 as the proposed
> feature
> > freeze date and Nov 7 as the date for the release cut / rc1.
> >
> > I think this is reasonable as I am not aware of any big features / bug
> > fixes being worked on right now. Given the size of the changes related to
> > the autoscaler module refactor we should try to focus the remaining time
> on
> > testing.
> >
> > I am happy to volunteer as a release manager but I am of course open to
> > working together with someone on this.
> >
> > What do you think?
> >
> > Cheers,
> > Gyula
> >
>

Re: [DISCUSS][FLINK-33240] Document deprecated options as well

2023-10-31 Thread Alexander Fedulov

Hi Zhanghao,

Thanks for the proposition.
In general +1, this sounds like a good idea as long it is clear that the
usage of these settings is discouraged.
Just one minor concern - the configuration page is already very long, do
you have a rough estimate of how many more options would be added with this
change?

Best,
Alexander Fedulov

On Mon, 30 Oct 2023 at 18:24, Matthias Pohl 
wrote:

> Thanks for your proposal, Zhanghao Chen. I think it adds more transparency
> to the configuration documentation.
>
> +1 from my side on the proposal
>
> On Wed, Oct 11, 2023 at 2:09 PM Zhanghao Chen 
> wrote:
>
> > Hi Flink users and developers,
> >
> > Currently, Flink won't generate doc for the deprecated options. This
> might
> > confuse users when upgrading from an older version of Flink: they have to
> > either carefully read the release notes or check the source code for
> > upgrade guidance on deprecated options.
> >
> > I propose to document deprecated options as well, with a "(deprecated)"
> > tag placed at the beginning of the option description to highlight the
> > deprecation status [1].
> >
> > Looking forward to your feedbacks on it.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-33240
> >
> > Best,
> > Zhanghao Chen
> >
>

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

2023-10-31 Thread Jing Ge

Since we didn't get any new feedback from the reporters, +1 for closing
FLINK-30238 according to the analysis done by Gordon, Martijn, and Alex.
Happy to see we can move a little bit forward. Thanks for your effort!

Best regards,
Jing

On Tue, Oct 31, 2023 at 3:03 AM Yun Gao  wrote:

> Hi Martijn and Gordon,
>
> Very sorry for the very late reply, +1 for close FLINK-30238 and open
> dedicated issues
> for the remaining issues.
>
> Best,
> Yun
>
> On Mon, Oct 30, 2023 at 6:42 PM Martijn Visser 
> wrote:
> >
> > Hi everyone,
> >
> > I would like to +1 Gordon's proposal to close FLINK-30238, create a
> > new follow-up ticket and try to address the specific
> > PostCommitTopology in the work that's currently being done by Peter on
> > SinkV2. If there's no feedback on this topic, I assume everyone's OK
> > with that.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Fri, Sep 29, 2023 at 8:11 AM Tzu-Li (Gordon) Tai 
> wrote:
> > >
> > > Hi everyone,
> > >
> > > It’s been a while since this topic was last discussed, but
> nevertheless, it
> > > still remains very desirable to figure out a clear path towards making
> > > SinkV2 @Public.
> > >
> > > There’s a new thread [1] that has a pretty good description on missing
> > > features in SinkV2 from the Iceberg connector’s perspective, which
> prevents
> > > them from migrating - anything related to those new requirements, let's
> > > discuss there.
> > >
> > > Nevertheless, I think we should also revive and reuse this thread to
> reach
> > > consensus / closure on all concerns already brought up here.
> > >
> > > It’s quite a long thread where a lot of various concerns were brought
> up,
> > > but I’d start by addressing two very specific ones: FLIP-287 [2] and
> > > FLINK-30238 [3]
> > >
> > > First of all, FLIP-287 has been approved and merged already, and will
> be
> > > released with 1.18.0. So, connector migrations that were waiting on
> this
> > > should hopefully be unblocked after the release. So this seems to no
> longer
> > > be a concern - let’s see things through with those connectors actually
> > > being migrated.
> > >
> > > FLINK-30238 is sort of a confusing one, and I do believe it is
> (partially)
> > > a false alarm. After looking into this, the problem reported there
> > > essentially breaks down to two things:
> > > 1) TwoPhaseCommittingSink is unable to take a new savepoint after
> restoring
> > > from a savepoint generated via `stop-with-savepoint --drain`
> > > 2) SinkV2 sinks with a PostCommitTopology do not properly have
> post-commits
> > > completed after a stop-with-savepoint operation, since committed
> > > commitables are not emitted to the post-commit topology after the
> committer
> > > receives the end-of-input signal.
> > >
> > > My latest comment in [3] explains this in a bit more detail.
> > >
> > > I believe we can conclude that problem 1) is a non-concern - users
> should
> > > not restore from a job that is drained on stop-with-savepoint and
> cannot
> > > expect the restored job to function normally.
> > > Problem 2) remains a real issue though, and to help clear things up I
> think
> > > we should close FLINK-30238 in favor of a new ticket scoped to the
> specific
> > > PostCommitTopology problem.
> > >
> > > The other open concerns seem to mostly be around graduation criteria
> and
> > > process - I've yet to go through those and will follow up with a
> separate
> > > reply (or perhaps Martijn can help wrap up that part?).
> > >
> > > Thanks,
> > > Gordon
> > >
> > > [1] https://lists.apache.org/thread/h3kg7jcgjstpvwlhnofq093vk93ylgsn
> > > [2]
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240880853
> > > [3] https://issues.apache.org/jira/browse/FLINK-30238
> > >
> > > On Mon, Feb 13, 2023 at 2:50 AM Jing Ge 
> wrote:
> > >
> > > > @Martijn
> > > >
> > > > What I shared previously is the fact of the current KafkaSink.
> Following
> > > > your suggestion, the KafkaSink should still be marked as
> @Experimental for
> > > > now which will need even longer time to graduate. BTW, KafkaSink
> does not
> > > > depend on any @Internal interfaces. The @Internal is used for methods
> > > > coming from @PublicEvolving SinkV2 interfaces, not interfaces
> themself.
> > > > Thanks for bringing this topic up. Currently, there is no rule
> defined to
> > > > say that no @Internal is allowed for methods implemented
> > > > from @PublicEvolving interfaces. Further (off-this-topic) discussion
> might
> > > > be required to check if it really makes sense to define such a rule,
> since
> > > > some methods defined in interfaces might only be used internally,
> i.e. no
> > > > connector user would be aware of them.
> > > >
> > > > @Dong
> > > >
> > > > I agree with everything you said and especially can't agree more to
> let
> > > > developers who will own it make the decision.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > >
> > > > On Sun, Feb 12, 2023 at 2:53 AM Dong Lin 
> wrote:
> > > >
> > > > > Hi

[jira] [Created] (FLINK-33412) Implement type inference for reinterpret_cast function

2023-10-31 Thread Dawid Wysakowicz (Jira)

Dawid Wysakowicz created FLINK-33412:


 Summary: Implement type inference for reinterpret_cast function
 Key: FLINK-33412
 URL: https://issues.apache.org/jira/browse/FLINK-33412
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Dawid Wysakowicz
 Fix For: 1.19.0


https://github.com/apache/flink/blob/91d81c427aa6312841ca868d54e8ce6ea721cd60/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/expressions/Reinterpret.scala



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33411) Implement type inference for window properties functions

2023-10-31 Thread Dawid Wysakowicz (Jira)

Dawid Wysakowicz created FLINK-33411:


 Summary: Implement type inference for window properties functions
 Key: FLINK-33411
 URL: https://issues.apache.org/jira/browse/FLINK-33411
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Dawid Wysakowicz
 Fix For: 1.19.0


https://github.com/apache/flink/blob/91d81c427aa6312841ca868d54e8ce6ea721cd60/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/expressions/windowProperties.scala

Functions:
* WINDOW_START
* WINDOW_END



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33410) Implement type inference for Over offsets functions

2023-10-31 Thread Dawid Wysakowicz (Jira)

Dawid Wysakowicz created FLINK-33410:


 Summary: Implement type inference for Over offsets functions
 Key: FLINK-33410
 URL: https://issues.apache.org/jira/browse/FLINK-33410
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Dawid Wysakowicz
 Fix For: 1.19.0


https://github.com/apache/flink/blob/91d81c427aa6312841ca868d54e8ce6ea721cd60/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/expressions/overOffsets.scala

Functions:
* CURRENT_RANGE
* CURRENT_ROW
* UNBOUNDED_ROW
* UNBOUNDED_RANGE



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33409) Bump Guava to address CVE-2020-8908/CVE-2023-2976

2023-10-31 Thread Danny Cranmer (Jira)

Danny Cranmer created FLINK-33409:
-

 Summary: Bump Guava to address CVE-2020-8908/CVE-2023-2976
 Key: FLINK-33409
 URL: https://issues.apache.org/jira/browse/FLINK-33409
 Project: Flink
  Issue Type: Bug
  Components: Connectors / AWS
Affects Versions: aws-connector-4.1.0, aws-connector-3.0.0
Reporter: Danny Cranmer
 Fix For: aws-connector-4.2.0


Bump Guava from {{32.0.0-jre}} to {{32.1.3-jre}} to mitigate



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

2023-10-31 Thread Jiabao Sun

Hi Becket,

Actually, for FileSystemSource, it is not always desired, only OCR file formats 
support filter pushdown.

We can disable predicate pushdown for FileSystemSource by setting 
'table.optimizer.source.predicate-pushdown-enabled' to false. 
I think we can also disable filter pushdown at a more granular level through 
fine-grained configuration.


Best,
Jiabao


> 2023年10月31日 16:50，Becket Qin  写道：
> 
> Hi Jiabao,
> 
> Thanks for the explanation. Maybe it's easier to explain with an example.
> 
> Let's take FileSystemTableSource as an example. Currently it implements
> SupportsFilterPushDown interface. With your proposal, does it have to
> support `source.filter-push-down.enabled` as well? But this configuration
> does not quite make sense for the FileSystemTableSource because filter
> pushdown is always desired. However, because this configuration is a part
> of the SupportsFilterPushDown interface (which sounds confusing to begin
> with), the FileSystemTableSource can only do one of the following:
> 
> 1. Ignore the user configuration to always apply the pushed down filters -
> this is an apparent anti-pattern because a configuration should always do
> what it says.
> 2. Throw an exception telling users that this configuration is not
> applicable to the FileSystemTableSource.
> 3. Implement this configuration to push back the pushed down filters, even
> though this is never desired.
> 
> None of the above options looks awkward. I am curious what your solution is
> here?
> 
> Thanks,
> 
> Jiangjie (Becket) Qin
> 
> On Tue, Oct 31, 2023 at 3:11 PM Jiabao Sun 
> wrote:
> 
>> Thanks Becket for the further explanation.
>> 
>> Perhaps I didn't explain it clearly.
>> 
>> 1. If a source does not implement the SupportsFilterPushDown interface,
>> the newly added configurations do not need to be added to either the
>> requiredOptions or optionalOptions.
>> Similar to LookupOptions, if a source does not implement
>> LookupTableSource, there is no need to add LookupOptions to either
>> requiredOptions or optionalOptions.
>> 
>> 2. "And these configs are specific to those sources, instead of common
>> configs."
>> The newly introduced configurations define standardized names and default
>> values.
>> They still belong to the configuration at the individual source level.
>> The purpose is to avoid scattered configuration items when different
>> sources implement the same logic.
>> Whether a source should accept these configurations is determined by the
>> source's Factory.
>> 
>> Best,
>> Jiabao
>> 
>> 
>>> 2023年10月31日 13:47，Becket Qin  写道：
>>> 
>>> Hi Jiabao,
>>> 
>>> Please see the replies inline.
>>> 
>>> Introducing common configurations does not mean that all sources must
 accept these configuration options.
 The configuration options supported by a source are determined by the
 requiredOptions and optionalOptions in the Factory interface.
>>> 
>>> This is not true. Both required and optional options are SUPPORTED. That
>>> means they are implemented and if one specifies an optional config it
>> will
>>> still take effect. An OptionalConfig is "Optional" because this
>>> configuration has a default value. Hence, it is OK that users do not
>>> specify their own value. In another word, it is "optional" for the end
>>> users to set the config, but the implementation and support for that
>> config
>>> is NOT optional. In case a source does not support a common config, an
>>> exception must be thrown when the config is provided by the end users.
>>> However, the config we are talking about in this FLIP is a common config
>>> optional to implement, meaning that sometimes the claimed behavior won't
>> be
>>> there even if users specify that config.
>>> 
>>> Similar to sources that do not implement the LookupTableSource interface,
 sources that do not implement the SupportsFilterPushDown interface also
>> do
 not need to accept newly introduced options.
>>> 
>>> First of all, filter pushdown is a behavior of the query optimizer, not
>> the
>>> behavior of Sources. The Sources tells the optimizer that it has the
>>> ability to accept pushed down filters by implementing the
>>> SupportsFilterPushDown interface. And this is the only contract between
>> the
>>> Source and Optimizer regarding whether filters should be pushed down. As
>>> long as a specific source implements this decorative interface, filter
>>> pushdown should always take place, i.e.
>>> *SupportsFilterPushDown.applyFilters()* will be called. There should be
>> no
>>> other config to disable that call. However, Sources can decide how to
>>> behave based on their own configurations after *applyFilters()* is
>> called.
>>> And these configs are specific to those sources, instead of common
>> configs.
>>> Please see the examples I mentioned in the previous email.
>>> 
>>> Thanks,
>>> 
>>> Jiangjie (Becket) Qin
>>> 
>>> On Tue, Oct 31, 2023 at 10:27 AM Jiabao Sun > .invalid>
>>> wrote:
>>> 
 Hi Becket,
 
 Sorry, there was a

[jira] [Created] (FLINK-33408) Fixing the container vulnerability by upgrade the SnakeYaml Maven dependency in flink-kubernetes module.

2023-10-31 Thread Zhou Shijie (Jira)

Zhou Shijie created FLINK-33408:
---

 Summary: Fixing the container vulnerability by upgrade the 
SnakeYaml Maven dependency in flink-kubernetes module.
 Key: FLINK-33408
 URL: https://issues.apache.org/jira/browse/FLINK-33408
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes
Reporter: Zhou Shijie
 Fix For: 1.18.0


_Fix the container vulnerability in 
[CVE-2022-1471|https://github.com/advisories/GHSA-mjmj-j48q-9wg2] by upgrade 
the SnakeYaml Maven dependency in flink-kubernetes module._

Upgrade the Kubernetes Client from 6.6.2 to 6.7.0, thereby upgrading the 
version of snakeyaml, which the Kubernetes Client indirectly depends on, from 
1.33 to 2.0.
h4.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

2023-10-31 Thread Becket Qin

Hi Jiabao,

Thanks for the explanation. Maybe it's easier to explain with an example.

Let's take FileSystemTableSource as an example. Currently it implements
SupportsFilterPushDown interface. With your proposal, does it have to
support `source.filter-push-down.enabled` as well? But this configuration
does not quite make sense for the FileSystemTableSource because filter
pushdown is always desired. However, because this configuration is a part
of the SupportsFilterPushDown interface (which sounds confusing to begin
with), the FileSystemTableSource can only do one of the following:

1. Ignore the user configuration to always apply the pushed down filters -
this is an apparent anti-pattern because a configuration should always do
what it says.
2. Throw an exception telling users that this configuration is not
applicable to the FileSystemTableSource.
3. Implement this configuration to push back the pushed down filters, even
though this is never desired.

None of the above options looks awkward. I am curious what your solution is
here?

Thanks,

Jiangjie (Becket) Qin

On Tue, Oct 31, 2023 at 3:11 PM Jiabao Sun 
wrote:

> Thanks Becket for the further explanation.
>
> Perhaps I didn't explain it clearly.
>
> 1. If a source does not implement the SupportsFilterPushDown interface,
> the newly added configurations do not need to be added to either the
> requiredOptions or optionalOptions.
> Similar to LookupOptions, if a source does not implement
> LookupTableSource, there is no need to add LookupOptions to either
> requiredOptions or optionalOptions.
>
> 2. "And these configs are specific to those sources, instead of common
> configs."
> The newly introduced configurations define standardized names and default
> values.
> They still belong to the configuration at the individual source level.
> The purpose is to avoid scattered configuration items when different
> sources implement the same logic.
> Whether a source should accept these configurations is determined by the
> source's Factory.
>
> Best,
> Jiabao
>
>
> > 2023年10月31日 13:47，Becket Qin  写道：
> >
> > Hi Jiabao,
> >
> > Please see the replies inline.
> >
> > Introducing common configurations does not mean that all sources must
> >> accept these configuration options.
> >> The configuration options supported by a source are determined by the
> >> requiredOptions and optionalOptions in the Factory interface.
> >
> > This is not true. Both required and optional options are SUPPORTED. That
> > means they are implemented and if one specifies an optional config it
> will
> > still take effect. An OptionalConfig is "Optional" because this
> > configuration has a default value. Hence, it is OK that users do not
> > specify their own value. In another word, it is "optional" for the end
> > users to set the config, but the implementation and support for that
> config
> > is NOT optional. In case a source does not support a common config, an
> > exception must be thrown when the config is provided by the end users.
> > However, the config we are talking about in this FLIP is a common config
> > optional to implement, meaning that sometimes the claimed behavior won't
> be
> > there even if users specify that config.
> >
> > Similar to sources that do not implement the LookupTableSource interface,
> >> sources that do not implement the SupportsFilterPushDown interface also
> do
> >> not need to accept newly introduced options.
> >
> > First of all, filter pushdown is a behavior of the query optimizer, not
> the
> > behavior of Sources. The Sources tells the optimizer that it has the
> > ability to accept pushed down filters by implementing the
> > SupportsFilterPushDown interface. And this is the only contract between
> the
> > Source and Optimizer regarding whether filters should be pushed down. As
> > long as a specific source implements this decorative interface, filter
> > pushdown should always take place, i.e.
> > *SupportsFilterPushDown.applyFilters()* will be called. There should be
> no
> > other config to disable that call. However, Sources can decide how to
> > behave based on their own configurations after *applyFilters()* is
> called.
> > And these configs are specific to those sources, instead of common
> configs.
> > Please see the examples I mentioned in the previous email.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Tue, Oct 31, 2023 at 10:27 AM Jiabao Sun  .invalid>
> > wrote:
> >
> >> Hi Becket,
> >>
> >> Sorry, there was a typo in the second point. Let me correct it:
> >>
> >> Introducing common configurations does not mean that all sources must
> >> accept these configuration options.
> >> The configuration options supported by a source are determined by the
> >> requiredOptions and optionalOptions in the Factory interface.
> >>
> >> Similar to sources that do not implement the LookupTableSource
> interface,
> >> sources that do not implement the SupportsFilterPushDown interface also
> do
> >> not need to accept newly introduced

[jira] [Created] (FLINK-33407) Port time functions to the new type inference stack

2023-10-31 Thread Dawid Wysakowicz (Jira)

Dawid Wysakowicz created FLINK-33407:


 Summary: Port time functions to the new type inference stack
 Key: FLINK-33407
 URL: https://issues.apache.org/jira/browse/FLINK-33407
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Dawid Wysakowicz
 Fix For: 1.19.0


The end goal for this task is to remove 
https://github.com/apache/flink/blob/91d81c427aa6312841ca868d54e8ce6ea721cd60/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/expressions/time.scala

For that to happen we need to port:
* EXTRACT
* CURRENT_DATE
* CURRENT_TIME
* CURRENT_TIMESTAMP
* LOCAL_TIME
* LOCAL_TIMESTAMP
* TEMPORAL_OVERLAPS
* DATE_FORMAT
* TIMESTAMP_DIFF
* TO_TIMESTAMP_LTZ
functions to the new type inference



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Kubernetes Operator 1.7.0 release planning

2023-10-31 Thread Samrat Deb

Thank you Gyula

(+1 non-binding) in support of you taking on the role of release manager.

> I think this is reasonable as I am not aware of any big features / bug
fixes being worked on right now. Given the size of the changes related to
the autoscaler module refactor we should try to focus the remaining time on
testing.

I completely agree with you. Since the changes are quite extensive, it's
crucial to allocate more time for thorough testing and verification.

Regarding working with you for the release, I might not have the necessary
privileges for that.

However, I'd be more than willing to assist with testing the changes,
validating various features, and checking for any potential regressions in
the flink-kubernetes-operator.
Just let me know how I can support the testing efforts.

Bests,
Samrat

On Tue, 31 Oct 2023 at 12:59 AM, Gyula Fóra  wrote:

> Hi all!
>
> I would like to kick off the release planning for the operator 1.7.0
> release. We have added quite a lot of new functionality over the last few
> weeks and I think the operator is in a good state to kick this off.
>
> Based on the original release schedule we had Nov 1 as the proposed feature
> freeze date and Nov 7 as the date for the release cut / rc1.
>
> I think this is reasonable as I am not aware of any big features / bug
> fixes being worked on right now. Given the size of the changes related to
> the autoscaler module refactor we should try to focus the remaining time on
> testing.
>
> I am happy to volunteer as a release manager but I am of course open to
> working together with someone on this.
>
> What do you think?
>
> Cheers,
> Gyula
>

[DISCUSS] FLIP-379: Dynamic source parallelism inference for batch jobs

2023-10-31 Thread Xia Sun

Hi everyone,
I would like to start a discussion on FLIP-379: Dynamic source parallelism
inference for batch jobs[1].

In general, there are three main ways to set source parallelism for batch
jobs:
(1) User-defined source parallelism.
(2) Connector static parallelism inference.
(3) Dynamic parallelism inference.

Compared to manually setting parallelism, automatic parallelism inference
is easier to use and can better adapt to varying data volumes each day.
However, static parallelism inference cannot leverage runtime information,
resulting in inaccurate parallelism inference. Therefore, for batch jobs,
dynamic parallelism inference is the most ideal, but currently, the support
for adaptive batch scheduler is not very comprehensive.

Therefore, we aim to introduce a general interface that enables the
adaptive batch scheduler to dynamically infer the source parallelism at
runtime. Please refer to the FLIP[1] document for more details about the
proposed design and implementation.

I also thank Zhu Zhu and LiJie Wang for their suggestions during the
pre-discussion.
Looking forward to your feedback and suggestions, thanks.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-379%3A+Dynamic+source+parallelism+inference+for+batch+jobs

Best regards,
Xia

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

2023-10-31 Thread Jiabao Sun

Thanks Becket for the further explanation.

Perhaps I didn't explain it clearly. 

1. If a source does not implement the SupportsFilterPushDown interface, the 
newly added configurations do not need to be added to either the 
requiredOptions or optionalOptions. 
Similar to LookupOptions, if a source does not implement LookupTableSource, 
there is no need to add LookupOptions to either requiredOptions or 
optionalOptions.

2. "And these configs are specific to those sources, instead of common configs."
The newly introduced configurations define standardized names and default 
values. 
They still belong to the configuration at the individual source level. 
The purpose is to avoid scattered configuration items when different sources 
implement the same logic. 
Whether a source should accept these configurations is determined by the 
source's Factory.

Best,
Jiabao


> 2023年10月31日 13:47，Becket Qin  写道：
> 
> Hi Jiabao,
> 
> Please see the replies inline.
> 
> Introducing common configurations does not mean that all sources must
>> accept these configuration options.
>> The configuration options supported by a source are determined by the
>> requiredOptions and optionalOptions in the Factory interface.
> 
> This is not true. Both required and optional options are SUPPORTED. That
> means they are implemented and if one specifies an optional config it will
> still take effect. An OptionalConfig is "Optional" because this
> configuration has a default value. Hence, it is OK that users do not
> specify their own value. In another word, it is "optional" for the end
> users to set the config, but the implementation and support for that config
> is NOT optional. In case a source does not support a common config, an
> exception must be thrown when the config is provided by the end users.
> However, the config we are talking about in this FLIP is a common config
> optional to implement, meaning that sometimes the claimed behavior won't be
> there even if users specify that config.
> 
> Similar to sources that do not implement the LookupTableSource interface,
>> sources that do not implement the SupportsFilterPushDown interface also do
>> not need to accept newly introduced options.
> 
> First of all, filter pushdown is a behavior of the query optimizer, not the
> behavior of Sources. The Sources tells the optimizer that it has the
> ability to accept pushed down filters by implementing the
> SupportsFilterPushDown interface. And this is the only contract between the
> Source and Optimizer regarding whether filters should be pushed down. As
> long as a specific source implements this decorative interface, filter
> pushdown should always take place, i.e.
> *SupportsFilterPushDown.applyFilters()* will be called. There should be no
> other config to disable that call. However, Sources can decide how to
> behave based on their own configurations after *applyFilters()* is called.
> And these configs are specific to those sources, instead of common configs.
> Please see the examples I mentioned in the previous email.
> 
> Thanks,
> 
> Jiangjie (Becket) Qin
> 
> On Tue, Oct 31, 2023 at 10:27 AM Jiabao Sun 
> wrote:
> 
>> Hi Becket,
>> 
>> Sorry, there was a typo in the second point. Let me correct it:
>> 
>> Introducing common configurations does not mean that all sources must
>> accept these configuration options.
>> The configuration options supported by a source are determined by the
>> requiredOptions and optionalOptions in the Factory interface.
>> 
>> Similar to sources that do not implement the LookupTableSource interface,
>> sources that do not implement the SupportsFilterPushDown interface also do
>> not need to accept newly introduced options.
>> 
>> Best,
>> Jiabao
>> 
>> 
>>> 2023年10月31日 10:13，Jiabao Sun  写道：
>>> 
>>> Thanks Becket for the feedback.
>>> 
>>> 1. Currently, the SupportsFilterPushDown#applyFilters method returns a
>> result that includes acceptedFilters and remainingFilters. The source can
>> decide to push down some filters or not accept any of them.
>>> 2. Introducing common configuration options does not mean that a source
>> that supports the SupportsFilterPushDown capability must accept this
>> configuration. Similar to LookupOptions, only sources that implement the
>> LookupTableSource interface are necessary to accept these configuration
>> options.
>>> 
>>> Best,
>>> Jiabao
>>> 
>>> 
 2023年10月31日 07:49，Becket Qin  写道：
 
 Hi Jiabao and Ruanhang,
 
 Adding a configuration of source.filter-push-down.enabled as a common
 source configuration seems problematic.
 1. The config name is misleading. filter pushdown should only be
>> determined
 by whether the SupportsFilterPushdown interface is implemented or not.
 2. The behavior of this configuration is only applicable to some source
 implementations. Why is it a common configuration?
 
 Here's my suggestion for design principles:
 1. Only add source impl specific configuration to corresponding

[jira] [Created] (FLINK-33406) Flink Job failed due to losing connection from ZK server

2023-10-31 Thread Deng Liwen (Jira)

Deng Liwen created FLINK-33406:
--

 Summary: Flink Job failed due to losing connection from ZK server
 Key: FLINK-33406
 URL: https://issues.apache.org/jira/browse/FLINK-33406
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream
Affects Versions: 1.14.3
 Environment: Flink version: 1.14.3

Zookeeper version: 3.4.10
Reporter: Deng Liwen


We are using Flink 1.14.3 and we faced an issue when losing connection from ZK 
server, the flink job connecting to the target ZK server will be failed 
directly. This case can be reproduced 100% when you kill the connected ZK 
server for simulating connection refused issue. Flink jobs connect to other 
running ZK server keep running as expected. The log output is:
{code:java}
org.apache.flink.client.program.ProgramInvocationException: The main method 
caused an error: java.util.concurrent.TimeoutException        at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
        at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
        at 
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)        
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812) 
       at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)     
   at 
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)      
  at 
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)   
     at java.security.AccessController.doPrivileged(Native Method)        at 
javax.security.auth.Subject.doAs(Subject.java:422)        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)
        at 
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
        at 
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)Caused by: 
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException  
      at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)    
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)  
      at 
org.apache.flink.client.program.StreamContextEnvironment.getJobExecutionResult(StreamContextEnvironment.java:123)
        at 
org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:80)
        at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1916)
 {code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-376: Add DISTRIBUTED BY clause

2023-10-31 Thread Jark Wu

Hi Timo,

Thank you for the update. The FLIP looks good to me now.
I only have one more question.

What does Flink check and throw exceptions for the bucketing?
For example, do we check interfaces when executing create/alter
DDL and when used as a source?

Best,
Jark

On Tue, 31 Oct 2023 at 00:25, Timo Walther  wrote:

> Hi Jing,
>
>  > Have you considered using BUCKET BY directly?
>
> Which vendor uses this syntax? Most vendors that I checked call this
> concept "distribution".
>
> In any case, the "BY" is optional, so certain DDL statements would
> declare it like "BUCKET INTO 6 BUCKETS"? And following the PARTITIONED,
> we should use the passive voice.
>
>  > Did you mean users can use their own algorithm? How to do it?
>
> "own algorithm" only refers to deciding between a list of partitioning
> strategies (namely hash and range partitioning) if the connector offers
> more than one.
>
> Regards,
> Timo
>
>
> On 30.10.23 12:39, Jing Ge wrote:
> > Hi Timo,
> >
> > The FLIP looks great! Thanks for bringing it to our attention! In order
> to
> > make sure we are on the same page, I would ask some questions:
> >
> > 1. DISTRIBUTED BY reminds me DISTRIBUTE BY from Hive like Benchao
> mentioned
> > which is used to distribute rows amond reducers, i.e. focusing on the
> > shuffle during the computation. The FLIP is focusing more on storage, if
> I
> > am not mistaken. Have you considered using BUCKET BY directly?
> >
> > 2. According to the FLIP: " CREATE TABLE MyTable (uid BIGINT, name
> STRING)
> > DISTRIBUTED BY HASH(uid) INTO 6 BUCKETS
> >
> > - For advanced users, the algorithm can be defined explicitly.
> > - Currently, either HASH() or RANGE().
> >
> > "
> > Did you mean users can use their own algorithm? How to do it?
> >
> > Best regards,
> > Jing
> >
> > On Mon, Oct 30, 2023 at 11:13 AM Timo Walther 
> wrote:
> >
> >> Let me reply to the feedback from Yunfan:
> >>
> >>   > Distribute by in DML is also supported by Hive
> >>
> >> I see DISTRIBUTED BY and DISTRIBUTE BY as two separate discussions. This
> >> discussion is about DDL. For DDL, we have more freedom as every vendor
> >> has custom syntax for CREATE TABLE clauses. Furthermore, this is tightly
> >> connector to the connector implementation, not the engine. However, for
> >> DML we need to watch out for standard compliance and introduce changes
> >> with high caution.
> >>
> >> How a LookupTableSource interprets the DISTRIBUTED BY is
> >> connector-dependent in my opinion. In general this FLIP is a sink
> >> ability, but we could have a follow FLIP that helps in distributing load
> >> of lookup joins.
> >>
> >>   > to avoid data skew problem
> >>
> >> I understand the use case and that it is important to solve it
> >> eventually. Maybe a solution might be to introduce helper Polymorphic
> >> Table Functions [1] in the future instead of new syntax.
> >>
> >> [1]
> >>
> >>
> https://www.ifis.uni-luebeck.de/~moeller/Lectures/WS-19-20/NDBDM/12-Literature/Michels-SQL-2016.pdf
> >>
> >>
> >> Let me reply to the feedback from Benchao:
> >>
> >>   > Do you think it's useful to add some extensibility for the hash
> >> strategy
> >>
> >> The hash strategy is fully determined by the connector, not the Flink
> >> SQL engine. We are not using Flink's hash strategy in any way. If the
> >> hash strategy for the regular Flink file system connector should be
> >> changed, this should be expressed via config option. Otherwise we should
> >> offer a dedicated `hive-filesystem` or `spark-filesystem` connector.
> >>
> >> Regards,
> >> Timo
> >>
> >>
> >> On 30.10.23 10:44, Timo Walther wrote:
> >>> Hi Jark,
> >>>
> >>> my intention was to avoid too complex syntax in the first version. In
> >>> the past years, we could enable use cases also without this clause, so
> >>> we should be careful with overloading it with too functionality in the
> >>> first version. We can still iterate on it later, the interfaces are
> >>> flexible enough to support more in the future.
> >>>
> >>> I agree that maybe an explicit HASH and RANGE doesn't harm. Also making
> >>> the bucket number optional.
> >>>
> >>> I updated the FLIP accordingly. Now the SupportsBucketing interface
> >>> declares more methods that help in validation and proving helpful error
> >>> messages to users.
> >>>
> >>> Let me know what you think.
> >>>
> >>> Regards,
> >>> Timo
> >>>
> >>>
> >>> On 27.10.23 10:20, Jark Wu wrote:
>  Hi Timo,
> 
>  Thanks for starting this discussion. I really like it!
>  The FLIP is already in good shape, I only have some minor comments.
> 
>  1. Could we also support HASH and RANGE distribution kind on the DDL
>  syntax?
>  I noticed that HASH and UNKNOWN are introduced in the Java API, but
>  not in
>  the syntax.
> 
>  2. Can we make "INTO n BUCKETS" optional in CREATE TABLE and ALTER
> >> TABLE?
>  Some storage engines support automatically determining the bucket
> number
>  based on
>  the cluster resources

Re: [VOTE] Release flink-connector-aws, v4.2.0 release candidate #1

Re: Pointers to computational models of Flink CEP

[jira] [Created] (FLINK-33422) Add restore tests for Calc node

[jira] [Created] (FLINK-33421) Implement ExecNode Restore Tests

[ANNOUNCE] Apache Flink Kafka Connectors 3.0.1 released

[jira] [Created] (FLINK-33420) Run e2e test fails Intermetiently with ClientCoordinationHandler : Unhandled exception

Pointers to computational models of Flink CEP

Re: [DISCUSS] AWS Connectors v4.2.0 release + 1.18 support

[VOTE] Release flink-connector-aws, v4.2.0 release candidate #1

[jira] [Created] (FLINK-33419) Port PROCTIME/ROWTIME functions to the new inference stack

[jira] [Created] (FLINK-33418) SqlGatewayE2ECase failed due to ConnectException

Re: Request to release flink 1.6.3

Re: Request to release flink 1.6.3

Re: [DISCUSS] FLIP-379: Dynamic source parallelism inference for batch jobs

Re: [DISCUSS] FLIP-376: Add DISTRIBUTED BY clause

[jira] [Created] (FLINK-33417) Update netty version to 4.1.83 for flink-shaded

[jira] [Created] (FLINK-33416) FineGrainedSlotManagerTest failed with fatal error

[jira] [Created] (FLINK-33415) HAJobRunOnHadoopS3FileSystemITCase failed due to NoSuchMethodError

[jira] [Created] (FLINK-33414) MiniClusterITCase.testHandleStreamingJobsWhenNotEnoughSlot fails due to unexpected TimeoutException

[jira] [Created] (FLINK-33413) Bump Avro in AWS Connectors to address CVE-2023-39410

Re: [DISCUSS] Kubernetes Operator 1.7.0 release planning

Re: [DISCUSS][FLINK-33240] Document deprecated options as well

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

[jira] [Created] (FLINK-33412) Implement type inference for reinterpret_cast function

[jira] [Created] (FLINK-33411) Implement type inference for window properties functions

[jira] [Created] (FLINK-33410) Implement type inference for Over offsets functions

[jira] [Created] (FLINK-33409) Bump Guava to address CVE-2020-8908/CVE-2023-2976

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

[jira] [Created] (FLINK-33408) Fixing the container vulnerability by upgrade the SnakeYaml Maven dependency in flink-kubernetes module.

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

[jira] [Created] (FLINK-33407) Port time functions to the new type inference stack

Re: [DISCUSS] Kubernetes Operator 1.7.0 release planning

[DISCUSS] FLIP-379: Dynamic source parallelism inference for batch jobs

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

[jira] [Created] (FLINK-33406) Flink Job failed due to losing connection from ZK server

Re: [DISCUSS] FLIP-376: Add DISTRIBUTED BY clause

36 matches

Site Navigation

Mail list logo

Footer information