Re: [ANNOUNCE] Apache Flink 1.19.1 released
Hi Rui, Thanks for flagging - this is the official docker repo, which requires action from external party to merge our PR[1] and replicate the Docker images. The Docker image is available on the ASF maintained Docker repo [2] [1] https://github.com/docker-library/official-images/pull/17013 [2] https://hub.docker.com/r/apache/flink/tags?page=1=1.19.1 On Wed, Jun 19, 2024 at 2:52 AM Rui Fan <1996fan...@gmail.com> wrote: > Thanks Hong for driving this release! > > I clicked the library/flink docker images link[1] from > 1.19.1-release-announcement[2]. > And I cannot find the 1.19.1 images there, I'm not sure if it's caused by > delay or > some steps are missed. > > [1] https://hub.docker.com/_/flink/tags?page=1=1.19.1 > [2] > > https://flink.apache.org/2024/06/14/apache-flink-1.19.1-release-announcement/ > > Best, > Rui > > On Wed, Jun 19, 2024 at 9:49 AM Leonard Xu wrote: > > > Congratulations! Thanks Hong for the release work and all involved! > > > > Best, > > Leonard > > > > > 2024年6月19日 上午4:20，Hong Liang 写道： > > > > > > The Apache Flink community is very happy to announce the release of > > Apache > > > Flink 1.19.1, which is the first bugfix release for the Apache Flink > 1.19 > > > series. > > > > > > Apache Flink® is an open-source stream processing framework for > > > distributed, high-performing, always-available, and accurate data > > streaming > > > applications. > > > > > > The release is available for download at: > > > https://flink.apache.org/downloads.html > > > > > > Please check out the release blog post for an overview of the > > improvements > > > for this bugfix release: > > > > > > https://flink.apache.org/2024/06/14/apache-flink-1.19.1-release-announcement/ > > > > > > The full release notes are available in Jira: > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12354399=12315522 > > > > > > We would like to thank all contributors of the Apache Flink community > who > > > made this release possible! > > > > > > Feel free to reach out to the release managers (or respond to this > > thread) > > > with feedback on the release process. Our goal is to constantly improve > > the > > > release process. Feedback on what could be improved or things that > didn't > > > go so well are appreciated. > > > > > > Regards, > > > Hong > > > > >
[ANNOUNCE] Apache Flink 1.19.1 released
The Apache Flink community is very happy to announce the release of Apache Flink 1.19.1, which is the first bugfix release for the Apache Flink 1.19 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https://flink.apache.org/downloads.html Please check out the release blog post for an overview of the improvements for this bugfix release: https://flink.apache.org/2024/06/14/apache-flink-1.19.1-release-announcement/ The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12354399=12315522 We would like to thank all contributors of the Apache Flink community who made this release possible! Feel free to reach out to the release managers (or respond to this thread) with feedback on the release process. Our goal is to constantly improve the release process. Feedback on what could be improved or things that didn't go so well are appreciated. Regards, Hong
Re: [VOTE] Release 1.19.1, release candidate #1
Thanks all for the testing and votes. The RC is approved and this thread is now closed. See results in [1]. [1] https://lists.apache.org/thread/yqr3jv4wr85brnz2ylzqo9pqn453jqvq Regards, Hong On Tue, Jun 11, 2024 at 9:39 AM Hang Ruan wrote: > +1(non-binding) > > - Verified signatures > - Verified hashsums > - Checked Github release tag > - Source archives with no binary files > - Reviewed the flink-web PR > - Checked the jar build with jdk 1.8 > > Best, > Hang > > gongzhongqiang 于2024年6月11日周二 15:53写道： > > > +1(non-binding) > > > > - Verified signatures and sha512 > > - Checked Github release tag exsit > > - Source archives with no binary files > > - Build the source with jdk8 on ubuntu 22.04 succeed > > - Reviewed the flink-web PR > > > > Best, > > Zhongqiang Gong > > > > Hong Liang 于2024年6月6日周四 23:39写道： > > > > > Hi everyone, > > > Please review and vote on the release candidate #1 for the flink > v1.19.1, > > > as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > > > > The complete staging area is available for your review, which includes: > > > * JIRA release notes [1], > > > * the official Apache source release and binary convenience releases to > > be > > > deployed to dist.apache.org [2], which are signed with the key with > > > fingerprint B78A5EA1 [3], > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > * source code tag "release-1.19.1-rc1" [5], > > > * website pull request listing the new release and adding announcement > > blog > > > post [6]. > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Thanks, > > > Hong > > > > > > [1] > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354399 > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.19.1-rc1/ > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > [4] > > > > https://repository.apache.org/content/repositories/orgapacheflink-1736/ > > > [5] https://github.com/apache/flink/releases/tag/release-1.19.1-rc1 > > > [6] https://github.com/apache/flink-web/pull/745 > > > > > >
[RESULT] [VOTE] Release 1.19.1, release candidate #1
Hi all, I'm happy to announce that we have unanimously approved this release. [1] There are 11 approving votes, 5 of which are binding: * Rui Fan (binding) * Xiqian Yu (non-binding) * Weijie Guo (binding) * Jeyhun (non-binding) * Ahmed Hamdy (non-binding) * Xintong Song (binding) * Matthias Pohl (binding) * Sergey Nuyanzin (non-binding) * Leonard Xu (binding) * Zhongqiang Gong (non-binding) * Hang Ruan (non-binding) There are no disapproving votes. Thanks everyone! [1] https://lists.apache.org/thread/hrptj22y6rjt61flzdzngxdsw134osk4 Regards, Hong
Re: [VOTE] Release 1.19.1, release candidate #1
Thanks for testing the release candidate, everyone. Nice to see coverage on different types of testing being done. I've addressed the comments on the web PR - thanks Rui Fan for good comments, and for the reminder from Ahmed :) We have <24 hours on the vote wait time, and still waiting on 1 more binding vote! Regards, Hong On Sat, Jun 8, 2024 at 11:33 PM Ahmed Hamdy wrote: > Hi Hong, > Thanks for driving > > +1 (non-binding) > > - Verified signatures and hashes > - Checked github release tag > - Verified licenses > - Checked that the source code does not contain binaries > - Reviewed Web PR, nit: Could we address the comment of adding FLINK-34633 > in the release > > > Best Regards > Ahmed Hamdy > > > On Sat, 8 Jun 2024 at 22:22, Jeyhun Karimov wrote: > > > Hi Hong, > > > > Thanks for driving the release. > > +1 (non-binding) > > > > - Verified gpg signature > > - Reviewed the PR > > - Verified sha512 > > - Checked github release tag > > - Checked that the source code does not contain binaries > > > > Regards, > > Jeyhun > > > > On Sat, Jun 8, 2024 at 1:52 PM weijie guo > > wrote: > > > > > Thanks Hong! > > > > > > +1(binding) > > > > > > - Verified gpg signature > > > - Verified sha512 hash > > > - Checked gh release tag > > > - Checked all artifacts deployed to maven repo > > > - Ran a simple wordcount job on local standalone cluster > > > - Compiled from source code with JDK 1.8.0_291. > > > > > > Best regards, > > > > > > Weijie > > > > > > > > > Xiqian YU 于2024年6月7日周五 18:23写道： > > > > > > > +1 (non-binding) > > > > > > > > > > > > * Checked download links & release tags > > > > * Verified that package checksums matched > > > > * Compiled Flink from source code with JDK 8 / 11 > > > > * Ran E2e data integration test jobs on local cluster > > > > > > > > Regards, > > > > yux > > > > > > > > De : Rui Fan <1996fan...@gmail.com> > > > > Date : vendredi, 7 juin 2024 à 17:14 > > > > À : dev@flink.apache.org > > > > Objet : Re: [VOTE] Release 1.19.1, release candidate #1 > > > > +1(binding) > > > > > > > > - Reviewed the flink-web PR (Left some comments) > > > > - Checked Github release tag > > > > - Verified signatures > > > > - Verified sha512 (hashsums) > > > > - The source archives do not contain any binaries > > > > - Build the source with Maven 3 and java8 (Checked the license as > well) > > > > - Start the cluster locally with jdk8, and run the > StateMachineExample > > > job, > > > > it works fine. > > > > > > > > Best, > > > > Rui > > > > > > > > On Thu, Jun 6, 2024 at 11:39 PM Hong Liang wrote: > > > > > > > > > Hi everyone, > > > > > Please review and vote on the release candidate #1 for the flink > > > v1.19.1, > > > > > as follows: > > > > > [ ] +1, Approve the release > > > > > [ ] -1, Do not approve the release (please provide specific > comments) > > > > > > > > > > > > > > > The complete staging area is available for your review, which > > includes: > > > > > * JIRA release notes [1], > > > > > * the official Apache source release and binary convenience > releases > > to > > > > be > > > > > deployed to dist.apache.org [2], which are signed with the key > with > > > > > fingerprint B78A5EA1 [3], > > > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > > > * source code tag "release-1.19.1-rc1" [5], > > > > > * website pull request listing the new release and adding > > announcement > > > > blog > > > > > post [6]. > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > majority > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > Thanks, > > > > > Hong > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354399 > > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.19.1-rc1/ > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > > > [4] > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheflink-1736/ > > > > > [5] > https://github.com/apache/flink/releases/tag/release-1.19.1-rc1 > > > > > [6] https://github.com/apache/flink-web/pull/745 > > > > > > > > > > > > > > >
[VOTE] Release 1.19.1, release candidate #1
Hi everyone, Please review and vote on the release candidate #1 for the flink v1.19.1, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint B78A5EA1 [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tag "release-1.19.1-rc1" [5], * website pull request listing the new release and adding announcement blog post [6]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Hong [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354399 [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.19.1-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1736/ [5] https://github.com/apache/flink/releases/tag/release-1.19.1-rc1 [6] https://github.com/apache/flink-web/pull/745
[jira] [Created] (FLINK-35532) Prevent Cross-Site Authentication (XSA) attacks on Flink dashboard
Hong Liang Teoh created FLINK-35532:
---
Summary: Prevent Cross-Site Authentication (XSA) attacks on Flink
dashboard
Key: FLINK-35532
URL: https://issues.apache.org/jira/browse/FLINK-35532
Project: Flink
Issue Type: Technical Debt
Components: Runtime / Web Frontend
Affects Versions: 1.19.0, 1.19.1
Reporter: Hong Liang Teoh
Assignee: Hong Liang Teoh
Fix For: 1.19.2
As part of FLINK-33325, we introduced a new tab on the Flink dashboard to
trigger the async profiler on the JobManager and TaskManager.
The HTML component introduced links out to async profiler page on Github ->
[https://github.com/async-profiler/async-profiler/wiki].
However, the anchor element introduced does not follow best practices around
preventing XSA attacks, by setting up the below:
{code:java}
target="_blank" rel="noopener noreferrer"{code}
We should add these attributes as best practice!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35512) ArtifactFetchManagerTest unit tests fail
Hong Liang Teoh created FLINK-35512:
---
Summary: ArtifactFetchManagerTest unit tests fail
Key: FLINK-35512
URL: https://issues.apache.org/jira/browse/FLINK-35512
Project: Flink
Issue Type: Technical Debt
Affects Versions: 1.19.1
Reporter: Hong Liang Teoh
Fix For: 1.19.1
The below three tests from *ArtifactFetchManagerTest* seem to fail consistently:
* ArtifactFetchManagerTest.testFileSystemFetchWithAdditionalUri
* ArtifactFetchManagerTest.testMixedArtifactFetch
* ArtifactFetchManagerTest.testHttpFetch
The error printed is
{code:java}
java.lang.AssertionError:
Expecting actual not to be empty
at
org.apache.flink.client.program.artifact.ArtifactFetchManagerTest.getFlinkClientsJar(ArtifactFetchManagerTest.java:248)
at
org.apache.flink.client.program.artifact.ArtifactFetchManagerTest.testMixedArtifactFetch(ArtifactFetchManagerTest.java:146)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[DISCUSS] Flink 1.19.1 release
Hi all, I would like to discuss creating a new 1.19 patch release (1.19.1). The last 1.19 release is two months old, and since then, 35 tickets have been closed [1], of which 7 are blocker/critical [2]. There are 5 items open at the moment; 2 of which are not blocking because they affect previous Flink versions as well [3][4], 3 of which have approved PRs [5][6][7], and will be merged soon. I propose that we create a release after [5][6][7] have been closed. Please let me know if there are any unresolved issues you'd like to be included in this release. If the community agrees to create this new patch release, I'm happy to volunteer as the release manager, if another PMC (Danny?) is happy to assist with the PMC-related release steps! Best regards, Hong [1] https://issues.apache.org/jira/projects/FLINK/versions/12354399 [2] https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.19.1%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC [3] https://issues.apache.org/jira/browse/FLINK-34672 [4] https://issues.apache.org/jira/browse/FLINK-35229 [5] https://issues.apache.org/jira/browse/FLINK-35358 [6] https://issues.apache.org/jira/browse/FLINK-34746 [7] https://issues.apache.org/jira/browse/FLINK-34379
Re: [VOTE] FLIP-451: Introduce timeout configuration to AsyncSink
+1 (binding) Thanks Ahmed On Tue, May 14, 2024 at 11:51 AM David Radley wrote: > Thanks for the clarification Ahmed > > +1 (non-binding) > > From: Ahmed Hamdy > Date: Monday, 13 May 2024 at 19:58 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [VOTE] FLIP-451: Introduce timeout configuration > to AsyncSink > Thanks David, > I have replied to your question in the discussion thread. > Best Regards > Ahmed Hamdy > > > On Mon, 13 May 2024 at 16:21, David Radley > wrote: > > > Hi, > > I raised a question on the discussion thread, around retriable errors, as > > a possible alternative, > > Kind regards, David. > > > > > > From: Aleksandr Pilipenko > > Date: Monday, 13 May 2024 at 16:07 > > To: dev@flink.apache.org > > Subject: [EXTERNAL] Re: [VOTE] FLIP-451: Introduce timeout configuration > > to AsyncSink > > Thanks for driving this! > > > > +1 (non-binding) > > > > Thanks, > > Aleksandr > > > > On Mon, 13 May 2024 at 14:08, > > wrote: > > > > > Thanks Ahmed! > > > > > > +1 non binding > > > On May 13, 2024 at 12:40 +0200, Jeyhun Karimov , > > > wrote: > > > > Thanks for driving this Ahmed. > > > > > > > > +1 (non-binding) > > > > > > > > Regards, > > > > Jeyhun > > > > > > > > On Mon, May 13, 2024 at 12:37 PM Muhammet Orazov > > > > wrote: > > > > > > > > > Thanks Ahmed, +1 (non-binding) > > > > > > > > > > Best, > > > > > Muhammet > > > > > > > > > > On 2024-05-13 09:50, Ahmed Hamdy wrote: > > > > > > > Hi all, > > > > > > > > > > > > > > Thanks for the feedback on the discussion thread[1], I would > like > > > to > > > > > > > start > > > > > > > a vote on FLIP-451[2]: Introduce timeout configuration to > > AsyncSink > > > > > > > > > > > > > > The vote will be open for at least 72 hours unless there is an > > > > > > > objection or > > > > > > > insufficient votes. > > > > > > > > > > > > > > 1- > > https://lists.apache.org/thread/ft7wcw7kyftvww25n5fm4l925tlgdfg0 > > > > > > > 2- > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Introduce+timeout+configuration+to+AsyncSink+API > > > > > > > Best Regards > > > > > > > Ahmed Hamdy > > > > > > > > > > > > Unless otherwise stated above: > > > > IBM United Kingdom Limited > > Registered in England and Wales with number 741598 > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU > > > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU >
Re: [DISCUSS] FLIP-451: Refactor Async sink API
Hi Ahmed, Thanks for putting this together! Should we still be marking getFatalExceptionCons() as @Deprecated in this FLIP, if we are not providing a replacement? Regards, Hong On Mon, May 13, 2024 at 7:58 PM Ahmed Hamdy wrote: > Hi David, > yes there error classification was initially left to sink implementers to > handle while we provided utilities to classify[1] and bubble up[2] fatal > exceptions to avoid retrying them. > Additionally some sink implementations provide an option to short circuit > the failures by exposing a `failOnError` flag as in KinesisStreamsSink[3], > however this FLIP scope doesn't include any changes for retry mechanisms. > > 1- > > https://github.com/apache/flink/blob/015867803ff0c128b1c67064c41f37ca0731ed86/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/throwable/FatalExceptionClassifier.java#L32 > 2- > > https://github.com/apache/flink/blob/015867803ff0c128b1c67064c41f37ca0731ed86/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java#L533 > 3- > > https://github.com/apache/flink-connector-aws/blob/c6e0abb65a0e51b40dd218b890a111886fbf797f/flink-connector-aws/flink-connector-aws-kinesis-streams/src/main/java/org/apache/flink/connector/kinesis/sink/KinesisStreamsSinkWriter.java#L106 > > Best Regards > Ahmed Hamdy > > > On Mon, 13 May 2024 at 16:20, David Radley > wrote: > > > Hi, > > I wonder if the way that the async request fails could be a retriable or > > non-retriable error, so it would retry only for retriable (transient) > > errors (like IOExceptions) . I see some talk on the internet around > > retriable SQL errors. > > If this was the case then we may need configuration to limit the number > > of retries of retriable errors. > > Kind regards, David > > > > > > From: Muhammet Orazov > > Date: Monday, 13 May 2024 at 10:30 > > To: dev@flink.apache.org > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-451: Refactor Async sink API > > Great, thanks for clarifying! > > > > Best, > > Muhammet > > > > > > On 2024-05-06 13:40, Ahmed Hamdy wrote: > > > Hi Muhammet, > > > Thanks for the feedback. > > > > > >> Could you please add more here why it is harder? Would the > > >> `completeExceptionally` > > >> method be related to it? Maybe you can add usage example for it also. > > >> > > > > > > this is mainly due to the current implementation of fatal exception > > > failures which depends on base `getFatalExceptionConsumer` method that > > > is > > > decoupled from the actual called method `submitRequestEntries`, Since > > > this > > > is now not the primary concern of the FLIP, I have removed it from the > > > motivation so that the scope is defined around introducing the timeout > > > configuration. > > > > > >> Should we add a list of possible connectors that this FLIP would > > >> improve? > > > > > > Good call, I have added under migration plan. > > > > > > Best Regards > > > Ahmed Hamdy > > > > > > > > > On Mon, 6 May 2024 at 08:49, Muhammet Orazov > > > wrote: > > > > > >> Hey Ahmed, > > >> > > >> Thanks for the FLIP! +1 (non-binding) > > >> > > >> > Additionally the current interface for passing fatal exceptions and > > >> > retrying records relies on java consumers which makes it harder to > > >> > understand. > > >> > > >> Could you please add more here why it is harder? Would the > > >> `completeExceptionally` > > >> method be related to it? Maybe you can add usage example for it also. > > >> > > >> > we should proceed by adding support in all supporting connector > repos. > > >> > > >> Should we add list of possible connectors that this FLIP would > > >> improve? > > >> > > >> Best, > > >> Muhammet > > >> > > >> > > >> On 2024-04-29 14:08, Ahmed Hamdy wrote: > > >> > Hi all, > > >> > I would like to start a discussion on FLIP-451[1] > > >> > The proposal comes on encountering a couple of issues while working > > >> > with > > >> > implementers for Async Sink. > > >> > The FLIP mainly proposes a new API similar to AsyncFunction and > > >> > ResultFuture as well as introducing timeout handling for AsyncSink > > >> > requests. > > >> > The FLIP targets 1.20 with backward compatible changes and we should > > >> > proceed by adding support in all supporting connector repos. > > >> > > > >> > 1- > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Refactor+Async+Sink+API > > >> > Best Regards > > >> > Ahmed Hamdy > > >> > > > > Unless otherwise stated above: > > > > IBM United Kingdom Limited > > Registered in England and Wales with number 741598 > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU > > >
Re: [Vote] FLIP-438: Amazon SQS Sink Connector
Hi Priya, Great to see the FLIP has passed the vote. It would be good to report the final result of the vote at the end of the thread, listing the binding / non-binding votes in an email. See example here [1] [2] Regards, Hong [1] https://lists.apache.org/thread/3sj88kk0104vzj4hklfgbn3rpdnjxj8v [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals On Mon, May 6, 2024 at 1:13 AM Dhingra, Priya wrote: > Thank you all! > > > Closing the vote. Will update the Flip with Jira link for tracking > implementation. > >
[jira] [Created] (FLINK-35299) FlinkKinesisConsumer does not respect StreamInitialPosition for new Kinesis Stream when restoring from snapshot
Hong Liang Teoh created FLINK-35299: --- Summary: FlinkKinesisConsumer does not respect StreamInitialPosition for new Kinesis Stream when restoring from snapshot Key: FLINK-35299 URL: https://issues.apache.org/jira/browse/FLINK-35299 Project: Flink Issue Type: Bug Components: Connectors / Kinesis Affects Versions: aws-connector-4.2.0 Reporter: Hong Liang Teoh Fix For: aws-connector-4.4.0 h3. What The FlinkKinesisConsumer allows users to read from [multiple Kinesis Streams|https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L224]. Users can also specify a STREAM_INITIAL_POSITION, which configures if the consumer starts reading the stream from TRIM_HORIZON / LATEST / AT_TIMESTAMP. When restoring the Kinesis Consumer from an existing snapshot, users can configure the consumer to read from additional Kinesis Streams. The expected behavior would be for the FlinkKinesisConsumer to start reading from the additional Kinesis Streams respecting the STREAM_INITIAL_POSITION configuration. However, we find that it currently reads from TRIM_HORIZON. This is surprising behavior and should be corrected. h3. Why Principle of Least Astonishment h3. How We recommend that we reconstruct the previously seen streams by iterating through the [sequenceNumsStateForCheckpoint in FlinkKinesisConsumer#initializeState()|https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L454]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] FLIP-444: Native file copy support
Hi Piotr, Thanks for the FLIP! Nice to see work to improve the filesystem performance. +1 to future work to improve the upload speed as well. This would be useful for jobs with large state and high Async checkpointing times. Some thoughts on the configuration, it might be good for us to introduce 2x points of configurability for future proofing: 1/ Configure the implementation of PathsCopyingFileSystem used, maybe by config, or by ServiceResources (this would allow us to use this for alternative clouds/Implement S3 SDKv2 support if we want this in the future). Also this could be used as a feature flag to determine if we should be using this new native file copy support. 2/ Configure the location of the s5cmd binary (version control etc.), as you have mentioned in the FLIP. Regards, Hong On Thu, May 2, 2024 at 9:40 AM Muhammet Orazov wrote: > Hey Piotr, > > Thanks for the proposal! It would be great improvement! > > Some questions from my side: > > > In order to configure s5cmd Flink’s user would need > > to specify path to the s5cmd binary. > > Could you please also add the configuration property > for this? An example showing how users would set this > parameter would be helpful. > > Would this affect any filesystem connectors that use > FileSystem[1][2] dependencies? > > Best, > Muhammet > > [1]: > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/ > [2]: > > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/filesystem/ > > On 2024-04-30 13:15, Piotr Nowojski wrote: > > Hi all! > > > > I would like to put under discussion: > > > > FLIP-444: Native file copy support > > https://cwiki.apache.org/confluence/x/rAn9EQ > > > > This proposal aims to speed up Flink recovery times, by speeding up > > state > > download times. However in the future, the same mechanism could be also > > used to speed up state uploading (checkpointing/savepointing). > > > > I'm curious to hear your thoughts. > > > > Best, > > Piotrek >
Re: [Vote] FLIP-438: Amazon SQS Sink Connector
+1 (binding) Thanks Priya for driving this! This has been a requested feature for a while now, and will benefit the community :) Hong On Tue, Apr 16, 2024 at 3:23 AM Muhammet Orazov wrote: > +1 (non-binding) > > Thanks Priya for the FLIP and driving it! > > Best, > Muhammet > > On 2024-04-12 21:56, Dhingra, Priya wrote: > > Hi devs, > > > > > > > > Thank you to everyone for the feedback on FLIP-438: Amazon SQS Sink > > Connector< > https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector > > > > > > > > > > I would like to start a vote for it. The vote will be open for at least > > 72 > > > > hours unless there is an objection or not enough votes. > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector > > > > Regards > > Priya >
Re: [VOTE] FLIP-418: Show data skew score on Flink Dashboard
+1 (binding) Thank you for driving this Emre! This is a good step towards better user experience when diagnosing performance issues with Flink jobs. Best, Hong On Wed, Jan 31, 2024 at 3:00 AM Aleksandr Pilipenko wrote: > Thanks for the FLIP! > > +1 (non-binding) > > Best, > Aleksandr > > On Mon, 29 Jan 2024 at 10:11, Kartoglu, Emre > wrote: > > > Hello, > > > > I'd like to call votes on FLIP-418: Show data skew score on Flink > > Dashboard. > > > > FLIP: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-418%3A+Show+data+skew+score+on+Flink+Dashboard > > Discussion: > > https://lists.apache.org/thread/m5ockoork0h2zr78h77dcrn71rbt35ql > > > > Kind regards, > > Emre > > > > >
Security fixes for Flink 1.18 (flink-shaded)
Hi all, Recently, we detected some active CVEs on the flink-shaded-guava and flink-shaded-zookeeper package used in Flink 1.18. Since Flink 1.18 is still in support for security fixes, we should consider fixing this. However, since the vulnerable package is coming from flink-shaded, I wanted to check if there are thoughts from the community around releasing a patch version of flink-shaded. Problem: Flink 1.18 uses guava 31.1-jre from flink-shaded-guava 17.0, which is affected by CVE-2023-2976 (HIGH) [1] and CVE-2020-8908 (LOW) [2]. Flink 1.18 also uses zookeeper 3.7.1, which is affected by CVE-2023-44981 (CRITICAL) [3]. To fix, I can think of two options: Option 1: Upgrade Flink 1.18 to use flink.shaded.version 18.0. This is easiest as we can backport the change for Flink 1.19 directly (after the performance regression is addressed) [4]. However, there are also upgrades to jackson, asm and netty in flink.shaded.version 1.18. Option 2: Release flink.shaded.version 17.1, with just a bump in zookeeper and guava versions. Then, upgrade Flink 1.18 to use this new flink.shaded.version 17.1. This is harder, but keeps the changes contained and minimal. Given the version bump is on flink-shaded, which is relocated to keep the usage of libraries contained within the flink runtime itself, I am inclined to go with Option 1, even though the change is slightly larger than just the security fixes. Do people have any objections? Regards, Hong [1] https://nvd.nist.gov/vuln/detail/CVE-2023-2976 [2] https://nvd.nist.gov/vuln/detail/CVE-2020-8908 [3] https://nvd.nist.gov/vuln/detail/CVE-2023-44981 [4] https://issues.apache.org/jira/browse/FLINK-33705
[jira] [Created] (FLINK-33872) Checkpoint history does not display for completed jobs
Hong Liang Teoh created FLINK-33872: --- Summary: Checkpoint history does not display for completed jobs Key: FLINK-33872 URL: https://issues.apache.org/jira/browse/FLINK-33872 Project: Flink Issue Type: Bug Components: Runtime / REST Affects Versions: 1.18.0 Reporter: Hong Liang Teoh Fix For: 1.19.0, 1.18.2 Attachments: image-2023-12-18-11-37-11-914.png, image-2023-12-18-11-37-29-596.png Prior to https://issues.apache.org/jira/browse/FLINK-32469, we see checkpoint history for completed jobs (CANCELED, FAILED, FINISHED). After https://issues.apache.org/jira/browse/FLINK-32469, the checkpoint history does not show up for completed jobs. *Reproduction steps:* # Start a Flink cluster. # Submit a job with checkpointing enabled. # Wait until at least 1 checkpoint completes. # Cancel job. # Open the Flink dashboard > Job > Checkpoints > History. We will see log line in JobManager saying "FlinkJobNotFoundException: Could not find Flink job ( )" *Snapshot of failure:* When job is running, we can see checkpoints. !image-2023-12-18-11-37-11-914.png! When job has been CANCELLED, we no longer see checkpoints data. !image-2023-12-18-11-37-29-596.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] Release Flink 1.18.1
+1. Thanks Jing for driving this. Hong On Mon, Dec 11, 2023 at 2:27 AM Yun Tang wrote: > Thanks Jing for driving 1.18.1 release, +1 for this. > > > Best > Yun Tang > > From: Rui Fan <1996fan...@gmail.com> > Sent: Saturday, December 9, 2023 21:46 > To: dev@flink.apache.org > Subject: Re: [DISCUSS] Release Flink 1.18.1 > > Thanks Jing for driving this release, +1 > > Best, > Rui > > On Sat, Dec 9, 2023 at 7:33 AM Leonard Xu wrote: > > > Thanks Jing for driving this release, +1 > > > > Best, > > Leonard > > > > > 2023年12月9日 上午1:23，Danny Cranmer 写道： > > > > > > +1 > > > > > > Thanks for driving this > > > > > > On Fri, 8 Dec 2023, 12:05 Timo Walther, wrote: > > > > > >> Thanks for taking care of this Jing. > > >> > > >> +1 to release 1.18.1 for this. > > >> > > >> Cheers, > > >> Timo > > >> > > >> > > >> On 08.12.23 10:00, Benchao Li wrote: > > >>> I've merged FLINK-33313 to release-1.18 branch. > > >>> > > >>> Péter Váry 于2023年12月8日周五 16:56写道： > > > > Hi Jing, > > Thanks for taking care of this! > > +1 (non-binding) > > Peter > > > > Sergey Nuyanzin ezt írta (időpont: 2023. dec. > > >> 8., P, > > 9:36): > > > > > Thanks Jing driving it > > > +1 > > > > > > also +1 to include FLINK-33313 mentioned by Benchao Li > > > > > > On Fri, Dec 8, 2023 at 9:17 AM Benchao Li > > >> wrote: > > > > > >> Thanks Jing for driving 1.18.1 releasing. > > >> > > >> I would like to include FLINK-33313[1] in 1.18.1, it's just a > > bugfix, > > >> not a blocker, but it's already merged into master, I plan to > merge > > it > > >> to 1.8/1.7 branches today after the CI passes. > > >> > > >> [1] https://issues.apache.org/jira/browse/FLINK-33313 > > >> > > >> Jing Ge 于2023年12月8日周五 16:06写道： > > >>> > > >>> Hi all, > > >>> > > >>> I would like to discuss creating a new 1.18 patch release > (1.18.1). > > >> The > > >>> last 1.18 release is nearly two months old, and since then, 37 > > >> tickets > > >> have > > >>> been closed [1], of which 6 are blocker/critical [2]. Some of > them > > >> are > > >>> quite important, such as FLINK-33598 [3] > > >>> > > >>> Most urgent and important one is FLINK-33523 [4] and according to > > the > > >>> discussion thread[5] on the ML, 1.18.1 should/must be released > asap > > > after > > >>> the breaking change commit has been reverted. > > >>> > > >>> I am not aware of any other unresolved blockers and there are no > > >> in-progress > > >>> tickets [6]. > > >>> Please let me know if there are any issues you'd like to be > > included > > >> in > > >>> this release but still not merged. > > >>> > > >>> If the community agrees to create this new patch release, I could > > >>> volunteer as the release manager. > > >>> > > >>> Best regards, > > >>> Jing > > >>> > > >>> [1] > > >>> > > >> > > > > > >> > > > https://issues.apache.org/jira/browse/FLINK-33567?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.18.1%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC > > >>> [2] > > >>> > > >> > > > > > >> > > > https://issues.apache.org/jira/browse/FLINK-33693?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.18.1%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC > > >>> [3] https://issues.apache.org/jira/browse/FLINK-33598 > > >>> [4] https://issues.apache.org/jira/browse/FLINK-33523 > > >>> [5] > > https://lists.apache.org/thread/m4c879y8mb7hbn2kkjh9h3d8g1jphh3j > > >>> [6] > > https://issues.apache.org/jira/projects/FLINK/versions/12353640 > > >>> Thanks, > > >> > > >> > > >> > > >> -- > > >> > > >> Best, > > >> Benchao Li > > >> > > > > > > > > > -- > > > Best regards, > > > Sergey > > > > > >>> > > >>> > > >>> > > >> > > >> > > > > >
[jira] [Created] (FLINK-33259) flink-connector-aws should use/extend the common connector workflow
Hong Liang Teoh created FLINK-33259: --- Summary: flink-connector-aws should use/extend the common connector workflow Key: FLINK-33259 URL: https://issues.apache.org/jira/browse/FLINK-33259 Project: Flink Issue Type: Technical Debt Reporter: Hong Liang Teoh We should use the common ci github workflow. [https://github.com/apache/flink-connector-shared-utils/blob/ci_utils/.github/workflows/ci.yml] Example used in flink-connector-elasticsearch [https://github.com/apache/flink-connector-elasticsearch/blob/main/.github/workflows/push_pr.yml] This improves our operational stance because we will now inherit any improvements/changes to the main ci workflow file -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33167) Run IT tests against Kinesalite if AWS credentials are not present
Hong Liang Teoh created FLINK-33167: --- Summary: Run IT tests against Kinesalite if AWS credentials are not present Key: FLINK-33167 URL: https://issues.apache.org/jira/browse/FLINK-33167 Project: Flink Issue Type: Technical Debt Components: Connectors / AWS, Connectors / Kinesis Reporter: Hong Liang Teoh *What* We want to run Kinesis IT tests against Kinesalite if there are no AWS credentials present. *Why* We want maximum test coverage (e.g. on PR build, we don't have AWS creds, so we run against Kinesalite to eagerly find mistakes in PRs) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33073) Implement end-to-end tests for the Kinesis Streams Sink
Hong Liang Teoh created FLINK-33073: --- Summary: Implement end-to-end tests for the Kinesis Streams Sink Key: FLINK-33073 URL: https://issues.apache.org/jira/browse/FLINK-33073 Project: Flink Issue Type: Sub-task Components: Connectors / AWS Reporter: Hong Liang Teoh Fix For: 2.0.0 *What* Implement end-to-end tests for KinesisStreamsSink. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33072) Implement end-to-end tests for AWS Kinesis Connectors
Hong Liang Teoh created FLINK-33072: --- Summary: Implement end-to-end tests for AWS Kinesis Connectors Key: FLINK-33072 URL: https://issues.apache.org/jira/browse/FLINK-33072 Project: Flink Issue Type: Improvement Components: Connectors / AWS Reporter: Hong Liang Teoh Fix For: 2.0.0 *What* We want to implement end-to-end tests that target real Kinesis Data Streams. *Why* This solidifies our testing to ensure we pick up any integration issues with Kinesis Data Streams API. We especially want to test happy cases and failure cases to ensure those cases are handled as expected by the KDS connector. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] New Apache Flink PMC Member - Matthias Pohl
Congratulations Matthias! Very well deserved! Regards, Hong On Fri, Aug 4, 2023 at 10:38 AM Markos Sfikas wrote: > Congrats Matthias! > Best > Markos > > On Fri, Aug 4, 2023 at 11:25 AM Alexander Fedulov < > alexander.fedu...@gmail.com> wrote: > > > Congrats! > > > > Best, > > Alex > > > > On Fri, 4 Aug 2023 at 11:09, Dan Zou wrote: > > > > > Congrats, Matthias! > > > > > > Best, > > > Dan Zou > > > > > > > > > > > > > > > > > > > 2023年8月4日 13:12，Samrat Deb 写道： > > > > > > > > Congrats, Matthias! > > > > > > > > >
Re: [VOTE] FLIP-333: Redesign Apache Flink website
+1 (binding) Thanks Deepthi! On Thu, Aug 3, 2023 at 7:44 PM Danny Cranmer wrote: > +1 (binding) > > Thanks Deepthi > > > On Thu, 3 Aug 2023, 12:03 Rui Fan, <1996fan...@gmail.com> wrote: > > > +1(binding), thanks for driving this proposal, it's cool ! > > > > Best, > > Rui Fan > > > > On Thu, Aug 3, 2023 at 6:06 PM Jing Ge > wrote: > > > > > +1, thanks for driving it! > > > > > > Best regards, > > > Jing > > > > > > On Thu, Aug 3, 2023 at 4:49 AM Mohan, Deepthi > > > > > > wrote: > > > > > > > Hi, > > > > > > > > Thank you all for your feedback on FLIP-333. I’d like to start a > vote. > > > > > > > > Discussion thread: > > > > https://lists.apache.org/thread/z9j0rqt61ftgbkr37gzwbjg0n4fl1hsf > > > > FLIP: > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-333%3A+Redesign+Apache+Flink+website > > > > > > > > > > > > Thanks, > > > > Deepthi > > > > > > > > > >
[jira] [Created] (FLINK-32703) [hotfix] flink-python POM has a typo for protobuf-java in shading config
Hong Liang Teoh created FLINK-32703:
---
Summary: [hotfix] flink-python POM has a typo for protobuf-java in
shading config
Key: FLINK-32703
URL: https://issues.apache.org/jira/browse/FLINK-32703
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 1.17.1, 1.16.2, 1.18.0
Reporter: Hong Liang Teoh
Fix For: 1.18.0
Fix typo. `inculde` -> `include`
{code:java}
net.razorvine:*
net.sf.py4j:*
org.apache.beam:*
com.fasterxml.jackson.core:*
joda-time:*
com.google.protobuf:*
org.apache.arrow:*
io.netty:*
com.google.flatbuffers:*
com.alibaba:pemja
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-32619) ConfigOptions to support fallback configuration
Hong Liang Teoh created FLINK-32619:
---
Summary: ConfigOptions to support fallback configuration
Key: FLINK-32619
URL: https://issues.apache.org/jira/browse/FLINK-32619
Project: Flink
Issue Type: Technical Debt
Components: Runtime / Configuration
Affects Versions: 1.17.1, 1.16.2
Reporter: Hong Liang Teoh
ConfigOptions has no option to specify a "fallback configuration" as the
default.
For example, if we want {{rest.cache.checkpoint-statistics.timeout}} to
fallback to web.refresh-interval instead of a static default value, we have to
specify
{code:java}
@Documentation.OverrideDefault("web.refresh-interval")
@Documentation.Section(Documentation.Sections.EXPERT_REST)
public static final ConfigOption CACHE_CHECKPOINT_STATISTICS_TIMEOUT =
key("rest.cache.checkpoint-statistics.timeout")
.durationType()
.noDefaultValue()
.withDescription(
"");
{code}
The {{.noDefault()}} is misleading as it actually has a default.
We should introduce a {{.fallbackConfiguration()}} that is handled gracefully
by doc generators.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-32537) Add compatibility annotation for REST API classes
Hong Liang Teoh created FLINK-32537: --- Summary: Add compatibility annotation for REST API classes Key: FLINK-32537 URL: https://issues.apache.org/jira/browse/FLINK-32537 Project: Flink Issue Type: Technical Debt Components: Runtime / REST Affects Versions: 1.17.1, 1.16.2 Reporter: Hong Liang Teoh Fix For: 1.18.0 *Why* We want to standardise the class labelling for Flink classes. Currently, the compatibility annotations like @Public, @PublicEvolving, @Internal are not present for REST API classes. *What* We should be added @Internal for most Flink classes, unless they change the REST API variables, so we know clearly which components will change our REST API when changed -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32535) CheckpointingStatisticsHandler periodically returns NullArgumentException after job restarts
Hong Liang Teoh created FLINK-32535:
---
Summary: CheckpointingStatisticsHandler periodically returns
NullArgumentException after job restarts
Key: FLINK-32535
URL: https://issues.apache.org/jira/browse/FLINK-32535
Project: Flink
Issue Type: Bug
Components: Runtime / REST
Affects Versions: 1.17.1, 1.16.2
Reporter: Hong Liang Teoh
Fix For: 1.18.0
*What*
When making requests to /checkpoints REST API after a job restart, we see 500
for a short period of time. We should handle this gracefully in the
CheckpointingStatisticsHandler.
*How to replicate*
* Checkpointing interval 1s
* Job is constantly restarting
* Make constant requests to /checkpoints REST API.
Stack trace:
{{org.apache.commons.math3.exception.NullArgumentException: input array}}
{{ at
org.apache.commons.math3.util.MathArrays.verifyValues(MathArrays.java:1753)}}
{{ at
org.apache.commons.math3.stat.descriptive.AbstractUnivariateStatistic.test(AbstractUnivariateStatistic.java:158)}}
{{ at
org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:272)}}
{{ at
org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:241)}}
{{ at
org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics$CommonMetricsSnapshot.getPercentile(DescriptiveStatisticsHistogramStatistics.java:159)}}
{{ at
org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics.getQuantile(DescriptiveStatisticsHistogramStatistics.java:53)}}
{{ at
org.apache.flink.runtime.checkpoint.StatsSummarySnapshot.getQuantile(StatsSummarySnapshot.java:108)}}
{{ at
org.apache.flink.runtime.rest.messages.checkpoints.StatsSummaryDto.valueOf(StatsSummaryDto.java:81)}}
{{ at
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:133)}}
{{ at
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:85)}}
{{ at
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:59)}}
{{ at
org.apache.flink.runtime.rest.handler.job.checkpoints.AbstractCheckpointStatsHandler.lambda$handleRequest$1(AbstractCheckpointStatsHandler.java:62)}}
{{ at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)}}
{{ at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)}}
{{ at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)}}
{{ at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)}}
{{ at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)}}
{{ at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)}}
{{ at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)}}
{{ at java.base/java.lang.Thread.run(Thread.java:829)\n}}
See graphs here for tests. The dips in the green line correspond to the
failures immediately after a job restart.
!https://user-images.githubusercontent.com/35062175/250529297-908a6714-ea15-4aac-a7fc-332589da2582.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-32499) Removing dependency of flink-connector-aws on a specific flink-shaded version
Hong Liang Teoh created FLINK-32499: --- Summary: Removing dependency of flink-connector-aws on a specific flink-shaded version Key: FLINK-32499 URL: https://issues.apache.org/jira/browse/FLINK-32499 Project: Flink Issue Type: Technical Debt Components: Connectors / AWS Reporter: Hong Liang Teoh We want to improve build compatibility of the `flink-connector-aws` repo on upgrading Flink versions. If there are changes in the `flink-shaded` in Flink, we can see broken builds due to upgraded shaded versions (e.g. Guava). We want to explicitly state the version of dependencies being used to prevent this. See https://issues.apache.org/jira/browse/FLINK-32462 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32488) Introduce configuration to control ExecutionGraph cache in REST API
Hong Liang Teoh created FLINK-32488:
---
Summary: Introduce configuration to control ExecutionGraph cache
in REST API
Key: FLINK-32488
URL: https://issues.apache.org/jira/browse/FLINK-32488
Project: Flink
Issue Type: Improvement
Components: Runtime / REST
Affects Versions: 1.17.1, 1.16.2
Reporter: Hong Liang Teoh
Fix For: 1.18.0
*What*
Currently, REST handlers that inherit from AbstractExecutionGraphHandler serve
information derived from a cached ExecutionGraph.
This ExecutionGraph cache currently derives it's timeout from
{*}web.refresh-interval{*}. The *web.refresh-interval* controls both the
refresh rate of the Flink dashboard and the ExecutionGraph cache timeout.
We should introduce a new configuration to control the ExecutionGraph cache,
namely {*}rest.cache.execution-graph.expiry{*}.
*Why*
Sharing configuration between REST handler and Flink dashboard is a sign that
we are coupling the two.
Ideally, we want our REST API behaviour to independent of the Flink dashboard
(e.g. supports programmatic access).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-32469) Simplify the implementation of the checkpoint handlers
Hong Liang Teoh created FLINK-32469:
---
Summary: Simplify the implementation of the checkpoint handlers
Key: FLINK-32469
URL: https://issues.apache.org/jira/browse/FLINK-32469
Project: Flink
Issue Type: Improvement
Components: Runtime / REST
Affects Versions: 1.17.1, 1.16.2
Reporter: Hong Liang Teoh
Fix For: 1.18.0
*What*
The checkpoint handlers currently retrieve checkpoint information from the
`ExecutionGraph`, which is cached in the `AbstractExecutionGraphHandler`. This
means that this information is potentially stale (depending on the
`web.refresh-interval`, which defaults to 3s).
*Why*
We want to enable programmatic use of the REST API, independent of the Flink
dashboard.
The current configuration of the `ExecutionGraph` cache is meant to facilitate
a fluid user experience of the Flink dashboard. On the Job details page, the
Flink dashboard makes a series of requests (e.g. /jobs/\{jobid},
/jobs/\{jobid}/vertices/\{vertexid}){color:#172b4d}. {color}
{color:#172b4d}To ensure that the requests return consistent results, we have
the execution graph cache.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-32324) Implement watermark alignment on KDS source
Hong Liang Teoh created FLINK-32324: --- Summary: Implement watermark alignment on KDS source Key: FLINK-32324 URL: https://issues.apache.org/jira/browse/FLINK-32324 Project: Flink Issue Type: Sub-task Components: Connectors / Kinesis Reporter: Hong Liang Teoh Implement watermark alignment interfaces suggested by this FLIP in the KDS Source. https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32229) Implement metrics and logging for Initial implementation
Hong Liang Teoh created FLINK-32229: --- Summary: Implement metrics and logging for Initial implementation Key: FLINK-32229 URL: https://issues.apache.org/jira/browse/FLINK-32229 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh Add/Ensure Kinesis specific metrics for MillisBehindLatest/numRecordsIn are published. List is here: https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32218) Implement support for parent/child shard ordering
Hong Liang Teoh created FLINK-32218: --- Summary: Implement support for parent/child shard ordering Key: FLINK-32218 URL: https://issues.apache.org/jira/browse/FLINK-32218 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh Implement support for parent/child shard ordering in the KDS connector -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32116) FlinkKinesisConsumer cannot stop-with-savepoint when configured with watermark assigner and watermark tracker
Hong Liang Teoh created FLINK-32116:
---
Summary: FlinkKinesisConsumer cannot stop-with-savepoint when
configured with watermark assigner and watermark tracker
Key: FLINK-32116
URL: https://issues.apache.org/jira/browse/FLINK-32116
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Affects Versions: 1.15.4, 1.16.1, 1.17.0
Reporter: Hong Liang Teoh
Problem:
When FlinkKinesisConsumer is configured with legacy watermarking system, it is
unable to take a savepoint during stop-with-savepoint, and will get stuck
indefinitely.
{code:java}
FlinkKinesisConsumer src = new FlinkKinesisConsumer("YourStreamHere", new
SimpleStringSchema(), consumerConfig);
// Set up watermark assigner on Kinesis source
src.setPeriodicWatermarkAssigner(...);
// Set up watermark tracker on Kinesis source
src.setWatermarkTracker(...);{code}
*Why does it get stuck?*
When watermarks are setup, the `shardConsumer` and `recordEmitter` thread
communicate using asynchronous queue.
On stop-with-savepoint, shardConsumer waits for queue to empty before
continuing. recordEmitter is terminated before queue is empty. As such, queue
is never going to be empty, and app gets stuck indefinitely.
*Workarounds*
Use the new watermark framework
{code:java}
FlinkKinesisConsumer src = new FlinkKinesisConsumer("YourStreamHere", new
SimpleStringSchema(), consumerConfig);
env.addSource(src)
// Set up watermark strategy with both watermark assigner and watermark tracker
.assignTimestampsAndWatermarks(WatermarkStrategy.forMonotonousTimestamps()){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-32097) Implement support for Kinesis aggregation
Hong Liang Teoh created FLINK-32097: --- Summary: Implement support for Kinesis aggregation Key: FLINK-32097 URL: https://issues.apache.org/jira/browse/FLINK-32097 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh Implement support for KPL aggregation on UserRecord. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31990) Use Flink Configuration to specify KDS Source configuration object
Hong Liang Teoh created FLINK-31990: --- Summary: Use Flink Configuration to specify KDS Source configuration object Key: FLINK-31990 URL: https://issues.apache.org/jira/browse/FLINK-31990 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh *What* Use the Flink Configuration object to standardise the method of specifying configurations for the KDS source. *Why* We want to standardise error messages + source serialization methods implemented by Flink on the Flink Configuration objects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31989) Update documentation
Hong Liang Teoh created FLINK-31989: --- Summary: Update documentation Key: FLINK-31989 URL: https://issues.apache.org/jira/browse/FLINK-31989 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh Update Flink documentation to explain the new KDS source. Include * Improvements available in new KDS source * Incompatible changes made * Example implementation * Example customisations -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31988) Implement Python wrapper for new KDS source
Hong Liang Teoh created FLINK-31988: --- Summary: Implement Python wrapper for new KDS source Key: FLINK-31988 URL: https://issues.apache.org/jira/browse/FLINK-31988 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh *What?* - Implement Python wrapper for KDS source - Write tests for this KDS source -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31987) Implement Table API support
Hong Liang Teoh created FLINK-31987: --- Summary: Implement Table API support Key: FLINK-31987 URL: https://issues.apache.org/jira/browse/FLINK-31987 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh Implement Table API support for KDS Source. Consider: * Configurations to support. Should have customisation parity with DataStream API * Testing should include both SQL client + Table API via Java -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31986) Implement Integration test for FLIP-27 DataStreams API
Hong Liang Teoh created FLINK-31986: --- Summary: Implement Integration test for FLIP-27 DataStreams API Key: FLINK-31986 URL: https://issues.apache.org/jira/browse/FLINK-31986 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh *What?* Implement an integration test that uses Kinesalite. It should spin up FLIP-27 KDS source and test: * Reading from single shard KDS stream * Reading from multiple shard KDS stream * Reading from resharded KDS stream * Failure modes (stream doesn't exist) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31980) Implement support for EFO in new Source
Hong Liang Teoh created FLINK-31980: --- Summary: Implement support for EFO in new Source Key: FLINK-31980 URL: https://issues.apache.org/jira/browse/FLINK-31980 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh Implement support for reading from Kinesis Stream using Enhanced Fan Out mechanism -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31942) Support Conditional Writes in DynamoDB connector
Hong Liang Teoh created FLINK-31942: --- Summary: Support Conditional Writes in DynamoDB connector Key: FLINK-31942 URL: https://issues.apache.org/jira/browse/FLINK-31942 Project: Flink Issue Type: New Feature Components: Connectors / DynamoDB Reporter: Hong Liang Teoh Currently, the AWS DynamoDB connector uses the BatchWrite API, which does not support conditional writes. This is not great because there might be some use cases where the Flink user might want to use conditional writes to implement idempotent writes. We propose to implement support for using `PutItem`, `UpdateItem` and `DeleteItem` in the DDB connector -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31922) Port over Kinesis Client configurations for retry and backoff
Hong Liang Teoh created FLINK-31922: --- Summary: Port over Kinesis Client configurations for retry and backoff Key: FLINK-31922 URL: https://issues.apache.org/jira/browse/FLINK-31922 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh Port over the Kinesis Client configurations for GetRecords, ListShards, DescribeStream -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31813) Initial implementation of Kinesis Source using FLIP-27
Hong Liang Teoh created FLINK-31813: --- Summary: Initial implementation of Kinesis Source using FLIP-27 Key: FLINK-31813 URL: https://issues.apache.org/jira/browse/FLINK-31813 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh Fix For: aws-connector-4.2.0 Implement a base implementation of the Kinesis source based on FLIP-27 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31662) Migrate flink-connectors-aws to new structure
Hong Liang Teoh created FLINK-31662:
---
Summary: Migrate flink-connectors-aws to new structure
Key: FLINK-31662
URL: https://issues.apache.org/jira/browse/FLINK-31662
Project: Flink
Issue Type: Sub-task
Reporter: Hong Liang Teoh
Migrate flink-connectors-aws to follow a new structure of
{code:java}
\-flink-connector-aws-base
\-flink-connector-aws-e2e-tests
\-flink-connector-aws-kinesis-firehose-e2e-tests
\-flink-connector-aws-kinesis-streams-e2e-tests
\-flink-connector-kinesis-e2e-tests
\-flink-formats-avro-glue-schema-registry-e2e-tests
\-flink-formats-json-glue-schema-registry-e2e-tests
\-flink-connector-aws
\-flink-connector-aws-kinesis-firehose
\-flink-connector-aws-kinesis-streams
\-flink-connector-dynamodb
\-flink-connector-kinesis
\-flink-sql-connector-aws-kinesis-firehose
\-flink-sql-connector-aws-kinesis-streams
\-flink-sql-connector-dynamodb
\-flink-sql-connector-kinesis
\-flink-formats-aws
\-flink-avro-glue-schema-registry
\-flink-json-glue-schema-registry{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31159) Release Testing: Verify FLINK-30583 Provide the flame graph to the subtask level
Hong Liang Teoh created FLINK-31159: --- Summary: Release Testing: Verify FLINK-30583 Provide the flame graph to the subtask level Key: FLINK-31159 URL: https://issues.apache.org/jira/browse/FLINK-31159 Project: Flink Issue Type: Sub-task Components: Runtime / REST, Runtime / Web Frontend Affects Versions: 1.17.0 Reporter: Hong Liang Teoh Fix For: 1.17.0 The issue aims to verify FLINK-30583. Please verify: # When below conditions are met: ** Job has more than 1 parallel subtask ** Some subtasks are busy, whilst others are not (e.g. some receive records, some don't) ** The FlameGraph accurately reflects the busy subtask -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31108) Use StreamARN for API calls in Kinesis Connector
Hong Liang Teoh created FLINK-31108: --- Summary: Use StreamARN for API calls in Kinesis Connector Key: FLINK-31108 URL: https://issues.apache.org/jira/browse/FLINK-31108 Project: Flink Issue Type: Improvement Components: Connectors / Kinesis Affects Versions: 1.16.1, 1.15.3 Reporter: Hong Liang Teoh Fix For: 1.17.1 Currently, the FlinkKinesisConsumer (Polling + EFO) + FlinkKinesisProducer uses the stream name during API calls We want to change this to the StreamARN. There are two reasons for this: - This allows lower latency calls to the Kinesis endpoint for GetRecords API - Paves the way for allowing user target cross-account streams without assume role (i.e. IAM role in account A but target stream in account B) The APIs that are currently called: * ** ListShards ** GetShardIterator ** GetRecords ** DescribeStream ** DescribeStreamSummary ** DescribeStreamConsumer (already uses StreamARN) ** RegisterStreamConsumer (already uses StreamARN) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30950) Remove flink-connector-aws-base from Flink main repo
Hong Liang Teoh created FLINK-30950: --- Summary: Remove flink-connector-aws-base from Flink main repo Key: FLINK-30950 URL: https://issues.apache.org/jira/browse/FLINK-30950 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh Remove flink-connector-aws-base from Flink main repo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30949) Sync flink-connector-aws-base from Flink main to flink-connector-aws
Hong Liang Teoh created FLINK-30949: --- Summary: Sync flink-connector-aws-base from Flink main to flink-connector-aws Key: FLINK-30949 URL: https://issues.apache.org/jira/browse/FLINK-30949 Project: Flink Issue Type: Sub-task Components: Connectors / Common Reporter: Hong Liang Teoh Sync flink-connector-aws-base from Flink main to flink-connector-aws -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30948) Remove flink-avro-glue-schema-registry and flink-json-glue-schema-registry from Flink main repo
Hong Liang Teoh created FLINK-30948: --- Summary: Remove flink-avro-glue-schema-registry and flink-json-glue-schema-registry from Flink main repo Key: FLINK-30948 URL: https://issues.apache.org/jira/browse/FLINK-30948 Project: Flink Issue Type: Sub-task Environment: Remove flink-avro-glue-schema-registry and flink-json-glue-schema-registry from Flink main repo, along with associated end-to-end tests Reporter: Hong Liang Teoh -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30947) Sync flink-avro-glue-schema-registry and flink-json-glue-schema-registry to flink-connector-aws
Hong Liang Teoh created FLINK-30947: --- Summary: Sync flink-avro-glue-schema-registry and flink-json-glue-schema-registry to flink-connector-aws Key: FLINK-30947 URL: https://issues.apache.org/jira/browse/FLINK-30947 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh Copy flink-avro-glue-schema-registry and flink-json-glue-schema-registry, along with associated end-to-end tests to flink-connector-aws -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30946) Sync master Avro GlueSchemaRegistry formats flink-connector-aws
Hong Liang Teoh created FLINK-30946: --- Summary: Sync master Avro GlueSchemaRegistry formats flink-connector-aws Key: FLINK-30946 URL: https://issues.apache.org/jira/browse/FLINK-30946 Project: Flink Issue Type: Technical Debt Reporter: Hong Liang Teoh Update flink-connector-aws to take the flink-avro-glue-schema-registry from the Flink codebase. h4. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30853) Make clear the public API for flink-avro
Hong Liang Teoh created FLINK-30853: --- Summary: Make clear the public API for flink-avro Key: FLINK-30853 URL: https://issues.apache.org/jira/browse/FLINK-30853 Project: Flink Issue Type: Technical Debt Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Reporter: Hong Liang Teoh flink-avro provides users with methods to register the Apache Avro format in their connectors. However, it is currently not clear which classes are @Public, @PublicEvolving and @Internal. We want to make clear the classes / interfaces that are @Public or @PublicEvolving so that users can know whether the API is expected to be stable. For classes that are already indirectly exposed as @Public or @PublicEvolving, we should ensure that they are not marked as anything lower (e.g. @Internal) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30837) Remove dependency of MutableByteArrayInputStream from flink-avro-glue-schema-registry
Hong Liang Teoh created FLINK-30837: --- Summary: Remove dependency of MutableByteArrayInputStream from flink-avro-glue-schema-registry Key: FLINK-30837 URL: https://issues.apache.org/jira/browse/FLINK-30837 Project: Flink Issue Type: Technical Debt Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Affects Versions: 1.16.1 Reporter: Hong Liang Teoh MutableByteArrayInputStream is a class implemented in flink-avro. We want to make this @Internal if possible, so we can reduce the surface area of interface that we support. At the moment, it is only used in flink-avro and flink-avro-glue-schema-registry. As such, we explore if we can remove the use of MutableByteArrayInputStream from flink-avro-glue-schema-registry. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30783) Add pull request template for AWS Connectors repo
Hong Liang Teoh created FLINK-30783: --- Summary: Add pull request template for AWS Connectors repo Key: FLINK-30783 URL: https://issues.apache.org/jira/browse/FLINK-30783 Project: Flink Issue Type: Technical Debt Components: Connectors / AWS Reporter: Hong Liang Teoh Add a pull request template for Apache Flink AWS Connectors [https://github.com/apache/flink-connector-aws] This improves our pull request and commit quality. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30156) [FLIP-242] Blogpost about the customisable RateLimitingStrategy in the AsyncSinkBase
Hong Liang Teoh created FLINK-30156: --- Summary: [FLIP-242] Blogpost about the customisable RateLimitingStrategy in the AsyncSinkBase Key: FLINK-30156 URL: https://issues.apache.org/jira/browse/FLINK-30156 Project: Flink Issue Type: Improvement Reporter: Hong Liang Teoh Create a blogpost to explain the customisability of the RateLimitingStrategy in the AsyncSinkBase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30098) Update DynamoDb Sink unit tests to JUnit 5
Hong Liang Teoh created FLINK-30098: --- Summary: Update DynamoDb Sink unit tests to JUnit 5 Key: FLINK-30098 URL: https://issues.apache.org/jira/browse/FLINK-30098 Project: Flink Issue Type: Improvement Components: Connectors / DynamoDB Reporter: Hong Liang Teoh Use JUnit 5 in all unit tests for dynamoDB sink -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30096) Rename DynamoDB config destinationTableName to tableName
Hong Liang Teoh created FLINK-30096: --- Summary: Rename DynamoDB config destinationTableName to tableName Key: FLINK-30096 URL: https://issues.apache.org/jira/browse/FLINK-30096 Project: Flink Issue Type: Improvement Reporter: Hong Liang Teoh The word destination is redundant since it is part of a DDB table sink. Renaming destinationTableName to tableName in all places -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30092) Improve Table API experience for Flink DOUBLE type
Hong Liang Teoh created FLINK-30092:
---
Summary: Improve Table API experience for Flink DOUBLE type
Key: FLINK-30092
URL: https://issues.apache.org/jira/browse/FLINK-30092
Project: Flink
Issue Type: Improvement
Components: Connectors / DynamoDB
Reporter: Hong Liang Teoh
Currently, the DynamoDB sink's Table API allows users to input records with
Flink type of `DOUBLE`. However, the limitations on Double on Flink/Java side
[1] is more permissive than DynamoDB's limitations on the Number type [2].
Java Double limitations:
MIN_VALUE: −1.7976931348623157×10{^}308{^}
MAX_VALUE: 1.7976931348623157×10{^}308{^}
DynamoDB number type limitations:
MIN_VALUE: -9.9E+125 to -1E-130
MAX_VALUE: 1E-130 to 9.9E+125
This means that a user can write values of type `DOUBLE` into the DDB sink, as
long as they are within the limitations of DynamoDB. Once they exceed the
limitation, the record will fail to write.
We should consider improving this experience by adding a configuration to allow
users to map DOUBLE to DDB String type instead.
[1] [https://docs.oracle.com/javase/7/docs/api/java/lang/Double.html#MAX_VALUE]
[2]
[https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#:~:text=Boolean%2C%20and%20null.-,Number,-Numbers%20can%20be]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29909) Standardise connector package names
Hong Liang Teoh created FLINK-29909: --- Summary: Standardise connector package names Key: FLINK-29909 URL: https://issues.apache.org/jira/browse/FLINK-29909 Project: Flink Issue Type: Sub-task Reporter: Hong Liang Teoh -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29900) Implement Table API for DynamoDB Sink
Hong Liang Teoh created FLINK-29900: --- Summary: Implement Table API for DynamoDB Sink Key: FLINK-29900 URL: https://issues.apache.org/jira/browse/FLINK-29900 Project: Flink Issue Type: Improvement Components: Connectors / DynamoDB Reporter: Hong Liang Teoh Implement table API support for DynamoDB sink -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29683) Introduce config parser for AWS BackoffStrategy
Hong Liang Teoh created FLINK-29683: --- Summary: Introduce config parser for AWS BackoffStrategy Key: FLINK-29683 URL: https://issues.apache.org/jira/browse/FLINK-29683 Project: Flink Issue Type: Improvement Reporter: Hong Liang Teoh *What* Different connectors use AWS SDK, and there are some de-duplicable logic here. # Parsing retry backoff strategy configuration from configs. # Converting retry backoff strategy into the AWS SDK Java objects. We want to introduce support for this in the `flink-connector-aws-base` so that other connectors can just use this config parser to retrieve the BackoffStrategy. See [here|https://github.com/apache/flink-connector-dynamodb/pull/1/files#diff-d383895c9604d623476abe5740a12bf4db5ec17110d6d7d4df748bcf634dd776] for some inspiration -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29574) Upgrade software.amazon.glue:schema-registry-common and software.amazon.glue:schema-registry-serde dependency from 1.1.8 to 1.1.14
Hong Liang Teoh created FLINK-29574: --- Summary: Upgrade software.amazon.glue:schema-registry-common and software.amazon.glue:schema-registry-serde dependency from 1.1.8 to 1.1.14 Key: FLINK-29574 URL: https://issues.apache.org/jira/browse/FLINK-29574 Project: Flink Issue Type: Technical Debt Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Reporter: Hong Liang Teoh Fix For: 1.17.0 We should update the software.amazon.glue:schema-registry-common and software.amazon.glue:schema-registry-serde dependencies from 1.1.8 to 1.1.14 to be up to date with the latest version -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29395) [Kinesis][EFO] Issue using EFO consumer at timestamp with empty shard
Hong Liang Teoh created FLINK-29395:
---
Summary: [Kinesis][EFO] Issue using EFO consumer at timestamp with
empty shard
Key: FLINK-29395
URL: https://issues.apache.org/jira/browse/FLINK-29395
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Reporter: Hong Liang Teoh
Assignee: Danny Cranmer
Fix For: 1.12.0
*Background*
The consumer fails when a Polling record publisher uses a timestamp sentinel
starting position and the first record batch is empty. This is because the
consumer tries to recalculate the start position from the timestamp sentinel,
this operation is not supported.
*Reproduction Steps*
Setup an application consuming from Kinesis with following properties and
consume from an empty shard:
{code:java}
String format = "-MM-dd'T'HH:mm:ss";
String date = new SimpleDateFormat(format).format(new Date());
consumerConfig.setProperty(ConsumerConfigConstants.STREAM_INITIAL_TIMESTAMP,
date);
consumerConfig.setProperty(ConsumerConfigConstants.STREAM_TIMESTAMP_DATE_FORMAT,
format);
consumerConfig.setProperty(ConsumerConfigConstants.STREAM_INITIAL_POSITION,
"AT_TIMESTAMP"); {code}
*Error*
{code:java}
Exception in thread "main"
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
at
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$2(MiniClusterJobClient.java:119)
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:229)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:996)
at akka.dispatch.OnComplete.internal(Future.scala:264)
at akka.dispatch.OnComplete.internal(Future.scala:261)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
at
scala.concurrent.impl.CallbackRunnable.run$$$capture(Promise.scala:36)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala)
at
org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74)
at
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
at
scala.concurrent.impl.CallbackRunnable.run$$$capture(Promise.scala:36)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala)
at
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
akka.dispatch.for
[jira] [Created] (FLINK-28978) [aws][connectors] Kinesis connector doesn't work for new AWS regions
Hong Liang Teoh created FLINK-28978: --- Summary: [aws][connectors] Kinesis connector doesn't work for new AWS regions Key: FLINK-28978 URL: https://issues.apache.org/jira/browse/FLINK-28978 Project: Flink Issue Type: Bug Components: Connectors / Kinesis Affects Versions: 1.15.1, 1.14.5, 1.13.6 Reporter: Hong Liang Teoh Fix For: 1.16.0, 1.13.7, 1.15.2, 1.14.6 The current validation in the Kinesis connector checks that the AWS Region string specified is present in the `Regions` enum attached in the `AWS SDK`. This is not desirable because every time AWS launches a new region, we will have to update the AWS SDK shaded into the connector. We want to change it such that we validate the shape of the string, allowing for future AWS Regions. Current list of regions: ap-south-1, eu-south-1, us-gov-east-1, ca-central-1, eu-central-1, us-west-1, us-west-2, af-south-1, eu-north-1, eu-west-3, eu-west-2, eu-west-1, ap-northeast-3, ap-northeast-2, ap-northeast-1, me-south-1, sa-east-1, ap-east-1, cn-north-1, us-gov-west-1, ap-southeast-1, ap-southeast-2, ap-southeast-3, us-iso-east-1, us-east-1, us-east-2, cn-northwest-1, us-isob-east-1, aws-global, aws-cn-global, aws-us-gov-global, aws-iso-global, aws-iso-b-global -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28487) Introduce configurable RateLimitingStrategy for Async Sink
Hong Liang Teoh created FLINK-28487: --- Summary: Introduce configurable RateLimitingStrategy for Async Sink Key: FLINK-28487 URL: https://issues.apache.org/jira/browse/FLINK-28487 Project: Flink Issue Type: New Feature Components: Connectors / Common Reporter: Hong Liang Teoh Fix For: 1.16.0 Introduce a configurable RateLimitingStrategy to the AsyncSinkWriter. This change will allow sink implementers using AsyncSinkWriter to configure their own RateLimitingStrategy instead of using the default AIMDRateLimitingStrategy. See [FLIP-242: Introduce configurable RateLimitingStrategy for Async Sink|https://cwiki.apache.org/confluence/display/FLINK/FLIP-242%3A+Introduce+configurable+RateLimitingStrategy+for+Async+Sink]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-19226) [Kinesis] [EFO] Connector reaches default max attempts for describeStream and describeStreamConsumer when parallelism is high
Hong Liang Teoh created FLINK-19226: --- Summary: [Kinesis] [EFO] Connector reaches default max attempts for describeStream and describeStreamConsumer when parallelism is high Key: FLINK-19226 URL: https://issues.apache.org/jira/browse/FLINK-19226 Project: Flink Issue Type: Improvement Components: Connectors / Kinesis Reporter: Hong Liang Teoh Fix For: 1.12.0 *Background* When lazily registering the stream consumer on apps with high parallelism, EFO connector hits default maximum number of attempts when calling describeStream and describeStreamConsumer on the Kinesis Streams API. The default FullJitterBackoff constants are tuned to prevent this when parallelism of 1024 is used. *Scope* * See [FLIP|https://cwiki.apache.org/confluence/display/FLINK/FLIP-128%3A+Enhanced+Fan+Out+for+AWS+Kinesis+Consumers] for full list of configuration options * Suggested changes: |flink.stream.describe.maxretries|50| |flink.stream.describe.backoff.base|2000L| |flink.stream.describe.backoff.max|5000L| |flink.stream.describestreamconsumer.maxretries|50| |flink.stream.describestreamconsumer.backoff.base|2000L| |flink.stream.describestreamconsumer.backoff.max|5000L| -- This message was sent by Atlassian Jira (v8.3.4#803005)
