Re: [VOTE] Release flink-connector-kafka v3.1.0, release candidate #1
+1 (non-binding) - Validated checksum hash - Verified signature - Verified that no binaries exist in the source archive - Build the source with Maven and jdk11 - Verified web PR - Check that the jar is built by jdk8 Best, Hang Martijn Visser 于2024年1月26日周五 21:05写道： > Hi everyone, > Please review and vote on the release candidate #1 for the Flink Kafka > connector version 3.1.0, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > This release is compatible with Flink 1.17.* and Flink 1.18.* > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org > [2], > which are signed with the key with fingerprint > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag v3.1.0-rc1 [5], > * website pull request listing the new release [6]. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Thanks, > Release Manager > > [1] > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353135 > [2] > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-kafka-3.1.0-rc1 > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] https://repository.apache.org/content/repositories/orgapacheflink-1700 > [5] > https://github.com/apache/flink-connector-kafka/releases/tag/v3.1.0-rc1 > [6] https://github.com/apache/flink-web/pull/718 >
[jira] [Created] (FLINK-34258) Incorrect example of accumulator usage within emitUpdateWithRetract for TableAggregateFunction
Jane Chan created FLINK-34258: - Summary: Incorrect example of accumulator usage within emitUpdateWithRetract for TableAggregateFunction Key: FLINK-34258 URL: https://issues.apache.org/jira/browse/FLINK-34258 Project: Flink Issue Type: Bug Components: Documentation, Table SQL / API Affects Versions: 1.18.1, 1.19.0 Reporter: Jane Chan Assignee: Jane Chan The [documentation|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/#retraction-example] provides an example of using `emitUpdateWithRetract`. However, the example is misleading as it incorrectly suggests that the accumulator can be updated within the `emitUpdateWithRetract method`. In reality, the order of invocation is to first call `getAccumulator` and then `emitUpdateWithRetract`, which means that updating the accumulator within `emitUpdateWithRetract` will not take effect. Please see [GroupTableAggFunction#L141|https://github.com/apache/flink/blob/20450485b20cb213b96318b0c3275e42c0300e15/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/aggregate/GroupTableAggFunction.java#L141] ~ [GroupTableAggFunction#L146|https://github.com/apache/flink/blob/20450485b20cb213b96318b0c3275e42c0300e15/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/aggregate/GroupTableAggFunction.java#L146] for more details. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34257) Update Flink YAML Parser to Support YAML 1.2 Specification
Junrui Li created FLINK-34257: - Summary: Update Flink YAML Parser to Support YAML 1.2 Specification Key: FLINK-34257 URL: https://issues.apache.org/jira/browse/FLINK-34257 Project: Flink Issue Type: Bug Components: API / Core Reporter: Junrui Li Fix For: 1.19.0 FLINK-33297 and FLINK-33577 added snakeyaml and pyyaml dependencies to support a standard YAML parser. However, these parsers support the YAML 1.1 specification, not the YAML 1.2 specification. Therefore, we need to update these dependencies that support YAML 1.2. The updated dependencies are as follows: 1. For Java: change from snakeyaml to snakeyaml-engine 2. For Python: change from pyyaml to ruamel.yaml -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34256) Add a documentation section for minibatch join
Shuai Xu created FLINK-34256: Summary: Add a documentation section for minibatch join Key: FLINK-34256 URL: https://issues.apache.org/jira/browse/FLINK-34256 Project: Flink Issue Type: Sub-task Components: Documentation Affects Versions: 1.19.0 Reporter: Shuai Xu We should add a minibatch join section in Performance Tuning to explain the usage and principle of minibatch-join. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34255) FLIP-406: Reorganize State & Checkpointing & Recovery Configuration
Zakelly Lan created FLINK-34255:
---
Summary: FLIP-406: Reorganize State & Checkpointing & Recovery
Configuration
Key: FLINK-34255
URL: https://issues.apache.org/jira/browse/FLINK-34255
Project: Flink
Issue Type: Improvement
Components: Runtime / Checkpointing, Runtime / State Backends
Reporter: Zakelly Lan
Assignee: Zakelly Lan
Fix For: 2.0.0
The FLIP:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560
Currently, the configuration options pertaining to checkpointing, recovery, and
state management are primarily grouped under the following prefixes:
* *state.backend.** : configurations related to state accessing and
checkpointing, as well as specific options for individual state backends
* *execution.checkpointing.** : configurations associated with checkpoint
execution and recovery
* {*}execution.savepoint.*{*}: configurations for recovery from savepoint
In addition, there are several individual options such as
_{{state.checkpoint-storage}}_ and _{{state.checkpoints.dir}}_ that fall
outside of these prefixes. The current arrangement of these options, which span
multiple modules, is somewhat haphazard and lacks a systematic structure. For
example, the options under the {{_CheckpointingOptions_ }}and
{{_ExecutionCheckpointingOptions_ }}are related and have no clear boundaries
from the user's perspective, but there is no unified prefix for them. With the
upcoming release of Flink 2.0, we have an excellent opportunity to overhaul and
restructure the configurations related to checkpointing, recovery, and state
management. This FLIP proposes to reorganize these settings, making it more
coherent by module, which would significantly lower the barriers for
understanding and reduce the development costs moving forward.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[RESULT][VOTE] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration
Hi devs, I'm glad to announce that the FLIP-406[1] has been accepted. The voting thread is here[2]. The proposal received 6 approving votes, 5 of which are binding: - Rui Fan (binding) - Hangxiang Yu (binding) - Yanfei Lei (binding) - Lijie Wang (binding) - Xuannan Su (non-binding) - Yuan Mei (binding) And there is no disapproving one. Thanks to all participants for discussion, suggestion and voting! [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560 [2] https://lists.apache.org/thread/lrqjg44v0s82shbpvbqp6ojqv873q1wr Best, Zakelly
Re: [VOTE] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration
Thank you all for the votes! I'm closing this thread and the result will be posted in a separate mail. Best, Zakelly On Thu, Jan 25, 2024 at 3:14 PM Yuan Mei wrote: > +1 > > On Thu, Jan 25, 2024 at 10:57 AM Xuannan Su wrote: > > > +1 (non-binding) > > > > Best, > > Xuannan > > > > On Thu, Jan 25, 2024 at 10:15 AM Lijie Wang > > wrote: > > > > > > +1 (binding) > > > > > > Best, > > > Lijie > > > > > > Yanfei Lei 于2024年1月25日周四 10:06写道： > > > > > > > +1 (binding) > > > > > > > > Hangxiang Yu 于2024年1月25日周四 10:00写道： > > > > > > > > > > +1 (binding) > > > > > > > > > > On Thu, Jan 25, 2024 at 8:49 AM Rui Fan <1996fan...@gmail.com> > > wrote: > > > > > > > > > > > +1(binding) > > > > > > > > > > > > Best, > > > > > > Rui > > > > > > > > > > > > On Wed, 24 Jan 2024 at 21:50, Zakelly Lan > > > > > wrote: > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > I'd like to start a vote on the FLIP-406: Reorganize State & > > > > > > Checkpointing > > > > > > > & Recovery Configuration [1]. The discussion thread is here > [2]. > > > > > > > > > > > > > > The vote will be open for at least 72 hours unless there is an > > > > objection > > > > > > or > > > > > > > insufficient votes. > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560 > > > > > > > [2] > > https://lists.apache.org/thread/0oc10cr2q2ms855dbo29s7v08xs3bvqg > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > Zakelly > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Best, > > > > > Hangxiang. > > > > > > > > > > > > > > > > -- > > > > Best, > > > > Yanfei > > > > > > >
Re: Confluence access
Hi, thanks for reply! My account in JIRA is ParyshevSergey, on confluence I didn't found how to create account. >Понедельник, 22 января 2024, 19:51 +07:00 от Martijn Visser >: > >Hi, > >What's your Confluence user name? > >Best regards, > >Martijn > >On Fri, Jan 19, 2024 at 8:04AM Сергей Парышев >< sergey_parys...@mail.ru.invalid > wrote: >> >> >> Hi devs! Can I get access to confluence? I want suggest a FLIP.
[jira] [Created] (FLINK-34254) `DESCRIBE` syntaxes like `DESCRIBE CATALOG xxx` throws strange exceptions
xuyang created FLINK-34254: -- Summary: `DESCRIBE` syntaxes like `DESCRIBE CATALOG xxx` throws strange exceptions Key: FLINK-34254 URL: https://issues.apache.org/jira/browse/FLINK-34254 Project: Flink Issue Type: Bug Components: Table SQL / API Reporter: xuyang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34253) Offsets out of range with no configured reset policy for partitions
Jepson created FLINK-34253:
--
Summary: Offsets out of range with no configured reset policy for
partitions
Key: FLINK-34253
URL: https://issues.apache.org/jira/browse/FLINK-34253
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.14.4
Environment: flink 1.14.4
Reporter: Jepson
java.lang.RuntimeException: SplitFetcher thread 0 received unexpected exception
while polling the records at
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:150)
~[flink-table_2.12-1.14.4.jar:1.14.4] at
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:105)
[flink-table_2.12-1.14.4.jar:1.14.4] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_241] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_241] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_241] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_241] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241] Caused by:
org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of
range with no configured reset policy for partitions: {dp-oracle-sllv-0=12734616
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34252) WatermarkAssignerOperator should not emit WatermarkStatus.IDLE under continuous data flow
David Christle created FLINK-34252:
--
Summary: WatermarkAssignerOperator should not emit
WatermarkStatus.IDLE under continuous data flow
Key: FLINK-34252
URL: https://issues.apache.org/jira/browse/FLINK-34252
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.18.1, 1.17.2, 1.16.3
Reporter: David Christle
The WatermarkAssignerOperator in the table runtime incorrectly transitions to
an IDLE state even when data is continuously flowing. This behavior, observed
under normal operating conditions where the interval between data elements is
shorter than the configured idleTimeout, leads to regular transitions between
ACTIVE and IDLE states, which are unnecessary.
_Detail:_
In the current implementation, the lastRecordTime variable, which tracks the
time of the last received data element, is updated only when the
WatermarkStatus transitions from IDLE to ACTIVE. However, it is not updated
when WatermarkStatus is ACTIVE, which means even under continuous data flow,
the condition `(currentTime - lastRecordTime > idleTimeout)` will eventually
always become true, and the WatermarkStatus will erroneously be marked IDLE.
It is unclear to me if this bug produces any incorrectness downstream, since
when the WatermarkStatus is in in the IDLE state, the next processElement will
cause a WatermarkStatus.ACTIVE to be emitted. Nevertheless, we should eliminate
this flip-flop behavior between states.
The test I wrote fails without the fix and illustrates the flip-flops:
{noformat}
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.030 s
<<< FAILURE! -- in
org.apache.flink.table.runtime.operators.wmassigners.WatermarkAssignerOperatorTest
[ERROR]
org.apache.flink.table.runtime.operators.wmassigners.WatermarkAssignerOperatorTest.testIdleStateAvoidanceWithConsistentDataFlow
-- Time elapsed: 0.013 s <<< FAILURE!
java.lang.AssertionError:
Expecting
[WatermarkStatus(IDLE),
WatermarkStatus(ACTIVE),
WatermarkStatus(IDLE),
WatermarkStatus(ACTIVE),
WatermarkStatus(IDLE),
WatermarkStatus(ACTIVE),
WatermarkStatus(IDLE),
WatermarkStatus(ACTIVE),
WatermarkStatus(IDLE)]
not to contain
[WatermarkStatus(IDLE)]
but found
[WatermarkStatus(IDLE)]
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
