[jira] [Created] (FLINK-35488) DataType Support Geometry Type
Leopold created FLINK-35488: --- Summary: DataType Support Geometry Type Key: FLINK-35488 URL: https://issues.apache.org/jira/browse/FLINK-35488 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: Leopold I want sync data from mysql to postgresql,but in Geometry Datatype filed i couldn't do it,mysql geometry datatype data can be transformed to string by spatialfuntion ,for example.st_astext(geom) .In other way,postgresql geometry datatype data also transformed to string . So,i hope Flink suport mysql and postgrdql databse geometry datatype can be transform. Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35487) ContinuousFileProcessingCheckpointITCase crashed as process exit with code 127
Weijie Guo created FLINK-35487: -- Summary: ContinuousFileProcessingCheckpointITCase crashed as process exit with code 127 Key: FLINK-35487 URL: https://issues.apache.org/jira/browse/FLINK-35487 Project: Flink Issue Type: Bug Components: Build System / CI Affects Versions: 1.20.0 Reporter: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35486) Potential sql expression generation issues on SQL gateway
Xingcan Cui created FLINK-35486:
---
Summary: Potential sql expression generation issues on SQL gateway
Key: FLINK-35486
URL: https://issues.apache.org/jira/browse/FLINK-35486
Project: Flink
Issue Type: Bug
Components: Table SQL / Gateway, Table SQL / Planner
Affects Versions: 1.18.1
Reporter: Xingcan Cui
We hit the following exceptions a few times when submitting queries to a
session cluster with the Flink SQL gateway. When the same queries were
submitted again, everything was good. There might be a concurrency problem for
the expression generator.
{code:java}
"process.thread.name":"sql-gateway-operation-pool-thread-111","log.logger":"org.apache.flink.table.gateway.service.operation.OperationManager","error.type":"org.apache.flink.table.planner.codegen.CodeGenException","error.message":"Mismatch
of expected output data type 'ARRAY NOT
NULL>' and function's output type 'ARRAY NOT
NULL>'.","error.stack_trace":"org.apache.flink.table.planner.codegen.CodeGenException:
Mismatch of expected output data type 'ARRAY
NOT NULL>' and function's output type 'ARRAY
NOT NULL>'.
at
org.apache.flink.table.planner.codegen.calls.BridgingFunctionGenUtil$.verifyOutputType(BridgingFunctionGenUtil.scala:369)
at
org.apache.flink.table.planner.codegen.calls.BridgingFunctionGenUtil$.verifyFunctionAwareOutputType(BridgingFunctionGenUtil.scala:359)
at
org.apache.flink.table.planner.codegen.calls.BridgingFunctionGenUtil$.generateFunctionAwareCallWithDataType(BridgingFunctionGenUtil.scala:107)
at
org.apache.flink.table.planner.codegen.calls.BridgingFunctionGenUtil$.generateFunctionAwareCall(BridgingFunctionGenUtil.scala:84)
at
org.apache.flink.table.planner.codegen.calls.BridgingSqlFunctionCallGen.generate(BridgingSqlFunctionCallGen.scala:79)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:820)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:481)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:56)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:189)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.$anonfun$visitCall$1(ExprCodeGenerator.scala:478)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at scala.collection.TraversableLike.map(TraversableLike.scala:233)
at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:469)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:56)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:189)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.$anonfun$visitCall$1(ExprCodeGenerator.scala:478)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at scala.collection.TraversableLike.map(TraversableLike.scala:233)
at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:469)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:56)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:189)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateExpression(ExprCodeGenerator.scala:134)
at
org.apache.flink.table.planner.codegen.CalcCodeGenerator$.$anonfun$generateProcessCode$4(CalcCodeGenerator.scala:140)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at scala.collection.TraversableLike.map(TraversableLike.scala:233)
at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
org.apache.flink.table.planner.codegen.CalcCodeGenerator$.produceProjectionCode$1(CalcCodeGenerator.scala:140)
at
[jira] [Created] (FLINK-35485) JobMaster failed with "the job xx has not been finished"
Xingcan Cui created FLINK-35485:
---
Summary: JobMaster failed with "the job xx has not been finished"
Key: FLINK-35485
URL: https://issues.apache.org/jira/browse/FLINK-35485
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.18.1
Reporter: Xingcan Cui
We ran a session cluster on K8s and used Flink SQL gateway to submit queries.
Hit the following rare exception once which caused the job manager to restart.
{code:java}
org.apache.flink.util.FlinkException: JobMaster for job
50d681ae1e8170f77b4341dda6aba9bc failed.
at
org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:1454)
at
org.apache.flink.runtime.dispatcher.Dispatcher.jobManagerRunnerFailed(Dispatcher.java:776)
at
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$runJob$6(Dispatcher.java:698)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown
Source)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown
Source)
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRunAsync$4(PekkoRpcActor.java:451)
at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRunAsync(PekkoRpcActor.java:451)
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:218)
at
org.apache.flink.runtime.rpc.pekko.FencedPekkoRpcActor.handleRpcMessage(FencedPekkoRpcActor.java:85)
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:168)
at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33)
at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29)
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
at
org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547)
at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545)
at org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229)
at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590)
at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557)
at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280)
at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241)
at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253)
at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown
Source)
at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: org.apache.flink.runtime.jobmaster.JobNotFinishedException: The job
(50d681ae1e8170f77b4341dda6aba9bc) has not been finished.
at
org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.closeAsync(DefaultJobMasterServiceProcess.java:157)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.stopJobMasterServiceProcess(JobMasterServiceLeadershipRunner.java:431)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.callIfRunning(JobMasterServiceLeadershipRunner.java:476)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.lambda$stopJobMasterServiceProcessAsync$12(JobMasterServiceLeadershipRunner.java:407)
at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(Unknown
Source)
at java.base/java.util.concurrent.CompletableFuture.thenCompose(Unknown
Source)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.stopJobMasterServiceProcessAsync(JobMasterServiceLeadershipRunner.java:405)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.runIfStateRunning(JobMasterServiceLeadershipRunner.java:463)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.revokeLeadership(JobMasterServiceLeadershipRunner.java:397)
at
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.notifyLeaderContenderOfLeadershipLoss(DefaultLeaderElectionService.java:484)
at java.base/java.util.HashMap.forEach(Unknown Source)
at
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.onRevokeLeadershipInternal(DefaultLeaderElectionService.java:452)
at
[jira] [Created] (FLINK-35484) Flink Document document file had removed but website can access
Zhongqiang Gong created FLINK-35484: --- Summary: Flink Document document file had removed but website can access Key: FLINK-35484 URL: https://issues.apache.org/jira/browse/FLINK-35484 Project: Flink Issue Type: Bug Components: Documentation Reporter: Zhongqiang Gong Flink 1.18 document had remove document about DataSet : issue link https://issues.apache.org/jira/browse/FLINK-32741. But I can still access the link : https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/dataset/formats/avro/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] Release flink-connector-opensearch v1.2.0, release candidate #1
+1 (non-binding) - Built from source code with JDK 1.8 on MaxOS- Run examples locally.- Checked release notes Best, Yuepeng Pan At 2024-05-28 22:53:10, "gongzhongqiang" wrote: >+1(non-binding) > >- Verified signatures and hash sums >- Reviewed the web PR >- Built from source code with JDK 1.8 on Ubuntu 22.04 >- Checked release notes > >Best, >Zhongqiang Gong > > >Sergey Nuyanzin 于2024年5月16日周四 06:03写道： > >> Hi everyone, >> Please review and vote on release candidate #1 for >> flink-connector-opensearch v1.2.0, as follows: >> [ ] +1, Approve the release >> [ ] -1, Do not approve the release (please provide specific comments) >> >> >> The complete staging area is available for your review, which includes: >> * JIRA release notes [1], >> * the official Apache source release to be deployed to dist.apache.org >> [2], >> which are signed with the key with fingerprint >> F7529FAE24811A5C0DF3CA741596BBF0726835D8 [3], >> * all artifacts to be deployed to the Maven Central Repository [4], >> * source code tag v1.2.0-rc1 [5], >> * website pull request listing the new release [6]. >> * CI build of the tag [7]. >> >> The vote will be open for at least 72 hours. It is adopted by majority >> approval, with at least 3 PMC affirmative votes. >> >> Note that this release is for Opensearch v1.x >> >> Thanks, >> Release Manager >> >> [1] https://issues.apache.org/jira/projects/FLINK/versions/12353812 >> [2] >> >> https://dist.apache.org/repos/dist/dev/flink/flink-connector-opensearch-1.2.0-rc1 >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS >> [4] https://repository.apache.org/content/repositories/orgapacheflink-1734 >> [5] >> >> https://github.com/apache/flink-connector-opensearch/releases/tag/v1.2.0-rc1 >> [6] https://github.com/apache/flink-web/pull/740 >> [7] >> >> https://github.com/apache/flink-connector-opensearch/actions/runs/9102334125 >>
[jira] [Created] (FLINK-35483) BatchJobRecoveryTest.testRecoverFromJMFailover produced no output for 900 second
Weijie Guo created FLINK-35483: -- Summary: BatchJobRecoveryTest.testRecoverFromJMFailover produced no output for 900 second Key: FLINK-35483 URL: https://issues.apache.org/jira/browse/FLINK-35483 Project: Flink Issue Type: Bug Components: Build System / CI Affects Versions: 1.20.0 Reporter: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] Flink CDC 3.1.1 Release
+1 Thanks Xiqian. Best, Zhongqiang Gong Xiqian YU 于2024年5月28日周二 19:44写道： > Hi devs, > > I would like to make a proposal about creating a new Flink CDC 3.1 patch > release (3.1.1). It’s been a week since the last CDC version 3.1.0 got > released [1], and since then, 7 tickets have been closed, 4 of them are of > high priority. > > Currently, there are 5 items open at the moment: 1 of them is a blocker, > which stops users from restoring with existed checkpoints after upgrading > [2]. There’s a PR ready and will be merged soon. Other 4 of them have > approved PRs, and will be merged soon [3][4][5][6]. I propose that a patch > version could be released after all pending tickets closed. > > Please reply if there are any unresolved blocking issues you’d like to > include in this release. > > Regards, > Xiqian > > [1] > https://flink.apache.org/2024/05/17/apache-flink-cdc-3.1.0-release-announcement/ > [2] https://issues.apache.org/jira/browse/FLINK-35464 > [3] https://issues.apache.org/jira/browse/FLINK-35149 > [4] https://issues.apache.org/jira/browse/FLINK-35323 > [5] https://issues.apache.org/jira/browse/FLINK-35430 > [6] https://issues.apache.org/jira/browse/FLINK-35447 > >
[jira] [Created] (FLINK-35482) ParquetInt64TimestampReaderTest unit tests fail due to system timezone
Rob Young created FLINK-35482:
-
Summary: ParquetInt64TimestampReaderTest unit tests fail due to
system timezone
Key: FLINK-35482
URL: https://issues.apache.org/jira/browse/FLINK-35482
Project: Flink
Issue Type: Bug
Components: Tests
Environment: linux version: uname -r
6.9.1-arch1-1
checked with:
openjdk version "21" 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
openjdk 17.0.5 2022-10-18
OpenJDK Runtime Environment Temurin-17.0.5+8 (build 17.0.5+8)
openjdk version "11.0.21" 2023-10-17 LTS
OpenJDK Runtime Environment Zulu11.68+17-CA (build 11.0.21+9-LTS)
openjdk version "1.8.0_382"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_382-b05)
Reporter: Rob Young
To reproduce:
{code:java}
export TZ=Pacific/Auckland
./mvnw -Dfast -DskipTests -Dskip.npm=true install -pl :flink-parquet -am
./mvnw test -pl :flink-parquet{code}
The parquet tests fail with:
{code:java}
[ERROR] Failures:
[ERROR] ParquetInt64TimestampReaderTest.testReadInt64TimestampMicros:46
expected:<2021-11-22T1[7]:50:20.000112> but was:<2021-11-22T1[8]:50:20.000112>
[ERROR] ParquetInt64TimestampReaderTest.testReadInt64TimestampMillis:66
expected:<2021-11-22T1[7]:50:20> but was:<2021-11-22T1[8]:50:20>
[ERROR] ParquetInt64TimestampReaderTest.testReadInt64TimestampNanos:78
expected:<2021-11-22T1[7]:50:20.000112233> but
was:<2021-11-22T1[8]:50:20.000112233>{code}
I think this is because the tests convert a LocalDateTime to epoch seconds
using `OffsetDateTime.now().getOffset()` as the offset, but now's offset is
different to what it would be at 2021-11-22T17:50:20.000112 NZST due to
daylight savings.
Instead of using now's offset we could convert the localDateTime to a
zonedDateTime using `localDateTime.atZone(ZoneId.systemDefault())`. If you're
happy with that idea please assign me and I'll make a PR.
Another possible idea would be to set the user.timezone to GMT in the base
argLine to use a consistent timezone for tests, which would be picked up by
surefire and IDEA. But improving the tests feels like a better solution.
Thanks
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
RE: [DISCUSS] FLIP-XXX Add K8S conditions to Flink CRD
Hi Gyula, Thank you for the quick response and confirmation we need a Flip. I am not an expert at K8s, Lajith will answer in more detail. Some questions I had anyway: I assume each of the ResourceLifecycleState do have a corresponding jobReady status. You point out some mistakes in the table, for example that STABLE should be NotReady; thankyou. If we put a reason mentioning the stable state, this would help us understand the jobStatus. I guess the jobReady is one perspective that we know is useful (with corrected mappings from ResourceLifecycleState and with reasons). Can I check that the 2 proposed conditions would also be useful additions? I assume that in your proposal when jobReady is true, then UpgradeCompleted condition would not be present and ClusterReady would always be true? I know conditions do not need to be orthogonal, but I wanted to check what your thoughts are. Kind regards, David. From: Gyula Fóra Date: Wednesday, 29 May 2024 at 15:28 To: dev@flink.apache.org Cc: morh...@apache.org Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Add K8S conditions to Flink CRD Hi David! This change definitely warrants a FLIP even if the code change is not huge, there are quite some implications going forward. Looping in @morh...@apache.org for this discussion. I have some questions / suggestions regarding the condition's meaning and naming. In your proposal you have: - Ready (True/False) -> This condition is intended for resources which are fully ready and operational - Error (True) -> This condition can be used in scenarios where any exception/error during resource reconcile process The problem with the above is that the implementation does not well reflect this. ResourceLifecycleState STABLE/ROLLED_BACK does not actually mean the job is running, it just means that the resource is fully reconciled and it will not be rolled back (so the current pending upgrade is completed). This is mainly a fault of the ResourceLifecycleState as it doesn't capture the job status but one could argue that it was "designed" this way. I think we should probably have more condition types to capture the difference: - JobReady (True/False) -> Flink job is running (Basically job status but with transition time) - ClusterReady (True/False) -> Session / Application cluster is deployed (Basically JM deployment status but with transition time) - UpgradeCompleted (True/False) -> Similar to what you call Ready now which should correspond to the STABLE/ROLLED_BACK states and mostly tracks in-progress CR updates This is my best idea at the moment, not great as it feels a little redundant with the current status fields. But maybe thats not a problem or a way to eliminate the old fields later? I am not so sure of the Error status and what this means in practice. Why do we want to track the last error in 2 places? It's already in the status. What do you think? Gyula On Wed, May 29, 2024 at 3:55 PM David Radley wrote: > Hi, > Thanks Lajith for raising this discussion thread under the Flip title. > > To summarise the concerns from the other discussion thread. > > “ > - I echo Gyula that including some examples and further explanations might > ease reader's work. With the current version, the FLIP is a bit hard to > follow. - Will the usage of Conditions be enabled by default? Or will there > be any disadvantages for Flink users? If Conditions with the same type > already exist in the Status Conditions > > - Do you think we should have clear rules about handling rules for how > these Conditions should be managed, especially when multiple Conditions of > the same type are present? For example, resource has multiple causes for > the same condition (e.g., Error due to network and Error due to I/O). Then, > overriding the old condition with the new one is not the best approach no? > Please correct me if I misunderstood. > “ > > I see the Google doc link has been reformatted to match the Flip template. > > To explicitly answer the questions from Jeyhun and Gyula: > - “Will the usage of Conditions be enabled by default?” Yes, but this is > just making the status content useful, whereas before it was not useful. > - in terms of examples, I am not sure what you would like to see, the > table Lajith provided shows the status for various ResourceLifecycleStates. > How the operator gets into these states is the current behaviour. The > change just shows the appropriate corresponding high level status – that > could be shown on the User Interfaces. > - “will there be any disadvantages for Flink users?” None , there is just > more information in the status, without this it is more difficult to work > out the status of the job. > - Multiple conditions question. The status is showing whether the job is > ready or not, so as long as the last condition is the one that is shown - > all is as expected. I don’t think this needs rules for precedence and the > like. > - The condition’s Reason is going to be more specific. >
Re: [DISCUSS] FLIP-XXX Add K8S conditions to Flink CRD
Hi David! This change definitely warrants a FLIP even if the code change is not huge, there are quite some implications going forward. Looping in @morh...@apache.org for this discussion. I have some questions / suggestions regarding the condition's meaning and naming. In your proposal you have: - Ready (True/False) -> This condition is intended for resources which are fully ready and operational - Error (True) -> This condition can be used in scenarios where any exception/error during resource reconcile process The problem with the above is that the implementation does not well reflect this. ResourceLifecycleState STABLE/ROLLED_BACK does not actually mean the job is running, it just means that the resource is fully reconciled and it will not be rolled back (so the current pending upgrade is completed). This is mainly a fault of the ResourceLifecycleState as it doesn't capture the job status but one could argue that it was "designed" this way. I think we should probably have more condition types to capture the difference: - JobReady (True/False) -> Flink job is running (Basically job status but with transition time) - ClusterReady (True/False) -> Session / Application cluster is deployed (Basically JM deployment status but with transition time) - UpgradeCompleted (True/False) -> Similar to what you call Ready now which should correspond to the STABLE/ROLLED_BACK states and mostly tracks in-progress CR updates This is my best idea at the moment, not great as it feels a little redundant with the current status fields. But maybe thats not a problem or a way to eliminate the old fields later? I am not so sure of the Error status and what this means in practice. Why do we want to track the last error in 2 places? It's already in the status. What do you think? Gyula On Wed, May 29, 2024 at 3:55 PM David Radley wrote: > Hi, > Thanks Lajith for raising this discussion thread under the Flip title. > > To summarise the concerns from the other discussion thread. > > “ > - I echo Gyula that including some examples and further explanations might > ease reader's work. With the current version, the FLIP is a bit hard to > follow. - Will the usage of Conditions be enabled by default? Or will there > be any disadvantages for Flink users? If Conditions with the same type > already exist in the Status Conditions > > - Do you think we should have clear rules about handling rules for how > these Conditions should be managed, especially when multiple Conditions of > the same type are present? For example, resource has multiple causes for > the same condition (e.g., Error due to network and Error due to I/O). Then, > overriding the old condition with the new one is not the best approach no? > Please correct me if I misunderstood. > “ > > I see the Google doc link has been reformatted to match the Flip template. > > To explicitly answer the questions from Jeyhun and Gyula: > - “Will the usage of Conditions be enabled by default?” Yes, but this is > just making the status content useful, whereas before it was not useful. > - in terms of examples, I am not sure what you would like to see, the > table Lajith provided shows the status for various ResourceLifecycleStates. > How the operator gets into these states is the current behaviour. The > change just shows the appropriate corresponding high level status – that > could be shown on the User Interfaces. > - “will there be any disadvantages for Flink users?” None , there is just > more information in the status, without this it is more difficult to work > out the status of the job. > - Multiple conditions question. The status is showing whether the job is > ready or not, so as long as the last condition is the one that is shown - > all is as expected. I don’t think this needs rules for precedence and the > like. > - The condition’s Reason is going to be more specific. > > Gyula and Jeyhun, is the google doc clear enough for you now? Do you feel > you feedback has been addressed? Lajith and I are happy to provide more > details. > > I wonder whether this change is big enough to warrant a Flip, as it is so > small. We could do this in an issue. WDYT? > > Kind regards, David. > > > From: Lajith Koova > Date: Wednesday, 29 May 2024 at 13:41 > To: dev@flink.apache.org > Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Add K8S conditions to Flink CRD > Hello , > > > Discussion thread here: > https://lists.apache.org/thread/dvy8w17pyjv68c3t962w49frl9odoz4z to > discuss a proposal to add Conditions field in the CR status of Flink > Deployment and FlinkSessionJob. > > > Note : Starting this new thread as discussion thread title has been > modified to follow the FLIP process. > > > Thank you. > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU >
Re: [DISCUSS] FLIP-XXX Add K8S conditions to Flink CRD
Hi, Thanks Lajith for raising this discussion thread under the Flip title. To summarise the concerns from the other discussion thread. “ - I echo Gyula that including some examples and further explanations might ease reader's work. With the current version, the FLIP is a bit hard to follow. - Will the usage of Conditions be enabled by default? Or will there be any disadvantages for Flink users? If Conditions with the same type already exist in the Status Conditions - Do you think we should have clear rules about handling rules for how these Conditions should be managed, especially when multiple Conditions of the same type are present? For example, resource has multiple causes for the same condition (e.g., Error due to network and Error due to I/O). Then, overriding the old condition with the new one is not the best approach no? Please correct me if I misunderstood. “ I see the Google doc link has been reformatted to match the Flip template. To explicitly answer the questions from Jeyhun and Gyula: - “Will the usage of Conditions be enabled by default?” Yes, but this is just making the status content useful, whereas before it was not useful. - in terms of examples, I am not sure what you would like to see, the table Lajith provided shows the status for various ResourceLifecycleStates. How the operator gets into these states is the current behaviour. The change just shows the appropriate corresponding high level status – that could be shown on the User Interfaces. - “will there be any disadvantages for Flink users?” None , there is just more information in the status, without this it is more difficult to work out the status of the job. - Multiple conditions question. The status is showing whether the job is ready or not, so as long as the last condition is the one that is shown - all is as expected. I don’t think this needs rules for precedence and the like. - The condition’s Reason is going to be more specific. Gyula and Jeyhun, is the google doc clear enough for you now? Do you feel you feedback has been addressed? Lajith and I are happy to provide more details. I wonder whether this change is big enough to warrant a Flip, as it is so small. We could do this in an issue. WDYT? Kind regards, David. From: Lajith Koova Date: Wednesday, 29 May 2024 at 13:41 To: dev@flink.apache.org Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Add K8S conditions to Flink CRD Hello , Discussion thread here: https://lists.apache.org/thread/dvy8w17pyjv68c3t962w49frl9odoz4z to discuss a proposal to add Conditions field in the CR status of Flink Deployment and FlinkSessionJob. Note : Starting this new thread as discussion thread title has been modified to follow the FLIP process. Thank you. Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
Re: [DISCUSS] Add Flink CDC Channel to Apache Flink Slack Workspace
Thanks Zhongqiang, +1 for sure. Best Regards Ahmed Hamdy On Wed, 29 May 2024 at 13:48, ConradJam wrote: > +1 best > > Hang Ruan 于2024年5月29日周三 11:28写道： > > > Hi, zhongqiang. > > > > Thanks for the proposal. +1 for it. > > > > Best, > > Hang > > > > Leonard Xu 于2024年5月28日周二 11:58写道： > > > > > > > > Thanks Zhongqiang for the proposal, we need the Channel and I should > have > > > been created it but not yet, +1 from my side. > > > > > > Best, > > > Leonard > > > > > > > 2024年5月28日 上午11:54，gongzhongqiang 写道： > > > > > > > > Hi devs, > > > > > > > > I would like to propose adding a dedicated Flink CDC channel to the > > > Apache > > > > Flink Slack workspace. > > > > > > > > Creating a channel focused on Flink CDC will help community members > > > easily > > > > find previous discussions > > > > and target new discussions and questions to the correct place. Flink > > CDC > > > is > > > > a sufficiently distinct component > > > > within the Apache Flink ecosystem, and having a dedicated channel > will > > > make > > > > it viable and useful for > > > > those specifically working with or interested in this technology. > > > > > > > > Looking forward to your feedback and support on this proposal. > > > > > > > > > > > > Best, > > > > Zhongqiang Gong > > > > > > > > > > > -- > Best > > ConradJam >
[jira] [Created] (FLINK-35481) Add HISTOGRAM function in SQL & Table API
Ufuk Celebi created FLINK-35481: --- Summary: Add HISTOGRAM function in SQL & Table API Key: FLINK-35481 URL: https://issues.apache.org/jira/browse/FLINK-35481 Project: Flink Issue Type: Improvement Components: Table SQL / API Reporter: Ufuk Celebi Fix For: 1.20.0 Consider adding a HISTOGRAM aggregate function similar to ksqlDB (https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/aggregate-functions/#histogram). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35480) Add FIELD function in SQL & Table API
Ufuk Celebi created FLINK-35480:
---
Summary: Add FIELD function in SQL & Table API
Key: FLINK-35480
URL: https://issues.apache.org/jira/browse/FLINK-35480
Project: Flink
Issue Type: Improvement
Components: Table SQL / API
Reporter: Ufuk Celebi
Fix For: 1.20.0
Add support for the {{FIELD}} function to return the position of {{str}} in
{{{}args{}}}, or 0 if not found.
*References*
* ksqlDB:
[https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/scalar-functions/#field]
* MySQL:
https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_elt
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [DISCUSS] Add Flink CDC Channel to Apache Flink Slack Workspace
+1 best Hang Ruan 于2024年5月29日周三 11:28写道： > Hi, zhongqiang. > > Thanks for the proposal. +1 for it. > > Best, > Hang > > Leonard Xu 于2024年5月28日周二 11:58写道： > > > > > Thanks Zhongqiang for the proposal, we need the Channel and I should have > > been created it but not yet, +1 from my side. > > > > Best, > > Leonard > > > > > 2024年5月28日 上午11:54，gongzhongqiang 写道： > > > > > > Hi devs, > > > > > > I would like to propose adding a dedicated Flink CDC channel to the > > Apache > > > Flink Slack workspace. > > > > > > Creating a channel focused on Flink CDC will help community members > > easily > > > find previous discussions > > > and target new discussions and questions to the correct place. Flink > CDC > > is > > > a sufficiently distinct component > > > within the Apache Flink ecosystem, and having a dedicated channel will > > make > > > it viable and useful for > > > those specifically working with or interested in this technology. > > > > > > Looking forward to your feedback and support on this proposal. > > > > > > > > > Best, > > > Zhongqiang Gong > > > > > -- Best ConradJam
[DISCUSS] FLIP-XXX Add K8S conditions to Flink CRD
Hello , Discussion thread here: https://lists.apache.org/thread/dvy8w17pyjv68c3t962w49frl9odoz4z to discuss a proposal to add Conditions field in the CR status of Flink Deployment and FlinkSessionJob. Note : Starting this new thread as discussion thread title has been modified to follow the FLIP process. Thank you.
[jira] [Created] (FLINK-35479) Add end-to-end test for materialized table
Feng Jin created FLINK-35479: Summary: Add end-to-end test for materialized table Key: FLINK-35479 URL: https://issues.apache.org/jira/browse/FLINK-35479 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Gateway, Tests Reporter: Feng Jin Fix For: 1.20.0 Add end-to-end test cases related to materialized tables, including the processes of dropping, refreshing, and dropping materialized tables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0
Hi Ferenc, Thanks for the FLIP, +1 from me for the proposal. I think these changes would be a great solution to all the confusion that comes from these two action parameters. Best regards, Mate Ferenc Csaky ezt írta (időpont: 2024. máj. 28., K, 16:13): > Thank you Xintong for your input. > > I prepared a FLIP for this change [1], looking forward for any > other opinions. > > Thanks, > Ferenc > > [1] > https://docs.google.com/document/d/1EX74rFp9bMKdfoGkz1ASOM6Ibw32rRxIadX72zs2zoY/edit?usp=sharing > > > > On Friday, 17 May 2024 at 07:04, Xintong Song > wrote: > > > > > > > AFAIK, the main purpose of having `run-application` was to make sure > > the user is aware that application mode is used, which executes the main > > method of the user program in JM rather than in client. This was > important > > at the time application mode was first introduced, but maybe not that > > important anymore, given that per-job mode is deprecated and likely > removed > > in 2.0. Therefore, +1 for the proposal. > > > > Best, > > > > Xintong > > > > > > > > On Thu, May 16, 2024 at 11:35 PM Ferenc Csaky ferenc.cs...@pm.me.invalid > > > > wrote: > > > > > Hello devs, > > > > > > I saw quite some examples when customers were confused about run, and > run- > > > application in the Flink CLI and I was wondering about the necessity of > > > deploying > > > Application Mode (AM) jobs with a different command, than Session and > > > Per-Job mode jobs. > > > > > > I can see a point that YarnDeploymentTarget [1] and > > > KubernetesDeploymentTarget > > > [2] are part of their own maven modules and not known in flink-clients, > > > so the > > > deployment mode validations are happening during cluster deployment in > > > their specific > > > ClusterDescriptor implementation [3]. Although these are implementation > > > details that > > > IMO should not define user-facing APIs. > > > > > > The command line setup is the same for both run and run-application, so > > > I think there > > > is a quite simple way to achieve a unified flink run experience, but I > > > might missed > > > something so I would appreciate any inputs on this topic. > > > > > > Based on my assumptions I think it would be possible to deprecate the > run- > > > application in Flink 1.20 and remove it completely in Flink 2.0. I > > > already put together a > > > PoC [4], and I was able to deploy AM jobs like this: > > > > > > flink run --target kubernetes-application ... > > > > > > If others also agree with this, I would be happy to open a FLIP. WDYT? > > > > > > Thanks, > > > Ferenc > > > > > > [1] > > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java > > > [2] > > > > https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java > > > [3] > > > > https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206 > > > [4] > > > > https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0 >
RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi Danny, Thank you for your feedback on this. I agree that using maps has pros and cons. The maps are flexible, but do require the sender and receiver to know what is in the map. When you say “That sounds like it would fit in better, I assume we cannot just take that approach?” The motivation behind this Flip is to support the headers which is the usual way that Apicurio runs. We will support the “schema id in the payload” as well. I agree with you when you say “ I am not 100% happy with the solution but I cannot offer a better option.” – this is a pragmatic way we have found to solve this issue. I am open to any suggestions to improve this as well. If we are going with the maps design (which is the best we have at the moment) ; it would be good to have the Flink core changes in base Flink version 2.0 as this would mean we do not need to use reflection in a Flink Kafka version 2 connector to work out if the runtime Flink has the new methods. At this stage we only have one committer (yourself) backing this. Do you know of other 2 committers who would support this Flip? Kind regards, David. From: Danny Cranmer Date: Friday, 24 May 2024 at 19:32 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format Hello, > I am curious what you mean by abused. I just meant we will end up adding more and more fields to this map over time, and it may be hard to undo. > For Apicurio it can be sent at the start of the payload like Confluent Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema id, at the start of the payload. Apicurio clients and SerDe libraries can be configured to not put the schema id in the headers in which case there is a magic byte followed by an 8 byte schema at the start of the payload. In the deserialization case, we would not need to look at the headers – though the encoding is also in the headers. That sounds like it would fit in better, I assume we cannot just take that approach? Thanks for the discussion. I am not 100% happy with the solution but I cannot offer a better option. I would be interested to hear if others have any suggestions. Playing devil's advocate against myself, we pass maps around to configure connectors so it is not too far away from that. Thanks, Danny On Fri, May 24, 2024 at 2:23 PM David Radley wrote: > Hi Danny, > No worries, thanks for replying. I have working prototype code that is > being reviewed. It needs some cleaning up and more complete testing before > it is ready, but will give you the general idea [1][2] to help to assess > this approach. > > > I am curious what you mean by abused. I guess the approaches are between > generic map, mechanism vs a more particular more granular things being > passed that might be used by another connector. > > Your first question: > “how would this work if the schema ID is not in the Kafka headers, as > hinted to in the FLIP "usually the global ID in a Kafka header"? > > For Apicurio it can be sent at the start of the payload like Confluent > Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema > id, at the start of the payload. Apicurio clients and SerDe libraries can > be configured to not put the schema id in the headers in which case there > is a magic byte followed by an 8 byte schema at the start of the payload. > In the deserialization case, we would not need to look at the headers – > though the encoding is also in the headers. > > Your second question: > “I am wondering if there are any other instances where the source would be > aware of the schema ID and pass it through in this way? > ” > The examples I can think of are: > - Avro can send the complete schema in a header, this is not recommended > but in theory fits the need for a message payload to require something else > to get the structure. > - I see [2] that Apicurio Protobuf uses headers. > - it might be that other message queuing projects like Rabbit MQ would > need this to be able to support Apicurio Avro & protobuf. > > Kind regards, David, > > > > > [1] https://github.com/apache/flink/pull/24715 > [2] https://github.com/apache/flink-connector-kafka/pull/99 > [3] > https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry > > > > > > > > > > > > > From: Danny Cranmer > Date: Friday, 24 May 2024 at 12:22 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format > Hello, > > Apologies, I am on vacation and have limited access to email. > > I can see the logic here and why you ended up where you did. I can also see > there are other useful metadata fields that we might want to pass through, > which might result in this Map being abused (Kafka Topic, Kinesis Shard, > etc). > > I have a follow up question, how would this work if the schema ID is not in > the Kafka headers, as hinted to in the FLIP "usually the global ID in a
Re: [DISCUSS] Flink CDC 3.1.1 Release
Thanks Xiqian, It is good idea thanks, +1 Best, Muhammet On 2024-05-28 11:43, Xiqian YU wrote: Hi devs, I would like to make a proposal about creating a new Flink CDC 3.1 patch release (3.1.1). It’s been a week since the last CDC version 3.1.0 got released [1], and since then, 7 tickets have been closed, 4 of them are of high priority. Currently, there are 5 items open at the moment: 1 of them is a blocker, which stops users from restoring with existed checkpoints after upgrading [2]. There’s a PR ready and will be merged soon. Other 4 of them have approved PRs, and will be merged soon [3][4][5][6]. I propose that a patch version could be released after all pending tickets closed. Please reply if there are any unresolved blocking issues you’d like to include in this release. Regards, Xiqian [1] https://flink.apache.org/2024/05/17/apache-flink-cdc-3.1.0-release-announcement/ [2] https://issues.apache.org/jira/browse/FLINK-35464 [3] https://issues.apache.org/jira/browse/FLINK-35149 [4] https://issues.apache.org/jira/browse/FLINK-35323 [5] https://issues.apache.org/jira/browse/FLINK-35430 [6] https://issues.apache.org/jira/browse/FLINK-35447
[DISCUSS] Flink CEP: to add processing of unmatched events
Hello. Early I asked how can I get access to unmatched events in CEP Pattern ( https://lists.apache.org/thread/p7n507jvm5hw0xmpoh0lcf87gf3yk18p). Unfortunately Biao Geng answered me, that "that currently there is no such API to access the middle NFA state". I have reviewed the realization of flink CEP library and got a proposal. I added new ValueState, named unmatchedEvents, in CEPOperator ( https://github.com/apache/flink/blob/master/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/operator/CepOperator.java ). Also I added new interface UnmatchedEventsHandler, like TimedOutPartialMatchHandler ( https://github.com/apache/flink/blob/master/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/functions/TimedOutPartialMatchHandler.java ). I put all events in umatchedEvents in bufferEvent(IN event, long currentTime) method. When CEPOperator starts processing match (or timeout match), it also deletes matching events from unmatchedEvents state. Then CEPOperator calls UnmatchedEventsHadler.processEvents(List events, Context ctx) for processing inmatchedEvents.value(). You can look on realization of my proposal in my github fork https://github.com/A-Kinski/flink/pull/1/files
